Robots.txt Usage Distribution in Spain

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	76,462	9.21
Googlebot Disallow	75,617	9.11
Google AdsBot Mobile Disallow	58,920	7.1
Baidu Bot Disallow	49,444	5.96
Some Robots Disallow	40,428	4.87
GPTBot Disallow	37,061	4.47
Google AdsBot Mobile Apps Disallow	34,285	4.13
Yandex Bot Disallow	34,151	4.12
Yahoo Slurp Disallow	34,077	4.11
Common Crawl Bot Disallow	32,147	3.87
Robots Disallow	31,094	3.75
Ahrefs Bot Disallow	30,616	3.69
Alexa Bot Disallow	28,724	3.46
Microsoft BingBot Disallow	26,286	3.17
Sogou Bot Disallow	25,885	3.12
Soso Bot Disallow	24,557	2.96
Anthropic Claude Bot Disallow	19,804	2.39
Majestic Bot Disallow	19,748	2.38
Google Bard Disallow	19,073	2.3
Facebook Bot Disallow	18,708	2.25
Cohere AI Disallow	18,253	2.2
AppleBot Disallow	15,593	1.88
Apple AI Bot Disallow	14,185	1.71
Perplexity	13,920	1.68
Semrush Bot Disallow	12,797	1.54
Microsoft MSNBot Disallow	12,715	1.53
Exalead Bot Disallow	5,350	0.64
Google Image Bot Disallow	3,789	0.46
Google Media Partners Disallow	2,741 ^*	0.33
360 Bot Disallow	2,555	0.31
ByteDance Bytespider Disallow	1,838	0.22
Youdao Bot Disallow	1,323	0.16
BuiltWithBot Disallow	1,283	0.15
Google News Bot Disallow	1,247	0.15
Google Video Bot Disallow	1,164	0.14
Webz Omgili Disallow	1,067	0.13
YOU Disallow	619	0.07
DuckDuckGo Bot Disallow	574	0.07
Google Favicon Disallow	440	0.05
Google API Bot Disallow	395	0.05
Google DuplexWeb Disallow	391	0.05
Pipl Disallow	243	0.03
Diffbot Disallow	195	0.02
Google FeedFetcher Disallow	9	0
Google Read Aloud Disallow	5	0