Robots.txt Usage Distribution in Brazil

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	151,330	14.82
Googlebot Disallow	142,065	13.91
Google AdsBot Mobile Disallow	119,056	11.66
Ahrefs Bot Disallow	54,733	5.36
Robots Disallow	47,076 ^*	4.61
Anthropic Claude Bot Disallow	43,872	4.3
Baidu Bot Disallow	39,637	3.88
Majestic Bot Disallow	38,922	3.81
Yahoo Slurp Disallow	37,111	3.63
Yandex Bot Disallow	35,450	3.47
Exalead Bot Disallow	28,523	2.79
GPTBot Disallow	27,106	2.65
Common Crawl Bot Disallow	25,908	2.54
Perplexity	23,202	2.27
Cohere AI Disallow	22,932	2.25
Some Robots Disallow	22,122	2.17
ByteDance Bytespider Disallow	21,111	2.07
Webz Omgili Disallow	20,226	1.98
Diffbot Disallow	19,817	1.94
Sogou Bot Disallow	13,128	1.29
Semrush Bot Disallow	12,478	1.22
Alexa Bot Disallow	10,740	1.05
Microsoft BingBot Disallow	10,603	1.04
Soso Bot Disallow	8,432	0.83
Google Image Bot Disallow	8,381	0.82
Google Media Partners Disallow	6,031 ^*	0.59
AppleBot Disallow	5,446	0.53
Google Bard Disallow	4,516	0.44
Facebook Bot Disallow	4,179	0.41
Apple AI Bot Disallow	3,927	0.38
Google AdsBot Mobile Apps Disallow	2,742	0.27
Microsoft MSNBot Disallow	2,274	0.22
360 Bot Disallow	1,937	0.19
DuckDuckGo Bot Disallow	1,260	0.12
BuiltWithBot Disallow	1,202	0.12
YOU Disallow	872	0.09
Google FeedFetcher Disallow	735	0.07
Youdao Bot Disallow	571	0.06
Pipl Disallow	322	0.03
Google News Bot Disallow	313	0.03
Google Video Bot Disallow	263	0.03
Google Favicon Disallow	189	0.02
Google API Bot Disallow	180	0.02
Google DuplexWeb Disallow	139	0.01
LLMS Text	24	0
Google Read Aloud Disallow	16	0
Google Web Light Disallow	4	0