Robots.txt Usage Distribution in the Top 10k Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	3,650	36.5
GPTBot Disallow	1,526	15.26
Some Robots Disallow	1,505	15.05
Googlebot Disallow	1,468	14.68
Common Crawl Bot Disallow	1,178	11.78
AppleBot Disallow	1,163	11.63
Anthropic Claude Bot Disallow	1,083	10.83
Ahrefs Bot Disallow	1,033	10.33
ByteDance Bytespider Disallow	873	8.73
Google AdsBot Disallow	867	8.67
Yandex Bot Disallow	842	8.42
Google Bard Disallow	785	7.85
Facebook Bot Disallow	780	7.8
Perplexity	780	7.8
Semrush Bot Disallow	757	7.57
Webz Omgili Disallow	729	7.29
Cohere AI Disallow	710	7.1
Apple AI Bot Disallow	695	6.95
Microsoft BingBot Disallow	668	6.68
Baidu Bot Disallow	616	6.16
Majestic Bot Disallow	608	6.08
Diffbot Disallow	534	5.34
Alexa Bot Disallow	440	4.4
Yahoo Slurp Disallow	380	3.8
Google Media Partners Disallow	372	3.72
Microsoft MSNBot Disallow	328	3.28
Sogou Bot Disallow	312	3.12
Google AdsBot Mobile Disallow	307	3.07
Google Image Bot Disallow	277	2.77
YOU Disallow	268	2.68
Exalead Bot Disallow	207	2.07
Google News Bot Disallow	196	1.96
Soso Bot Disallow	174	1.74
Google AdsBot Mobile Apps Disallow	169	1.69
DuckDuckGo Bot Disallow	152	1.52
360 Bot Disallow	151	1.51
Google Video Bot Disallow	117	1.17
Pipl Disallow	82	0.82
Google API Bot Disallow	79	0.79
Google Favicon Disallow	75	0.75
Google DuplexWeb Disallow	71	0.71
BuiltWithBot Disallow	63	0.63
LLMS Text	37	0.37
Youdao Bot Disallow	27	0.27
Google FeedFetcher Disallow	4	0.04
Google Read Aloud Disallow	2	0.02
Google Web Light Disallow	1	0.01
Google BIG Bench Disallow	1	0.01