Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	163,830	16.38
Ahrefs Bot Disallow	71,886	7.19
Googlebot Disallow	58,251	5.83
Google AdsBot Disallow	50,603	5.06
Yandex Bot Disallow	46,822	4.68
Some Robots Disallow	41,316	4.13
GPTBot Disallow	35,308	3.53
Semrush Bot Disallow	35,058	3.51
Majestic Bot Disallow	34,277	3.43
Baidu Bot Disallow	33,147	3.31
Microsoft BingBot Disallow	27,979	2.8
Common Crawl Bot Disallow	26,065	2.61
Anthropic Claude Bot Disallow	23,296	2.33
AppleBot Disallow	22,245	2.22
Yahoo Slurp Disallow	21,138	2.11
Sogou Bot Disallow	19,525	1.95
Google Media Partners Disallow	19,432	1.94
Alexa Bot Disallow	18,774	1.88
Google Bard Disallow	17,232	1.72
Facebook Bot Disallow	15,487	1.55
ByteDance Bytespider Disallow	14,446	1.44
Microsoft MSNBot Disallow	14,033	1.4
Perplexity	13,176	1.32
Google AdsBot Mobile Disallow	13,012	1.3
Google Image Bot Disallow	11,685	1.17
Cohere AI Disallow	11,356	1.14
Apple AI Bot Disallow	11,261	1.13
Webz Omgili Disallow	10,859	1.09
Soso Bot Disallow	10,809	1.08
Exalead Bot Disallow	9,944	0.99
Google AdsBot Mobile Apps Disallow	7,843	0.78
Diffbot Disallow	7,265	0.73
Google News Bot Disallow	7,091	0.71
DuckDuckGo Bot Disallow	6,521	0.65
360 Bot Disallow	6,376	0.64
Google Video Bot Disallow	5,678	0.57
YOU Disallow	5,573	0.56
Google Favicon Disallow	4,959	0.5
Google API Bot Disallow	4,910	0.49
Google DuplexWeb Disallow	4,765	0.48
Pipl Disallow	3,320	0.33
BuiltWithBot Disallow	2,791	0.28
Youdao Bot Disallow	906	0.09
LLMS Text	487	0.05
Google FeedFetcher Disallow	169	0.02
Google Read Aloud Disallow	91	0.01
Google Web Light Disallow	31	0
Google BIG Bench Disallow	4	0