Robots.txt Usage Distribution in the Top 100k Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	30,537	30.54
Googlebot Disallow	10,367	10.37
Some Robots Disallow	9,473	9.47
Ahrefs Bot Disallow	9,167	9.17
GPTBot Disallow	8,682	8.68
Yandex Bot Disallow	6,882	6.88
Semrush Bot Disallow	6,383	6.38
AppleBot Disallow	6,333	6.33
Common Crawl Bot Disallow	6,141	6.14
Google AdsBot Disallow	5,750	5.75
Anthropic Claude Bot Disallow	5,576	5.58
Majestic Bot Disallow	5,462	5.46
Baidu Bot Disallow	4,963	4.96
Microsoft BingBot Disallow	4,838	4.84
Google Bard Disallow	4,521	4.52
ByteDance Bytespider Disallow	4,042	4.04
Facebook Bot Disallow	3,744	3.74
Perplexity	3,519	3.52
Webz Omgili Disallow	3,223	3.22
Apple AI Bot Disallow	3,096	3.1
Cohere AI Disallow	3,066	3.07
Alexa Bot Disallow	2,996	3
Yahoo Slurp Disallow	2,978	2.98
Google Media Partners Disallow	2,786	2.79
Microsoft MSNBot Disallow	2,554	2.55
Sogou Bot Disallow	2,350	2.35
Google Image Bot Disallow	2,238	2.24
Google AdsBot Mobile Disallow	2,187	2.19
Diffbot Disallow	2,067	2.07
Exalead Bot Disallow	1,708	1.71
Google News Bot Disallow	1,543	1.54
Soso Bot Disallow	1,384	1.38
YOU Disallow	1,384	1.38
Google AdsBot Mobile Apps Disallow	1,332	1.33
DuckDuckGo Bot Disallow	1,246	1.25
360 Bot Disallow	1,075	1.08
Google Video Bot Disallow	1,042	1.04
Google API Bot Disallow	820	0.82
Google Favicon Disallow	817	0.82
Google DuplexWeb Disallow	776	0.78
Pipl Disallow	602	0.6
BuiltWithBot Disallow	562	0.56
Youdao Bot Disallow	202	0.2
LLMS Text	158	0.16
Google FeedFetcher Disallow	31	0.03
Google Read Aloud Disallow	23	0.02
Google Web Light Disallow	8	0.01
Google BIG Bench Disallow	3	0