Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	157,014	15.7
Ahrefs Bot Disallow	61,428	6.14
Googlebot Disallow	60,356	6.04
Yandex Bot Disallow	49,180	4.92
Google AdsBot Disallow	41,609	4.16
Some Robots Disallow	38,426	3.84
Baidu Bot Disallow	36,357	3.64
Semrush Bot Disallow	30,953	3.1
Majestic Bot Disallow	30,757	3.08
GPTBot Disallow	28,118	2.81
Microsoft BingBot Disallow	27,950	2.8
Google Media Partners Disallow	24,048	2.4
Yahoo Slurp Disallow	23,038	2.3
Alexa Bot Disallow	22,667	2.27
Common Crawl Bot Disallow	20,825	2.08
Sogou Bot Disallow	17,857	1.79
Anthropic Claude Bot Disallow	15,493	1.55
Soso Bot Disallow	15,159	1.52
Google Bard Disallow	13,210	1.32
Microsoft MSNBot Disallow	13,084	1.31
AppleBot Disallow	11,912	1.19
Google AdsBot Mobile Disallow	11,826	1.18
Google Image Bot Disallow	10,692	1.07
Facebook Bot Disallow	9,918	0.99
Exalead Bot Disallow	8,981	0.9
ByteDance Bytespider Disallow	8,973	0.9
Perplexity	7,792	0.78
Google AdsBot Mobile Apps Disallow	7,768	0.78
Google News Bot Disallow	6,874	0.69
Cohere AI Disallow	6,496	0.65
Webz Omgili Disallow	6,388	0.64
Apple AI Bot Disallow	6,367	0.64
DuckDuckGo Bot Disallow	5,896	0.59
Google Video Bot Disallow	5,388	0.54
360 Bot Disallow	5,261	0.53
Google Favicon Disallow	4,755	0.48
Google API Bot Disallow	4,660	0.47
Google DuplexWeb Disallow	4,553	0.46
Diffbot Disallow	2,786	0.28
BuiltWithBot Disallow	2,433	0.24
Pipl Disallow	2,062	0.21
YOU Disallow	1,784	0.18
Youdao Bot Disallow	933	0.09
Google FeedFetcher Disallow	146	0.01
Google Read Aloud Disallow	49	0
Google Web Light Disallow	16	0
Google BIG Bench Disallow	4	0