Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet

Technology	Websites	%
Some Robots Disallow	39,138,226	18.6
Googlebot Disallow	17,828,157	8.47
Baidu Bot Disallow	13,097,374	6.22
Sogou Bot Disallow	12,199,348	5.8
Google Media Partners Disallow	11,942,797	5.67
Microsoft BingBot Disallow	11,728,319	5.57
Yandex Bot Disallow	11,686,141	5.55
Yahoo Slurp Disallow	10,921,546	5.19
Google AdsBot Disallow	9,790,845	4.65
Robots Disallow	8,533,496	4.05
Google AdsBot Mobile Disallow	7,817,180	3.71
Alexa Bot Disallow	5,217,515	2.48
Soso Bot Disallow	4,790,953	2.28
Ahrefs Bot Disallow	3,952,255	1.88
GPTBot Disallow	3,819,558	1.81
AppleBot Disallow	3,757,288	1.79
Common Crawl Bot Disallow	3,722,656	1.77
Anthropic Claude Bot Disallow	3,591,156	1.71
Google Bard Disallow	3,430,304	1.63
Facebook Bot Disallow	3,384,465	1.61
Apple AI Bot Disallow	3,293,219	1.56
Cohere AI Disallow	3,282,715	1.56
Perplexity	3,229,070	1.53
Majestic Bot Disallow	2,223,915	1.06
Google AdsBot Mobile Apps Disallow	2,080,353	0.99
Semrush Bot Disallow	2,069,960	0.98
ByteDance Bytespider Disallow	673,782	0.32
Exalead Bot Disallow	455,319	0.22
Microsoft MSNBot Disallow	437,707	0.21
Webz Omgili Disallow	392,093	0.19
Google Image Bot Disallow	347,468	0.17
DuckDuckGo Bot Disallow	275,546	0.13
Diffbot Disallow	268,227	0.13
360 Bot Disallow	255,709	0.12
YOU Disallow	164,979	0.08
BuiltWithBot Disallow	115,195	0.05
Google Video Bot Disallow	91,151	0.04
Youdao Bot Disallow	90,415	0.04
Google News Bot Disallow	90,384	0.04
Google Favicon Disallow	72,713	0.03
LLMS Text	49,070	0.02
Google API Bot Disallow	46,654	0.02
Google DuplexWeb Disallow	41,438	0.02
Pipl Disallow	36,337	0.02
Google Read Aloud Disallow	5,596	0
Google FeedFetcher Disallow	4,158	0
Anthropic Claude User Bot Disallow	3,763	0
Anthropic Claude Search Bot Disallow	3,666	0
Google Web Light Disallow	743	0
Google BIG Bench Disallow	41	0