Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet

Technology	Websites	%
Some Robots Disallow	39,138,226	18.64
Googlebot Disallow	18,777,473	8.94
Baidu Bot Disallow	13,233,594	6.3
Sogou Bot Disallow	12,367,809	5.89
Google Media Partners Disallow	11,590,120	5.52
Microsoft BingBot Disallow	11,275,970	5.37
Yandex Bot Disallow	11,233,892	5.35
Google AdsBot Disallow	10,701,849	5.1
Yahoo Slurp Disallow	10,485,295	4.99
Google AdsBot Mobile Disallow	8,344,481	3.97
Robots Disallow	8,014,950	3.82
Alexa Bot Disallow	5,029,184	2.4
Soso Bot Disallow	4,615,935	2.2
Ahrefs Bot Disallow	3,916,812	1.87
GPTBot Disallow	3,751,266	1.79
AppleBot Disallow	3,698,915	1.76
Common Crawl Bot Disallow	3,681,953	1.75
Anthropic Claude Bot Disallow	3,551,800	1.69
Google Bard Disallow	3,391,288	1.62
Facebook Bot Disallow	3,347,039	1.59
Apple AI Bot Disallow	3,256,425	1.55
Cohere AI Disallow	3,248,417	1.55
Perplexity	3,192,806	1.52
Majestic Bot Disallow	2,182,652	1.04
Google AdsBot Mobile Apps Disallow	2,080,357	0.99
Semrush Bot Disallow	2,026,465	0.97
ByteDance Bytespider Disallow	665,525	0.32
Exalead Bot Disallow	446,285	0.21
Microsoft MSNBot Disallow	428,704	0.2
Webz Omgili Disallow	386,711	0.18
Google Image Bot Disallow	339,010	0.16
DuckDuckGo Bot Disallow	268,468	0.13
Diffbot Disallow	262,940	0.13
360 Bot Disallow	253,257	0.12
YOU Disallow	160,638	0.08
BuiltWithBot Disallow	114,753	0.05
Google Video Bot Disallow	90,091	0.04
Google News Bot Disallow	89,417	0.04
Youdao Bot Disallow	87,418	0.04
Google Favicon Disallow	72,144	0.03
Google API Bot Disallow	46,170	0.02
Google DuplexWeb Disallow	41,014	0.02
Pipl Disallow	33,922	0.02
Google Read Aloud Disallow	5,456	0
LLMS Text	5,454	0
Google FeedFetcher Disallow	4,106	0
Anthropic Claude User Bot Disallow	1,653	0
Anthropic Claude Search Bot Disallow	1,632	0
Google Web Light Disallow	737	0
Google BIG Bench Disallow	39	0