Robots.txt Usage Distribution on the Entire Internet

Top In Robots.txt Usage Distribution on the Entire Internet

Technology	Websites	%
Some Robots Disallow	39,138,226	19.22
Googlebot Disallow	14,319,434	7.03
Baidu Bot Disallow	13,411,818	6.59
Sogou Bot Disallow	12,557,537	6.17
Google Media Partners Disallow	12,272,964	6.03
Yandex Bot Disallow	10,674,796	5.24
Microsoft BingBot Disallow	10,639,648	5.22
Yahoo Slurp Disallow	9,901,934	4.86
Google AdsBot Disallow	8,717,405	4.28
Robots Disallow	7,971,308	3.91
Google AdsBot Mobile Disallow	7,269,025	3.57
Alexa Bot Disallow	5,502,757	15.52
Soso Bot Disallow	5,123,375	2.52
Ahrefs Bot Disallow	4,320,735	2.12
GPTBot Disallow	3,819,787	10.78
AppleBot Disallow	3,734,152	1.83
Common Crawl Bot Disallow	3,689,029	10.41
Anthropic Claude Bot Disallow	3,542,190	9.99
Google Bard Disallow	3,382,611	9.54
Facebook Bot Disallow	3,340,030	9.42
Cohere AI Disallow	3,236,580	9.13
Apple AI Bot Disallow	3,225,055	9.1
Perplexity	3,161,273	8.92
Majestic Bot Disallow	2,309,821	1.13
Google AdsBot Mobile Apps Disallow	2,262,185	1.11
Semrush Bot Disallow	2,113,718	1.04
ByteDance Bytespider Disallow	734,933	2.07
Exalead Bot Disallow	481,062	0.24
Microsoft MSNBot Disallow	446,491	0.22
Webz Omgili Disallow	417,754	1.18
Google Image Bot Disallow	351,709	0.17
360 Bot Disallow	302,825	0.15
Diffbot Disallow	257,932	0.73
DuckDuckGo Bot Disallow	228,358	0.11
YOU Disallow	155,792	0.44
BuiltWithBot Disallow	147,510	0.07
Google Video Bot Disallow	96,468	0.05
Google News Bot Disallow	96,467	0.05
Youdao Bot Disallow	93,084	0.05
Google Favicon Disallow	78,524	0.04
Google API Bot Disallow	51,946	0.03
Google DuplexWeb Disallow	46,838	0.02
Pipl Disallow	27,660	0.01
Google Read Aloud Disallow	5,568	0
Google FeedFetcher Disallow	4,458	0
LLMS Text	3,827	0
Google Web Light Disallow	954	0
Google BIG Bench Disallow	40	0

203,667,593 Detections

of Robots.txt on the Entire Internet. Last updated 07th Jun 2025.

Google Media Partners Disallow is currently the most popular technology in this category.