Robots.txt Usage Distribution in the Top 1 Million Sites

Top In Robots.txt Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
Robots Disallow	154,655	15.47
Ahrefs Bot Disallow	69,338	6.93
Googlebot Disallow	55,267	5.53
Google AdsBot Disallow	48,731	4.87
Yandex Bot Disallow	45,588	4.56
Some Robots Disallow	38,359	3.84
GPTBot Disallow	34,539	3.45
Semrush Bot Disallow	34,212	3.42
Majestic Bot Disallow	33,230	3.32
Baidu Bot Disallow	31,393	3.14
Microsoft BingBot Disallow	27,160	2.72
Common Crawl Bot Disallow	25,232	2.52
Anthropic Claude Bot Disallow	23,101	2.31
Yahoo Slurp Disallow	20,597	2.06
AppleBot Disallow	20,580	2.06
Sogou Bot Disallow	18,614	1.86
Alexa Bot Disallow	18,287	1.83
Google Media Partners Disallow	17,850	1.78
Google Bard Disallow	17,171	1.72
Facebook Bot Disallow	15,419	1.54
ByteDance Bytespider Disallow	14,388	1.44
Perplexity	13,250	1.32
Microsoft MSNBot Disallow	13,126	1.31
Google AdsBot Mobile Disallow	12,158	1.22
Apple AI Bot Disallow	11,340	1.13
Cohere AI Disallow	10,972	1.1
Webz Omgili Disallow	10,880	1.09
Google Image Bot Disallow	10,879	1.09
Soso Bot Disallow	10,576	1.06
Exalead Bot Disallow	9,530	0.95
Diffbot Disallow	7,500	0.75
Google AdsBot Mobile Apps Disallow	6,911	0.69
Google News Bot Disallow	6,433	0.64
360 Bot Disallow	6,152	0.62
YOU Disallow	6,033	0.6
DuckDuckGo Bot Disallow	5,927	0.59
Google Video Bot Disallow	5,063	0.51
Google Favicon Disallow	4,375	0.44
Google API Bot Disallow	4,325	0.43
Google DuplexWeb Disallow	4,178	0.42
Pipl Disallow	3,745	0.37
BuiltWithBot Disallow	2,647	0.26
Youdao Bot Disallow	863	0.09
LLMS Text	719	0.07
Anthropic Claude User Bot Disallow	306	0.03
Anthropic Claude Search Bot Disallow	299	0.03
Google FeedFetcher Disallow	164	0.02
Google Read Aloud Disallow	105	0.01
Google Web Light Disallow	29	0
Google BIG Bench Disallow	2	0

932,198 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 07th Jun 2025.

Robots Disallow is currently the most popular technology in this category.