Robots.txt 使用分布 跻身前 10,000 个网站

排名靠前 Robots.txt 使用分布 跻身前 10,000 个网站

技术	网站	%
Robots Disallow	3,579	35.79
Some Robots Disallow	1,495	14.95
Googlebot Disallow	1,456	14.56
GPTBot Disallow	1,378	13.78
Common Crawl Bot Disallow	1,026	10.26
Ahrefs Bot Disallow	1,011	10.11
AppleBot Disallow	974	9.74
Anthropic Claude Bot Disallow	929	9.29
Yandex Bot Disallow	845	8.45
Google AdsBot Disallow	842	8.42
Google Bard Disallow	789	7.89
Semrush Bot Disallow	753	7.53
ByteDance Bytespider Disallow	723	7.23
Microsoft BingBot Disallow	659	6.59
Perplexity	634	6.34
Facebook Bot Disallow	619	6.19
Baidu Bot Disallow	611	6.11
Majestic Bot Disallow	601	6.01
Webz Omgili Disallow	576	5.76
Cohere AI Disallow	550	5.5
Apple AI Bot Disallow	542	5.42
Alexa Bot Disallow	433	4.33
Diffbot Disallow	387	3.87
Yahoo Slurp Disallow	379	3.79
Google Media Partners Disallow	369	3.69
Microsoft MSNBot Disallow	324	3.24
Sogou Bot Disallow	312	3.12
Google AdsBot Mobile Disallow	295	2.95
YOU Disallow	295	2.95
Google Image Bot Disallow	275	2.75
Exalead Bot Disallow	203	2.03
Google News Bot Disallow	195	1.95
Soso Bot Disallow	174	1.74
Google AdsBot Mobile Apps Disallow	160	1.6
DuckDuckGo Bot Disallow	148	1.48
360 Bot Disallow	144	1.44
Google Video Bot Disallow	115	1.15
Pipl Disallow	85	0.85
Google API Bot Disallow	74	0.74
Google Favicon Disallow	70	0.7
LLMS Text	67	0.67
Google DuplexWeb Disallow	66	0.66
BuiltWithBot Disallow	62	0.62
Anthropic Claude User Bot Disallow	31	0.31
Youdao Bot Disallow	27	0.27
Anthropic Claude Search Bot Disallow	25	0.25
Google FeedFetcher Disallow	4	0.04
Google Read Aloud Disallow	2	0.02

25,313 检测

的 Robots.txt 跻身前 10,000 个网站. 上次更新 27th 7月 2025.

Robots Disallow 是目前该类别中最流行的技术。