Robots.txt 使用分布 跻身前 100 万个网站之列

排名靠前 Robots.txt 使用分布 跻身前 100 万个网站之列

技术	网站	%
Robots Disallow	158,185	15.82
Ahrefs Bot Disallow	69,725	6.97
Googlebot Disallow	56,385	5.64
Google AdsBot Disallow	48,850	4.88
Yandex Bot Disallow	46,646	4.66
Some Robots Disallow	39,136	3.91
GPTBot Disallow	35,653	3.57
Semrush Bot Disallow	34,606	3.46
Majestic Bot Disallow	33,578	3.36
Baidu Bot Disallow	32,187	3.22
Microsoft BingBot Disallow	28,024	2.8
Common Crawl Bot Disallow	25,309	2.53
Anthropic Claude Bot Disallow	23,218	2.32
AppleBot Disallow	21,607	2.16
Yahoo Slurp Disallow	21,357	2.14
Sogou Bot Disallow	19,347	1.93
Alexa Bot Disallow	18,867	1.89
Google Media Partners Disallow	18,580	1.86
Google Bard Disallow	17,271	1.73
Facebook Bot Disallow	15,505	1.55
ByteDance Bytespider Disallow	14,484	1.45
Perplexity	13,348	1.33
Microsoft MSNBot Disallow	13,179	1.32
Google AdsBot Mobile Disallow	12,245	1.22
Apple AI Bot Disallow	11,429	1.14
Soso Bot Disallow	11,065	1.11
Cohere AI Disallow	11,011	1.1
Google Image Bot Disallow	10,981	1.1
Webz Omgili Disallow	10,962	1.1
Exalead Bot Disallow	9,584	0.96
Diffbot Disallow	7,583	0.76
Google AdsBot Mobile Apps Disallow	6,911	0.69
Google News Bot Disallow	6,484	0.65
360 Bot Disallow	6,182	0.62
YOU Disallow	6,117	0.61
DuckDuckGo Bot Disallow	5,993	0.6
Google Video Bot Disallow	5,098	0.51
Google Favicon Disallow	4,399	0.44
Google API Bot Disallow	4,353	0.44
Google DuplexWeb Disallow	4,202	0.42
Pipl Disallow	3,795	0.38
BuiltWithBot Disallow	2,659	0.27
LLMS Text	1,510	0.15
Youdao Bot Disallow	865	0.09
Anthropic Claude User Bot Disallow	385	0.04
Anthropic Claude Search Bot Disallow	369	0.04
Google FeedFetcher Disallow	169	0.02
Google Read Aloud Disallow	107	0.01
Google Web Light Disallow	29	0
Google BIG Bench Disallow	2	0

949,536 检测

的 Robots.txt 跻身前 100 万个网站之列. 上次更新 01st 7月 2025.

Robots Disallow 是目前该类别中最流行的技术。