Robots.txt 使用分布 跻身前 100 万个网站之列

排名靠前 Robots.txt 使用分布 跻身前 100 万个网站之列

技术	网站	%
Robots Disallow	163,669	16.37
Ahrefs Bot Disallow	69,476	6.95
Googlebot Disallow	57,398	5.74
Google AdsBot Disallow	48,482	4.85
Yandex Bot Disallow	44,307	4.43
Some Robots Disallow	41,977	4.2
Semrush Bot Disallow	33,897	3.39
Baidu Bot Disallow	33,893	3.39
GPTBot Disallow	33,651	3.37
Majestic Bot Disallow	33,459	3.35
Common Crawl Bot Disallow	25,450	2.54
Microsoft BingBot Disallow	24,860	2.49
Anthropic Claude Bot Disallow	21,387	2.14
AppleBot Disallow	20,992	2.1
Google Media Partners Disallow	20,621	2.06
Sogou Bot Disallow	19,904	1.99
Alexa Bot Disallow	18,877	1.89
Yahoo Slurp Disallow	18,858	1.89
Google Bard Disallow	15,940	1.59
Facebook Bot Disallow	14,223	1.42
Microsoft MSNBot Disallow	13,762	1.38
ByteDance Bytespider Disallow	12,936	1.29
Google AdsBot Mobile Disallow	12,931	1.29
Perplexity	11,571	1.16
Google Image Bot Disallow	11,292	1.13
Soso Bot Disallow	10,895	1.09
Cohere AI Disallow	10,774	1.08
Apple AI Bot Disallow	9,895	0.99
Exalead Bot Disallow	9,759	0.98
Webz Omgili Disallow	9,727	0.97
Google AdsBot Mobile Apps Disallow	8,442	0.84
Google News Bot Disallow	7,193	0.72
DuckDuckGo Bot Disallow	6,461	0.65
360 Bot Disallow	6,297	0.63
Diffbot Disallow	5,995	0.6
Google Video Bot Disallow	5,759	0.58
Google Favicon Disallow	5,115	0.51
Google API Bot Disallow	5,022	0.5
Google DuplexWeb Disallow	4,904	0.49
YOU Disallow	4,300	0.43
BuiltWithBot Disallow	2,838	0.28
Pipl Disallow	2,427	0.24
Youdao Bot Disallow	918	0.09
Google FeedFetcher Disallow	142	0.01
LLMS Text	133	0.01
Google Read Aloud Disallow	58	0.01
Google Web Light Disallow	15	0
Google BIG Bench Disallow	5	0

940,887 检测

的 Robots.txt 跻身前 100 万个网站之列. 上次更新 07th 3月 2025.

Google Media Partners Disallow 是目前该类别中最流行的技术。