Robots.txt Usage Distribution in China

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	91,341	9.28
Ahrefs Bot Disallow	86,675	8.81
Semrush Bot Disallow	79,482	8.08
Microsoft BingBot Disallow	79,014	8.03
Yahoo Slurp Disallow	70,540	7.17
Baidu Bot Disallow	64,709	6.58
Alexa Bot Disallow	64,491	6.55
Sogou Bot Disallow	58,290	5.92
Yandex Bot Disallow	52,935	5.38
AppleBot Disallow	51,070	5.19
Soso Bot Disallow	48,702	4.95
Majestic Bot Disallow	39,577	4.02
Some Robots Disallow	31,023	3.15
Robots Disallow	23,789 ^*	2.42
Google AdsBot Disallow	22,836	2.32
Microsoft MSNBot Disallow	22,425	2.28
Google Image Bot Disallow	17,132	1.74
Google AdsBot Mobile Disallow	15,763	1.6
Anthropic Claude Bot Disallow	9,397	0.95
GPTBot Disallow	9,291	0.94
ByteDance Bytespider Disallow	5,859	0.6
Exalead Bot Disallow	5,786	0.59
Youdao Bot Disallow	4,999	0.51
360 Bot Disallow	4,330	0.44
Common Crawl Bot Disallow	3,792	0.39
Google Bard Disallow	3,031	0.31
Facebook Bot Disallow	2,969	0.3
Google Media Partners Disallow	2,525 ^*	0.26
Apple AI Bot Disallow	2,404	0.24
Cohere AI Disallow	2,325	0.24
Perplexity	2,290	0.23
Google AdsBot Mobile Apps Disallow	1,598	0.16
Webz Omgili Disallow	674	0.07
Diffbot Disallow	535	0.05
DuckDuckGo Bot Disallow	511	0.05
BuiltWithBot Disallow	306	0.03
Google News Bot Disallow	247	0.03
YOU Disallow	233	0.02
Google Video Bot Disallow	230	0.02
Google Favicon Disallow	203	0.02
Google API Bot Disallow	201	0.02
Google DuplexWeb Disallow	199	0.02
Google FeedFetcher Disallow	181	0.02
Pipl Disallow	126	0.01
LLMS Text	68	0.01
Google Read Aloud Disallow	3	0
Google Web Light Disallow	2	0