Robots.txt Usage Distribution in China

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	91,341	9.99
Microsoft BingBot Disallow	79,014	8.64
Ahrefs Bot Disallow	71,670	7.84
Baidu Bot Disallow	64,709	7.08
Semrush Bot Disallow	63,959	7
Yahoo Slurp Disallow	60,364	6.6
Sogou Bot Disallow	58,290	6.38
Alexa Bot Disallow	57,779	6.32
Yandex Bot Disallow	52,935	5.79
Soso Bot Disallow	47,029	5.14
Robots Disallow	38,916	4.26
AppleBot Disallow	38,570	4.22
Majestic Bot Disallow	35,332	3.86
Some Robots Disallow	31,023	3.39
Google AdsBot Disallow	22,836	2.5
Microsoft MSNBot Disallow	16,606	1.82
Google AdsBot Mobile Disallow	15,763	1.72
Google Image Bot Disallow	12,303	1.35
GPTBot Disallow	8,492	0.93
Anthropic Claude Bot Disallow	7,659	0.84
ByteDance Bytespider Disallow	5,424	0.59
Exalead Bot Disallow	4,789	0.52
360 Bot Disallow	3,993	0.44
Youdao Bot Disallow	3,405	0.37
Common Crawl Bot Disallow	3,360	0.37
Google Bard Disallow	2,675	0.29
Facebook Bot Disallow	2,628	0.29
Google Media Partners Disallow	2,383 ^*	0.26
Apple AI Bot Disallow	2,082	0.23
Cohere AI Disallow	2,074	0.23
Perplexity	1,959	0.21
Google AdsBot Mobile Apps Disallow	1,662	0.18
Webz Omgili Disallow	592	0.06
DuckDuckGo Bot Disallow	452	0.05
Diffbot Disallow	450	0.05
Google News Bot Disallow	251	0.03
BuiltWithBot Disallow	248	0.03
Google Video Bot Disallow	233	0.03
Google Favicon Disallow	210	0.02
Google API Bot Disallow	208	0.02
Google DuplexWeb Disallow	207	0.02
Google FeedFetcher Disallow	167	0.02
YOU Disallow	136	0.01
LLMS Text	42	0
Pipl Disallow	40	0
Google Read Aloud Disallow	4	0
Google Web Light Disallow	2	0