Robots.txt Usage Distribution in Hong Kong

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	15,092	16.12
Googlebot Disallow	10,009	10.69
Google AdsBot Mobile Disallow	9,907	10.58
Ahrefs Bot Disallow	6,937	7.41
Robots Disallow	5,136 ^*	5.49
Anthropic Claude Bot Disallow	3,580	3.82
AppleBot Disallow	3,104	3.32
GPTBot Disallow	3,089	3.3
Majestic Bot Disallow	2,806	3
Common Crawl Bot Disallow	2,740	2.93
Google Bard Disallow	2,673	2.85
Facebook Bot Disallow	2,655	2.84
Apple AI Bot Disallow	2,630	2.81
Perplexity	2,583	2.76
Cohere AI Disallow	2,562	2.74
Google AdsBot Mobile Apps Disallow	1,965	2.1
Baidu Bot Disallow	1,895	2.02
Yandex Bot Disallow	1,688	1.8
Yahoo Slurp Disallow	1,664	1.78
Sogou Bot Disallow	1,662	1.78
Semrush Bot Disallow	1,528	1.63
Alexa Bot Disallow	1,456	1.56
Microsoft BingBot Disallow	1,336	1.43
Some Robots Disallow	1,270	1.36
Soso Bot Disallow	1,144	1.22
Google Media Partners Disallow	666 ^*	0.71
Exalead Bot Disallow	292	0.31
Microsoft MSNBot Disallow	261	0.28
ByteDance Bytespider Disallow	245	0.26
Google Image Bot Disallow	230	0.25
Webz Omgili Disallow	195	0.21
DuckDuckGo Bot Disallow	109	0.12
Diffbot Disallow	87	0.09
360 Bot Disallow	78	0.08
YOU Disallow	62	0.07
Google News Bot Disallow	56	0.06
Google Video Bot Disallow	53	0.06
Google Favicon Disallow	37	0.04
Youdao Bot Disallow	34	0.04
BuiltWithBot Disallow	29	0.03
Google API Bot Disallow	27	0.03
Pipl Disallow	26	0.03
Google DuplexWeb Disallow	25	0.03
Google FeedFetcher Disallow	4	0