Robots.txt Usage Distribution in New Zealand

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	45,206	11
Google AdsBot Mobile Disallow	32,418	7.89
GPTBot Disallow	26,356	12.55
AppleBot Disallow	25,770	6.27
Common Crawl Bot Disallow	25,696	12.23
Anthropic Claude Bot Disallow	25,494	12.14
Google Bard Disallow	25,353	12.07
Facebook Bot Disallow	25,280	12.03
Apple AI Bot Disallow	25,168	11.98
Perplexity	25,060	11.93
Cohere AI Disallow	25,026	11.91
Googlebot Disallow	19,143	4.66
Google AdsBot Mobile Apps Disallow	18,516	4.5
Ahrefs Bot Disallow	16,051	3.91
Robots Disallow	7,641 ^*	1.86
Baidu Bot Disallow	7,512	1.83
Microsoft BingBot Disallow	6,223	1.51
Yahoo Slurp Disallow	4,690	1.14
Yandex Bot Disallow	4,158	1.01
Alexa Bot Disallow	2,969	1.41
Sogou Bot Disallow	2,858	0.7
Soso Bot Disallow	2,664	0.65
Majestic Bot Disallow	2,518	0.61
Semrush Bot Disallow	2,121	0.52
Some Robots Disallow	1,805	0.44
Google Media Partners Disallow	824 ^*	0.2
ByteDance Bytespider Disallow	794	0.38
Microsoft MSNBot Disallow	697	0.17
Exalead Bot Disallow	590	0.14
Google Image Bot Disallow	587	0.14
Webz Omgili Disallow	483	0.23
Diffbot Disallow	234	0.11
360 Bot Disallow	178	0.04
YOU Disallow	148	0.07
Google News Bot Disallow	147	0.04
Google Video Bot Disallow	145	0.04
DuckDuckGo Bot Disallow	119	0.03
BuiltWithBot Disallow	109	0.03
Google Favicon Disallow	99	0.02
Pipl Disallow	55	0.01
Google API Bot Disallow	42	0.01
Google DuplexWeb Disallow	41	0.01
Youdao Bot Disallow	37	0.01
LLMS Text	3	0
Google FeedFetcher Disallow	2	0
Google Read Aloud Disallow	1	0