Robots.txt Usage Distribution in Anguilla

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	28,843	13.72
Baidu Bot Disallow	23,771	11.31
Sogou Bot Disallow	23,641	11.25
Some Robots Disallow	22,900	10.9
Microsoft BingBot Disallow	21,147	10.06
Yandex Bot Disallow	21,146	10.06
Yahoo Slurp Disallow	21,058	10.02
Google AdsBot Disallow	6,791	3.23
Google AdsBot Mobile Disallow	6,241	2.97
GPTBot Disallow	2,854	1.36
Robots Disallow	2,821 ^*	1.34
AppleBot Disallow	2,660	1.27
Soso Bot Disallow	2,572	1.22
Alexa Bot Disallow	2,453	1.17
Google Bard Disallow	2,396	1.14
Anthropic Claude Bot Disallow	2,360	1.12
Common Crawl Bot Disallow	2,331	1.11
Apple AI Bot Disallow	2,282	1.09
Facebook Bot Disallow	2,183	1.04
Perplexity	2,168	1.03
Cohere AI Disallow	2,027	0.96
Google AdsBot Mobile Apps Disallow	1,304	0.62
Ahrefs Bot Disallow	887	0.42
Majestic Bot Disallow	540	0.26
Semrush Bot Disallow	514	0.24
Google Media Partners Disallow	373 ^*	0.18
ByteDance Bytespider Disallow	345	0.16
LLMS Text	311	0.15
Webz Omgili Disallow	213	0.1
Diffbot Disallow	134	0.06
Google Image Bot Disallow	120	0.06
YOU Disallow	120	0.06
Microsoft MSNBot Disallow	90	0.04
DuckDuckGo Bot Disallow	86	0.04
Pipl Disallow	86	0.04
360 Bot Disallow	63	0.03
Google Video Bot Disallow	58	0.03
Google News Bot Disallow	58	0.03
Google API Bot Disallow	51	0.02
Google Favicon Disallow	49	0.02
Google DuplexWeb Disallow	48	0.02
Exalead Bot Disallow	40	0.02
BuiltWithBot Disallow	19	0.01
Youdao Bot Disallow	4	0
Google Read Aloud Disallow	4	0
Anthropic Claude User Bot Disallow	3	0
Google FeedFetcher Disallow	3	0
Anthropic Claude Search Bot Disallow	2	0
Google BIG Bench Disallow	1	0