Robots.txt Usage Distribution in the United States Virgin Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Anthropic Claude Bot Disallow	148	7.7
GPTBot Disallow	146	7.6
AppleBot Disallow	144	7.49
Common Crawl Bot Disallow	141	7.34
Facebook Bot Disallow	141	7.34
Google Bard Disallow	140	7.28
Apple AI Bot Disallow	137	7.13
Cohere AI Disallow	134	6.97
Perplexity	133	6.92
Google AdsBot Mobile Apps Disallow	101	5.25
Robots Disallow	98 ^*	5.1
Ahrefs Bot Disallow	80	4.16
Baidu Bot Disallow	60	3.12
Yandex Bot Disallow	44	2.29
Sogou Bot Disallow	32	1.66
Some Robots Disallow	31	1.61
Microsoft BingBot Disallow	29	1.51
Yahoo Slurp Disallow	29	1.51
Alexa Bot Disallow	28	1.46
Majestic Bot Disallow	27	1.4
Semrush Bot Disallow	23	1.2
Soso Bot Disallow	22	1.14
Google Media Partners Disallow	17 ^*	0.88
ByteDance Bytespider Disallow	7	0.36
Exalead Bot Disallow	5	0.26
Microsoft MSNBot Disallow	5	0.26
Webz Omgili Disallow	5	0.26
Google Image Bot Disallow	5	0.26
Diffbot Disallow	2	0.1
360 Bot Disallow	2	0.1
YOU Disallow	1	0.05
BuiltWithBot Disallow	1	0.05
Google Video Bot Disallow	1	0.05
Google News Bot Disallow	1	0.05
Google Favicon Disallow	1	0.05
Pipl Disallow	1	0.05