Robots.txt Usage Distribution in British Virgin Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Soso Bot Disallow	9,224	17.01
Yahoo Slurp Disallow	9,112	16.8
Googlebot Disallow	5,895	10.87
Baidu Bot Disallow	5,888	10.86
Sogou Bot Disallow	5,885	10.85
Yandex Bot Disallow	5,867	10.82
Microsoft BingBot Disallow	5,863	10.81
Some Robots Disallow	5,824	10.74
Alexa Bot Disallow	208	0.38
Robots Disallow	57 ^*	0.11
Google Media Partners Disallow	35 ^*	0.06
GPTBot Disallow	33	0.06
AppleBot Disallow	32	0.06
Anthropic Claude Bot Disallow	31	0.06
Google Bard Disallow	30	0.06
Facebook Bot Disallow	30	0.06
Apple AI Bot Disallow	30	0.06
Common Crawl Bot Disallow	28	0.05
Cohere AI Disallow	26	0.05
Perplexity	26	0.05
Google AdsBot Mobile Apps Disallow	20	0.04
Semrush Bot Disallow	14	0.03
Ahrefs Bot Disallow	12	0.02
Majestic Bot Disallow	12	0.02
Webz Omgili Disallow	8	0.01
ByteDance Bytespider Disallow	6	0.01
Diffbot Disallow	6	0.01
Microsoft MSNBot Disallow	4	0.01
Google Image Bot Disallow	4	0.01
360 Bot Disallow	4	0.01
YOU Disallow	3	0.01
Pipl Disallow	3	0.01
DuckDuckGo Bot Disallow	2	0
Exalead Bot Disallow	1	0
Google Video Bot Disallow	1	0
Google News Bot Disallow	1	0
Youdao Bot Disallow	1	0
Google Favicon Disallow	1	0
Google API Bot Disallow	1	0
Google DuplexWeb Disallow	1	0