Robots.txt Usage Distribution in Turks and Caicos Islands

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	102 ^*	8.48
Googlebot Disallow	73	6.07
GPTBot Disallow	71	13.1
Common Crawl Bot Disallow	65	11.99
Google Bard Disallow	64	11.81
Anthropic Claude Bot Disallow	63	11.62
AppleBot Disallow	62	5.15
Facebook Bot Disallow	62	11.44
Apple AI Bot Disallow	62	11.44
Perplexity	62	11.44
Cohere AI Disallow	61	11.25
Google AdsBot Disallow	60	4.99
Baidu Bot Disallow	47	3.91
Google AdsBot Mobile Apps Disallow	45	3.74
Some Robots Disallow	44	3.66
Sogou Bot Disallow	42	3.49
Google Media Partners Disallow	35 ^*	2.91
Yandex Bot Disallow	29	2.41
Yahoo Slurp Disallow	29	2.41
Alexa Bot Disallow	25	4.61
Soso Bot Disallow	22	1.83
Microsoft BingBot Disallow	20	1.66
Ahrefs Bot Disallow	14	1.16
Majestic Bot Disallow	7	0.58
Microsoft MSNBot Disallow	7	0.58
Semrush Bot Disallow	6	0.5
Google Image Bot Disallow	5	0.42
ByteDance Bytespider Disallow	2	0.37
DuckDuckGo Bot Disallow	2	0.17
Google Video Bot Disallow	2	0.17
Google News Bot Disallow	2	0.17
Google Favicon Disallow	2	0.17
Google API Bot Disallow	2	0.17
Google DuplexWeb Disallow	2	0.17
Exalead Bot Disallow	1	0.08
Webz Omgili Disallow	1	0.18
360 Bot Disallow	1	0.08
Google FeedFetcher Disallow	1	0.08