Robots.txt Usage Distribution in Samoa

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yahoo Slurp Disallow	6,670	15.05
Alexa Bot Disallow	5,014	11.31
Soso Bot Disallow	4,948	11.16
Googlebot Disallow	4,740	10.69
Baidu Bot Disallow	4,691	10.58
Sogou Bot Disallow	4,634	10.45
Some Robots Disallow	3,727	8.41
Yandex Bot Disallow	3,547	8
Microsoft BingBot Disallow	3,522	7.95
Google Media Partners Disallow	894 ^*	2.02
Robots Disallow	410 ^*	0.92
Ahrefs Bot Disallow	111	0.25
Google AdsBot Disallow	96	0.22
Majestic Bot Disallow	96	0.22
Common Crawl Bot Disallow	86	0.19
GPTBot Disallow	79	0.18
Semrush Bot Disallow	78	0.18
AppleBot Disallow	77	0.17
Microsoft MSNBot Disallow	72	0.16
Exalead Bot Disallow	63	0.14
Google Bard Disallow	61	0.14
Anthropic Claude Bot Disallow	60	0.14
Google Image Bot Disallow	56	0.13
Facebook Bot Disallow	52	0.12
Apple AI Bot Disallow	52	0.12
Perplexity	38	0.09
360 Bot Disallow	37	0.08
Webz Omgili Disallow	35	0.08
Google Video Bot Disallow	35	0.08
Google News Bot Disallow	35	0.08
Google Favicon Disallow	35	0.08
DuckDuckGo Bot Disallow	34	0.08
Cohere AI Disallow	32	0.07
Google API Bot Disallow	28	0.06
Google DuplexWeb Disallow	28	0.06
ByteDance Bytespider Disallow	27	0.06
Google AdsBot Mobile Apps Disallow	22	0.05
Diffbot Disallow	19	0.04
Google Read Aloud Disallow	17	0.04
Google FeedFetcher Disallow	16	0.04
Google Web Light Disallow	16	0.04
YOU Disallow	15	0.03
BuiltWithBot Disallow	12	0.03
Pipl Disallow	7	0.02
Youdao Bot Disallow	2	0
LLMS Text	1	0