Robots.txt Usage Distribution in Tonga

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	14,058	51.55
Googlebot Disallow	1,717	6.3
Baidu Bot Disallow	1,639	6.01
Sogou Bot Disallow	1,615	5.92
Google Media Partners Disallow	1,589 ^*	5.83
Some Robots Disallow	1,478	5.42
Yahoo Slurp Disallow	1,054	3.86
Alexa Bot Disallow	828	3.04
Soso Bot Disallow	777	2.85
Yandex Bot Disallow	757	2.78
Microsoft BingBot Disallow	747	2.74
Ahrefs Bot Disallow	100	0.37
GPTBot Disallow	88	0.32
Semrush Bot Disallow	88	0.32
Majestic Bot Disallow	81	0.3
Google AdsBot Disallow	78	0.29
Google Bard Disallow	60	0.22
Common Crawl Bot Disallow	57	0.21
AppleBot Disallow	54	0.2
Anthropic Claude Bot Disallow	51	0.19
Facebook Bot Disallow	50	0.18
Apple AI Bot Disallow	42	0.15
Perplexity	37	0.14
Cohere AI Disallow	29	0.11
ByteDance Bytespider Disallow	27	0.1
Webz Omgili Disallow	27	0.1
Google AdsBot Mobile Apps Disallow	21	0.08
Microsoft MSNBot Disallow	20	0.07
Google Image Bot Disallow	18	0.07
Diffbot Disallow	16	0.06
Exalead Bot Disallow	15	0.06
YOU Disallow	12	0.04
Pipl Disallow	10	0.04
360 Bot Disallow	6	0.02
Google News Bot Disallow	5	0.02
DuckDuckGo Bot Disallow	4	0.01
Youdao Bot Disallow	4	0.01
Google Video Bot Disallow	2	0.01
Google Favicon Disallow	2	0.01
Google API Bot Disallow	2	0.01
Google DuplexWeb Disallow	2	0.01
LLMS Text	2	0.01
BuiltWithBot Disallow	1	0
Google FeedFetcher Disallow	1	0