Robots.txt Usage Distribution in Tonga

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yahoo Slurp Disallow	1,799	11.81
Google Media Partners Disallow	1,718 ^*	11.28
Googlebot Disallow	1,717	11.27
Baidu Bot Disallow	1,639	10.76
Sogou Bot Disallow	1,615	10.6
Some Robots Disallow	1,478	9.7
Alexa Bot Disallow	893	55.78
Soso Bot Disallow	833	5.47
Robots Disallow	820 ^*	5.38
Yandex Bot Disallow	757	4.97
Microsoft BingBot Disallow	747	4.9
GPTBot Disallow	104	6.5
Ahrefs Bot Disallow	99	0.65
Semrush Bot Disallow	98	0.64
Majestic Bot Disallow	80	0.53
Google AdsBot Disallow	78	0.51
Google Bard Disallow	78	4.87
AppleBot Disallow	69	0.45
Anthropic Claude Bot Disallow	69	4.31
Facebook Bot Disallow	64	4
Common Crawl Bot Disallow	62	3.87
Apple AI Bot Disallow	59	3.69
Perplexity	54	3.37
Webz Omgili Disallow	40	2.5
ByteDance Bytespider Disallow	38	2.37
Cohere AI Disallow	34	2.12
Diffbot Disallow	30	1.87
YOU Disallow	26	1.62
Google Image Bot Disallow	23	0.15
Microsoft MSNBot Disallow	22	0.14
Google AdsBot Mobile Apps Disallow	20	0.13
Pipl Disallow	20	0.13
Exalead Bot Disallow	15	0.1
Google News Bot Disallow	6	0.04
DuckDuckGo Bot Disallow	5	0.03
Google Video Bot Disallow	5	0.03
360 Bot Disallow	4	0.03
Youdao Bot Disallow	4	0.03
Google Favicon Disallow	4	0.03
Google API Bot Disallow	3	0.02
Google DuplexWeb Disallow	3	0.02
LLMS Text	3	0.02
BuiltWithBot Disallow	1	0.01