Robots.txt Usage Distribution in Wallis and Futuna

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	72	10.99
Baidu Bot Disallow	72	10.99
Sogou Bot Disallow	69	10.53
Yahoo Slurp Disallow	69	10.53
Alexa Bot Disallow	66	10.08
Soso Bot Disallow	63	9.62
Yandex Bot Disallow	58	8.85
Microsoft BingBot Disallow	57	8.7
Google Media Partners Disallow	18 ^*	2.75
Robots Disallow	11 ^*	1.68
Ahrefs Bot Disallow	8	1.22
GPTBot Disallow	7	1.07
Semrush Bot Disallow	7	1.07
Common Crawl Bot Disallow	6	0.92
Microsoft MSNBot Disallow	6	0.92
Anthropic Claude Bot Disallow	5	0.76
Google Bard Disallow	5	0.76
Perplexity	5	0.76
ByteDance Bytespider Disallow	5	0.76
Webz Omgili Disallow	5	0.76
AppleBot Disallow	4	0.61
Facebook Bot Disallow	4	0.61
Apple AI Bot Disallow	4	0.61
Majestic Bot Disallow	4	0.61
Diffbot Disallow	4	0.61
Cohere AI Disallow	3	0.46
Exalead Bot Disallow	3	0.46
Google Image Bot Disallow	3	0.46
YOU Disallow	2	0.31
Google AdsBot Mobile Apps Disallow	1	0.15
360 Bot Disallow	1	0.15
DuckDuckGo Bot Disallow	1	0.15
BuiltWithBot Disallow	1	0.15
Google Video Bot Disallow	1	0.15
Google News Bot Disallow	1	0.15
Youdao Bot Disallow	1	0.15
Google Favicon Disallow	1	0.15
Google API Bot Disallow	1	0.15
Google DuplexWeb Disallow	1	0.15