Robots.txt Usage Distribution in Reunion

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	2,360	16.31
Robots Disallow	1,337	9.24
Baidu Bot Disallow	1,312	9.06
Sogou Bot Disallow	1,274	8.8
Google AdsBot Disallow	1,078	7.45
Some Robots Disallow	992	6.85
Google AdsBot Mobile Disallow	783	5.41
Yahoo Slurp Disallow	671	4.64
Ahrefs Bot Disallow	516	3.57
Yandex Bot Disallow	461	3.19
Microsoft BingBot Disallow	451	3.12
Alexa Bot Disallow	414	2.86
Soso Bot Disallow	411	2.84
Semrush Bot Disallow	246	1.7
Google Media Partners Disallow	238 ^*	1.64
Majestic Bot Disallow	202	1.4
Anthropic Claude Bot Disallow	176	1.22
GPTBot Disallow	143	0.99
Common Crawl Bot Disallow	127	0.88
Google Bard Disallow	121	0.84
Facebook Bot Disallow	112	0.77
AppleBot Disallow	110	0.76
Apple AI Bot Disallow	94	0.65
Cohere AI Disallow	89	0.61
Perplexity	89	0.61
Google AdsBot Mobile Apps Disallow	79	0.55
DuckDuckGo Bot Disallow	73	0.5
Webz Omgili Disallow	67	0.46
Microsoft MSNBot Disallow	53	0.37
Google Image Bot Disallow	53	0.37
ByteDance Bytespider Disallow	47	0.32
Diffbot Disallow	40	0.28
YOU Disallow	38	0.26
Google News Bot Disallow	36	0.25
Google Video Bot Disallow	36	0.25
Google Favicon Disallow	34	0.23
Google API Bot Disallow	34	0.23
Google DuplexWeb Disallow	34	0.23
Exalead Bot Disallow	33	0.23
Pipl Disallow	4	0.03
Youdao Bot Disallow	3	0.02
360 Bot Disallow	1	0.01
BuiltWithBot Disallow	1	0.01
LLMS Text	1	0.01