Robots.txt Usage Distribution in Botswana

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	192	17.19
Google AdsBot Disallow	140	12.53
Googlebot Disallow	135	12.09
Google AdsBot Mobile Disallow	123	11.01
GPTBot Disallow	43	3.85
Ahrefs Bot Disallow	34	3.04
AppleBot Disallow	34	3.04
Common Crawl Bot Disallow	33	2.95
Anthropic Claude Bot Disallow	33	2.95
Google Bard Disallow	30	2.69
Facebook Bot Disallow	30	2.69
Cohere AI Disallow	29	2.6
Apple AI Bot Disallow	29	2.6
Perplexity	29	2.6
Google AdsBot Mobile Apps Disallow	21	1.88
Some Robots Disallow	19	1.7
Yandex Bot Disallow	19	1.7
Google Media Partners Disallow	16 ^*	1.43
Semrush Bot Disallow	15	1.34
Yahoo Slurp Disallow	14	1.25
Microsoft BingBot Disallow	13	1.16
Majestic Bot Disallow	13	1.16
ByteDance Bytespider Disallow	13	1.16
Soso Bot Disallow	12	1.07
Alexa Bot Disallow	10	0.9
Webz Omgili Disallow	8	0.72
Diffbot Disallow	7	0.63
YOU Disallow	7	0.63
360 Bot Disallow	4	0.36
Microsoft MSNBot Disallow	3	0.27
Google Image Bot Disallow	3	0.27
Exalead Bot Disallow	2	0.18
BuiltWithBot Disallow	2	0.18
DuckDuckGo Bot Disallow	2	0.18