Robots.txt Usage Distribution in Belarus

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yandex Bot Disallow	4,679	31.66
Robots Disallow	2,152 ^*	14.56
Googlebot Disallow	1,704	11.53
Semrush Bot Disallow	563	3.81
Ahrefs Bot Disallow	478	3.23
Microsoft BingBot Disallow	460	3.11
Majestic Bot Disallow	457	3.09
Alexa Bot Disallow	447	3.02
Google Media Partners Disallow	403 ^*	2.73
Some Robots Disallow	361	2.44
Yahoo Slurp Disallow	287	1.94
GPTBot Disallow	280	1.89
Common Crawl Bot Disallow	265	1.79
Baidu Bot Disallow	256	1.73
Microsoft MSNBot Disallow	252	1.71
Anthropic Claude Bot Disallow	223	1.51
Sogou Bot Disallow	209	1.41
Google Bard Disallow	209	1.41
Soso Bot Disallow	204	1.38
Webz Omgili Disallow	186	1.26
Google AdsBot Disallow	97	0.66
Google Image Bot Disallow	91	0.62
Exalead Bot Disallow	84	0.57
Google AdsBot Mobile Disallow	76	0.51
AppleBot Disallow	45	0.3
Facebook Bot Disallow	39	0.26
Apple AI Bot Disallow	32	0.22
Cohere AI Disallow	29	0.2
Perplexity	29	0.2
ByteDance Bytespider Disallow	23	0.16
360 Bot Disallow	22	0.15
Google AdsBot Mobile Apps Disallow	21	0.14
Google News Bot Disallow	18	0.12
Google Video Bot Disallow	17	0.12
DuckDuckGo Bot Disallow	16	0.11
BuiltWithBot Disallow	11	0.07
Google API Bot Disallow	11	0.07
Google Favicon Disallow	10	0.07
Google DuplexWeb Disallow	10	0.07
Diffbot Disallow	8	0.05
YOU Disallow	6	0.04
Pipl Disallow	5	0.03
Google FeedFetcher Disallow	2	0.01
LLMS Text	1	0.01