Robots.txt Usage Distribution in Ukraine

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Yandex Bot Disallow	21,387	24.64
Robots Disallow	12,424 ^*	14.31
Googlebot Disallow	10,029	11.55
Google AdsBot Disallow	3,772	4.34
Google AdsBot Mobile Disallow	3,130	3.61
Ahrefs Bot Disallow	3,091	3.56
Alexa Bot Disallow	2,662	3.07
Google Media Partners Disallow	2,647 ^*	3.05
Majestic Bot Disallow	2,589	2.98
Baidu Bot Disallow	2,550	2.94
Yahoo Slurp Disallow	2,530	2.91
Microsoft BingBot Disallow	2,495	2.87
Sogou Bot Disallow	2,357	2.71
Semrush Bot Disallow	1,917	2.21
Soso Bot Disallow	1,888	2.17
Some Robots Disallow	1,597	1.84
GPTBot Disallow	1,251	1.44
Common Crawl Bot Disallow	1,041	1.2
Microsoft MSNBot Disallow	791	0.91
Exalead Bot Disallow	647	0.75
Anthropic Claude Bot Disallow	615	0.71
Google Image Bot Disallow	610	0.7
Google Bard Disallow	559	0.64
360 Bot Disallow	463	0.53
AppleBot Disallow	462	0.53
Facebook Bot Disallow	404	0.47
Apple AI Bot Disallow	344	0.4
Perplexity	339	0.39
Cohere AI Disallow	315	0.36
ByteDance Bytespider Disallow	308	0.35
Webz Omgili Disallow	224	0.26
Youdao Bot Disallow	217	0.25
YOU Disallow	211	0.24
Google AdsBot Mobile Apps Disallow	206	0.24
BuiltWithBot Disallow	176	0.2
Diffbot Disallow	126	0.15
DuckDuckGo Bot Disallow	99	0.11
Google News Bot Disallow	82	0.09
Google Video Bot Disallow	67	0.08
Google API Bot Disallow	52	0.06
Google Favicon Disallow	50	0.06
Google DuplexWeb Disallow	48	0.06
Pipl Disallow	36	0.04
Google FeedFetcher Disallow	3	0
LLMS Text	3	0
Google Read Aloud Disallow	1	0