Robots.txt Usage Distribution in Russia

Top In Robots.txt Usage Distribution in Russia

Technology	Websites	%
Ahrefs Bot Disallow	404,820	14.57
Majestic Bot Disallow	396,278	14.26
Semrush Bot Disallow	333,028	11.99
Microsoft BingBot Disallow	232,549	8.37
Baidu Bot Disallow	231,757	8.34
Yandex Bot Disallow	231,236	8.32
AppleBot Disallow	216,062	7.78
Robots Disallow	157,912 *	5.68
Googlebot Disallow	123,725	4.45
Alexa Bot Disallow	97,977	3.53
Some Robots Disallow	77,405	2.79
Common Crawl Bot Disallow	31,206	1.12
Exalead Bot Disallow	29,493	1.06
Microsoft MSNBot Disallow	28,928	1.04
Yahoo Slurp Disallow	24,620	0.89
Sogou Bot Disallow	24,288	0.87
Soso Bot Disallow	21,791	0.78
GPTBot Disallow	18,605	0.67
Google Media Partners Disallow	11,782 *	0.42
Google AdsBot Disallow	10,172	0.37
Google Bard Disallow	9,701	0.35
Google AdsBot Mobile Disallow	9,011	0.32
Facebook Bot Disallow	8,730	0.31
Anthropic Claude Bot Disallow	8,439	0.3
Perplexity	6,436	0.23
Webz Omgili Disallow	5,507	0.2
Google Image Bot Disallow	5,088	0.18
360 Bot Disallow	4,682	0.17
ByteDance Bytespider Disallow	4,156	0.15
BuiltWithBot Disallow	3,876	0.14
Cohere AI Disallow	3,380	0.12
Apple AI Bot Disallow	1,247	0.04
Diffbot Disallow	703	0.03
YOU Disallow	588	0.02
Google AdsBot Mobile Apps Disallow	570	0.02
Google News Bot Disallow	534	0.02
Google Video Bot Disallow	438	0.02
DuckDuckGo Bot Disallow	430	0.02
Pipl Disallow	289	0.01
Youdao Bot Disallow	263	0.01
Google Favicon Disallow	249	0.01
Google API Bot Disallow	249	0.01
Google DuplexWeb Disallow	223	0.01
Google FeedFetcher Disallow	90	0
LLMS Text	85	0

2,778,598 Detections

of Robots.txt in Russia. Last updated 07th Jun 2025.

Ahrefs Bot Disallow is currently the most popular technology in this category.