Robots.txt Usage Distribution in Romania

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	22,066	10.04
Baidu Bot Disallow	20,275	9.22
Yandex Bot Disallow	19,920	9.06
Microsoft BingBot Disallow	19,642	8.94
Sogou Bot Disallow	19,069	8.68
Yahoo Slurp Disallow	18,735	8.52
Alexa Bot Disallow	18,525	8.43
Soso Bot Disallow	18,284	8.32
Ahrefs Bot Disallow	9,423	4.29
Robots Disallow	8,632 ^*	3.93
Google AdsBot Disallow	8,481	3.86
Majestic Bot Disallow	4,847	2.21
Google AdsBot Mobile Disallow	3,721	1.69
Some Robots Disallow	3,225	1.47
Common Crawl Bot Disallow	2,734	1.24
Anthropic Claude Bot Disallow	2,572	1.17
Facebook Bot Disallow	2,374	1.08
Semrush Bot Disallow	2,174	0.99
Exalead Bot Disallow	1,976	0.9
ByteDance Bytespider Disallow	1,958	0.89
GPTBot Disallow	1,507	0.69
Google Media Partners Disallow	1,350 ^*	0.61
AppleBot Disallow	1,290	0.59
Google Bard Disallow	1,041	0.47
Perplexity	918	0.42
Apple AI Bot Disallow	904	0.41
Cohere AI Disallow	894	0.41
Google AdsBot Mobile Apps Disallow	587	0.27
Microsoft MSNBot Disallow	584	0.27
Google Image Bot Disallow	343	0.16
360 Bot Disallow	319	0.15
Webz Omgili Disallow	292	0.13
BuiltWithBot Disallow	174	0.08
Diffbot Disallow	153	0.07
YOU Disallow	141	0.06
Google News Bot Disallow	135	0.06
DuckDuckGo Bot Disallow	132	0.06
Google Video Bot Disallow	108	0.05
Google Favicon Disallow	100	0.05
Google API Bot Disallow	65	0.03
Google DuplexWeb Disallow	64	0.03
Pipl Disallow	60	0.03
Youdao Bot Disallow	10	0
Google Read Aloud Disallow	3	0
Google FeedFetcher Disallow	2	0
Google Web Light Disallow	1	0