Robots.txt Usage Distribution in Iran

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Majestic Bot Disallow	4,790	14.54
Ahrefs Bot Disallow	4,199	12.74
Semrush Bot Disallow	3,810	11.56
Robots Disallow	2,206	6.7
Yandex Bot Disallow	1,668	5.06
Microsoft BingBot Disallow	1,627	4.94
Microsoft MSNBot Disallow	1,484	4.5
Common Crawl Bot Disallow	1,387	4.21
Some Robots Disallow	1,332	4.04
Exalead Bot Disallow	1,279	3.88
Googlebot Disallow	1,226	3.72
Baidu Bot Disallow	1,038	3.15
ByteDance Bytespider Disallow	994	3.02
Yahoo Slurp Disallow	915	2.78
Sogou Bot Disallow	808	2.45
Soso Bot Disallow	558	1.69
Alexa Bot Disallow	499	1.51
GPTBot Disallow	431	1.31
Google Media Partners Disallow	397 ^*	1.2
DuckDuckGo Bot Disallow	274	0.83
AppleBot Disallow	258	0.78
360 Bot Disallow	231	0.7
BuiltWithBot Disallow	197	0.6
Google Image Bot Disallow	192	0.58
Google Bard Disallow	153	0.46
Anthropic Claude Bot Disallow	132	0.4
Google AdsBot Disallow	112	0.34
Google AdsBot Mobile Disallow	80	0.24
Perplexity	65	0.2
Facebook Bot Disallow	62	0.19
Google News Bot Disallow	62	0.19
Google Video Bot Disallow	61	0.19
Google AdsBot Mobile Apps Disallow	58	0.18
Apple AI Bot Disallow	56	0.17
Cohere AI Disallow	49	0.15
Google Favicon Disallow	44	0.13
Google API Bot Disallow	44	0.13
Google DuplexWeb Disallow	44	0.13
Webz Omgili Disallow	42	0.13
Diffbot Disallow	32	0.1
Pipl Disallow	26	0.08
YOU Disallow	22	0.07
Youdao Bot Disallow	3	0.01
LLMS Text	1	0