Robots.txt Usage Distribution in Poland

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	43,728	8.8
Baidu Bot Disallow	35,202	7.08
Yandex Bot Disallow	33,493	6.74
Yahoo Slurp Disallow	31,813	6.4
Sogou Bot Disallow	30,355	6.11
Alexa Bot Disallow	29,422	5.92
Robots Disallow	27,683 ^*	5.57
Soso Bot Disallow	27,333	5.5
Microsoft BingBot Disallow	26,746	5.38
Ahrefs Bot Disallow	20,594	4.14
Majestic Bot Disallow	19,544	3.93
Google AdsBot Disallow	16,813	3.38
Semrush Bot Disallow	12,531	2.52
Google AdsBot Mobile Disallow	11,970	2.41
Some Robots Disallow	11,667	2.35
GPTBot Disallow	9,848	1.98
Common Crawl Bot Disallow	9,810	1.97
Anthropic Claude Bot Disallow	8,438	1.7
Exalead Bot Disallow	8,195	1.65
AppleBot Disallow	7,836	1.58
Google Bard Disallow	7,397	1.49
Facebook Bot Disallow	7,382	1.49
ByteDance Bytespider Disallow	7,113	1.43
Cohere AI Disallow	6,875	1.38
Perplexity	6,791	1.37
Apple AI Bot Disallow	6,753	1.36
Webz Omgili Disallow	5,967	1.2
Diffbot Disallow	5,418	1.09
YOU Disallow	5,150	1.04
Google Media Partners Disallow	2,875 ^*	0.58
Google Image Bot Disallow	2,429	0.49
360 Bot Disallow	2,055	0.41
Microsoft MSNBot Disallow	1,925	0.39
Google AdsBot Mobile Apps Disallow	1,354	0.27
BuiltWithBot Disallow	1,330	0.27
DuckDuckGo Bot Disallow	757	0.15
Google News Bot Disallow	433	0.09
Google Video Bot Disallow	387	0.08
Google Favicon Disallow	354	0.07
Google API Bot Disallow	336	0.07
Google DuplexWeb Disallow	320	0.06
Youdao Bot Disallow	298	0.06
Pipl Disallow	213	0.04
LLMS Text	23	0
Google FeedFetcher Disallow	12	0
Google Read Aloud Disallow	6	0
Google Web Light Disallow	2	0