Robots.txt Nutzungsverteilung in Weißrussland

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Yandex Bot Disallow	4.679	30,36
Robots Disallow	2.845	18,46
Googlebot Disallow	1.704	11,06
Semrush Bot Disallow	572	3,71
Ahrefs Bot Disallow	475	3,08
Microsoft BingBot Disallow	460	2,99
Majestic Bot Disallow	452	2,93
Alexa Bot Disallow	441	21,8
Google Media Partners Disallow	397 ^*	2,58
Some Robots Disallow	361	2,34
Yahoo Slurp Disallow	282	1,83
GPTBot Disallow	279	13,79
Common Crawl Bot Disallow	262	12,95
Microsoft MSNBot Disallow	257	1,67
Baidu Bot Disallow	256	1,66
Anthropic Claude Bot Disallow	223	11,02
Sogou Bot Disallow	209	1,36
Google Bard Disallow	207	10,23
Soso Bot Disallow	202	1,31
Webz Omgili Disallow	192	9,49
Google AdsBot Disallow	97	0,63
Google Image Bot Disallow	83	0,54
Exalead Bot Disallow	79	0,51
Google AdsBot Mobile Disallow	76	0,49
AppleBot Disallow	40	0,26
Facebook Bot Disallow	31	1,53
Apple AI Bot Disallow	25	1,24
Google AdsBot Mobile Apps Disallow	23	0,15
ByteDance Bytespider Disallow	23	1,14
Cohere AI Disallow	22	1,09
Perplexity	22	1,09
360 Bot Disallow	21	0,14
Google News Bot Disallow	18	0,12
Google Video Bot Disallow	17	0,11
DuckDuckGo Bot Disallow	16	0,1
Google API Bot Disallow	11	0,07
BuiltWithBot Disallow	10	0,06
Google Favicon Disallow	10	0,06
Google DuplexWeb Disallow	10	0,06
Diffbot Disallow	8	0,4
YOU Disallow	6	0,3
Pipl Disallow	5	0,03
Google FeedFetcher Disallow	2	0,01