Robots.txt Nutzungsverteilung in Schweden

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	34.814	8,48
Googlebot Disallow	26.300	6,4
Google AdsBot Mobile Disallow	23.629	5,75
Ahrefs Bot Disallow	20.343	4,95
GPTBot Disallow	18.680	4,55
Common Crawl Bot Disallow	18.324	4,46
Anthropic Claude Bot Disallow	17.637	4,29
AppleBot Disallow	17.596	4,28
Facebook Bot Disallow	17.050	4,15
Google Bard Disallow	17.047	4,15
Apple AI Bot Disallow	16.564	4,03
Perplexity	16.505	4,02
Cohere AI Disallow	16.318	3,97
Robots Disallow	14.705	3,58
Baidu Bot Disallow	12.838	3,13
Sogou Bot Disallow	11.056	2,69
Yandex Bot Disallow	10.120	2,46
Google AdsBot Mobile Apps Disallow	10.104	2,46
Yahoo Slurp Disallow	9.950	2,42
Majestic Bot Disallow	9.499	2,31
Alexa Bot Disallow	8.805	2,14
Microsoft BingBot Disallow	8.060	1,96
Semrush Bot Disallow	7.992	1,95
Soso Bot Disallow	7.455	1,81
ByteDance Bytespider Disallow	6.518	1,59
Webz Omgili Disallow	5.924	1,44
Some Robots Disallow	5.884	1,43
Diffbot Disallow	5.308	1,29
YOU Disallow	5.243	1,28
Google Media Partners Disallow	2.987 ^*	0,73
Microsoft MSNBot Disallow	1.991	0,48
Exalead Bot Disallow	1.144	0,28
DuckDuckGo Bot Disallow	1.004	0,24
Google Image Bot Disallow	929	0,23
Google Video Bot Disallow	454	0,11
Google News Bot Disallow	450	0,11
Google Favicon Disallow	371	0,09
Google API Bot Disallow	329	0,08
Google DuplexWeb Disallow	328	0,08
360 Bot Disallow	290	0,07
BuiltWithBot Disallow	146	0,04
Pipl Disallow	50	0,01
Youdao Bot Disallow	9	0
LLMS Text	6	0
Google FeedFetcher Disallow	3	0
Google Read Aloud Disallow	1	0