Robots.txt Nutzungsverteilung in Portugal

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	19.526	12,96
Googlebot Disallow	15.990	10,61
Google AdsBot Mobile Disallow	13.024	8,64
Ahrefs Bot Disallow	10.155	6,74
Robots Disallow	8.211	5,45
GPTBot Disallow	5.067	3,36
Anthropic Claude Bot Disallow	4.970	3,3
Yandex Bot Disallow	4.900	3,25
Common Crawl Bot Disallow	4.533	3,01
AppleBot Disallow	4.311	2,86
Google Bard Disallow	4.298	2,85
Facebook Bot Disallow	4.202	2,79
Baidu Bot Disallow	4.185	2,78
Apple AI Bot Disallow	3.996	2,65
Perplexity	3.969	2,63
Cohere AI Disallow	3.943	2,62
Yahoo Slurp Disallow	3.726	2,47
Sogou Bot Disallow	3.694	2,45
Google AdsBot Mobile Apps Disallow	3.420	2,27
Alexa Bot Disallow	3.313	2,2
Majestic Bot Disallow	3.234	2,15
Some Robots Disallow	3.207	2,13
Microsoft BingBot Disallow	3.126	2,07
Soso Bot Disallow	2.720	1,8
Semrush Bot Disallow	2.599	1,72
Google Media Partners Disallow	1.763 ^*	1,17
Microsoft MSNBot Disallow	1.023	0,68
ByteDance Bytespider Disallow	666	0,44
Exalead Bot Disallow	571	0,38
Google Image Bot Disallow	501	0,33
Webz Omgili Disallow	427	0,28
Diffbot Disallow	202	0,13
360 Bot Disallow	192	0,13
DuckDuckGo Bot Disallow	164	0,11
Google News Bot Disallow	140	0,09
Google Video Bot Disallow	125	0,08
YOU Disallow	119	0,08
Google Favicon Disallow	112	0,07
Google API Bot Disallow	97	0,06
Google DuplexWeb Disallow	95	0,06
BuiltWithBot Disallow	72	0,05
Pipl Disallow	67	0,04
Youdao Bot Disallow	33	0,02
LLMS Text	8	0,01
Google FeedFetcher Disallow	3	0
Google Web Light Disallow	3	0
Google Read Aloud Disallow	2	0