Robots.txt Nutzungsverteilung in Jordan

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	543	12,22
Googlebot Disallow	442	9,95
Google AdsBot Mobile Disallow	391	8,8
Robots Disallow	308	6,93
Ahrefs Bot Disallow	241	5,42
GPTBot Disallow	157	3,53
Common Crawl Bot Disallow	153	3,44
AppleBot Disallow	149	3,35
Anthropic Claude Bot Disallow	148	3,33
Google Bard Disallow	147	3,31
Baidu Bot Disallow	145	3,26
Facebook Bot Disallow	145	3,26
Cohere AI Disallow	142	3,2
Apple AI Bot Disallow	140	3,15
Perplexity	139	3,13
Some Robots Disallow	127	2,86
Google AdsBot Mobile Apps Disallow	126	2,84
Sogou Bot Disallow	119	2,68
Yandex Bot Disallow	97	2,18
Yahoo Slurp Disallow	97	2,18
Alexa Bot Disallow	81	1,82
Microsoft BingBot Disallow	80	1,8
Soso Bot Disallow	68	1,53
Google Media Partners Disallow	64 ^*	1,44
Semrush Bot Disallow	50	1,13
Majestic Bot Disallow	42	0,95
Microsoft MSNBot Disallow	21	0,47
Google Image Bot Disallow	13	0,29
ByteDance Bytespider Disallow	10	0,23
Webz Omgili Disallow	9	0,2
Google Video Bot Disallow	8	0,18
Google News Bot Disallow	7	0,16
Exalead Bot Disallow	6	0,14
BuiltWithBot Disallow	4	0,09
DuckDuckGo Bot Disallow	4	0,09
Google Favicon Disallow	4	0,09
360 Bot Disallow	3	0,07
YOU Disallow	3	0,07
Google API Bot Disallow	3	0,07
Google DuplexWeb Disallow	3	0,07
Diffbot Disallow	2	0,05
Youdao Bot Disallow	1	0,02
Pipl Disallow	1	0,02