Robots.txt Nutzungsverteilung in Marokko

Verteilung für Websites mit Robots.txt Technologien

Technologie	Websites	%
Google AdsBot Disallow	1.630	14,16
Robots Disallow	1.347	11,7
Ahrefs Bot Disallow	1.283	11,15
Googlebot Disallow	1.024	8,9
Google AdsBot Mobile Disallow	572	4,97
Baidu Bot Disallow	455	3,95
Sogou Bot Disallow	405	3,52
Yahoo Slurp Disallow	351	3,05
Yandex Bot Disallow	328	2,85
Alexa Bot Disallow	311	2,7
Microsoft BingBot Disallow	295	2,56
Soso Bot Disallow	293	2,55
Some Robots Disallow	280	2,43
GPTBot Disallow	248	2,16
Common Crawl Bot Disallow	237	2,06
AppleBot Disallow	227	1,97
Google Bard Disallow	222	1,93
Anthropic Claude Bot Disallow	212	1,84
Facebook Bot Disallow	209	1,82
Apple AI Bot Disallow	203	1,76
Perplexity	202	1,76
Cohere AI Disallow	195	1,69
Google AdsBot Mobile Apps Disallow	164	1,43
Majestic Bot Disallow	159	1,38
Google Media Partners Disallow	159 ^*	1,38
Semrush Bot Disallow	149	1,29
Microsoft MSNBot Disallow	38	0,33
Google Image Bot Disallow	38	0,33
ByteDance Bytespider Disallow	34	0,3
Exalead Bot Disallow	31	0,27
Webz Omgili Disallow	30	0,26
DuckDuckGo Bot Disallow	29	0,25
Diffbot Disallow	25	0,22
YOU Disallow	23	0,2
360 Bot Disallow	16	0,14
Google News Bot Disallow	14	0,12
Google Video Bot Disallow	12	0,1
Google Favicon Disallow	12	0,1
Google API Bot Disallow	12	0,1
Google DuplexWeb Disallow	12	0,1
Pipl Disallow	9	0,08
BuiltWithBot Disallow	8	0,07
Youdao Bot Disallow	4	0,03
LLMS Text	1	0,01