Robots.txt Nutzungsverteilung in den Top 1 Million Websites

Verteilung für Websites mit Robots.txt Technologien

Oben In Robots.txt Nutzungsverteilung in den Top 1 Million Websites

Technologie	Websites	%
Robots Disallow	158.577	15,86
Ahrefs Bot Disallow	70.002	7
Googlebot Disallow	56.550	5,66
Google AdsBot Disallow	48.878	4,89
Yandex Bot Disallow	46.771	4,68
Some Robots Disallow	39.333	3,93
GPTBot Disallow	38.247	3,82
Semrush Bot Disallow	34.786	3,48
Majestic Bot Disallow	33.728	3,37
Baidu Bot Disallow	32.294	3,23
Microsoft BingBot Disallow	28.151	2,82
Common Crawl Bot Disallow	27.097	2,71
Anthropic Claude Bot Disallow	24.783	2,48
AppleBot Disallow	22.767	2,28
Yahoo Slurp Disallow	21.454	2,15
Sogou Bot Disallow	19.438	1,94
Alexa Bot Disallow	18.867	1,89
Google Bard Disallow	18.736	1,87
Google Media Partners Disallow	18.673	1,87
Facebook Bot Disallow	15.642	1,56
ByteDance Bytespider Disallow	15.611	1,56
Perplexity	13.909	1,39
Microsoft MSNBot Disallow	13.228	1,32
Apple AI Bot Disallow	12.507	1,25
Google AdsBot Mobile Disallow	12.257	1,23
Soso Bot Disallow	11.264	1,13
Cohere AI Disallow	11.130	1,11
Webz Omgili Disallow	11.077	1,11
Google Image Bot Disallow	11.051	1,11
Exalead Bot Disallow	9.612	0,96
Diffbot Disallow	7.694	0,77
Google AdsBot Mobile Apps Disallow	6.911	0,69
Google News Bot Disallow	6.522	0,65
YOU Disallow	6.454	0,65
360 Bot Disallow	6.194	0,62
DuckDuckGo Bot Disallow	6.048	0,6
Google Video Bot Disallow	5.131	0,51
Google Favicon Disallow	4.426	0,44
Google API Bot Disallow	4.382	0,44
Google DuplexWeb Disallow	4.229	0,42
Pipl Disallow	3.826	0,38
BuiltWithBot Disallow	2.664	0,27
LLMS Text	1.614	0,16
Youdao Bot Disallow	870	0,09
Anthropic Claude User Bot Disallow	828	0,08
Anthropic Claude Search Bot Disallow	785	0,08
Google FeedFetcher Disallow	174	0,02
Google Read Aloud Disallow	110	0,01
Google Web Light Disallow	29	0
Google BIG Bench Disallow	2	0