Robots.txt Nutzungsverteilung im gesamten Internet

Verteilung für Websites mit Robots.txt Technologien

Oben In Robots.txt Nutzungsverteilung im gesamten Internet

Technologie	Websites	%
Some Robots Disallow	39.138.226	18,11
Googlebot Disallow	17.828.157	8,25
Baidu Bot Disallow	13.581.393	6,29
Sogou Bot Disallow	12.684.407	5,87
Google Media Partners Disallow	12.427.091	5,75
Microsoft BingBot Disallow	12.311.912	5,7
Yandex Bot Disallow	12.258.509	5,67
Yahoo Slurp Disallow	11.474.186	5,31
Google AdsBot Disallow	9.790.845	4,53
Robots Disallow	9.099.895	4,21
Google AdsBot Mobile Disallow	7.817.180	3,62
Alexa Bot Disallow	5.217.515	2,41
Soso Bot Disallow	5.040.372	2,33
GPTBot Disallow	4.075.441	1,89
Ahrefs Bot Disallow	4.019.066	1,86
AppleBot Disallow	3.990.455	1,85
Common Crawl Bot Disallow	3.838.613	1,78
Anthropic Claude Bot Disallow	3.714.262	1,72
Google Bard Disallow	3.559.411	1,65
Facebook Bot Disallow	3.430.596	1,59
Apple AI Bot Disallow	3.410.037	1,58
Cohere AI Disallow	3.328.080	1,54
Perplexity	3.285.151	1,52
Majestic Bot Disallow	2.277.342	1,05
Semrush Bot Disallow	2.137.525	0,99
Google AdsBot Mobile Apps Disallow	2.080.345	0,96
ByteDance Bytespider Disallow	752.357	0,35
Exalead Bot Disallow	465.497	0,22
Microsoft MSNBot Disallow	448.759	0,21
Webz Omgili Disallow	396.357	0,18
Google Image Bot Disallow	363.858	0,17
DuckDuckGo Bot Disallow	288.467	0,13
Diffbot Disallow	272.325	0,13
360 Bot Disallow	257.988	0,12
LLMS Text	193.522	0,09
YOU Disallow	168.669	0,08
BuiltWithBot Disallow	115.744	0,05
Youdao Bot Disallow	92.725	0,04
Google Video Bot Disallow	92.443	0,04
Google News Bot Disallow	91.702	0,04
Google Favicon Disallow	73.221	0,03
Google API Bot Disallow	47.055	0,02
Google DuplexWeb Disallow	41.774	0,02
Pipl Disallow	36.661	0,02
Anthropic Claude User Bot Disallow	17.173	0,01
Anthropic Claude Search Bot Disallow	13.248	0,01
Google Read Aloud Disallow	5.732	0
Google FeedFetcher Disallow	4.220	0
Google Web Light Disallow	758	0
Google BIG Bench Disallow	41	0