Robots.txt Répartition des utilisations à Palau

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Robots Disallow	2 926	16,41
Googlebot Disallow	2 237	12,54
Baidu Bot Disallow	2 010	11,27
Sogou Bot Disallow	1 980	11,1
Some Robots Disallow	1 623	9,1
Yahoo Slurp Disallow	1 175	6,59
Alexa Bot Disallow	960	68,92
Yandex Bot Disallow	938	5,26
Soso Bot Disallow	924	5,18
Google Media Partners Disallow	906 ^*	5,08
Microsoft BingBot Disallow	898	5,04
Google AdsBot Disallow	221	1,24
Ahrefs Bot Disallow	123	0,69
Majestic Bot Disallow	116	0,65
Semrush Bot Disallow	92	0,52
GPTBot Disallow	73	5,24
Common Crawl Bot Disallow	60	4,31
Microsoft MSNBot Disallow	54	0,3
Anthropic Claude Bot Disallow	40	2,87
360 Bot Disallow	40	0,22
Google Bard Disallow	38	2,73
Facebook Bot Disallow	36	2,58
AppleBot Disallow	35	0,2
Apple AI Bot Disallow	32	2,3
Exalead Bot Disallow	30	0,17
Google Image Bot Disallow	29	0,16
Cohere AI Disallow	26	1,87
ByteDance Bytespider Disallow	26	1,87
Perplexity	25	1,79
Webz Omgili Disallow	22	1,58
Google AdsBot Mobile Apps Disallow	20	0,11
BuiltWithBot Disallow	19	0,11
DuckDuckGo Bot Disallow	13	0,07
Diffbot Disallow	12	0,86
Google News Bot Disallow	10	0,06
Google Video Bot Disallow	10	0,06
Youdao Bot Disallow	10	0,06
Google Favicon Disallow	10	0,06
Google API Bot Disallow	10	0,06
Google DuplexWeb Disallow	10	0,06
YOU Disallow	8	0,57
Pipl Disallow	5	0,03
Google Read Aloud Disallow	1	0,01
Google FeedFetcher Disallow	1	0,01
Google Web Light Disallow	1	0,01