Robots.txt Répartition des utilisations aux Samoa

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Googlebot Disallow	4 740	11,8
Baidu Bot Disallow	4 691	11,68
Sogou Bot Disallow	4 634	11,54
Yahoo Slurp Disallow	4 501	11,21
Alexa Bot Disallow	3 975	89,55
Soso Bot Disallow	3 929	9,78
Some Robots Disallow	3 727	9,28
Yandex Bot Disallow	3 547	8,83
Microsoft BingBot Disallow	3 522	8,77
Robots Disallow	879	2,19
Google Media Partners Disallow	826 ^*	2,06
Ahrefs Bot Disallow	114	0,28
Google AdsBot Disallow	96	0,24
Majestic Bot Disallow	90	0,22
Semrush Bot Disallow	72	0,18
Common Crawl Bot Disallow	67	1,51
GPTBot Disallow	61	1,37
Microsoft MSNBot Disallow	60	0,15
AppleBot Disallow	51	0,13
Anthropic Claude Bot Disallow	48	1,08
Google Bard Disallow	48	1,08
Exalead Bot Disallow	46	0,11
Google Image Bot Disallow	42	0,1
Facebook Bot Disallow	41	0,92
Apple AI Bot Disallow	39	0,88
360 Bot Disallow	33	0,08
Webz Omgili Disallow	29	0,65
Perplexity	24	0,54
Cohere AI Disallow	23	0,52
Google AdsBot Mobile Apps Disallow	23	0,06
Google News Bot Disallow	21	0,05
Google Video Bot Disallow	21	0,05
Google Favicon Disallow	21	0,05
ByteDance Bytespider Disallow	20	0,45
DuckDuckGo Bot Disallow	19	0,05
Google API Bot Disallow	16	0,04
Google DuplexWeb Disallow	16	0,04
Diffbot Disallow	14	0,32
BuiltWithBot Disallow	11	0,03
YOU Disallow	9	0,2
Google Read Aloud Disallow	5	0,01
Pipl Disallow	4	0,01
Google FeedFetcher Disallow	4	0,01
Google Web Light Disallow	4	0,01
Youdao Bot Disallow	2	0
LLMS Text	1	0