Robots.txt Répartition des utilisations en Islande

Distribution pour les sites Web utilisant Robots.txt technologies

Technologie	Sites Web	%
Google AdsBot Disallow	3 674	12,44
Googlebot Disallow	2 735	9,26
Google AdsBot Mobile Disallow	2 485	8,42
Ahrefs Bot Disallow	1 443	4,89
GPTBot Disallow	1 299	4,4
AppleBot Disallow	1 204	4,08
Robots Disallow	1 193	4,04
Common Crawl Bot Disallow	1 178	3,99
Anthropic Claude Bot Disallow	1 156	3,92
Google Bard Disallow	1 152	3,9
Facebook Bot Disallow	1 139	3,86
Baidu Bot Disallow	1 131	3,83
Apple AI Bot Disallow	1 126	3,81
Perplexity	1 097	3,72
Cohere AI Disallow	1 092	3,7
Sogou Bot Disallow	1 042	3,53
Google AdsBot Mobile Apps Disallow	956	3,24
Some Robots Disallow	900	3,05
Google Media Partners Disallow	572 ^*	1,94
Yahoo Slurp Disallow	514	1,74
Yandex Bot Disallow	413	1,4
Microsoft BingBot Disallow	406	1,38
Alexa Bot Disallow	361	1,22
Soso Bot Disallow	313	1,06
Majestic Bot Disallow	227	0,77
Semrush Bot Disallow	226	0,77
ByteDance Bytespider Disallow	83	0,28
Webz Omgili Disallow	79	0,27
Microsoft MSNBot Disallow	48	0,16
Diffbot Disallow	37	0,13
Google Image Bot Disallow	35	0,12
YOU Disallow	33	0,11
Exalead Bot Disallow	24	0,08
Google News Bot Disallow	24	0,08
Google Video Bot Disallow	24	0,08
Google Favicon Disallow	24	0,08
DuckDuckGo Bot Disallow	16	0,05
Google API Bot Disallow	16	0,05
Google DuplexWeb Disallow	16	0,05
Pipl Disallow	15	0,05
360 Bot Disallow	8	0,03
Youdao Bot Disallow	6	0,02
BuiltWithBot Disallow	3	0,01
LLMS Text	1	0
Google BIG Bench Disallow	1	0