Robots.txt Répartition des utilisations dans le Top 1 Million de sites

Distribution pour les sites Web utilisant Robots.txt technologies

Haut de page Robots.txt Répartition des utilisations dans le Top 1 Million de sites

Technologie	Sites Web	%
Robots Disallow	154 655	15,47
Ahrefs Bot Disallow	69 338	6,93
Googlebot Disallow	55 267	5,53
Google AdsBot Disallow	48 731	4,87
Yandex Bot Disallow	45 588	4,56
Some Robots Disallow	38 359	3,84
GPTBot Disallow	34 539	3,45
Semrush Bot Disallow	34 212	3,42
Majestic Bot Disallow	33 230	3,32
Baidu Bot Disallow	31 393	3,14
Microsoft BingBot Disallow	27 160	2,72
Common Crawl Bot Disallow	25 232	2,52
Anthropic Claude Bot Disallow	23 101	2,31
Yahoo Slurp Disallow	20 597	2,06
AppleBot Disallow	20 580	2,06
Sogou Bot Disallow	18 614	1,86
Alexa Bot Disallow	18 287	1,83
Google Media Partners Disallow	17 850	1,78
Google Bard Disallow	17 171	1,72
Facebook Bot Disallow	15 419	1,54
ByteDance Bytespider Disallow	14 388	1,44
Perplexity	13 250	1,32
Microsoft MSNBot Disallow	13 126	1,31
Google AdsBot Mobile Disallow	12 158	1,22
Apple AI Bot Disallow	11 340	1,13
Cohere AI Disallow	10 972	1,1
Webz Omgili Disallow	10 880	1,09
Google Image Bot Disallow	10 879	1,09
Soso Bot Disallow	10 576	1,06
Exalead Bot Disallow	9 530	0,95
Diffbot Disallow	7 500	0,75
Google AdsBot Mobile Apps Disallow	6 911	0,69
Google News Bot Disallow	6 433	0,64
360 Bot Disallow	6 152	0,62
YOU Disallow	6 033	0,6
DuckDuckGo Bot Disallow	5 927	0,59
Google Video Bot Disallow	5 063	0,51
Google Favicon Disallow	4 375	0,44
Google API Bot Disallow	4 325	0,43
Google DuplexWeb Disallow	4 178	0,42
Pipl Disallow	3 745	0,37
BuiltWithBot Disallow	2 647	0,26
Youdao Bot Disallow	863	0,09
LLMS Text	719	0,07
Anthropic Claude User Bot Disallow	306	0,03
Anthropic Claude Search Bot Disallow	299	0,03
Google FeedFetcher Disallow	164	0,02
Google Read Aloud Disallow	105	0,01
Google Web Light Disallow	29	0
Google BIG Bench Disallow	2	0