Robot.txt Distribuzione dell'utilizzo tra i primi 1 milione di siti

Distribuzione per siti web che utilizzano Robot.txt tecnologie

Tecnologia	Siti web	%
Robots Disallow	163.986	16,4
Ahrefs Bot Disallow	72.684	7,27
Googlebot Disallow	58.627	5,86
Google AdsBot Disallow	51.286	5,13
Yandex Bot Disallow	46.685	4,67
Some Robots Disallow	41.343	4,13
GPTBot Disallow	35.302	3,53
Semrush Bot Disallow	35.040	3,5
Majestic Bot Disallow	34.628	3,46
Baidu Bot Disallow	33.548	3,35
Microsoft BingBot Disallow	27.523	2,75
Common Crawl Bot Disallow	26.338	2,63
AppleBot Disallow	23.111	2,31
Anthropic Claude Bot Disallow	22.972	2,3
Yahoo Slurp Disallow	20.876	2,09
Sogou Bot Disallow	19.793	1,98
Google Media Partners Disallow	19.710	1,97
Alexa Bot Disallow	18.728	1,87
Google Bard Disallow	17.056	1,71
Facebook Bot Disallow	15.316	1,53
Microsoft MSNBot Disallow	14.243	1,42
ByteDance Bytespider Disallow	14.102	1,41
Google AdsBot Mobile Disallow	13.106	1,31
Perplexity	12.815	1,28
Google Image Bot Disallow	11.856	1,19
Cohere AI Disallow	11.422	1,14
Apple AI Bot Disallow	10.908	1,09
Soso Bot Disallow	10.724	1,07
Webz Omgili Disallow	10.541	1,05
Exalead Bot Disallow	10.095	1,01
Google AdsBot Mobile Apps Disallow	8.193	0,82
Google News Bot Disallow	7.230	0,72
Diffbot Disallow	6.853	0,69
DuckDuckGo Bot Disallow	6.642	0,66
360 Bot Disallow	6.481	0,65
Google Video Bot Disallow	5.774	0,58
YOU Disallow	5.127	0,51
Google Favicon Disallow	5.053	0,51
Google API Bot Disallow	4.999	0,5
Google DuplexWeb Disallow	4.857	0,49
Pipl Disallow	2.918	0,29
BuiltWithBot Disallow	2.867	0,29
Youdao Bot Disallow	919	0,09
LLMS Text	352	0,04
Google FeedFetcher Disallow	169	0,02
Google Read Aloud Disallow	85	0,01
Google Web Light Disallow	31	0
Google BIG Bench Disallow	5	0