Robot.txt Distribuzione dell'utilizzo in Lussemburgo

Distribuzione per siti web che utilizzano Robot.txt tecnologie

Tecnologia	Siti web	%
Googlebot Disallow	8.876	13,02
Baidu Bot Disallow	5.289	7,76
Yandex Bot Disallow	5.210	7,64
Microsoft BingBot Disallow	5.179	7,6
Sogou Bot Disallow	5.067	7,43
Yahoo Slurp Disallow	5.065	7,43
Alexa Bot Disallow	4.787	7,02
Soso Bot Disallow	4.706	6,9
Google AdsBot Disallow	4.139	6,07
Google AdsBot Mobile Disallow	3.760	5,52
Robots Disallow	2.392 ^*	3,51
Majestic Bot Disallow	1.318	1,93
Anthropic Claude Bot Disallow	1.250	1,83
Ahrefs Bot Disallow	1.123	1,65
GPTBot Disallow	1.101	1,62
Common Crawl Bot Disallow	802	1,18
Semrush Bot Disallow	795	1,17
Google Bard Disallow	763	1,12
AppleBot Disallow	758	1,11
Facebook Bot Disallow	715	1,05
Cohere AI Disallow	702	1,03
Perplexity	701	1,03
Apple AI Bot Disallow	697	1,02
Some Robots Disallow	669	0,98
Google AdsBot Mobile Apps Disallow	489	0,72
Google Media Partners Disallow	355 ^*	0,52
DuckDuckGo Bot Disallow	177	0,26
ByteDance Bytespider Disallow	148	0,22
Microsoft MSNBot Disallow	145	0,21
Google Image Bot Disallow	143	0,21
Webz Omgili Disallow	123	0,18
Diffbot Disallow	84	0,12
Exalead Bot Disallow	83	0,12
Google News Bot Disallow	76	0,11
Google Video Bot Disallow	74	0,11
Google Favicon Disallow	69	0,1
Google API Bot Disallow	66	0,1
Google DuplexWeb Disallow	66	0,1
YOU Disallow	63	0,09
360 Bot Disallow	42	0,06
LLMS Text	31	0,05
BuiltWithBot Disallow	21	0,03
Anthropic Claude User Bot Disallow	11	0,02
Anthropic Claude Search Bot Disallow	11	0,02
Pipl Disallow	10	0,01
Youdao Bot Disallow	9	0,01
Google Web Light Disallow	1	0