Robot.txt Distribuzione dell'utilizzo nello Sri Lanka

Distribuzione per siti web che utilizzano Robot.txt tecnologie

Tecnologia	Siti web	%
Robots Disallow	1.023	13,88
Google AdsBot Disallow	855	11,6
Googlebot Disallow	811	11,01
Google AdsBot Mobile Disallow	521	7,07
Ahrefs Bot Disallow	453	6,15
Baidu Bot Disallow	269	3,65
Google Media Partners Disallow	238 ^*	3,23
Some Robots Disallow	233	3,16
Sogou Bot Disallow	233	3,16
Yahoo Slurp Disallow	185	2,51
Alexa Bot Disallow	185	11,95
Yandex Bot Disallow	183	2,48
GPTBot Disallow	183	11,82
AppleBot Disallow	166	2,25
Common Crawl Bot Disallow	165	10,66
Google Bard Disallow	164	10,59
Anthropic Claude Bot Disallow	159	10,27
Facebook Bot Disallow	157	10,14
Apple AI Bot Disallow	157	10,14
Perplexity	154	9,95
Soso Bot Disallow	151	2,05
Cohere AI Disallow	150	9,69
Microsoft BingBot Disallow	146	1,98
Google AdsBot Mobile Apps Disallow	133	1,8
Semrush Bot Disallow	96	1,3
Majestic Bot Disallow	74	1
Google Image Bot Disallow	24	0,33
ByteDance Bytespider Disallow	18	1,16
Webz Omgili Disallow	12	0,78
Exalead Bot Disallow	10	0,14
Microsoft MSNBot Disallow	10	0,14
Diffbot Disallow	8	0,52
DuckDuckGo Bot Disallow	8	0,11
YOU Disallow	7	0,45
360 Bot Disallow	5	0,07
Google News Bot Disallow	5	0,07
BuiltWithBot Disallow	3	0,04
Google Video Bot Disallow	3	0,04
Google Favicon Disallow	3	0,04
Google API Bot Disallow	3	0,04
Google DuplexWeb Disallow	3	0,04
Youdao Bot Disallow	1	0,01
Pipl Disallow	1	0,01
LLMS Text	1	0,01