Robots.txt Usage Distribution in Indonesia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Googlebot Disallow	21,944	11.82
Baidu Bot Disallow	17,863	9.62
Robots Disallow	16,728 ^*	9.01
Some Robots Disallow	14,730	7.93
Yahoo Slurp Disallow	14,617	7.87
Alexa Bot Disallow	12,085	6.51
Sogou Bot Disallow	8,659	4.66
Microsoft MSNBot Disallow	7,961	4.29
Google Image Bot Disallow	7,531	4.06
Google Media Partners Disallow	5,190 ^*	2.79
Google AdsBot Disallow	5,040	2.71
Yandex Bot Disallow	4,968	2.68
Ahrefs Bot Disallow	4,730	2.55
Microsoft BingBot Disallow	4,438	2.39
Majestic Bot Disallow	4,430	2.39
Soso Bot Disallow	4,060	2.19
Semrush Bot Disallow	3,344	1.8
Google AdsBot Mobile Disallow	3,315	1.79
Common Crawl Bot Disallow	2,926	1.58
GPTBot Disallow	2,809	1.51
Anthropic Claude Bot Disallow	2,363	1.27
Perplexity	1,672	0.9
Facebook Bot Disallow	1,665	0.9
Google Bard Disallow	1,577	0.85
Exalead Bot Disallow	1,514	0.82
AppleBot Disallow	1,431	0.77
Apple AI Bot Disallow	1,266	0.68
Cohere AI Disallow	1,101	0.59
Webz Omgili Disallow	934	0.5
ByteDance Bytespider Disallow	833	0.45
Google AdsBot Mobile Apps Disallow	824	0.44
Pipl Disallow	758	0.41
Diffbot Disallow	608	0.33
YOU Disallow	562	0.3
Youdao Bot Disallow	292	0.16
360 Bot Disallow	216	0.12
BuiltWithBot Disallow	149	0.08
DuckDuckGo Bot Disallow	125	0.07
Google News Bot Disallow	114	0.06
Google Video Bot Disallow	94	0.05
Google Favicon Disallow	76	0.04
Google API Bot Disallow	73	0.04
Google DuplexWeb Disallow	69	0.04
LLMS Text	5	0
Google FeedFetcher Disallow	3	0