Robots.txt Usage Distribution in Australia

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	255,551	9.33
Googlebot Disallow	182,665	6.67
Google AdsBot Mobile Disallow	181,742	6.64
GPTBot Disallow	136,231	4.98
AppleBot Disallow	135,094	4.93
Common Crawl Bot Disallow	133,794	4.89
Anthropic Claude Bot Disallow	132,144	4.83
Google Bard Disallow	131,703	4.81
Facebook Bot Disallow	131,398	4.8
Apple AI Bot Disallow	130,705	4.77
Perplexity	130,165	4.75
Cohere AI Disallow	130,137	4.75
Baidu Bot Disallow	126,761	4.63
Microsoft BingBot Disallow	100,717	3.68
Google AdsBot Mobile Apps Disallow	98,541	3.6
Ahrefs Bot Disallow	90,488	3.31
Sogou Bot Disallow	87,830	3.21
Yahoo Slurp Disallow	82,864	3.03
Yandex Bot Disallow	66,299	2.42
Alexa Bot Disallow	65,128	2.38
Soso Bot Disallow	61,578	2.25
Some Robots Disallow	50,286	1.84
Robots Disallow	44,525 ^*	1.63
Majestic Bot Disallow	12,746	0.47
Semrush Bot Disallow	10,954	0.4
Google Media Partners Disallow	5,330 ^*	0.19
Microsoft MSNBot Disallow	3,681	0.13
ByteDance Bytespider Disallow	3,385	0.12
Exalead Bot Disallow	3,140	0.11
Webz Omgili Disallow	2,484	0.09
Google Image Bot Disallow	2,362	0.09
Diffbot Disallow	1,495	0.05
360 Bot Disallow	1,395	0.05
Pipl Disallow	729	0.03
DuckDuckGo Bot Disallow	610	0.02
Google News Bot Disallow	543	0.02
YOU Disallow	532	0.02
Google Video Bot Disallow	478	0.02
BuiltWithBot Disallow	414	0.02
Google Favicon Disallow	393	0.01
Google API Bot Disallow	221	0.01
Google DuplexWeb Disallow	202	0.01
Youdao Bot Disallow	129	0
LLMS Text	30	0
Google FeedFetcher Disallow	15	0
Google Read Aloud Disallow	7	0
Google Web Light Disallow	4	0