Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	98,467	9.85
Googlebot Disallow	56,406	5.64
Ahrefs Bot Disallow	53,813	5.38
Yandex Bot Disallow	51,789	5.18
Google AdsBot Disallow	36,380	3.64
Baidu Bot Disallow	33,781	3.38
Some Robots Disallow	30,359	3.04
Microsoft BingBot Disallow	27,591	2.76
Semrush Bot Disallow	27,023	2.7
Majestic Bot Disallow	25,966	2.6
Yahoo Slurp Disallow	24,232	2.42
Sogou Bot Disallow	21,553	2.16
Soso Bot Disallow	19,675	1.97
Google Media Partners Disallow	11,680	1.17
Microsoft MSNBot Disallow	10,919	1.09
Google AdsBot Mobile Disallow	10,684	1.07
Alexa Bot Disallow	10,213	1.02
Exalead Bot Disallow	9,107	0.91
Google Image Bot Disallow	8,843	0.88
GPTBot Disallow	7,283	0.73
Google AdsBot Mobile Apps Disallow	6,755	0.68
360 Bot Disallow	5,673	0.57
Google News Bot Disallow	5,516	0.55
DuckDuckGo Bot Disallow	4,484	0.45
Google Video Bot Disallow	4,149	0.41
Google API Bot Disallow	3,725	0.37
Google DuplexWeb Disallow	3,683	0.37
Google Favicon Disallow	3,674	0.37
BuiltWithBot Disallow	3,625	0.36
AppleBot Disallow	1,648	0.16
Youdao Bot Disallow	840	0.08
Google FeedFetcher Disallow	59	0.01
Google Read Aloud Disallow	13	0
Google Web Light Disallow	8	0
Google BIG Bench Disallow	2	0