Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Robots Disallow	98,074	9.81
Googlebot Disallow	49,343	4.93
Yandex Bot Disallow	45,123	4.51
Ahrefs Bot Disallow	45,067	4.51
Google AdsBot Disallow	31,618	3.16
Baidu Bot Disallow	29,219	2.92
Some Robots Disallow	27,874	2.79
Semrush Bot Disallow	25,071	2.51
Microsoft BingBot Disallow	23,280	2.33
Majestic Bot Disallow	23,024	2.3
Yahoo Slurp Disallow	20,180	2.02
Sogou Bot Disallow	15,451	1.55
Soso Bot Disallow	13,830	1.38
Google AdsBot Mobile Disallow	12,282	1.23
Microsoft MSNBot Disallow	11,441	1.14
Google Media Partners Disallow	10,982	1.1
Alexa Bot Disallow	10,539	1.05
Google Image Bot Disallow	8,176	0.82
Google AdsBot Mobile Apps Disallow	7,382	0.74
Exalead Bot Disallow	7,339	0.73
Google News Bot Disallow	4,888	0.49
DuckDuckGo Bot Disallow	3,773	0.38
Google Video Bot Disallow	3,484	0.35
360 Bot Disallow	3,291	0.33
Google API Bot Disallow	3,096	0.31
Google DuplexWeb Disallow	3,066	0.31
Google Favicon Disallow	3,038	0.3
BuiltWithBot Disallow	1,736	0.17
AppleBot Disallow	1,350	0.14
Youdao Bot Disallow	751	0.08
Google FeedFetcher Disallow	73	0.01
Google Read Aloud Disallow	7	0
Google Web Light Disallow	6	0