Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	57,599	5.76
Ahrefs Bot Disallow	55,947	5.59
Robots Disallow	53,342	5.33
Googlebot Disallow	35,632	3.56
Yandex Bot Disallow	31,593	3.16
Baidu Bot Disallow	23,606	2.36
Some Robots Disallow	23,387	2.34
Majestic Bot Disallow	23,056	2.31
Semrush Bot Disallow	21,365	2.14
Google AdsBot Mobile Disallow	20,967	2.1
Microsoft BingBot Disallow	15,878	1.59
Yahoo Slurp Disallow	14,058	1.41
Google AdsBot Mobile Apps Disallow	11,636	1.16
Microsoft MSNBot Disallow	10,956	1.1
Alexa Bot Disallow	10,356	1.04
Sogou Bot Disallow	9,974	1
Google Media Partners Disallow	8,289	0.83
Soso Bot Disallow	8,037	0.8
Exalead Bot Disallow	6,562	0.66
Google Image Bot Disallow	5,860	0.59