Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Some Robots Disallow	23,396,165	30.97
Google AdsBot Disallow	8,391,198	11.11
Google AdsBot Mobile Disallow	6,968,691	9.22
Googlebot Disallow	5,304,775	7.02
Baidu Bot Disallow	4,541,900	6.01
Robots Disallow	3,843,280	5.09
Majestic Bot Disallow	3,179,190	4.21
Ahrefs Bot Disallow	2,781,536	3.68
Google Media Partners Disallow	2,658,808	3.52
Google AdsBot Mobile Apps Disallow	2,294,137	3.04
Yandex Bot Disallow	2,181,987	2.89
Microsoft BingBot Disallow	2,008,777	2.66
Yahoo Slurp Disallow	1,763,119	2.33
Sogou Bot Disallow	1,389,968	1.84
Soso Bot Disallow	1,340,669	1.77
Semrush Bot Disallow	1,022,516	1.35
Alexa Bot Disallow	733,846	0.97
Microsoft MSNBot Disallow	506,401	0.67
Exalead Bot Disallow	481,923	0.64
Google Image Bot Disallow	362,864	0.48