Robots.txt Usage Distribution in the Top 1 Million Sites

Top In Robots.txt Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
Googlebot Disallow	1,197	0.12
Robots Disallow	309	0.03
Baidu Bot Disallow	175	0.02
Alexa Bot Disallow	154	0.02
Exalead Bot Disallow	64	0.01
Microsoft BingBot Disallow	41	0
Yahoo Slurp Disallow	40	0
Yandex Bot Disallow	34	0
Microsoft MSNBot Disallow	22	0
DuckDuckGo Bot Disallow	1	0

2,037 Detections

of Robots.txt in the Top 1 Million Sites.

Googlebot Disallow is currently the most popular technology in this category.