Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

2,037 Detections

of Robots.txt in the Top 1 Million Sites.

Googlebot Disallow is currently the most popular technology in this category.