Robots.txt Usage Distribution in the Top 1 Million Sites

Top In Robots.txt Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
Robots Disallow	158,185	15.82
Ahrefs Bot Disallow	69,725	6.97
Googlebot Disallow	56,385	5.64
Google AdsBot Disallow	48,850	4.88
Yandex Bot Disallow	46,646	4.66
Some Robots Disallow	39,136	3.91
GPTBot Disallow	35,653	3.57
Semrush Bot Disallow	34,606	3.46
Majestic Bot Disallow	33,578	3.36
Baidu Bot Disallow	32,187	3.22
Microsoft BingBot Disallow	28,024	2.8
Common Crawl Bot Disallow	25,309	2.53
Anthropic Claude Bot Disallow	23,218	2.32
AppleBot Disallow	21,607	2.16
Yahoo Slurp Disallow	21,357	2.14
Sogou Bot Disallow	19,347	1.93
Alexa Bot Disallow	18,867	1.89
Google Media Partners Disallow	18,580	1.86
Google Bard Disallow	17,271	1.73
Facebook Bot Disallow	15,505	1.55
ByteDance Bytespider Disallow	14,484	1.45
Perplexity	13,348	1.33
Microsoft MSNBot Disallow	13,179	1.32
Google AdsBot Mobile Disallow	12,245	1.22
Apple AI Bot Disallow	11,429	1.14
Soso Bot Disallow	11,065	1.11
Cohere AI Disallow	11,011	1.1
Google Image Bot Disallow	10,981	1.1
Webz Omgili Disallow	10,962	1.1
Exalead Bot Disallow	9,584	0.96
Diffbot Disallow	7,583	0.76
Google AdsBot Mobile Apps Disallow	6,911	0.69
Google News Bot Disallow	6,484	0.65
360 Bot Disallow	6,182	0.62
YOU Disallow	6,117	0.61
DuckDuckGo Bot Disallow	5,993	0.6
Google Video Bot Disallow	5,098	0.51
Google Favicon Disallow	4,399	0.44
Google API Bot Disallow	4,353	0.44
Google DuplexWeb Disallow	4,202	0.42
Pipl Disallow	3,795	0.38
BuiltWithBot Disallow	2,659	0.27
LLMS Text	1,510	0.15
Youdao Bot Disallow	865	0.09
Anthropic Claude User Bot Disallow	385	0.04
Anthropic Claude Search Bot Disallow	369	0.04
Google FeedFetcher Disallow	169	0.02
Google Read Aloud Disallow	107	0.01
Google Web Light Disallow	29	0
Google BIG Bench Disallow	2	0

949,536 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 01st Jul 2025.

Robots Disallow is currently the most popular technology in this category.