Robots.txt Usage Distribution in the Top 1 Million Sites

Top In Robots.txt Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
Robots Disallow	160,200	16.02
Ahrefs Bot Disallow	70,094	7.01
Googlebot Disallow	57,152	5.72
Google AdsBot Disallow	48,883	4.89
Yandex Bot Disallow	47,243	4.72
Some Robots Disallow	39,785	3.98
GPTBot Disallow	39,657	3.97
Semrush Bot Disallow	34,878	3.49
Majestic Bot Disallow	33,809	3.38
Baidu Bot Disallow	32,724	3.27
Microsoft BingBot Disallow	28,591	2.86
Common Crawl Bot Disallow	28,554	2.86
Anthropic Claude Bot Disallow	27,879	2.79
AppleBot Disallow	23,533	2.35
Google Bard Disallow	21,872	2.19
Yahoo Slurp Disallow	21,843	2.18
Sogou Bot Disallow	19,799	1.98
Alexa Bot Disallow	19,335	1.93
Google Media Partners Disallow	19,032	1.9
ByteDance Bytespider Disallow	16,975	1.7
Perplexity	15,862	1.59
Facebook Bot Disallow	15,855	1.59
Apple AI Bot Disallow	13,858	1.39
Microsoft MSNBot Disallow	13,233	1.32
Google AdsBot Mobile Disallow	12,308	1.23
Soso Bot Disallow	11,448	1.14
Cohere AI Disallow	11,240	1.12
Webz Omgili Disallow	11,186	1.12
Google Image Bot Disallow	11,067	1.11
Exalead Bot Disallow	9,625	0.96
Diffbot Disallow	7,809	0.78
Google AdsBot Mobile Apps Disallow	6,911	0.69
YOU Disallow	6,650	0.66
Google News Bot Disallow	6,532	0.65
360 Bot Disallow	6,200	0.62
DuckDuckGo Bot Disallow	6,058	0.61
Google Video Bot Disallow	5,138	0.51
Google Favicon Disallow	4,429	0.44
Google API Bot Disallow	4,386	0.44
Google DuplexWeb Disallow	4,232	0.42
Pipl Disallow	3,855	0.39
Anthropic Claude User Bot Disallow	2,719	0.27
BuiltWithBot Disallow	2,666	0.27
Anthropic Claude Search Bot Disallow	2,635	0.26
LLMS Text	1,999	0.2
Youdao Bot Disallow	872	0.09
Google FeedFetcher Disallow	174	0.02
Google Read Aloud Disallow	111	0.01
Google Web Light Disallow	29	0
Google BIG Bench Disallow	2	0

990,927 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 01st Jul 2025.

Robots Disallow is currently the most popular technology in this category.