AI Bot Usage Distribution in the Top 10k Sites

Top In AI Bot Usage Distribution in the Top 10k Sites

Technology	Websites	%
GPTBot Disallow	1,396	13.96
Common Crawl Bot Disallow	1,051	10.51
Anthropic Claude Bot Disallow	947	9.47
Google Bard Disallow	807	8.07
ByteDance Bytespider Disallow	746	7.46
Perplexity	643	6.43
Facebook Bot Disallow	629	6.29
AmazonBot Disallow	629	6.29
Webz Omgili Disallow	584	5.84
Apple AI Bot Disallow	561	5.61
Cohere AI Disallow	558	5.58
Alexa Bot Disallow	436	4.36
Diffbot Disallow	394	3.94
YOU Disallow	306	3.06
Anthropic Claude User Bot Disallow	37	0.37

9,724 Detections

of AI Bot in the Top 10k Sites. Last updated 01st Jul 2025.

GPTBot Disallow is currently the most popular technology in this category.