AI Bot Usage Distribution in the Top 1 Million Sites

Top In AI Bot Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
GPTBot Disallow	35,308	3.53
Common Crawl Bot Disallow	26,065	2.61
Anthropic Claude Bot Disallow	23,296	2.33
AmazonBot Disallow	18,776	1.88
Alexa Bot Disallow	18,774	1.88
Google Bard Disallow	17,232	1.72
Facebook Bot Disallow	15,487	1.55
ByteDance Bytespider Disallow	14,446	1.44
Perplexity	13,176	1.32
Cohere AI Disallow	11,356	1.14
Apple AI Bot Disallow	11,261	1.13
Webz Omgili Disallow	10,859	1.09
Diffbot Disallow	7,265	0.73
YOU Disallow	5,573	0.56

228,874 Detections

of AI Bot in the Top 1 Million Sites. Last updated 13th May 2025.

GPTBot Disallow is currently the most popular technology in this category.