AI Bot Usage Distribution in the Top 10k Sites

Top In AI Bot Usage Distribution in the Top 10k Sites

Technology	Websites	%
GPTBot Disallow	1,356	13.56
Common Crawl Bot Disallow	1,011	10.11
Anthropic Claude Bot Disallow	917	9.17
Google Bard Disallow	778	7.78
ByteDance Bytespider Disallow	710	7.1
AmazonBot Disallow	620	6.2
Perplexity	618	6.18
Facebook Bot Disallow	612	6.12
Webz Omgili Disallow	569	5.69
Cohere AI Disallow	542	5.42
Apple AI Bot Disallow	531	5.31
Alexa Bot Disallow	433	4.33
Diffbot Disallow	380	3.8
YOU Disallow	280	2.8
Anthropic Claude User Bot Disallow	18	0.18

9,375 Detections

of AI Bot in the Top 10k Sites. Last updated 01st Jul 2025.

GPTBot Disallow is currently the most popular technology in this category.