AI Bot Usage Distribution in the Top 1 Million Sites

Top In AI Bot Usage Distribution in the Top 1 Million Sites

Technology	Websites	%
GPTBot Disallow	34,539	3.45
Common Crawl Bot Disallow	25,232	2.52
Anthropic Claude Bot Disallow	23,101	2.31
AmazonBot Disallow	20,402	2.04
Alexa Bot Disallow	18,287	1.83
Google Bard Disallow	17,171	1.72
Facebook Bot Disallow	15,419	1.54
ByteDance Bytespider Disallow	14,388	1.44
Perplexity	13,250	1.32
Apple AI Bot Disallow	11,340	1.13
Cohere AI Disallow	10,972	1.1
Webz Omgili Disallow	10,880	1.09
Diffbot Disallow	7,500	0.75
YOU Disallow	6,033	0.6
Anthropic Claude User Bot Disallow	306	0.03

228,820 Detections

of AI Bot in the Top 1 Million Sites. Last updated 07th Jun 2025.

GPTBot Disallow is currently the most popular technology in this category.