Robots.txt Usage Distribution in Jersey

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	209	8.17
GPTBot Disallow	195	7.62
Google Bard Disallow	188	7.35
Common Crawl Bot Disallow	187	7.31
Perplexity	187	7.31
AppleBot Disallow	186	7.27
Anthropic Claude Bot Disallow	183	7.15
Facebook Bot Disallow	182	7.11
Cohere AI Disallow	182	7.11
Apple AI Bot Disallow	182	7.11
Google AdsBot Mobile Apps Disallow	135	5.28
Googlebot Disallow	118	4.61
Baidu Bot Disallow	52	2.03
Sogou Bot Disallow	46	1.8
Yahoo Slurp Disallow	42	1.64
Yandex Bot Disallow	41	1.6
Microsoft BingBot Disallow	39	1.52
Google Media Partners Disallow	35 ^*	1.37
Alexa Bot Disallow	32	1.25
Soso Bot Disallow	31	1.21
Some Robots Disallow	27	1.06
Ahrefs Bot Disallow	23	0.9
Robots Disallow	11 ^*	0.43
Majestic Bot Disallow	8	0.31
Semrush Bot Disallow	6	0.23
Microsoft MSNBot Disallow	5	0.2
Google Image Bot Disallow	4	0.16
Exalead Bot Disallow	3	0.12
ByteDance Bytespider Disallow	2	0.08
Webz Omgili Disallow	2	0.08
Diffbot Disallow	2	0.08
Google Video Bot Disallow	2	0.08
Google News Bot Disallow	2	0.08
Google Favicon Disallow	2	0.08
360 Bot Disallow	1	0.04
DuckDuckGo Bot Disallow	1	0.04
YOU Disallow	1	0.04
Youdao Bot Disallow	1	0.04
Google API Bot Disallow	1	0.04
Google DuplexWeb Disallow	1	0.04
Pipl Disallow	1	0.04