Robots.txt Usage Distribution in Jersey

Distribution for websites using Robots.txt technologies

Technology	Websites	%
Google AdsBot Disallow	209	8.37
GPTBot Disallow	178	7.13
AppleBot Disallow	171	6.85
Google Bard Disallow	171	6.85
Common Crawl Bot Disallow	169	6.77
Perplexity	169	6.77
Anthropic Claude Bot Disallow	166	6.65
Facebook Bot Disallow	165	6.61
Cohere AI Disallow	165	6.61
Apple AI Bot Disallow	164	6.57
Google AdsBot Mobile Apps Disallow	144	5.77
Googlebot Disallow	118	4.73
Robots Disallow	82	3.29
Baidu Bot Disallow	52	2.08
Sogou Bot Disallow	46	1.84
Yahoo Slurp Disallow	44	1.76
Yandex Bot Disallow	41	1.64
Microsoft BingBot Disallow	39	1.56
Alexa Bot Disallow	38	1.52
Soso Bot Disallow	38	1.52
Google Media Partners Disallow	32 ^*	1.28
Some Robots Disallow	27	1.08
Ahrefs Bot Disallow	22	0.88
Majestic Bot Disallow	7	0.28
Semrush Bot Disallow	5	0.2
Microsoft MSNBot Disallow	5	0.2
Google Image Bot Disallow	4	0.16
ByteDance Bytespider Disallow	3	0.12
Webz Omgili Disallow	3	0.12
Diffbot Disallow	3	0.12
Exalead Bot Disallow	2	0.08
YOU Disallow	2	0.08
Google News Bot Disallow	2	0.08
Google Video Bot Disallow	2	0.08
Google Favicon Disallow	2	0.08
Pipl Disallow	2	0.08
DuckDuckGo Bot Disallow	1	0.04
Youdao Bot Disallow	1	0.04
Google API Bot Disallow	1	0.04
Google DuplexWeb Disallow	1	0.04