登录 · 免费注册

EN DE FR ES IT Pусский 한국어 日本語简体中文 عربي

Common Crawl Bot Disallow 使用情况统计

顶部 10k

顶部 100k

顶部 1m

全部互联网

阻止某个站点被 Common Crawl Crawler 抓取。

https://commoncrawl.org/

Robots.txt · 人工智能机器人

Common Crawl Bot Disallow 顾客

获取以下数据 4,796,720 网站即 Common Crawl Bot Disallow 顾客. 我们知道 3,722,656 直播网站使用 Common Crawl Bot Disallow 以及额外的 1,074,064 使用的网站 Common Crawl Bot Disallow 历史上和 2,568,345 网站美国.

下载潜在客户名单

Common Crawl Bot Disallow 奖项

2nd 最受欢迎美国在人工智能机器人类别.
2nd 最受欢迎 in the Top 10k sites 在人工智能机器人类别.
2nd 最受欢迎 in the Top 100k sites 在人工智能机器人类别.
2nd 最受欢迎 in the Top 1 Million sites 在人工智能机器人类别.
3rd 最受欢迎 on the Entire Internet 在人工智能机器人类别.
这最受欢迎 in Mauritius 在人工智能机器人类别.
这最受欢迎 in Honduras 在人工智能机器人类别.
这最受欢迎 in Iran 在人工智能机器人类别.
这最受欢迎 in Kuwait 在人工智能机器人类别.
这最受欢迎 in Saudi Arabia 在人工智能机器人类别.
2nd 最受欢迎美国在 Robots.txt 类别.
6th most popular in the Top 10k sites 在 Robots.txt 类别.
10th most popular in the Top 100k sites 在 Robots.txt 类别.
13th most popular in the Top 1 Million sites 在 Robots.txt 类别.
17th most popular on the Entire Internet 在 Robots.txt 类别.
2nd 最受欢迎 in Saudi Arabia 在 Robots.txt 类别.
3rd 最受欢迎 in Åland Islands 在 Robots.txt 类别.
3rd 最受欢迎 in Turks and Caicos Islands 在 Robots.txt 类别.
3rd 最受欢迎 in Barbados 在 Robots.txt 类别.
4th 最受欢迎 in New Zealand 在 Robots.txt 类别.

Common Crawl Bot Disallow 竞争对手和类似公司

GPTBot Disallow
3,819,558 直播网站

Anthropic Claude Bot Disallow
3,591,156 直播网站

Google Bard Disallow
3,430,304 直播网站

Facebook Bot Disallow
3,384,465 直播网站

Apple AI Bot Disallow
3,293,219 直播网站

Cohere AI Disallow
3,282,715 直播网站

Perplexity
3,229,070 直播网站

Alexa Bot Disallow
5,217,515 直播网站

AmazonBot Disallow
2,039,981 直播网站

ByteDance Bytespider Disallow
673,782 直播网站

Webz Omgili Disallow
392,093 直播网站

Diffbot Disallow
268,227 直播网站

YOU Disallow
164,979 直播网站

Anthropic Claude User Bot Disallow
3,763 直播网站

下载潜在客户名单

获取列表 4,796,720 网站使用 Common Crawl Bot Disallow 其中包括位置信息、托管数据、联系方式、 3,722,656 当前直播网站以及额外的 1,819,486 重定向至此列表中的站点的域名。 1,074,064 以前使用过该技术的网站和 2,568,345 网站美国目前正在使用 Common Crawl Bot Disallow.

网站总计

总直播: 3,722,656
1,819,486 额外的网站重定向^?.
美国直播网站: 2,568,345
现场和历史: 4,796,720

顶部 1m: 2.53%
25,309
顶部 100k: 5.85%
5,850
顶部 10k: 10.11%
1,011

国家

Financial

Group

African Union: 9,228
Arab League: 6,968
Asia-Pacific Economic Cooperation: 2,923,119
Assembly of European Regions: 502,785
Association of Caribbean State Members: 12,187
Association of Southeast Asian Nations: 14,148
Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation: 8,320
Caribbean Community Members: 1,170
Caribbean Free Trade Association: 919
Central American Integration System: 2,562
Central European Free Trade Agreement Members: 865
Club de Paris: 3,388,404
Commonwealth of Independent States Members: 43,250
Commonwealth of Nations: 473,569
Community of Portuguese Language Countries: 28,441
Council of Europe: 589,808
Development Assistance Committee: 3,383,421
East African Community: 1,112
Eastern Partnership: 306,683
EC Inner Six: 156,039
EC Outer Seven: 286,051
Economic Cooperation Organization: 3,159
Eurasian Economic Union: 43,115
Euronest Parliamentary Assembly: 306,392
European Economic Area: 320,491
European Free Trade Association: 50,543
European Union: 305,055
European Union, European Economic Area and Switzerland: 355,598
Federation of Euro-Asian Stock Exchanges: 16,134
G2 Nations: 2,572,123
G20 Nations: 3,221,849
G7 Nations: 2,983,174
Global Governance Group: 88,822
Group of 77: 73,123
GUAM Organization for Democracy and Economic Development: 1,279
Gulf Cooperation Council: 5,527
IBSA Dialogue Forum: 33,922
International Solar Alliance: 3,084,853
Interparliamentary Assembly on Orthodoxy: 82,346
Majority German-speaking states of Central Europe: 103,564
North Atlantic Treaty Organization: 3,126,735
Organisation for Economic Co-operation and Development: 3,398,699
Organisation of Eastern Caribbean States: 2,717
Organisation of Islamic Cooperation: 15,384
Organization of American States: 2,709,863
Organization of Ibero-American States: 87,820
Organization of the Black Sea Economic Cooperation: 52,535
Organization of the Petroleum Exporting Countries: 6,968
Organization of Turkic States: 1,692
Pacific Alliance: 10,197
Shanghai Cooperation Organisation: 52,108
South Asian Association for Regional Cooperation: 6,086
Southern African Development Community: 6,264
Southern Common Market: 25,499
Sovereign states and dependent territories of North America: 2,684,679
Union of South American Nations: 377
Visegrad Group: 23,469

Region

浏览技术组