登录 · 免费注册

EN DE FR ES IT Pусский 한국어 日本語简体中文 عربي

Common Crawl Bot Disallow 使用情况统计

顶部 10k

顶部 100k

顶部 1m

全部互联网

阻止某个站点被 Common Crawl Crawler 抓取。

https://commoncrawl.org/

Robots.txt · 人工智能机器人

Common Crawl Bot Disallow 顾客

获取以下数据 4,707,876 网站即 Common Crawl Bot Disallow 顾客. 我们知道 3,689,029 直播网站使用 Common Crawl Bot Disallow 以及额外的 1,018,847 使用的网站 Common Crawl Bot Disallow 历史上和 2,532,686 网站美国.

下载潜在客户名单

Common Crawl Bot Disallow 奖项

2nd 最受欢迎美国在人工智能机器人类别.
2nd 最受欢迎 in the Top 10k sites 在人工智能机器人类别.
2nd 最受欢迎 in the Top 100k sites 在人工智能机器人类别.
2nd 最受欢迎 in the Top 1 Million sites 在人工智能机器人类别.
3rd 最受欢迎 on the Entire Internet 在人工智能机器人类别.
这最受欢迎 in Cyprus 在人工智能机器人类别.
这最受欢迎 in Honduras 在人工智能机器人类别.
这最受欢迎 in Mauritius 在人工智能机器人类别.
这最受欢迎 in Iran 在人工智能机器人类别.
这最受欢迎 in Kuwait 在人工智能机器人类别.
2nd 最受欢迎美国在 Robots.txt 类别.
6th most popular in the Top 10k sites 在 Robots.txt 类别.
10th most popular in the Top 100k sites 在 Robots.txt 类别.
13th most popular in the Top 1 Million sites 在 Robots.txt 类别.
17th most popular on the Entire Internet 在 Robots.txt 类别.
这最受欢迎 in Barbados 在 Robots.txt 类别.
3rd 最受欢迎 in the United States Virgin Islands 在 Robots.txt 类别.
3rd 最受欢迎 in Saudi Arabia 在 Robots.txt 类别.
3rd 最受欢迎 in Rwanda 在 Robots.txt 类别.
4th 最受欢迎 in Cyprus 在 Robots.txt 类别.

Common Crawl Bot Disallow 竞争对手和类似公司

GPTBot Disallow
3,819,787 直播网站

Anthropic Claude Bot Disallow
3,542,190 直播网站

Google Bard Disallow
3,382,611 直播网站

Facebook Bot Disallow
3,340,030 直播网站

Cohere AI Disallow
3,236,580 直播网站

Apple AI Bot Disallow
3,225,055 直播网站

Perplexity
3,161,273 直播网站

Alexa Bot Disallow
5,502,757 直播网站

AmazonBot Disallow
981,982 直播网站

ByteDance Bytespider Disallow
734,933 直播网站

Webz Omgili Disallow
417,754 直播网站

Diffbot Disallow
257,932 直播网站

YOU Disallow
155,792 直播网站

下载潜在客户名单

获取列表 4,707,876 网站使用 Common Crawl Bot Disallow 其中包括位置信息、托管数据、联系方式、 3,689,029 当前直播网站以及额外的 1,831,128 重定向至此列表中的站点的域名。 1,018,847 以前使用过该技术的网站和 2,532,686 网站美国目前正在使用 Common Crawl Bot Disallow.

网站总计

总直播: 3,689,029
1,831,128 额外的网站重定向^?.
美国直播网站: 2,532,686
现场和历史: 4,707,876

顶部 1m: 2.61%
26,065
顶部 100k: 6.14%
6,141
顶部 10k: 11.78%
1,178

国家

Financial

Group

African Union: 9,107
Arab League: 7,330
Asia-Pacific Economic Cooperation: 2,872,112
Assembly of European Regions: 485,244
Association of Caribbean State Members: 11,895
Association of Southeast Asian Nations: 14,234
Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation: 8,382
Caribbean Community Members: 1,155
Caribbean Free Trade Association: 912
Central American Integration System: 2,506
Central European Free Trade Agreement Members: 874
Club de Paris: 3,331,641
Commonwealth of Independent States Members: 31,902
Commonwealth of Nations: 466,351
Community of Portuguese Language Countries: 30,978
Council of Europe: 569,801
Development Assistance Committee: 3,335,068
East African Community: 1,093
Eastern Partnership: 302,025
EC Inner Six: 153,394
EC Outer Seven: 281,079
Economic Cooperation Organization: 3,329
Eurasian Economic Union: 31,767
Euronest Parliamentary Assembly: 301,727
European Economic Area: 315,760
European Free Trade Association: 49,507
European Union: 300,399
European Union, European Economic Area and Switzerland: 349,906
Federation of Euro-Asian Stock Exchanges: 16,325
G2 Nations: 2,536,478
G20 Nations: 3,169,471
G7 Nations: 2,941,606
Global Governance Group: 87,119
Group of 77: 76,223
GUAM Organization for Democracy and Economic Development: 1,261
Gulf Cooperation Council: 5,897
IBSA Dialogue Forum: 36,544
International Solar Alliance: 3,045,975
Interparliamentary Assembly on Orthodoxy: 70,790
Majority German-speaking states of Central Europe: 100,004
North Atlantic Treaty Organization: 3,083,114
Organisation for Economic Co-operation and Development: 3,350,105
Organisation of Eastern Caribbean States: 2,760
Organisation of Islamic Cooperation: 15,958
Organization of American States: 2,675,297
Organization of Ibero-American States: 89,068
Organization of the Black Sea Economic Cooperation: 41,284
Organization of the Petroleum Exporting Countries: 7,486
Organization of Turkic States: 1,704
Pacific Alliance: 9,981
Shanghai Cooperation Organisation: 40,826
South Asian Association for Regional Cooperation: 6,151
Southern African Development Community: 6,154
Southern Common Market: 28,119
Sovereign states and dependent territories of North America: 2,647,529
Union of South American Nations: 364
Visegrad Group: 23,740

Region

浏览技术组