登录 · 免费注册

EN DE FR ES IT Pусский 한국어 日本語简体中文 عربي

Common Crawl Bot Disallow 使用情况统计

顶部 10k

顶部 100k

顶部 1m

全部互联网

阻止某个站点被 Common Crawl Crawler 抓取。

https://commoncrawl.org/

Robots.txt · 人工智能机器人

Common Crawl Bot Disallow 顾客

获取以下数据 4,766,999 网站即 Common Crawl Bot Disallow 顾客. 我们知道 3,681,953 直播网站使用 Common Crawl Bot Disallow 以及额外的 1,085,046 使用的网站 Common Crawl Bot Disallow 历史上和 2,540,879 网站美国.

下载潜在客户名单

Common Crawl Bot Disallow 奖项

2nd 最受欢迎美国在人工智能机器人类别.
2nd 最受欢迎 in the Top 10k sites 在人工智能机器人类别.
2nd 最受欢迎 in the Top 100k sites 在人工智能机器人类别.
2nd 最受欢迎 in the Top 1 Million sites 在人工智能机器人类别.
3rd 最受欢迎 on the Entire Internet 在人工智能机器人类别.
这最受欢迎 in Greece 在人工智能机器人类别.
这最受欢迎 in Mauritius 在人工智能机器人类别.
这最受欢迎 in Honduras 在人工智能机器人类别.
这最受欢迎 in Iran 在人工智能机器人类别.
这最受欢迎 in Maldives 在人工智能机器人类别.
2nd 最受欢迎美国在 Robots.txt 类别.
6th most popular in the Top 10k sites 在 Robots.txt 类别.
9th most popular in the Top 100k sites 在 Robots.txt 类别.
13th most popular in the Top 1 Million sites 在 Robots.txt 类别.
17th most popular on the Entire Internet 在 Robots.txt 类别.
3rd 最受欢迎 in the United States Virgin Islands 在 Robots.txt 类别.
3rd 最受欢迎 in Saudi Arabia 在 Robots.txt 类别.
3rd 最受欢迎 in Åland Islands 在 Robots.txt 类别.
3rd 最受欢迎 in Bermuda 在 Robots.txt 类别.
3rd 最受欢迎 in Barbados 在 Robots.txt 类别.

Common Crawl Bot Disallow 竞争对手和类似公司

GPTBot Disallow
3,751,266 直播网站

Anthropic Claude Bot Disallow
3,551,800 直播网站

Google Bard Disallow
3,391,288 直播网站

Facebook Bot Disallow
3,347,039 直播网站

Cohere AI Disallow
3,248,417 直播网站

Apple AI Bot Disallow
3,256,425 直播网站

Perplexity
3,192,806 直播网站

Alexa Bot Disallow
5,029,184 直播网站

AmazonBot Disallow
1,841,069 直播网站

ByteDance Bytespider Disallow
665,525 直播网站

Webz Omgili Disallow
386,711 直播网站

Diffbot Disallow
262,940 直播网站

YOU Disallow
160,638 直播网站

下载潜在客户名单

获取列表 4,766,999 网站使用 Common Crawl Bot Disallow 其中包括位置信息、托管数据、联系方式、 3,681,953 当前直播网站以及额外的 1,797,361 重定向至此列表中的站点的域名。 1,085,046 以前使用过该技术的网站和 2,540,879 网站美国目前正在使用 Common Crawl Bot Disallow.

网站总计

总直播: 3,681,953
1,797,361 额外的网站重定向^?.
美国直播网站: 2,540,879
现场和历史: 4,766,999

顶部 1m: 2.52%
25,232
顶部 100k: 5.83%
5,834
顶部 10k: 10.08%
1,008

国家

Financial

Group

African Union: 9,115
Arab League: 6,904
Asia-Pacific Economic Cooperation: 2,891,938
Assembly of European Regions: 497,436
Association of Caribbean State Members: 11,990
Association of Southeast Asian Nations: 13,943
Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation: 8,245
Caribbean Community Members: 1,163
Caribbean Free Trade Association: 913
Central American Integration System: 2,522
Central European Free Trade Agreement Members: 857
Club de Paris: 3,352,820
Commonwealth of Independent States Members: 42,402
Commonwealth of Nations: 469,039
Community of Portuguese Language Countries: 28,265
Council of Europe: 583,478
Development Assistance Committee: 3,348,154
East African Community: 1,102
Eastern Partnership: 303,574
EC Inner Six: 154,384
EC Outer Seven: 283,287
Economic Cooperation Organization: 3,138
Eurasian Economic Union: 42,267
Euronest Parliamentary Assembly: 303,282
European Economic Area: 317,334
European Free Trade Association: 50,216
European Union: 302,009
European Union, European Economic Area and Switzerland: 352,225
Federation of Euro-Asian Stock Exchanges: 16,063
G2 Nations: 2,544,586
G20 Nations: 3,187,546
G7 Nations: 2,951,521
Global Governance Group: 88,032
Group of 77: 72,417
GUAM Organization for Democracy and Economic Development: 1,215
Gulf Cooperation Council: 5,480
IBSA Dialogue Forum: 33,724
International Solar Alliance: 3,052,444
Interparliamentary Assembly on Orthodoxy: 81,126
Majority German-speaking states of Central Europe: 102,583
North Atlantic Treaty Organization: 3,093,614
Organisation for Economic Co-operation and Development: 3,363,215
Organisation of Eastern Caribbean States: 2,676
Organisation of Islamic Cooperation: 15,158
Organization of American States: 2,681,239
Organization of Ibero-American States: 86,884
Organization of the Black Sea Economic Cooperation: 51,564
Organization of the Petroleum Exporting Countries: 6,907
Organization of Turkic States: 1,676
Pacific Alliance: 10,031
Shanghai Cooperation Organisation: 51,144
South Asian Association for Regional Cooperation: 6,031
Southern African Development Community: 6,191
Southern Common Market: 25,376
Sovereign states and dependent territories of North America: 2,656,163
Union of South American Nations: 373
Visegrad Group: 23,235

Region

浏览技术组