Common Crawl Bot Disallow Usage Statistics

Common Crawl Bot Disallow

Block a site from being crawled by Common Crawl Crawler.

https://commoncrawl.org/

Robots.txt · AI Bot

Common Crawl Bot Disallow Customers

Get access to data on 4,882,308 websites that are Common Crawl Bot Disallow Customers. We know of 3,838,613 live websites using Common Crawl Bot Disallow and an additional 1,043,695 sites that used Common Crawl Bot Disallow historically and 2,622,467 websites in the United States.

Common Crawl Bot Disallow Competitors and Similar

Get a list of 4,882,308 websites using Common Crawl Bot Disallow which includes location information, hosting data, contact details, 3,838,613 currently live websites and an additional 1,859,681 domains that redirect to sites in this list. 1,043,695 sites that used this technology previouslyand 2,622,467 websites in the United States currently using Common Crawl Bot Disallow.

Site Totals

Total Live
3,838,613
1,859,681 additional website redirects?.
USA Live Sites
2,622,467
Live and Historical
4,882,308
Top 1m
2.71%
27,097
Top 100k
6.06%
6,065
Top 10k
10.26%
1,026

Countries

United States
2,622,467
United Kingdom
192,812
Australia
138,767
Canada
102,865
Germany
60,090
Russia
46,671
Spain
44,325
Switzerland
35,667
Italy
29,714
France
28,669
Netherlands
27,945
New Zealand
26,589
Brazil
24,341
.co
22,837
Sweden
21,433
Norway
13,809
Denmark
13,697
Belgium
12,658
Ireland
11,866
Japan
10,360
Poland
10,215
Austria
10,013
.me
9,911
Finland
9,282
MX Mexico
8,062
IN India
5,967
KR South Korea
5,741
ZA South Africa
5,574
PT Portugal
5,326
HU Hungary
5,323
IO .io
5,126
SG Singapore
5,006
CZ Czech Republic
4,565
CN China
4,119
SK Slovakia
4,014
GR Greece
3,769
ID Indonesia
3,589
SA Saudi Arabia
2,892
HK Hong Kong
2,841
TW Taiwan
2,834
TH Thailand
2,786
RO Romania
2,734
AI Anguilla
2,331
AE UAE
2,198
AR Argentina
2,058
TR Turkey
1,748
CC Cocos Islands
1,579
VN Vietnam
1,442
IS Iceland
1,422
CL Chile
1,371
TV .tv
1,363
MY Malaysia
1,356
SI Slovenia
1,344
PH Philippines
1,236
IR Iran
1,214
UA Ukraine
1,154
IM Isle of Man
1,049
LT Lithuania
1,038
IL Israel
1,023
CR Costa Rica
1,007
CY Cyprus
973
EE Estonia
856
LU Luxembourg
802
PE Peru
690
PR Puerto Rico
684
BG Bulgaria
681
LV Latvia
634
HR Croatia
612
EC Ecuador
558
KE Kenya
510
LB Lebanon
487
GT Guatemala
480
MT Malta
462
LI Liechtenstein
391
RS Serbia
375
DO Dominican Republic
368
NG Nigeria
343
PK Pakistan
334
UY Uruguay
333
MA Morocco
309
NU Niue
309
PA Panama
308
EG Egypt
291
VE Venezuela
290
VC Saint Vincent
282
GG Guernsey
277
AL Albania
275
BY Belarus
270
RW Rwanda
222
LK Sri Lanka
220
BD Bangladesh
214
KW Kuwait
211
KZ Kazakhstan
210
TZ Tanzania
208
GH Ghana
198
BM Bermuda
197
JE Jersey
196
KY Cayman Islands
190
BS Bahamas
185
QA Qatar
177
JO Jordan
176
UG Uganda
173
FM Micronesia
168
HN Honduras
162
NP Nepal
161
VI Virgin Islands
143
GE Georgia
141
RE Reunion
139
TT Trinidad
136
KH Cambodia
135
SH Saint Helena
134
SV El Salvador
123
BZ Belize
122
BH Bahrain
119
BB Barbados
118
MC Monaco
117
AM Armenia
109
BA Bosnia
105
JM Jamaica
103
MU Mauritius
99
NI Nicaragua
93
ET Ethiopia
92
WS Samoa
89
NA Namibia
89
OM Oman
88
FO Faroe Islands
86
PW Palau
79
TO Tonga
78
ZM Zambia
77
AG Antigua
74
FJ Fiji
74
AX Åland Islands
73
MD Moldova
72
PY Paraguay
72
MV Maldives
69
TC Turks
69
MK Macedonia
68
TN Tunisia
65
MM Myanmar
65
BO Bolivia
65
ZW Zimbabwe
63
MZ Mozambique
60
SC Seychelles
59
SN Senegal
58
GP Guadeloupe
53
CD Congo
53
MN Mongolia
53
AZ Azerbaijan
51
GL Greenland
51
AS American Samoa
51
GI Gibraltar
50
CI Cote d'Ivoire
50
HT Haiti
49
BW Botswana
48
AF Afghanistan
48
DZ Algeria
45
ST Sao Tome
43
SO Somalia
42
CX Christmas Island
42
UZ Uzbekistan
41
AD Andorra
41
AO Angola
40
IQ Iraq
37
MG Madagascar
37
GU Guam
36
PG Papua New Guinea
36
MW Malawi
35
PS Palestine
35
CM Cameroon
33
MQ Martinique
32
VG British Virgin Islands
32
KG Kyrgyzstan
31
GY Guyana
31
AW Aruba
31
VU Vanuatu
28
BN Brunei
27
CU Cuba
24
LR Liberia
24
GD Grenada
23
SL Sierra Leone
22
CK Cook Islands
22
LC Saint Lucia
21
LA Lao People's Democratic Republic
21
GM Gambia
21
SR Suriname
19
MS Montserrat
19
PM Saint Pierre
17
CW Curacao
14
ML Mali
13
PF French Polynesia
13
TL East Timor
12
SD Sudan
12
SZ Swaziland
12
GS South Georgia
11
LY Libya
11
BI Burundi
11
SY Syria
11
GA Gabon
11
SJ Svalbard
11
SB Solomon Islands
10
TG Togo
10
BJ Benin
10
MO Macao
10
MP Northern Mariana Islands
9
DJ Djibouti
9
GF French Guiana
9
DM Dominica
9
KN Saint Kitts
9
BF Burkina Faso
9
SM San Marino
9
MR Mauritania
8
LS Lesotho
8
NC New Caledonia
8
TJ Tajikistan
8
WF Wallis
7
YT Mayotte
7
BT Bhutan
7
YE Yemen
6
SX Sint Maarten
6
KI Kiribati
5
UM United States Minor Outlying Islands
5
FK Falkland Islands
5

Financial

AMER AMER
2,758,185
ANZ ANZ
165,356
ANZUK ANZUK
358,168
APAC APAC
216,678
ASEAN3 ASEAN+3
35,883
ASEAN5 ASEAN+5
203,087
ASEAN6 ASEAN+6
207,206
FOURASIANTIGERS ASIAN TIGERS
17,707
AUKUS AUKUS
2,954,046
BASIC BASIC
40,001
BRIC BRIC
81,098
BRICS BRICS
86,672
CANZUK CANZUK
461,033
CIVETS CIVETS
13,238
CLMV CLMV
1,663
EMEA EMEA
626,666
FRIT FRIT
58,383
FRITES FRITES
102,708
IMEA IMEA
26,770
LATAM LATAM
41,829
LATCAR LATCAR
45,742
MART MART
67,474
MEA MEA
20,803
MENA MENA
12,136
MEP MEP
12,020
MIKTA MIKTA
157,907
MINT MINT
13,742
PIGS PIGS
83,134
SEA SEA
15,675
TIGERCUB Tiger Cub Economies
10,409
VISTA VISTA
14,411

Group

AFRICANUNION African Union
9,665
ARABLEAGUE Arab League
7,218
APEC Asia-Pacific Economic Cooperation
2,988,855
AER Assembly of European Regions
518,701
ACS Association of Caribbean State Members
12,692
ASEAN Association of Southeast Asian Nations
15,663
BIMSTEC Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
9,420
CARICOM Caribbean Community Members
1,200
CARIFTA Caribbean Free Trade Association
947
CAIS Central American Integration System
2,663
CEFTA Central European Free Trade Agreement Members
932
PARISCLUB Club de Paris
3,464,215
CIS Commonwealth of Independent States Members
47,463
COMMONWEALTH Commonwealth of Nations
484,896
CPLP Community of Portuguese Language Countries
29,828
COE Council of Europe
608,103
DAC Development Assistance Committee
3,455,549
EAC East African Community
1,181
EAP Eastern Partnership
314,837
INNERSIX EC Inner Six
159,878
OUTERSEVEN EC Outer Seven
292,757
ECO Economic Cooperation Organization
3,687
EAEU Eurasian Economic Union
47,291
EURONEST Euronest Parliamentary Assembly
314,516
EEA European Economic Area
328,662
EFTA European Free Trade Association
51,289
EU European Union
313,040
EUEEACH European Union, European Economic Area and Switzerland
364,329
FEAS Federation of Euro-Asian Stock Exchanges
17,237
G2 G2 Nations
2,626,586
G20 G20 Nations
3,296,506
G7 G7 Nations
3,046,977
3G Global Governance Group
90,922
G77 Group of 77
78,316
GUAM GUAM Organization for Democracy and Economic Development
1,418
GCC Gulf Cooperation Council
5,685
IBSA IBSA Dialogue Forum
35,882
ISA International Solar Alliance
3,152,877
IAO Interparliamentary Assembly on Orthodoxy
88,064
DACH Majority German-speaking states of Central Europe
105,770
NATO North Atlantic Treaty Organization
3,194,503
OECD Organisation for Economic Co-operation and Development
3,471,882
OECS Organisation of Eastern Caribbean States
2,885
OIC Organisation of Islamic Cooperation
17,306
OAS Organization of American States
2,767,488
OEI Organization of Ibero-American States
91,375
BSEC Organization of the Black Sea Economic Cooperation
57,405
OPEC Organization of the Petroleum Exporting Countries
7,295
TURKIC Organization of Turkic States
2,081
PACIFICALLIANCE Pacific Alliance
10,717
SCO Shanghai Cooperation Organisation
57,381
SAARC South Asian Association for Regional Cooperation
7,020
SADC Southern African Development Community
6,462
MERCOSUR Southern Common Market
26,804
NORAM Sovereign states and dependent territories of North America
2,740,958
USAN Union of South American Nations
405
V4 Visegrad Group
24,117

Region

AFRICA Africa
9,408
BALKANS Balkan States
11,748
BALTICS Baltic States
2,528
BENELUX Benelux Union
41,405
CENTRALAFRICA Central Africa
417
CENTRALAMERICA Central America
2,295
CENTRALASIA Central Asia
292
CENTRALEUROPE Central Europe
131,622
CHAPS CHAPS
34,130
EASTAFRICA East Africa
1,714
EASTASIA East Asia
25,958
EUROPE Europe
608,584
LEVANT Levant Narrow
4,453
MIDDLEEAST Middle East
11,686
NORDICS Nordics
59,864
CUSA North America
2,725,332
NORTHAFRICA Northern Africa
741
NORTHERNEUROPE Northern Europe
74,037
SCANDINAVIA Scandinavia
48,939
SOUTHAMERICA South America
30,866
SOUTHASIA South Asia
7,020
SOUTHERNAFRICA Southern Africa
5,731
SOUTHERNCONE Southern Cone
3,834
SOUTHERNEUROPE Southern Europe
86,501
SUBSAHARA Sub-Saharan Africa
8,629
WESTAFRICA Western Africa
777
WESTASIA Western Asia
11,987
WESTERNEUROPE Western Europe
381,071
Browse Technology Groups