Common Crawl Bot Disallow Usage Statistics

Common Crawl Bot Disallow

Block a site from being crawled by Common Crawl Crawler.

https://commoncrawl.org/

Robots.txt · AI Bot

Common Crawl Bot Disallow Customers

Get access to data on 4,707,876 websites that are Common Crawl Bot Disallow Customers. We know of 3,689,029 live websites using Common Crawl Bot Disallow and an additional 1,018,847 sites that used Common Crawl Bot Disallow historically and 2,532,686 websites in the United States.

Common Crawl Bot Disallow Competitors and Similar

Get a list of 4,707,876 websites using Common Crawl Bot Disallow which includes location information, hosting data, contact details, 3,689,029 currently live websites and an additional 1,831,128 domains that redirect to sites in this list. 1,018,847 sites that used this technology previouslyand 2,532,686 websites in the United States currently using Common Crawl Bot Disallow.

Site Totals

Total Live
3,689,029
1,831,128 additional website redirects?.
USA Live Sites
2,532,686
Live and Historical
4,707,876
Top 1m
2.61%
26,065
Top 100k
6.14%
6,141
Top 10k
11.78%
1,178

Countries

United States
2,532,686
United Kingdom
184,918
Australia
133,794
Canada
100,129
Germany
56,357
Spain
42,086
Switzerland
34,146
Russia
31,206
Italy
29,787
France
27,527
Netherlands
26,794
Brazil
25,908
New Zealand
25,696
.co
22,232
Sweden
20,807
Norway
13,590
Denmark
13,199
Belgium
12,179
Ireland
11,407
Japan
10,202
Poland
9,810
.me
9,687
Austria
9,501
Finland
8,954
MX Mexico
7,517
HU Hungary
5,619
KR South Korea
5,571
ZA South Africa
5,319
IN India
5,317
PT Portugal
4,918
IO .io
4,913
SG Singapore
4,834
CZ Czech Republic
4,410
SK Slovakia
3,901
CN China
3,792
GR Greece
3,644
SA Saudi Arabia
3,250
ID Indonesia
2,926
TW Taiwan
2,819
HK Hong Kong
2,740
RO Romania
2,734
TH Thailand
2,553
AI Anguilla
2,221
AE UAE
2,073
AR Argentina
1,819
TR Turkey
1,446
IS Iceland
1,386
VN Vietnam
1,370
TV .tv
1,325
SI Slovenia
1,300
IR Iran
1,293
CL Chile
1,256
CC Cocos Islands
1,248
PH Philippines
1,185
MY Malaysia
1,135
UA Ukraine
1,041
IM Isle of Man
1,018
LT Lithuania
991
CR Costa Rica
957
IL Israel
893
EE Estonia
805
LU Luxembourg
750
CY Cyprus
671
PR Puerto Rico
652
BG Bulgaria
645
PE Peru
643
LV Latvia
607
HR Croatia
559
EC Ecuador
532
LB Lebanon
466
KE Kenya
463
GT Guatemala
455
MT Malta
437
LI Liechtenstein
385
RS Serbia
362
DO Dominican Republic
335
UY Uruguay
326
NU Niue
321
PA Panama
288
PK Pakistan
278
VC Saint Vincent
277
MA Morocco
276
EG Egypt
275
NG Nigeria
273
BY Belarus
265
VE Venezuela
261
GG Guernsey
258
AL Albania
248
RW Rwanda
214
KW Kuwait
203
TZ Tanzania
198
JE Jersey
187
GH Ghana
187
KY Cayman Islands
186
KZ Kazakhstan
186
BM Bermuda
181
LK Sri Lanka
180
BS Bahamas
180
QA Qatar
171
JO Jordan
168
FM Micronesia
164
UG Uganda
161
HN Honduras
155
BD Bangladesh
154
VI Virgin Islands
142
RE Reunion
137
KH Cambodia
133
TT Trinidad
132
SH Saint Helena
127
GE Georgia
122
BH Bahrain
116
NP Nepal
116
BB Barbados
115
SV El Salvador
113
BZ Belize
113
MC Monaco
113
AM Armenia
100
BA Bosnia
99
JM Jamaica
96
MU Mauritius
95
ET Ethiopia
92
NI Nicaragua
90
WS Samoa
86
NA Namibia
86
OM Oman
84
FO Faroe Islands
82
ZM Zambia
74
AG Antigua
73
PW Palau
72
FJ Fiji
68
PY Paraguay
66
MK Macedonia
66
MD Moldova
65
AX Åland Islands
65
TC Turks
65
TO Tonga
62
BO Bolivia
60
ZW Zimbabwe
60
TN Tunisia
57
SN Senegal
56
SC Seychelles
56
MV Maldives
55
MM Myanmar
55
MZ Mozambique
55
GP Guadeloupe
53
AS American Samoa
52
GI Gibraltar
51
MN Mongolia
49
HT Haiti
48
GL Greenland
47
BW Botswana
46
ST Sao Tome
45
AF Afghanistan
44
CD Congo
44
CI Cote d'Ivoire
43
AD Andorra
43
DZ Algeria
43
CX Christmas Island
42
SO Somalia
38
AO Angola
36
GU Guam
35
MG Madagascar
35
PG Papua New Guinea
35
PS Palestine
34
AZ Azerbaijan
33
CM Cameroon
33
MW Malawi
31
AW Aruba
31
IQ Iraq
31
MQ Martinique
30
UZ Uzbekistan
29
VG British Virgin Islands
28
GY Guyana
28
VU Vanuatu
27
LR Liberia
25
CU Cuba
23
BN Brunei
23
CK Cook Islands
23
GD Grenada
21
LC Saint Lucia
20
LA Lao People's Democratic Republic
20
GM Gambia
20
MS Montserrat
19
SL Sierra Leone
18
PM Saint Pierre
15
SR Suriname
15
GS South Georgia
14
SB Solomon Islands
14
ML Mali
13
CW Curacao
13
SZ Swaziland
13
PF French Polynesia
13
LY Libya
11
SD Sudan
11
KG Kyrgyzstan
10
TL East Timor
10
BF Burkina Faso
10
MO Macao
10
SM San Marino
10
SJ Svalbard
10
BI Burundi
9
DJ Djibouti
9
BJ Benin
9
DM Dominica
9
KN Saint Kitts
9
SY Syria
9
GA Gabon
9
MP Northern Mariana Islands
8
MR Mauritania
8
TG Togo
8
TJ Tajikistan
8
GF French Guiana
8
YT Mayotte
7
NC New Caledonia
7
BT Bhutan
7
YE Yemen
6
LS Lesotho
6
WF Wallis
6
SX Sint Maarten
5
KI Kiribati
5
UM United States Minor Outlying Islands
5
FK Falkland Islands
5

Financial

AMER AMER
2,666,597
ANZ ANZ
159,490
ANZUK ANZUK
344,408
APAC APAC
207,381
ASEAN3 ASEAN+3
33,799
ASEAN5 ASEAN+5
194,814
ASEAN6 ASEAN+6
198,606
FOURASIANTIGERS ASIAN TIGERS
16,937
AUKUS AUKUS
2,851,398
BASIC BASIC
40,336
BRIC BRIC
66,223
BRICS BRICS
71,542
CANZUK CANZUK
444,537
CIVETS CIVETS
11,901
CLMV CLMV
1,578
EMEA EMEA
587,911
FRIT FRIT
57,314
FRITES FRITES
99,400
IMEA IMEA
25,092
LATAM LATAM
42,163
LATCAR LATCAR
45,915
MART MART
50,981
MEA MEA
19,775
MENA MENA
11,595
MEP MEP
11,467
MIKTA MIKTA
151,254
MINT MINT
12,162
PIGS PIGS
80,435
SEA SEA
14,244
TIGERCUB Tiger Cub Economies
9,169
VISTA VISTA
12,880

Group

AFRICANUNION African Union
9,107
ARABLEAGUE Arab League
7,330
APEC Asia-Pacific Economic Cooperation
2,872,112
AER Assembly of European Regions
485,244
ACS Association of Caribbean State Members
11,895
ASEAN Association of Southeast Asian Nations
14,234
BIMSTEC Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
8,382
CARICOM Caribbean Community Members
1,155
CARIFTA Caribbean Free Trade Association
912
CAIS Central American Integration System
2,506
CEFTA Central European Free Trade Agreement Members
874
PARISCLUB Club de Paris
3,331,641
CIS Commonwealth of Independent States Members
31,902
COMMONWEALTH Commonwealth of Nations
466,351
CPLP Community of Portuguese Language Countries
30,978
COE Council of Europe
569,801
DAC Development Assistance Committee
3,335,068
EAC East African Community
1,093
EAP Eastern Partnership
302,025
INNERSIX EC Inner Six
153,394
OUTERSEVEN EC Outer Seven
281,079
ECO Economic Cooperation Organization
3,329
EAEU Eurasian Economic Union
31,767
EURONEST Euronest Parliamentary Assembly
301,727
EEA European Economic Area
315,760
EFTA European Free Trade Association
49,507
EU European Union
300,399
EUEEACH European Union, European Economic Area and Switzerland
349,906
FEAS Federation of Euro-Asian Stock Exchanges
16,325
G2 G2 Nations
2,536,478
G20 G20 Nations
3,169,471
G7 G7 Nations
2,941,606
3G Global Governance Group
87,119
G77 Group of 77
76,223
GUAM GUAM Organization for Democracy and Economic Development
1,261
GCC Gulf Cooperation Council
5,897
IBSA IBSA Dialogue Forum
36,544
ISA International Solar Alliance
3,045,975
IAO Interparliamentary Assembly on Orthodoxy
70,790
DACH Majority German-speaking states of Central Europe
100,004
NATO North Atlantic Treaty Organization
3,083,114
OECD Organisation for Economic Co-operation and Development
3,350,105
OECS Organisation of Eastern Caribbean States
2,760
OIC Organisation of Islamic Cooperation
15,958
OAS Organization of American States
2,675,297
OEI Organization of Ibero-American States
89,068
BSEC Organization of the Black Sea Economic Cooperation
41,284
OPEC Organization of the Petroleum Exporting Countries
7,486
TURKIC Organization of Turkic States
1,704
PACIFICALLIANCE Pacific Alliance
9,981
SCO Shanghai Cooperation Organisation
40,826
SAARC South Asian Association for Regional Cooperation
6,151
SADC Southern African Development Community
6,154
MERCOSUR Southern Common Market
28,119
NORAM Sovereign states and dependent territories of North America
2,647,529
USAN Union of South American Nations
364
V4 Visegrad Group
23,740

Region

AFRICA Africa
8,861
BALKANS Balkan States
11,137
BALTICS Baltic States
2,403
BENELUX Benelux Union
39,723
CENTRALAFRICA Central Africa
393
CENTRALAMERICA Central America
2,171
CENTRALASIA Central Asia
235
CENTRALEUROPE Central Europe
125,429
CHAPS CHAPS
33,241
EASTAFRICA East Africa
1,606
EASTASIA East Asia
25,183
EUROPE Europe
570,253
LEVANT Levant Narrow
3,687
MIDDLEEAST Middle East
11,189
NORDICS Nordics
58,140
CUSA North America
2,632,815
NORTHAFRICA Northern Africa
681
NORTHERNEUROPE Northern Europe
71,746
SCANDINAVIA Scandinavia
47,596
SOUTHAMERICA South America
31,899
SOUTHASIA South Asia
6,151
SOUTHERNAFRICA Southern Africa
5,470
SOUTHERNCONE Southern Cone
3,467
SOUTHERNEUROPE Southern Europe
83,633
SUBSAHARA Sub-Saharan Africa
8,146
WESTAFRICA Western Africa
681
WESTASIA Western Asia
11,444
WESTERNEUROPE Western Europe
364,120
Browse Technology Groups