Common Crawl Bot Disallow Usage Statistics

Common Crawl Bot Disallow

Block a site from being crawled by Common Crawl Crawler.

https://commoncrawl.org/

Robots.txt · AI Bot

Common Crawl Bot Disallow Customers

Get access to data on 4,766,999 websites that are Common Crawl Bot Disallow Customers. We know of 3,681,953 live websites using Common Crawl Bot Disallow and an additional 1,085,046 sites that used Common Crawl Bot Disallow historically and 2,540,879 websites in the United States.

Common Crawl Bot Disallow Competitors and Similar

Get a list of 4,766,999 websites using Common Crawl Bot Disallow which includes location information, hosting data, contact details, 3,681,953 currently live websites and an additional 1,797,361 domains that redirect to sites in this list. 1,085,046 sites that used this technology previouslyand 2,540,879 websites in the United States currently using Common Crawl Bot Disallow.

Site Totals

Total Live
3,681,953
1,797,361 additional website redirects?.
USA Live Sites
2,540,879
Live and Historical
4,766,999
Top 1m
2.52%
25,232
Top 100k
5.83%
5,834
Top 10k
10.08%
1,008

Countries

United States
2,540,879
United Kingdom
185,871
Australia
134,925
Canada
100,546
Germany
57,958
Spain
42,536
Russia
41,719
Switzerland
34,891
Italy
28,598
France
27,550
Netherlands
27,144
New Zealand
25,841
Brazil
23,160
.co
22,045
Sweden
20,913
Norway
13,567
Denmark
13,352
Belgium
12,359
Ireland
11,536
Japan
10,119
Austria
9,734
Poland
9,705
.me
9,533
Finland
9,025
MX Mexico
7,591
KR South Korea
5,543
ZA South Africa
5,354
IN India
5,210
HU Hungary
5,206
PT Portugal
4,959
SG Singapore
4,877
IO .io
4,796
CZ Czech Republic
4,377
SK Slovakia
3,947
CN China
3,707
GR Greece
3,610
SA Saudi Arabia
2,843
ID Indonesia
2,728
HK Hong Kong
2,726
TW Taiwan
2,692
RO Romania
2,624
TH Thailand
2,529
AI Anguilla
2,139
AE UAE
2,065
AR Argentina
1,829
TR Turkey
1,416
IS Iceland
1,378
TV .tv
1,313
SI Slovenia
1,310
VN Vietnam
1,259
CC Cocos Islands
1,243
CL Chile
1,234
PH Philippines
1,167
MY Malaysia
1,150
IR Iran
1,137
IM Isle of Man
1,040
UA Ukraine
995
LT Lithuania
985
CR Costa Rica
961
IL Israel
895
EE Estonia
814
LU Luxembourg
775
CY Cyprus
728
PR Puerto Rico
656
PE Peru
646
BG Bulgaria
644
LV Latvia
605
HR Croatia
568
EC Ecuador
532
LB Lebanon
466
KE Kenya
465
GT Guatemala
459
MT Malta
447
LI Liechtenstein
380
RS Serbia
351
DO Dominican Republic
334
UY Uruguay
322
NU Niue
306
PA Panama
294
MA Morocco
277
VC Saint Vincent
275
EG Egypt
273
PK Pakistan
273
VE Venezuela
266
NG Nigeria
266
BY Belarus
259
GG Guernsey
247
AL Albania
245
RW Rwanda
217
KW Kuwait
203
TZ Tanzania
198
JE Jersey
194
KZ Kazakhstan
189
BM Bermuda
188
KY Cayman Islands
187
GH Ghana
186
BS Bahamas
183
LK Sri Lanka
179
JO Jordan
170
QA Qatar
169
FM Micronesia
162
UG Uganda
162
HN Honduras
155
BD Bangladesh
147
VI Virgin Islands
141
TT Trinidad
131
RE Reunion
131
KH Cambodia
128
GE Georgia
122
SH Saint Helena
119
BZ Belize
118
BH Bahrain
116
BB Barbados
114
SV El Salvador
113
NP Nepal
112
MC Monaco
111
JM Jamaica
99
BA Bosnia
99
MU Mauritius
94
AM Armenia
91
ET Ethiopia
88
NI Nicaragua
88
NA Namibia
87
OM Oman
84
WS Samoa
82
FO Faroe Islands
80
AX Åland Islands
73
ZM Zambia
73
AG Antigua
72
FJ Fiji
68
PW Palau
66
PY Paraguay
65
MD Moldova
65
TC Turks
65
MV Maldives
61
MM Myanmar
61
MK Macedonia
60
BO Bolivia
60
ZW Zimbabwe
60
SN Senegal
57
SC Seychelles
57
TO Tonga
56
MZ Mozambique
56
GP Guadeloupe
54
TN Tunisia
54
AS American Samoa
52
MN Mongolia
50
HT Haiti
49
GI Gibraltar
49
GL Greenland
48
BW Botswana
47
CD Congo
47
CI Cote d'Ivoire
43
AF Afghanistan
42
AD Andorra
40
CX Christmas Island
39
ST Sao Tome
39
DZ Algeria
38
SO Somalia
36
GU Guam
35
PG Papua New Guinea
35
AO Angola
34
PS Palestine
34
MG Madagascar
34
AZ Azerbaijan
33
MW Malawi
31
VG British Virgin Islands
31
IQ Iraq
31
MQ Martinique
30
CM Cameroon
30
AW Aruba
30
UZ Uzbekistan
29
GY Guyana
29
VU Vanuatu
27
BN Brunei
25
CU Cuba
24
LR Liberia
23
CK Cook Islands
23
SL Sierra Leone
22
GM Gambia
21
GD Grenada
20
LC Saint Lucia
19
LA Lao People's Democratic Republic
19
SR Suriname
18
MS Montserrat
18
PM Saint Pierre
15
CW Curacao
14
ML Mali
13
SZ Swaziland
13
PF French Polynesia
13
SD Sudan
12
GS South Georgia
11
TL East Timor
11
SB Solomon Islands
11
LY Libya
11
MO Macao
10
GA Gabon
10
SJ Svalbard
10
MP Northern Mariana Islands
9
KG Kyrgyzstan
9
DJ Djibouti
9
BI Burundi
9
GF French Guiana
9
DM Dominica
9
KN Saint Kitts
9
BF Burkina Faso
9
SM San Marino
9
SY Syria
9
MR Mauritania
8
TG Togo
8
TJ Tajikistan
8
BJ Benin
8
YT Mayotte
7
BT Bhutan
7
LS Lesotho
6
WF Wallis
6
NC New Caledonia
6
YE Yemen
5
SX Sint Maarten
5
KI Kiribati
5
UM United States Minor Outlying Islands
5
FK Falkland Islands
5

Financial

AMER AMER
2,672,465
ANZ ANZ
160,766
ANZUK ANZUK
346,637
APAC APAC
207,867
ASEAN3 ASEAN+3
33,312
ASEAN5 ASEAN+5
195,581
ASEAN6 ASEAN+6
199,288
FOURASIANTIGERS ASIAN TIGERS
16,853
AUKUS AUKUS
2,861,675
BASIC BASIC
37,431
BRIC BRIC
73,796
BRICS BRICS
79,150
CANZUK CANZUK
447,183
CIVETS CIVETS
11,590
CLMV CLMV
1,467
EMEA EMEA
601,017
FRIT FRIT
56,148
FRITES FRITES
98,684
IMEA IMEA
24,444
LATAM LATAM
39,494
LATCAR LATCAR
43,170
MART MART
60,953
MEA MEA
19,234
MENA MENA
11,044
MEP MEP
10,917
MIKTA MIKTA
152,203
MINT MINT
12,001
PIGS PIGS
79,703
SEA SEA
13,954
TIGERCUB Tiger Cub Economies
8,833
VISTA VISTA
12,586

Group

AFRICANUNION African Union
9,115
ARABLEAGUE Arab League
6,904
APEC Asia-Pacific Economic Cooperation
2,891,938
AER Assembly of European Regions
497,436
ACS Association of Caribbean State Members
11,990
ASEAN Association of Southeast Asian Nations
13,943
BIMSTEC Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
8,245
CARICOM Caribbean Community Members
1,163
CARIFTA Caribbean Free Trade Association
913
CAIS Central American Integration System
2,522
CEFTA Central European Free Trade Agreement Members
857
PARISCLUB Club de Paris
3,352,820
CIS Commonwealth of Independent States Members
42,402
COMMONWEALTH Commonwealth of Nations
469,039
CPLP Community of Portuguese Language Countries
28,265
COE Council of Europe
583,478
DAC Development Assistance Committee
3,348,154
EAC East African Community
1,102
EAP Eastern Partnership
303,574
INNERSIX EC Inner Six
154,384
OUTERSEVEN EC Outer Seven
283,287
ECO Economic Cooperation Organization
3,138
EAEU Eurasian Economic Union
42,267
EURONEST Euronest Parliamentary Assembly
303,282
EEA European Economic Area
317,334
EFTA European Free Trade Association
50,216
EU European Union
302,009
EUEEACH European Union, European Economic Area and Switzerland
352,225
FEAS Federation of Euro-Asian Stock Exchanges
16,063
G2 G2 Nations
2,544,586
G20 G20 Nations
3,187,546
G7 G7 Nations
2,951,521
3G Global Governance Group
88,032
G77 Group of 77
72,417
GUAM GUAM Organization for Democracy and Economic Development
1,215
GCC Gulf Cooperation Council
5,480
IBSA IBSA Dialogue Forum
33,724
ISA International Solar Alliance
3,052,444
IAO Interparliamentary Assembly on Orthodoxy
81,126
DACH Majority German-speaking states of Central Europe
102,583
NATO North Atlantic Treaty Organization
3,093,614
OECD Organisation for Economic Co-operation and Development
3,363,215
OECS Organisation of Eastern Caribbean States
2,676
OIC Organisation of Islamic Cooperation
15,158
OAS Organization of American States
2,681,239
OEI Organization of Ibero-American States
86,884
BSEC Organization of the Black Sea Economic Cooperation
51,564
OPEC Organization of the Petroleum Exporting Countries
6,907
TURKIC Organization of Turkic States
1,676
PACIFICALLIANCE Pacific Alliance
10,031
SCO Shanghai Cooperation Organisation
51,144
SAARC South Asian Association for Regional Cooperation
6,031
SADC Southern African Development Community
6,191
MERCOSUR Southern Common Market
25,376
NORAM Sovereign states and dependent territories of North America
2,656,163
USAN Union of South American Nations
373
V4 Visegrad Group
23,235

Region

AFRICA Africa
8,863
BALKANS Balkan States
10,964
BALTICS Baltic States
2,404
BENELUX Benelux Union
40,278
CENTRALAFRICA Central Africa
390
CENTRALAMERICA Central America
2,188
CENTRALASIA Central Asia
237
CENTRALEUROPE Central Europe
127,508
CHAPS CHAPS
32,969
EASTAFRICA East Africa
1,606
EASTASIA East Asia
24,847
EUROPE Europe
583,927
LEVANT Levant Narrow
3,718
MIDDLEEAST Middle East
10,644
NORDICS Nordics
58,446
CUSA North America
2,641,425
NORTHAFRICA Northern Africa
673
NORTHERNEUROPE Northern Europe
72,175
SCANDINAVIA Scandinavia
47,832
SOUTHAMERICA South America
29,146
SOUTHASIA South Asia
6,031
SOUTHERNAFRICA Southern Africa
5,507
SOUTHERNCONE Southern Cone
3,450
SOUTHERNEUROPE Southern Europe
82,907
SUBSAHARA Sub-Saharan Africa
8,158
WESTAFRICA Western Africa
675
WESTASIA Western Asia
10,890
WESTERNEUROPE Western Europe
368,349
Browse Technology Groups