Common Crawl Bot Disallow Usage Statistics

No chart yet for Common Crawl Bot Disallow. Learn how we update data.

Common Crawl Bot Disallow

Block a site from being crawled by Common Crawl Crawler.

https://commoncrawl.org/

Robots.txt ·

Common Crawl Bot Disallow Customers

Get access to data on 2,610,024 websites Common Crawl Bot Disallow Customers. We know of 2,610,107 live websites using Common Crawl Bot Disallow and an additional -83 sites that used Common Crawl Bot Disallow historically and 1,984,530 websites in the United States.

Get a list of 2,610,024 websites using Common Crawl Bot Disallow which includes location information, hosting data, contact details, 2,610,107 currently live websites and an additional 1,380,584 domains that redirect to sites in this list. 0 sites that used this technology previouslyand 1,984,530 websites in the United States currently using Common Crawl Bot Disallow.

Site Totals

Total Live
2,610,107
1,380,584 additional website redirects?.
United States Live Sites
1,984,530
Live and Historical
2,610,024
Top 1m
1.64%
16,448
Top 100k
4.16%
4,156
Top 10k
6.93%
693

Countries

United States
1,984,530
United Kingdom
101,688
Australia
78,782
Canada
63,875
Germany
32,377
Russia
31,097
Spain
25,019
New Zealand
17,525
France
16,135
Netherlands
15,032
Italy
11,076
.co
10,514
Switzerland
10,418
Sweden
8,749
Norway
8,458
Ireland
6,792
Belgium
6,731
Denmark
6,439
Finland
5,620
Japan
5,102
.me
5,076
Mexico
4,724
Austria
4,371
South Korea
3,716
IN India
3,650
SG Singapore
3,500
BR Brazil
3,451
PT Portugal
2,802
PL Poland
2,716
IO .io
2,670
ZA South Africa
2,565
CN China
2,396
ID Indonesia
2,368
TH Thailand
2,099
GR Greece
2,049
IR Iran
2,029
HK Hong Kong
2,012
SA Saudi Arabia
1,869
CZ Czech Republic
1,850
RO Romania
1,703
AE UAE
1,393
TW Taiwan
1,227
HU Hungary
1,203
AR Argentina
1,187
VN Vietnam
1,126
TR Turkey
972
IS Iceland
868
PH Philippines
830
AI Anguilla
816
MY Malaysia
794
CR Costa Rica
757
IL Israel
720
CL Chile
709
TV .tv
683
SK Slovakia
618
UA Ukraine
557
PR Puerto Rico
521
EE Estonia
502
LU Luxembourg
478
CC Cocos Islands
462
SI Slovenia
451
BG Bulgaria
447
LV Latvia
419
PE Peru
415
EC Ecuador
412
HR Croatia
410
LT Lithuania
405
LB Lebanon
387
KE Kenya
336
GT Guatemala
326
MT Malta
250
RS Serbia
237
DO Dominican Republic
237
PA Panama
217
CY Cyprus
215
UY Uruguay
210
NG Nigeria
200
EG Egypt
190
RW Rwanda
181
MA Morocco
179
AL Albania
172
PK Pakistan
172
GG Guernsey
162
GH Ghana
157
KW Kuwait
148
TZ Tanzania
143
VE Venezuela
141
VC Saint Vincent
138
JE Jersey
134
FM Micronesia
127
KY Cayman Islands
125
LK Sri Lanka
125
JO Jordan
122
BS Bahamas
121
UG Uganda
121
HN Honduras
121
BM Bermuda
121
QA Qatar
116
NU Niue
116
VI Virgin Islands
113
GE Georgia
111
BD Bangladesh
110
KH Cambodia
102
LI Liechtenstein
99
SV El Salvador
90
BB Barbados
88
NP Nepal
87
TT Trinidad
85
BH Bahrain
84
IM Isle of Man
81
ET Ethiopia
77
NI Nicaragua
76
JM Jamaica
75
BZ Belize
75
MC Monaco
74
KZ Kazakhstan
68
BY Belarus
67
ZM Zambia
64
MU Mauritius
63
OM Oman
62
RE Reunion
59
AM Armenia
58
NA Namibia
57
BA Bosnia
56
FO Faroe Islands
56
WS Samoa
54
ZW Zimbabwe
54
SC Seychelles
49
SN Senegal
48
TC Turks
47
PW Palau
46
PY Paraguay
45
MM Myanmar
45
SH Saint Helena
44
HT Haiti
43
MK Macedonia
42
MZ Mozambique
42
FJ Fiji
41
MD Moldova
40
MV Maldives
39
CD Congo
38
GP Guadeloupe
35
TO Tonga
35
BO Bolivia
35
GI Gibraltar
35
GU Guam
32
BW Botswana
32
MN Mongolia
31
AG Antigua
31
PG Papua New Guinea
31
GL Greenland
30
AD Andorra
30
PS Palestine
27
TN Tunisia
26
AS American Samoa
25
DZ Algeria
25
MG Madagascar
25
AF Afghanistan
24
CI Cote d'Ivoire
24
IQ Iraq
24
VU Vanuatu
24
MW Malawi
23
VG British Virgin Islands
23
MQ Martinique
21
LR Liberia
21
AO Angola
21
AW Aruba
21
GY Guyana
20
BN Brunei
20
UZ Uzbekistan
20
CM Cameroon
18
AZ Azerbaijan
18
GD Grenada
18
SO Somalia
18
AX Åland Islands
18
SL Sierra Leone
18
CX Christmas Island
18
CU Cuba
16
GM Gambia
16
MS Montserrat
16
ST Sao Tome
16
CW Curacao
14
CK Cook Islands
14
SR Suriname
13
LA Lao People's Democratic Republic
11
LC Saint Lucia
11
PF French Polynesia
11
ML Mali
10
MO Macao
10
TL East Timor
9
DJ Djibouti
9
GA Gabon
9
NC New Caledonia
9
YE Yemen
8
SD Sudan
8
KN Saint Kitts
8
GF French Guiana
7
SB Solomon Islands
7
SM San Marino
7
BJ Benin
7
PM Saint Pierre
7
SZ Swaziland
7
KG Kyrgyzstan
6
GS South Georgia
6
SJ Svalbard
6
BI Burundi
6
LY Libya
6
TG Togo
6
SX Sint Maarten
6
TJ Tajikistan
5
DM Dominica
5

Financial

AMER AMER
2,057,184
ANZ ANZ
96,307
ANZUK ANZUK
197,995
APAC APAC
126,989
ASEAN3 ASEAN+3
22,109
ASEAN5 ASEAN+5
119,670
ASEAN6 ASEAN+6
122,066
FOURASIANTIGERS ASIAN TIGERS
11,624
AUKUS AUKUS
2,165,000
BASIC BASIC
12,062
BRIC BRIC
40,594
BRICS BRICS
43,159
CANZUK CANZUK
261,870
CIVETS CIVETS
7,615
CLMV CLMV
1,284
EMEA EMEA
322,061
FRIT FRIT
27,211
FRITES FRITES
52,230
IMEA IMEA
16,885
LATAM LATAM
14,194
LATCAR LATCAR
16,025
MART MART
44,332
MEA MEA
13,235
MENA MENA
8,616
MEP MEP
8,541
MIKTA MIKTA
90,562
MINT MINT
8,264
PIGS PIGS
40,946
SEA SEA
10,904
TIGERCUB Tiger Cub Economies
7,217
VISTA VISTA
8,218

Group

AFRICANUNION African Union
5,302
ARABLEAGUE Arab League
4,704
APEC Asia-Pacific Economic Cooperation
2,206,878
AER Assembly of European Regions
266,531
ACS Association of Caribbean State Members
7,830
ASEAN Association of Southeast Asian Nations
10,895
BIMSTEC Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
6,117
CARICOM Caribbean Community Members
747
CARIFTA Caribbean Free Trade Association
570
CAIS Central American Integration System
1,899
CEFTA Central European Free Trade Agreement Members
568
PARISCLUB Club de Paris
2,430,178
CIS Commonwealth of Independent States Members
31,379
COMMONWEALTH Commonwealth of Nations
275,717
CPLP Community of Portuguese Language Countries
6,348
COE Council of Europe
309,876
DAC Development Assistance Committee
2,425,470
EAC East African Community
829
EAP Eastern Partnership
155,710
INNERSIX EC Inner Six
81,829
OUTERSEVEN EC Outer Seven
142,925
ECO Economic Cooperation Organization
3,316
EAEU Eurasian Economic Union
31,296
EURONEST Euronest Parliamentary Assembly
155,625
EEA European Economic Area
164,284
EFTA European Free Trade Association
19,843
EU European Union
154,859
EUEEACH European Union, European Economic Area and Switzerland
174,702
FEAS Federation of Euro-Asian Stock Exchanges
10,981
G2 G2 Nations
1,986,926
G20 G20 Nations
2,351,560
G7 G7 Nations
2,214,783
3G Global Governance Group
45,877
G77 Group of 77
38,680
GUAM GUAM Organization for Democracy and Economic Development
726
GCC Gulf Cooperation Council
3,672
IBSA IBSA Dialogue Forum
9,666
ISA International Solar Alliance
2,260,823
IAO Interparliamentary Assembly on Orthodoxy
48,966
DACH Majority German-speaking states of Central Europe
47,166
NATO North Atlantic Treaty Organization
2,289,482
OECD Organisation for Economic Co-operation and Development
2,435,072
OECS Organisation of Eastern Caribbean States
1,122
OIC Organisation of Islamic Cooperation
12,091
OAS Organization of American States
2,062,683
OEI Organization of Ibero-American States
41,935
BSEC Organization of the Black Sea Economic Cooperation
37,224
OPEC Organization of the Petroleum Exporting Countries
5,868
TURKIC Organization of Turkic States
1,084
PACIFICALLIANCE Pacific Alliance
6,242
SCO Shanghai Cooperation Organisation
37,414
SAARC South Asian Association for Regional Cooperation
4,208
SADC Southern African Development Community
3,187
MERCOSUR Southern Common Market
4,893
NORAM Sovereign states and dependent territories of North America
2,057,582
USAN Union of South American Nations
209
V4 Visegrad Group
6,387

Region

AFRICA Africa
5,056
BALKANS Balkan States
6,560
BALTICS Baltic States
1,326
BENELUX Benelux Union
22,241
CENTRALAFRICA Central Africa
294
CENTRALAMERICA Central America
1,662
CENTRALASIA Central Asia
101
CENTRALEUROPE Central Europe
54,103
CHAPS CHAPS
10,758
EASTAFRICA East Africa
1,224
EASTASIA East Asia
14,494
EUROPE Europe
310,013
LEVANT Levant Narrow
2,446
MIDDLEEAST Middle East
8,369
NORDICS Nordics
30,244
CUSA North America
2,048,405
NORTHAFRICA Northern Africa
437
NORTHERNEUROPE Northern Europe
38,252
SCANDINAVIA Scandinavia
23,646
SOUTHAMERICA South America
7,334
SOUTHASIA South Asia
4,208
SOUTHERNAFRICA Southern Africa
2,665
SOUTHERNCONE Southern Cone
2,151
SOUTHERNEUROPE Southern Europe
42,651
SUBSAHARA Sub-Saharan Africa
4,600
WESTAFRICA Western Africa
527
WESTASIA Western Asia
8,556
WESTERNEUROPE Western Europe
194,225
Browse Technology Groups