Common Crawl Bot Disallow Usage Statistics

Common Crawl Bot Disallow

Block a site from being crawled by Common Crawl Crawler.

https://commoncrawl.org/

Robots.txt · AI Bot

Common Crawl Bot Disallow Customers

Get access to data on 5,133,646 websites that are Common Crawl Bot Disallow Customers. We know of 4,137,593 live websites using Common Crawl Bot Disallow and an additional 996,053 sites that used Common Crawl Bot Disallow historically and 2,795,422 websites in the United States.

Common Crawl Bot Disallow Competitors and Similar

Get a list of 5,133,646 websites using Common Crawl Bot Disallow which includes location information, hosting data, contact details, 4,137,593 currently live websites and an additional 1,980,601 domains that redirect to sites in this list. 996,053 sites that used this technology previouslyand 2,795,422 websites in the United States currently using Common Crawl Bot Disallow.

Site Totals

Total Live
4,137,593
1,980,601 additional website redirects?.
USA Live Sites
2,795,422
Live and Historical
5,133,646
Top 1m
2.86%
28,554
Top 100k
6.27%
6,270
Top 10k
10.51%
1,051

Countries

United States
2,795,422
United Kingdom
203,067
Australia
144,661
Canada
107,412
Germany
64,299
Russia
52,805
Spain
46,669
Switzerland
36,752
Italy
31,277
France
30,626
Netherlands
29,395
Brazil
28,158
New Zealand
27,698
.co
25,170
Sweden
22,123
Denmark
14,293
Norway
14,227
Belgium
13,180
Ireland
12,278
Poland
11,563
Japan
10,941
.me
10,916
Austria
10,389
Finland
9,659
IN India
8,882
MX Mexico
8,813
ID Indonesia
6,650
PT Portugal
6,125
ZA South Africa
6,122
IO .io
6,111
KR South Korea
6,096
HU Hungary
5,578
SG Singapore
5,246
CN China
5,101
CZ Czech Republic
4,923
GR Greece
4,231
SK Slovakia
4,172
TH Thailand
3,634
TW Taiwan
3,141
HK Hong Kong
3,110
RO Romania
3,077
SA Saudi Arabia
2,998
AI Anguilla
2,897
TR Turkey
2,868
CC Cocos Islands
2,719
AR Argentina
2,672
AE UAE
2,574
VN Vietnam
2,064
MY Malaysia
1,899
CL Chile
1,724
UA Ukraine
1,568
TV .tv
1,505
IR Iran
1,493
IS Iceland
1,481
PH Philippines
1,429
SI Slovenia
1,419
IL Israel
1,387
LT Lithuania
1,146
CR Costa Rica
1,094
IM Isle of Man
1,092
EE Estonia
923
CY Cyprus
899
LU Luxembourg
861
PE Peru
833
BG Bulgaria
782
LV Latvia
717
PR Puerto Rico
705
HR Croatia
679
KE Kenya
677
PK Pakistan
615
NG Nigeria
601
EC Ecuador
600
LB Lebanon
531
GT Guatemala
520
MT Malta
508
RS Serbia
435
BD Bangladesh
433
LI Liechtenstein
421
MA Morocco
409
DO Dominican Republic
406
UY Uruguay
364
LK Sri Lanka
357
GG Guernsey
351
EG Egypt
347
VE Venezuela
345
NP Nepal
343
PA Panama
333
NU Niue
328
AL Albania
326
VC Saint Vincent
311
BY Belarus
287
KZ Kazakhstan
269
TZ Tanzania
248
RW Rwanda
236
UG Uganda
233
KW Kuwait
230
GH Ghana
225
QA Qatar
212
JE Jersey
207
BM Bermuda
202
JO Jordan
197
KY Cayman Islands
192
BS Bahamas
191
SH Saint Helena
188
FM Micronesia
180
GE Georgia
178
HN Honduras
173
RE Reunion
161
KH Cambodia
158
VI Virgin Islands
150
AM Armenia
149
BZ Belize
142
TT Trinidad
138
MU Mauritius
135
SV El Salvador
135
BH Bahrain
134
AZ Azerbaijan
129
PW Palau
129
TO Tonga
125
BB Barbados
121
MC Monaco
118
BA Bosnia
117
WS Samoa
115
MD Moldova
114
JM Jamaica
107
ET Ethiopia
107
OM Oman
101
PY Paraguay
99
NI Nicaragua
99
TN Tunisia
96
FO Faroe Islands
96
NA Namibia
95
ZW Zimbabwe
86
ZM Zambia
85
MK Macedonia
82
BO Bolivia
82
MV Maldives
80
MM Myanmar
76
FJ Fiji
76
AG Antigua
75
AX Åland Islands
75
UZ Uzbekistan
74
TC Turks
71
MZ Mozambique
71
KG Kyrgyzstan
67
MN Mongolia
66
AF Afghanistan
64
SC Seychelles
62
CI Cote d'Ivoire
61
SN Senegal
61
IQ Iraq
60
CD Congo
60
DZ Algeria
59
GP Guadeloupe
58
BW Botswana
58
CX Christmas Island
58
AD Andorra
56
SO Somalia
56
GL Greenland
55
HT Haiti
53
AS American Samoa
52
GI Gibraltar
51
ST Sao Tome
50
CM Cameroon
47
PS Palestine
45
AO Angola
44
MG Madagascar
41
MW Malawi
39
PG Papua New Guinea
39
GU Guam
37
VG British Virgin Islands
36
AW Aruba
35
MQ Martinique
33
GY Guyana
31
VU Vanuatu
29
BN Brunei
27
CU Cuba
26
LC Saint Lucia
25
GD Grenada
25
LR Liberia
25
LA Lao People's Democratic Republic
24
MS Montserrat
23
SL Sierra Leone
23
SR Suriname
22
GM Gambia
22
CK Cook Islands
22
PM Saint Pierre
21
GS South Georgia
17
CW Curacao
16
SY Syria
16
TG Togo
15
MR Mauritania
14
ML Mali
14
PF French Polynesia
14
MP Northern Mariana Islands
13
TL East Timor
13
SZ Swaziland
13
LY Libya
12
BJ Benin
12
BI Burundi
12
TJ Tajikistan
12
GF French Guiana
11
GA Gabon
11
SJ Svalbard
11
LS Lesotho
10
SB Solomon Islands
10
DJ Djibouti
10
NC New Caledonia
10
SD Sudan
10
BF Burkina Faso
10
SM San Marino
10
MO Macao
10
BT Bhutan
10
YT Mayotte
9
SX Sint Maarten
9
KN Saint Kitts
9
YE Yemen
8
WF Wallis
8
DM Dominica
8
KI Kiribati
5
UM United States Minor Outlying Islands
5
SS South Sudan
5
FK Falkland Islands
5

Financial

AMER AMER
2,941,058
ANZ ANZ
172,359
ANZUK ANZUK
375,426
APAC APAC
236,825
ASEAN3 ASEAN+3
43,345
ASEAN5 ASEAN+5
219,485
ASEAN6 ASEAN+6
224,586
FOURASIANTIGERS ASIAN TIGERS
19,553
AUKUS AUKUS
3,143,150
BASIC BASIC
48,263
BRIC BRIC
94,946
BRICS BRICS
101,068
CANZUK CANZUK
482,838
CIVETS CIVETS
18,711
CLMV CLMV
2,322
EMEA EMEA
668,268
FRIT FRIT
61,903
FRITES FRITES
108,572
IMEA IMEA
33,639
LATAM LATAM
47,999
LATCAR LATCAR
52,573
MART MART
77,562
MEA MEA
24,757
MENA MENA
14,700
MEP MEP
14,715
MIKTA MIKTA
169,088
MINT MINT
18,932
PIGS PIGS
88,302
SEA SEA
21,220
TIGERCUB Tiger Cub Economies
15,676
VISTA VISTA
20,376

Group

AFRICANUNION African Union
11,242
ARABLEAGUE Arab League
8,103
APEC Asia-Pacific Economic Cooperation
3,188,745
AER Assembly of European Regions
550,068
ACS Association of Caribbean State Members
13,862
ASEAN Association of Southeast Asian Nations
21,207
BIMSTEC Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
13,735
CARICOM Caribbean Community Members
1,281
CARIFTA Caribbean Free Trade Association
1,015
CAIS Central American Integration System
2,902
CEFTA Central European Free Trade Agreement Members
1,114
PARISCLUB Club de Paris
3,685,116
CIS Commonwealth of Independent States Members
53,906
COMMONWEALTH Commonwealth of Nations
512,380
CPLP Community of Portuguese Language Countries
34,467
COE Council of Europe
646,721
DAC Development Assistance Committee
3,670,817
EAC East African Community
1,471
EAP Eastern Partnership
334,216
INNERSIX EC Inner Six
169,638
OUTERSEVEN EC Outer Seven
306,976
ECO Economic Cooperation Organization
5,593
EAEU Eurasian Economic Union
53,577
EURONEST Euronest Parliamentary Assembly
333,800
EEA European Economic Area
347,920
EFTA European Free Trade Association
52,881
EU European Union
331,791
EUEEACH European Union, European Economic Area and Switzerland
384,672
FEAS Federation of Euro-Asian Stock Exchanges
19,770
G2 G2 Nations
2,800,523
G20 G20 Nations
3,518,870
G7 G7 Nations
3,243,044
3G Global Governance Group
96,422
G77 Group of 77
96,723
GUAM GUAM Organization for Democracy and Economic Development
1,989
GCC Gulf Cooperation Council
6,249
IBSA IBSA Dialogue Forum
43,162
ISA International Solar Alliance
3,360,742
IAO Interparliamentary Assembly on Orthodoxy
98,250
DACH Majority German-speaking states of Central Europe
111,440
NATO North Atlantic Treaty Organization
3,400,847
OECD Organisation for Economic Co-operation and Development
3,690,149
OECS Organisation of Eastern Caribbean States
3,500
OIC Organisation of Islamic Cooperation
24,353
OAS Organization of American States
2,951,202
OEI Organization of Ibero-American States
100,691
BSEC Organization of the Black Sea Economic Cooperation
66,227
OPEC Organization of the Petroleum Exporting Countries
8,430
TURKIC Organization of Turkic States
3,407
PACIFICALLIANCE Pacific Alliance
12,030
SCO Shanghai Cooperation Organisation
67,825
SAARC South Asian Association for Regional Cooperation
10,784
SADC Southern African Development Community
7,169
MERCOSUR Southern Common Market
31,293
NORAM Sovereign states and dependent territories of North America
2,920,137
USAN Union of South American Nations
480
V4 Visegrad Group
26,236

Region

AFRICA Africa
11,004
BALKANS Balkan States
14,056
BALTICS Baltic States
2,786
BENELUX Benelux Union
43,436
CENTRALAFRICA Central Africa
464
CENTRALAMERICA Central America
2,496
CENTRALASIA Central Asia
424
CENTRALEUROPE Central Europe
139,516
CHAPS CHAPS
36,625
EASTAFRICA East Africa
2,114
EASTASIA East Asia
28,465
EUROPE Europe
647,278
LEVANT Levant Narrow
5,943
MIDDLEEAST Middle East
14,100
NORDICS Nordics
62,020
CUSA North America
2,902,834
NORTHAFRICA Northern Africa
947
NORTHERNEUROPE Northern Europe
76,847
SCANDINAVIA Scandinavia
50,643
SOUTHAMERICA South America
36,061
SOUTHASIA South Asia
10,784
SOUTHERNAFRICA Southern Africa
6,298
SOUTHERNCONE Southern Cone
4,859
SOUTHERNEUROPE Southern Europe
92,012
SUBSAHARA Sub-Saharan Africa
10,010
WESTAFRICA Western Africa
1,094
WESTASIA Western Asia
14,556
WESTERNEUROPE Western Europe
401,442
Browse Technology Groups