Robots Disallow Usage Statistics

Robots Disallow

A robots disallow all directive with no other options.

https://robotstxt.org

Robots.txt

Robots Disallow Customers

Get access to data on 8,881,281 websites that are Robots Disallow Customers. We know of 4,517,112 live websites using Robots Disallow and an additional 4,364,169 sites that used Robots Disallow historically and 2,083,905 websites in the United States.

Get a list of 8,881,281 websites using Robots Disallow which includes location information, hosting data, contact details, 4,517,112 currently live websites and an additional 2,752,229 domains that redirect to sites in this list. 4,364,169 sites that used this technology previously and 2,083,905 websites in the United States currently using Robots Disallow.

Site Totals

Total Live
4,517,112
2,752,229 additional website redirects?.
USA Live Sites
2,083,905
Live and Historical
8,881,281
Top 1m
9.74%
97,415
Top 100k
21.51%
21,509
Top 10k
31.28%
3,128

Countries

United States
2,083,905
Russia
420,617
Germany
187,607
United Kingdom
81,216
Netherlands
61,080
China
54,435
Australia
52,935
France
43,293
Canada
42,698
Japan
40,223
Ukraine
33,021
Italy
31,906
Brazil
28,528
India
24,914
Switzerland
24,474
Spain
20,657
South Korea
18,646
Poland
18,592
.co
17,549
New Zealand
16,247
.me
15,805
Czech Republic
14,977
Taiwan
13,806
Austria
13,496
AR Argentina
13,397
BE Belgium
11,852
SE Sweden
11,795
IO .io
11,516
DK Denmark
9,282
ZA South Africa
8,380
FI Finland
8,281
CC Cocos Islands
8,180
HU Hungary
8,043
MX Mexico
7,578
NO Norway
6,817
RO Romania
6,628
PW Palau
5,773
ID Indonesia
5,486
VN Vietnam
5,479
CL Chile
5,468
SK Slovakia
4,808
PT Portugal
4,755
TV .tv
4,289
TR Turkey
4,154
HK Hong Kong
4,065
GR Greece
3,686
SG Singapore
3,419
IR Iran
3,193
MY Malaysia
3,140
ML Mali
3,060
IE Ireland
2,865
IL Israel
2,647
BY Belarus
2,569
BG Bulgaria
2,226
LT Lithuania
2,093
KE Kenya
1,844
KZ Kazakhstan
1,783
AE UAE
1,760
EE Estonia
1,736
NU Niue
1,661
PH Philippines
1,604
PE Peru
1,559
LV Latvia
1,526
SI Slovenia
1,483
TH Thailand
1,449
PK Pakistan
1,410
RS Serbia
1,339
HR Croatia
1,338
UZ Uzbekistan
1,205
LU Luxembourg
1,081
AI Anguilla
1,071
NG Nigeria
873
SA Saudi Arabia
836
IS Iceland
818
BD Bangladesh
792
MA Morocco
692
EC Ecuador
689
UY Uruguay
669
LI Liechtenstein
645
SH Saint Helena
643
WS Samoa
619
LK Sri Lanka
617
NP Nepal
568
TN Tunisia
566
AM Armenia
546
GE Georgia
491
RE Reunion
486
IM Isle of Man
480
CY Cyprus
457
MD Moldova
441
MK Macedonia
426
AZ Azerbaijan
413
MT Malta
393
BZ Belize
381
CR Costa Rica
372
TO Tonga
369
DO Dominican Republic
355
GG Guernsey
327
AL Albania
325
VE Venezuela
298
EG Egypt
292
CX Christmas Island
290
GT Guatemala
279
CM Cameroon
274
TZ Tanzania
273
UG Uganda
268
AG Antigua
248
PY Paraguay
243
BA Bosnia
217
BO Bolivia
206
PR Puerto Rico
199
MN Mongolia
199
PA Panama
196
KG Kyrgyzstan
192
VC Saint Vincent
187
FM Micronesia
187
QA Qatar
185
DZ Algeria
180
SO Somalia
174
MU Mauritius
172
ZW Zimbabwe
168
ST Sao Tome
167
MS Montserrat
140
TJ Tajikistan
137
PM Saint Pierre
136
AF Afghanistan
128
HN Honduras
125
GH Ghana
122
SV El Salvador
120
MZ Mozambique
115
LB Lebanon
112
TT Trinidad
110
GP Guadeloupe
107
CI Cote d'Ivoire
106
BB Barbados
105
KH Cambodia
104
SC Seychelles
101
KY Cayman Islands
96
PS Palestine
96
AS American Samoa
96
GL Greenland
94
MV Maldives
93
BW Botswana
92
LC Saint Lucia
91
JO Jordan
88
KW Kuwait
86
MC Monaco
85
BH Bahrain
85
KN Saint Kitts
85
NC New Caledonia
82
SN Senegal
81
RW Rwanda
80
JM Jamaica
77
BS Bahamas
76
NI Nicaragua
74
BM Bermuda
72
GS South Georgia
71
OM Oman
71
GY Guyana
70
MG Madagascar
68
AO Angola
67
NA Namibia
67
VG British Virgin Islands
65
VI Virgin Islands
64
MM Myanmar
62
FO Faroe Islands
61
TC Turks
59
JE Jersey
56
ZM Zambia
55
IQ Iraq
53
YT Mayotte
51
WF Wallis
50
PG Papua New Guinea
50
GD Grenada
49
BF Burkina Faso
49
SX Sint Maarten
48
DJ Djibouti
46
SR Suriname
45
BJ Benin
44
HT Haiti
44
SM San Marino
42
TL East Timor
41
SL Sierra Leone
40
CD Congo
40
VU Vanuatu
39
AD Andorra
38
UM United States Minor Outlying Islands
38
BN Brunei
36
GI Gibraltar
35
PF French Polynesia
34
MW Malawi
34
ET Ethiopia
33
FJ Fiji
33
SD Sudan
32
TM Turkmenistan
31
GM Gambia
31
BI Burundi
28
TG Togo
28
MQ Martinique
25
SB Solomon Islands
24
SY Syria
21
CU Cuba
18
DM Dominica
18
HM Heard Island
17
YE Yemen
15
CW Curacao
15
LS Lesotho
12
BT Bhutan
11
GF French Guiana
10
GN Guinea
10
CK Cook Islands
9
MP Northern Mariana Islands
9
AW Aruba
9
MO Macao
8
AX Åland Islands
8
TD Chad
8
KI Kiribati
7
SZ Swaziland
7
NE Niger
7
MR Mauritania
7
GU Guam
7
NF Norfolk Island
6
PN Pitcairn
6
MH Marshall Islands
6

Financial

AMER AMER
2,180,038
ANZ ANZ
69,182
ANZUK ANZUK
150,398
APAC APAC
267,499
ASEAN3 ASEAN+3
134,084
ASEAN5 ASEAN+5
173,745
ASEAN6 ASEAN+6
228,180
FOURASIANTIGERS ASIAN TIGERS
80,565
AUKUS AUKUS
2,218,056
BASIC BASIC
116,257
BRIC BRIC
528,494
BRICS BRICS
536,874
CANZUK CANZUK
193,096
CIVETS CIVETS
24,397
CLMV CLMV
5,646
EMEA EMEA
1,085,652
FRIT FRIT
75,199
FRITES FRITES
95,856
IMEA IMEA
58,762
LATAM LATAM
61,165
LATCAR LATCAR
64,111
MART MART
454,465
MEA MEA
33,848
MENA MENA
15,631
MEP MEP
15,561
MIKTA MIKTA
88,799
MINT MINT
18,091
PIGS PIGS
61,004
SEA SEA
20,821
TIGERCUB Tiger Cub Economies
17,158
VISTA VISTA
36,896

Group

AFRICANUNION African Union
19,374
ARABLEAGUE Arab League
5,382
APEC Asia-Pacific Economic Cooperation
2,782,845
AER Assembly of European Regions
810,213
ACS Association of Caribbean State Members
11,607
ASEAN Association of Southeast Asian Nations
20,780
BIMSTEC Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
28,413
CARICOM Caribbean Community Members
1,726
CARIFTA Caribbean Free Trade Association
1,561
CAIS Central American Integration System
1,902
CEFTA Central European Free Trade Agreement Members
2,751
PARISCLUB Club de Paris
3,204,820
CIS Commonwealth of Independent States Members
427,903
COMMONWEALTH Commonwealth of Nations
243,847
CPLP Community of Portuguese Language Countries
33,676
COE Council of Europe
1,052,063
DAC Development Assistance Committee
2,827,518
EAC East African Community
2,534
EAP Eastern Partnership
513,417
INNERSIX EC Inner Six
336,819
OUTERSEVEN EC Outer Seven
151,835
ECO Economic Cooperation Organization
12,646
EAEU Eurasian Economic Union
425,707
EURONEST Euronest Parliamentary Assembly
510,435
EEA European Economic Area
484,216
EFTA European Free Trade Association
32,754
EU European Union
475,936
EUEEACH European Union, European Economic Area and Switzerland
508,690
FEAS Federation of Euro-Asian Stock Exchanges
33,782
G2 G2 Nations
2,138,340
G20 G20 Nations
3,150,754
G7 G7 Nations
2,510,848
3G Global Governance Group
77,188
G77 Group of 77
186,103
GUAM GUAM Organization for Democracy and Economic Development
34,366
GCC Gulf Cooperation Council
3,023
IBSA IBSA Dialogue Forum
61,822
ISA International Solar Alliance
2,527,876
IAO Interparliamentary Assembly on Orthodoxy
526,144
DACH Majority German-speaking states of Central Europe
225,577
NATO North Atlantic Treaty Organization
2,659,010
OECD Organisation for Economic Co-operation and Development
2,853,698
OECS Organisation of Eastern Caribbean States
2,086
OIC Organisation of Islamic Cooperation
33,011
OAS Organization of American States
2,188,951
OEI Organization of Ibero-American States
86,430
BSEC Organization of the Black Sea Economic Cooperation
472,548
OPEC Organization of the Petroleum Exporting Countries
7,352
TURKIC Organization of Turkic States
7,747
PACIFICALLIANCE Pacific Alliance
15,211
SCO Shanghai Cooperation Organisation
504,693
SAARC South Asian Association for Regional Cooperation
28,533
SADC Southern African Development Community
9,654
MERCOSUR Southern Common Market
42,837
NORAM Sovereign states and dependent territories of North America
2,139,343
USAN Union of South American Nations
619
V4 Visegrad Group
46,420

Region

AFRICA Africa
19,989
BALKANS Balkan States
21,825
BALTICS Baltic States
5,355
BENELUX Benelux Union
74,013
CENTRALAFRICA Central Africa
667
CENTRALAMERICA Central America
1,547
CENTRALASIA Central Asia
3,348
CENTRALEUROPE Central Europe
274,125
CHAPS CHAPS
59,916
EASTAFRICA East Africa
3,495
EASTASIA East Asia
131,382
EUROPE Europe
1,056,415
LEVANT Levant Narrow
7,575
MIDDLEEAST Middle East
14,151
NORDICS Nordics
37,159
CUSA North America
2,126,603
NORTHAFRICA Northern Africa
1,772
NORTHERNEUROPE Northern Europe
45,213
SCANDINAVIA Scandinavia
27,894
SOUTHAMERICA South America
52,084
SOUTHASIA South Asia
28,533
SOUTHERNAFRICA Southern Africa
8,558
SOUTHERNCONE Southern Cone
19,777
SOUTHERNEUROPE Southern Europe
66,642
SUBSAHARA Sub-Saharan Africa
18,049
WESTAFRICA Western Africa
4,464
WESTASIA Western Asia
15,601
WESTERNEUROPE Western Europe
427,732
Browse Technology Groups