619,618 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 12th Sep 2023.

Robots Disallow Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
619,618
Top 100k
Top 100,000 sites by traffic
86,268
Top 10k
Top 10,000 sites by traffic
11,473
Entire Internet
144,873,209
United States
21,070,347
Germany
5,745,922
United Kingdom
4,958,754
Australia
1,735,433
Russia
1,716,923
Italy
1,672,946
Canada
1,479,089
.co
1,450,018
India
1,159,911
France
1,085,323
Brazil
986,229
Netherlands
860,287
China
809,367
.me
697,830
Japan
662,615
Spain
579,666
CH Switzerland
547,143
PL Poland
444,321
CC Cocos Islands
411,706
IO .io
315,466
BE Belgium
311,301
SE Sweden
287,122
AT Austria
250,920
MX Mexico
249,455
NZ New Zealand
239,989
ZA South Africa
229,665
TV .tv
229,080
CZ Czech Republic
207,194
DK Denmark
189,936
KR South Korea
182,042
NO Norway
173,269
TR Turkey
168,625
ID Indonesia
168,249
RO Romania
158,674
IE Ireland
136,664
FI Finland
122,257
VN Vietnam
104,142
TW Taiwan
100,709
UA Ukraine
97,552
PT Portugal
96,016
SG Singapore
87,228
CL Chile
85,712
AR Argentina
79,130
TH Thailand
76,663
HK Hong Kong
75,167
ML Mali
72,890
HU Hungary
71,673
PH Philippines
69,117
PW Palau
67,300
VG British Virgin Islands
65,778
IL Israel
60,551
GR Greece
59,327
MY Malaysia
54,448
SK Slovakia
53,224
WS Samoa
49,272
AE UAE
47,122
IR Iran
43,232
CM Cameroon
42,217
PK Pakistan
35,649
LU Luxembourg
35,366
AI Anguilla
29,151
PE Peru
28,479
BG Bulgaria
28,079
LT Lithuania
25,501
LV Latvia
23,643
IM Isle of Man
21,964
EE Estonia
21,768
SA Saudi Arabia
20,212
NG Nigeria
20,040
SI Slovenia
19,546
SH Saint Helena
17,630
IS Iceland
17,460
TO Tonga
16,915
NU Niue
16,571
RS Serbia
15,941
BY Belarus
15,638
HR Croatia
15,123
BD Bangladesh
14,912
GG Guernsey
13,269
LI Liechtenstein
12,738
KE Kenya
12,728
VC Saint Vincent and the Grenadines
12,038
KZ Kazakhstan
11,817
PR Puerto Rico
11,470
UY Uruguay
10,591
EC Ecuador
10,305
RE Reunion
10,113
CR Costa Rica
9,600
BZ Belize
9,313
MA Morocco
8,920
UZ Uzbekistan
7,757
EG Egypt
7,746
SO Somalia
7,518
FM Micronesia
7,355
MT Malta
6,992
LK Sri Lanka
6,854
UG Uganda
6,833
SX Sint Maarten
6,661
VE Venezuela
5,850
AM Armenia
5,691
DO Dominican Republic
5,549
GE Georgia
5,108
NP Nepal
5,090
AL Albania
5,085
GT Guatemala
5,082
GH Ghana
5,079
PA Panama
4,639
LB Lebanon
4,497
QA Qatar
4,326
MK Macedonia
4,045
CY Cyprus
4,034
CX Christmas Island
3,868
KW Kuwait
3,565
MN Mongolia
3,528
AG Antigua and Barbuda
3,381
JO Jordan
3,226
MD Moldova
3,188
LC Saint Lucia
2,995
TN Tunisia
2,870
AZ Azerbaijan
2,495
ST Sao Tome and Principe
2,493
BO Bolivia
2,418
PY Paraguay
2,413
MU Mauritius
2,208
KY Cayman Islands
2,188
ET Ethiopia
2,033
NA Namibia
1,877
AF Afghanistan
1,803
MS Montserrat
1,800
HN Honduras
1,793
RW Rwanda
1,791
MQ Martinique
1,776
DZ Algeria
1,774
SC Seychelles
1,770
ZW Zimbabwe
1,697
MC Monaco
1,695
BS Bahamas
1,662
GP Guadeloupe
1,619
SV El Salvador
1,574
OM Oman
1,540
TZ Tanzania
1,428
ZM Zambia
1,394
VI United States Virgin Islands
1,390
PM Saint Pierre and Miquelon
1,384
JM Jamaica
1,177
KG Kyrgyzstan
1,164
BM Bermuda
1,117
NC New Caledonia
1,048
MM Myanmar
1,035
IQ Iraq
1,025
BA Bosnia and Herzegovina
989
KH Cambodia
953
GL Greenland
927
NI Nicaragua
864
MZ Mozambique
804
GI Gibraltar
762
CD Congo
754
PS Palestine
713
AD Andorra
712
AS American Samoa
683
TT Trinidad and Tobago
664
FO Faroe Islands
622
WF Wallis and Futuna
612
TG Togo
601
FJ Fiji
575
MG Madagascar
558
GF French Guiana
542
CI Cote d'Ivoire
517
GD Grenada
517
GY Guyana
508
BH Bahrain
491
SN Senegal
482
TJ Tajikistan
466
MV Maldives
464
BB Barbados
434
HT Haiti
389
BJ Benin
361
GU Guam
361
YT Mayotte
357
BF Burkina Faso
353
AO Angola
317
JE Jersey
314
BW Botswana
274
LR Liberia
274
TL East Timor
271
GM Gambia
268
GS South Georgia and the South Sandwich Islands
255
TC Turks and Caicos Islands
243
MW Malawi
234
TM Turkmenistan
221
CK Cook Islands
212
HM Heard Island and McDonald Islands
209
AW Aruba
200
SR Suriname
197
BI Burundi
186
DJ Djibouti
168
CU Cuba
166
VU Vanuatu
157
SB Solomon Islands
142
KN Saint Kitts and Nevis
140
YE Yemen
140
BN Brunei
129
SD Sudan
125
MO Macao
123
PF French Polynesia
123
SM San Marino
112
BT Bhutan
103
PG Papua New Guinea
102
SL Sierra Leone
89
NE Niger
86
MH Marshall Islands
80
DM Dominica
76
UM United States Minor Outlying Islands
64
AX Åland Islands
59
SY Syria
59
CW Curacao
45
GA Gabon
43
MP Northern Mariana Islands
43
NF Norfolk Island
40
LS Lesotho
39
LA Lao People's Democratic Republic
38
MR Mauritania
38
SZ Swaziland
38
GN Guinea
31
CV Cabo Verde
28
LY Libya
26
TD Chad
24
CG Congo
20
FK Falkland Islands
20
GW Guinea-Bissau
18
SS South Sudan
14
KI Kiribati
12