586,084 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 14th Jan 2024.

Robots Disallow Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
586,084
Top 100k
Top 100,000 sites by traffic
89,232
Top 10k
Top 10,000 sites by traffic
11,768
Entire Internet
137,052,248
United States
31,700,627
Germany
4,884,497
United Kingdom
4,802,982
Russia
2,025,219
Australia
1,914,676
Canada
1,683,599
Italy
1,620,920
.co
1,211,883
India
1,144,443
Brazil
1,063,565
France
1,017,361
Japan
827,708
Netherlands
722,872
China
646,232
.me
599,715
Spain
587,140
CH Switzerland
576,358
IO .io
549,675
PL Poland
414,179
BE Belgium
314,330
SE Sweden
305,384
MX Mexico
298,558
CC Cocos Islands
297,362
NZ New Zealand
286,168
AT Austria
235,853
CZ Czech Republic
220,645
TR Turkey
213,974
DK Denmark
211,122
NO Norway
204,948
KR South Korea
201,867
ZA South Africa
200,046
TV .tv
192,477
RO Romania
169,750
IE Ireland
153,440
PH Philippines
152,162
ID Indonesia
146,787
FI Finland
137,611
PT Portugal
104,788
SG Singapore
104,512
TW Taiwan
102,822
VN Vietnam
97,285
UA Ukraine
96,470
TH Thailand
95,433
CL Chile
94,566
AR Argentina
88,149
HK Hong Kong
86,921
HU Hungary
79,831
IL Israel
73,475
GR Greece
68,796
VG British Virgin Islands
67,362
MY Malaysia
59,622
AE UAE
58,717
SK Slovakia
55,120
WS Samoa
43,452
AI Anguilla
38,226
LU Luxembourg
37,723
PK Pakistan
37,370
IR Iran
36,906
PW Palau
33,325
SA Saudi Arabia
32,734
BG Bulgaria
29,917
LT Lithuania
28,270
PE Peru
28,244
CM Cameroon
27,384
EE Estonia
24,707
LV Latvia
24,267
SI Slovenia
21,126
IS Iceland
20,331
NG Nigeria
18,564
HR Croatia
17,032
RS Serbia
15,438
NU Niue
15,251
PR Puerto Rico
14,877
BD Bangladesh
14,775
GG Guernsey
13,700
UY Uruguay
13,552
CR Costa Rica
13,289
EC Ecuador
12,982
KE Kenya
12,933
BY Belarus
12,884
IM Isle of Man
10,783
KZ Kazakhstan
10,703
SH Saint Helena
9,409
MA Morocco
9,363
EG Egypt
9,210
RE Reunion
9,100
LI Liechtenstein
8,848
TO Tonga
8,771
MT Malta
8,324
LK Sri Lanka
7,806
SO Somalia
7,767
BZ Belize
7,631
GT Guatemala
7,214
LC Saint Lucia
7,111
DO Dominican Republic
6,832
GH Ghana
6,740
UG Uganda
6,707
GE Georgia
6,475
VE Venezuela
6,076
PA Panama
6,047
QA Qatar
5,696
LB Lebanon
5,460
FM Micronesia
5,422
AM Armenia
5,201
NP Nepal
5,065
CY Cyprus
5,029
UZ Uzbekistan
4,927
AL Albania
4,532
TZ Tanzania
4,406
KW Kuwait
4,327
VC Saint Vincent and the Grenadines
3,868
MK Macedonia
3,845
JO Jordan
3,484
MD Moldova
3,070
GP Guadeloupe
2,977
MU Mauritius
2,974
PY Paraguay
2,939
TN Tunisia
2,891
AZ Azerbaijan
2,811
CX Christmas Island
2,632
OM Oman
2,590
HN Honduras
2,589
AG Antigua and Barbuda
2,502
BO Bolivia
2,471
NA Namibia
2,326
MQ Martinique
2,309
KY Cayman Islands
2,282
ZW Zimbabwe
2,197
MC Monaco
2,190
BS Bahamas
2,173
SV El Salvador
1,896
ST Sao Tome and Principe
1,855
MN Mongolia
1,838
ZM Zambia
1,750
AF Afghanistan
1,734
BH Bahrain
1,712
MS Montserrat
1,682
VI United States Virgin Islands
1,668
SC Seychelles
1,610
BM Bermuda
1,594
DZ Algeria
1,592
ET Ethiopia
1,575
KH Cambodia
1,570
BA Bosnia and Herzegovina
1,455
RW Rwanda
1,426
NC New Caledonia
1,412
JM Jamaica
1,388
NI Nicaragua
1,354
IQ Iraq
1,277
MV Maldives
1,220
PM Saint Pierre and Miquelon
1,190
MM Myanmar
1,169
CD Congo
1,118
KG Kyrgyzstan
1,067
MZ Mozambique
1,064
TT Trinidad and Tobago
959
SX Sint Maarten
944
GI Gibraltar
924
CI Cote d'Ivoire
826
MG Madagascar
808
FJ Fiji
790
GL Greenland
774
YT Mayotte
750
TC Turks and Caicos Islands
726
GD Grenada
723
SN Senegal
697
AS American Samoa
655
GF French Guiana
639
JE Jersey
625
AD Andorra
609
PS Palestine
604
AO Angola
579
BB Barbados
545
WF Wallis and Futuna
537
FO Faroe Islands
534
BW Botswana
532
TG Togo
529
GM Gambia
470
MW Malawi
452
BJ Benin
449
HT Haiti
441
AW Aruba
434
GY Guyana
426
BF Burkina Faso
403
GU Guam
394
GS South Georgia and the South Sandwich Islands
388
TJ Tajikistan
323
BT Bhutan
237
HM Heard Island and McDonald Islands
226
SR Suriname
224
CU Cuba
208
SB Solomon Islands
202
LR Liberia
201
NE Niger
201
SD Sudan
197
TM Turkmenistan
195
TL East Timor
185
BN Brunei
184
SM San Marino
166
YE Yemen
164
SL Sierra Leone
157
DJ Djibouti
156
KN Saint Kitts and Nevis
154
VU Vanuatu
154
PF French Polynesia
148
BI Burundi
145
CK Cook Islands
142
PG Papua New Guinea
139
SY Syria
130
ML Mali
113
AX Åland Islands
94
LS Lesotho
90
MR Mauritania
82
GA Gabon
72
LA Lao People's Democratic Republic
72
LY Libya
67
SZ Swaziland
67
DM Dominica
65
UM United States Minor Outlying Islands
64
MO Macao
63
NF Norfolk Island
62
MP Northern Mariana Islands
61
CW Curacao
59
MH Marshall Islands
57
GN Guinea
39
CG Congo
36
CV Cabo Verde
36
VA Vatican City
34
GW Guinea-Bissau
30
SS South Sudan
29
TD Chad
21
SJ Svalbard and Jan Mayen
20
KI Kiribati
17
FK Falkland Islands
15
BL Saint Barthélemy
12
PN Pitcairn
11