88,969 Detections

of Robots.txt in the Top 100k Sites. Last updated 02nd Aug 2022.

Robots Disallow Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
543,854
Top 100k
Top 100,000 sites by traffic
88,969
Top 10k
Top 10,000 sites by traffic
12,806
Entire Internet
142,065,946
United States
20,698,053
Germany
5,457,019
United Kingdom
4,338,206
Russia
1,984,552
Australia
1,582,026
Italy
1,550,451
Canada
1,291,704
.co
1,286,046
China
970,965
France
953,468
India
946,783
Brazil
935,911
Netherlands
771,746
.me
747,218
Poland
535,987
Japan
529,112
ES Spain
491,527
CH Switzerland
470,607
BE Belgium
295,286
CC Cocos Islands
286,277
SE Sweden
276,366
IO .io
256,446
AT Austria
231,828
NZ New Zealand
211,934
DK Denmark
210,665
ZA South Africa
209,457
MX Mexico
204,620
KR South Korea
187,733
TV .tv
186,351
CZ Czech Republic
179,193
NO Norway
156,907
TR Turkey
131,176
RO Romania
126,520
FI Finland
121,202
UA Ukraine
104,926
IE Ireland
99,104
ID Indonesia
96,205
TW Taiwan
90,954
AR Argentina
87,599
PT Portugal
81,537
VN Vietnam
81,085
CL Chile
78,592
PW Palau
76,714
SG Singapore
70,814
HK Hong Kong
66,258
HU Hungary
62,601
IR Iran
59,306
VG British Virgin Islands
57,727
CM Cameroon
51,521
IL Israel
50,594
GR Greece
50,257
TH Thailand
49,442
MY Malaysia
48,472
SK Slovakia
47,342
PH Philippines
41,379
TO Tonga
37,523
WS Samoa
26,949
BG Bulgaria
24,029
LU Luxembourg
21,608
AE UAE
21,428
LV Latvia
19,353
LT Lithuania
19,079
EE Estonia
18,267
IM Isle of Man
18,094
BY Belarus
17,886
PE Peru
17,126
SA Saudi Arabia
16,748
PK Pakistan
16,122
NU Niue
16,024
SI Slovenia
15,994
IS Iceland
14,062
AI Anguilla
13,068
RS Serbia
12,694
KZ Kazakhstan
11,513
NG Nigeria
10,870
LI Liechtenstein
10,818
SH Saint Helena
10,654
PR Puerto Rico
10,386
BD Bangladesh
10,196
HR Croatia
10,020
GG Guernsey
9,102
RE Reunion
8,894
UY Uruguay
7,875
KE Kenya
7,204
BZ Belize
7,124
VC Saint Vincent and the Grenadines
6,555
EC Ecuador
6,406
SO Somalia
6,300
MA Morocco
6,164
FM Micronesia
5,942
CR Costa Rica
5,918
UZ Uzbekistan
5,344
AM Armenia
5,226
EG Egypt
4,738
DO Dominican Republic
4,406
AL Albania
4,125
GE Georgia
4,057
MT Malta
4,027
VE Venezuela
3,808
GT Guatemala
3,261
CY Cyprus
3,056
LK Sri Lanka
3,056
CX Christmas Island
2,824
MD Moldova
2,607
MK Macedonia
2,601
LB Lebanon
2,536
SC Seychelles
2,365
AZ Azerbaijan
2,332
ST Sao Tome and Principe
2,292
TN Tunisia
2,134
PA Panama
2,118
FO Faroe Islands
2,117
QA Qatar
2,114
AG Antigua and Barbuda
1,951
NP Nepal
1,904
ML Mali
1,838
MN Mongolia
1,597
UG Uganda
1,558
BM Bermuda
1,367
BS Bahamas
1,365
MC Monaco
1,345
RW Rwanda
1,335
MS Montserrat
1,329
KW Kuwait
1,300
JO Jordan
1,287
BO Bolivia
1,243
YT Mayotte
1,212
SV El Salvador
1,164
HN Honduras
1,134
AF Afghanistan
1,060
PY Paraguay
1,047
NA Namibia
1,036
DZ Algeria
1,029
NC New Caledonia
996
GL Greenland
969
GH Ghana
946
KY Cayman Islands
930
PM Saint Pierre and Miquelon
870
SX Sint Maarten
852
KG Kyrgyzstan
824
TZ Tanzania
821
ZW Zimbabwe
816
MU Mauritius
806
BA Bosnia and Herzegovina
789
LC Saint Lucia
692
BB Barbados
681
BH Bahrain
661
KH Cambodia
652
TT Trinidad and Tobago
612
PS Palestine
611
AS American Samoa
592
MZ Mozambique
531
VI United States Virgin Islands
519
GP Guadeloupe
517
GI Gibraltar
513
TC Turks and Caicos Islands
500
JM Jamaica
473
GF French Guiana
446
SN Senegal
426
CD Congo
418
GD Grenada
390
BW Botswana
376
GU Guam
375
MQ Martinique
370
GY Guyana
359
TJ Tajikistan
359
MG Madagascar
345
NI Nicaragua
338
MM Myanmar
334
OM Oman
328
CI Cote d'Ivoire
304
JE Jersey
299
FJ Fiji
292
GS South Georgia and the South Sandwich Islands
270
AO Angola
226
ET Ethiopia
225
CU Cuba
223
MV Maldives
201
TM Turkmenistan
198
HT Haiti
192
AD Andorra
182
TL East Timor
182
ZM Zambia
181
YE Yemen
171
WF Wallis and Futuna
157
DJ Djibouti
156
IQ Iraq
148
VU Vanuatu
147
KN Saint Kitts and Nevis
141
PF French Polynesia
136
BJ Benin
125
MO Macao
121
HM Heard Island and McDonald Islands
103
AW Aruba
100
CK Cook Islands
99
MW Malawi
98
BI Burundi
93
PG Papua New Guinea
93
BN Brunei
92
SR Suriname
91
SB Solomon Islands
85
SM San Marino
78
TG Togo
73
UM United States Minor Outlying Islands
72
SL Sierra Leone
71
BF Burkina Faso
68
CW Curacao
66
AX Åland Islands
62
NE Niger
59
SD Sudan
59
NF Norfolk Island
58
BT Bhutan
45
GM Gambia
44
LS Lesotho
40
MR Mauritania
33
MP Northern Mariana Islands
30
SJ Svalbard and Jan Mayen
29
SY Syria
29
DM Dominica
27
SZ Swaziland
21
LR Liberia
19
LY Libya
17
TD Chad
17
GN Guinea
16
LA Lao People's Democratic Republic
16
GW Guinea-Bissau
14
AQ Antarctica
13
KI Kiribati
12
FK Falkland Islands
11
MH Marshall Islands
11