510,300 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 23rd Nov 2021.

Some Robots Disallow Some Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
510,300
Top 100k
Top 100,000 sites by traffic
82,389
Top 10k
Top 10,000 sites by traffic
13,459
Entire Internet
116,237,615
United States
17,839,436
Germany
1,833,697
Russia
980,268
United Kingdom
955,432
China
551,270
Italy
477,331
Australia
381,305
Canada
365,238
France
306,021
Netherlands
296,087
.co
281,484
Brazil
234,315
Poland
199,057
India
183,162
Spain
168,479
.me
136,177
Japan
130,819
Switzerland
124,295
Sweden
116,764
Denmark
98,431
Belgium
77,553
Austria
72,956
Czech Republic
67,506
South Africa
58,385
South Korea
58,156
.io
55,318
New Zealand
54,284
Cocos Islands
49,396
Norway
45,242
.tv
42,529
Mexico
39,547
Romania
39,383
Indonesia
37,593
Finland
37,503
Iran
36,436
Turkey
35,970
Taiwan
34,383
Ukraine
31,311
Argentina
31,272
Ireland
31,052
Vietnam
29,464
Hungary
28,046
Portugal
27,044
Hong Kong
25,385
Singapore
23,908
Thailand
21,490
Slovakia
19,647
Philippines
19,027
Greece
18,269
Palau
18,222
Cameroon
17,767
Chile
17,174
Mali
16,028
British Virgin Islands
14,145
Malaysia
12,857
Israel
12,315
Tonga
11,616
Bulgaria
10,884
Samoa
8,512
Luxembourg
7,655
Latvia
7,603
UAE
7,482
Slovenia
7,424
Belarus
6,934
Estonia
6,913
Niue
6,664
Pakistan
6,499
Peru
6,265
Lithuania
6,116
Kazakhstan
5,708
Serbia
4,952
Isle of Man
4,849
Iceland
4,495
Nigeria
4,249
Anguilla
4,048
Croatia
3,913
Uzbekistan
3,570
Bangladesh
3,089
Liechtenstein
3,071
Saint Helena
2,752
Saudi Arabia
2,730
Costa Rica
2,581
Puerto Rico
2,577
Ecuador
2,496
Egypt
2,463
Morocco
2,318
Belize
2,308
Saint Vincent and the Grenadines
2,221
Kenya
2,134
Uruguay
2,129
Micronesia
2,071
Guernsey
1,894
Armenia
1,814
Reunion
1,781
Malta
1,749
Somalia
1,700
Dominican Republic
1,612
Sri Lanka
1,587
Azerbaijan
1,420
Albania
1,388
Macedonia
1,361
Georgia
1,339
Moldova
1,335
Venezuela
1,328
Seychelles
1,278
Lebanon
1,263
Cyprus
1,226
Panama
1,215
Qatar
1,175
Sao Tome and Principe
1,142
Guatemala
1,141
Christmas Island
1,084
Kuwait
1,054
Jordan
957
Faroe Islands
911
Rwanda
876
Bolivia
842
Mongolia
828
United States Virgin Islands
825
Tunisia
781
Montserrat
748
Afghanistan
731
Antigua and Barbuda
728
Nepal
725
Bahamas
692
Ghana
649
Uganda
599
Kyrgyzstan
596
Congo
580
Saint Pierre and Miquelon
570
Tanzania
558
El Salvador
546
Monaco
528
Cayman Islands
525
Trinidad and Tobago
520
Namibia
514
Algeria
513
Honduras
493
Paraguay
481
Oman
469
Bermuda
466
Guadeloupe
444
Saint Lucia
441
Bosnia and Herzegovina
436
Palestine
414
Turks and Caicos Islands
404
Mauritius
391
Greenland
371
Myanmar
370
Gibraltar
363
Mayotte
362
Grenada
353
Cambodia
350
South Georgia and the South Sandwich Islands
328
New Caledonia
323
Jamaica
321
Cuba
320
Andorra
317
Cote d'Ivoire
309
Haiti
299
Guyana
298
Ethiopia
292
Bahrain
282
Sint Maarten
274
Madagascar
272
Barbados
266
Jersey
252
American Samoa
246
Nicaragua
246
Angola
244
Tajikistan
236
Zimbabwe
230
Zambia
202
Iraq
191
Senegal
184
Mozambique
178
Yemen
164
Guam
164
Vanuatu
163
Wallis and Futuna
163
Macao
162
Maldives
140
Turkmenistan
139
Botswana
132
Bhutan
131
Aruba
125
Djibouti
116
Martinique
116
East Timor
114
French Polynesia
109
Malawi
106
Fiji
105
Chad
95
Burundi
80
Heard Island and McDonald Islands
76
Cook Islands
70
French Guiana
67
Suriname
60
Papua New Guinea
58
Curacao
57
Norfolk Island
57
Brunei
54
Solomon Islands
54
Burkina Faso
53
Libya
52
Lesotho
52
Niger
48
Syria
47
Benin
43
Togo
42
San Marino
38
Åland Islands
35
Dominica
35
Sierra Leone
34
United States Minor Outlying Islands
34
Antarctica
34
Saint Kitts and Nevis
31
Sudan
28
Mauritania
27
Gambia
25
Northern Mariana Islands
20
Marshall Islands
18
Swaziland
17
Lao People's Democratic Republic
17
Guinea-Bissau
14
Liberia
11