Robots.txt Usage Distribution in the Top 10k Sites

Distribution for websites using Robots.txt technologies

13,546 Detections

of Robots.txt in the Top 10k Sites. Last updated 23rd Oct 2021.

Some Robots Disallow Some Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
527,350
Top 100k
Top 100,000 sites by traffic
84,058
Top 10k
Top 10,000 sites by traffic
13,546
Entire Internet
111,350,937
United States
11,077,020
United Kingdom
670,493
Germany
620,477
China
548,077
Russia
545,523
Canada
302,714
Australia
274,948
Brazil
237,114
France
232,152
Netherlands
197,457
Italy
190,634
Poland
122,995
Spain
113,990
.co
109,511
India
104,334
Japan
99,366
Sweden
88,684
Denmark
88,176
Switzerland
86,367
Belgium
49,595
New Zealand
48,900
Czech Republic
46,726
.me
44,050
South Korea
41,899
Austria
41,486
Iran
33,655
Turkey
32,838
Mexico
31,418
Indonesia
30,666
Finland
30,612
Norway
28,449
South Africa
27,785
.io
27,458
Hungary
26,212
Ukraine
25,939
Taiwan
24,536
Vietnam
24,138
Cocos Islands
23,225
Argentina
23,212
Portugal
22,839
Hong Kong
22,259
Thailand
22,028
Singapore
21,253
Ireland
19,491
Romania
18,558
.tv
15,629
Slovakia
15,538
Philippines
14,269
Greece
14,252
Chile
13,740
Malaysia
11,706
Israel
10,538
Bulgaria
8,923
Mali
7,329
Belarus
6,287
UAE
6,280
Slovenia
6,154
Latvia
5,865
Kazakhstan
5,680
Palau
5,484
Luxembourg
5,303
Pakistan
5,297
Estonia
5,280
Peru
5,148
Tonga
5,039
Lithuania
4,651
Niue
4,183
Iceland
3,863
Samoa
3,792
Uzbekistan
3,520
Serbia
3,495
Croatia
3,363
Nigeria
2,984
Anguilla
2,551
Puerto Rico
2,491
Saudi Arabia
2,327
Bangladesh
2,261
Ecuador
2,161
Costa Rica
2,097
Egypt
2,065
Morocco
1,992
Uruguay
1,819
Kenya
1,817
Isle of Man
1,692
Liechtenstein
1,630
Sri Lanka
1,494
Malta
1,435
Micronesia
1,396
Azerbaijan
1,382
Saint Helena
1,331
Belize
1,295
Guernsey
1,291
Cyprus
1,175
Panama
1,171
Moldova
1,169
Dominican Republic
1,162
Lebanon
1,088
Armenia
1,068
Macedonia
991
Guatemala
966
Reunion
964
Qatar
942
Jordan
922
Georgia
915
Seychelles
908
Kuwait
905
Venezuela
829
Saint Vincent and the Grenadines
794
Sao Tome and Principe
789
Bolivia
780
United States Virgin Islands
769
Albania
750
Afghanistan
743
Rwanda
739
Cameroon
732
Nepal
702
Christmas Island
672
Faroe Islands
618
Bahamas
594
Ghana
576
Cayman Islands
574
Congo
531
Tanzania
516
Paraguay
498
Namibia
490
Monaco
476
Somalia
470
Kyrgyzstan
457
Algeria
456
Tunisia
446
Trinidad and Tobago
446
Bermuda
445
Montserrat
436
Oman
425
Antigua and Barbuda
423
Uganda
422
Bosnia and Herzegovina
409
Honduras
404
British Virgin Islands
402
El Salvador
397
Palestine
381
Cuba
371
Bahrain
352
Haiti
348
Mauritius
346
Greenland
334
Saint Pierre and Miquelon
318
Myanmar
315
Barbados
299
Guadeloupe
295
New Caledonia
288
Gibraltar
286
Jamaica
285
Turks and Caicos Islands
281
Saint Lucia
277
Mongolia
268
Cambodia
254
Andorra
250
Zimbabwe
249
Mozambique
241
Nicaragua
231
Ethiopia
225
Jersey
212
Zambia
200
Angola
199
American Samoa
183
Madagascar
170
Senegal
161
Sint Maarten
152
Vanuatu
151
Guam
148
Bhutan
145
Guyana
143
Grenada
143
Iraq
139
Tajikistan
129
Yemen
125
Botswana
124
Cote d'Ivoire
114
Maldives
114
Papua New Guinea
113
Curacao
110
Macao
107
French Polynesia
99
Aruba
96
Martinique
93
South Georgia and the South Sandwich Islands
92
Fiji
86
Turkmenistan
79
Libya
73
Mayotte
72
Sierra Leone
70
Malawi
68
Cook Islands
61
Burkina Faso
49
Brunei
48
French Guiana
48
Suriname
40
Syria
40
Wallis and Futuna
39
Benin
38
Burundi
36
East Timor
36
Lesotho
36
Solomon Islands
33
Norfolk Island
31
Sudan
28
San Marino
28
Åland Islands
27
United States Minor Outlying Islands
25
Saint Kitts and Nevis
18
Djibouti
18
Togo
17
Lao People's Democratic Republic
17
Northern Mariana Islands
17
Mauritania
11