Robots.txt Usage Distribution in the Top 10k Sites

Distribution for websites using Robots.txt technologies

10,159 Detections

of Robots.txt in the Top 10k Sites.

Some Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
385,191
Top 100k
Top 100,000 sites by traffic
75,521
Top 10k
Top 10,000 sites by traffic
10,159
Entire Internet
72,159,388
Russia
1,105,426
United States
16,719,742
Germany
2,205,436
United Kingdom
759,545
China
595,265
Italy
550,733
Australia
380,317
Canada
324,131
Netherlands
317,211
France
288,876
Brazil
243,225
.co
192,490
Poland
187,733
India
169,776
Sweden
134,216
Spain
131,527
Switzerland
116,161
Mali
115,347
Denmark
104,810
.me
93,008
Belgium
91,304
Czech Republic
80,747
Japan
70,594
New Zealand
67,173
Austria
66,661
South Africa
63,594
Norway
57,201
South Korea
48,586
Romania
46,412
Ukraine
46,301
.tv
33,889
Finland
33,180
.io
32,915
Mexico
32,848
Taiwan
31,750
Cocos Islands
30,481
Iran
27,707
British Virgin Islands
25,920
Hungary
24,469
Palau
23,038
Hong Kong
20,201
Ireland
19,417
Turkey
18,914
Slovakia
18,825
Argentina
18,792
Vietnam
17,836
Greece
17,360
Singapore
16,936
Chile
16,911
Israel
15,894
Portugal
15,712
Indonesia
14,728
Niue
11,798
Philippines
11,767
Belarus
10,527
Kazakhstan
9,486
Malaysia
9,169
Bulgaria
8,899
UAE
7,243
Pakistan
7,106
Thailand
7,082
Samoa
6,806
Luxembourg
6,514
Slovenia
6,250
Cameroon
6,171
Tonga
6,027
Peru
5,993
Latvia
5,983
Lithuania
5,738
Estonia
5,215
Iceland
4,938
Serbia
4,760
Uzbekistan
4,554
Morocco
3,393
Liechtenstein
3,320
Nigeria
3,308
Croatia
3,039
Kenya
2,973
Isle of Man
2,789
Costa Rica
2,438
Anguilla
2,407
Egypt
2,389
Ecuador
2,385
Belize
2,304
Bosnia and Herzegovina
2,255
Saudi Arabia
2,250
Bangladesh
2,215
Reunion
2,211
Uruguay
2,093
Armenia
1,943
Dominican Republic
1,800
Saint Helena
1,677
Venezuela
1,482
Georgia
1,440
Cyprus
1,365
Puerto Rico
1,328
Moldova
1,320
Sri Lanka
1,273
Albania
1,254
Macedonia
1,244
Guernsey
1,233
Micronesia
1,221
Nepal
1,209
Panama
1,206
Guatemala
1,186
Antigua and Barbuda
1,132
Lebanon
1,059
Tunisia
1,032
Sao Tome and Principe
958
Qatar
898
Saint Vincent and the Grenadines
897
Mongolia
812
Somalia
796
Christmas Island
772
Malta
768
Azerbaijan
761
Jordan
738
Afghanistan
731
Bolivia
692
Seychelles
655
Kuwait
640
Paraguay
606
Tanzania
593
Kyrgyzstan
574
Faroe Islands
563
Montserrat
547
Uganda
526
Trinidad and Tobago
507
Cayman Islands
501
Honduras
497
Ghana
493
Saint Pierre and Miquelon
479
El Salvador
476
Mauritius
452
Jamaica
442
Greenland
432
United States Virgin Islands
431
Namibia
419
South Georgia and the South Sandwich Islands
383
Cambodia
379
Cote d'Ivoire
378
Mayotte
378
Palestine
376
Oman
371
Barbados
347
Bahamas
343
Jersey
339
Bahrain
329
Algeria
327
Haiti
327
Sint Maarten
324
Gibraltar
322
Tajikistan
311
Guyana
303
Grenada
297
Turks and Caicos Islands
292
Bermuda
283
American Samoa
282
Saint Lucia
280
Nicaragua
262
Zimbabwe
253
Wallis and Futuna
246
East Timor
245
Myanmar
225
Senegal
201
Botswana
199
Maldives
199
Madagascar
194
Cuba
186
Zambia
184
Congo
178
Ethiopia
174
New Caledonia
172
Iraq
166
Mozambique
154
Angola
154
Rwanda
152
Saint Kitts and Nevis
133
Guadeloupe
131
Burundi
118
Monaco
118
Fiji
117
Yemen
110
Vanuatu
110
Aruba
110
French Polynesia
109
Norfolk Island
108
Suriname
107
United States Minor Outlying Islands
101
Guam
97
Turkmenistan
97
Djibouti
96
Brunei
93
Andorra
89
Papua New Guinea
86
Benin
85
Malawi
83
Niger
83
Burkina Faso
80
Curacao
73
Heard Island and McDonald Islands
67
Macao
66
Dominica
61
Gambia
57
Sierra Leone
56
Martinique
52
San Marino
51
Mauritania
50
Togo
45
Sudan
43
Solomon Islands
36
Cook Islands
35
French Guiana
27
Lao People's Democratic Republic
26
Lesotho
24
Northern Mariana Islands
18
Guinea-Bissau
17
Chad
17
Syria
16
Swaziland
16
Bhutan
15
Guinea
14
Congo
12
Cabo Verde
12
Liberia
12
Åland Islands
11