Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution in the Top 1 Million Sites
Technology Websites %
Robots Disallow 142,424 14.24
Googlebot Disallow 59,490 5.95
Ahrefs Bot Disallow 57,087 5.71
Yandex Bot Disallow 48,896 4.89
Google AdsBot Disallow 38,547 3.85
Some Robots Disallow 36,920 3.69
Baidu Bot Disallow 36,830 3.68
Microsoft BingBot Disallow 29,750 2.97
Semrush Bot Disallow 29,313 2.93
Majestic Bot Disallow 28,834 2.88
Google Media Partners Disallow 24,850 2.48
Alexa Bot Disallow 24,570 2.46
Yahoo Slurp Disallow 24,509 2.45
GPTBot Disallow 23,315 2.33
Sogou Bot Disallow 19,463 1.95
Soso Bot Disallow 16,849 1.68
Common Crawl Bot Disallow 16,448 1.64
Microsoft MSNBot Disallow 12,634 1.26
Google AdsBot Mobile Disallow 10,698 1.07
Google Image Bot Disallow 10,421 1.04
Anthropic Claude Bot Disallow 9,346 0.93
Google Bard Disallow 8,526 0.85
Exalead Bot Disallow 8,440 0.84
Google AdsBot Mobile Apps Disallow 6,995 0.7
Google News Bot Disallow 6,467 0.65
Facebook Bot Disallow 5,528 0.55
DuckDuckGo Bot Disallow 5,114 0.51
360 Bot Disallow 4,995 0.5
Google Video Bot Disallow 4,937 0.49
ByteDance Bytespider Disallow 4,695 0.47
AppleBot Disallow 4,558 0.46
Google Favicon Disallow 4,312 0.43
Google API Bot Disallow 4,203 0.42
Google DuplexWeb Disallow 4,108 0.41
Cohere AI Disallow 3,726 0.37
Webz Omgili Disallow 3,358 0.34
BuiltWithBot Disallow 2,115 0.21
Perplexity 1,887 0.19
Diffbot Disallow 1,437 0.14
Pipl Disallow 1,426 0.14
Apple AI Bot Disallow 1,051 0.11
YOU Disallow 999 0.1
Youdao Bot Disallow 969 0.1
Google FeedFetcher Disallow 136 0.01
Google Read Aloud Disallow 40 0
Google Web Light Disallow 13 0
Google BIG Bench Disallow 2 0

791,231 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 06th Jul 2024.

Google Media Partners Disallow Google Media Partners Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
791,231
Top 100k
Top 100,000 sites by traffic
131,611
Top 10k
Top 10,000 sites by traffic
18,044
Entire Internet
162,083,857
United States
40,831,249
Germany
7,069,709
United Kingdom
4,508,268
Russia
4,034,055
Australia
2,213,154
Italy
1,850,820
Canada
1,715,444
France
1,204,000
Brazil
1,129,600
.co
1,049,780
Japan
946,433
India
944,238
Netherlands
885,190
Spain
717,785
Switzerland
682,392
China
679,642
IO .io
677,892
PL Poland
458,369
ME .me
405,744
BE Belgium
372,432
NZ New Zealand
347,468
SE Sweden
343,553
AT Austria
297,790
MX Mexico
286,838
CZ Czech Republic
262,555
NO Norway
253,379
DK Denmark
242,916
KR South Korea
242,137
ZA South Africa
239,309
TR Turkey
233,565
CC Cocos Islands
220,546
RO Romania
211,533
IE Ireland
180,980
ID Indonesia
175,979
PH Philippines
166,518
FI Finland
164,910
TV .tv
125,996
PT Portugal
124,285
TW Taiwan
119,757
TH Thailand
118,299
SG Singapore
113,163
CL Chile
108,091
AR Argentina
102,144
HU Hungary
97,069
HK Hong Kong
95,737
UA Ukraine
92,593
GR Greece
90,068
VN Vietnam
89,454
IL Israel
81,796
VG British Virgin Islands
68,943
AE UAE
64,614
SK Slovakia
63,298
MY Malaysia
62,107
AI Anguilla
59,536
WS Samoa
45,011
LU Luxembourg
44,514
IR Iran
38,637
PK Pakistan
37,492
SA Saudi Arabia
37,240
BG Bulgaria
34,597
CM Cameroon
34,381
LT Lithuania
33,705
PE Peru
30,962
EE Estonia
30,519
LV Latvia
30,087
SI Slovenia
25,665
IS Iceland
24,793
NU Niue
23,171
HR Croatia
20,943
PW Palau
19,058
PR Puerto Rico
18,498
RS Serbia
18,349
BY Belarus
17,833
NG Nigeria
17,145
CR Costa Rica
15,743
EC Ecuador
15,653
UY Uruguay
15,149
KE Kenya
14,813
IM Isle of Man
13,269
KZ Kazakhstan
13,159
TO Tonga
12,105
LI Liechtenstein
12,007
BD Bangladesh
11,876
RE Reunion
11,766
GG Guernsey
11,487
MT Malta
10,858
VC Saint Vincent and the Grenadines
10,847
DO Dominican Republic
10,656
MA Morocco
10,303
EG Egypt
9,905
SH Saint Helena
8,938
GT Guatemala
8,841
GE Georgia
7,787
BZ Belize
7,602
PA Panama
7,399
VE Venezuela
7,152
LB Lebanon
6,710
SO Somalia
6,678
LK Sri Lanka
6,561
UG Uganda
6,390
AL Albania
6,243
GH Ghana
6,200
CY Cyprus
6,061
QA Qatar
6,024
AM Armenia
5,964
UZ Uzbekistan
5,894
FM Micronesia
5,669
NP Nepal
5,127
KW Kuwait
4,889
MK Macedonia
4,421
JO Jordan
4,216
MD Moldova
4,003
CX Christmas Island
3,894
TZ Tanzania
3,608
PY Paraguay
3,472
TN Tunisia
3,286
MU Mauritius
3,228
HN Honduras
3,210
AZ Azerbaijan
3,197
BO Bolivia
2,987
NA Namibia
2,738
OM Oman
2,737
AG Antigua and Barbuda
2,660
MC Monaco
2,652
MN Mongolia
2,562
DZ Algeria
2,481
BS Bahamas
2,408
KY Cayman Islands
2,348
ZW Zimbabwe
2,343
SV El Salvador
2,310
BH Bahrain
2,263
ST Sao Tome and Principe
2,252
BA Bosnia and Herzegovina
2,171
RW Rwanda
2,168
AF Afghanistan
2,158
GP Guadeloupe
2,011
BM Bermuda
1,973
VI United States Virgin Islands
1,965
ZM Zambia
1,958
KH Cambodia
1,905
SC Seychelles
1,842
MS Montserrat
1,825
LC Saint Lucia
1,763
TT Trinidad and Tobago
1,749
IQ Iraq
1,709
ET Ethiopia
1,676
NI Nicaragua
1,659
NC New Caledonia
1,591
MZ Mozambique
1,386
MV Maldives
1,345
MM Myanmar
1,330
PM Saint Pierre and Miquelon
1,325
JM Jamaica
1,299
CI Cote d'Ivoire
1,288
KG Kyrgyzstan
1,279
CD Congo
1,237
MQ Martinique
1,196
GI Gibraltar
1,164
SX Sint Maarten
1,142
BW Botswana
1,125
JE Jersey
1,094
YT Mayotte
1,092
SN Senegal
1,080
AS American Samoa
1,071
GL Greenland
957
FJ Fiji
942
TC Turks and Caicos Islands
930
MG Madagascar
899
FO Faroe Islands
892
AD Andorra
888
BB Barbados
865
AO Angola
831
GS South Georgia and the South Sandwich Islands
792
PS Palestine
773
AW Aruba
746
TG Togo
723
GF French Guiana
718
GY Guyana
663
HT Haiti
643
WF Wallis and Futuna
638
MW Malawi
592
GD Grenada
547
TJ Tajikistan
529
GM Gambia
483
VU Vanuatu
457
BF Burkina Faso
428
ML Mali
427
BJ Benin
413
SR Suriname
325
SL Sierra Leone
324
SB Solomon Islands
294
GU Guam
287
SD Sudan
286
YE Yemen
285
BT Bhutan
282
CU Cuba
282
TL East Timor
276
PG Papua New Guinea
275
HM Heard Island and McDonald Islands
269
TM Turkmenistan
251
BN Brunei
244
DJ Djibouti
236
NE Niger
233
BI Burundi
211
LR Liberia
201
SM San Marino
199
CK Cook Islands
197
PF French Polynesia
194
SY Syria
168
LA Lao People's Democratic Republic
162
LS Lesotho
153
AX Åland Islands
147
KN Saint Kitts and Nevis
145
MP Northern Mariana Islands
139
LY Libya
137
SZ Swaziland
128
CW Curacao
116
GA Gabon
112
MR Mauritania
111
DM Dominica
101
MO Macao
91
UM United States Minor Outlying Islands
82
NF Norfolk Island
71
MH Marshall Islands
59
CV Cabo Verde
55
TD Chad
54
SS South Sudan
53
VA Vatican City
53
SJ Svalbard and Jan Mayen
49
CG Congo
44
GN Guinea
44
GW Guinea-Bissau
44
KI Kiribati
24
FK Falkland Islands
19
BL Saint Barthélemy
18
PN Pitcairn
16
GQ Equatorial Guinea
14
AQ Antarctica
11
Categories