Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution in the Top 1 Million Sites
Technology Websites %
Robots Disallow 163,830 16.38
Ahrefs Bot Disallow 71,886 7.19
Googlebot Disallow 58,251 5.83
Google AdsBot Disallow 50,603 5.06
Yandex Bot Disallow 46,822 4.68
Some Robots Disallow 41,316 4.13
GPTBot Disallow 35,308 3.53
Semrush Bot Disallow 35,058 3.51
Majestic Bot Disallow 34,277 3.43
Baidu Bot Disallow 33,147 3.31
Microsoft BingBot Disallow 27,979 2.8
Common Crawl Bot Disallow 26,065 2.61
Anthropic Claude Bot Disallow 23,296 2.33
AppleBot Disallow 22,245 2.22
Yahoo Slurp Disallow 21,138 2.11
Sogou Bot Disallow 19,525 1.95
Google Media Partners Disallow 19,432 1.94
Alexa Bot Disallow 18,774 1.88
Google Bard Disallow 17,232 1.72
Facebook Bot Disallow 15,487 1.55
ByteDance Bytespider Disallow 14,446 1.44
Microsoft MSNBot Disallow 14,033 1.4
Perplexity 13,176 1.32
Google AdsBot Mobile Disallow 13,012 1.3
Google Image Bot Disallow 11,685 1.17
Cohere AI Disallow 11,356 1.14
Apple AI Bot Disallow 11,261 1.13
Webz Omgili Disallow 10,859 1.09
Soso Bot Disallow 10,809 1.08
Exalead Bot Disallow 9,944 0.99
Google AdsBot Mobile Apps Disallow 7,843 0.78
Diffbot Disallow 7,265 0.73
Google News Bot Disallow 7,091 0.71
DuckDuckGo Bot Disallow 6,521 0.65
360 Bot Disallow 6,376 0.64
Google Video Bot Disallow 5,678 0.57
YOU Disallow 5,573 0.56
Google Favicon Disallow 4,959 0.5
Google API Bot Disallow 4,910 0.49
Google DuplexWeb Disallow 4,765 0.48
Pipl Disallow 3,320 0.33
BuiltWithBot Disallow 2,791 0.28
Youdao Bot Disallow 906 0.09
LLMS Text 487 0.05
Google FeedFetcher Disallow 169 0.02
Google Read Aloud Disallow 91 0.01
Google Web Light Disallow 31 0
Google BIG Bench Disallow 4 0

971,032 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 13th May 2025.

Robots Disallow Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
971,032
Top 100k
Top 100,000 sites by traffic
180,637
Top 10k
Top 10,000 sites by traffic
27,142
Entire Internet
203,667,593
United States
40,359,756
Germany
6,760,216
United Kingdom
4,369,021
Russia
2,778,598
Australia
2,737,625
Canada
1,986,848
France
1,271,026
Italy
1,208,148
Brazil
1,021,104
Spain
986,747
China
984,108
Netherlands
970,613
Switzerland
876,483
.co
834,097
India
765,086
Japan
710,070
PL Poland
496,977
SE Sweden
445,942
BE Belgium
417,997
CC Cocos Islands
414,575
NZ New Zealand
411,031
AT Austria
375,478
ME .me
351,412
NO Norway
307,957
CZ Czech Republic
296,526
DK Denmark
287,640
MX Mexico
276,526
IO .io
274,844
IE Ireland
256,573
ZA South Africa
247,041
KR South Korea
228,013
RO Romania
219,810
TR Turkey
210,698
FI Finland
198,742
ID Indonesia
185,692
PT Portugal
158,587
PH Philippines
139,237
HU Hungary
133,709
AI Anguilla
121,588
SG Singapore
119,129
TV .tv
112,947
GR Greece
103,014
CL Chile
95,110
HK Hong Kong
93,627
SK Slovakia
89,422
TH Thailand
86,877
UA Ukraine
86,815
AR Argentina
86,361
TW Taiwan
83,648
VN Vietnam
83,349
AE UAE
72,010
MY Malaysia
67,391
IL Israel
67,299
LU Luxembourg
57,426
VG British Virgin Islands
54,229
WS Samoa
44,327
PK Pakistan
40,899
SI Slovenia
38,539
SA Saudi Arabia
37,585
LT Lithuania
36,898
BG Bulgaria
35,982
EE Estonia
33,626
IR Iran
32,213
IS Iceland
31,978
LV Latvia
31,806
PE Peru
30,533
CM Cameroon
28,851
IM Isle of Man
25,382
EC Ecuador
24,199
HR Croatia
21,984
CR Costa Rica
19,130
NU Niue
18,775
RS Serbia
18,270
NG Nigeria
18,242
PR Puerto Rico
18,143
PW Palau
17,376
KE Kenya
17,089
TO Tonga
15,236
BD Bangladesh
15,087
LI Liechtenstein
15,055
BY Belarus
14,779
RE Reunion
14,619
GG Guernsey
14,573
UY Uruguay
14,317
SH Saint Helena
14,303
VC Saint Vincent and the Grenadines
14,133
KZ Kazakhstan
12,923
MT Malta
12,918
CY Cyprus
11,937
DO Dominican Republic
11,739
MA Morocco
11,644
EG Egypt
10,872
GT Guatemala
9,958
AM Armenia
9,170
BZ Belize
9,085
PM Saint Pierre and Miquelon
8,856
LB Lebanon
8,636
AL Albania
8,419
SO Somalia
8,306
VE Venezuela
8,233
PA Panama
7,963
LK Sri Lanka
7,618
UG Uganda
7,499
GE Georgia
7,309
QA Qatar
7,273
FM Micronesia
7,070
GH Ghana
6,423
UZ Uzbekistan
6,058
NP Nepal
5,837
TZ Tanzania
5,475
CX Christmas Island
5,470
KW Kuwait
5,465
ET Ethiopia
4,760
JO Jordan
4,759
MD Moldova
4,689
MK Macedonia
4,553
HN Honduras
3,918
PY Paraguay
3,616
AG Antigua and Barbuda
3,606
KH Cambodia
3,569
GP Guadeloupe
3,463
BS Bahamas
3,378
BO Bolivia
3,359
DZ Algeria
3,265
TN Tunisia
3,263
SV El Salvador
3,216
MU Mauritius
3,198
ST Sao Tome and Principe
3,117
BH Bahrain
3,093
MC Monaco
2,998
AZ Azerbaijan
2,921
OM Oman
2,907
BA Bosnia and Herzegovina
2,852
RW Rwanda
2,799
BM Bermuda
2,786
MN Mongolia
2,778
KY Cayman Islands
2,768
NA Namibia
2,570
JE Jersey
2,558
AD Andorra
2,488
SC Seychelles
2,475
AF Afghanistan
2,427
FO Faroe Islands
2,219
TT Trinidad and Tobago
2,134
ZW Zimbabwe
2,124
JM Jamaica
2,076
ZM Zambia
1,923
SX Sint Maarten
1,914
VI United States Virgin Islands
1,910
NI Nicaragua
1,829
MS Montserrat
1,811
IQ Iraq
1,714
LC Saint Lucia
1,707
MQ Martinique
1,597
BB Barbados
1,547
YT Mayotte
1,497
MZ Mozambique
1,469
GD Grenada
1,467
KG Kyrgyzstan
1,456
CI Cote d'Ivoire
1,389
MV Maldives
1,372
MM Myanmar
1,370
AO Angola
1,297
SN Senegal
1,265
GL Greenland
1,251
GY Guyana
1,241
NC New Caledonia
1,233
TC Turks and Caicos Islands
1,203
FJ Fiji
1,200
BW Botswana
1,173
GI Gibraltar
1,167
PS Palestine
1,136
AS American Samoa
1,121
MG Madagascar
1,004
GS South Georgia and the South Sandwich Islands
971
CD Congo
941
AW Aruba
935
HT Haiti
847
AX Åland Islands
817
DJ Djibouti
758
ML Mali
728
WF Wallis and Futuna
655
GF French Guiana
635
MW Malawi
602
GM Gambia
590
VU Vanuatu
579
TJ Tajikistan
577
CU Cuba
567
TG Togo
560
PG Papua New Guinea
557
BJ Benin
467
BF Burkina Faso
460
TL East Timor
426
BN Brunei
388
GU Guam
386
SL Sierra Leone
375
SY Syria
327
SB Solomon Islands
316
LY Libya
315
LR Liberia
280
SR Suriname
269
BT Bhutan
266
BI Burundi
266
SM San Marino
263
TM Turkmenistan
263
LA Lao People's Democratic Republic
260
CK Cook Islands
253
SD Sudan
235
HM Heard Island and McDonald Islands
228
MR Mauritania
228
PF French Polynesia
227
MO Macao
221
YE Yemen
214
SZ Swaziland
201
LS Lesotho
191
NE Niger
168
SJ Svalbard and Jan Mayen
162
CW Curacao
159
KN Saint Kitts and Nevis
138
DM Dominica
134
MP Northern Mariana Islands
118
GA Gabon
114
NF Norfolk Island
94
SS South Sudan
84
CV Cabo Verde
64
UM United States Minor Outlying Islands
62
VA Vatican City
62
KI Kiribati
61
FK Falkland Islands
57
GN Guinea
52
CG Congo
42
PN Pitcairn
42
GW Guinea-Bissau
36
TD Chad
36
AQ Antarctica
31
MH Marshall Islands
30
BL Saint Barthélemy
29
NR Nauru
22
BQ Bonaire
20
Robots.txt Categories