Robots.txt Usage Distribution in the Top 1 Million Sites

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution in the Top 1 Million Sites
Technology Websites %
Robots Disallow 154,655 15.47
Ahrefs Bot Disallow 69,338 6.93
Googlebot Disallow 55,267 5.53
Google AdsBot Disallow 48,731 4.87
Yandex Bot Disallow 45,588 4.56
Some Robots Disallow 38,359 3.84
GPTBot Disallow 34,539 3.45
Semrush Bot Disallow 34,212 3.42
Majestic Bot Disallow 33,230 3.32
Baidu Bot Disallow 31,393 3.14
Microsoft BingBot Disallow 27,160 2.72
Common Crawl Bot Disallow 25,232 2.52
Anthropic Claude Bot Disallow 23,101 2.31
Yahoo Slurp Disallow 20,597 2.06
AppleBot Disallow 20,580 2.06
Sogou Bot Disallow 18,614 1.86
Alexa Bot Disallow 18,287 1.83
Google Media Partners Disallow 17,850 1.78
Google Bard Disallow 17,171 1.72
Facebook Bot Disallow 15,419 1.54
ByteDance Bytespider Disallow 14,388 1.44
Perplexity 13,250 1.32
Microsoft MSNBot Disallow 13,126 1.31
Google AdsBot Mobile Disallow 12,158 1.22
Apple AI Bot Disallow 11,340 1.13
Cohere AI Disallow 10,972 1.1
Webz Omgili Disallow 10,880 1.09
Google Image Bot Disallow 10,879 1.09
Soso Bot Disallow 10,576 1.06
Exalead Bot Disallow 9,530 0.95
Diffbot Disallow 7,500 0.75
Google AdsBot Mobile Apps Disallow 6,911 0.69
Google News Bot Disallow 6,433 0.64
360 Bot Disallow 6,152 0.62
YOU Disallow 6,033 0.6
DuckDuckGo Bot Disallow 5,927 0.59
Google Video Bot Disallow 5,063 0.51
Google Favicon Disallow 4,375 0.44
Google API Bot Disallow 4,325 0.43
Google DuplexWeb Disallow 4,178 0.42
Pipl Disallow 3,745 0.37
BuiltWithBot Disallow 2,647 0.26
Youdao Bot Disallow 863 0.09
LLMS Text 719 0.07
Anthropic Claude User Bot Disallow 306 0.03
Anthropic Claude Search Bot Disallow 299 0.03
Google FeedFetcher Disallow 164 0.02
Google Read Aloud Disallow 105 0.01
Google Web Light Disallow 29 0
Google BIG Bench Disallow 2 0

932,198 Detections

of Robots.txt in the Top 1 Million Sites. Last updated 07th Jun 2025.

Robots Disallow Robots Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
932,198
Top 100k
Top 100,000 sites by traffic
172,896
Top 10k
Top 10,000 sites by traffic
24,867
Entire Internet
209,940,517
United States
42,511,217
Germany
7,874,628
United Kingdom
5,233,029
Australia
3,041,080
Russia
2,876,830
Canada
2,247,463
France
1,399,851
Italy
1,285,784
Netherlands
1,156,875
Brazil
1,113,861
Spain
1,081,562
China
1,064,785
Switzerland
1,006,400
.co
999,900
India
896,944
Japan
888,458
CC Cocos Islands
574,146
PL Poland
523,882
BE Belgium
484,521
SE Sweden
482,875
NZ New Zealand
459,111
AT Austria
445,305
ME .me
398,921
IO .io
357,839
NO Norway
331,579
CZ Czech Republic
321,177
MX Mexico
320,221
DK Denmark
303,562
ZA South Africa
291,659
IE Ireland
291,328
KR South Korea
262,058
TR Turkey
241,668
RO Romania
227,339
FI Finland
219,277
ID Indonesia
195,658
PH Philippines
195,053
AI Anguilla
193,389
PT Portugal
176,327
TV .tv
139,011
SG Singapore
136,890
HU Hungary
135,665
GR Greece
111,562
CL Chile
110,720
HK Hong Kong
106,283
TH Thailand
98,531
AR Argentina
96,269
SK Slovakia
95,850
UA Ukraine
94,499
TW Taiwan
89,285
VN Vietnam
86,797
IL Israel
82,865
MY Malaysia
82,553
AE UAE
80,344
VG British Virgin Islands
73,503
LU Luxembourg
67,889
WS Samoa
60,377
SI Slovenia
44,026
PK Pakistan
42,065
LT Lithuania
40,318
EE Estonia
39,856
BG Bulgaria
38,398
IR Iran
37,954
IS Iceland
37,551
SA Saudi Arabia
35,951
LV Latvia
35,408
PE Peru
33,914
CM Cameroon
30,563
IM Isle of Man
30,439
EC Ecuador
28,006
HR Croatia
24,698
CY Cyprus
23,882
PW Palau
20,960
CR Costa Rica
20,948
NU Niue
20,703
GG Guernsey
20,597
PR Puerto Rico
20,014
NG Nigeria
19,801
RS Serbia
19,713
VC Saint Vincent and the Grenadines
19,560
SH Saint Helena
19,375
TO Tonga
19,057
KE Kenya
18,910
RE Reunion
18,034
LI Liechtenstein
17,805
UY Uruguay
16,323
BD Bangladesh
16,138
BY Belarus
15,198
MT Malta
14,437
KZ Kazakhstan
13,390
DO Dominican Republic
12,468
MA Morocco
12,361
SO Somalia
11,838
EG Egypt
11,559
AM Armenia
11,184
GT Guatemala
10,964
BZ Belize
10,128
PM Saint Pierre and Miquelon
9,423
VE Venezuela
9,361
AL Albania
9,339
LB Lebanon
9,309
PA Panama
8,913
FM Micronesia
8,893
LK Sri Lanka
8,624
QA Qatar
8,274
GE Georgia
8,265
UG Uganda
8,259
CX Christmas Island
7,395
GH Ghana
7,077
NP Nepal
6,624
UZ Uzbekistan
6,044
KW Kuwait
5,876
TZ Tanzania
5,785
MD Moldova
5,177
JO Jordan
5,155
MK Macedonia
4,987
ET Ethiopia
4,870
AG Antigua and Barbuda
4,331
HN Honduras
4,223
PY Paraguay
4,072
GP Guadeloupe
3,845
KH Cambodia
3,834
BO Bolivia
3,594
BS Bahamas
3,560
SV El Salvador
3,484
AD Andorra
3,383
TN Tunisia
3,359
ST Sao Tome and Principe
3,342
BH Bahrain
3,319
AZ Azerbaijan
3,235
MC Monaco
3,222
OM Oman
3,216
KY Cayman Islands
3,168
BA Bosnia and Herzegovina
3,127
MU Mauritius
3,107
DZ Algeria
3,098
NA Namibia
3,071
BM Bermuda
3,027
MN Mongolia
2,927
RW Rwanda
2,896
SC Seychelles
2,809
JE Jersey
2,708
ZW Zimbabwe
2,575
AF Afghanistan
2,402
FO Faroe Islands
2,390
TT Trinidad and Tobago
2,330
MS Montserrat
2,267
JM Jamaica
2,163
SX Sint Maarten
2,137
ZM Zambia
2,127
VI United States Virgin Islands
1,915
LC Saint Lucia
1,863
NI Nicaragua
1,856
IQ Iraq
1,855
YT Mayotte
1,828
MQ Martinique
1,816
CI Cote d'Ivoire
1,663
BB Barbados
1,656
MV Maldives
1,625
MZ Mozambique
1,612
GD Grenada
1,609
KG Kyrgyzstan
1,519
MM Myanmar
1,493
AO Angola
1,475
NC New Caledonia
1,418
GL Greenland
1,378
GY Guyana
1,366
SN Senegal
1,361
BW Botswana
1,324
AS American Samoa
1,313
PS Palestine
1,202
TC Turks and Caicos Islands
1,160
GI Gibraltar
1,152
FJ Fiji
1,137
MG Madagascar
1,095
DJ Djibouti
1,044
GS South Georgia and the South Sandwich Islands
1,009
CD Congo
999
AW Aruba
985
AX Åland Islands
922
HT Haiti
842
TJ Tajikistan
677
WF Wallis and Futuna
670
VU Vanuatu
660
GM Gambia
647
MW Malawi
623
PG Papua New Guinea
600
TG Togo
583
ML Mali
554
BF Burkina Faso
508
TL East Timor
479
CU Cuba
453
SL Sierra Leone
443
BN Brunei
409
GU Guam
381
BJ Benin
349
SY Syria
332
SR Suriname
310
SB Solomon Islands
304
TM Turkmenistan
301
LY Libya
291
BI Burundi
267
SM San Marino
266
CK Cook Islands
253
GF French Guiana
253
LA Lao People's Democratic Republic
250
MR Mauritania
247
LR Liberia
245
PF French Polynesia
229
MO Macao
220
SD Sudan
217
BT Bhutan
214
SZ Swaziland
204
YE Yemen
195
LS Lesotho
189
CW Curacao
164
SJ Svalbard and Jan Mayen
164
NE Niger
154
KN Saint Kitts and Nevis
140
DM Dominica
134
GA Gabon
124
MP Northern Mariana Islands
105
NF Norfolk Island
94
HM Heard Island and McDonald Islands
86
SS South Sudan
84
VA Vatican City
63
UM United States Minor Outlying Islands
62
CV Cabo Verde
61
KI Kiribati
61
FK Falkland Islands
57
GN Guinea
55
TD Chad
47
GW Guinea-Bissau
34
PN Pitcairn
33
AQ Antarctica
31
CG Congo
31
BL Saint Barthélemy
29
NR Nauru
22
MH Marshall Islands
21
BQ Bonaire
20
GQ Equatorial Guinea
18
Robots.txt Categories