Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet
Technology Websites %
Some Robots Disallow 39,138,226 18.6
Googlebot Disallow 17,828,157 8.47
Baidu Bot Disallow 13,097,374 6.22
Sogou Bot Disallow 12,199,348 5.8
Google Media Partners Disallow 11,942,797 5.67
Microsoft BingBot Disallow 11,728,319 5.57
Yandex Bot Disallow 11,686,141 5.55
Yahoo Slurp Disallow 10,921,546 5.19
Google AdsBot Disallow 9,790,845 4.65
Robots Disallow 8,533,496 4.05
Google AdsBot Mobile Disallow 7,817,180 3.71
Alexa Bot Disallow 5,217,515 2.48
Soso Bot Disallow 4,790,953 2.28
Ahrefs Bot Disallow 3,952,255 1.88
GPTBot Disallow 3,819,558 1.81
AppleBot Disallow 3,757,288 1.79
Common Crawl Bot Disallow 3,722,656 1.77
Anthropic Claude Bot Disallow 3,591,156 1.71
Google Bard Disallow 3,430,304 1.63
Facebook Bot Disallow 3,384,465 1.61
Apple AI Bot Disallow 3,293,219 1.56
Cohere AI Disallow 3,282,715 1.56
Perplexity 3,229,070 1.53
Majestic Bot Disallow 2,223,915 1.06
Google AdsBot Mobile Apps Disallow 2,080,353 0.99
Semrush Bot Disallow 2,069,960 0.98
ByteDance Bytespider Disallow 673,782 0.32
Exalead Bot Disallow 455,319 0.22
Microsoft MSNBot Disallow 437,707 0.21
Webz Omgili Disallow 392,093 0.19
Google Image Bot Disallow 347,468 0.17
DuckDuckGo Bot Disallow 275,546 0.13
Diffbot Disallow 268,227 0.13
360 Bot Disallow 255,709 0.12
YOU Disallow 164,979 0.08
BuiltWithBot Disallow 115,195 0.05
Google Video Bot Disallow 91,151 0.04
Youdao Bot Disallow 90,415 0.04
Google News Bot Disallow 90,384 0.04
Google Favicon Disallow 72,713 0.03
LLMS Text 49,070 0.02
Google API Bot Disallow 46,654 0.02
Google DuplexWeb Disallow 41,438 0.02
Pipl Disallow 36,337 0.02
Google Read Aloud Disallow 5,596 0
Google FeedFetcher Disallow 4,158 0
Anthropic Claude User Bot Disallow 3,763 0
Anthropic Claude Search Bot Disallow 3,666 0
Google Web Light Disallow 743 0
Google BIG Bench Disallow 41 0

210,450,965 Detections

of Robots.txt on the Entire Internet. Last updated 01st Jul 2025.

Google Media Partners Disallow Google Media Partners Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
949,536
Top 100k
Top 100,000 sites by traffic
174,646
Top 10k
Top 10,000 sites by traffic
25,082
Entire Internet
210,450,965
United States
42,689,478
Germany
8,022,120
United Kingdom
5,205,631
Australia
2,994,080
Russia
2,880,635
Canada
2,224,444
France
1,386,048
Italy
1,273,571
Netherlands
1,177,636
Brazil
1,087,021
Spain
1,085,314
China
1,043,429
Switzerland
1,008,716
.co
992,703
India
869,719
Japan
861,794
CC Cocos Islands
577,094
PL Poland
516,045
BE Belgium
482,496
SE Sweden
480,630
NZ New Zealand
449,532
AT Austria
448,527
ME .me
399,398
IO .io
368,400
NO Norway
329,020
CZ Czech Republic
320,737
MX Mexico
317,559
DK Denmark
301,258
IE Ireland
289,136
ZA South Africa
287,730
KR South Korea
259,737
TR Turkey
235,690
RO Romania
225,489
FI Finland
223,294
AI Anguilla
208,024
ID Indonesia
193,364
PH Philippines
193,134
PT Portugal
175,403
TV .tv
138,952
HU Hungary
135,575
SG Singapore
133,230
GR Greece
109,313
CL Chile
108,911
HK Hong Kong
104,627
AR Argentina
95,807
SK Slovakia
95,568
TH Thailand
95,338
UA Ukraine
93,433
TW Taiwan
89,031
VN Vietnam
83,502
MY Malaysia
81,910
IL Israel
80,591
AE UAE
78,418
VG British Virgin Islands
73,813
LU Luxembourg
67,582
WS Samoa
60,997
SI Slovenia
44,250
PK Pakistan
40,187
LT Lithuania
39,830
EE Estonia
39,563
BG Bulgaria
38,125
IR Iran
37,519
IS Iceland
37,447
SA Saudi Arabia
35,259
LV Latvia
34,979
PE Peru
33,762
CM Cameroon
30,031
IM Isle of Man
29,816
EC Ecuador
28,241
HR Croatia
24,877
CY Cyprus
23,274
GG Guernsey
21,255
CR Costa Rica
20,703
NU Niue
20,400
PW Palau
20,215
VC Saint Vincent and the Grenadines
20,111
NG Nigeria
19,996
SH Saint Helena
19,768
PR Puerto Rico
19,745
RS Serbia
19,612
TO Tonga
19,279
KE Kenya
18,411
RE Reunion
18,067
LI Liechtenstein
17,665
UY Uruguay
16,144
BD Bangladesh
15,929
BY Belarus
14,982
MT Malta
14,281
KZ Kazakhstan
13,194
DO Dominican Republic
12,416
SO Somalia
12,038
MA Morocco
11,941
EG Egypt
11,460
AM Armenia
11,257
GT Guatemala
10,879
BZ Belize
9,932
VE Venezuela
9,627
AL Albania
9,344
LB Lebanon
9,250
PM Saint Pierre and Miquelon
9,241
FM Micronesia
8,879
PA Panama
8,871
LK Sri Lanka
8,610
QA Qatar
8,282
GE Georgia
8,118
UG Uganda
8,011
CX Christmas Island
7,578
GH Ghana
6,935
NP Nepal
6,826
UZ Uzbekistan
6,050
KW Kuwait
5,810
TZ Tanzania
5,602
MD Moldova
5,215
JO Jordan
5,077
MK Macedonia
4,933
ET Ethiopia
4,665
AG Antigua and Barbuda
4,240
HN Honduras
4,104
PY Paraguay
3,958
KH Cambodia
3,872
GP Guadeloupe
3,787
BO Bolivia
3,666
MU Mauritius
3,579
AD Andorra
3,535
BS Bahamas
3,501
SV El Salvador
3,468
BM Bermuda
3,412
TN Tunisia
3,359
BH Bahrain
3,277
ST Sao Tome and Principe
3,236
AZ Azerbaijan
3,231
MC Monaco
3,204
BA Bosnia and Herzegovina
3,186
OM Oman
3,139
KY Cayman Islands
3,121
DZ Algeria
3,095
NA Namibia
3,066
RW Rwanda
2,904
MN Mongolia
2,835
SC Seychelles
2,760
JE Jersey
2,696
ZW Zimbabwe
2,534
AF Afghanistan
2,467
SX Sint Maarten
2,359
MS Montserrat
2,330
TT Trinidad and Tobago
2,301
FO Faroe Islands
2,292
JM Jamaica
2,106
ZM Zambia
2,095
VI United States Virgin Islands
1,922
IQ Iraq
1,878
NI Nicaragua
1,866
LC Saint Lucia
1,857
YT Mayotte
1,796
MQ Martinique
1,758
CI Cote d'Ivoire
1,647
BB Barbados
1,644
MV Maldives
1,615
MZ Mozambique
1,595
GD Grenada
1,551
KG Kyrgyzstan
1,536
MM Myanmar
1,512
AO Angola
1,490
SN Senegal
1,406
NC New Caledonia
1,388
GL Greenland
1,371
BW Botswana
1,287
AS American Samoa
1,281
GY Guyana
1,242
PS Palestine
1,226
FJ Fiji
1,153
GI Gibraltar
1,138
MG Madagascar
1,100
DJ Djibouti
1,092
TC Turks and Caicos Islands
1,085
GS South Georgia and the South Sandwich Islands
1,043
CD Congo
994
AW Aruba
980
AX Åland Islands
921
HT Haiti
844
TJ Tajikistan
681
VU Vanuatu
654
GM Gambia
646
MW Malawi
639
WF Wallis and Futuna
631
PG Papua New Guinea
606
ML Mali
588
TG Togo
585
BF Burkina Faso
517
CU Cuba
475
TL East Timor
469
SL Sierra Leone
452
BN Brunei
413
GU Guam
384
BJ Benin
362
SY Syria
338
SR Suriname
314
TM Turkmenistan
298
LY Libya
291
SB Solomon Islands
286
BI Burundi
275
GF French Guiana
273
SM San Marino
272
LA Lao People's Democratic Republic
264
SD Sudan
259
CK Cook Islands
254
MR Mauritania
249
LR Liberia
245
PF French Polynesia
232
MO Macao
228
BT Bhutan
224
SZ Swaziland
201
LS Lesotho
195
YE Yemen
190
SJ Svalbard and Jan Mayen
167
CW Curacao
165
NE Niger
159
KN Saint Kitts and Nevis
141
GA Gabon
136
DM Dominica
135
MP Northern Mariana Islands
105
NF Norfolk Island
97
HM Heard Island and McDonald Islands
87
SS South Sudan
86
VA Vatican City
63
CV Cabo Verde
63
KI Kiribati
62
UM United States Minor Outlying Islands
62
GN Guinea
59
FK Falkland Islands
57
TD Chad
48
GW Guinea-Bissau
35
PN Pitcairn
33
CG Congo
31
MH Marshall Islands
31
AQ Antarctica
30
BL Saint Barthélemy
29
NR Nauru
23
BQ Bonaire
20
GQ Equatorial Guinea
18
Robots.txt Categories