Robots.txt Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Top In Robots.txt Usage Distribution on the Entire Internet
Technology Websites %
Some Robots Disallow 39,138,226 18.64
Googlebot Disallow 18,777,473 8.94
Baidu Bot Disallow 13,233,594 6.3
Sogou Bot Disallow 12,367,809 5.89
Google Media Partners Disallow 11,590,120 5.52
Microsoft BingBot Disallow 11,275,970 5.37
Yandex Bot Disallow 11,233,892 5.35
Google AdsBot Disallow 10,701,849 5.1
Yahoo Slurp Disallow 10,485,295 4.99
Google AdsBot Mobile Disallow 8,344,481 3.97
Robots Disallow 8,014,950 3.82
Alexa Bot Disallow 5,029,184 2.4
Soso Bot Disallow 4,615,935 2.2
Ahrefs Bot Disallow 3,916,812 1.87
GPTBot Disallow 3,751,266 1.79
AppleBot Disallow 3,698,915 1.76
Common Crawl Bot Disallow 3,681,953 1.75
Anthropic Claude Bot Disallow 3,551,800 1.69
Google Bard Disallow 3,391,288 1.62
Facebook Bot Disallow 3,347,039 1.59
Apple AI Bot Disallow 3,256,425 1.55
Cohere AI Disallow 3,248,417 1.55
Perplexity 3,192,806 1.52
Majestic Bot Disallow 2,182,652 1.04
Google AdsBot Mobile Apps Disallow 2,080,357 0.99
Semrush Bot Disallow 2,026,465 0.97
ByteDance Bytespider Disallow 665,525 0.32
Exalead Bot Disallow 446,285 0.21
Microsoft MSNBot Disallow 428,704 0.2
Webz Omgili Disallow 386,711 0.18
Google Image Bot Disallow 339,010 0.16
DuckDuckGo Bot Disallow 268,468 0.13
Diffbot Disallow 262,940 0.13
360 Bot Disallow 253,257 0.12
YOU Disallow 160,638 0.08
BuiltWithBot Disallow 114,753 0.05
Google Video Bot Disallow 90,091 0.04
Google News Bot Disallow 89,417 0.04
Youdao Bot Disallow 87,418 0.04
Google Favicon Disallow 72,144 0.03
Google API Bot Disallow 46,170 0.02
Google DuplexWeb Disallow 41,014 0.02
Pipl Disallow 33,922 0.02
Google Read Aloud Disallow 5,456 0
LLMS Text 5,454 0
Google FeedFetcher Disallow 4,106 0
Anthropic Claude User Bot Disallow 1,653 0
Anthropic Claude Search Bot Disallow 1,632 0
Google Web Light Disallow 737 0
Google BIG Bench Disallow 39 0

209,940,517 Detections

of Robots.txt on the Entire Internet. Last updated 07th Jun 2025.

Google Media Partners Disallow Google Media Partners Disallow is currently the most popular technology in this category.

Top 1m
Top 1,000,000 sites by traffic
932,198
Top 100k
Top 100,000 sites by traffic
172,896
Top 10k
Top 10,000 sites by traffic
24,867
Entire Internet
209,940,517
United States
42,511,217
Germany
7,874,628
United Kingdom
5,233,029
Australia
3,041,080
Russia
2,876,830
Canada
2,247,463
France
1,399,851
Italy
1,285,784
Netherlands
1,156,875
Brazil
1,113,861
Spain
1,081,562
China
1,064,785
Switzerland
1,006,400
.co
999,900
India
896,944
Japan
888,458
CC Cocos Islands
574,146
PL Poland
523,882
BE Belgium
484,521
SE Sweden
482,875
NZ New Zealand
459,111
AT Austria
445,305
ME .me
398,921
IO .io
357,839
NO Norway
331,579
CZ Czech Republic
321,177
MX Mexico
320,221
DK Denmark
303,562
ZA South Africa
291,659
IE Ireland
291,328
KR South Korea
262,058
TR Turkey
241,668
RO Romania
227,339
FI Finland
219,277
ID Indonesia
195,658
PH Philippines
195,053
AI Anguilla
193,389
PT Portugal
176,327
TV .tv
139,011
SG Singapore
136,890
HU Hungary
135,665
GR Greece
111,562
CL Chile
110,720
HK Hong Kong
106,283
TH Thailand
98,531
AR Argentina
96,269
SK Slovakia
95,850
UA Ukraine
94,499
TW Taiwan
89,285
VN Vietnam
86,797
IL Israel
82,865
MY Malaysia
82,553
AE UAE
80,344
VG British Virgin Islands
73,503
LU Luxembourg
67,889
WS Samoa
60,377
SI Slovenia
44,026
PK Pakistan
42,065
LT Lithuania
40,318
EE Estonia
39,856
BG Bulgaria
38,398
IR Iran
37,954
IS Iceland
37,551
SA Saudi Arabia
35,951
LV Latvia
35,408
PE Peru
33,914
CM Cameroon
30,563
IM Isle of Man
30,439
EC Ecuador
28,006
HR Croatia
24,698
CY Cyprus
23,882
PW Palau
20,960
CR Costa Rica
20,948
NU Niue
20,703
GG Guernsey
20,597
PR Puerto Rico
20,014
NG Nigeria
19,801
RS Serbia
19,713
VC Saint Vincent and the Grenadines
19,560
SH Saint Helena
19,375
TO Tonga
19,057
KE Kenya
18,910
RE Reunion
18,034
LI Liechtenstein
17,805
UY Uruguay
16,323
BD Bangladesh
16,138
BY Belarus
15,198
MT Malta
14,437
KZ Kazakhstan
13,390
DO Dominican Republic
12,468
MA Morocco
12,361
SO Somalia
11,838
EG Egypt
11,559
AM Armenia
11,184
GT Guatemala
10,964
BZ Belize
10,128
PM Saint Pierre and Miquelon
9,423
VE Venezuela
9,361
AL Albania
9,339
LB Lebanon
9,309
PA Panama
8,913
FM Micronesia
8,893
LK Sri Lanka
8,624
QA Qatar
8,274
GE Georgia
8,265
UG Uganda
8,259
CX Christmas Island
7,395
GH Ghana
7,077
NP Nepal
6,624
UZ Uzbekistan
6,044
KW Kuwait
5,876
TZ Tanzania
5,785
MD Moldova
5,177
JO Jordan
5,155
MK Macedonia
4,987
ET Ethiopia
4,870
AG Antigua and Barbuda
4,331
HN Honduras
4,223
PY Paraguay
4,072
GP Guadeloupe
3,845
KH Cambodia
3,834
BO Bolivia
3,594
BS Bahamas
3,560
SV El Salvador
3,484
AD Andorra
3,383
TN Tunisia
3,359
ST Sao Tome and Principe
3,342
BH Bahrain
3,319
AZ Azerbaijan
3,235
MC Monaco
3,222
OM Oman
3,216
KY Cayman Islands
3,168
BA Bosnia and Herzegovina
3,127
MU Mauritius
3,107
DZ Algeria
3,098
NA Namibia
3,071
BM Bermuda
3,027
MN Mongolia
2,927
RW Rwanda
2,896
SC Seychelles
2,809
JE Jersey
2,708
ZW Zimbabwe
2,575
AF Afghanistan
2,402
FO Faroe Islands
2,390
TT Trinidad and Tobago
2,330
MS Montserrat
2,267
JM Jamaica
2,163
SX Sint Maarten
2,137
ZM Zambia
2,127
VI United States Virgin Islands
1,915
LC Saint Lucia
1,863
NI Nicaragua
1,856
IQ Iraq
1,855
YT Mayotte
1,828
MQ Martinique
1,816
CI Cote d'Ivoire
1,663
BB Barbados
1,656
MV Maldives
1,625
MZ Mozambique
1,612
GD Grenada
1,609
KG Kyrgyzstan
1,519
MM Myanmar
1,493
AO Angola
1,475
NC New Caledonia
1,418
GL Greenland
1,378
GY Guyana
1,366
SN Senegal
1,361
BW Botswana
1,324
AS American Samoa
1,313
PS Palestine
1,202
TC Turks and Caicos Islands
1,160
GI Gibraltar
1,152
FJ Fiji
1,137
MG Madagascar
1,095
DJ Djibouti
1,044
GS South Georgia and the South Sandwich Islands
1,009
CD Congo
999
AW Aruba
985
AX Åland Islands
922
HT Haiti
842
TJ Tajikistan
677
WF Wallis and Futuna
670
VU Vanuatu
660
GM Gambia
647
MW Malawi
623
PG Papua New Guinea
600
TG Togo
583
ML Mali
554
BF Burkina Faso
508
TL East Timor
479
CU Cuba
453
SL Sierra Leone
443
BN Brunei
409
GU Guam
381
BJ Benin
349
SY Syria
332
SR Suriname
310
SB Solomon Islands
304
TM Turkmenistan
301
LY Libya
291
BI Burundi
267
SM San Marino
266
CK Cook Islands
253
GF French Guiana
253
LA Lao People's Democratic Republic
250
MR Mauritania
247
LR Liberia
245
PF French Polynesia
229
MO Macao
220
SD Sudan
217
BT Bhutan
214
SZ Swaziland
204
YE Yemen
195
LS Lesotho
189
CW Curacao
164
SJ Svalbard and Jan Mayen
164
NE Niger
154
KN Saint Kitts and Nevis
140
DM Dominica
134
GA Gabon
124
MP Northern Mariana Islands
105
NF Norfolk Island
94
HM Heard Island and McDonald Islands
86
SS South Sudan
84
VA Vatican City
63
UM United States Minor Outlying Islands
62
CV Cabo Verde
61
KI Kiribati
61
FK Falkland Islands
57
GN Guinea
55
TD Chad
47
GW Guinea-Bissau
34
PN Pitcairn
33
AQ Antarctica
31
CG Congo
31
BL Saint Barthélemy
29
NR Nauru
22
MH Marshall Islands
21
BQ Bonaire
20
GQ Equatorial Guinea
18
Robots.txt Categories