Log In · Signup for Free
EN  DE  FR  ES  IT  Pусский  한국어  日本語  简体中文  عربي

Robots Disallow Usage Statistics

R Robots Disallow

A robots disallow all directive with no other options.

https://robotstxt.org

Robots.txt

Robots Disallow · Customers

Get access to data on 19,155,474 websites that are Robots Disallow Customers. We know of 8,326,302 live websites using Robots Disallow and an additional 10,829,172 sites that used Robots Disallow historically and 3,326,218 websites in the United States

Download Lead List
Pie Excluded Technology

This is an excluded technology from charts.

General robots disallow rule.

Competitors and Similar Technologies
Microsoft MSNBot Disallow favicon
Microsoft MSNBot Disallow

333,596 live sites

DuckDuckGo Bot Disallow favicon
DuckDuckGo Bot Disallow

278,430 live sites

Exalead Bot Disallow favicon
Exalead Bot Disallow

335,791 live sites

BuiltWithBot Disallow favicon
BuiltWithBot Disallow

44,424 live sites

Sogou Bot Disallow favicon
Sogou Bot Disallow

12,003,031 live sites

Soso Bot Disallow favicon
Soso Bot Disallow

4,299,023 live sites

360 Bot Disallow favicon
360 Bot Disallow

121,377 live sites

Youdao Bot Disallow favicon
Youdao Bot Disallow

141,795 live sites

Google API Bot Disallow favicon
Google API Bot Disallow

25,985 live sites

Google Media Partners Disallow favicon
Google Media Partners Disallow

11,796,610 live sites

Google AdsBot Mobile Disallow favicon
Google AdsBot Mobile Disallow

8,297,265 live sites

Google AdsBot Disallow favicon
Google AdsBot Disallow

8,906,110 live sites

Google Image Bot Disallow favicon
Google Image Bot Disallow

290,384 live sites

Google News Bot Disallow favicon
Google News Bot Disallow

65,589 live sites

Google Video Bot Disallow favicon
Google Video Bot Disallow

67,367 live sites

Google AdsBot Mobile Apps Disallow favicon
Google AdsBot Mobile Apps Disallow

539,515 live sites

Google FeedFetcher Disallow favicon
Google FeedFetcher Disallow

3,434 live sites

Google Read Aloud Disallow favicon
Google Read Aloud Disallow

5,323 live sites

Google DuplexWeb Disallow favicon
Google DuplexWeb Disallow

21,224 live sites

Google Favicon Disallow favicon
Google Favicon Disallow

50,817 live sites

Google Web Light Disallow favicon
Google Web Light Disallow

559 live sites

Majestic Bot Disallow favicon
Majestic Bot Disallow

1,545,212 live sites

Semrush Bot Disallow favicon
Semrush Bot Disallow

1,518,445 live sites

Ahrefs Bot Disallow favicon
Ahrefs Bot Disallow

1,856,245 live sites

AppleBot Disallow favicon
AppleBot Disallow

4,439,451 live sites

Google BIG Bench Disallow favicon
Google BIG Bench Disallow

38 live sites

GPTBot Disallow favicon
GPTBot Disallow

4,430,048 live sites

Common Crawl Bot Disallow favicon
Common Crawl Bot Disallow

4,324,319 live sites

Anthropic Claude Bot Disallow favicon
Anthropic Claude Bot Disallow

4,238,680 live sites

Cohere AI Disallow favicon
Cohere AI Disallow

3,366,401 live sites

Download Lead List

Get a list of 8,542,848 websites using Robots Disallow which includes location information, hosting data and contact details. The list includes 8,326,302 live websites and 216,546 websites redirecting to those sites. 3,326,218 of these sites are in the United States.

We also know of 10,829,172 sites that have used Robots Disallow previously.

Site Totals
Total Live
8,326,302
216,546 additional website redirects?.
Expired
10,829,172
🇺🇸 American Live Sites
3,326,218
Live and Expired
19,155,474
Expired
10,829,172
Traffic
Top 1m
91,488
Top 100k
17,241
Top 10k
2,194
Top 1000
243
Outside Top 1m
8,211,541
Technology Spend
$100+
619,433
$500+
281,491
$1000+
190,340
$5k+
44,675
$10k+
21,613
Social Followers
10+
150,498
100+
130,307
500+
96,477
1000+
77,545
5000+
37,781
10k+
26,505
100k+
5,925
1m+
784
Sales Revenue
$100
213,097
$500
197,665
$1000
190,153
$5000
159,367
$10k
88,913
$100k
18,136
$1m
4,919
$10m
2,552
$100m
1,293
SKU Products
10+
102,073
50+
38,883
500+
13,646
100+
8,506
5000+
2,516
10k+
1,134
100k+
65
Live Country Totals
🇺🇸 United States
3,326,218
🇷🇺 Russia
907,663
🇩🇪 Germany
269,752
🇬🇧 United Kingdom
124,524
🇮🇳 India
115,903
🇧🇷 Brazil
86,261
🇫🇷 France
76,025
🇳🇱 Netherlands
69,270
🇨🇦 Canada
57,897
🇨🇳 China
48,205
🇯🇵 Japan
46,010
🇨🇭 Switzerland
42,782
🇦🇺 Australia
40,658
🇪🇺 European Union
37,108
🇮🇹 Italy
34,947
🇪🇸 Spain
32,522
🇮🇩 Indonesia
29,985
🇨🇨 Cocos Islands
28,839
🇨🇴 .co
27,175
🇳🇴 Norway
26,793
🇮🇴 .io
25,378
🇭🇺 Hungary
23,510
🇲🇪 .me
22,743
🇵🇱 Poland
21,232
🇿🇦 South Africa
19,396
🇺🇦 Ukraine
18,296
🇨🇿 Czech Republic
17,310
🇰🇷 South Korea
16,630
🇦🇹 Austria
16,028
🇧🇪 Belgium
15,653
🇲🇽 Mexico
14,333
🇵🇭 Philippines
13,488
🇸🇪 Sweden
13,146
🇹🇷 Turkey
12,462
🇫🇮 Finland
10,439
🇨🇱 Chile
10,021
🇼🇸 Samoa
8,579
🇩🇰 Denmark
8,416
🇱🇹 Lithuania
8,259
🇦🇮 Anguilla
7,850
🇦🇷 Argentina
7,543
🇵🇹 Portugal
7,507
🇻🇳 Vietnam
7,385
🇷🇴 Romania
6,835
🇦🇪 UAE
6,517
🇲🇾 Malaysia
6,212
🇬🇷 Greece
5,723
🇹🇼 Taiwan
5,592
🇳🇿 New Zealand
5,490
🇹🇻 .tv
4,933
🇸🇰 Slovakia
4,566
🇸🇬 Singapore
4,363
🇵🇰 Pakistan
3,796
🇹🇭 Thailand
3,346
🇭🇰 Hong Kong
3,330
🇧🇬 Bulgaria
3,047
🇮🇷 Iran
3,038
🇵🇼 Palau
2,854
🇵🇪 Peru
2,837
🇮🇪 Ireland
2,727
🇮🇱 Israel
2,552
🇧🇾 Belarus
2,541
🇪🇪 Estonia
2,435
🇰🇿 Kazakhstan
2,412
🇸🇮 Slovenia
2,305
🇧🇩 Bangladesh
2,197
🇷🇸 Serbia
2,194
🇱🇻 Latvia
2,166
🇳🇬 Nigeria
2,145
🇭🇷 Croatia
1,966
🇸🇦 Saudi Arabia
1,923
🇰🇪 Kenya
1,784
🇱🇺 Luxembourg
1,762
🇺🇿 Uzbekistan
1,700
🇳🇵 Nepal
1,637
🇳🇺 Niue
1,416
🇲🇦 Morocco
1,271
🇪🇨 Ecuador
1,249
🇨🇾 Cyprus
1,241
🇱🇰 Sri Lanka
1,239
🇺🇾 Uruguay
1,200
🇪🇬 Egypt
1,181
🇱🇮 Liechtenstein
1,118
🇸🇭 Saint Helena
999
🇹🇴 Tonga
953
🇨🇲 Cameroon
888
🇮🇸 Iceland
863
🇷🇪 Reunion
837
🇦🇲 Armenia
793
🇬🇬 Guernsey
736
🇨🇷 Costa Rica
712
🇬🇪 Georgia
701
🇩🇴 Dominican Republic
665
🇹🇿 Tanzania
649
🇦🇿 Azerbaijan
643
🇻🇪 Venezuela
591
🇶🇦 Qatar
591
🇦🇱 Albania
585
🇧🇿 Belize
574
🇮🇲 Isle of Man
564
🇬🇹 Guatemala
551
🇲🇹 Malta
551
🇬🇭 Ghana
545
🇿🇼 Zimbabwe
542
🇲🇰 Macedonia
520
🇲🇩 Moldova
514
🇺🇬 Uganda
510
🇵🇾 Paraguay
507
🇦🇸 American Samoa
484
🇸🇴 Somalia
463
🇱🇧 Lebanon
457
🇱🇦 Lao People's Democratic Republic
446
🇧🇴 Bolivia
424
🇵🇦 Panama
418
🇹🇳 Tunisia
402
🇩🇿 Algeria
398
🇨🇽 Christmas Island
386
🇰🇼 Kuwait
380
🇻🇨 Saint Vincent and the Grenadines
358
🇴🇲 Oman
339
🇸🇹 Sao Tome and Principe
330
🇵🇷 Puerto Rico
318
🇧🇦 Bosnia and Herzegovina
305
🇯🇴 Jordan
303
🇲🇳 Mongolia
301
🇮🇶 Iraq
296
🇲🇺 Mauritius
294
🇫🇲 Micronesia
285
🇰🇬 Kyrgyzstan
270
🇰🇭 Cambodia
257
🇭🇳 Honduras
243
🇵🇲 Saint Pierre and Miquelon
241
🇧🇭 Bahrain
240
🇱🇾 Libya
235
🇨🇮 Cote d'Ivoire
232
🇿🇲 Zambia
226
🇦🇴 Angola
226
🇪🇹 Ethiopia
223
🇸🇳 Senegal
212
🇨🇩 Congo
212
🇸🇻 El Salvador
208
🇲🇲 Myanmar
202
🇨🇻 Cabo Verde
201
🇲🇻 Maldives
201
🇧🇼 Botswana
197
🇸🇽 Sint Maarten
195
🇦🇬 Antigua and Barbuda
186
🇲🇬 Madagascar
183
🇲🇿 Mozambique
180
🇹🇯 Tajikistan
166
🇦🇫 Afghanistan
161
🇲🇨 Monaco
155
🇳🇦 Namibia
153
🇷🇼 Rwanda
149
🇸🇨 Seychelles
144
🇹🇹 Trinidad and Tobago
135
🇲🇸 Montserrat
132
🇹🇫 French Southern Territories
131
🇳🇮 Nicaragua
127
🇯🇲 Jamaica
123
🇵🇸 Palestine
123
🇧🇧 Barbados
121
🇳🇨 New Caledonia
119
🇾🇹 Mayotte
119
🇸🇧 Solomon Islands
118
🇦🇩 Andorra
117
🇹🇬 Togo
113
🇧🇸 Bahamas
112
🇬🇱 Greenland
107
🇰🇾 Cayman Islands
103
🇸🇷 Suriname
96
🇧🇫 Burkina Faso
95
🇬🇸 South Georgia and the South Sandwich Islands
95
🇫🇴 Faroe Islands
94
🇬🇾 Guyana
91
🇧🇯 Benin
89
🇸🇩 Sudan
88
🇬🇵 Guadeloupe
86
🇲🇼 Malawi
84
🇸🇱 Sierra Leone
84
🇻🇬 British Virgin Islands
83
🇫🇯 Fiji
80
🇲🇱 Mali
79
🇧🇲 Bermuda
78
🇦🇽 Åland Islands
77
🇯🇪 Jersey
75
🇵🇬 Papua New Guinea
75
🇱🇨 Saint Lucia
74
🇩🇯 Djibouti
70
🇸🇾 Syria
68
🇹🇨 Turks and Caicos Islands
67
🇼🇫 Wallis and Futuna
63
🇾🇪 Yemen
60
🇻🇺 Vanuatu
60
🇻🇮 United States Virgin Islands
58
🇧🇮 Burundi
54
🇲🇴 Macao
54
🇭🇹 Haiti
54
🇲🇶 Martinique
53
🇱🇷 Liberia
46
🇬🇦 Gabon
45
🇬🇩 Grenada
45
🇧🇹 Bhutan
43
🇬🇮 Gibraltar
42
🇵🇫 French Polynesia
41
🇬🇲 Gambia
38
🇨🇺 Cuba
38
🇺🇲 United States Minor Outlying Islands
37
🇳🇪 Niger
37
🇸🇲 San Marino
36
🇸🇸 South Sudan
36
🇦🇼 Aruba
35
🇹🇱 East Timor
33
🇹🇲 Turkmenistan
32
🇹🇩 Chad
32
🇱🇸 Lesotho
30
🇲🇷 Mauritania
29
🇧🇳 Brunei
29
🇬🇫 French Guiana
29
🇲🇵 Northern Mariana Islands
25
🇩🇲 Dominica
24
🇬🇳 Guinea
24
🇰🇳 Saint Kitts and Nevis
18
🇭🇲 Heard Island and McDonald Islands
16
🇸🇿 Swaziland
15
🇨🇼 Curacao
14
🇨🇬 Congo
13
🇳🇫 Norfolk Island
12
🇲🇭 Marshall Islands
10
🇨🇰 Cook Islands
10
🇰🇮 Kiribati
9
🇬🇺 Guam
7
🇳🇷 Nauru
5
🇰🇲 Comoros
4
🇬🇼 Guinea-Bissau
4
🇨🇫 Central African Republic
4
🇫🇰 Falkland Islands
2
🇵🇳 Pitcairn
2
🇻🇦 Vatican City
2
🇬🇶 Equatorial Guinea
2
🇸🇯 Svalbard and Jan Mayen
2
🇪🇭 Western Sahara
1
🇹🇰 .tk
1
🇦🇶 Antarctica
1
Live Region Totals
North America
3,384,115
Europe
1,828,102
Western Europe
619,913
Central Europe
398,603
South America
138,548
South Asia
125,177
East Asia
120,122
Southern Europe
112,063
B Benelux Union
86,685
CHAPS
82,646
Northern Europe
75,244
N Nordics
59,937
Balkan States
58,685
Scandinavia
48,355
Africa
36,773
Western Asia
33,908
Sub-Saharan Africa
32,699
Middle East
31,771
Southern Africa
19,791
Southern Cone
19,271
Levant Narrow
17,206
Baltic States
12,860
East Africa
5,683
Central Asia
4,580
Western Africa
3,973
Northern Africa
3,604
Central America
2,833
Central Africa
1,955
Live Financial Totals
AMER
3,525,078
AUKUS
3,491,400
EMEA
1,881,762
B BRICS
1,177,428
BRIC
1,158,032
MART
975,026
APAC
431,974
ASEAN+6
338,609
ASEAN+5
290,404
BASIC
269,765
CANZUK
228,569
IMEA
183,266
ASEAN+3
176,558
ANZUK
170,672
LATCAR
165,755
LATAM
155,643
FRITES
143,494
M MIKTA
114,068
FRIT
110,972
CIVETS
97,584
PIGS
80,699
VISTA
76,771
ASIAN TIGERS
72,528
MEA
67,363
SEA
65,746
Tiger Cub Economies
60,416
MINT
58,925
ANZ
46,148
MEP
35,567
MENA
34,194
CLMV
8,290
Live Group Totals
G G20 Nations
5,236,335
Club de Paris
5,226,913
Asia-Pacific Economic Cooperation
4,549,767
O Organisation for Economic Co-operation and Development
4,400,820
D Development Assistance Committee
4,320,705
N North Atlantic Treaty Organization
4,187,813
I International Solar Alliance
4,051,903
G7 Nations
3,935,373
O Organization of American States
3,541,191
Sovereign states and dependent territories of North America
3,412,459
G2 Nations
3,374,423
C Council of Europe
1,823,147
A Assembly of European Regions
1,482,073
S Shanghai Cooperation Organisation
1,080,115
I Interparliamentary Assembly on Orthodoxy
1,041,695
B Organization of the Black Sea Economic Cooperation
957,262
C Commonwealth of Independent States Members
916,702
E Eurasian Economic Union
913,679
E European Union, European Economic Area and Switzerland
730,896
E European Economic Area
688,114
E Eastern Partnership
682,828
E Euronest Parliamentary Assembly
679,644
E European Union
659,340
EC Inner Six
467,409
G Group of 77
442,246
C Commonwealth of Nations
408,485
Majority German-speaking states of Central Europe
328,562
EC Outer Seven
239,196
I IBSA Dialogue Forum
221,560
O Organization of Ibero-American States
195,569
3 Global Governance Group
142,688
S South Asian Association for Regional Cooperation
125,177
B Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
124,567
M Southern Common Market
95,511
C Community of Portuguese Language Countries
94,744
O Organisation of Islamic Cooperation
84,253
E European Free Trade Association
71,556
V Visegrad Group
66,618
A Association of Southeast Asian Nations
65,713
A African Union
61,636
P Pacific Alliance
54,366
F Federation of Euro-Asian Stock Exchanges
48,752
A Association of Caribbean State Members
47,072
C Central European Free Trade Agreement Members
26,861
E Economic Cooperation Organization
24,680
S Southern African Development Community
22,535
G GUAM Organization for Democracy and Economic Development
20,154
T Organization of Turkic States
17,487
O Organization of the Petroleum Exporting Countries
15,809
A Arab League
15,370
G Gulf Cooperation Council
9,990
O Organisation of Eastern Caribbean States
8,909
C Central American Integration System
3,498
E East African Community
3,394
C Caribbean Community Members
2,143
Caribbean Free Trade Association
1,881
Union of South American Nations
1,202