Log In · Signup for Free
EN  DE  FR  ES  IT  Pусский  한국어  日本語  简体中文  عربي

Some Robots Disallow Usage Statistics

S Some Robots Disallow

Blocks some but not all robots from indexing the website.

https://robotstxt.org

Robots.txt

Some Robots Disallow · Customers

Get access to data on 63,578,813 websites that are Some Robots Disallow Customers. We know of 10,759,718 live websites using Some Robots Disallow and an additional 52,819,095 sites that used Some Robots Disallow historically and 6,582,545 websites in the United States

Download Lead List
Pie Excluded Technology

This is an excluded technology from charts.

General robots disallow rule.

Competitors and Similar Technologies
Googlebot Disallow favicon
Googlebot Disallow

18,145,897 live sites

Microsoft BingBot Disallow favicon
Microsoft BingBot Disallow

12,568,342 live sites

Yahoo Slurp Disallow favicon
Yahoo Slurp Disallow

11,856,471 live sites

Microsoft MSNBot Disallow favicon
Microsoft MSNBot Disallow

333,596 live sites

DuckDuckGo Bot Disallow favicon
DuckDuckGo Bot Disallow

278,430 live sites

Baidu Bot Disallow favicon
Baidu Bot Disallow

12,739,715 live sites

Yandex Bot Disallow favicon
Yandex Bot Disallow

12,495,810 live sites

Exalead Bot Disallow favicon
Exalead Bot Disallow

335,791 live sites

Alexa Bot Disallow favicon
Alexa Bot Disallow

4,719,358 live sites

BuiltWithBot Disallow favicon
BuiltWithBot Disallow

44,424 live sites

Sogou Bot Disallow favicon
Sogou Bot Disallow

12,003,031 live sites

Soso Bot Disallow favicon
Soso Bot Disallow

4,299,023 live sites

360 Bot Disallow favicon
360 Bot Disallow

121,377 live sites

Youdao Bot Disallow favicon
Youdao Bot Disallow

141,795 live sites

Google API Bot Disallow favicon
Google API Bot Disallow

25,985 live sites

Google Media Partners Disallow favicon
Google Media Partners Disallow

11,796,610 live sites

Google AdsBot Mobile Disallow favicon
Google AdsBot Mobile Disallow

8,297,265 live sites

Google AdsBot Disallow favicon
Google AdsBot Disallow

8,906,110 live sites

Google Image Bot Disallow favicon
Google Image Bot Disallow

290,384 live sites

Google News Bot Disallow favicon
Google News Bot Disallow

65,589 live sites

Google Video Bot Disallow favicon
Google Video Bot Disallow

67,367 live sites

Google AdsBot Mobile Apps Disallow favicon
Google AdsBot Mobile Apps Disallow

539,515 live sites

Google FeedFetcher Disallow favicon
Google FeedFetcher Disallow

3,434 live sites

Google Read Aloud Disallow favicon
Google Read Aloud Disallow

5,323 live sites

Google DuplexWeb Disallow favicon
Google DuplexWeb Disallow

21,224 live sites

Google Favicon Disallow favicon
Google Favicon Disallow

50,817 live sites

Google Web Light Disallow favicon
Google Web Light Disallow

559 live sites

Majestic Bot Disallow favicon
Majestic Bot Disallow

1,545,212 live sites

Semrush Bot Disallow favicon
Semrush Bot Disallow

1,518,445 live sites

Ahrefs Bot Disallow favicon
Ahrefs Bot Disallow

1,856,245 live sites

Download Lead List

Get a list of 10,827,923 websites using Some Robots Disallow which includes location information, hosting data and contact details. The list includes 10,759,718 live websites and 68,205 websites redirecting to those sites. 6,582,545 of these sites are in the United States.

We also know of 52,819,095 sites that have used Some Robots Disallow previously.

Site Totals
Total Live
10,759,718
68,205 additional website redirects?.
Expired
52,819,095
🇺🇸 American Live Sites
6,582,545
Live and Expired
63,578,813
Expired
52,819,095
Traffic
Top 1m
26,156
Top 100k
5,574
Top 10k
942
Top 1000
116
Outside Top 1m
10,721,522
Technology Spend
$100+
222,356
$500+
69,293
$1000+
43,109
$5k+
11,927
$10k+
6,666
Social Followers
10+
64,819
100+
56,117
500+
41,102
1000+
32,425
5000+
14,585
10k+
10,091
100k+
2,200
1m+
318
Sales Revenue
$100
80,496
$500
77,693
$1000
67,612
$5000
59,227
$10k
34,141
$100k
7,839
$1m
1,930
$10m
983
$100m
520
SKU Products
10+
49,943
50+
26,759
500+
10,867
100+
5,979
5000+
1,715
10k+
783
100k+
48
Live Country Totals
🇺🇸 United States
6,582,545
🇩🇪 Germany
370,779
🇬🇧 United Kingdom
208,092
🇷🇺 Russia
102,036
🇵🇭 Philippines
91,640
🇨🇴 .co
80,133
🇨🇳 China
60,775
🇦🇺 Australia
58,617
🇨🇦 Canada
50,969
🇳🇱 Netherlands
49,323
🇮🇳 India
46,013
🇮🇴 .io
44,115
🇪🇸 Spain
41,068
🇦🇮 .ai
38,179
🇫🇷 France
35,756
🇨🇨 Cocos Islands
34,627
🇲🇪 .me
29,367
🇪🇺 European Union
27,030
🇧🇷 Brazil
26,408
🇮🇹 Italy
23,318
🇰🇷 South Korea
19,101
🇨🇭 Switzerland
17,038
🇦🇹 Austria
16,691
🇯🇵 Japan
13,721
🇮🇩 Indonesia
12,202
🇵🇱 Poland
12,174
🇹🇴 Tonga
12,009
🇹🇻 .tv
11,651
🇿🇦 South Africa
11,387
🇧🇪 Belgium
11,337
🇻🇬 British Virgin Islands
9,260
🇲🇽 Mexico
7,499
🇸🇪 Sweden
6,921
🇹🇭 Thailand
5,467
🇮🇷 Iran
4,994
🇼🇸 Samoa
4,981
🇷🇴 Romania
4,520
🇵🇹 Portugal
4,380
🇩🇰 Denmark
4,178
🇨🇿 Czech Republic
3,936
🇲🇾 Malaysia
3,804
🇫🇮 Finland
3,753
🇸🇰 Slovakia
3,321
🇹🇷 Turkey
3,282
🇹🇼 Taiwan
3,049
🇻🇳 Vietnam
2,969
🇦🇷 Argentina
2,905
🇸🇭 Saint Helena
2,703
🇬🇬 Guernsey
2,679
🇬🇷 Greece
2,598
🇳🇴 Norway
2,575
🇳🇿 New Zealand
2,373
🇻🇨 Saint Vincent and the Grenadines
2,302
🇮🇲 Isle of Man
2,151
🇨🇱 Chile
2,149
🇵🇼 Palau
1,965
🇸🇮 Slovenia
1,906
🇭🇺 Hungary
1,887
🇪🇪 Estonia
1,875
🇺🇦 Ukraine
1,818
🇭🇰 Hong Kong
1,752
🇸🇴 Somalia
1,674
🇧🇿 Belize
1,651
🇵🇰 Pakistan
1,649
🇸🇬 Singapore
1,525
🇦🇪 UAE
1,509
🇷🇪 Reunion
1,486
🇮🇸 Iceland
1,460
🇮🇪 Ireland
1,171
🇨🇽 Christmas Island
1,109
🇱🇦 Lao People's Democratic Republic
1,091
🇦🇲 Armenia
1,024
🇱🇻 Latvia
966
🇱🇮 Liechtenstein
918
🇧🇩 Bangladesh
879
🇵🇪 Peru
845
🇫🇲 Micronesia
810
🇳🇬 Nigeria
791
🇷🇸 Serbia
769
🇱🇺 Luxembourg
653
🇧🇬 Bulgaria
650
🇮🇱 Israel
630
🇱🇹 Lithuania
620
🇳🇺 Niue
615
🇰🇪 Kenya
574
🇦🇩 Andorra
533
🇦🇱 Albania
441
🇸🇦 Saudi Arabia
433
🇪🇬 Egypt
426
🇬🇪 Georgia
417
🇨🇲 Cameroon
400
🇳🇵 Nepal
392
🇧🇾 Belarus
392
🇪🇨 Ecuador
382
🇭🇷 Croatia
356
🇨🇾 Cyprus
349
🇻🇪 Venezuela
328
🇲🇸 Montserrat
323
🇶🇦 Qatar
306
🇱🇰 Sri Lanka
302
🇲🇦 Morocco
299
🇬🇭 Ghana
286
🇲🇹 Malta
283
🇰🇿 Kazakhstan
275
🇨🇷 Costa Rica
261
🇦🇬 Antigua and Barbuda
247
🇾🇹 Mayotte
238
🇺🇾 Uruguay
223
🇲🇩 Moldova
211
🇱🇧 Lebanon
206
🇺🇬 Uganda
203
🇨🇻 Cabo Verde
203
🇩🇴 Dominican Republic
192
🇲🇳 Mongolia
176
🇵🇦 Panama
163
🇵🇷 Puerto Rico
163
🇬🇸 South Georgia and the South Sandwich Islands
158
🇸🇹 Sao Tome and Principe
156
🇷🇼 Rwanda
155
🇯🇴 Jordan
154
🇹🇨 Turks and Caicos Islands
152
🇺🇿 Uzbekistan
150
🇬🇹 Guatemala
140
🇹🇿 Tanzania
132
🇩🇯 Djibouti
127
🇸🇨 Seychelles
127
🇰🇼 Kuwait
118
🇵🇲 Saint Pierre and Miquelon
111
🇱🇾 Libya
109
🇭🇳 Honduras
106
🇰🇭 Cambodia
104
🇧🇴 Bolivia
104
🇬🇾 Guyana
92
🇲🇰 Macedonia
92
🇦🇿 Azerbaijan
87
🇹🇯 Tajikistan
85
🇩🇿 Algeria
83
🇱🇨 Saint Lucia
83
🇧🇭 Bahrain
82
🇴🇲 Oman
80
🇬🇩 Grenada
79
🇹🇳 Tunisia
77
🇮🇶 Iraq
77
🇸🇻 El Salvador
73
🇪🇹 Ethiopia
70
🇦🇫 Afghanistan
68
🇬🇵 Guadeloupe
66
🇨🇮 Cote d'Ivoire
64
🇸🇽 Sint Maarten
64
🇬🇱 Greenland
62
🇳🇮 Nicaragua
61
🇹🇫 French Southern Territories
61
🇵🇸 Palestine
57
🇧🇦 Bosnia and Herzegovina
57
🇨🇩 Congo
53
🇿🇼 Zimbabwe
52
🇦🇸 American Samoa
50
🇲🇲 Myanmar
50
🇳🇦 Namibia
50
🇲🇺 Mauritius
50
🇳🇨 New Caledonia
50
🇿🇲 Zambia
49
🇵🇾 Paraguay
49
🇲🇶 Martinique
46
🇸🇳 Senegal
42
🇯🇲 Jamaica
41
🇲🇨 Monaco
41
🇧🇸 Bahamas
38
🇲🇻 Maldives
38
🇹🇬 Togo
35
🇹🇹 Trinidad and Tobago
34
🇫🇴 Faroe Islands
33
🇰🇬 Kyrgyzstan
32
🇧🇹 Bhutan
32
🇾🇪 Yemen
32
🇲🇱 Mali
31
🇯🇪 Jersey
28
🇦🇴 Angola
28
🇲🇬 Madagascar
27
🇻🇮 United States Virgin Islands
27
🇬🇫 French Guiana
27
🇧🇯 Benin
26
🇼🇫 Wallis and Futuna
25
🇸🇩 Sudan
25
🇹🇱 East Timor
24
🇲🇴 Macao
24
🇧🇼 Botswana
23
🇸🇷 Suriname
23
🇧🇫 Burkina Faso
22
🇬🇮 Gibraltar
21
🇦🇼 Aruba
21
🇧🇧 Barbados
21
🇭🇲 Heard Island and McDonald Islands
20
🇲🇼 Malawi
19
🇧🇲 Bermuda
18
🇭🇹 Haiti
18
🇫🇯 Fiji
18
🇸🇾 Syria
18
🇲🇷 Mauritania
16
🇸🇱 Sierra Leone
15
🇰🇾 Cayman Islands
15
🇱🇷 Liberia
14
🇲🇿 Mozambique
13
🇧🇮 Burundi
12
🇦🇽 Åland Islands
12
🇳🇪 Niger
12
🇨🇺 Cuba
12
🇬🇺 Guam
11
🇹🇲 Turkmenistan
10
🇩🇲 Dominica
9
🇻🇺 Vanuatu
9
🇺🇲 United States Minor Outlying Islands
7
🇨🇰 Cook Islands
7
🇵🇬 Papua New Guinea
6
🇬🇲 Gambia
6
🇰🇳 Saint Kitts and Nevis
6
🇸🇧 Solomon Islands
6
🇹🇩 Chad
6
🇸🇸 South Sudan
5
🇸🇲 San Marino
5
🇬🇦 Gabon
5
🇳🇫 Norfolk Island
5
🇵🇫 French Polynesia
5
🇲🇵 Northern Mariana Islands
5
🇱🇸 Lesotho
4
🇸🇿 Swaziland
4
🇧🇳 Brunei
3
🇬🇼 Guinea-Bissau
2
🇨🇼 Curacao
2
🇻🇦 Vatican City
2
🇬🇳 Guinea
2
🇲🇭 Marshall Islands
2
🇨🇬 Congo
1
🇰🇮 Kiribati
1
🇵🇳 Pitcairn
1
🇬🇶 Equatorial Guinea
1
🇦🇶 Antarctica
1
Live Region Totals
North America
6,633,514
Europe
975,701
Western Europe
712,332
Central Europe
428,650
South America
113,838
Southern Europe
105,194
East Asia
98,598
B Benelux Union
61,313
South Asia
49,373
Balkan States
44,038
CHAPS
38,009
Northern Europe
23,519
Africa
22,481
Sub-Saharan Africa
19,783
N Nordics
18,994
Western Asia
14,281
Scandinavia
13,674
Middle East
12,753
Southern Africa
11,468
Southern Cone
5,326
Levant Narrow
4,696
Baltic States
3,461
East Africa
3,314
Central America
2,455
Western Africa
1,567
Northern Africa
1,035
Central Africa
817
Central Asia
552
Live Financial Totals
AUKUS
6,849,254
AMER
6,749,644
EMEA
1,006,878
APAC
439,930
CANZUK
320,051
ASEAN+6
319,455
ANZUK
269,082
ASEAN+5
258,680
B BRICS
246,619
BRIC
235,232
ASEAN+3
212,452
LATCAR
174,843
BASIC
144,583
MART
136,844
LATAM
122,353
SEA
118,879
Tiger Cub Economies
116,082
CIVETS
110,399
M MIKTA
100,701
FRITES
100,142
ASIAN TIGERS
83,153
IMEA
80,821
PIGS
71,364
ANZ
60,990
FRIT
59,074
MEA
34,808
VISTA
32,745
MINT
23,774
MEP
14,402
MENA
13,362
CLMV
4,214
Live Group Totals
O Organisation for Economic Co-operation and Development
7,649,056
Club de Paris
7,646,027
G G20 Nations
7,635,838
D Development Assistance Committee
7,551,641
N North Atlantic Treaty Organization
7,454,424
G7 Nations
7,285,180
I International Solar Alliance
7,216,732
Asia-Pacific Economic Cooperation
7,023,047
O Organization of American States
6,760,179
Sovereign states and dependent territories of North America
6,694,995
G2 Nations
6,643,320
C Council of Europe
975,032
E European Union, European Economic Area and Switzerland
626,760
E European Economic Area
609,722
E Eastern Partnership
608,718
E Euronest Parliamentary Assembly
608,239
E European Union
604,769
A Assembly of European Regions
573,773
EC Inner Six
491,166
C Commonwealth of Nations
422,468
Majority German-speaking states of Central Europe
404,508
G Group of 77
392,464
EC Outer Seven
259,875
S Shanghai Cooperation Organisation
211,015
I Interparliamentary Assembly on Orthodoxy
171,512
O Organization of Ibero-American States
168,178
3 Global Governance Group
162,111
A Association of Southeast Asian Nations
118,855
B Organization of the Black Sea Economic Cooperation
117,084
C Commonwealth of Independent States Members
104,292
E Eurasian Economic Union
103,759
A African Union
98,072
A Association of Caribbean State Members
93,612
P Pacific Alliance
90,626
I IBSA Dialogue Forum
83,808
B Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
53,135
O Organisation of Eastern Caribbean States
50,600
S South Asian Association for Regional Cooperation
49,373
O Organisation of Islamic Cooperation
35,679
C Community of Portuguese Language Countries
31,215
C Central European Free Trade Agreement Members
30,937
M Southern Common Market
29,585
F Federation of Euro-Asian Stock Exchanges
22,538
E European Free Trade Association
21,991
V Visegrad Group
21,318
S Southern African Development Community
12,018
E Economic Cooperation Organization
10,632
O Organization of the Petroleum Exporting Countries
8,477
A Arab League
5,890
C Caribbean Community Members
4,967
Caribbean Free Trade Association
4,888
T Organization of Turkic States
3,826
C Central American Integration System
2,647
G GUAM Organization for Democracy and Economic Development
2,533
G Gulf Cooperation Council
2,528
E East African Community
1,134
Union of South American Nations
547