Log In · Signup for Free
EN  DE  FR  ES  IT  Pусский  한국어  日本語  简体中文  عربي

Diffbot Disallow Usage Statistics

D Diffbot Disallow

Diffbot uses AI to extract data from websites - this website blocks it.

https://diffbot.com

Robots.txt · AI Bot

Diffbot Disallow · Customers

Get access to data on 315,191 websites that are Diffbot Disallow Customers. We know of 259,620 live websites using Diffbot Disallow and an additional 55,571 sites that used Diffbot Disallow historically and 114,134 websites in the United States

Download Lead List
Technology Awards
🥉 3rd in AI Bot - Robots.txt growth in the last 7 days.
🥉 3rd in AI Bot - Robots.txt growth in the last 90 days.
4th in AI Bot - Robots.txt growth in the last 14 days.
4th in AI Bot - Robots.txt growth in the last 30 days.
4th in AI Bot - Robots.txt growth in the last 60 days.
7th in Robots.txt growth in the last 7 days.
8th in Robots.txt growth in the last 14 days.
8th in Robots.txt growth in the last 30 days.
7th Most popular technology in Åland Islands 🇦🇽 under AI Bot.
8th Most popular technology in Wallis and Futuna 🇼🇫 under AI Bot.
9th Most popular technology in Brazil 🇧🇷 under AI Bot.
9th Most popular technology in South Sudan 🇸🇸 under AI Bot.
10th Most popular technology in Åland Islands 🇦🇽 under Robots.txt.
10th Most popular technology in South Korea 🇰🇷 under AI Bot.
Investor Center
Competitors and Similar Technologies
Webz Omgili Disallow favicon
Webz Omgili Disallow

342,493 live sites

YOU Disallow favicon
YOU Disallow

163,249 live sites

DuckDuckGo Bot Disallow favicon
DuckDuckGo Bot Disallow

280,847 live sites

Google Image Bot Disallow favicon
Google Image Bot Disallow

325,695 live sites

ByteDance Bytespider Disallow favicon
ByteDance Bytespider Disallow

1,008,012 live sites

360 Bot Disallow favicon
360 Bot Disallow

187,605 live sites

Microsoft MSNBot Disallow favicon
Microsoft MSNBot Disallow

393,526 live sites

Exalead Bot Disallow favicon
Exalead Bot Disallow

409,286 live sites

LLMS Text favicon
LLMS Text

531,459 live sites

Youdao Bot Disallow favicon
Youdao Bot Disallow

113,665 live sites

Anthropic Claude User Bot Disallow favicon
Anthropic Claude User Bot Disallow

36,203 live sites

BuiltWithBot Disallow favicon
BuiltWithBot Disallow

77,647 live sites

Google Video Bot Disallow favicon
Google Video Bot Disallow

73,308 live sites

Google News Bot Disallow favicon
Google News Bot Disallow

72,320 live sites

Google Favicon Disallow favicon
Google Favicon Disallow

55,792 live sites

Google API Bot Disallow favicon
Google API Bot Disallow

30,771 live sites

Pipl Disallow favicon
Pipl Disallow

30,301 live sites

Google DuplexWeb Disallow favicon
Google DuplexWeb Disallow

25,609 live sites

Anthropic Claude Search Bot Disallow favicon
Anthropic Claude Search Bot Disallow

18,370 live sites

Google Read Aloud Disallow favicon
Google Read Aloud Disallow

5,539 live sites

Google FeedFetcher Disallow favicon
Google FeedFetcher Disallow

3,643 live sites

Google Web Light Disallow favicon
Google Web Light Disallow

564 live sites

Google BIG Bench Disallow favicon
Google BIG Bench Disallow

36 live sites

Download Lead List

Get a list of 363,786 websites using Diffbot Disallow which includes location information, hosting data and contact details. The list includes 259,620 live websites and 104,166 websites redirecting to those sites. 114,134 of these sites are in the United States.

We also know of 55,571 sites that have used Diffbot Disallow previously.

Site Totals
Total Live
259,620
104,166 additional website redirects?.
Expired
55,571
🇺🇸 American Live Sites
114,134
Live and Expired
315,191
Expired
55,571
Traffic
Top 1m
6,378
Top 100k
1,671
Top 10k
315
Top 1000
34
Outside Top 1m
251,956
Technology Spend
$100+
71,380
$500+
14,014
$1000+
5,654
$5k+
1,056
$10k+
117
Social Followers
10+
33,046
100+
29,873
500+
21,759
1000+
16,813
5000+
7,068
10k+
4,717
100k+
836
1m+
119
Sales Revenue
$100
26,822
$500
26,258
$1000
25,754
$5000
19,060
$10k
16,866
$100k
944
$1m
350
$10m
165
$100m
49
SKU Products
10+
3,593
50+
2,583
500+
1,049
100+
889
5000+
528
10k+
100
Live Country Totals
🇺🇸 United States
114,134
🇨🇭 Switzerland
15,858
🇧🇷 Brazil
13,367
🇬🇧 United Kingdom
13,254
🇮🇹 Italy
12,243
🇩🇪 Germany
11,633
🇪🇸 Spain
7,034
🇸🇪 Sweden
6,815
🇵🇱 Poland
5,803
🇳🇱 Netherlands
4,026
🇫🇷 France
3,769
🇩🇰 Denmark
3,559
🇨🇦 Canada
3,061
🇰🇷 South Korea
2,809
🇸🇰 Slovakia
2,779
🇦🇹 Austria
2,553
🇷🇺 Russia
2,369
🇭🇺 Hungary
2,347
🇨🇿 Czech Republic
1,730
🇮🇪 Ireland
1,690
🇦🇺 Australia
1,527
🇫🇮 Finland
1,482
🇪🇺 European Union
1,401
🇳🇴 Norway
1,292
🇬🇷 Greece
1,182
🇿🇦 South Africa
1,080
🇧🇪 Belgium
867
🇮🇩 Indonesia
631
🇮🇳 India
576
🇸🇮 Slovenia
538
🇲🇪 .me
510
🇱🇹 Lithuania
500
🇲🇽 Mexico
471
🇨🇳 China
407
🇨🇴 .co
360
🇮🇴 .io
301
🇯🇵 Japan
284
🇳🇿 New Zealand
267
🇵🇹 Portugal
259
🇹🇷 Turkey
224
🇨🇨 Cocos Islands
191
🇦🇪 UAE
179
🇦🇷 Argentina
175
🇷🇴 Romania
174
🇭🇰 Hong Kong
161
🇹🇼 Taiwan
159
🇳🇺 Niue
151
🇱🇮 Liechtenstein
144
🇦🇽 Åland Islands
126
🇺🇦 Ukraine
124
🇸🇬 Singapore
124
🇦🇮 Anguilla
114
🇲🇾 Malaysia
109
🇻🇳 Vietnam
107
🇹🇻 .tv
100
🇨🇱 Chile
86
🇱🇺 Luxembourg
85
🇧🇬 Bulgaria
76
🇹🇭 Thailand
72
🇻🇪 Venezuela
70
🇸🇭 Saint Helena
64
🇪🇪 Estonia
61
🇷🇸 Serbia
57
🇨🇾 Cyprus
56
🇰🇪 Kenya
52
🇭🇷 Croatia
49
🇵🇰 Pakistan
49
🇵🇭 Philippines
48
🇷🇪 Reunion
45
🇮🇸 Iceland
43
🇳🇬 Nigeria
42
🇮🇱 Israel
40
🇵🇪 Peru
38
🇮🇷 Iran
38
🇬🇬 Guernsey
36
🇺🇾 Uruguay
33
🇲🇦 Morocco
32
🇱🇻 Latvia
32
🇹🇴 Tonga
25
🇧🇩 Bangladesh
24
🇨🇷 Costa Rica
22
🇸🇦 Saudi Arabia
22
🇦🇱 Albania
22
🇮🇲 Isle of Man
21
🇼🇸 Samoa
21
🇱🇰 Sri Lanka
19
🇲🇹 Malta
19
🇪🇬 Egypt
18
🇫🇲 Micronesia
17
🇵🇼 Palau
17
🇳🇵 Nepal
17
🇦🇸 American Samoa
17
🇱🇧 Lebanon
15
🇵🇷 Puerto Rico
14
🇲🇰 Macedonia
14
🇪🇨 Ecuador
14
🇧🇦 Bosnia and Herzegovina
14
🇻🇨 Saint Vincent and the Grenadines
13
🇳🇦 Namibia
13
🇱🇦 Lao People's Democratic Republic
13
🇸🇹 Sao Tome and Principe
13
🇦🇲 Armenia
12
🇲🇩 Moldova
12
🇺🇬 Uganda
12
🇬🇪 Georgia
11
🇨🇽 Christmas Island
11
🇧🇸 Bahamas
11
🇬🇹 Guatemala
11
🇪🇹 Ethiopia
11
🇧🇼 Botswana
10
🇸🇴 Somalia
10
🇶🇦 Qatar
9
🇩🇴 Dominican Republic
9
🇰🇿 Kazakhstan
8
🇲🇷 Mauritania
8
🇬🇵 Guadeloupe
8
🇱🇾 Libya
8
🇦🇬 Antigua and Barbuda
8
🇧🇿 Belize
7
🇹🇿 Tanzania
7
🇵🇲 Saint Pierre and Miquelon
7
🇬🇭 Ghana
7
🇰🇭 Cambodia
7
🇹🇳 Tunisia
7
🇹🇫 French Southern Territories
7
🇵🇦 Panama
7
🇧🇾 Belarus
7
🇵🇾 Paraguay
7
🇦🇫 Afghanistan
6
🇬🇱 Greenland
6
🇫🇴 Faroe Islands
6
🇧🇴 Bolivia
6
🇩🇿 Algeria
5
🇺🇿 Uzbekistan
5
🇧🇭 Bahrain
5
🇲🇺 Mauritius
5
🇸🇻 El Salvador
5
🇮🇶 Iraq
5
🇭🇹 Haiti
5
🇹🇹 Trinidad and Tobago
5
🇻🇬 British Virgin Islands
5
🇨🇲 Cameroon
4
🇰🇼 Kuwait
4
🇲🇨 Monaco
4
🇼🇫 Wallis and Futuna
4
🇸🇱 Sierra Leone
4
🇲🇻 Maldives
4
🇯🇴 Jordan
4
🇲🇬 Madagascar
4
🇬🇲 Gambia
3
🇲🇲 Myanmar
3
🇲🇼 Malawi
3
🇳🇮 Nicaragua
3
🇬🇸 South Georgia and the South Sandwich Islands
3
🇲🇸 Montserrat
3
🇵🇸 Palestine
3
🇬🇾 Guyana
3
🇰🇾 Cayman Islands
3
🇸🇾 Syria
3
🇲🇶 Martinique
3
🇬🇩 Grenada
3
🇸🇩 Sudan
3
🇩🇯 Djibouti
3
🇾🇹 Mayotte
3
🇧🇲 Bermuda
2
🇸🇲 San Marino
2
🇿🇼 Zimbabwe
2
🇯🇪 Jersey
2
🇸🇽 Sint Maarten
2
🇸🇨 Seychelles
2
🇲🇵 Northern Mariana Islands
2
🇲🇳 Mongolia
2
🇭🇳 Honduras
2
🇧🇮 Burundi
2
🇨🇩 Congo
2
🇦🇴 Angola
2
🇧🇧 Barbados
2
🇸🇧 Solomon Islands
2
🇦🇿 Azerbaijan
1
🇸🇿 Swaziland
1
🇴🇲 Oman
1
🇯🇲 Jamaica
1
🇿🇲 Zambia
1
🇱🇷 Liberia
1
🇸🇷 Suriname
1
🇬🇺 Guam
1
🇨🇺 Cuba
1
🇻🇺 Vanuatu
1
🇱🇨 Saint Lucia
1
🇧🇳 Brunei
1
🇸🇳 Senegal
1
🇻🇮 United States Virgin Islands
1
🇱🇸 Lesotho
1
🇸🇸 South Sudan
1
🇬🇦 Gabon
1
🇲🇱 Mali
1
🇲🇴 Macao
1
🇸🇯 Svalbard and Jan Mayen
1
🇰🇬 Kyrgyzstan
1
🇹🇬 Togo
1
Live Region Totals
North America
117,195
Europe
105,343
Western Europe
53,883
Central Europe
43,385
Southern Europe
21,943
Northern Europe
15,474
CHAPS
15,212
South America
14,172
N Nordics
13,330
Scandinavia
11,666
B Benelux Union
4,978
East Asia
3,823
Balkan States
2,860
Africa
1,502
Sub-Saharan Africa
1,418
Southern Africa
1,105
South Asia
695
Western Asia
650
Middle East
626
Baltic States
593
Levant Narrow
345
Southern Cone
301
East Africa
118
Northern Africa
81
Western Africa
68
Central America
57
Central Africa
24
Central Asia
14
Live Financial Totals
AMER
131,417
AUKUS
128,915
EMEA
107,173
FRITES
23,046
PIGS
20,718
CANZUK
18,109
B BRICS
17,799
BRIC
16,719
FRIT
16,012
BASIC
15,430
ANZUK
15,048
LATCAR
14,894
LATAM
14,717
APAC
8,288
ASEAN+6
6,985
ASEAN+5
6,578
M MIKTA
5,662
ASEAN+3
4,615
MART
4,479
ASIAN TIGERS
3,501
IMEA
2,686
CIVETS
2,420
VISTA
2,217
MEA
2,110
ANZ
1,794
MINT
1,368
SEA
1,115
Tiger Cub Economies
967
MENA
689
MEP
675
CLMV
130
Live Group Totals
O Organisation for Economic Co-operation and Development
224,719
Club de Paris
223,666
D Development Assistance Committee
222,923
N North Atlantic Treaty Organization
191,300
G G20 Nations
182,036
I International Solar Alliance
172,232
G7 Nations
158,378
O Organization of American States
131,941
Asia-Pacific Economic Cooperation
126,865
Sovereign states and dependent territories of North America
117,948
G2 Nations
114,541
C Council of Europe
105,328
E European Union, European Economic Area and Switzerland
88,698
A Assembly of European Regions
81,194
E European Economic Area
72,840
E Eastern Partnership
71,528
E Euronest Parliamentary Assembly
71,520
E European Union
71,361
EC Outer Seven
43,590
EC Inner Six
32,623
Majority German-speaking states of Central Europe
30,044
O Organization of Ibero-American States
21,994
C Commonwealth of Nations
20,540
3 Global Governance Group
19,698
G Group of 77
18,233
E European Free Trade Association
17,337
I Interparliamentary Assembly on Orthodoxy
17,007
I IBSA Dialogue Forum
15,023
C Community of Portuguese Language Countries
13,641
M Southern Common Market
13,582
V Visegrad Group
12,659
B Organization of the Black Sea Economic Cooperation
4,207
S Shanghai Cooperation Organisation
3,415
F Federation of Euro-Asian Stock Exchanges
2,539
C Commonwealth of Independent States Members
2,415
E Eurasian Economic Union
2,397
A African Union
1,741
O Organisation of Islamic Cooperation
1,537
S Southern African Development Community
1,133
A Association of Southeast Asian Nations
1,115
A Association of Caribbean State Members
1,021
P Pacific Alliance
955
B Bay of Bengal Initiative for Multi-Sectoral Technical and Economic Cooperation
711
S South Asian Association for Regional Cooperation
695
C Central European Free Trade Agreement Members
629
O Organization of the Petroleum Exporting Countries
376
A Arab League
341
E Economic Cooperation Organization
332
T Organization of Turkic States
239
G Gulf Cooperation Council
220
O Organisation of Eastern Caribbean States
158
G GUAM Organization for Democracy and Economic Development
148
Union of South American Nations
80
E East African Community
76
C Central American Integration System
66
C Caribbean Community Members
63
Caribbean Free Trade Association
46