Log In · Signup for Free

EN DE FR ES IT Pусский 한국어 日本語简体中文 عربي

AI Bot Web Usage Distribution in the Top 1 Million sites

Distribution for websites using AI Bot Robots.txt technologies

Market Share Evolution · AI Bot Web Usage Distribution in the Top 1 Million sites

Top in AI Bot Web Usage Distribution in the Top 1 Million sites

	Technology	Websites
	GPTBot Disallow	28,492
	Common Crawl Bot Disallow	22,999
	Anthropic Claude Bot Disallow	21,283
	AmazonBot Disallow	18,233
	Google Bard Disallow	17,362
	ByteDance Bytespider Disallow	17,103
	Apple AI Bot Disallow	13,418
	Alexa Bot Disallow	11,493
	Facebook Bot Disallow	10,366
	Perplexity	9,839
	Cohere AI Disallow	7,967
	Webz Omgili Disallow	6,820
	Diffbot Disallow	5,191
	YOU Disallow	4,588
	Anthropic Claude User Bot Disallow	667

195,821 Detections

of AI Bot Robots.txt in the Top 1 Million sites. Last updated 16 Sep 2025.

GPTBot Disallow is currently the most used technology in this category.

Totals

Top 1m Top 1,000,000 sites by traffic rank: 195,821
Top 100k Top 100,000 sites by traffic rank: 39,309
Top 10k Top 10,000 sites by traffic rank: 6,539
Top 1k Top 1,000 sites by traffic rank: 759
Entire Internet: 42,212,421
🇺🇸 United States: 27,336,694
🇺🇸 United States: 27,336,694
🇬🇧 United Kingdom: 1,895,806
🇦🇺 Australia: 1,339,988
🇩🇪 Germany: 1,093,981
🇨🇦 Canada: 1,001,248
🇪🇸 Spain: 436,294
🇨🇭 Switzerland: 391,224
🇷🇺 Russia: 373,461
🇮🇹 Italy: 301,741
🇨🇴 .co: 291,831
🇫🇷 France: 288,817
🇳🇱 Netherlands: 279,881
🇧🇷 Brazil: 260,646
🇳🇿 New Zealand: 253,799
🇸🇪 Sweden: 222,203
🇵🇱 Poland: 153,384
🇨🇳 China: 148,896
🇮🇳 India: 145,902
🇩🇰 Denmark: 142,794
🇳🇴 Norway: 141,773
🇮🇪 Ireland: 129,412
🇧🇪 Belgium: 129,007
🇲🇪 .me: 127,976
🇮🇩 Indonesia: 119,238
🇦🇹 Austria: 111,521
🇨🇨 Cocos Islands: 108,391
🇲🇽 Mexico: 102,699
🇫🇮 Finland: 97,860
🇯🇵 Japan: 90,686
🇪🇺 European Union: 86,867
🇮🇴 .io: 84,075
🇨🇿 Czech Republic: 69,212
🇿🇦 South Africa: 69,149
🇰🇷 South Korea: 61,028
🇵🇹 Portugal: 54,711
🇦🇷 Argentina: 52,295
🇸🇬 Singapore: 48,188
🇸🇰 Slovakia: 47,944
🇹🇷 Turkey: 45,591
🇬🇷 Greece: 42,316
🇦🇮 .ai: 42,268
🇷🇴 Romania: 38,419
🇭🇺 Hungary: 37,294
🇭🇰 Hong Kong: 30,486
🇨🇱 Chile: 29,625
🇹🇭 Thailand: 28,685
🇲🇾 Malaysia: 28,137
🇦🇪 UAE: 26,221
🇻🇳 Vietnam: 24,440
🇹🇼 Taiwan: 22,455
🇹🇻 .tv: 21,784
🇺🇦 Ukraine: 20,623
🇵🇭 Philippines: 18,232
🇮🇱 Israel: 17,728
🇮🇸 Iceland: 14,308
🇱🇹 Lithuania: 13,518
🇸🇮 Slovenia: 13,185
🇮🇷 Iran: 12,927
🇱🇺 Luxembourg: 12,475
🇨🇾 Cyprus: 10,339
🇵🇰 Pakistan: 10,018
🇳🇬 Nigeria: 9,386
🇨🇷 Costa Rica: 9,335
🇪🇪 Estonia: 9,271
🇵🇪 Peru: 9,150
🇰🇪 Kenya: 8,542
🇪🇨 Ecuador: 8,207
🇧🇩 Bangladesh: 8,065
🇱🇻 Latvia: 7,990
🇧🇬 Bulgaria: 7,371
🇮🇲 Isle of Man: 6,925
🇭🇷 Croatia: 6,831
🇳🇵 Nepal: 6,831
🇵🇷 Puerto Rico: 6,257
🇱🇮 Liechtenstein: 6,165
🇸🇦 Saudi Arabia: 6,134
🇱🇰 Sri Lanka: 5,537
🇲🇦 Morocco: 5,305
🇱🇦 Lao People's Democratic Republic: 5,272
🇬🇬 Guernsey: 4,971
🇳🇺 Niue: 4,836
🇬🇹 Guatemala: 4,687
🇼🇸 Samoa: 4,459
🇱🇧 Lebanon: 4,414
🇩🇴 Dominican Republic: 4,308
🇷🇸 Serbia: 4,300
🇸🇭 Saint Helena: 4,257
🇲🇹 Malta: 4,237
🇻🇨 Saint Vincent and the Grenadines: 3,753
🇦🇱 Albania: 3,743
🇵🇼 Palau: 3,583
🇪🇬 Egypt: 3,537
🇻🇪 Venezuela: 3,514
🇺🇾 Uruguay: 3,414
🇹🇴 Tonga: 3,152
🇸🇽 Sint Maarten: 3,103
🇨🇲 Cameroon: 3,067
🇵🇦 Panama: 2,860
🇹🇿 Tanzania: 2,818
🇺🇬 Uganda: 2,719
🇰🇿 Kazakhstan: 2,475
🇦🇿 Azerbaijan: 2,431
🇬🇪 Georgia: 2,307
🇬🇭 Ghana: 2,096
🇶🇦 Qatar: 2,039
🇰🇼 Kuwait: 2,004
🇫🇲 Micronesia: 1,997
🇷🇪 Reunion: 1,991
🇧🇿 Belize: 1,938
🇯🇪 Jersey: 1,896
🇦🇲 Armenia: 1,787
🇯🇴 Jordan: 1,785
🇧🇲 Bermuda: 1,783
🇦🇽 Åland Islands: 1,731
🇰🇾 Cayman Islands: 1,726
🇧🇾 Belarus: 1,701
🇧🇸 Bahamas: 1,688
🇨🇻 Cabo Verde: 1,671
🇰🇭 Cambodia: 1,584
🇲🇩 Moldova: 1,546
🇹🇳 Tunisia: 1,489
🇲🇺 Mauritius: 1,430
🇭🇳 Honduras: 1,403
🇺🇿 Uzbekistan: 1,393
🇸🇴 Somalia: 1,383
🇹🇹 Trinidad and Tobago: 1,310
🇻🇮 United States Virgin Islands: 1,298
🇸🇻 El Salvador: 1,291
🇪🇹 Ethiopia: 1,287
🇵🇲 Saint Pierre and Miquelon: 1,268
🇿🇼 Zimbabwe: 1,254
🇲🇰 Macedonia: 1,249
🇵🇾 Paraguay: 1,237
🇱🇾 Libya: 1,178
🇧🇭 Bahrain: 1,175
🇦🇬 Antigua and Barbuda: 1,159
🇧🇧 Barbados: 1,125
🇫🇴 Faroe Islands: 1,101
🇰🇬 Kyrgyzstan: 1,094
🇷🇼 Rwanda: 1,071
🇨🇽 Christmas Island: 1,069
🇧🇦 Bosnia and Herzegovina: 1,056
🇯🇲 Jamaica: 1,026
🇲🇳 Mongolia: 1,015
🇲🇨 Monaco: 946
🇦🇩 Andorra: 945
🇳🇮 Nicaragua: 906
🇴🇲 Oman: 890
🇳🇦 Namibia: 884
🇧🇴 Bolivia: 881
🇲🇻 Maldives: 830
🇮🇶 Iraq: 794
🇸🇹 Sao Tome and Principe: 778
🇱🇨 Saint Lucia: 768
🇿🇲 Zambia: 756
🇲🇲 Myanmar: 727
🇻🇺 Vanuatu: 712
🇫🇯 Fiji: 705
🇹🇨 Turks and Caicos Islands: 692
🇲🇿 Mozambique: 669
🇦🇫 Afghanistan: 665
🇨🇮 Cote d'Ivoire: 647
🇦🇸 American Samoa: 640
🇵🇸 Palestine: 635
🇸🇳 Senegal: 585
🇧🇼 Botswana: 580
🇻🇬 British Virgin Islands: 554
🇩🇿 Algeria: 549
🇨🇩 Congo: 543
🇬🇵 Guadeloupe: 537
🇸🇨 Seychelles: 514
🇭🇹 Haiti: 509
🇬🇱 Greenland: 495
🇦🇴 Angola: 482
🇬🇮 Gibraltar: 471
🇬🇩 Grenada: 460
🇲🇬 Madagascar: 421
🇲🇸 Montserrat: 377
🇵🇬 Papua New Guinea: 363
🇹🇫 French Southern Territories: 356
🇲🇱 Mali: 341
🇲🇶 Martinique: 341
🇬🇾 Guyana: 339
🇲🇼 Malawi: 332
🇬🇸 South Georgia and the South Sandwich Islands: 327
🇬🇺 Guam: 325
🇦🇼 Aruba: 318
🇬🇲 Gambia: 290
🇸🇷 Suriname: 249
🇸🇱 Sierra Leone: 242
🇲🇷 Mauritania: 233
🇾🇹 Mayotte: 226
🇧🇳 Brunei: 225
🇱🇷 Liberia: 225
🇨🇰 Cook Islands: 221
🇸🇾 Syria: 217
🇨🇺 Cuba: 212
🇹🇬 Togo: 200
🇧🇯 Benin: 165
🇧🇮 Burundi: 152
🇲🇴 Macao: 150
🇨🇼 Curacao: 150
🇸🇿 Swaziland: 149
🇸🇧 Solomon Islands: 146
🇼🇫 Wallis and Futuna: 145
🇸🇩 Sudan: 143
🇩🇯 Djibouti: 140
🇹🇯 Tajikistan: 136
🇱🇸 Lesotho: 128
🇵🇫 French Polynesia: 128
🇳🇨 New Caledonia: 121
🇲🇵 Northern Mariana Islands: 115
🇹🇱 East Timor: 109
🇧🇹 Bhutan: 107
🇸🇲 San Marino: 104
🇸🇯 Svalbard and Jan Mayen: 103
🇧🇫 Burkina Faso: 99
🇳🇪 Niger: 95
🇬🇦 Gabon: 85
🇰🇳 Saint Kitts and Nevis: 81
🇩🇲 Dominica: 79
🇾🇪 Yemen: 71
🇬🇫 French Guiana: 67
🇸🇸 South Sudan: 48
🇫🇰 Falkland Islands: 47
🇰🇮 Kiribati: 46
🇺🇲 United States Minor Outlying Islands: 36
🇭🇲 Heard Island and McDonald Islands: 32
🇨🇬 Congo: 31
🇳🇫 Norfolk Island: 29
🇧🇱 Saint Barthélemy: 27
🇵🇳 Pitcairn: 26
🇹🇲 Turkmenistan: 22
🇹🇩 Chad: 22
🇬🇼 Guinea-Bissau: 21
🇬🇳 Guinea: 18
🇲🇭 Marshall Islands: 18
🇳🇷 Nauru: 18
🇧🇶 Bonaire: 18
🇻🇦 Vatican City: 10
🇪🇷 Eritrea: 9
🇦🇶 Antarctica: 9
🇬🇶 Equatorial Guinea: 2

AI Bot Robots.txt Categories

All Robots.txt AI Bot

Excluded from Pie Charts

Robots Disallow

Some Robots Disallow

General robots disallow rule.