Log In · Signup for Free

EN DE FR ES IT Pусский 한국어 日本語简体中文 عربي

AI Bot Web Usage Distribution in the Top 1 Million sites

Distribution for websites using AI Bot Robots.txt technologies

Market Share Evolution · AI Bot Web Usage Distribution in the Top 1 Million sites

Top in AI Bot Web Usage Distribution in the Top 1 Million sites

	Technology	Websites
	GPTBot Disallow	32,915
	Common Crawl Bot Disallow	29,254
	Anthropic Claude Bot Disallow	27,701
	AmazonBot Disallow	26,356
	Google Bard Disallow	24,789
	ByteDance Bytespider Disallow	24,706
	Apple AI Bot Disallow	22,041
	Alexa Bot Disallow	11,493
	Facebook Bot Disallow	9,317
	Perplexity	8,430
	Cohere AI Disallow	6,485
	Webz Omgili Disallow	5,293
	YOU Disallow	4,720
	Diffbot Disallow	4,256
	Anthropic Claude User Bot Disallow	1,027

238,783 Detections

of AI Bot Robots.txt in the Top 1 Million sites. Last updated 11 Oct 2025.

GPTBot Disallow is currently the most used technology in this category.

Totals

Top 1m Top 1,000,000 sites by traffic rank: 238,783
Top 100k Top 100,000 sites by traffic rank: 34,824
Top 10k Top 10,000 sites by traffic rank: 4,844
Top 1k Top 1,000 sites by traffic rank: 701
Entire Internet: 46,317,170
🇺🇸 United States: 29,561,597
🇺🇸 United States: 29,561,597
🇬🇧 United Kingdom: 1,978,865
🇦🇺 Australia: 1,395,417
🇩🇪 Germany: 1,110,026
🇨🇦 Canada: 1,049,347
🇪🇸 Spain: 447,416
🇨🇭 Switzerland: 397,416
🇷🇺 Russia: 394,076
🇨🇴 .co: 332,223
🇫🇷 France: 308,728
🇧🇷 Brazil: 305,313
🇮🇹 Italy: 298,657
🇳🇱 Netherlands: 293,211
🇳🇿 New Zealand: 266,227
🇸🇪 Sweden: 229,121
🇮🇳 India: 194,257
🇮🇩 Indonesia: 185,264
🇵🇱 Poland: 180,536
🇨🇳 China: 171,954
🇩🇰 Denmark: 146,983
🇲🇪 .me: 146,815
🇳🇴 Norway: 145,590
🇨🇨 Cocos Islands: 139,381
🇮🇪 Ireland: 134,128
🇧🇪 Belgium: 134,038
🇦🇹 Austria: 115,386
🇲🇽 Mexico: 114,096
🇮🇴 .io: 109,247
🇫🇮 Finland: 101,629
🇪🇺 European Union: 97,370
🇯🇵 Japan: 85,787
🇿🇦 South Africa: 77,990
🇨🇿 Czech Republic: 72,871
🇹🇷 Turkey: 71,240
🇰🇷 South Korea: 64,104
🇦🇷 Argentina: 63,527
🇵🇹 Portugal: 61,185
🇦🇮 .ai: 57,849
🇸🇬 Singapore: 51,367
🇬🇷 Greece: 49,751
🇸🇰 Slovakia: 49,099
🇷🇴 Romania: 44,200
🇲🇾 Malaysia: 39,080
🇻🇳 Vietnam: 38,105
🇨🇱 Chile: 38,075
🇹🇭 Thailand: 36,851
🇭🇰 Hong Kong: 32,764
🇦🇪 UAE: 31,396
🇭🇺 Hungary: 30,891
🇺🇦 Ukraine: 29,922
🇹🇼 Taiwan: 26,028
🇮🇱 Israel: 25,169
🇹🇻 .tv: 23,800
🇮🇷 Iran: 22,298
🇵🇭 Philippines: 20,714
🇵🇰 Pakistan: 16,939
🇮🇸 Iceland: 15,558
🇱🇹 Lithuania: 15,530
🇳🇬 Nigeria: 14,319
🇸🇮 Slovenia: 13,861
🇧🇩 Bangladesh: 12,951
🇱🇺 Luxembourg: 12,758
🇰🇪 Kenya: 12,180
🇵🇪 Peru: 11,991
🇨🇾 Cyprus: 11,519
🇳🇵 Nepal: 10,955
🇪🇪 Estonia: 10,428
🇨🇷 Costa Rica: 9,997
🇱🇻 Latvia: 9,799
🇪🇨 Ecuador: 9,436
🇧🇬 Bulgaria: 8,950
🇱🇰 Sri Lanka: 8,819
🇸🇦 Saudi Arabia: 8,276
🇭🇷 Croatia: 8,209
🇮🇲 Isle of Man: 7,810
🇲🇦 Morocco: 7,396
🇬🇬 Guernsey: 6,684
🇱🇮 Liechtenstein: 6,559
🇵🇷 Puerto Rico: 6,317
🇱🇦 Lao People's Democratic Republic: 5,903
🇷🇸 Serbia: 5,615
🇸🇭 Saint Helena: 5,438
🇳🇺 Niue: 5,235
🇬🇹 Guatemala: 5,174
🇵🇼 Palau: 5,162
🇼🇸 Samoa: 5,024
🇩🇴 Dominican Republic: 4,783
🇦🇱 Albania: 4,758
🇲🇹 Malta: 4,746
🇱🇧 Lebanon: 4,696
🇦🇿 Azerbaijan: 4,665
🇹🇴 Tonga: 4,531
🇪🇬 Egypt: 4,381
🇻🇪 Venezuela: 4,211
🇻🇨 Saint Vincent and the Grenadines: 4,126
🇺🇾 Uruguay: 3,935
🇰🇿 Kazakhstan: 3,662
🇹🇿 Tanzania: 3,619
🇬🇪 Georgia: 3,320
🇨🇲 Cameroon: 3,289
🇺🇬 Uganda: 3,204
🇸🇽 Sint Maarten: 3,170
🇵🇦 Panama: 3,073
🇲🇩 Moldova: 2,528
🇬🇭 Ghana: 2,497
🇶🇦 Qatar: 2,488
🇦🇲 Armenia: 2,437
🇺🇿 Uzbekistan: 2,437
🇧🇿 Belize: 2,307
🇷🇪 Reunion: 2,300
🇹🇳 Tunisia: 2,269
🇨🇻 Cabo Verde: 2,261
🇰🇼 Kuwait: 2,258
🇫🇲 Micronesia: 2,184
🇯🇴 Jordan: 2,104
🇯🇪 Jersey: 1,962
🇿🇼 Zimbabwe: 1,925
🇵🇾 Paraguay: 1,876
🇰🇭 Cambodia: 1,872
🇲🇰 Macedonia: 1,848
🇧🇲 Bermuda: 1,822
🇰🇾 Cayman Islands: 1,807
🇧🇾 Belarus: 1,803
🇸🇴 Somalia: 1,794
🇰🇬 Kyrgyzstan: 1,760
🇦🇽 Åland Islands: 1,735
🇧🇸 Bahamas: 1,732
🇱🇾 Libya: 1,681
🇨🇽 Christmas Island: 1,668
🇲🇺 Mauritius: 1,636
🇲🇳 Mongolia: 1,611
🇪🇹 Ethiopia: 1,568
🇭🇳 Honduras: 1,532
🇦🇬 Antigua and Barbuda: 1,440
🇹🇹 Trinidad and Tobago: 1,421
🇸🇻 El Salvador: 1,401
🇷🇼 Rwanda: 1,397
🇧🇦 Bosnia and Herzegovina: 1,378
🇧🇭 Bahrain: 1,357
🇵🇲 Saint Pierre and Miquelon: 1,349
🇻🇮 United States Virgin Islands: 1,333
🇫🇴 Faroe Islands: 1,253
🇦🇩 Andorra: 1,217
🇲🇻 Maldives: 1,196
🇮🇶 Iraq: 1,162
🇧🇧 Barbados: 1,157
🇸🇹 Sao Tome and Principe: 1,139
🇴🇲 Oman: 1,136
🇧🇴 Bolivia: 1,131
🇯🇲 Jamaica: 1,075
🇳🇦 Namibia: 1,067
🇲🇨 Monaco: 993
🇳🇮 Nicaragua: 980
🇲🇲 Myanmar: 958
🇿🇲 Zambia: 950
🇻🇺 Vanuatu: 940
🇨🇮 Cote d'Ivoire: 871
🇱🇨 Saint Lucia: 860
🇲🇿 Mozambique: 825
🇦🇫 Afghanistan: 794
🇵🇸 Palestine: 779
🇹🇨 Turks and Caicos Islands: 756
🇸🇳 Senegal: 739
🇫🇯 Fiji: 737
🇧🇼 Botswana: 708
🇦🇴 Angola: 683
🇻🇬 British Virgin Islands: 669
🇦🇸 American Samoa: 655
🇩🇿 Algeria: 640
🇨🇩 Congo: 640
🇸🇨 Seychelles: 586
🇬🇱 Greenland: 578
🇭🇹 Haiti: 557
🇬🇵 Guadeloupe: 534
🇬🇩 Grenada: 521
🇲🇬 Madagascar: 517
🇬🇮 Gibraltar: 487
🇲🇸 Montserrat: 485
🇹🇫 French Southern Territories: 463
🇲🇱 Mali: 442
🇲🇼 Malawi: 415
🇵🇬 Papua New Guinea: 409
🇬🇸 South Georgia and the South Sandwich Islands: 403
🇬🇾 Guyana: 355
🇲🇶 Martinique: 345
🇦🇼 Aruba: 342
🇬🇺 Guam: 308
🇹🇬 Togo: 305
🇸🇷 Suriname: 300
🇸🇾 Syria: 298
🇸🇱 Sierra Leone: 297
🇬🇲 Gambia: 296
🇲🇷 Mauritania: 296
🇾🇹 Mayotte: 266
🇧🇯 Benin: 263
🇨🇰 Cook Islands: 239
🇧🇳 Brunei: 227
🇨🇺 Cuba: 218
🇱🇷 Liberia: 215
🇹🇯 Tajikistan: 210
🇸🇿 Swaziland: 178
🇧🇮 Burundi: 171
🇨🇼 Curacao: 169
🇸🇧 Solomon Islands: 166
🇲🇴 Macao: 163
🇱🇸 Lesotho: 158
🇩🇯 Djibouti: 154
🇸🇩 Sudan: 152
🇼🇫 Wallis and Futuna: 146
🇳🇨 New Caledonia: 142
🇹🇱 East Timor: 141
🇧🇫 Burkina Faso: 131
🇧🇹 Bhutan: 123
🇵🇫 French Polynesia: 122
🇲🇵 Northern Mariana Islands: 118
🇳🇪 Niger: 115
🇾🇪 Yemen: 107
🇸🇯 Svalbard and Jan Mayen: 103
🇬🇦 Gabon: 94
🇸🇲 San Marino: 92
🇩🇲 Dominica: 85
🇸🇸 South Sudan: 81
🇰🇳 Saint Kitts and Nevis: 73
🇹🇲 Turkmenistan: 59
🇬🇫 French Guiana: 59
🇫🇰 Falkland Islands: 48
🇰🇮 Kiribati: 46
🇭🇲 Heard Island and McDonald Islands: 46
🇺🇲 United States Minor Outlying Islands: 46
🇳🇫 Norfolk Island: 42
🇹🇩 Chad: 35
🇨🇬 Congo: 32
🇧🇱 Saint Barthélemy: 27
🇵🇳 Pitcairn: 26
🇬🇼 Guinea-Bissau: 21
🇬🇳 Guinea: 19
🇲🇭 Marshall Islands: 18
🇳🇷 Nauru: 18
🇧🇶 Bonaire: 18
🇻🇦 Vatican City: 17
🇬🇶 Equatorial Guinea: 17
🇪🇷 Eritrea: 9
🇦🇶 Antarctica: 9

AI Bot Robots.txt Categories

All Robots.txt AI Bot

Excluded from Pie Charts

Robots Disallow

Some Robots Disallow

General robots disallow rule.