Log In · Signup for Free

EN DE FR ES IT Pусский 한국어 日本語简体中文 عربي

Robots.txt Web Usage Distribution on the Entire Internet

Distribution for websites using Robots.txt technologies

Market Share Evolution · Robots.txt Web Usage Distribution on the Entire Internet

Top in Robots.txt Web Usage Distribution on the Entire Internet

	Technology	Websites
	Googlebot Disallow	18,145,897
	Baidu Bot Disallow	12,739,715
	Microsoft BingBot Disallow	12,568,342
	Yandex Bot Disallow	12,495,810
	Sogou Bot Disallow	12,003,031
	Yahoo Slurp Disallow	11,856,471
	Google Media Partners Disallow	11,796,610
	Google AdsBot Disallow	8,906,110
	Google AdsBot Mobile Disallow	8,297,265
	AppleBot Disallow	5,264,472
	Common Crawl Bot Disallow	5,190,420
	Anthropic Claude Bot Disallow	5,119,278
	Google Bard Disallow	5,015,490
	GPTBot Disallow	4,992,257
	Apple AI Bot Disallow	4,923,004
	Alexa Bot Disallow	4,719,358
	Soso Bot Disallow	4,299,023
	Facebook Bot Disallow	3,518,939
	Cohere AI Disallow	3,471,126
	Perplexity	3,461,325
	ByteDance Bytespider Disallow	1,954,780
	Ahrefs Bot Disallow	1,856,245
	Semrush Bot Disallow	1,417,242
	Majestic Bot Disallow	1,335,258
	LLMS Text	844,473
	Exalead Bot Disallow	316,384
	Microsoft MSNBot Disallow	298,512
	DuckDuckGo Bot Disallow	284,759
	Google Image Bot Disallow	270,110
	Webz Omgili Disallow	250,649
	Diffbot Disallow	233,900
	Youdao Bot Disallow	181,817
	YOU Disallow	156,388
	Anthropic Claude User Bot Disallow	90,943
	360 Bot Disallow	84,870
	Anthropic Claude Search Bot Disallow	63,355
	Google Video Bot Disallow	54,616
	Google News Bot Disallow	51,290
	Google Favicon Disallow	36,783
	BuiltWithBot Disallow	27,662
	Pipl Disallow	22,873
	Google API Bot Disallow	12,010
	Google DuplexWeb Disallow	7,468
	Google Read Aloud Disallow	5,177
	Google FeedFetcher Disallow	3,444
	Google AdsBot Mobile Apps Disallow	1,371
	Google Web Light Disallow	529
	Google BIG Bench Disallow	41

168,646,892 Detections

of Robots.txt on the Entire Internet. Last updated 4 Nov 2025.

Googlebot Disallow is currently the most used technology in this category.

Totals

Top 1m Top 1,000,000 sites by traffic rank: 554,504
Top 100k Top 100,000 sites by traffic rank: 76,886
Top 10k Top 10,000 sites by traffic rank: 10,612
Top 1k Top 1,000 sites by traffic rank: 1,520
Entire Internet: 168,646,892
🇺🇸 United States: 97,981,890
🇺🇸 United States: 97,981,890
🇩🇪 Germany: 7,828,491
🇬🇧 United Kingdom: 5,366,705
🇦🇺 Australia: 2,991,776
🇨🇦 Canada: 2,221,470
🇷🇺 Russia: 1,563,000
🇨🇳 China: 1,302,631
🇫🇷 France: 1,253,001
🇮🇹 Italy: 1,169,092
🇨🇴 .co: 1,141,823
🇳🇱 Netherlands: 1,107,199
🇪🇸 Spain: 1,029,373
🇨🇭 Switzerland: 999,939
🇧🇷 Brazil: 970,523
🇮🇳 India: 950,753
🇨🇨 Cocos Islands: 845,200
🇯🇵 Japan: 750,236
🇵🇱 Poland: 534,227
🇪🇺 European Union: 497,378
🇲🇪 .me: 494,588
🇮🇴 .io: 493,269
🇧🇪 Belgium: 461,500
🇳🇿 New Zealand: 453,508
🇸🇪 Sweden: 448,754
🇦🇹 Austria: 423,253
🇦🇮 .ai: 348,626
🇲🇽 Mexico: 337,170
🇨🇿 Czech Republic: 327,235
🇳🇴 Norway: 321,750
🇮🇩 Indonesia: 318,255
🇮🇪 Ireland: 305,995
🇿🇦 South Africa: 293,271
🇩🇰 Denmark: 262,325
🇹🇷 Turkey: 256,184
🇰🇷 South Korea: 227,691
🇷🇴 Romania: 226,915
🇫🇮 Finland: 224,661
🇵🇹 Portugal: 161,993
🇹🇻 .tv: 146,695
🇦🇷 Argentina: 123,154
🇸🇬 Singapore: 121,567
🇲🇾 Malaysia: 119,685
🇨🇱 Chile: 118,837
🇬🇷 Greece: 112,852
🇹🇭 Thailand: 102,495
🇭🇰 Hong Kong: 96,851
🇵🇭 Philippines: 95,158
🇹🇴 Tonga: 93,463
🇮🇱 Israel: 92,076
🇺🇦 Ukraine: 90,872
🇭🇺 Hungary: 90,640
🇻🇳 Vietnam: 89,848
🇸🇰 Slovakia: 88,289
🇹🇼 Taiwan: 83,685
🇦🇪 UAE: 78,262
🇱🇺 Luxembourg: 68,159
🇻🇬 British Virgin Islands: 65,992
🇼🇸 Samoa: 48,653
🇮🇷 Iran: 44,155
🇵🇰 Pakistan: 40,727
🇸🇮 Slovenia: 39,998
🇮🇸 Iceland: 39,381
🇮🇲 Isle of Man: 38,525
🇱🇹 Lithuania: 38,254
🇪🇪 Estonia: 36,067
🇨🇾 Cyprus: 35,332
🇸🇭 Saint Helena: 34,950
🇱🇻 Latvia: 34,918
🇵🇪 Peru: 33,154
🇧🇬 Bulgaria: 32,314
🇬🇬 Guernsey: 29,626
🇳🇬 Nigeria: 29,468
🇪🇨 Ecuador: 28,277
🇸🇽 Sint Maarten: 27,045
🇨🇲 Cameroon: 26,416
🇱🇦 Lao People's Democratic Republic: 26,240
🇻🇨 Saint Vincent and the Grenadines: 25,892
🇵🇼 Palau: 24,229
🇧🇩 Bangladesh: 23,808
🇰🇪 Kenya: 23,652
🇭🇷 Croatia: 22,477
🇳🇺 Niue: 21,395
🇱🇮 Liechtenstein: 20,399
🇨🇷 Costa Rica: 19,241
🇸🇦 Saudi Arabia: 18,678
🇷🇸 Serbia: 18,177
🇷🇪 Reunion: 18,065
🇸🇴 Somalia: 16,762
🇵🇷 Puerto Rico: 16,681
🇳🇵 Nepal: 16,183
🇺🇾 Uruguay: 14,955
🇲🇦 Morocco: 14,649
🇱🇰 Sri Lanka: 14,621
🇲🇹 Malta: 13,380
🇦🇲 Armenia: 12,168
🇩🇴 Dominican Republic: 11,597
🇧🇿 Belize: 11,056
🇨🇽 Christmas Island: 10,960
🇪🇬 Egypt: 10,907
🇬🇹 Guatemala: 10,780
🇦🇱 Albania: 10,603
🇵🇲 Saint Pierre and Miquelon: 10,305
🇰🇿 Kazakhstan: 10,188
🇫🇲 Micronesia: 10,151
🇻🇪 Venezuela: 9,659
🇺🇬 Uganda: 9,611
🇬🇪 Georgia: 9,334
🇱🇧 Lebanon: 8,024
🇧🇾 Belarus: 7,828
🇶🇦 Qatar: 7,772
🇵🇦 Panama: 7,647
🇹🇿 Tanzania: 7,467
🇦🇿 Azerbaijan: 7,329
🇬🇭 Ghana: 6,825
🇦🇩 Andorra: 6,707
🇨🇻 Cabo Verde: 6,570
🇦🇬 Antigua and Barbuda: 5,801
🇲🇩 Moldova: 5,763
🇱🇨 Saint Lucia: 5,348
🇲🇰 Macedonia: 5,183
🇪🇹 Ethiopia: 5,020
🇺🇿 Uzbekistan: 4,804
🇰🇼 Kuwait: 4,748
🇯🇴 Jordan: 4,674
🇵🇾 Paraguay: 4,449
🇹🇳 Tunisia: 4,392
🇲🇳 Mongolia: 4,268
🇲🇺 Mauritius: 4,068
🇿🇼 Zimbabwe: 3,963
🇰🇭 Cambodia: 3,835
🇭🇳 Honduras: 3,817
🇸🇹 Sao Tome and Principe: 3,338
🇷🇼 Rwanda: 3,318
🇬🇵 Guadeloupe: 3,268
🇸🇻 El Salvador: 3,267
🇧🇦 Bosnia and Herzegovina: 3,253
🇧🇲 Bermuda: 3,214
🇧🇴 Bolivia: 3,189
🇧🇸 Bahamas: 3,155
🇯🇪 Jersey: 3,132
🇧🇭 Bahrain: 3,108
🇳🇦 Namibia: 3,092
🇱🇾 Libya: 3,064
🇰🇾 Cayman Islands: 3,007
🇴🇲 Oman: 2,954
🇸🇨 Seychelles: 2,870
🇮🇶 Iraq: 2,856
🇰🇬 Kyrgyzstan: 2,788
🇫🇴 Faroe Islands: 2,769
🇻🇮 United States Virgin Islands: 2,712
🇲🇨 Monaco: 2,615
🇲🇸 Montserrat: 2,467
🇹🇹 Trinidad and Tobago: 2,391
🇨🇮 Cote d'Ivoire: 2,240
🇳🇮 Nicaragua: 2,236
🇲🇶 Martinique: 2,236
🇩🇿 Algeria: 2,206
🇦🇽 Åland Islands: 2,194
🇬🇩 Grenada: 2,160
🇦🇫 Afghanistan: 2,100
🇿🇲 Zambia: 2,062
🇲🇻 Maldives: 1,996
🇦🇸 American Samoa: 1,988
🇯🇲 Jamaica: 1,956
🇸🇳 Senegal: 1,933
🇲🇲 Myanmar: 1,871
🇨🇩 Congo: 1,848
🇲🇿 Mozambique: 1,763
🇦🇴 Angola: 1,714
🇧🇧 Barbados: 1,692
🇲🇱 Mali: 1,690
🇾🇹 Mayotte: 1,530
🇬🇾 Guyana: 1,503
🇹🇫 French Southern Territories: 1,488
🇵🇸 Palestine: 1,485
🇧🇼 Botswana: 1,418
🇫🇯 Fiji: 1,388
🇻🇺 Vanuatu: 1,355
🇬🇱 Greenland: 1,303
🇩🇯 Djibouti: 1,269
🇹🇨 Turks and Caicos Islands: 1,263
🇭🇹 Haiti: 1,256
🇲🇬 Madagascar: 1,220
🇬🇸 South Georgia and the South Sandwich Islands: 1,129
🇬🇮 Gibraltar: 1,123
🇦🇼 Aruba: 1,117
🇲🇼 Malawi: 1,000
🇳🇨 New Caledonia: 965
🇵🇬 Papua New Guinea: 927
🇹🇬 Togo: 858
🇸🇷 Suriname: 849
🇹🇯 Tajikistan: 817
🇸🇱 Sierra Leone: 748
🇼🇫 Wallis and Futuna: 740
🇬🇫 French Guiana: 681
🇧🇯 Benin: 666
🇬🇲 Gambia: 629
🇧🇮 Burundi: 626
🇬🇺 Guam: 624
🇸🇾 Syria: 600
🇨🇺 Cuba: 516
🇲🇷 Mauritania: 507
🇱🇷 Liberia: 504
🇨🇰 Cook Islands: 476
🇧🇫 Burkina Faso: 467
🇹🇱 East Timor: 446
🇾🇪 Yemen: 433
🇸🇩 Sudan: 432
🇲🇴 Macao: 413
🇭🇲 Heard Island and McDonald Islands: 409
🇧🇳 Brunei: 391
🇳🇪 Niger: 387
🇸🇿 Swaziland: 363
🇧🇹 Bhutan: 360
🇵🇫 French Polynesia: 332
🇱🇸 Lesotho: 331
🇲🇵 Northern Mariana Islands: 320
🇨🇼 Curacao: 304
🇬🇦 Gabon: 291
🇸🇧 Solomon Islands: 277
🇸🇲 San Marino: 261
🇹🇲 Turkmenistan: 244
🇩🇲 Dominica: 204
🇰🇳 Saint Kitts and Nevis: 186
🇸🇯 Svalbard and Jan Mayen: 154
🇸🇸 South Sudan: 133
🇬🇳 Guinea: 122
🇹🇩 Chad: 114
🇨🇬 Congo: 111
🇳🇫 Norfolk Island: 110
🇺🇲 United States Minor Outlying Islands: 103
🇫🇰 Falkland Islands: 80
🇰🇮 Kiribati: 77
🇲🇭 Marshall Islands: 58
🇵🇳 Pitcairn: 45
🇬🇼 Guinea-Bissau: 44
🇻🇦 Vatican City: 44
🇬🇶 Equatorial Guinea: 44
🇧🇱 Saint Barthélemy: 33
🇳🇷 Nauru: 28
🇧🇶 Bonaire: 22
🇪🇷 Eritrea: 12
🇦🇶 Antarctica: 11
🇨🇫 Central African Republic: 4
🇰🇲 Comoros: 3
🇰🇵 North Korea: 3

Robots.txt Categories

All Robots.txt AI Bot

Excluded from Pie Charts

Robots Disallow

Some Robots Disallow

General robots disallow rule.