Blocks some but not all robots from indexing the website.
https://robotstxt.org
Robots.txt
Get access to data on 65,040,337 websites Some Robots Disallow Customers. We know of 39,138,226 live websites using Some Robots Disallow and an additional 25,902,111 sites that used Some Robots Disallow historically and 1,044,687 websites in the United States.
Get a list of 65,040,337 websites using Some Robots Disallow which includes location information, hosting data, contact details, 39,138,226 currently live websites and an additional 363,066 domains that redirect to sites in this list. 25,902,111 sites that used this technology previouslyand 1,044,687 websites in the United States currently using Some Robots Disallow.
Countries
Financial
Group
Region