Blocks some but not all robots from indexing the website.
https://robotstxt.org
Robots.txt
Get access to data on 60,830,604 websites that are Some Robots Disallow Customers. We know of 39,138,226 live websites using Some Robots Disallow and an additional 21,692,378 sites that used Some Robots Disallow historically and 596,988 websites in the United States.
Get a list of 60,830,604 websites using Some Robots Disallow which includes location information, hosting data, contact details, 39,138,226 currently live websites and an additional 1,058,844 domains that redirect to sites in this list. 21,692,378 sites that used this technology previouslyand 596,988 websites in the United States currently using Some Robots Disallow.
Countries
Financial
Group
Region