Robots.txt is a file that is is accessible as part of an http request. It’s a backend configuration file that sets rules for what automatically running web crawlers are allowed. It can set both who is and who isn’t allowed. Google is usually the most widely allowed domain for bots just because their crawler is how they find websites for search results. But it’s basically the honor system. You could write a scraper today that goes to websites that it is being told it doesn’t have permission to view this page, ignore it, and still get the information
I do not think it is even part of the HTTP protocol I think it’s just a pseudo add-on. It’s barely even a protocol it’s basically just a page that bots can look at with no really pre-agreed syntax.
If you want to make a bot that doesn’t respect robots.txt you don’t even need to do anything complicated, you just need to not include the requirement to look at the page. It’s not enforceable at all.
TIL that robots.txt is a thing
what is it?
robots.txt is a file available in a standard location on web servers (example.com/robots.txt) which set guidelines for how scrapers should behave.
That can range from saying “don’t bother indexing the login page” to “Googlebot go away”.
IT’s also in the first paragraph of the article.
Robots.txt is a file that is is accessible as part of an http request. It’s a backend configuration file that sets rules for what automatically running web crawlers are allowed. It can set both who is and who isn’t allowed. Google is usually the most widely allowed domain for bots just because their crawler is how they find websites for search results. But it’s basically the honor system. You could write a scraper today that goes to websites that it is being told it doesn’t have permission to view this page, ignore it, and still get the information
I do not think it is even part of the HTTP protocol I think it’s just a pseudo add-on. It’s barely even a protocol it’s basically just a page that bots can look at with no really pre-agreed syntax.
If you want to make a bot that doesn’t respect robots.txt you don’t even need to do anything complicated, you just need to not include the requirement to look at the page. It’s not enforceable at all.
just wait until you hear about humans.txt, it really exitst here