They also don’t give a single flying fuck about robots.txt …
If you try to rate-limit them, they’ll just switch to other IPs all the time. If you try to block them by User Agent string, they’ll just switch to a non-bot UA string (no, really). This is literally a DDoS on the entire internet.
It then become a game of whac a mole with big tech 😓
The more infuriating for me is that it’s done by the big names, and not some random startup.
Edit: Now that I think about it, this doesn’t prove it is done by Google or Amazon: it can be someone using random popular user agents
The only simple possibles ways are:
From the article, they try to bypass all of them:
It then become a game of whac a mole with big tech 😓
The more infuriating for me is that it’s done by the big names, and not some random startup.Edit: Now that I think about it, this doesn’t prove it is done by Google or Amazon: it can be someone using random popular user agentsI do believe there’s blocklists for their IPs out there, that should mitigate things a little