LLM scraping is a parasite on the internet. In the actual ecological definition of parasite: they place a burden on other unwitting organisms computer systems, making it harder for the host to survive or carry out their own necessary processes, solely for the parasite’s own benefit while giving nothing to the host in return.

I know there’s an ongoing debate (both in the courts and on social media) about whether AI should have to pay royalties to its training data under copyright law, but I think they should at the very least be paying to use infrastructure while collecting the data, even free data, given that it costs the organisation hosting said data real money and resources to be scraped, and it’s orders of magnitude more money and resources compared to serving that data to individual people.

The case can certainly be made that copying is not theft, but copying is by no means free either, especially when done at the scales LLMs do.

While AI crawlers are a problem I’m also kind of astonished why so many projects don’t use tools like ratelimiters or IP-blocklists. These are pretty simple to setup, cause no/very little additional load and don’t cause collateral damage for legitimate users that just happend to use a different browser.

I have a small site that mirrors hacker news but with dark mode and stuff, and it is getting blasted by bot traffic. All the data is freely available from the official api but they’re scraping my piddling site which runs on an anemic VPS because it looks like user generated content. Bot protection does little to help from my provider. Gonna have to rethink my whole architecture now. Very annoying

In a blogpost called, “AI crawlers need to be more respectful”, they claim that blocking all AI crawlers immediately decreased their traffic by 75%, going from 800GB/day to 200GB/day. This made the project save up around $1500 a month.

“AI” companies are a plague on humanity. From now on, I’m mentally designating them as terrorists.

If you’re wondering if it’s really that bad, have this quote:

GNOME sysadmin, Bart Piotrowski, kindly shared some numbers to let people fully understand the scope of the problem. According to him, in around two hours and a half they received 81k total requests, and out of those only 3% passed Anubi’s proof of work, hinting at 97% of the traffic being bots

And this is just one quote. The article is full of quotes of people all over reporting they can’t focus on their work because either the infra they rely on is constantly down, or because they’re the ones fighting to keep it functional.

This shit is unsustainable. Fuck all of these AI companies.

Great write-up by Niccolò.

I actually agree with the commenter on that post, the lack of quoting and using images is pretty bad, especially for screen-readers (which I use), and not directly linking sources (though they are made clear regardless) is a bit of a pain.

One of my sites was close to being DoS’d by openAI’s crawler along with a couple of other crawlers. Blocking them made the site much faster.

I’d admit the software design offering search suggestions as HTML links didn’t exactly help (this is a FOSS software used for hundreds of sites, and this issue likely applies to similar sites) but their rapid speed of requests turned this from pointless queries into a negligent security threat.

You’d think these centralised LLM search providers would be caching a lot of this stuff, eg perplexity or claude.

Sad there’s no mention of running an Onion Service. That has built-in PoW for DoS protection. So you dont have to be an asshole and block all if Brazil or China or Edge users.

Just use Tor, silly sysadmins