"Nepenthes generates random links that always point back to itself - the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself."

A pseudonymous coder has created and released an open source “tar pit” to indefinitely trap AI training web crawlers in an infinitely, randomly-generating series of pages to waste their time and computing power. The program, called Nepenthes after the genus of carnivorous pitcher plants which trap and consume their prey, can be deployed by webpage owners to protect their own content from being scraped or can be deployed “offensively” as a honeypot trap to waste AI companies’ resources.

“It’s less like flypaper and more an infinite maze holding a minotaur, except the crawler is the minotaur that cannot get out. The typical web crawler doesn’t appear to have a lot of logic. It downloads a URL, and if it sees links to other URLs, it downloads those too. Nepenthes generates random links that always point back to itself - the crawler downloads those new links. Nepenthes happily just returns more and more lists of links pointing back to itself,” Aaron B, the creator of Nepenthes, told 404 Media.

  • patrick@lemmy.bestiver.seEnglish
    77·
    7 hours ago

    This showed up on HN recently. Several people who wrote web crawlers pointed out that this won’t even come close to working except on terribly written crawlers. Most just limit the number of pages crawled per domain based on popularity of the domain. So they’ll index all of Wikipedia but they definitely won’t crawl all 1 million pages of your unranked website expecting to find quality content.

  • renzev@lemmy.worldEnglish
    18·
    6 hours ago

    This reminds me of that one time a guy figured out how to make “gzip bombs” that bricked automated vuln scanners.

    • GreenKnight23@lemmy.worldEnglish
      16·
      6 hours ago

      I had a scanner that was relentless smashing a server at work and configured one of those.

      evidently it was one of our customers. it filled their storage up and increased their storage costs by like 500%.

      they complained that we purposefully sabotaged their scans. when I told them I spent two weeks tracking down and confirmed their scan were causing performance issues on our infrastructure I had every right to protect the experience of all our users.

      I also reminded them they were effectively DDOSing our services which I could file a request to investigate with cyber crimes division of the FBI.

      they shut up, paid their bill, and didn’t renew their measly $2k mrr account with us when their contract ended.

      bitch ass small companies are always the biggest pita.

  • Jordan117@lemmy.worldEnglish
    41·
    7 hours ago

    More accurately, it traps any web crawler, including regular search engines and benign projects like the Internet Archive. This should not be used without an allowlist for known trusted crawlers at least.

    • Treczoks@lemmy.worldEnglish
      11·
      4 hours ago

      Just put the trap in a space roped off by robots.txt - any crawler that ventures there deserves being roasted.

    • DreamlandLividity@lemmy.worldEnglish
      2·
      2 hours ago

      More accurately, it traps any web crawler

      More accurately, it does not trap any competent crawlers, which have per domain limits on how many pages they crawl.

  • FaceDeer@fedia.io
    37·
    9 hours ago

    This sort of thing has been a strategy for dealing with unwanted web crawlers since web crawlers were a thing. It’s an arms race, though; crawlers do things to detect these “mazes” and so the maze-makers keep needing to up their game as well.

    As we enter an age where AI is effectively passing the Turing Test, it’s going to be tricky making traps for them that don’t also ensnare the actual humans you’re trying to serve pages to.

    • doylio@lemmy.caEnglish
      49·
      11 hours ago

      Picking words at random from a dictionary would not be very compute intensive, the content doesn’t need to be sensical

      • BrianTheeBiscuiteer@lemmy.worldEnglish
        7·
        10 hours ago

        Yes, the scraper is going to mindlessly gobble up information. At best they’d expend more resources later to try and determine the value of the content but how do you do that really? Mostly I think they’re hoping the good will outweigh the bad.

        • tempest@lemmy.caEnglish
          3·
          8 hours ago

          It honestly depends. There are random drive by scrapers that will just do what they can, usually within a specific budget for a domain and move on. If you have something specific though that someone wants you end up in an arms race pretty quickly as they will pay attention and tune their crawler daily.

      • Jack@slrpnk.netEnglish
        2·
        8 hours ago

        I was thinking exactly that, generating something like lorem ipsum to cost both time, compute and storage for the crawler.

        It will be more complex and require more resources tho.

    • meyotch@slrpnk.netEnglish
      19·
      11 hours ago

      I would think yes. The compute needed to make a hyperlink maze is low, compared to the AI processing of the random content, which costs nearly nothing to make, but still costs the same to process as genuine content.

      Am I missing something?

      • RaoulDook@lemmy.worldEnglish
        8·
        10 hours ago

        I’m wondering about the cost to the server’s resources / bandwidth to serve up unlimited random junk also.

        But kudos to the developer for making this anyway

  • count_dongulus@lemmy.worldEnglish
    33·
    11 hours ago

    This won’t work against commercial crawlers. They check page contents with something similar to a simhash and don’t recrawl these pages. They also have limiters like for depth to avoid getting stuck in circular links.

    You could generate random content for each new page, but you’ll still eventually hit the depth limit. There are probably other rules related to content quality to limit crawling too.

    • meyotch@slrpnk.netEnglish
      31·
      11 hours ago

      True, this is an arms race situation after all. The real benefit of this is creating garbage training data that makes garbage models. So it’s not just increasing the cost of crawling, it increases the cost of stealing everybody’s shit because you need extra data quality checks. Poisoning the well.

        • Thrashy@lemmy.worldEnglish
          8·
          10 hours ago

          Say it with me now: model collapse! I think this approach is especially insidious in that rather than dumping obvious nonsense into the training corpus that can then be scrubbed, it pushes the downstream LLM invisibly towards spontaneously imploding.

        • meyotch@slrpnk.netEnglish
          2·
          9 hours ago

          Exactly! That’s ideal because LLM or simple pattern matching can’t be used to easily winnow out random strings. If it’s sensible language but the usual LLM hallucinations, then you need humans to curate your data. Fuck you, Sam Altman.

  • Nougat@fedia.io
    10·
    11 hours ago

    The modern equivalent of making a page that loads in two frames, left and right, which each load in two frames, top and bottom, which each load in two frames, left and right …

    As I recall, this was five lines of HTML.

    • palordrolap@fedia.io
      2·
      10 hours ago

      I remember making one of those.

      It had a faux URL bar at the top of both the left and right frame and used a little JavaScript to turn each side into its own functioning browser window. This was long before browser tabs were a mainstream thing. At the time, relatively small 4:3 or 5:4 ratio monitors were the norm, and I couldn’t bear the skinny page rendering at each side, so I gave it up as a failed experiment.

      And yes I did open it inside itself. The loaded pages were even more ridiculously skinny.

      • Nougat@fedia.io
        2·
        10 hours ago

        When I did my five lines, recursively opening frames inside frames ad infinitum, it would crash browsers of the time in a matter of twenty seconds.