Reddit CEO Steve Huffman is standing by Reddit’s decision to block companies from scraping the site without an AI agreement.

Last week, 404 Media noticed that search engines that weren’t Google were no longer listing recent Reddit posts in results. This was because Reddit updated its Robots Exclusion Protocol (txt file) to block bots from scraping the site. The file reads: “Reddit believes in an open Internet, but not the misuse of public content.” Since the news broke, OpenAI announced SearchGPT, which can show recent Reddit results.

The change came a year after Reddit began its efforts to stop free scraping, which Huffman initially framed as an attempt to stop AI companies from making money off of Reddit content for free. This endeavor also led Reddit to begin charging for API access (the high pricing led to many third-party Reddit apps closing).

In an interview with The Verge today, Huffman stood by the changes that led to Google temporarily being the only search engine able to show recent discussions from Reddit. Reddit and Google signed an AI training deal in February said to be worth $60 million a year. It’s unclear how much Reddit’s OpenAI deal is worth.

Huffman said:

Without these agreements, we don’t have any say or knowledge of how our data is displayed and what it’s used for, which has put us in a position now of blocking folks who haven’t been willing to come to terms with how we’d like our data to be used or not used.

“[It’s been] a real pain in the ass to block these companies,” Huffman told The Verge.

  • Sordid@lemmy.worldEnglish
    66·
    4 months ago

    The enshittification cycle:

    Phase one, attract users by providing a good service.
    Phase two, once the users are locked in, squeeze them for all they’re worth by selling them to business customers (advertisers and/or data buyers).
    Phase three, once the business customers are locked in, squeeze them for all they’re worth by threatening to deny them access to the users on whom they now depend.

    Spez seems to think Reddit has the pull to make phase 3 happen. I rather doubt it, but we’ll see.

    • lemmyvore@feddit.nlEnglish
      13·
      4 months ago

      If he really had balls he’d restrict access to the site and improve the built-in search engine.

      If reddit’s own search worked well nobody would care. Engines like DDG even have bang codes that send you to a site’s own engine. So instead of having to add “site:reddit.com” to the search on DDG I’d just add “r!” and it would end up being the same thing. IF the internal search didn’t suck.

    • _haha_oh_wow_@sh.itjust.worksEnglish
      9·
      4 months ago

      Yeah, as soon as the API thing happened I switched to Lemmy for mobile browsing and like it more than Reddit (Connect is pretty good, but even the mobile browser site is solid).

      The more they squeeze, the more popular alternatives like Lemmy, Kbin/Mbin, Tildes, etc. will become.

    • Boozilla@lemmy.worldEnglish
      6·
      4 months ago

      My guess is that phase three will work for a while. But I think you’re right that eventually they are going to drive that thing into the ground. Because it’s never enough pure profit for rent-seeking scum, and there is no lower limit to the abuse they’ll inflict on their content creators (who they call users but think of as products).