Server maintenance | ani.social
status.ani.social
Welcome to ani.social status page for real-time and historical data on system performance.

It’s been a while since I’ve made these type of posts but there’ll be a server maintenance that will (hopefully) address the recent Lemmy-UI problems.

edit: I’m going to cancel this because the problem has been solved by blocking excessive crawling/scraping bots. No more crashing and zombie processes.

  • hitagi@ani.socialOPMEnglish
    10·
    23 days ago

    Update: I’ll postpone this for a week as it seems like the issue comes from a Chinese datacenter in Singapore that’s scraping the site very aggressively.

    I’ll block the IP range and see if that solves the problem.

    • wjs018@ani.socialEnglish
      5·
      23 days ago

      Damn AI scraper bots (probably)! Thanks for the backend work!

      Out of curiosity, are they just hitting the lemmy-ui frontend and not hitting the others?

      • hitagi@ani.socialOPMEnglish
        6·
        23 days ago

        They only seem to be scraping the lemmy-ui frontend and not the others. I’m actually not 100% sure if it’s scraping but this is what it looks like:

        All from the same IP range in Singapore. A huge spike in requests (up to almost 5k per minute) matches the time when Lemmy-UI crashed last night.

        I captcha’d the ASN (it’s Alibaba btw) for now (I don’t know if that’s okay to do, hopefully I’m not blocking real people) but perhaps there are better solutions like ratelimiting.

        Anyway, let me know if you’re still having trouble accessing the site. The filter should only cover that particular ASN for now.

        • SatouKazuma@ani.socialEnglish
          2·
          22 days ago

          Whoa. Was there anything else going down in that time frame? I’d be interested to have some profile on the attacker. Do you have any information about where the majority of ani.social’s traffic came from before that?

          • hitagi@ani.socialOPMEnglish
            5·
            22 days ago

            Nothing else went down. After blocking the ASN, I found out that they’ve been scraping the site from Hong Kong too but not as aggressively with Singapore. Or rather, the Hong Kong servers are more consistent but not as frequent bursty (I don’t know how best to describe it lol). The only info I have is the ASN which points to the Alibaba Group. It also doesn’t disclose what it is other than “Microsoft Edge Windows” which is not very nice in my opinion.

            Cloudflare only gives me logs for the past 24 hours but most of ani.social’s traffic comes from other instances federating in (unsurprisingly, Lemmy.World is #1).

            Some graphs if you're interested:

            The green line is Singapore. This happened a few hours after the crash the other night.

            This is a closer look of the spike. The lines are 5 of their IP addresses. You can see that they’re dormant then suddenly they start hammering the instance with requests.

            These are the top countries for the past 6 hours that use GET after filtering out Alibaba. United States is high because they have a lot of web crawlers/scrapers like Open AI (at least they’re honest) but I might block the AI ones too.

      • hitagi@ani.socialOPMEnglish
        3·
        20 days ago

        Thanks! I’ve used this before but decided not to set it up for this server because of Fediverse things. I’m afraid I might break federation with it. But I’ll see if I can work around that.

    • hitagi@ani.socialOPMEnglish
      2·
      16 days ago

      That’s actually really cool. Maybe I should put something up like this too. I’ve been curious about our stats lately

  • wjs018@ani.socialEnglish
    2·
    23 days ago

    Update after trying to use the site for a bit:

    It seems like something got messed up. My mobile apps can’t connect to ani.social any longer and the alternative frontends are throwing 500 errors. Meanwhile, in Firefox, I am getting prompted occasionally by cloudflare to verify that I am a human.

    I am guessing that some cloudflare setting is causing it to block certain consumers of the api. Federation still seems to be working, so ActivityPub is alive and well. Similarly, I was able to connect via @rikka@ani.social, so she should still be ok. However, I am getting 403’s most other ways to connect to the site.

    • hitagi@ani.socialOPMEnglish
      3·
      23 days ago

      My bad. I think I messed up somewhere but I’ll turn off the filter I made in the meantime. I’ll figure it out again when I get home from class.