Hi Beeple!

Here’s a vague version of events :

  • 11PM EST: Lemmy.world got hacked

  • 12:20AM EST: Blahaj.zone got hacked

  • 12:25AM EST: I shut down the server

  • 12:30AM EST: I make announcements to tell people about this

  • 12:45AM EST: I have an idea of what the problem is but there is no fix

  • 2:20AM EST: I go to sleep

  • 8:50AM EST: The server is booted back up, steps are applied to mitigate issues (Rotating JWTs, Clearing DB of the source of vulnerability, deleting custom emoji), UI is updated with the fix, CSP and other security options are applied

  • 11:40AM EST: We start testing things to make sure are working And well, now here we are.

If you have issues logging in or using an app:

  1. Log out if you somehow are still logged in

  2. Clear all cache, site data, etc.

  3. Hard refresh Beehaw using CTRL+F5

  4. Log back in.

If you still have issues, write to us at support@beehaw.org

To be clear : We have not been hacked as far as we know, we were completely unaffected. This was done preemptively.

Oh yeah, in case, you haven’t, this is a good opportunity and reminder to follow us on Mastodon as the communication line was still up despite Beehaw being down : https://hachyderm.io/@beehaw

  • frogman [he/him]@beehaw.orgEnglish
    31·
    1 year ago

    shutting down the server early was best. the nature of open source software is what allows these incidents to be mitigated as quickly as they are. thanks a lot to you guys, and to all of the team at Lemmy who worked to resolve this.

    heroes <3

  • Lee@beehaw.orgEnglish
    13·
    1 year ago

    This is why I am on Beehaw. The Admins really care about the Instance and the content on it.

    That’s why I want to bring attention to the fact, that U can support them. https://opencollective.com/beehaw

    I am not a Admin, Mod or anything else. I just really like Beehaw and support them. And you should too.

  • alehel@beehaw.orgEnglish
    11·
    1 year ago

    Thank you for shutting down rather than “wait and see”! It was the right choice.

  • Hirom@beehaw.orgEnglish
    10·
    1 year ago

    The shutdown is a good call given the circumstances.

    An idea of less-radical preventive action is placing the instance in read-only mode, either as a Lemmy feature, or through reverse proxy settings (eg reply 503 for any POST/PUT/DELETE request). But that’d require some development and/or preparation.

    Doing that on the reserve proxy side would block any user-submitted content and more (logins, searches, …). This would hopefully be efficient at blocking many attack vectors, while still keeping the instance partially online, even if that’s a degraded mode.

    • Lionir [he/him]@beehaw.orgOPMEnglish
      7·
      1 year ago

      Note that if this were a Lemmy feature, if we had been infected, an admin could’ve gotten hacked and as a result, disabled that feature. I’m not really sure what can be done to make Beehaw foolproof. That said, the UI has since been hardened by CSP headers so this type of attack should no longer be possible.

    • interolivary@beehaw.orgEnglish
      7·
      1 year ago

      Would read-only mode help with XSS exploits though, like this particular one? Since the “damage was already done” by the time anybody noticed, wouldn’t putting the site in read-only mode still have kept serving up the XSS payload? It’d stop “infected” people from making any state mutations on Lemmy, but eg. data exliftration would still happen

  • fracture [he/him] @beehaw.orgEnglish
    9·
    1 year ago

    huge Ws, excellent work

    also, thanks for the Mastodon link, i wasn’t sure where to check on beehaw status during the outage

  • emma@beehaw.orgEnglish
    8·
    1 year ago

    morning thought: I’ve definitely joined the right instance. (also the start from the assumption of good faith guidelines linked to in Gaywallet’s recent post)

  • YourHeroes4Ghosts@beehaw.orgEnglish
    7·
    1 year ago

    Thank you for all you do, from what I was hearing I was in no way expecting you to have the site back up within 12 hours. Many kudos.

  • pwacata@beehaw.orgEnglish
    7·
    1 year ago

    Awesome response, and a great succinct postmortem. Thanks for doing what you do!

  • Pepper@beehaw.orgEnglish
    6·
    1 year ago

    12:30AM EST: I make announcements to tell people about this

    I think it’d be beneficial to have more backup lines of communication for announcements than just Mastodon.

  • GameGod@beehaw.orgEnglish
    5·
    1 year ago

    Content-Security-Policy will really help save your bacon beans and protect against XSS. Hopefully the Lemmy devs can apply a super strict policy to help. IMHO it’s a must for any site with user generated content.