This post explains the incident well but long story short some hackers were able to compromise user and admin accounts through stolen authentication cookies on some instances.
Before things were clear on exactly how this happened, we pulled the plug on our instance to mitigate the risk. We probabaly should have hastily wrote an announcment post before doing that but the situation seemed critical so we didn’t want to waste any time.
Few hours later, people were able to figure out the issue and promptly fix it. Turns out this vulnerabilty could only be exploited if an instance had custom emojis which thankfully ours didn’t, so users using this instance should be safe from the hack. lemmy.fmhy.ml now runs on v18.2rc which has fixed this vuln to be extra secure.
Sorry for the downtime and we will try to communicate the problem better in the future.
P.S. After somone mentioned exploding-heads on a recent post and why we are still federated with it, we took some time to view it carefully and decided it’s an instance that systematically breaks our rules and to defederate with it. We will shortly post our defederation policy soon to give a better idea on how we will decide on which instance to defederate from moving forward.
I also want to mention this happened at 2 or 3 AM EST, which only had 2 admins on, me and the hoster. I should’ve made a post, but given the severity, lack of info, and lack of staff members, we both decided to shut it down immediately until it was patched. I personally apologize for the downtime but I hope it’s understandable why we did it this way. In case something like this happens again, I’ll post it in the FMHY Divolt server.
Sometimes you unfortunately have to choose between safety and communication. In this case you chose safety, which even if it didn’t end up being warranted was the right call IMO. Thanks for putting in all the effort.
Not only understandable, but I’m glad that’s how you guys handled it. Leaving it up seems irresponsible to me, although I’m not blaming any admins who did.
I’m glad you guys had the time to do it at all. Also shows this layered universe stuff is not that bad, so much harder to do harm.
Website going down was confusing but under the circumstances best course of action.
On the bright side apps still worked :)
Awesome to have the site back up and running.