Hey folks
This is a heads up that I will be performing some maintenance and hardware upgrades on our database this Saturday.
We are currently experiencing several spikes throughout the day which cause our database to become overloaded - this results in degraded performance for many users. The spikes are happening due to a combination of continued growth of the database, some expensive periodic scheduled tasks which Lemmy runs, and fluctuating traffic patterns. Some of this can be optimized on the code level in the future, but it seems that the best way to deal with it right now is to add some additional resources to our database server.
I am intending to switch to slightly different hardware in this upgrade, and will be unable to make this switch without downtime, so unfortunately lemm.ee will be unavailable for the duration.
As our database has grown quite a bit, cloning it will most likely take a few hours, so I expect the downtime to last 2-3 hours. Sorry for the inconvenience, I am hopeful that it will be worth it and that this upgrade will significantly reduce some of our recent long page load times!
Edit: upgrade complete!
I have now migrated the lemm.ee database from the original DigitalOcean managed database service to a dedicated server on Hetzner.
As part of this migration, I have also moved all of our Lemmy servers from the DigitalOcean cloud to Hetzner’s Cloud. I always want the servers to be as close as possible to the database, in order to keep latencies low. At the same time, I am very interested in having the ability to dynamically spin up and down servers as needed, so a cloud-type solution is really ideal for that. Fortunately, Hetzner allows connecting cloud servers to their dedicated servers through a private network, so we are able to take advantage of a powerful dedicated server for the database, while retaining the flexibility of the cloud approach for the rest of our servers. I’m really happy with the solution now.
In terms of results, I am already seeing far better page load times and far less resource use on the new hardware, so I think the migration has been a success. I will keep monitoring things and tuning as necessary.
In other words, lemm.ee will be down starting at UTC 2024-01-06T10:00 for 2+ hours. Convert to your local time via https://www.utctime.net/utc-time-zone-converter if you want.
Thank you sunaurus, for the heads up and for the ongoing maintenance you do!
I prefer using timeanddate.com since it has a fixed time URL.
So I wasn’t going crazy, the long loads times were real. Glad to see upgrades coming soon!
Although the long page load times have been real, you still might be going crazy. Definitely a possibility on Lemmy (づ￣ ³￣)づ
My experience has been downgraded since the latest Lemmy update, all these caveats/workarounds are pretty much welcome in order to get this in a stable state again!
Hell yeah!!!
That’s awesome! Are you monitoring personally or is this from somewhere public?
I run a bunch of small websites/tools. I setup UptimeKuma a while back to monitor my own apps. Since it’s just a simple web monitor, I can monitor any http/s URLs. I setup a lemm.ee monitor a few days ago when things were starting to get unstable.
Oh nice, thanks! I think I’m gonna go monitor everything now
When this comment is 2 hours old, I’ll have completed taking a colossal dump.
1 hr old; hang in there champ! 🫡💩
Nothing after 5h. Died doing what he loved RIP Cluck
Why the downvotes? I’m laughing like an idiot over here
Thank you for all the work you do
No worries about the downtime. If that’s what’s needed to get back to 0.18 levels of performance, go for it! Thanks for all the work y’all do.
Np, thanks so much for all you do!
Thank you for your work on lemm.ee! I was actually looking to post somewhere to see if there was some slowness or if it was just me, looks like you answered my question.
It’s flying now! Thanks for the upgrade @sunaurus@lemm.ee.
Really appreciate the advanced heads up. Thanks for being such a awesome instance admin!
Thanks for the update and information. You really set an example for communication and level headedness in decision making. Thanks for providing this space for us.
I just wanted to tell you both, “Good luck,” We’re all counting on you.
-Frank Drebbin, Police Squad
Frank Drebbin
great movie lol
Yeap. Load times are waaaaay better. Many thanks for this!
It’s become extremely responsive compared to before! Good job :)
