I am honestly kinda curious how exactly you manage your servers and keep them up to date, I feel like there is a linux patch every week or so, which would usually require a reboot. Do you all deploy live patching, how you become aware of critical stuff in your otherwise busy lives, RSS?
An LTS distro like Debian and Ubuntu doesn’t update too frequently. I’ve never tried livepatching. I install updates on the weekends and reboot only if necessary. Downtime is usually about a minute and my uptime monitors don’t usually catch it.
My lemmy.fan instance died. Something broke with federation and I’ve never been able to get it running again, even with a new database and a new subdomain. I gave up for now, at least until better error checking and recovery is implemented.
Perform automatic updates and reboot when necessary.
If one is serious about hosting this, it’s best to isolate the services. One container or VM or reach service, with (probably) physical hosts for the DBs.
Schema change is more involved, but backup then update. If you have read only db, it should sync the changes when reconnected.
Realistically, federated data will be re-sent if the recipient doesn’t respond, so a few minutes of downtime is not the end of the world. At least that’s how mastodon works - not sure about Lemmy but I’m presuming it operates in a similar fashion.
Couldn’t auto updates break things if they arnt tested?
They should be tested upstream of you, assuming you aren’t using customized (eg roll your own) versions of any of the ancillary software (php, pgsql, redis, etc). Generally configs are either merged or not adopted, and you can restrict version upgrades to non major releases, if there’s chances of breakages between them (eg moving from pgsql 9 to 10, etc).
Keep instance small, with all users in the same timezone. Use NixOS, let it update everynight automatically and safely. It’s good enough for a small service, downtime is mostly when people are sleeping.
