2026-09-17 (No issues)
Current Status (1430 AEST)
1430AEST: And MNZ is also back up
1330AEST: MAU is back up, MNZ is being recovered now that I have proven the recovery process works.
Details
1200AEST: Well. This is bad. Both mastodon.au and mastodon.nz had their database (postgresql) on the same storage server.
That storage server has lost TWO drives. Even more annoyingly, I knew that SOMETHING had lost a drive and needed to be replaced, but I couldn’t remember what.
It did, of course, bite me in the arse.
I am restoring and moving things around to get them back up and working. I was trying to leave them up, but it wasn’t worth the heartache.
They are both down for the next few hours (probably until around 3-4pm AEST).
1330pm (AEST)
Mastodon AU is back up, proving the recovery concept. MNZ is next. I’ve decided that the storage server itself is inherently unreliable, so I’m just going to get it de-racked, as it has AT LEAST three failed drives in it now.
