Back in March and April, there were some severe outages on Magic Pages. Some led to a downtime of 2-3 hours. That was everything but good.
Maybe you remember, I took some time to explore ways to avoid that in the future:
The solution back then was quite simple: Replication.
Instead of one server doing all the work, every single service should be replicated not once, but twice. So…three servers doing all the work – three times.
Yes, that means, the Magic Pages infrastructure got quite a bit more expensive. But today I got the proof that it was all worth it 🙌
Around an hour ago I got an email from my upstream server provider:
One of your servers is currently unavailable due to an outage in our data center. We are aware of this and are actively working on a solution.
Well, no solution in sight yet. But since there were still two other servers running the same services, there was just a tiny little downtime of a few minutes. I bet most people didn't even notice.
All that needed to happen was for an IP address to switch over to one of the other servers. Easy-peasy, automated lemon squeezy. I literally just sat there, put a quick update into Magic Pages's status page, waited for a few minutes, and resolved the issue on the status page.
Now, I am still waiting the server provider to solve the underlying issue. But I know: even if a second server goes down, there is still one more holding all the necessary strings.
It's a great feeling when a plan pans out.