Don’t Be a Hero: Why Safe Migrations Beat Smart Hacks
A real-world lesson in why confidence isn’t a strategy.
On a rainy Friday, your product owner asks you to do a complex database migration. After reading the requirements and acceptance criteria of the Jira ticket, you think: “F**k! This cannot be done easily!”
The problem is that the change is so complex, it cannot be done gracefully. You have to stop the backend, make a dump of the database, delete the old database, create a new database, and then run a complex migration script. Finally, you have to make sure that the backend works with the new database.
If anything breaks during the process, you have to roll everything back manually. Hopefully, that’s even possible.
What if the backend doesn’t work with the new database? Then you have to make the hard choice: spend all the effort and roll it back, or do ad-hoc changes to the backend in the middle of the night while you’re tired.
Even if this process works, you’ll be extremely exhausted — and this will have a negative impact on your private life during your free weekend days.
And if it doesn’t work? You’re screwed! Maybe you can summon a few colleagues and spend the rest of the weekend fixing it. Maybe you can’t fix it — and on Monday, you’ll be called by some angry customers. If your company is big, you’ll even read about it in the news.
The root cause of this is hubris: you think you’re a 10X developer who can do things with an all-at-once approach. Your genius, you believe, allows you to take a shortcut and avoid the rough road of a blue-green deployment with zero downtime.
What would be the better approach?
Provision a new database in parallel with the existing one. Write to both databases simultaneously. Use the old schema for the old one and the new schema for the new one. Run a migration script that reads historical data from the old database, converts it to the new schema, and writes it to the new database. After a grace period, start reading from both databases and compare the results. If, after another grace period, the results are always the same, you can start using only the new database and archive the old one.
So what’s the difference between the approaches?
The first is an all-at-once approach that relies on your genius as a 10X developer to take a leap of faith into the unknown. The second is a professional software development approach that does things incrementally, safely, and predictably.
How do I know this? I’ve taken the leap of faith countless times. And in 99% of cases, I regretted it.
Do you think confidence in tech is often mistaken for competence — especially when taking all-or-nothing risks?
Is it ever justifiable to go with an all-at-once approach in software — or is it always a sign of hubris?