I was in my first year out, at a marketing company of the fast kind—everything due yesterday, nobody with time to walk the new guy through the systems. I had been handed a client’s contest submissions to review. I was still learning the site.
I ran a delete against roughly twelve thousand records, looked at the result, and understood with total clarity that I had just destroyed production data for a paying client.
I did not calmly assess the situation. I sat there and seriously considered picking up my bag and walking out of the building, because there was no version of the next conversation I was going to survive.
Then I checked again. It was QA. I had confused the two environments. Production was untouched, the client’s submissions were fine, and nobody ever knew it happened.
The relief was physical. The lesson landed exactly as hard as if I had actually done it.
What it actually taught me
The obvious lesson is “check which environment you are in,” and that is the least interesting thing I took from it. Everyone already knows that. I knew it that morning.
The useful lesson is that I had been handed a system where one ambiguous moment and one confident keystroke could erase a client’s data, and nothing in it would have stopped me. No confirmation. No visual difference between the two environments. No undo. The system trusted me completely, and I was a junior developer four months into his first real job.
That is a design decision. Someone made it, probably by not thinking about it.
Since that day I have never written an outright hard delete. Everything soft deletes or archives. But the habit generalised well past deletion, and that is the part worth describing to anyone deciding whether to hand me their system.
Build for the tired person
The engineer touching your production system at 4pm on a Friday is not operating at full capacity. Neither is the support rep clearing a queue, or the founder poking at the admin panel to answer a customer question. Systems that only behave when everyone is sharp are systems that are going to have a bad quarter.
So I build in the assumption of human error, structurally:
- Destructive actions are reversible by default, or they ask a real question—not an “Are you sure?” that everyone clicks through on reflex.
- Environments look obviously different. Different colour, a banner, something impossible to miss when you are moving fast.
- Admin tools get the same care as customer-facing ones, because that is where the sharpest edges live and the least design attention usually goes.
- The dangerous path is never the convenient one.
None of this is exotic. It is mostly a willingness to spend a little time on the failure case at the moment you are building the success case, which is the only moment it is cheap.
The part I got wrong for years
I want to be honest about the limits of my own rule, because “never hard delete” sounds like a complete answer and is not one.
A deleted_at column protects you from the mistake I nearly made. It does nothing about the harder question, which is what should genuinely disappear, and from where. That row still exists in the read replica, last night’s backup, the search index, the analytics warehouse, and a CSV somebody scheduled to email themselves two years ago.
Soft delete solved my fear. It did not solve deletion. When “delete my data” arrived as a legal obligation with a deadline rather than a feature request, plenty of teams discovered that a flag on one copy in one system was not the same thing as the data being gone.
So the rule grew a second half: soft delete by default, and decide up front what each piece of data actually is. Reversible and user-owned. Shared with other people, and therefore not the departing user’s to erase. Financial or historical, which is an archive with a retention policy and should stop being called deletion. Or genuinely required to vanish, in which case you enumerate every copy and handle them, deliberately.
Why I tell this story
Nearly twenty years on, it is still the clearest example I have of how experience actually accumulates in this work. Not through the disasters. Through the near misses that teach you what the disaster would have cost.
I confused two environments, felt the full weight of a consequence I had not caused, and permanently changed how I build software on the strength of a mistake that cost the company nothing.
The lesson did not require the catastrophe. It only required believing, for about ninety seconds, that the catastrophe was real.