A Small-Team Rollback Plan You Can Actually Rehearse

Executive summary

Almost every small team has a rollback plan. Very few have ever run one. The plan sits in a document, full of sensible steps, and it is treated as finished the moment it is written. This post explains why that document is not really a plan until it has been rehearsed, and offers a lightweight rehearsal format that a one- or two-person team can run in a single 30-minute session with no staging, no downtime window, and no special tooling.

The plan that was never run

A rollback plan fails quietly. It looks complete on paper: revert the change, restore the previous version, verify, communicate. Then a real bad release happens at 6 PM on a Friday, and the team discovers what the document never said. The revert command assumes a migration that was already applied and will not roll forward again. The "previous version" pointer was overwritten by an intermediate hotfix nobody recorded. The verification step requires data that only exists in an environment nobody can reach from the incident call. Nobody is discovering anything new about engineering in that moment. They are discovering the gap between a document and a practiced capability.

That is the deckplate gap in release operations: the operator standing in the incident has one job — restore service — but the tool they were handed (the rollback plan) has never been tested under the conditions it exists for. A plan that has never been rehearsed is not a plan. It is a hope with formatting.

Why rehearsal matters more for small teams

Large organizations rehearse recoveries because they can afford to: they have change advisory boards, staging environments that mirror production, and on-call rotations that can absorb a drill. Small teams skip rehearsal for exactly the opposite reason — there is no slack time, no spare environment, and no second shift to cover the risk.

But that logic runs backwards. A small team has less capacity to absorb a botched rollback, not more. When the rollback itself fails — because of a step that only works in theory — a two-person team has no bench to fall back on, no war room to assemble, and usually a customer waiting. The less margin a team has, the more valuable the rehearsal becomes, because the rehearsal is the cheapest possible moment to find out the plan is wrong.

The 30-minute rehearsal format

The goal is not a full disaster exercise. It is a walkthrough that surfaces broken steps while the stakes are zero. The format has four parts:

1. Pick the last real release (10 minutes). Do not invent a scenario. Take the most recent change the team actually shipped and walk the rollback plan against it, out loud, step by step. Using a real release matters because it forces the plan to confront real artifacts: the actual migration, the actual configuration change, the actual version tags that exist or do not.

2. Touch every command before speaking it (10 minutes). For each step in the plan, answer one question: "When I run this today, exactly what happens?" Look up the actual version number. Open the actual migration file. Check whether the artifact the plan references still exists. The most common finding is a plan that references a previous version by a convention the team stopped following three months ago.

3. Write down every "huh" (5 minutes). Any moment where someone pauses, guesses, or says "I think it works like this" is a defect in the plan. Do not fix it on the spot and do not argue about it — record it verbatim. Those moments are the entire output of the rehearsal. A rehearsal that produces zero notes almost certainly was not honest.

4. Fix the plan, not the memory (5 minutes). Close the session by turning each recorded note into a concrete plan edit: add the missing precondition, delete the step that no longer applies, add the one command someone had to look up. The plan should get shorter and more specific every rehearsal, not longer. Steps that survive three rehearsals without ever being questioned are candidates for deletion, not celebration.

What a rehearsal is not

A rehearsal is not a test environment exercise, and a small team does not need one to benefit. Walking the steps and verifying that each referenced artifact actually exists catches the majority of plan rot — the slow drift between what the document says and what the system now does. If the team does have a disposable environment, running one full rollback there once a quarter is a genuine upgrade. But waiting for that environment is the most common way rehearsal never happens at all.

A rehearsal is also not a blame session. If the plan is wrong, the plan was wrong — the person who wrote it made a reasonable assumption that the system has since outgrown. The rehearsal exists to find those moments when nothing is on fire.

Keeping it honest over time

Two habits keep a rehearsal from becoming ceremony. First, rehearse after every meaningful release while the artifacts are fresh, not on a calendar cadence the team will quietly stop honoring. Second, the person who shipped the change should not be the one who runs the rehearsal — anyone who can read the plan and follow it literally will find the gaps fastest. The measure of a good rollback plan is that the newest person on the team can execute it, which means the newest person on the team is exactly who should be reading it out loud.

The deckplate test is simple: if a bad deploy happened tonight, could the team restore service using only the document and no tribal knowledge? If the answer requires saying someone's name, the rehearsal has work left to do. Close the gap while the release is boring, because the whole point of a rehearsed rollback is that the day it is needed, nobody has to think.

> Independence and use disclaimer: This article is independent educational content from BlueVioletApps, LLC. It is not official government policy, legal advice, or a compliance determination. Readers should validate requirements against the authoritative source and their own operating context.