Module

The rollback decision

A decision framework — thresholds, blast radius, timing correlation with deploys — when to roll back immediately vs investigate live.

The question nobody wants to answer at 2am

Something is broken. You deployed two hours ago. Do you roll back?

Rolling back is fast and feels safe. It’s also destructive — you lose forward progress, might hit irreversible migrations, and you still don’t know what went wrong.

Not rolling back lets you investigate. It also lets a bad deploy burn users while you grep logs.

You need a decision framework, not a gut feeling.

Roll back immediately if ANY of these are true

Signal Threshold
Error rate > 1% and climbing, OR > 5% at any point
Latency p99 > 2× baseline and still rising
Deploy correlation Metrics changed at the deploy marker, not before
Customer impact Support tickets, status page reports, revenue alerts
Uncertainty + worsening You don’t know why, and it’s getting worse

If multiple boxes are checked, stop reading and roll back. Investigate from the previous version.

Investigate live (15-minute time box) if…

  • Error rate is elevated but stable (not climbing)
  • Only one pod/instance is affected — might be a bad node, not a bad deploy
  • The pattern predates the most recent deploy
  • You have a strong hypothesis and a safe mitigation (scale up, flip feature flag, circuit break)
  • Rolling back is risky (schema migration, one-way data transform)

Set a timer. Fifteen minutes. If you don’t have a hypothesis by then, roll back anyway.

Blast radius: who cares?

Not every incident needs the same urgency.

Blast radius Urgency
Internal admin tool, 3 users Investigate calmly
Checkout / payment path Roll back first, ask questions later
Background job, retriable Maybe scale and monitor
Data corruption risk Stop writes immediately, don’t just roll back

Know which services are in the “roll back without discussion” tier before you go on-call.

Deploy timing: correlation vs causation

The deploy marker is a clue, not a verdict.

Roll back candidate: error rate was 0.05% before deploy, 3% after. Line up is tight.

Not a rollback: error rate was climbing for an hour before deploy. Deploy didn’t cause it — might even be unrelated.

Tricky case: deploy at 02:00, incident at 02:45. Could be a slow leak or a delayed config propagation. Check memory and traffic panels before deciding.

Before you roll back: checklist

  1. Note the target version — What SHA are you rolling back to? Write it down.
  2. Check reversibility — Did this deploy run a migration? Is rollback actually safe?
  3. Tell someone — Slack message counts. #incidents, your team channel, whatever exists.
  4. Watch the rollback — Don’t walk away. Confirm error rate drops after the old version is live.
  5. Preserve evidence — Save logs, screenshot dashboards, note timestamps. You’ll need them for the postmortem.

When NOT to roll back

  • Memory leak — Rolling back doesn’t fix a leak. Restart might, but the leak comes back.
  • Downstream outage — Your code is fine; rolling back changes nothing.
  • Traffic spike — Site is slow because it’s busy, not because deploy broke it.
  • Single bad pod — Cordon and drain the node. Don’t roll back the whole fleet.

The rollback-and-investigate playbook

  1. Roll back to last known good
  2. Confirm metrics recover (5–10 minutes)
  3. Capture logs and dashboard screenshots from the bad version
  4. Write a one-line incident summary: what broke, when, what fixed it
  5. Investigate root cause in the morning, with coffee

You don’t need to understand the bug to stop the bleeding.

Practice the decision

Use the interactive scenarios below. Each describes a situation — pick roll back now, investigate 15 minutes, or other action. Then reveal the reasoning.

Better to practice here than on production.

Rollback decision practice

Errors spike at deploy

Error rate jumps from 0.05% to 4% within 2 minutes of a deploy. p99 latency triples. Memory and traffic look normal.

What do you do?