Cheat Sheet

Quick reference for 2am. Print this. Tape it to your monitor. Ignore it until you need it.

The 4-panel check

Open Grafana (or equivalent). Check these four panels first, in this order. Read them together — one panel lying is common; four panels lying is rare.

  1. Error rate — Is something failing? Spikes here mean users are hitting errors. Start here.
  2. Latency p99 — Are requests slow? High p99 with flat error rate often means timeouts or downstream slowness.
  3. Resource usage — CPU, memory, disk. Staircase memory = leak. CPU pegged = saturation or infinite loop.
  4. Request volume — Is traffic actually up? Decouples "site is busy" from "something is broken."

Rollback decision framework

Roll back immediately if any of these are true. Otherwise, investigate live (with a time box).

Roll back now

  • Error rate > 1% and climbing (or > 5% at any point)
  • p99 latency > 2× baseline and still rising
  • Deploy marker lines up with the start of the incident
  • Customer-facing impact confirmed (support tickets, status page)
  • You're not sure what's wrong and it's getting worse

Investigate live (15 min max)

  • Error rate elevated but stable (not climbing)
  • Single pod/instance affected — might be a bad node
  • Pattern predates the most recent deploy
  • You have a strong hypothesis and a safe mitigation (scale up, circuit break)

Before you roll back

  • Note the deploy SHA / version you're rolling back to
  • Check if rollback is actually possible (migrations, schema changes)
  • Tell someone — even if it's just a Slack message

Memory leak quick check

  • RSS staircase + flat traffic = leak
  • RSS rises with traffic = probably fine
  • Deploy marker ≠ cause (check if pattern started before deploy)
  • Don't restart until you've captured a heap dump (if safe)

Log triage

  • Start narrow: last 15 min, ERROR level, alerted service
  • Find top error message by count — that's usually the symptom
  • Copy request_id from one line, search across all services
  • Note first error timestamp — correlate with deploys and dashboards
  • Grep context: grep -B3 -A3 "pattern" app.log

Kibana / KQL essentials

  • Confirm index pattern = correct environment (prod, not staging)
  • Use absolute time range once you find incident start
  • service:checkout and level:error
  • message:"connection refused"
  • level:error and not message:"health check"
  • Zero results? Check index pattern and timezone before rewriting query

Baseline checklist

  • Is this metric abnormal for this hour and this day?
  • Compare against last 7 days at the same time — not a global threshold
  • Document normal ranges: error rate, p99, memory floor, traffic
  • Known noise: health check errors on restart, scheduled batch jobs
  • One instance vs whole fleet — bad node ≠ bad deploy