Grafana essentials
The 4 panels to check first — error rate, latency p99, resource usage, request volume — and how to read them together.
One dashboard, four panels, five minutes
When you get paged, you don’t need to understand every dashboard your team has built. You need four panels, read in order, together.
Grafana is a window into metrics. The skill is knowing which metrics and how to read them as a set.
The 4-panel check
Open your service dashboard. Find these four panels. If any are missing, that’s a gap to fix next sprint — not tonight.
| Order | Panel | Question it answers |
|---|---|---|
| 1 | Error rate | Is something failing? |
| 2 | Latency p99 | Are requests slow? |
| 3 | Resource usage (CPU / memory) | Is the process under pressure? |
| 4 | Request volume | Is traffic actually up? |
Read them together. One panel lying is common. Four panels telling the same story is a diagnosis.
Panel 1: Error rate
Start here. Errors mean users are hitting failures right now.
- Flat near zero — Probably not a user-facing outage (check latency anyway)
- Spike at a deploy marker — Bad deploy candidate
- Gradual climb — Degrading dependency, config drift, or capacity issue
- Sudden cliff — Downstream service went down
Error rate alone doesn’t tell you what broke. It tells you that something broke.
Panel 2: Latency p99
p99 is the latency 99% of requests beat. It catches tail latency that averages hide.
| Pattern | Likely cause |
|---|---|
| p99 up, errors flat | Timeouts, slow downstream, thread pool exhaustion |
| p99 up, errors up | Failing requests (check error logs) |
| p99 up, traffic up | Might just be load — check if p99 scales linearly |
| p99 flat, errors up | Fast failures (4xx/5xx returning quickly) |
Don’t panic at a single p99 spike. Look for sustained elevation over multiple intervals.
Panel 3: Resource usage
Memory and CPU tell you about process health:
- Staircase memory, flat traffic — Memory leak (see the memory leak module)
- Sawtooth memory — Healthy GC, ignore it
- CPU pegged at 100% — Infinite loop, crypto mining, or genuine saturation
- CPU correlates with traffic — Probably fine, maybe needs scaling
Resource panels explain why latency might be climbing even when errors look fine.
Panel 4: Request volume
This panel decouples “the site is broken” from “the site is busy.”
- Traffic up + latency up + errors flat → might need to scale, not roll back
- Traffic flat + errors up + latency up → something internal broke
- Traffic flat + memory climbing → leak, not load
Always check volume before assuming an incident is traffic-driven.
Reading panels together: three common stories
Story 1: Bad deploy
- Errors spike at deploy marker
- p99 spikes at same time
- Memory and traffic look normal
- → Roll back, investigate later
Story 2: Memory leak
- Memory staircase over hours
- Traffic flat
- Errors and p99 rise late
- Deploy marker may be unrelated
- → Not a rollback problem; find the leak
Story 3: Downstream failure
- Errors and p99 spike together
- Memory and CPU normal
- No deploy correlation
- → Check dependencies, circuit breakers, upstream status
Time range and variables
Time range traps:
- Default “last 6 hours” hides a 10-minute incident spike — zoom in
- “Last 5 minutes” misses a slow-building leak — zoom out after initial triage
Dashboard variables:
$environmentset tostagingwhile production burns — check the dropdown$servicefiltering to the wrong microservice — confirm before concluding “all healthy”
Annotations and deploy markers
Deploy annotations overlay release events on graphs. Useful for correlation — dangerous for assumption.
Ask: did the metric change at the deploy, or was it already trending? A deploy marker at 02:14 doesn’t cause a leak that started at 01:30.
Try it yourself
The interactive panel below shows a healthy 4-panel dashboard. Read each panel and note what “normal” looks like — you’ll need that baseline when things go wrong.
Then head to the incident lab to practice on broken states.
Read all four together. When one panel looks wrong, check the others before acting.