Building a baseline

You can’t spot abnormal without knowing normal

Every incident investigation starts the same way: “is this actually bad?”

Without a baseline, you’re guessing. Is 200ms p99 bad? Maybe. Maybe it’s Tuesday afternoon and that’s normal. Maybe it’s 3am and that’s catastrophic.

Baselines turn “I think something’s wrong” into “this is 4× higher than usual for this hour.”

What is a baseline?

A baseline is what normal looks like for your service — by metric, by time of day, by day of week.

Not a single number. A range:

• Error rate: 0.02–0.08% on weekdays, 0.01–0.05% on weekends
• p99 latency: 45–80ms for checkout, 120–200ms for search
• Memory: sawtooth between 400–500MB, floor stable over 24h
• Traffic: peaks at 14:00 UTC, trough at 04:00 UTC

You build baselines by observing, not by setting SLOs in a spreadsheet.

Why baselines matter at 2am

Without baseline	With baseline
“Error rate is 0.3%” — is that bad?	“Error rate is 0.3%, normally 0.05%. Yes, bad.”
“Traffic is half usual” — outage or quiet night?	“Traffic is half usual for 3pm Tuesday. Something’s wrong.”
“Memory is 600MB” — leak or normal?	“Memory floor was 420MB yesterday. Staircase started 2h ago.”

The dashboard looks the same in both cases. The baseline tells you which one it is.

Seasonal patterns: time is a variable

Metrics aren’t constant. They breathe:

• Time of day — US traffic peaks when Europe sleeps
• Day of week — Monday morning ≠ Saturday night
• Monthly cycles — Billing runs on the 1st. Reports on Fridays.
• External events — Product launch, marketing email, Black Friday

An alert threshold of “error rate > 1%” fires every quiet night if your baseline is 0.05%. Compare against expected for this hour, not a global threshold.

How to build a baseline (practical)

You don’t need a data science project. Start with one service and four metrics:

1. Open Grafana for your most critical service
2. Set time range to “last 7 days”
3. Note the range for error rate, p99, memory, and traffic at the same hour you’re usually on-call
4. Write it down — a sticky note, a doc, a #baselines channel post

checkout-service (weekday 02:00 UTC):
  error_rate: 0.03–0.08%
  p99: 50–90ms
  memory: 400–520MB (sawtooth)
  traffic: 800–1200 req/min

Repeat for peak hours. Now you have a reference card.

Documenting for the next on-call

Your baseline dies when you go on vacation unless someone else can read it.

Minimum viable baseline doc:

• Service name and dashboard link
• Normal ranges for the 4-panel check
• Known seasonal patterns (“traffic drops 60% after 22:00 UTC”)
• Known noise (“health check errors on pod restart — ignore”)
• Recent deploy cadence (“deploys Mon/Wed/Fri 18:00 UTC”)

The next on-call shouldn’t need to ask you what normal looks like.

Baseline vs alerting

Baseline	Alert
What you know is normal	What fires when something crosses a threshold
Context for judgment	Automation that wakes you up
Built by observing	Built by configuring

Good alerts use baselines: “error rate > 3× 7-day average for this hour” beats “error rate > 1%.”

Bad alerts ignore baselines: firing every night at 3am because traffic is low and percentage errors look high.

The baseline sanity check during incidents

Before you roll back, restart, or scale — ask:

1. Is this metric actually abnormal for this time?
2. Did this pattern exist yesterday at the same hour?
3. Is only one instance affected, or the whole fleet?

If it’s normal for this context, you might be investigating a non-incident. Go back to sleep. (After you confirm.)

Start tonight

Pick your most-paged service. Open Grafana. Set “last 7 days.” Write down the four panel ranges for your on-call hours.

Ten minutes of work. Saves an hour of panic next time the alert fires and the number “looks high” but you can’t prove it.