Prometheus alerting that humans actually read
Alert on symptoms, not causes. A practical framework for routing, deduplication and runbook links.
Most alert fatigue isn't a tooling problem — it's a design problem. We cut alert volume by 60% while catching more real incidents.
Alert on symptoms
Users feel latency and errors, not CPU saturation. Page on SLO burn rate; ticket on causes.
Every alert needs a runbook
If you can't write the first response in the alert annotation, you don't understand the failure mode yet.
Routing
Severity 1 pages immediately, severity 2 goes to chat with a timeout escalation, severity 3 becomes a morning ticket. Deduplicate by alert fingerprint so one broken node doesn't create forty pages.