beginner · Module 1.4
Alerting & Alertmanager
Learning Objectives
- Author alerting rules with for: durations to suppress flapping.
- Explain Alertmanager grouping, inhibition, and silences.
- Route OTT alerts to the right on-call receiver by severity and team label.
Key Concepts
Alerting rules and for:
An alerting rule fires when its PromQL expression is true; the for: clause requires it to stay true for a duration before firing, filtering out momentary blips during match traffic ramps.
Grouping and inhibition
Alertmanager groups related alerts into one notification and can inhibit lower-severity alerts when a higher-severity one for the same scope is already firing, cutting noise during a broad incident.
Routing tree and silences
A routing tree matches labels (team, severity) to receivers (PagerDuty, Slack). Silences temporarily mute known-noisy alerts during planned maintenance.
Warning
Alerting on raw error counts instead of ratios or burn rates produces pages that scale with traffic — you will get paged simply because a match is popular.
Insight
Inhibition rules are how you avoid 200 pages when one upstream dependency takes everything down.
Knowledge Check
1. What does the for: clause in a Prometheus alerting rule do?
2. Which Alertmanager feature suppresses lower-severity alerts when a related higher-severity alert is already firing?
3. Why is alerting on a raw error COUNT problematic for a live-match OTT service?