Alerting

On this page 8

Alerting is the part of server operations that continuously evaluates monitoring data against defined rules and notifies a person or system when a condition requires action. In the field of Monitoring & logs, it turns passive telemetry such as metrics, log events and health check results into timely, actionable signals that reach the right responder through the right channel.

Good alerting is judged less by how much it detects and more by how reliably it wakes the right person for the right reason. An alert that fires constantly, or for conditions nobody can fix, erodes trust and is eventually ignored, which is why mature teams treat alert quality as an engineering discipline rather than a configuration afterthought.

How an alerting pipeline works

An alerting pipeline moves from raw data to a delivered notification in four distinct stages. Collectors gather metrics, logs and probe results; a rule engine evaluates expressions at a fixed interval; an alert manager groups, deduplicates and routes firing alerts; and a notification channel delivers them to people or automation.

Rules usually include a pending duration, often written as a for clause, so that a condition must hold for several evaluation cycles before the alert fires. This debounce window suppresses short spikes, such as a brief CPU burst during a deployment, that would otherwise cause noise.

The following table shows the stages and what each one contributes.

StageFunctionTypical failure if misconfigured
CollectionScrapes metrics, ships logs, runs probesMissing data hides real outages
EvaluationApplies threshold or query rules every 15-60 sRules fire on noise or never fire
RoutingGroups, deduplicates and selects receiversAlerts reach the wrong team
NotificationPages, messages, emails or webhooksResponders miss or ignore alerts
AcknowledgmentRecords who owns the incidentSeveral people work the same issue
Stages of a typical alerting pipeline

A frequently overlooked element is the absence alert, which fires when expected data stops arriving. Without it, a crashed collection agent looks exactly like a perfectly healthy server, because no data produces no alerts.

Alert types and conditions

Alert conditions fall into several families, each suited to a different kind of problem. Static thresholds are simple and predictable, while rate-of-change and anomaly detection rules catch problems that a fixed number would miss.

Condition typeExampleBest suited for
Static thresholdDisk usage above 90% for 10 minutesHard capacity limits
Rate of changeError count rising 5x in 5 minutesSudden regressions
Absence of dataNo heartbeat for 3 minutesDead agents or hosts
PredictiveDisk projected full within 4 hoursSlow resource exhaustion
Anomaly detectionLatency outside learned seasonal bandTraffic with strong daily patterns
SLO burn rateError budget consumed 14x faster than allowedUser-facing reliability
Common alert condition types

Burn-rate alerting against a service level objective has become the preferred method for user-facing services. It measures how quickly the error budget is being spent and typically combines a fast window (for example 5 minutes and 1 hour) with a slow window (for example 6 hours) so that both sharp outages and gradual degradation are caught with few false positives.

Severity levels and routing

Severity determines how urgently and through which channel an alert reaches a responder. A clear severity scheme prevents the common failure in which every alert pages someone at night regardless of impact.

SeverityMeaningChannel and response
CriticalUsers affected now or data at riskPage on-call immediately, any hour
WarningDegradation likely to become criticalTeam chat, handle in working hours
InfoNotable event, no action neededDashboard or log only
A practical severity scheme

Routing rules match alert labels such as service, environment and team to the correct receiver and escalation policy. An escalation policy defines what happens if the first responder does not acknowledge within a set time, commonly 5 to 15 minutes, after which a secondary engineer or manager is paged.

Grouping combines related alerts into one notification, so that fifty web servers losing the same database produce a single page instead of fifty. Inhibition suppresses dependent alerts while a root-cause alert is active, and silences mute alerts during planned maintenance windows.

Alerting is often confused with the monitoring activities that feed it. The distinction matters because each practice answers a different operational question.

PracticeQuestion it answersRelationship to alerting
MonitoringWhat is the state of the systemSupplies the data alerts evaluate
Health checkIs this component responding correctlyA common input to alert rules
Uptime monitoringIs the service reachable from outsideTriggers availability alerts
Log aggregationWhat happened, in detailEnables log-based alert queries
Incident managementWho fixes it and howBegins where the alert ends
Alerting and neighboring concepts

Tools such as Nagios and Zabbix bundle checks and alerting in one product, while stacks built on the ELK stack or Splunk typically attach alert rules to saved log queries. The underlying principles of thresholds, routing and escalation remain the same across tools.

Designing alerts that people trust

The most widely accepted design rule is to alert on symptoms, not causes. Users experience high latency, errors and unavailability; they do not experience high CPU. A cause-based alert such as CPU above 80% fires often without user impact, whereas a symptom-based alert fires only when service quality actually suffers.

  • Every page must be actionable: if the responder cannot do anything, the alert should not page.
  • Attach a runbook link or short remediation note to each alert so that responders start from known steps.
  • Include context in the message: affected service, current value, threshold and a dashboard reference.
  • Review alerts regularly, deleting or demoting any rule that fired without leading to action.
  • Test alerts deliberately after changes, because a rule that never fires may simply be broken.

Teams commonly track the ratio of actionable to total pages and the number of pages per on-call shift. A sustainable target in many organizations is no more than two incidents per 12-hour shift, although the right figure depends on team size and service criticality.

Common failure modes

Most alerting problems come from configuration habits rather than tool limitations. The table below links typical symptoms to their causes and remedies.

SymptomLikely causeRemedy
Alert fatigueToo many low-value pagesDelete, demote or merge noisy rules
Flapping alertsThreshold sits at normal fluctuationAdd hysteresis or a longer pending period
Silent outageCollector down, no absence ruleAdd heartbeat and no-data alerts
Alert stormOne failure triggers hundreds of rulesUse grouping and inhibition
Late detectionEvaluation interval or window too longShorten windows for critical rules
Alerting problems and their fixes

Alert fatigue is the most damaging of these because it is gradual. When responders learn that most pages are harmless, they begin to acknowledge without investigating, and a genuine incident is eventually missed.

In plain words

Alerting is like a smoke detector for servers. Computers constantly report numbers about their health. Alerting watches those numbers so that people do not have to stare at screens all day.

Sometimes a number crosses a line that matters. A website may return errors, or a disk may be almost full. Then the system sends a message or a phone call to the person on duty. The key idea is balance. Too few alarms let real problems slip by. Too many alarms teach people to ignore them, like a car alarm that goes off every night.

A real-world example

An online retailer runs its checkout service on twelve application servers. Originally, the operations team had alerts for CPU above 75%, memory above 80% and every single HTTP 500 error. The on-call engineer received around 40 pages per week, most of which resolved on their own, and response times to real incidents grew longer.

The team replaced these rules with two symptom-based alerts: a burn-rate alert on the checkout error budget for a 99.9% success objective, and a latency alert on the 95th percentile exceeding 800 ms for ten minutes. Resource alerts were demoted to warnings in team chat, and a heartbeat alert was added for the metrics agent.

Within a month, pages dropped to about four per week, nearly all of them actionable. When a faulty payment gateway release later raised checkout errors to 3%, the fast burn-rate rule paged the engineer within six minutes, and the release was rolled back before most customers noticed.