A watchdog timer is a countdown mechanism that triggers a recovery action, usually a reset or restart, when the monitored system fails to signal within a set interval that it is still healthy. In services & processes on servers, watchdog timers exist at two levels: hardware watchdogs that reboot the whole machine when the operating system freezes, and software watchdogs that restart an individual daemon when it hangs.
The core idea is simple and robust: a healthy component must periodically prove it is alive by resetting the timer, an action called kicking, petting or feeding the watchdog. If that proof stops arriving, the watchdog assumes the component is stuck and acts without waiting for a human.
How a watchdog timer detects a hung system
A watchdog timer counts down from a configured timeout. The monitored software periodically writes to the watchdog, which reloads the counter. As long as these writes keep arriving, the counter never reaches zero; when they stop, the timer expires and fires its action.
The strength of the design is that the watchdog does not need to understand what went wrong. A deadlock, an infinite loop, a kernel panic or a process starved of CPU all look the same from outside: the heartbeat stops. This makes watchdogs the last line of defense against hung processes that other monitoring cannot repair.
| Phase | What happens | Typical duration |
|---|---|---|
| Arming | Watchdog device or service monitoring is activated | At boot or service start |
| Feeding | Healthy software resets the counter | Every few seconds |
| Pre-timeout | Optional warning interrupt or log entry | Seconds before expiry |
| Expiry | Counter reaches zero without a reset | After the full timeout |
| Recovery | Reset, restart, crash dump or fencing | Immediate |
A good watchdog feeder only kicks the timer after checking real health, not from a separate thread that runs regardless of the main work. Otherwise the main loop can be frozen while the feeder thread keeps the watchdog satisfied, which defeats the purpose.
Hardware and software watchdogs on servers
Servers use watchdogs at several layers, and each layer catches a different class of failure. A hardware watchdog sits on the motherboard, in the chipset or in the baseboard management controller (BMC), and keeps counting even when the operating system has stopped completely.
| Layer | Example | Recovery action | Catches |
|---|---|---|---|
| Hardware or BMC | IPMI watchdog, chipset timer | Hard reset or power cycle | Kernel freeze, firmware hang |
| Kernel interface | Linux /dev/watchdog driver | Reboot through the hardware timer | Userspace lockup |
| Kernel soft lockup | softlockup and hardlockup detectors | Panic and reboot if configured | CPU stuck in kernel mode |
| Service manager | systemd WatchdogSec | Restart the unit | Hung daemon |
| Application | Internal heartbeat thread | Self-restart or alert | Stalled internal queue |
On Linux, systemd acts both as a consumer and a provider. With RuntimeWatchdogSec it feeds the hardware watchdog for the whole machine, and with WatchdogSec in a unit file it expects the service to send WATCHDOG=1 through sd_notify. If the notification is late, systemd kills and restarts the service according to its restart policy.
Watchdog timers compared with health checks and supervisors
A watchdog is a push-based liveness signal: the monitored component must report on its own. A health check is pull-based: an external system asks the component whether it is healthy. A process supervisor such as Supervisord, PM2 or Monit restarts processes that exit, but on its own it does not notice a process that is alive yet frozen.
| Mechanism | Signal direction | Detects hang | Detects crash |
|---|---|---|---|
| Watchdog timer | Component pushes heartbeat | Yes | Yes, heartbeat stops |
| Process supervisor | Watches process exit | Not by itself | Yes |
| Health check endpoint | Monitor polls component | Yes, if probe is meaningful | Yes |
| Uptime monitoring | External service polls | Only user-facing symptoms | Yes |
In practice these mechanisms are layered rather than chosen. A supervisor handles clean crashes, a watchdog handles silent hangs, and external monitoring confirms that users can actually reach the service.
Choosing timeouts and recovery actions
The timeout is the central trade-off: too short and normal pauses such as garbage collection, heavy I/O or a busy CPU cause false resets; too long and a hung service stays unavailable. A common rule is to feed the watchdog at about half the timeout or more often, so one delayed heartbeat does not trigger recovery.
| Context | Typical timeout | Reasoning |
|---|---|---|
| Hardware watchdog for the OS | 30 seconds to a few minutes | Must survive heavy load and boot phases |
| Service watchdog in systemd | 10 to 60 seconds | Fast recovery of a single daemon |
| Cluster fencing watchdog | Seconds to tens of seconds | Must expire before peers take over |
| Embedded controllers | Milliseconds to seconds | Tight real-time control loops |
The recovery action should match the scope of the failure. Restarting one service is preferred when only that service hangs; a full machine reset is justified only when the kernel or hardware is unresponsive. Collecting a crash dump before the reset, through a pre-timeout interrupt or kdump, preserves evidence for root cause analysis.
Pitfalls when relying on watchdogs
Watchdogs recover symptoms, not causes, so a restart loop can hide a real defect. Every watchdog expiry should be logged and alerted, and repeated expiries should stop automatic restarts rather than cycle forever.
- Feeding from an independent thread keeps the watchdog quiet while the real work is frozen.
- Timeouts shorter than normal pauses produce resets during backups, patching or peak load.
- Disabling the watchdog during maintenance and forgetting to re-enable it removes protection silently.
- No restart limit turns a persistent bug into an endless crash loop.
- Resetting without diagnostics loses the state needed to find the root cause.
In high availability clusters, watchdogs also serve as a self-fencing mechanism: a node that loses contact with the cluster stops feeding its watchdog and reboots itself, so peers can safely take over shared storage without risking split-brain data corruption.
The idea in brief
A watchdog timer is like a dead man’s switch on a train. The driver must press a pedal every few seconds. If the pedal stops being pressed, the train brakes by itself.
On a server, the software must keep saying it is fine. If it falls silent for too long, the watchdog restarts it automatically. Nobody has to notice the problem or walk to the machine. The main benefit is that frozen systems come back on their own, often within a minute.
A practical scenario
A hosting provider runs a message queue daemon on Linux servers. Occasionally a rare deadlock freezes the daemon: the process still exists, so the process supervisor sees nothing wrong, but no messages are delivered.
The team adds WatchdogSec=30 to the systemd unit and changes the daemon so that its main event loop sends a heartbeat only after it has processed its queue check. They also set a restart limit of five attempts in ten minutes and an alert on every watchdog event. At the machine level, systemd feeds the BMC hardware watchdog with a two-minute timeout to cover kernel hangs.
The next time the deadlock occurs, systemd restarts the daemon after about 30 seconds instead of waiting for a customer report. The outage shrinks from hours to under a minute, and the alert history gives developers the timestamps they need to reproduce and fix the bug.