A service restart policy is the set of rules that tells a process supervisor whether, when and how often to restart a service after its main process exits, crashes or stops responding. In services & processes on servers, the policy turns a single failure into a short, automatic recovery instead of an outage that waits for a human, while rate limits prevent a broken service from restarting endlessly.
Restart policies are configured in the supervisor that owns the process: systemd units on most Linux servers, Supervisord or PM2 for application processes, the restart setting of Docker containers, or the restartPolicy of a Kubernetes pod. The vocabulary differs, but every implementation answers the same four questions: which exit conditions trigger a restart, how long to wait, how many attempts are allowed, and what happens when the limit is reached.
How a restart policy decides to act
The supervisor starts the service as a child process and watches it through its process ID. When the process ends, the supervisor receives the exit status or the signal that killed it and compares that result with the policy. A clean exit with code 0 usually means an intentional stop, while a non-zero code, a fatal signal such as SIGSEGV or SIGKILL, or an expired watchdog timer signals a failure.
If the policy matches, the supervisor waits for the configured restart delay, starts the process again and increments a counter. If too many restarts happen within a defined time window, it stops trying and marks the service as failed. That final state is the signal for alerting, failover or an operator.
In systemd, the Restart= directive expresses the trigger conditions. The table below shows the main values and when each one fits.
| Restart value | Restarts after | Typical use |
|---|---|---|
| no | Never | One-shot jobs, services managed elsewhere |
| on-failure | Non-zero exit, fatal signal, timeout, watchdog | Most long-running daemons |
| on-abnormal | Signal, timeout or watchdog, not exit codes | Services that exit non-zero by design |
| on-abort | Uncaught signal only | Narrow crash recovery |
| on-watchdog | Watchdog timeout only | Services that send keepalive notifications |
| always | Any exit, including clean exit code 0 | Agents that must run continuously |
Delays, backoff and rate limits
The restart delay gives dependent resources time to recover and stops a crashing service from consuming a CPU core in a tight loop. A fixed delay of 100 milliseconds to a few seconds suits most daemons. Exponential backoff, in which each attempt waits longer than the previous one up to a ceiling, is better when the cause is often external, such as an unreachable database.
The start rate limit is the circuit breaker of the policy. In systemd it is StartLimitBurst and StartLimitIntervalSec, with a default of five starts within ten seconds. Once the limit is exceeded, the unit enters the failed state and stays there until an operator resets it or the window expires and a new start is requested.
| Parameter | Typical range | What it controls |
|---|---|---|
| Restart delay | 100 ms to 30 s | Pause before each new attempt |
| Backoff ceiling | 1 to 5 minutes | Longest wait in exponential backoff |
| Burst limit | 3 to 10 starts | Attempts allowed inside the window |
| Rate-limit window | 10 s to 10 minutes | Period over which attempts are counted |
| Stop timeout | 10 to 90 s | Grace period before a forced kill |
| Watchdog interval | Seconds to minutes | Maximum silence before a hung process is restarted |
Newer systemd releases add RestartSteps and RestartMaxDelaySec, which implement progressive backoff natively. Without them, a fixed delay combined with a reasonable burst limit is still a sound default.
Restart policies across supervisors
Every supervisor implements the same idea with its own names, and the defaults differ enough to surprise teams that move a service between platforms. The comparison below lists the equivalents.
| Supervisor | Policy setting | Limit mechanism | Default behavior |
|---|---|---|---|
| systemd | Restart=, RestartSec= | StartLimitBurst, StartLimitIntervalSec | No restart unless configured |
| Docker | –restart no, on-failure, always, unless-stopped | on-failure:N maximum retries | No restart |
| Kubernetes | restartPolicy Always, OnFailure, Never | CrashLoopBackOff, up to 5 minutes | Always for Deployment pods |
| Supervisord | autorestart true, false, unexpected | startretries, startsecs | Restart on unexpected exit |
| PM2 | autorestart, max_restarts | min_uptime, exponential backoff option | Restart enabled |
A restart policy is narrower than a process supervisor as a whole: the supervisor also handles logging, dependencies and resource limits. It is also different from a graceful restart, which is a planned reload that lets in-flight requests finish, whereas a restart policy reacts to unplanned exits.
Choosing the right policy for a service
The right policy depends on what a restart can fix. Transient faults, such as a memory leak, a lost connection or a rare race condition, are often cleared by a fresh process. Deterministic faults, such as a broken configuration file or a missing secret, fail again on every attempt, so the policy should give up quickly and alert.
| Situation | Recommended policy | Reason |
|---|---|---|
| Web or API daemon | on-failure, 2-5 s delay, burst limit | Recovers crashes, stops crash loops |
| Monitoring or log agent | always, short delay | Must run even after a clean exit |
| Batch or one-shot job | no, or on-failure with few retries | Repeated runs may duplicate work |
| Service depending on a database | on-failure with backoff | Waits out the dependency outage |
| Stateful primary database | on-failure with low limits plus alerting | Repeated restarts can worsen corruption |
A good policy is paired with health checks and a watchdog, because a process that hangs without exiting never triggers an exit-based restart. Readiness checks in front of a load balancer then ensure that traffic returns only after the restarted process is actually serving.
Common mistakes with restart policies
- Restarting forever without limits: an endless crash loop fills logs, hides the real error and can exhaust disk space or connection pools.
- Masking a persistent bug: a service that restarts every hour looks healthy on dashboards; restart counts must be monitored and alerted on.
- Restart delay of zero: a tight loop burns CPU and hammers the failing dependency.
- Ignoring the stop timeout: a too-short grace period kills the process before it flushes data, turning a clean stop into a failure.
- Using always for jobs that exit on purpose: the supervisor relaunches a finished task again and again.
- Restarting a dependent service before its dependency: without ordering, the service fails until the dependency is ready.
The guiding principle is that a restart policy is a first response, not a fix. It buys time and keeps users served, while logs, core dumps and exit codes from each failure must still reach the people who can remove the cause.
Explained without jargon
Think of a night guard who watches a set of machines in a factory. If one stops, the guard waits a moment and switches it back on. If the same machine stops five times in a few minutes, the guard stops switching it on and calls the engineer, because something is clearly broken.
A service restart policy is that guard’s instruction sheet for server software. It says which kinds of stops deserve a restart, how long to wait, and when to give up and raise the alarm.
A typical use case
An online shop runs its order API as a systemd service on three servers behind a load balancer. The unit uses Restart=on-failure, RestartSec=3, and a limit of five starts within 300 seconds. A rare bug in a payment library occasionally crashes the process with a segmentation fault.
When the crash happens during a sales event, systemd restarts the API within about three seconds, the load balancer health check marks the server healthy again, and customers barely notice. Later, a faulty configuration deployment makes the service exit immediately on every start. After five failed attempts the unit enters the failed state, the monitoring system raises an alert, and the deployment is rolled back. The policy absorbed the transient crash and surfaced the persistent fault instead of hiding it.