Graceful restart

On this page 7

A graceful restart is a controlled restart of a server process in which new workers start with the new configuration or code while old workers finish the requests they already accepted and then exit. In the services and processes layer of servers, it is the standard way to apply changes to a running daemon such as a web server, an application server or a PHP-FPM pool without refusing or cutting off client connections.

The essential property is that the listening socket never closes. Connections keep arriving at the same address and port throughout the operation, and only the processes that serve them are replaced. A hard restart, by contrast, stops everything first and starts again afterward, which creates a short window of refused connections and aborted responses.

How a graceful restart works

A graceful restart relies on a master and worker architecture. A long-lived master process owns the listening sockets and supervises a set of worker processes that actually handle traffic. When an operator requests a restart, the master rereads its configuration, validates it and spawns a new generation of workers that inherit the already open sockets.

The old generation receives a signal to stop accepting new connections. Each old worker completes its current requests, closes idle keep-alive connections and exits on its own. This phase is called draining, and it usually ends within seconds, although long uploads, streaming responses or WebSocket sessions can extend it considerably.

The sequence below shows the typical order of events. The exact signals and timeouts differ between server programs, but the logic is the same in every implementation.

  1. The operator or a tool such as systemctl reload sends a reload signal to the master process.
  2. The master validates the new configuration; if validation fails, it keeps the old workers and logs an error.
  3. New workers start, open log files again and begin accepting connections on the shared listening socket.
  4. Old workers stop accepting, finish in-flight requests and exit.
  5. If an old worker exceeds the drain timeout, the master terminates it forcibly.

Signals and commands in common server software

Each major daemon exposes graceful restart through a specific Unix signal or a wrapper command. Knowing the exact mapping matters, because some signals look similar but produce an abrupt stop instead of a graceful one.

SoftwareGraceful command or signalDrain limit settingAbrupt alternative
Nginxnginx -s reload (SIGHUP to master)worker_shutdown_timeoutnginx -s stop (SIGTERM)
Apache HTTP Serverapachectl -k graceful (SIGUSR1)GracefulShutdownTimeout (for graceful-stop)apachectl -k restart (SIGHUP)
PHP-FPMSIGUSR2 to master, often via systemctl reloadprocess_control_timeoutSIGTERM or SIGINT
HAProxyReload through the master-worker modehard-stop-afterSIGTERM to all processes
GunicornSIGHUP to the arbitergraceful_timeout (default 30 s)SIGINT or SIGQUIT
Graceful restart controls in widely used server software

A frequent trap is Apache, where SIGHUP means an immediate restart that kills active children, while SIGUSR1 is the graceful variant. In PHP-FPM the default process_control_timeout is 0, meaning no wait, so a reload can cut long requests unless the value is raised to a few seconds or more.

On systemd hosts, the unit file connects these signals to standard commands. The ExecReload directive defines what systemctl reload does, and a unit without that directive cannot be reloaded gracefully through systemd at all.

Graceful restart is often confused with neighboring operations that sound similar but have different effects on availability. The comparison below separates them by what happens to the listening socket, to active requests and to the master process.

OperationListening socketActive requestsTypical use
Graceful restartStays openFinish normallyConfiguration or code change
Hard restartClosed brieflyAbortedRecovery from a hung process
Configuration reloadStays openUsually finishSettings change only
Graceful shutdownClosed after drainingFinish, then service stopsMaintenance or host reboot
Rolling restartHandled by load balancerShifted to other nodesClusters with several instances
How graceful restart differs from similar operations

In many programs a configuration reload and a graceful restart are the same mechanism, because new settings take effect only in new workers. The distinction matters with a binary upgrade, where Nginx for example can start a second master from a new executable using SIGUSR2 and later retire the old one with SIGWINCH and SIGQUIT.

A process supervisor or init system usually issues these commands, while each worker process carries out the draining itself. The process ID of the master normally stays the same during a reload, which keeps PID files and monitoring valid.

Configuring drain time and readiness

The most important parameter is the drain timeout, the maximum time old workers may spend finishing requests. Too short a value aborts slow requests; too long a value lets two generations of workers coexist, which doubles memory use and can keep stale code serving traffic.

WorkloadTypical request durationReasonable drain timeoutReason
Static files and APIsMilliseconds to 1 s5 to 30 sRequests end quickly
Dynamic PHP pages0.1 to 10 s10 to 60 sCovers slow database queries
File uploads and reportsUp to several minutes60 to 300 sAvoids corrupt uploads
WebSocket and streamingHoursBounded, plus client reconnectConnections never end on their own
Choosing a drain timeout by workload

Before triggering a reload, operators should run the program’s configuration test, such as nginx -t or apachectl configtest. Most daemons refuse an invalid configuration during reload, but a test catches the problem before anyone depends on the change being live.

Applications behind a graceful restart must also be ready before they accept traffic. Workers that load large caches or open database pools should report readiness only when that work is complete, and a health check should confirm the new generation before old workers are retired.

Limitations and common mistakes

A graceful restart protects connections, but it does not guarantee application consistency. If new code changes a database schema or a session format, old and new workers may run side by side for the length of the drain period and must remain compatible with each other.

SymptomLikely causeRemedy
Brief 502 or 503 errorsHard restart used instead of reloadUse the graceful signal or systemctl reload
Long requests cut offDrain timeout too short or zeroRaise the timeout to match real request length
Memory spike after reloadOld workers never exitSet a bounded drain limit and close idle connections
Change not appliedReload failed validation silentlyRun the config test and read the error log
Mixed old and new behaviorTwo generations serving at onceKeep changes backward compatible
Symptoms of a faulty graceful restart
  • Changes to listening ports or user accounts may still require a full restart in some programs.
  • A reload does not help a master process that has crashed; supervision must restart it.
  • In containers, the main process must receive and handle the signal, which fails when a shell wrapper swallows it.

In simple terms

Think of a busy restaurant changing its menu at lunchtime. Instead of locking the door and sending every guest away, it lets the current diners finish their meals while new guests receive the new menu. No one is turned away, and no plate is taken from the table.

A graceful restart does the same for a server. New helper processes start with the new settings, the old ones finish what they started and then leave. Visitors never notice the change.

A practical example

An operations engineer at an online shop needs to add a new security header to the Nginx configuration of a server that handles about 400 requests per second during the day. A hard restart would abort checkouts in progress and show error pages to customers for a moment.

The engineer first runs nginx -t, which confirms the syntax, and then runs systemctl reload nginx. The master process starts new workers that send the new header, while old workers finish their requests within a few seconds and exit. The worker_shutdown_timeout of 30 s caps the wait for any slow connection.

Monitoring shows no rise in 5xx errors and no dropped connections during the change. The same procedure is then built into the deployment script, so every future configuration change is applied with a graceful restart by default.