Connection draining is the load balancing mechanism that stops sending new connections or requests to a backend while allowing the connections it already holds to complete gracefully within a defined timeout. In networking and application delivery, it is the standard way to remove a server from a pool for maintenance, scaling in or deployment without dropping in-flight requests.
Vendors use different names for the same idea: deregistration delay, graceful shutdown, drain mode or connection quiescing. The underlying contract is identical: the backend is marked as leaving, the balancer stops choosing it, and the remaining sessions are given a bounded amount of time to finish before they are forcibly closed.
How connection draining works
Connection draining works by changing a backend from an active state to a draining state in which the load balancer excludes it from new selection decisions but keeps forwarding packets for existing flows. The balancer continues to track the connection table or session state for that backend until each flow ends or the drain timer expires.
The sequence is predictable. An operator, an autoscaling group or an orchestrator triggers deregistration; the balancer stops including the target in its algorithm, whether that is round robin, least connections or consistent hashing; existing TCP connections and HTTP requests continue; and when the drain timeout is reached, any remaining connections are reset or closed. Only then is the backend reported as fully removed.
The following table summarizes the backend states that most implementations expose during this process.
| State | New connections | Existing connections | Typical trigger |
|---|---|---|---|
| Active | Accepted | Served normally | Healthy and registered |
| Draining | Refused or rerouted | Allowed to finish | Deregistration or scale-in |
| Drained | Refused | None remaining | All flows ended |
| Forced removal | Refused | Reset at timeout | Drain timer expired |
Draining at layer 4 and layer 7
The behavior of connection draining depends on which layer the balancer operates at, because a layer 4 device sees TCP or UDP flows while a layer 7 proxy sees individual HTTP requests. This difference determines what counts as finished work.
In layer 4 load balancing, draining keeps existing five-tuple flows pinned to the leaving backend until the client or server closes them. A long-lived TCP session, such as a database connection pool, may therefore hold the backend for the entire timeout. In layer 7 load balancing, the proxy can be smarter: it can finish the current request, then send Connection: close in HTTP/1.1 or a GOAWAY frame in HTTP/2, prompting clients to reconnect elsewhere at a clean boundary.
| Traffic type | Unit of work | Graceful signal | Main difficulty |
|---|---|---|---|
| Short HTTP/1.1 requests | Single request | Connection: close header | Keep-alive reuse |
| HTTP/2 and gRPC | Stream on shared connection | GOAWAY frame | Many streams per connection |
| WebSocket | Long-lived session | Application close message | Sessions lasting hours |
| Raw TCP | Entire connection | FIN from either side | No protocol-level hint |
| UDP | Pseudo-flow by tuple | None | Idle timeout only |
UDP has no connection semantics, so draining usually means keeping tuple affinity until an idle timer expires. Sticky sessions add another dimension: a draining backend may still receive requests from clients whose session cookie points to it, depending on whether the balancer honors affinity during the drain.
Choosing the drain timeout
The drain timeout should be slightly longer than the longest legitimate request the service handles, and no longer. A timeout that is too short cuts off uploads, reports or streaming responses; a timeout that is too long slows deployments and scale-in and keeps paying for capacity that serves almost nothing.
Typical defaults range from 30 to 300 seconds, and some cloud balancers allow up to one hour. Engineers should measure the 99th or 99.9th percentile request duration from access logs before choosing a value, then add a safety margin of about 20 to 50 percent.
| Workload | Typical duration | Suggested timeout | Reason |
|---|---|---|---|
| REST APIs | Under 2 s | 10 to 30 s | Requests finish quickly |
| Web pages and assets | Under 5 s | 30 to 60 s | Keep-alive tails are short |
| File uploads and exports | Seconds to minutes | 120 to 600 s | Large bodies take time |
| WebSocket or streaming | Minutes to hours | 300 s plus app reconnect | Timeout alone cannot cover it |
| Batch or database TCP | Variable | Match pool recycle time | Pools hold idle connections |
For long-lived sessions, a timeout is not enough on its own. The application must cooperate by closing sessions voluntarily, ideally with randomized delays so that thousands of clients do not reconnect at the same instant and create a reconnection storm on the remaining backends.
How draining fits deployments and autoscaling
Connection draining is the foundation of zero-downtime operations, because rolling updates, blue-green deployments and scale-in events all remove backends while traffic flows. Without it, every removal produces a burst of HTTP 502 or connection reset errors.
In container orchestrators, draining must be coordinated with process shutdown. A common pattern is a preStop delay: the pod is first removed from service endpoints, the process keeps serving for several seconds while the balancer converges, and only then does it receive SIGTERM and stop accepting work. The application should also fail its readiness check as soon as shutdown begins, so health-check-driven balancers stop routing to it.
- Rolling deployment: each old instance drains before termination, so capacity never drops to zero.
- Blue-green switch: the old environment drains after traffic moves, then is retired.
- Autoscaling scale-in: the scaling group waits for the drain to finish before terminating the instance.
- Maintenance window: an operator sets drain mode manually, patches the server and returns it to active.
Limitations and common mistakes
Connection draining guarantees only that the balancer stops choosing the backend; it does not guarantee that clients stop using it. Several failure modes appear repeatedly in production, and most of them result from mismatched timers between layers.
| Symptom | Likely cause | Fix |
|---|---|---|
| Errors at the end of every deploy | Process exits before drain ends | Align shutdown grace period with drain timeout |
| Errors at the start of every deploy | Process stops before balancer converges | Add a preStop delay of a few seconds |
| Deployments take far too long | Timeout much longer than requests | Base timeout on measured percentiles |
| Backend never empties | Persistent keep-alive or WebSocket clients | Send close signals from the application |
| Load spike on remaining servers | All clients reconnect at once | Stagger closes and drain fewer nodes per step |
A frequent misconception is that health checks alone provide draining. A failing health check removes a backend after several intervals, and many balancers then reset existing connections immediately. Proper draining is an explicit, separate state with its own timer, and it should be used whenever a removal is planned rather than caused by failure.
Put simply
Connection draining is like closing a checkout lane at a supermarket. The cashier puts up a lane closed sign so no new shoppers join the line, but everyone already waiting is still served. Only when the last customer has paid does the cashier leave.
A load balancer does the same with servers. It stops sending new visitors to a server that is about to be switched off, lets the current visitors finish what they are doing, and then removes the server. The result is that people using the website never notice the change.
In practice
An online retailer runs its checkout API on twelve containers behind a layer 7 load balancer and deploys new versions several times a day. Before enabling draining, each rolling update produced a few hundred failed payment requests, because containers were terminated while still processing orders.
The platform team measured request durations and found that the 99.9th percentile was 8 seconds, driven by calls to an external payment gateway. They set the deregistration delay to 15 seconds, added a 5-second preStop pause, raised the container shutdown grace period to 30 seconds and made the service fail its readiness probe on SIGTERM.
After the change, deployments replaced two containers at a time, each one draining before shutdown. Failed checkout requests during releases fell to effectively zero, and a full rollout still completed in under three minutes.