Flow control

On this page 7

Flow control is the mechanism by which a receiver limits how much data a sender may transmit before waiting for confirmation, so that the receiver’s buffer is never overrun. In transport protocols such as TCP and SCTP, flow control is an end-to-end agreement between two hosts, and it protects the receiving application rather than the network in between.

The concept is easily confused with congestion control. Flow control answers whether the receiver can absorb more data, while congestion control answers whether the network path can carry it. A sender always obeys the smaller of the two limits.

How flow control works in TCP

TCP implements flow control with a sliding window. Every segment the receiver sends carries a 16-bit receive window (rwnd) field that announces how many bytes of free buffer space remain beyond the last acknowledged byte. The sender may have at most that many unacknowledged bytes in flight.

As the application reads data from the socket buffer, space frees up and the receiver advertises a larger window; when the application falls behind, the advertised window shrinks. If it reaches zero, the sender must stop and periodically send zero window probes until the receiver reports free space again, which prevents a deadlock if a window update is lost.

The 16-bit field alone allows at most 65,535 bytes. The window scale option, negotiated in the TCP handshake and defined in RFC 7323, multiplies the value by a power of two up to 2 to the 14th, permitting windows of roughly 1 GiB. Without scaling, a single connection could not fill fast long-distance links.

ElementWhere it livesFunction
Receive windowTCP header of every segmentAdvertises free receive buffer space
Window scale optionSYN and SYN-ACK segmentsExtends windows beyond 64 KiB
Zero window probeSender timerDetects when a closed window reopens
Socket receive bufferReceiver operating systemHolds data until the application reads it
Send bufferSender operating systemKeeps data until it is acknowledged
Elements of TCP flow control

Window size, bandwidth and latency

Throughput under flow control is bounded by the window divided by the round-trip time. To keep a path fully used, the window must be at least the bandwidth-delay product (BDP), the amount of data that fits in transit during one round trip.

The following values show why receive buffers must grow with distance and speed; they apply to a single connection with no packet loss.

Receive windowRound-trip timeMaximum throughput
64 KiB1 ms on a LANAbout 520 Mbit/s
64 KiB100 ms intercontinentalAbout 5 Mbit/s
4 MiB100 ms intercontinentalAbout 335 Mbit/s
16 MiB100 ms intercontinentalAbout 1.3 Gbit/s
Maximum throughput for a given receive window

Modern operating systems apply receive buffer autotuning, growing the window toward the measured BDP, typically within limits of a few megabytes to tens of megabytes. When transfers across long paths stall at a fixed low rate, a receive window that is too small is one of the first causes to check.

Flow control appears at several layers of a network stack, and each version protects a different resource. The comparison below helps place the transport-layer meaning.

MechanismLayerWhat it protectsSignal used
TCP flow controlTransportReceiver bufferAdvertised receive window
TCP congestion controlTransportNetwork pathPacket loss, delay, ECN marks
Ethernet pause framesData linkSwitch or NIC bufferIEEE 802.3x pause frame
Priority flow controlData linkPer-class buffersIEEE 802.1Qbb per-priority pause
HTTP/2 and QUIC flow controlApplication or streamPer-stream buffersWINDOW_UPDATE or MAX_DATA frames
Flow control and neighboring mechanisms

UDP has no flow control at all: a fast sender can overflow the receiving socket buffer, and the excess datagrams are silently dropped. Protocols built on UDP, such as QUIC, therefore add their own credit-based flow control at the connection and stream level.

Flow control in SCTP and QUIC

Newer transport protocols keep the receiver-driven principle of TCP but apply it per association or per stream. SCTP advertises an a_rwnd value in every SACK chunk, which plays the same role as the TCP receive window but covers all streams of one association together.

QUIC uses credit-based limits at two levels. A MAX_DATA frame caps the total bytes the peer may send on the whole connection, while MAX_STREAM_DATA frames cap each individual stream. This prevents one large download from consuming the entire buffer and blocking a small, urgent request on another stream, a form of head-of-line blocking that a single TCP window cannot avoid. Because QUIC runs in user space, applications can also tune these credits directly instead of relying on kernel settings.

Pitfalls in flow control behavior

Flow control problems usually appear as low throughput without packet loss, which distinguishes them from congestion problems. Packet captures make them visible in the advertised window values.

  • Silly window syndrome: a receiver advertising tiny windows makes the sender transmit many small segments; receivers delay updates until a reasonable amount of space is free, and Nagle’s algorithm addresses the sender side.
  • Disabled window scaling: a middlebox that strips the option caps every connection at 64 KiB in flight.
  • Slow application reads: a busy application leaves data in the buffer, the window closes and the sender pauses.
  • Fixed small buffers: manually set socket buffer sizes disable autotuning and limit long-distance transfers.

In packet analyzers such as Wireshark, the warnings TCP ZeroWindow and TCP Window Full indicate that the receiver, not the network, is the bottleneck. The correct fix is then on the receiving host: a faster consumer, larger buffers, or working window scaling.

Understanding it simply

Imagine pouring water into a glass while someone drinks from it. If you pour faster than they drink, the glass overflows. Flow control is the drinker telling you how much room is left in the glass, so you pour only that much and then wait.

Computers do the same with data. The receiving computer keeps announcing its free space, and the sender never sends more than the receiver can hold. Nothing is wasted, and a slow receiver is never flooded.

Applied in practice

A media company copies nightly backups of about 2 TB from its data center in Frankfurt to a recovery site in Virginia over a 1 Gbit/s link with a round-trip time of about 90 ms. The transfer reaches only around 6 Mbit/s and would take weeks to finish, even though the link shows no packet loss.

A packet capture shows that the receiving server advertises a fixed window of 64 KiB, because a firewall removes the window scale option from the handshake. After the firewall is reconfigured to pass the option and the backup server’s buffers are allowed to autotune up to 16 MiB, a single stream reaches roughly 800 Mbit/s, and the nightly backup fits comfortably within its time window.