Replication lag

On this page 7

Replication lag is the delay between a change being committed on a primary database and the moment the same change is applied and visible on a replica. In replication, it is the central health metric of any asynchronous primary-replica setup, because it measures how stale the data on a replica is.

Lag is usually expressed in seconds or in bytes of log not yet applied. A healthy system typically stays below one second, but lag can grow to minutes or hours under heavy writes, long transactions or network trouble, and stale reads then become visible to users.

Where replication lag comes from

A change travels through several stages before it appears on a replica, and each stage adds delay. The primary writes it to its log, the log is shipped over the network, the replica stores it, and finally a replica process replays the change against its own data files.

StageWhat happensTypical cause of delay
Log write on primaryCommit recorded in WAL or binary logSlow storage on the primary
Log shippingRecords sent to the replicaNetwork bandwidth or latency
Receive and storeReplica writes records to its relay logReplica disk throughput
Apply or replayChanges executed on replica dataSingle-threaded apply, heavy queries
VisibilityNew data seen by replica queriesConflicts with long-running reads
Stages of replication and their typical delays

In practice, the apply stage is the most common bottleneck. The primary executes many transactions in parallel, but a replica that replays changes on one thread cannot keep up when the write rate exceeds what a single thread can sustain.

How replication lag is measured

Each database system exposes lag differently, and the numbers do not always mean the same thing. Time-based metrics estimate how old the newest applied change is, while position-based metrics count how much log remains to be applied.

SystemIndicatorWhat it measuresCaveat
MySQL and MariaDBSeconds_Behind_Master or Seconds_Behind_SourceAge of the event being appliedReads zero when the receiver is stalled
PostgreSQLpg_stat_replication lag columnsWrite, flush and replay delayMeasured on the primary
PostgreSQL replicanow() minus pg_last_xact_replay_timestamp()Time since last replayed commitGrows on an idle primary
Any systemHeartbeat tableTimestamp written by primary, read on replicaRequires clock synchronization
Lag indicators in common systems

A heartbeat table is the most reliable general method: the primary updates a timestamp every second, and monitoring compares it on the replica. It measures end-to-end lag as users experience it, regardless of how the engine reports internal positions.

Replication lag in synchronous and asynchronous setups

Whether lag exists at all depends on the replication mode. Synchronous modes move the delay into the commit itself, while asynchronous modes leave it on the replica, where it shows up as stale data.

ModeCommit waits forLag visible to readersCost
AsynchronousPrimary onlyYes, unboundedLowest commit latency
Semi-synchronousReplica receipt of the logYes, apply lag remainsOne network round trip
Synchronous with remote applyReplica apply of the changeNo for that replicaHighest commit latency
Virtually synchronous clusterCertification on all nodesSmall apply delay possibleLatency of slowest node
Replication modes and their relation to lag

Semi-synchronous replication is often misunderstood: it protects against data loss on failover, but a replica can still serve stale reads because receiving the log is not the same as applying it.

Reducing and living with replication lag

Lag is reduced at its source and tolerated in the application. On the database side, the goal is a replica that applies faster than the primary writes; on the application side, the goal is to route reads that must see fresh data to the primary.

CauseRemedyEffect
Single-threaded applyEnable parallel replication workersApply keeps pace with parallel writes
Huge transactionsSplit bulk updates into small batchesAvoids long apply stalls
Underpowered replicaMatch primary CPU and storageRemoves hardware bottleneck
Missing primary keysAdd keys to replicated tablesRow events apply without table scans
Read-your-writes failuresRoute the user to the primary after a writeUsers see their own changes
Long reporting queriesUse a dedicated read replicaConflicts do not delay other replicas
Remedies by cause

Common mistakes

  • Trusting a zero lag reading without checking that the replication threads are actually running.
  • Sending writes and immediate reads to different servers, which shows users outdated data after they save something.
  • Promoting a lagging replica during failover, which silently discards transactions that never arrived.
  • Running schema changes such as large ALTER TABLE operations that replay serially and block everything behind them.
  • Ignoring clock drift when comparing timestamps between primary and replica.

In everyday language

Picture a main notebook and a copy that an assistant keeps up to date by reading over the writer’s shoulder. If the writer works faster than the assistant can copy, the copy falls behind. That gap is replication lag.

Anyone reading the copy may see an older version of the truth: a balance before the latest payment, or a profile before the latest edit. A small gap of a fraction of a second is normal. A growing gap is a warning sign that the copy cannot keep up.

Seen in the real world

A news website sends all reads to two read replicas and all writes to one primary. During an evening data import, editors notice that articles they just published do not appear on the site for several minutes. Monitoring based on a heartbeat table shows the replicas falling over 300 seconds behind.

Investigation finds that the import updates millions of rows in one large transaction, which each replica replays on a single thread. The team splits the import into batches of 5,000 rows, enables parallel replication workers on the replicas, and routes editors’ own page views to the primary for a short time after they publish. Lag during imports now stays under two seconds, and editors see their changes immediately.