High availability

On this page 7

High availability (HA) is the property of a server system that keeps a service operational and reachable for an agreed, very high share of time, typically 99.9% or more, despite hardware faults, software crashes and maintenance. In server reliability engineering, it is achieved by eliminating single points of failure, adding redundant components, and detecting and recovering from faults automatically instead of waiting for a person to intervene.

High availability is not a product or a single feature but an architectural outcome. It combines redundancy, failover, health monitoring and disciplined operations, and it is always measured against a stated target, usually expressed as a percentage of uptime over a month or a year.

How high availability is measured

Availability is the fraction of time a service is usable, and it is commonly calculated as MTBF divided by the sum of MTBF and MTTR. Mean time between failures (MTBF) describes how often something breaks, while mean time to repair (MTTR) describes how long it stays broken. This formula shows the key insight of HA design: shortening recovery time raises availability as effectively as making failures rarer.

Targets are expressed in “nines”, and each additional nine cuts the permitted downtime by a factor of ten. The table converts common targets into a downtime budget, using a 365-day year and a 30-day month.

Availability targetDowntime per yearDowntime per 30-day monthTypical use
99% (two nines)3.65 days7.2 hoursInternal tools, batch systems
99.9% (three nines)8.76 hours43.2 minutesBusiness web applications
99.95%4.38 hours21.6 minutesCommon cloud service level agreements
99.99% (four nines)52.56 minutes4.32 minutesPayments, e-commerce checkout
99.999% (five nines)5.26 minutes25.9 secondsTelecom cores, critical infrastructure
Availability targets and the downtime they allow

A crucial detail is what counts as downtime. A service that responds with errors or takes 30 seconds per request is effectively down for its users, so mature teams define availability through service level indicators such as the share of successful requests within a latency threshold, not merely whether a process is running.

Building blocks of a highly available server platform

Every layer of the stack needs a redundant counterpart and a way to switch to it, because the weakest layer sets the availability of the whole service. The table lists the usual techniques by layer.

LayerTypical techniqueWhat it protects against
Power and hardwareDual power supplies, RAID, ECC memoryFailed power supply, disk or memory module
NetworkBonded network interfaces, redundant switches, VRRPFailed cable, port, switch or gateway
ComputeMultiple stateless application nodes behind a load balancerCrashed process or failed server
Load balancingActive-passive balancer pair with a floating IP addressFailure of the balancer itself
DataDatabase replication with automatic promotionLoss of the primary database server
SiteMultiple availability zones or data centersPower, cooling or network outage of one facility
Common high availability techniques by layer

Three mechanisms tie these components together. Health checks probe each node and remove unhealthy ones from service. Failover logic promotes a standby when the active node stops responding. Quorum and fencing ensure that only one node acts as primary, which prevents the dangerous split-brain condition in which two nodes both accept writes and corrupt shared data.

Active-active and active-passive designs

Highly available clusters follow one of two basic patterns: active-active, where all nodes serve traffic, and active-passive, where a standby waits to take over. The choice depends on whether the workload can share state safely and on how much idle capacity the budget allows.

PropertyActive-activeActive-passive
Resource useAll nodes carry loadStandby is mostly idle
Failover timeNear zero for stateless trafficSeconds to minutes for promotion
ComplexityHigh for stateful data, conflicts possibleLower, one writer at a time
Capacity planningSurvivors must absorb the failed node’s loadStandby must match the primary
Typical fitWeb and API tiersRelational databases, legacy services
Comparison of the two main cluster patterns

A frequent planning error in active-active designs is running every node above 50% utilization in a two-node cluster. When one node fails, the survivor receives the full load, saturates and fails as well. The usual rule is N+1 capacity: the cluster must carry peak load with any single node missing.

How redundancy changes the arithmetic

Components placed in series multiply their availabilities, so each additional dependency lowers the total. Two components at 99.9% each in series give 0.999 × 0.999 = about 99.80%, which is worse than either part alone.

Components placed in parallel multiply their unavailabilities. Two independent nodes at 99.9% each have a combined unavailability of 0.001 × 0.001 = 0.000001, which yields a theoretical 99.9999%. In practice the real figure is much lower, because the nodes share common-cause failures such as the same software bug, the same configuration error or the same power feed, and because failover itself is never instantaneous or perfectly reliable.

  • Independence is the hidden assumption behind every parallel calculation; shared racks, shared code and shared operators break it.
  • Failover success rate matters: a standby that fails to take over 5% of the time caps the benefit of redundancy.
  • Dependencies count: DNS, authentication, storage and external APIs sit in series with the service.

Limitations and common mistakes

High availability protects against component failure, not against bad data. If an application deletes records, replication copies the deletion to every replica within milliseconds, so HA never replaces backups. It also differs from disaster recovery, which restores service after the loss of an entire site and is governed by RTO and RPO targets.

  • Untested failover: a standby that has never been promoted in production conditions often fails when needed.
  • Hidden single points of failure: one DNS provider, one certificate, one shared NFS mount or one configuration database.
  • Overly aggressive health checks that trigger failover during brief load spikes and cause flapping.
  • Ignoring planned downtime: upgrades that require stopping the cluster consume the same downtime budget as outages.
  • Chasing extra nines without need: each nine typically raises cost and operational complexity sharply.

In everyday language

High availability means building a service so that it keeps working when a part of it breaks. Instead of relying on one server, the system has spare servers, spare network paths and spare power ready to take over automatically.

Think of a hospital with a backup generator. When the city power fails, the lights flicker and the generator starts on its own, so surgeries continue. The goal is not that nothing ever fails, but that users barely notice when something does.

Seen in the real world

An online retailer sets a target of 99.99% monthly availability for its checkout service, which allows about 4.3 minutes of downtime per month. The team runs four stateless application servers across two availability zones behind a pair of load balancers, and a PostgreSQL primary with a synchronous standby in the second zone. A cluster manager monitors the primary and promotes the standby automatically.

During a sale, the host running the primary database suffers a memory failure. The cluster manager fences the failed node, promotes the standby in about 20 seconds, and the application reconnects. Customers see a short pause on a few requests. With about 20 seconds of disruption in a 30-day month, the service ends the month at about 99.999% availability, well within the target.