GPU cluster

On this page 7

A GPU cluster is a set of servers equipped with graphics processing units, linked by a high-bandwidth, low-latency network and managed as one pool of compute, so that a single AI workload can run across many accelerators at once. In AI infrastructure, GPU clusters are the standard platform for distributed training of large models and for serving models too large or too busy for one machine.

The defining property of a cluster is not the number of GPUs but how efficiently they cooperate. Communication between GPUs, storage throughput, scheduling and fault tolerance determine whether adding hardware actually shortens training time, which is why cluster design is as much a networking discipline as a computing one.

How a GPU cluster is built

A GPU cluster is organized as a hierarchy of interconnects. Inside each node, typically with four or eight GPUs, a fast local fabric connects the accelerators; between nodes, a separate network links each GPU to the rest of the cluster through dedicated network adapters.

ComponentRoleTypical form
GPU nodeHolds the accelerators and host CPUsServer with 4 or 8 GPUs
Intra-node interconnectLinks GPUs inside one serverNVLink or Infinity Fabric
Inter-node networkLinks GPUs across serversInfiniBand or RoCE Ethernet
Storage tierFeeds training data and stores checkpointsParallel file system or object store
SchedulerAllocates GPUs to jobsSlurm or Kubernetes
Management networkMonitoring, provisioning, control trafficSeparate Ethernet fabric
Main building blocks of a GPU cluster

Large training clusters use a non-blocking or rail-optimized network topology, in which each GPU in a node attaches to its own switch rail. This keeps the bandwidth between any two GPUs predictable, which matters because collective operations such as all-reduce run at the speed of the slowest link.

Scale tiers and key parameters

GPU clusters range from a few servers in a research lab to tens of thousands of accelerators in dedicated AI data centers. The parameters that matter most are aggregate memory, interconnect bandwidth and power, not only peak floating-point throughput.

TierApproximate GPU countTypical use
Single node4 to 8Fine-tuning, inference of mid-size models
Small cluster16 to 256Research training, multi-node inference
Large cluster1,000 to 10,000Pretraining of large language models
Frontier-scale clusterAbove 10,000Pretraining of the largest frontier models
Typical GPU cluster scale tiers

Power and cooling scale with the cluster. A modern eight-GPU training server can draw roughly 10 kW or more, so racks of such servers often require direct liquid cooling and facility power measured in tens or hundreds of megawatts at the largest scale. These figures vary by GPU generation and should be read as orders of magnitude, not specifications.

Parallelism strategies that run on a cluster

A cluster becomes useful only when the workload is split across GPUs with a parallelism strategy. Each strategy divides a different dimension of the work and places different demands on the network, so large jobs usually combine several of them.

StrategyWhat is splitNetwork demand
Data parallelismTraining batches across model replicasGradient all-reduce each step
Tensor parallelismIndividual layer matricesVery high, kept inside one node
Pipeline parallelismGroups of layers into stagesModerate, point-to-point
Expert parallelismExperts of a mixture-of-experts modelHigh all-to-all traffic
FSDP or ZeRO shardingParameters, gradients and optimizer stateFrequent gather and scatter
Parallelism strategies and their network demands

The usual rule is to map the most communication-heavy strategy to the fastest link. Tensor parallelism stays within a node on the intra-node fabric, while data parallelism spans nodes over the slower inter-node network.

Training clusters compared with inference clusters

Training and inference place different priorities on cluster design. Training jobs are long, synchronous and sensitive to any single failure, while inference workloads are many short, independent requests that must meet latency targets.

AspectTraining clusterInference cluster
Primary goalMaximum sustained throughputLow latency at acceptable cost
NetworkFull-bandwidth fabric for collectivesOften modest between replicas
Failure handlingCheckpoint and restart the jobRoute requests to healthy replicas
Key metricModel FLOPs utilizationTokens per second, time to first token
Job shapeFew very large jobsMany small, elastic workloads
Design priorities for training and inference clusters

Inference of very large models increasingly uses multi-node techniques such as disaggregated serving, which separates the prefill phase and the decoding phase onto different GPU pools, so the boundary between the two cluster types is becoming less sharp.

Efficiency limits and operational risks

Adding GPUs does not deliver proportional speed-up. Real clusters lose efficiency to communication overhead, stragglers, data loading stalls and failures, and well-tuned large training runs often report model FLOPs utilization in the range of about 35 to 55 percent.

  • Hardware failures scale with size: on clusters with thousands of GPUs, faulty accelerators, memory errors or network links interrupt jobs regularly, so frequent checkpointing is mandatory.
  • Network congestion: oversubscribed switches or poor job placement slow every collective operation.
  • Storage bottlenecks: slow data pipelines leave GPUs idle while waiting for input.
  • Fragmented scheduling: partially used nodes block large jobs that need contiguous allocations.
  • Silent data corruption: rare numerical faults can damage a run without obvious errors, so health checks and loss monitoring are part of operations.

A non-technical view

A GPU cluster is many powerful computers wired together so tightly that they can work on one huge job as a team. Modern AI models are too large to learn on a single machine in reasonable time, so the work is divided among hundreds or thousands of graphics chips.

Imagine a large kitchen preparing a banquet. Many cooks speed things up only if they can pass ingredients quickly and stay coordinated. A GPU cluster works the same way: the speed of the connections between chips matters as much as the speed of the chips themselves.

A concrete example

A university research group wants to pretrain a 7-billion-parameter language model on about 1 trillion tokens. On a single eight-GPU server the run would take many months, so the group reserves 64 nodes with 512 GPUs on a shared national research cluster connected by InfiniBand.

The engineers use tensor parallelism within each node and data parallelism with sharded optimizer state across nodes. During the first week, utilization sits near 30 percent because the data loader cannot keep up; moving preprocessed data to the parallel file system raises it above 45 percent. Two node failures occur during the run, and hourly checkpoints limit each loss to less than an hour of compute.

The final run completes in under three weeks, a result the group could not have reached on its own hardware.