A GPU cluster is a set of servers equipped with graphics processing units, linked by a high-bandwidth, low-latency network and managed as one pool of compute, so that a single AI workload can run across many accelerators at once. In AI infrastructure, GPU clusters are the standard platform for distributed training of large models and for serving models too large or too busy for one machine.
The defining property of a cluster is not the number of GPUs but how efficiently they cooperate. Communication between GPUs, storage throughput, scheduling and fault tolerance determine whether adding hardware actually shortens training time, which is why cluster design is as much a networking discipline as a computing one.
How a GPU cluster is built
A GPU cluster is organized as a hierarchy of interconnects. Inside each node, typically with four or eight GPUs, a fast local fabric connects the accelerators; between nodes, a separate network links each GPU to the rest of the cluster through dedicated network adapters.
| Component | Role | Typical form |
|---|---|---|
| GPU node | Holds the accelerators and host CPUs | Server with 4 or 8 GPUs |
| Intra-node interconnect | Links GPUs inside one server | NVLink or Infinity Fabric |
| Inter-node network | Links GPUs across servers | InfiniBand or RoCE Ethernet |
| Storage tier | Feeds training data and stores checkpoints | Parallel file system or object store |
| Scheduler | Allocates GPUs to jobs | Slurm or Kubernetes |
| Management network | Monitoring, provisioning, control traffic | Separate Ethernet fabric |
Large training clusters use a non-blocking or rail-optimized network topology, in which each GPU in a node attaches to its own switch rail. This keeps the bandwidth between any two GPUs predictable, which matters because collective operations such as all-reduce run at the speed of the slowest link.
Scale tiers and key parameters
GPU clusters range from a few servers in a research lab to tens of thousands of accelerators in dedicated AI data centers. The parameters that matter most are aggregate memory, interconnect bandwidth and power, not only peak floating-point throughput.
| Tier | Approximate GPU count | Typical use |
|---|---|---|
| Single node | 4 to 8 | Fine-tuning, inference of mid-size models |
| Small cluster | 16 to 256 | Research training, multi-node inference |
| Large cluster | 1,000 to 10,000 | Pretraining of large language models |
| Frontier-scale cluster | Above 10,000 | Pretraining of the largest frontier models |
Power and cooling scale with the cluster. A modern eight-GPU training server can draw roughly 10 kW or more, so racks of such servers often require direct liquid cooling and facility power measured in tens or hundreds of megawatts at the largest scale. These figures vary by GPU generation and should be read as orders of magnitude, not specifications.
Parallelism strategies that run on a cluster
A cluster becomes useful only when the workload is split across GPUs with a parallelism strategy. Each strategy divides a different dimension of the work and places different demands on the network, so large jobs usually combine several of them.
| Strategy | What is split | Network demand |
|---|---|---|
| Data parallelism | Training batches across model replicas | Gradient all-reduce each step |
| Tensor parallelism | Individual layer matrices | Very high, kept inside one node |
| Pipeline parallelism | Groups of layers into stages | Moderate, point-to-point |
| Expert parallelism | Experts of a mixture-of-experts model | High all-to-all traffic |
| FSDP or ZeRO sharding | Parameters, gradients and optimizer state | Frequent gather and scatter |
The usual rule is to map the most communication-heavy strategy to the fastest link. Tensor parallelism stays within a node on the intra-node fabric, while data parallelism spans nodes over the slower inter-node network.
Training clusters compared with inference clusters
Training and inference place different priorities on cluster design. Training jobs are long, synchronous and sensitive to any single failure, while inference workloads are many short, independent requests that must meet latency targets.
| Aspect | Training cluster | Inference cluster |
|---|---|---|
| Primary goal | Maximum sustained throughput | Low latency at acceptable cost |
| Network | Full-bandwidth fabric for collectives | Often modest between replicas |
| Failure handling | Checkpoint and restart the job | Route requests to healthy replicas |
| Key metric | Model FLOPs utilization | Tokens per second, time to first token |
| Job shape | Few very large jobs | Many small, elastic workloads |
Inference of very large models increasingly uses multi-node techniques such as disaggregated serving, which separates the prefill phase and the decoding phase onto different GPU pools, so the boundary between the two cluster types is becoming less sharp.
Efficiency limits and operational risks
Adding GPUs does not deliver proportional speed-up. Real clusters lose efficiency to communication overhead, stragglers, data loading stalls and failures, and well-tuned large training runs often report model FLOPs utilization in the range of about 35 to 55 percent.
- Hardware failures scale with size: on clusters with thousands of GPUs, faulty accelerators, memory errors or network links interrupt jobs regularly, so frequent checkpointing is mandatory.
- Network congestion: oversubscribed switches or poor job placement slow every collective operation.
- Storage bottlenecks: slow data pipelines leave GPUs idle while waiting for input.
- Fragmented scheduling: partially used nodes block large jobs that need contiguous allocations.
- Silent data corruption: rare numerical faults can damage a run without obvious errors, so health checks and loss monitoring are part of operations.
A non-technical view
A GPU cluster is many powerful computers wired together so tightly that they can work on one huge job as a team. Modern AI models are too large to learn on a single machine in reasonable time, so the work is divided among hundreds or thousands of graphics chips.
Imagine a large kitchen preparing a banquet. Many cooks speed things up only if they can pass ingredients quickly and stay coordinated. A GPU cluster works the same way: the speed of the connections between chips matters as much as the speed of the chips themselves.
A concrete example
A university research group wants to pretrain a 7-billion-parameter language model on about 1 trillion tokens. On a single eight-GPU server the run would take many months, so the group reserves 64 nodes with 512 GPUs on a shared national research cluster connected by InfiniBand.
The engineers use tensor parallelism within each node and data parallelism with sharded optimizer state across nodes. During the first week, utilization sits near 30 percent because the data loader cannot keep up; moving preprocessed data to the parallel file system raises it above 45 percent. Two node failures occur during the run, and hourly checkpoints limit each loss to less than an hour of compute.
The final run completes in under three weeks, a result the group could not have reached on its own hardware.