AI data center

Also known as: AI factory

On this page 7

An AI data center is a facility designed specifically to house dense clusters of GPUs or other accelerators and to supply the power, cooling and high-bandwidth networking that large-scale model training and inference demand. Within AI infrastructure it is the physical foundation beneath GPU clusters, GPU clouds and model serving platforms, and vendors increasingly market it as an AI factory that turns electricity and data into tokens.

It differs from a conventional enterprise or cloud facility mainly in power density per rack, the shift toward liquid cooling and a network fabric built for collective communication between thousands of accelerators.

What makes an AI data center different

An AI data center differs from a general-purpose facility because its workload is dominated by tightly coupled accelerator jobs rather than many independent virtual machines. A single training run can occupy thousands of GPUs for weeks, and every one of them must exchange gradients or activations at very high speed.

This drives three design changes. Racks draw far more power, often tens of kilowatts and in the newest designs well above 100 kW. Heat must be removed with direct-to-chip or immersion liquid cooling because air alone cannot carry it away efficiently. The network becomes a non-blocking, low-latency fabric where a slow link can stall an entire job.

The table summarizes the contrasts that matter most to operators and planners.

PropertyConventional data centerAI data center
Rack power densityAbout 5-15 kW per rackAbout 30-140 kW per rack
Primary coolingAir with hot and cold aislesDirect-to-chip liquid, sometimes immersion
Network designOversubscribed EthernetNon-blocking InfiniBand or tuned Ethernet
Workload patternMany small independent servicesFew large synchronized jobs
Failure impactOne service degradesA whole training job can stall
Site powerTens of megawatts typicalHundreds of megawatts to gigawatt plans
Conventional data center compared with an AI data center

Core building blocks of an AI data center

An AI data center is built from compute, interconnect, storage, power and cooling layers that must be sized together, because a bottleneck in any one of them leaves expensive accelerators idle. Planners usually think in scalable units of a few hundred to a few thousand GPUs that share a network spine.

LayerTypical componentsRole in AI workloads
ComputeServers with 4-8 accelerators eachRun matrix operations for training and inference
Scale-up interconnectNVLink-class links, switch traysFast memory sharing inside a node or rack
Scale-out networkInfiniBand or RoCE Ethernet, 400-800 Gb/s portsAll-reduce and parallelism across nodes
StorageParallel file systems, NVMe tiersFeed data sets and save checkpoints
Power trainSubstation, UPS, busways, PDUsDeliver stable power at high density
Cooling plantCold plates, CDUs, dry coolers, chillersRemove heat from chips and racks
Main layers and their role

The scale-up domain links accelerators so closely that they behave almost like one large device, which suits tensor parallelism. The scale-out network connects those domains for data parallelism and pipeline parallelism, and its bandwidth determines how efficiently collective operations such as all-reduce complete.

Power and cooling at high density

Power and cooling are the hardest constraints on AI data center growth, often more limiting than chip supply. A large training campus may need hundreds of megawatts, and grid connections in many regions take years to approve and build.

Efficiency is commonly reported as power usage effectiveness (PUE), the ratio of total facility power to IT power. Liquid-cooled AI halls can reach values around 1.1 to 1.3, while older air-cooled sites often sit higher. PUE ignores water use and the carbon intensity of the electricity, so operators increasingly also report water usage effectiveness and the share of carbon-free energy.

The following table shows how cooling methods compare at different densities.

Cooling methodPractical density rangeAdvantagesDrawbacks
Air coolingUp to about 20-30 kWSimple and well understoodFans and airflow limits
Rear-door heat exchangersAbout 30-50 kWRetrofits existing hallsStill relies on server fans
Direct-to-chip liquidAbout 50-140 kWRemoves most heat at the sourcePlumbing and leak management
Immersion coolingAbove 100 kW possibleVery high heat transferSpecial tanks and service procedures
Cooling options by rack density

Training sites compared with inference sites

AI data centers are increasingly specialized for either training or inference, because the two workloads have different priorities. Training rewards maximum interconnect bandwidth and large contiguous clusters, while inference rewards low latency to users, high utilization and cost per token.

AspectTraining-focused siteInference-focused site
Cluster shapeVery large, tightly coupledMany smaller replicas
Location driverCheap, abundant powerProximity to users and networks
Key metricModel FLOPs utilizationTime to first token and cost per token
Reliability approachCheckpointing and fast restartsRedundant replicas and model fallback
Load profileSteady and near fullVariable with user demand
Design priorities for training and inference

Training jobs also create rapid power swings when thousands of GPUs pause together for checkpoints or synchronization. Operators smooth these swings with batteries, power capping or software scheduling so that the grid connection is not stressed.

Risks, limitations and common planning mistakes

The largest risk in building an AI data center is misjudging the pace of hardware change. Accelerator generations arrive every one to two years, and each tends to raise power and cooling requirements, so a hall designed for today’s racks may need costly retrofits before its expected life ends.

  • Underestimating network cost: optics and switches can represent a large share of cluster spending.
  • Ignoring failure rates: at tens of thousands of GPUs, hardware faults occur daily, so automatic job recovery is essential.
  • Overlooking water and permits: local communities increasingly question water use, noise and grid impact.
  • Assuming high utilization: idle accelerators waste capital, so scheduling and MLOps tooling matter as much as hardware.

Claims about the scale of announced AI campuses should be treated with caution, because announced capacity is not delivered capacity. Many projects depend on future grid upgrades, financing and equipment availability.

Put simply

An AI data center is a building full of very powerful computer chips working together to train and run AI systems. Ordinary data centers host websites, email and business software. An AI data center is built for one heavy job: doing enormous amounts of math at once.

Those chips use large amounts of electricity and become very hot, much like a crowd of ovens packed into a small kitchen. So these buildings need huge power connections, water-based cooling and very fast cables linking every chip to the others. The key point is that power and cooling, not just chips, decide how big an AI data center can grow.

In practice

A cloud provider converts one hall of an existing campus to host a cluster of about 4,000 GPUs for customers who fine-tune and serve language models. The original hall was designed for 12 kW air-cooled racks, but the new servers need roughly 100 kW per rack.

The team installs direct-to-chip cold plates, coolant distribution units and a new busway system, and replaces the old aggregation network with a non-blocking 400 Gb/s fabric. Because floor space is no longer the constraint, the hall ends up with fewer than a third of its former racks, yet it draws several times more power.

After commissioning, the hall reaches a PUE of about 1.2. Customer training jobs run with stable throughput, and the provider uses checkpoint-aware scheduling to restart jobs within minutes after a node failure, keeping utilization above 85 percent in the first quarter.