GPU cloud

Also known as: neocloud

On this page 7

A GPU cloud is a cloud computing service that rents graphics processing units and complete accelerator servers on demand or by reservation, together with the high-bandwidth networking, storage and orchestration needed to train and serve machine learning models. In AI infrastructure, GPU clouds are the main way that organizations obtain large amounts of parallel compute without building their own data centers.

The category includes the GPU offerings of general-purpose hyperscalers and a newer group of specialized providers often called a neocloud, which focus almost entirely on AI workloads. Both sell the same basic resource, but they differ in pricing, network design, software stack and contract terms.

How a GPU cloud delivers accelerated compute

A GPU cloud packages accelerators into rentable units, from a fraction of one GPU up to clusters of thousands, and exposes them through virtual machines, bare-metal servers, containers or managed services. The provider owns the hardware, power, cooling and network, while the customer controls the workload and pays for GPU-hours or reserved capacity.

A typical training node holds eight high-end GPUs connected inside the server by a fast GPU-to-GPU fabric such as NVLink, plus several high-speed network adapters. Nodes are linked by an InfiniBand or RDMA over Converged Ethernet (RoCE) fabric, commonly at 400 Gb/s per GPU, so that collective operations such as all-reduce do not stall the computation.

The main building blocks and their role are shown in the following table.

ComponentFunctionWhy it matters
AcceleratorsRun matrix operations for training and inferenceDetermine raw throughput and memory capacity
Intra-node fabricConnects GPUs inside one serverEnables tensor parallelism at low latency
Cluster networkConnects servers with RDMALimits scaling efficiency of distributed training
Parallel storageFeeds data and stores checkpointsPrevents idle GPUs during data loading
Orchestration layerSchedules jobs with Kubernetes or SlurmControls utilization and multi-tenant isolation
Core components of a GPU cloud

Service models and pricing options

GPU clouds sell capacity in several commercial forms, and the choice usually matters more to total cost than the list price per GPU-hour. Short-term flexibility costs more, while long commitments buy lower rates and guaranteed availability.

ModelHow it worksBest fitMain drawback
On-demandPay per hour with no commitmentExperiments and short jobsHighest rate and limited availability
Reserved capacityCommit for months or yearsSustained training programsPayment continues when GPUs sit idle
Spot or preemptibleUse spare capacity at a discountFault-tolerant batch inferenceInstances can be reclaimed at short notice
Managed inferencePay per request or tokenProduction model servingLess control over hardware and tuning
Dedicated clusterSingle-tenant reserved clusterFrontier model trainingLarge upfront commitment
Common GPU cloud purchasing models

Prices vary with GPU generation, region, contract length and supply. They have changed sharply during periods of scarcity, so any hourly figure should be treated as a snapshot, and buyers should compare the effective cost per useful training step rather than the headline rate.

Hyperscaler GPU clouds compared with neoclouds

Hyperscalers and neoclouds both rent GPUs, but they are built around different priorities. Hyperscalers offer breadth and integration with databases, identity and compliance services, while neoclouds offer AI-specific depth, often with faster access to the newest accelerators and clusters tuned for large training jobs.

AspectHyperscaler GPU cloudNeocloud
Service rangeHundreds of general cloud servicesFocused on compute, storage and AI tooling
Hardware accessBroad catalog, newest parts sometimes limitedOften early access to latest GPUs
Network designVaries by instance familyCommonly non-blocking RDMA fabrics
PricingHigher on-demand rates, many discountsOften lower per-GPU rates for reserved terms
Ecosystem lock-inStrong through managed servicesLower, but fewer adjacent services
Typical differences between provider types

A GPU cloud also differs from an on-premises GPU cluster. Owning hardware can be cheaper at very high, steady utilization, but it requires capital, power capacity and specialist staff, and it carries the risk of hardware obsolescence within a few accelerator generations.

Choosing and operating GPU cloud capacity

Selecting a GPU cloud starts with the workload profile: training needs fast interconnects and large contiguous clusters, while inference needs low latency, autoscaling and good regional coverage. Matching these needs to the offer avoids paying for capabilities that the workload never uses.

WorkloadRecommended setupReason
Large model pretrainingReserved cluster with RDMA fabricCommunication overhead dominates at scale
Fine-tuningOn-demand nodes with 1-8 GPUsShort runs and modest memory needs
Real-time inferenceManaged or serverless inference endpointsAutoscaling absorbs traffic peaks
Offline batch inferenceSpot instances with checkpointingInterruptions are cheap to recover from
Decision guide by workload

Operational quality is measured by model FLOPs utilization, job failure rates and time to recover from a failed node. On large clusters, hardware faults occur regularly, so frequent checkpointing, health checks and automatic job restarts are essential parts of the operating model.

Risks and limitations of GPU clouds

The most common risks are capacity shortages, cost overruns and data movement charges. A GPU cloud removes the burden of owning hardware, but it shifts risk into contracts, availability and engineering discipline.

  • Capacity scarcity: popular GPU types can be unavailable on demand for weeks, delaying projects.
  • Idle reserved capacity: committed GPUs cost money even when no job is scheduled.
  • Egress fees: moving large datasets or checkpoints out of a provider can be expensive.
  • Data residency and compliance: regulated data may require specific regions and certifications.
  • Provider concentration: depending on one supplier exposes training plans to its outages and pricing changes.

Mitigation usually combines a portable software stack, such as containers and open orchestration tools, with capacity planning and clear service-level terms on availability and node replacement.

Stripped to the essentials

A GPU cloud is a way to rent very powerful AI computers over the internet instead of buying them. It works much like renting a professional kitchen by the hour instead of building one at home.

Companies pay for the time they use, or sign longer deals for lower prices. The main benefit is speed and flexibility; the main risks are high bills and not finding free machines when demand is high.

Where it is applied

A mid-sized software company wants to fine-tune an open-weight language model on 40,000 internal support tickets. Buying servers would take months, so the team rents one node with eight GPUs from a GPU cloud on demand.

The fine-tuning job runs for about 14 hours, with checkpoints saved to the provider’s object storage every hour. After evaluation, the team deploys the model on a managed inference endpoint that scales from two GPUs at night to six during business hours.

The outcome is a working assistant in under three weeks, with no hardware purchase. After three months of steady traffic, the company moves the inference capacity to a one-year reservation, which lowers its monthly compute bill while keeping the same performance.