Serverless inference

On this page 7

Serverless inference is a deployment model in which a provider runs a trained model on demand, allocates compute only while requests arrive and bills by request, token or execution time rather than by reserved machines. In AI infrastructure, it removes the need to provision, patch and scale GPU or CPU servers, because the platform handles automatic scaling, including scaling to zero when traffic stops.

The approach suits intermittent or unpredictable workloads, but it introduces its own trade-offs: cold starts, limits on model size and runtime, and a per-unit price that can exceed the cost of dedicated capacity at high, steady volume.

How serverless inference executes a request

Serverless inference works by attaching model execution to an event-driven control plane that starts workers when requests arrive and stops them when they go idle. The caller sends a request to an inference endpoint; a router checks whether a warm worker with the model already loaded is available and, if not, schedules a new one.

A new worker must pull a container image, load model weights into memory or GPU memory and initialize the runtime before it can answer. This startup delay is the cold start. Once warm, the worker serves subsequent requests quickly, often with dynamic batching that groups concurrent requests to raise accelerator utilization.

The table below lists the stages of a request and what typically dominates their latency.

StageWhat happensMain latency driver
RoutingGateway authenticates and selects a workerNetwork and queueing
ProvisioningPlatform allocates a container or microVMCapacity availability
Model loadingWeights move from storage into memoryModel size and storage bandwidth
Warm-upRuntime compiles kernels and allocates cachesFramework and graph compilation
ExecutionModel computes the prediction or tokensModel size and batch size
Scale-downIdle worker is released after a timeoutPlatform idle policy
Stages of a serverless inference request

Key parameters that shape cost and latency

The behavior of a serverless deployment is controlled by a small set of parameters that trade responsiveness against cost. Most platforms expose them directly, and tuning them is the main lever an engineer has.

ParameterTypical settingEffect when increased
Minimum instances0 to a fewFewer cold starts, higher idle cost
Maximum concurrency1 to dozens per workerBetter utilization, possible queueing
Idle timeoutSeconds to minutesLonger warm periods, higher cost
Memory or GPU classFrom CPU to large GPUsFaster execution, higher unit price
Maximum instancesSet by quota or budgetHigher peak throughput, cost exposure
Configuration parameters and their effects

Cold start duration varies widely. A small CPU model may start in well under a second, whereas a large language model with tens of gigabytes of weights can take tens of seconds to minutes, depending on storage throughput, caching of images and weights, and whether the platform keeps snapshots of initialized memory.

Serverless inference compared with other serving models

Serverless inference is one of several ways to perform model serving, and the right choice depends mainly on traffic shape and latency requirements. Dedicated endpoints and batch inference remain better fits for many workloads.

Serving modelBilling basisLatency profileBest fit
Serverless inferencePer request, token or secondCold starts possibleSpiky or low traffic
Dedicated endpointPer reserved instance hourStable and predictableSteady high traffic
Batch inferencePer job or compute hourMinutes to hoursOffline scoring of large datasets
On-device AIHardware already ownedNo network round tripPrivacy and offline use
Serving options compared

Two variants are common for large language models. Shared multi-tenant APIs serve a popular model to many customers from a pool that is always warm and charge per token, so cold starts are invisible to the user. Serverless custom endpoints run the customer’s own model or fine-tuned weights and are therefore more exposed to cold starts.

When to choose serverless inference

Serverless inference is the right choice when utilization of dedicated hardware would be low. A GPU that sits idle most of the day costs the same per hour whether it serves one request or thousands, so paying only for active time saves money below a certain break-even utilization.

The following decision table summarizes common situations.

SituationRecommended approachReason
Prototype or internal toolServerlessNear-zero cost when unused
Unpredictable traffic spikesServerless with maximum capElastic capacity, bounded spend
Strict latency target under 200 msServerless with warm minimum, or dedicatedAvoids cold start penalties
Sustained high request volumeDedicated or reserved capacityLower cost per prediction
Nightly scoring of millions of rowsBatch inferenceThroughput matters more than latency
Decision guide for serverless deployment

Teams usually estimate the break-even point by comparing the monthly cost of a reserved instance with the projected per-request bill. Many end with a hybrid pattern: a small dedicated baseline plus serverless overflow for peaks.

Limitations and common mistakes

Serverless inference hides servers but not physics: the model still has to be loaded, and limits imposed by the platform shape what can run. The most frequent problems come from ignoring these constraints.

  • Underestimating cold starts: user-facing applications that scale to zero may show multi-second delays on the first request after an idle period.
  • Exceeding platform limits: maximum memory, container size, request duration or payload size can rule out large models or long generations.
  • Ignoring cost at scale: per-unit pricing that looks cheap for a pilot can become more expensive than reserved GPUs at sustained volume.
  • Weak observability: short-lived workers complicate tracing, so teams need model monitoring and logging designed for ephemeral instances.
  • State assumptions: workers are stateless, so caches such as a KV cache or session data do not persist between invocations unless stored externally.

Mitigations include provisioned or minimum warm instances, smaller or quantized models produced by model compression, weight caching close to compute and budget alerts tied to maximum instance counts.

In simple terms

Serverless inference means renting an AI model’s brainpower only for the moments you actually use it. You do not buy or run any computers. You send a question, the provider wakes up a machine, gets the answer and puts the machine back to sleep.

It works like a taxi instead of owning a car. For occasional trips it is cheaper and simpler, but if you drive all day every day, owning the car wins, and sometimes you wait a little for the taxi to arrive.

A practical example

A mid-sized insurance company builds a tool that reads scanned claim forms and extracts policy numbers, dates and amounts. Claims arrive unevenly: a few dozen per hour most of the time, but several thousand within a day after a major storm.

The team deploys the document model as a serverless endpoint with a minimum of one warm instance during business hours and zero overnight. On quiet days the bill stays small, and after a storm the platform scales to dozens of workers without manual intervention. Overnight requests occasionally wait about ten seconds for a cold start, which is acceptable for a back-office process. The outcome is that the company pays for capacity roughly in proportion to claim volume instead of keeping idle GPUs reserved for rare peaks.