Model serving is the practice of running a trained machine learning model in production so that applications can send it inputs and receive predictions through a stable interface, usually an HTTP or gRPC API. It is the stage of AI infrastructure where a model stops being a file of weights and becomes a service with latency, throughput, availability and cost targets.
In artificial intelligence systems, serving covers far more than loading a model into memory. It includes request batching, hardware scheduling, scaling, version management and monitoring, and for large language models it adds specialized memory management that often decides whether a deployment is economically viable.
How a model serving system handles requests
A model serving system handles requests by receiving inputs at an API gateway, queuing and batching them, running inference on CPUs or accelerators, and returning post-processed results. Each stage adds latency, and well-designed systems measure every stage separately.
| Stage | Function | Typical concern |
|---|---|---|
| Gateway and routing | Authentication, rate limits, model selection | Fair use across clients |
| Preprocessing | Tokenization, image resizing, feature lookup | CPU bottlenecks before the GPU |
| Scheduler and batcher | Groups requests to use hardware efficiently | Balancing throughput against waiting time |
| Inference runtime | Executes the model on the target hardware | Memory capacity and kernel efficiency |
| Postprocessing | Decoding, thresholds, formatting, safety filters | Consistency with training-time logic |
| Observability | Metrics, traces, logs, prediction sampling | Detecting latency spikes and drift |
The central engineering lever is batching. Accelerators are far more efficient on many inputs at once, so servers use dynamic batching, waiting a few milliseconds to collect requests into one batch. For language models, continuous batching goes further and inserts new requests into the running batch at each generation step instead of waiting for the whole batch to finish.
Serving patterns and when each fits
Serving patterns differ in when predictions are computed and how quickly they must arrive. Choosing the pattern before choosing tools avoids paying for real-time infrastructure where an overnight job would do.
| Pattern | How it works | Latency expectation | Typical use |
|---|---|---|---|
| Online real-time | Synchronous API call per request | Milliseconds to a few seconds | Chat assistants, fraud checks |
| Streaming generation | Tokens returned as they are produced | First token within about a second | Interactive LLM interfaces |
| Asynchronous | Request queued, result fetched later | Seconds to minutes | Long documents, video analysis |
| Batch offline | Scheduled job over a whole dataset | Hours | Nightly scoring, embedding backfills |
| Edge or on-device | Model runs on the client hardware | Milliseconds, no network | Mobile vision, offline use |
Capacity planning for online serving follows Little’s law: average concurrency equals throughput multiplied by average latency. A service that must sustain 50 requests per second at an average latency of 2 seconds therefore holds about 100 requests in flight, and every replica must be sized for its share of that load.
Serving large language models
Serving large language models is dominated by memory rather than raw compute, because generation is autoregressive and each request keeps a growing key-value (KV) cache of attention states for all previous tokens. The cache competes with the model weights for accelerator memory and limits how many requests can run at once.
A 7-billion-parameter model with 32 layers, a hidden size of 4,096 and 16-bit values stores keys and values of 2 x 32 x 4,096 x 2 bytes, or about 0.5 MiB per token. A single 4,096-token sequence therefore needs about 2 GiB of KV cache, on top of roughly 14 GB for the weights themselves. On an 80 GB accelerator, this leaves room for only about 30 full-length sequences, which is why KV cache management is a primary design concern.
| Technique | What it changes | Main benefit | Trade-off |
|---|---|---|---|
| Paged KV cache | Allocates cache in fixed blocks | Less fragmentation, more concurrent requests | More complex memory manager |
| Quantization | Weights or cache in 8 or 4 bits | Smaller footprint, often faster | Possible accuracy loss |
| Prefix or prompt caching | Reuses states for repeated prompt prefixes | Lower latency and cost on shared context | Cache invalidation and memory use |
| Speculative decoding | Small draft model proposes tokens to verify | Fewer expensive decoding steps | Gains depend on draft acceptance rate |
| Tensor parallelism | Splits layers across several GPUs | Serves models too large for one device | Interconnect overhead |
LLM latency is reported with two separate metrics: time to first token (TTFT), which reflects processing of the prompt, and time per output token, which determines how fast the answer streams. Optimizing one can worsen the other, so service-level targets should name both.
Model serving versus related infrastructure terms
Model serving is the overall discipline, while several related AI infrastructure terms name individual components or techniques inside it. Keeping them apart clarifies which layer to change when a problem appears.
| Concept | Scope | Role in serving |
|---|---|---|
| Model serving | End-to-end production delivery of predictions | The whole system and its operations |
| Inference server | Software process that hosts models | Executes requests, batches, exposes APIs |
| GPU inference | Running the forward pass on GPUs | The hardware execution layer |
| Model routing | Choosing a model per request | Directs traffic by cost, quality or capability |
| Prompt caching | Reusing computation for repeated prompts | An optimization inside LLM serving |
Operating models reliably in production
Reliable model serving depends on safe release practices and continuous monitoring, because a model can fail silently by returning plausible but wrong predictions while every health check stays green.
- Version every model artifact together with its preprocessing code, so that a rollback restores matching logic.
- Release new models through shadow or canary deployments, comparing outputs on live traffic before a full switch.
- Track tail latency such as p95 and p99, not only averages, because users experience the slowest requests.
- Monitor input and prediction drift, which signals that production data has moved away from the training distribution.
- Set autoscaling on queue depth or GPU utilization, and plan for cold starts, since loading large weights can take minutes.
A frequent mistake is training-serving skew: preprocessing implemented differently in training and in production, for example a different tokenizer version or feature scaling. It produces degraded accuracy with no error messages and is best prevented by sharing one preprocessing implementation across both environments.
The idea in brief
Model serving is how a trained AI model is put to work for real users. Training is like writing a recipe; serving is running the restaurant kitchen that cooks it for hundreds of guests at once, quickly and without mistakes.
The kitchen must group orders efficiently, keep enough cooks on duty for busy hours and notice when something tastes wrong. The key takeaway: a good model is only useful if it is served fast, reliably and at a sustainable cost.
A practical scenario
A software company launches an internal assistant built on a 7-billion-parameter language model and expects about 20 requests per second at peak, with prompts of around 1,500 tokens and answers of about 300 tokens. Its first deployment runs a basic server with static batching on two 80 GB GPUs, and p95 time to first token exceeds 4 seconds at peak load.
The team moves to an inference server with continuous batching and a paged KV cache, enables prefix caching for the shared system prompt and quantizes the weights to 8 bits after confirming no measurable accuracy loss on its evaluation set. With the same two GPUs, p95 time to first token drops to under 1 second, and the service sustains the peak load with headroom. The outcome is that the planned purchase of two additional GPUs is canceled, because the gains came from serving software rather than hardware.