Inference endpoint

Also known as: inference API

On this page 8

An inference endpoint is a network-addressable interface, usually an HTTPS URL, that accepts input data, runs it through a deployed machine learning model and returns the prediction or generated output to the caller. In AI infrastructure it is the boundary where a trained model becomes a consumable service, and it is often called an inference API.

Within artificial intelligence, the endpoint hides everything behind it, including the inference server, the GPU cluster, batching logic and autoscaling. Client applications see only a stable contract: a request schema, a response schema, authentication rules and latency and availability targets.

How an inference endpoint processes a request

An inference endpoint processes a request in a fixed sequence: authenticate, validate, route, compute and respond. A gateway or load balancer receives the call, checks the API key or token, applies rate limits and forwards the payload to a healthy model replica.

The replica runs on an inference server that loads model weights into accelerator memory, tokenizes or preprocesses the input, executes the forward pass and post-processes the result. For large language models, the server separates the prefill phase, which processes the prompt, from the decode phase, which produces output tokens one by one, and it may stream tokens back to the client as they appear.

The table below lists the layers that usually sit behind a single endpoint URL.

LayerTypical functionFailure if missing
API gatewayAuthentication, quotas, request loggingUnauthorized or unmetered access
Load balancerDistributes calls across replicasHot spots and uneven latency
Inference serverLoads model, batches and executes requestsNo model execution at all
Accelerator hardwareGPU or other accelerator computeVery slow CPU-only inference
AutoscalerAdds or removes replicas with demandOverload at peaks, waste at idle
Monitoring stackMetrics, traces and alertsSilent degradation and drift
Layers behind a typical inference endpoint

Types of inference endpoints

Inference endpoints differ mainly in how they are provisioned and how they deliver results. The choice determines cost structure, cold-start behavior and the latency a client can expect.

A dedicated endpoint reserves capacity for one model and one owner. A serverless endpoint scales to zero and bills per request or per token, at the price of possible cold starts while weights load. Shared multi-tenant endpoints offered by model providers expose popular models behind one API, while batch endpoints accept large files and return results hours later at lower cost.

Endpoint typeBilling basisLatency profileTypical use
Dedicated real-timePer instance hourLow and predictableProduction apps with steady traffic
ServerlessPer request or tokenCold starts possibleSpiky or low-volume workloads
Shared provider APIPer tokenVaries with provider loadFast access to hosted models
AsynchronousPer instance or jobSeconds to minutesLong jobs such as video analysis
BatchPer job, often discountedHoursOffline scoring and dataset labeling
Common endpoint types and their trade-offs

Performance metrics for inference endpoints

Endpoint performance is measured by latency, throughput, availability and cost per request, and the relevant latency metric depends on the model type. Classic models report end-to-end latency, while generative models add token-level metrics.

For language model endpoints, time to first token shows how quickly a user sees a response begin, and time per output token determines how fast the rest streams. Teams track tail latency, usually the p95 or p99 percentile, because averages hide the slow requests that users actually notice.

MetricWhat it measuresMain influences
Time to first tokenDelay before output startsPrompt length, queueing, prefill speed
Time per output tokenSpeed of streaming outputMemory bandwidth, batch size, model size
ThroughputRequests or tokens per secondContinuous batching, hardware, parallelism
p99 latencySlowest one percent of requestsLoad spikes, cold starts, long inputs
AvailabilityShare of successful requestsRedundancy, health checks, fallback
Cost per tokenSpend per unit of outputUtilization, quantization, instance price
Key metrics and what influences them

An inference endpoint is the externally visible access point, while model serving and inference servers describe the machinery that fulfills its requests. The distinction matters because one endpoint can front several models, versions or hardware pools.

Model serving is the overall discipline of running models in production. An inference server such as vLLM is the software process that executes the model. An AI gateway sits in front of many endpoints and adds model routing, quotas and model fallback across providers. Model deployment is the act of placing a model version behind an endpoint, typically drawn from a model registry.

Designing and operating an inference endpoint

Good endpoint design starts from the service level objective and works backward to hardware and configuration. A chat interface may need time to first token below about one second, while a nightly scoring job cares only about total cost.

Operators use continuous batching to raise GPU utilization, prompt caching to reuse repeated prefixes and quantization to fit larger models on fewer accelerators. Safe rollout patterns such as canary releases and blue-green deployment send a small share of traffic to a new model version before a full switch.

RequirementRecommended choiceReason
Strict interactive latencyDedicated replicas with warm poolsAvoids cold starts and queueing
Unpredictable low trafficServerless endpointPays only for actual requests
Very large offline datasetBatch endpointLower unit price and no timeouts
High availability targetMulti-zone replicas with fallback modelSurvives zone or model failures
Frequent model updatesVersioned routes with canary trafficLimits impact of a bad release
Configuration choices by requirement

Risks and common mistakes

The most common endpoint problems are underestimated tail latency, uncontrolled cost and weak security. An endpoint that looks fast in a single-user test can degrade sharply under concurrent load when queues build up.

  • Exposed endpoints without authentication invite abuse and unexpected bills.
  • Missing input limits let very long prompts monopolize accelerator memory.
  • No timeout or retry policy turns brief overloads into cascading failures.
  • Unpinned model versions change outputs silently when the provider updates a model.
  • Logging full payloads can store personal data in violation of privacy rules.
  • Ignoring model monitoring hides drift until users report wrong predictions.

Mature teams treat an endpoint like any other production API, with versioned contracts, documented error codes, load tests at expected peak and cost alerts tied to token or request volume.

The idea in brief

An inference endpoint is like a service window for a trained AI model. An app sends a question or a piece of data to a web address. Behind that address, powerful computers run the model. The answer comes back in a moment.

The app never needs to know which machines did the work. It only needs the address, a key and the agreed format. That makes it possible to improve or swap the model without changing the app.

A practical scenario

An online retailer adds an AI assistant that answers product questions on its website. The engineering team deploys a fine-tuned 8-billion-parameter language model behind a dedicated inference endpoint with two GPU replicas in separate zones, protected by API keys and a limit of 4,000 input tokens per request.

During a seasonal sale, traffic triples. The autoscaler adds four replicas, and continuous batching keeps time to first token below 800 milliseconds at p95. When one zone has an outage, the gateway routes requests to the remaining replicas and to a smaller fallback model.

A month later the team releases an improved model version to 5 percent of traffic through a canary route. Metrics show fewer escalations to human support, so the new version is promoted to full traffic without any change to the website code.