An inference endpoint is a network-addressable interface, usually an HTTPS URL, that accepts input data, runs it through a deployed machine learning model and returns the prediction or generated output to the caller. In AI infrastructure it is the boundary where a trained model becomes a consumable service, and it is often called an inference API.
Within artificial intelligence, the endpoint hides everything behind it, including the inference server, the GPU cluster, batching logic and autoscaling. Client applications see only a stable contract: a request schema, a response schema, authentication rules and latency and availability targets.
How an inference endpoint processes a request
An inference endpoint processes a request in a fixed sequence: authenticate, validate, route, compute and respond. A gateway or load balancer receives the call, checks the API key or token, applies rate limits and forwards the payload to a healthy model replica.
The replica runs on an inference server that loads model weights into accelerator memory, tokenizes or preprocesses the input, executes the forward pass and post-processes the result. For large language models, the server separates the prefill phase, which processes the prompt, from the decode phase, which produces output tokens one by one, and it may stream tokens back to the client as they appear.
The table below lists the layers that usually sit behind a single endpoint URL.
| Layer | Typical function | Failure if missing |
|---|---|---|
| API gateway | Authentication, quotas, request logging | Unauthorized or unmetered access |
| Load balancer | Distributes calls across replicas | Hot spots and uneven latency |
| Inference server | Loads model, batches and executes requests | No model execution at all |
| Accelerator hardware | GPU or other accelerator compute | Very slow CPU-only inference |
| Autoscaler | Adds or removes replicas with demand | Overload at peaks, waste at idle |
| Monitoring stack | Metrics, traces and alerts | Silent degradation and drift |
Types of inference endpoints
Inference endpoints differ mainly in how they are provisioned and how they deliver results. The choice determines cost structure, cold-start behavior and the latency a client can expect.
A dedicated endpoint reserves capacity for one model and one owner. A serverless endpoint scales to zero and bills per request or per token, at the price of possible cold starts while weights load. Shared multi-tenant endpoints offered by model providers expose popular models behind one API, while batch endpoints accept large files and return results hours later at lower cost.
| Endpoint type | Billing basis | Latency profile | Typical use |
|---|---|---|---|
| Dedicated real-time | Per instance hour | Low and predictable | Production apps with steady traffic |
| Serverless | Per request or token | Cold starts possible | Spiky or low-volume workloads |
| Shared provider API | Per token | Varies with provider load | Fast access to hosted models |
| Asynchronous | Per instance or job | Seconds to minutes | Long jobs such as video analysis |
| Batch | Per job, often discounted | Hours | Offline scoring and dataset labeling |
Performance metrics for inference endpoints
Endpoint performance is measured by latency, throughput, availability and cost per request, and the relevant latency metric depends on the model type. Classic models report end-to-end latency, while generative models add token-level metrics.
For language model endpoints, time to first token shows how quickly a user sees a response begin, and time per output token determines how fast the rest streams. Teams track tail latency, usually the p95 or p99 percentile, because averages hide the slow requests that users actually notice.
| Metric | What it measures | Main influences |
|---|---|---|
| Time to first token | Delay before output starts | Prompt length, queueing, prefill speed |
| Time per output token | Speed of streaming output | Memory bandwidth, batch size, model size |
| Throughput | Requests or tokens per second | Continuous batching, hardware, parallelism |
| p99 latency | Slowest one percent of requests | Load spikes, cold starts, long inputs |
| Availability | Share of successful requests | Redundancy, health checks, fallback |
| Cost per token | Spend per unit of output | Utilization, quantization, instance price |
Inference endpoint compared with related concepts
An inference endpoint is the externally visible access point, while model serving and inference servers describe the machinery that fulfills its requests. The distinction matters because one endpoint can front several models, versions or hardware pools.
Model serving is the overall discipline of running models in production. An inference server such as vLLM is the software process that executes the model. An AI gateway sits in front of many endpoints and adds model routing, quotas and model fallback across providers. Model deployment is the act of placing a model version behind an endpoint, typically drawn from a model registry.
Designing and operating an inference endpoint
Good endpoint design starts from the service level objective and works backward to hardware and configuration. A chat interface may need time to first token below about one second, while a nightly scoring job cares only about total cost.
Operators use continuous batching to raise GPU utilization, prompt caching to reuse repeated prefixes and quantization to fit larger models on fewer accelerators. Safe rollout patterns such as canary releases and blue-green deployment send a small share of traffic to a new model version before a full switch.
| Requirement | Recommended choice | Reason |
|---|---|---|
| Strict interactive latency | Dedicated replicas with warm pools | Avoids cold starts and queueing |
| Unpredictable low traffic | Serverless endpoint | Pays only for actual requests |
| Very large offline dataset | Batch endpoint | Lower unit price and no timeouts |
| High availability target | Multi-zone replicas with fallback model | Survives zone or model failures |
| Frequent model updates | Versioned routes with canary traffic | Limits impact of a bad release |
Risks and common mistakes
The most common endpoint problems are underestimated tail latency, uncontrolled cost and weak security. An endpoint that looks fast in a single-user test can degrade sharply under concurrent load when queues build up.
- Exposed endpoints without authentication invite abuse and unexpected bills.
- Missing input limits let very long prompts monopolize accelerator memory.
- No timeout or retry policy turns brief overloads into cascading failures.
- Unpinned model versions change outputs silently when the provider updates a model.
- Logging full payloads can store personal data in violation of privacy rules.
- Ignoring model monitoring hides drift until users report wrong predictions.
Mature teams treat an endpoint like any other production API, with versioned contracts, documented error codes, load tests at expected peak and cost alerts tied to token or request volume.
The idea in brief
An inference endpoint is like a service window for a trained AI model. An app sends a question or a piece of data to a web address. Behind that address, powerful computers run the model. The answer comes back in a moment.
The app never needs to know which machines did the work. It only needs the address, a key and the agreed format. That makes it possible to improve or swap the model without changing the app.
A practical scenario
An online retailer adds an AI assistant that answers product questions on its website. The engineering team deploys a fine-tuned 8-billion-parameter language model behind a dedicated inference endpoint with two GPU replicas in separate zones, protected by API keys and a limit of 4,000 input tokens per request.
During a seasonal sale, traffic triples. The autoscaler adds four replicas, and continuous batching keeps time to first token below 800 milliseconds at p95. When one zone has an outage, the gateway routes requests to the remaining replicas and to a smaller fallback model.
A month later the team releases an improved model version to 5 percent of traffic through a canary route. Metrics show fewer escalations to human support, so the new version is promoted to full traffic without any change to the website code.