Time per output token (TPOT), also called inter-token latency, is the average time a large language model takes to generate each token after the first one in a streamed response. In AI infrastructure, it is the standard metric for the speed of the decode phase, and it determines how quickly text appears to a user once the answer has started.
TPOT is usually reported in milliseconds per token and is measured per request, which separates it from aggregate throughput. Together with time to first token (TTFT), it defines the latency a user actually experiences, and serving teams set service level objectives on both.
How time per output token is calculated
TPOT is computed by dividing the decode duration by the number of tokens after the first. The common formula is TPOT = (end-to-end latency minus TTFT) divided by (output tokens minus 1). Excluding the first token isolates steady-state generation speed from prompt processing and queueing.
Some tools report inter-token latency as the distribution of individual gaps between consecutive tokens rather than their mean. The two values coincide on average, but the distribution reveals jitter and stalls that a single mean hides, which is why mature benchmarks publish p50, p90 and p99 values.
The table below shows how the main latency metrics relate to one another for a single request.
| Metric | What it measures | Dominant phase |
|---|---|---|
| Time to first token | Wait until the first token arrives | Queueing and prefill |
| Time per output token | Average gap between later tokens | Decode |
| Inter-token latency | Each individual token gap | Decode, including stalls |
| End-to-end latency | Total time for the full response | Prefill plus decode |
For example, a response of 501 tokens with a TTFT of 400 ms and an end-to-end time of 15.4 seconds has a TPOT of 30 ms, equal to about 33 tokens per second for that user.
Why decoding speed is limited by memory bandwidth
TPOT is governed mainly by memory bandwidth, not raw compute. Each decode step generates one token per sequence, yet the accelerator must read all model weights and the KV cache from high-bandwidth memory to do so, which makes decoding a memory-bound operation at small batch sizes.
A useful lower bound is the model size in bytes divided by memory bandwidth. A 70-billion-parameter model in 16-bit precision holds about 140 GB of weights; on hardware with roughly 3.3 TB/s of bandwidth, a single step therefore needs at least about 40 ms before parallelism. Splitting the model with tensor parallelism across several GPUs divides that traffic and lowers TPOT, at the cost of extra communication.
The following table lists the factors that most influence TPOT and the direction of their effect.
| Factor | Effect on TPOT | Reason |
|---|---|---|
| Model size | Larger model raises TPOT | More weights read per step |
| Weight precision | Lower precision reduces TPOT | Fewer bytes moved per step |
| Batch size | Larger batch raises TPOT | More work and KV cache per step |
| Context length | Longer context raises TPOT | Attention reads a larger KV cache |
| Tensor parallelism | More GPUs usually reduce TPOT | Bandwidth is pooled across devices |
| Speculative decoding | Often reduces effective TPOT | Several tokens accepted per step |
Time per output token versus throughput
TPOT measures the experience of one request, while throughput measures the total tokens per second of the whole system. The two pull in opposite directions: larger batches raise aggregate throughput and lower cost per token, but every sequence in the batch waits for a heavier step, so per-user TPOT rises.
This creates the central latency-throughput trade-off of LLM serving. Operators pick an operating point by setting a TPOT ceiling and then admitting as many concurrent requests as the ceiling allows, a practice often summarized as maximizing goodput within latency limits.
| Concurrent requests | Per-user TPOT | Aggregate throughput | Cost per token |
|---|---|---|---|
| 1 | Lowest | Lowest | Highest |
| 8 | Slightly higher | Several times higher | Much lower |
| 32 | Noticeably higher | High | Low |
| 128 | Highest, may breach targets | Near saturation | Lowest |
Techniques that reduce TPOT in serving systems
Inference engines such as vLLM combine several methods to keep TPOT low under load. Continuous batching inserts and removes sequences at every step, and PagedAttention manages the KV cache in blocks so memory is not wasted. Neither directly speeds a single step, but both let a server hold more users at the same TPOT.
Disaggregated serving runs prefill and decode on separate GPU pools so that long prompts from new requests do not interrupt token generation for existing ones. Without such separation, or chunked prefill, a large incoming prompt can cause visible TPOT spikes for every active stream.
Weight quantization to 8-bit or 4-bit formats and KV cache quantization cut the bytes read per step, while speculative decoding uses a small draft model to propose tokens that the large model verifies in parallel. The latter can lower effective TPOT by roughly 1.5 to 3 times when the acceptance rate is high, and it brings little benefit when drafts are often rejected.
Setting targets and measuring TPOT correctly
A practical TPOT target follows from human reading speed. Most people read about 4 to 8 words per second, roughly 5 to 10 tokens per second, so a TPOT of 50 to 100 ms already feels fluent in chat. Agentic pipelines and code generation, where machines consume the output, often demand far lower TPOT because total task time matters.
| Workload | Typical TPOT target | Main concern |
|---|---|---|
| Interactive chat | 50 to 100 ms | Fluent reading pace |
| Voice assistants | Under 50 ms | Speech synthesis must not stall |
| Code completion | Under 30 ms | Developer flow |
| Agent tool loops | As low as feasible | Many sequential calls |
| Offline batch jobs | Not critical | Throughput and cost |
Common measurement mistakes distort comparisons:
- Including the first token mixes prefill time into TPOT and inflates the value.
- Reporting only the mean hides tail stalls that users notice.
- Testing with one concurrent request gives a TPOT that production load will never match.
- Comparing different tokenizers distorts results, since tokens per word vary between models.
Explained without jargon
When a chatbot answers, the text appears piece by piece. Time per output token is how long you wait between each piece once the answer has started. A smaller number means the words flow faster.
Think of a printer that has warmed up and is now printing line after line. The warm-up is a separate measure. TPOT is the speed of the printing itself. Serving more people at once on the same machine usually slows that printing a little for each of them.
A typical use case
A company runs an internal coding assistant on a cluster of GPUs and sets a service level objective of p90 TPOT under 40 ms. Monitoring shows that during morning peaks the p90 rises to 75 ms. Engineers trace the cause to long code files sent as prompts, whose prefill interrupts generation for everyone else on the same GPUs.
The team enables chunked prefill, quantizes the model weights to 8-bit precision and caps each GPU at 48 concurrent sequences. After the change, peak p90 TPOT falls to about 35 ms while aggregate throughput drops by only a few percent. Developers report smoother completions, and the operators gain a clear rule for when to add capacity: whenever p90 TPOT approaches the 40 ms limit for more than ten minutes.