Time per output token

Also known as: TPOT · inter-token latency

On this page 7

Time per output token (TPOT), also called inter-token latency, is the average time a large language model takes to generate each token after the first one in a streamed response. In AI infrastructure, it is the standard metric for the speed of the decode phase, and it determines how quickly text appears to a user once the answer has started.

TPOT is usually reported in milliseconds per token and is measured per request, which separates it from aggregate throughput. Together with time to first token (TTFT), it defines the latency a user actually experiences, and serving teams set service level objectives on both.

How time per output token is calculated

TPOT is computed by dividing the decode duration by the number of tokens after the first. The common formula is TPOT = (end-to-end latency minus TTFT) divided by (output tokens minus 1). Excluding the first token isolates steady-state generation speed from prompt processing and queueing.

Some tools report inter-token latency as the distribution of individual gaps between consecutive tokens rather than their mean. The two values coincide on average, but the distribution reveals jitter and stalls that a single mean hides, which is why mature benchmarks publish p50, p90 and p99 values.

The table below shows how the main latency metrics relate to one another for a single request.

MetricWhat it measuresDominant phase
Time to first tokenWait until the first token arrivesQueueing and prefill
Time per output tokenAverage gap between later tokensDecode
Inter-token latencyEach individual token gapDecode, including stalls
End-to-end latencyTotal time for the full responsePrefill plus decode
Latency metrics for a single LLM request

For example, a response of 501 tokens with a TTFT of 400 ms and an end-to-end time of 15.4 seconds has a TPOT of 30 ms, equal to about 33 tokens per second for that user.

Why decoding speed is limited by memory bandwidth

TPOT is governed mainly by memory bandwidth, not raw compute. Each decode step generates one token per sequence, yet the accelerator must read all model weights and the KV cache from high-bandwidth memory to do so, which makes decoding a memory-bound operation at small batch sizes.

A useful lower bound is the model size in bytes divided by memory bandwidth. A 70-billion-parameter model in 16-bit precision holds about 140 GB of weights; on hardware with roughly 3.3 TB/s of bandwidth, a single step therefore needs at least about 40 ms before parallelism. Splitting the model with tensor parallelism across several GPUs divides that traffic and lowers TPOT, at the cost of extra communication.

The following table lists the factors that most influence TPOT and the direction of their effect.

FactorEffect on TPOTReason
Model sizeLarger model raises TPOTMore weights read per step
Weight precisionLower precision reduces TPOTFewer bytes moved per step
Batch sizeLarger batch raises TPOTMore work and KV cache per step
Context lengthLonger context raises TPOTAttention reads a larger KV cache
Tensor parallelismMore GPUs usually reduce TPOTBandwidth is pooled across devices
Speculative decodingOften reduces effective TPOTSeveral tokens accepted per step
Factors that change time per output token

Time per output token versus throughput

TPOT measures the experience of one request, while throughput measures the total tokens per second of the whole system. The two pull in opposite directions: larger batches raise aggregate throughput and lower cost per token, but every sequence in the batch waits for a heavier step, so per-user TPOT rises.

This creates the central latency-throughput trade-off of LLM serving. Operators pick an operating point by setting a TPOT ceiling and then admitting as many concurrent requests as the ceiling allows, a practice often summarized as maximizing goodput within latency limits.

Concurrent requestsPer-user TPOTAggregate throughputCost per token
1LowestLowestHighest
8Slightly higherSeveral times higherMuch lower
32Noticeably higherHighLow
128Highest, may breach targetsNear saturationLowest
Typical effect of batch size on a single GPU deployment

Techniques that reduce TPOT in serving systems

Inference engines such as vLLM combine several methods to keep TPOT low under load. Continuous batching inserts and removes sequences at every step, and PagedAttention manages the KV cache in blocks so memory is not wasted. Neither directly speeds a single step, but both let a server hold more users at the same TPOT.

Disaggregated serving runs prefill and decode on separate GPU pools so that long prompts from new requests do not interrupt token generation for existing ones. Without such separation, or chunked prefill, a large incoming prompt can cause visible TPOT spikes for every active stream.

Weight quantization to 8-bit or 4-bit formats and KV cache quantization cut the bytes read per step, while speculative decoding uses a small draft model to propose tokens that the large model verifies in parallel. The latter can lower effective TPOT by roughly 1.5 to 3 times when the acceptance rate is high, and it brings little benefit when drafts are often rejected.

Setting targets and measuring TPOT correctly

A practical TPOT target follows from human reading speed. Most people read about 4 to 8 words per second, roughly 5 to 10 tokens per second, so a TPOT of 50 to 100 ms already feels fluent in chat. Agentic pipelines and code generation, where machines consume the output, often demand far lower TPOT because total task time matters.

WorkloadTypical TPOT targetMain concern
Interactive chat50 to 100 msFluent reading pace
Voice assistantsUnder 50 msSpeech synthesis must not stall
Code completionUnder 30 msDeveloper flow
Agent tool loopsAs low as feasibleMany sequential calls
Offline batch jobsNot criticalThroughput and cost
Typical TPOT targets by workload

Common measurement mistakes distort comparisons:

  • Including the first token mixes prefill time into TPOT and inflates the value.
  • Reporting only the mean hides tail stalls that users notice.
  • Testing with one concurrent request gives a TPOT that production load will never match.
  • Comparing different tokenizers distorts results, since tokens per word vary between models.

Explained without jargon

When a chatbot answers, the text appears piece by piece. Time per output token is how long you wait between each piece once the answer has started. A smaller number means the words flow faster.

Think of a printer that has warmed up and is now printing line after line. The warm-up is a separate measure. TPOT is the speed of the printing itself. Serving more people at once on the same machine usually slows that printing a little for each of them.

A typical use case

A company runs an internal coding assistant on a cluster of GPUs and sets a service level objective of p90 TPOT under 40 ms. Monitoring shows that during morning peaks the p90 rises to 75 ms. Engineers trace the cause to long code files sent as prompts, whose prefill interrupts generation for everyone else on the same GPUs.

The team enables chunked prefill, quantizes the model weights to 8-bit precision and caps each GPU at 48 concurrent sequences. After the change, peak p90 TPOT falls to about 35 ms while aggregate throughput drops by only a few percent. Developers report smoother completions, and the operators gain a clear rule for when to add capacity: whenever p90 TPOT approaches the 40 ms limit for more than ten minutes.