Tokens per second is the rate at which a language model processes or generates tokens, and it is the central throughput metric in AI infrastructure. It is reported in two very different ways: as per-user decoding speed, which describes how fast text appears for one request, and as aggregate token throughput, which describes how much work an entire inference server completes across all concurrent requests.
The metric matters because it links model quality to user experience and cost. A response streamed at 50 tokens per second reads faster than most people can follow, while an infrastructure team cares more about the total of thousands of tokens per second per GPU, because that number sets the cost per token of a deployment.
What tokens per second actually counts
Tokens per second divides a number of tokens by elapsed time, but every serious benchmark must state which tokens and which time window it uses. A token is a subword unit produced by the model’s tokenizer; in English, one token averages roughly 0.75 words, although the ratio varies by tokenizer, language and content such as code.
Inference has two phases with very different speeds. The prefill phase processes the whole prompt in parallel and is compute-bound, so input tokens per second can reach tens of thousands. The decode phase generates output one token at a time per sequence and is usually memory-bandwidth-bound, because each step must read the model weights and the KV cache from GPU memory.
| Variant | What it measures | Who cares most |
|---|---|---|
| Output tokens per second per user | Decode speed for one stream after the first token | Product and UX teams |
| Input tokens per second | Prompt processing speed during prefill | Teams serving long documents or RAG |
| Aggregate output throughput | Total generated tokens across all concurrent requests | Capacity planners |
| Total throughput | Input plus output tokens per second | Cost and billing analysis |
| Tokens per second per GPU | Aggregate throughput normalized by hardware | Infrastructure procurement |
Per-user speed is closely related to time per output token: 20 milliseconds per token equals 50 tokens per second. It is separate from time to first token, which captures queueing and prefill delay before streaming begins.
Why per-user speed and total throughput pull apart
Per-user speed and aggregate throughput trade against each other through batch size. A GPU decoding a single sequence wastes most of its compute, because it reads gigabytes of weights to produce one token. Batching many sequences reuses each weight read, so aggregate throughput rises sharply while each individual stream slows somewhat.
The table below shows the typical shape of this trade-off for a mid-sized model on one modern data center GPU. The figures are illustrative ranges, since real values depend on model size, precision, sequence length and serving software.
| Concurrent requests | Per-user output speed | Aggregate output throughput | Main bottleneck |
|---|---|---|---|
| 1 | High, often 60 to 150 tokens/s | Equal to per-user speed | Memory bandwidth |
| 8 | Slightly lower | Several times higher | Memory bandwidth |
| 64 | Noticeably lower | Often 10 to 30 times single-user | KV cache capacity |
| 256 or more | Can fall below reading speed | Near the hardware ceiling | Compute and scheduling |
Operators therefore set latency targets first and then maximize throughput under them, for example requiring at least 30 output tokens per second per user at the 95th percentile. This framing, sometimes called goodput, counts only tokens served within the agreed service level.
Techniques that raise token throughput
Most inference optimization work aims to raise tokens per second without breaking latency targets. Each technique attacks a specific bottleneck, which is why they are usually combined rather than chosen individually.
| Technique | Mechanism | Main effect |
|---|---|---|
| Continuous batching | Adds and removes requests at every decode step | Higher aggregate throughput |
| PagedAttention | Stores the KV cache in fixed-size pages | More concurrent sequences per GPU |
| Quantization | Uses 8-bit or 4-bit weights and caches | Less memory traffic per token |
| Speculative decoding | A draft model proposes tokens the main model verifies | Higher per-user speed |
| FlashAttention | Fuses attention into memory-efficient kernels | Faster prefill on long contexts |
| Tensor parallelism | Splits each layer across GPUs | Lower latency for large models |
| Disaggregated serving | Runs prefill and decode on separate hardware | Less interference between phases |
Serving engines such as vLLM implement several of these at once. Prompt caching also helps indirectly: reusing a cached prefix skips prefill work, which frees compute for decoding and shortens time to first token.
Measuring tokens per second correctly
A tokens per second figure is meaningful only together with its test conditions. Two numbers from different benchmarks are rarely comparable, and published vendor figures often reflect best-case settings.
- State the tokenizer, because models with larger vocabularies produce fewer tokens for the same text and therefore look slower per token while being equally fast per word.
- Report input and output lengths, since long prompts reduce decode speed through larger KV cache reads.
- Separate time to first token from steady-state decode speed instead of averaging them together.
- Report percentiles, such as p50 and p95, rather than a single mean.
- Fix concurrency and hardware, including GPU type, count and numeric precision.
Comparisons across tokenizers are a frequent trap. Normalizing to words or characters per second gives a fairer view when models from different families are compared on the same workload.
Limitations and common misconceptions
Tokens per second is a necessary but never sufficient metric. A system can post excellent throughput while delivering a poor experience or poor answers, and several misconceptions persist among buyers and engineers.
| Misconception | Reality | Better practice |
|---|---|---|
| Higher is always better for users | Above reading speed, gains are hard to notice in chat | Optimize time to first token as well |
| Vendor peak equals production speed | Peaks use short prompts and ideal batching | Benchmark with real traffic patterns |
| Throughput equals quality | Aggressive quantization can lower accuracy | Evaluate quality at each setting |
| Reasoning models are slow | They often generate many hidden tokens | Measure time to final answer |
For agentic and reasoning workloads, the relevant figure is often total time to complete a task, because a model may generate thousands of intermediate tokens before producing visible output. In those cases, high tokens per second reduces waiting, but the number of tokens required matters just as much.
In everyday language
Tokens per second tells you how fast an AI model writes. A token is a small piece of a word, so a model that produces 50 tokens per second writes roughly 35 to 40 words every second, faster than most people read.
There are two ways to look at the number. For one person, it is how quickly the answer appears on screen. For the company running the system, it is how much text all its machines produce together, much like a restaurant counting meals per hour rather than how fast one plate arrives. Serving more people at once usually raises the total but makes each answer a little slower.
Seen in the real world
An online retailer runs a customer support assistant on a cluster of eight GPUs. During a holiday sale, traffic triples and users start complaining that answers stall. Monitoring shows that aggregate throughput is high, but per-user output speed has fallen to about 12 tokens per second at the 95th percentile, slower than comfortable reading.
The infrastructure team enables continuous batching with PagedAttention, switches the model weights to 8-bit precision after confirming that answer quality is unchanged on its evaluation set, and adds prompt caching for the long shared system prompt. They also set a service target of at least 30 tokens per second per user. After the changes, the same eight GPUs serve roughly twice as many concurrent conversations while keeping p95 speed above 35 tokens per second, which postpones a planned hardware purchase.