Tokens per second

Also known as: token throughput

On this page 7

Tokens per second is the rate at which a language model processes or generates tokens, and it is the central throughput metric in AI infrastructure. It is reported in two very different ways: as per-user decoding speed, which describes how fast text appears for one request, and as aggregate token throughput, which describes how much work an entire inference server completes across all concurrent requests.

The metric matters because it links model quality to user experience and cost. A response streamed at 50 tokens per second reads faster than most people can follow, while an infrastructure team cares more about the total of thousands of tokens per second per GPU, because that number sets the cost per token of a deployment.

What tokens per second actually counts

Tokens per second divides a number of tokens by elapsed time, but every serious benchmark must state which tokens and which time window it uses. A token is a subword unit produced by the model’s tokenizer; in English, one token averages roughly 0.75 words, although the ratio varies by tokenizer, language and content such as code.

Inference has two phases with very different speeds. The prefill phase processes the whole prompt in parallel and is compute-bound, so input tokens per second can reach tens of thousands. The decode phase generates output one token at a time per sequence and is usually memory-bandwidth-bound, because each step must read the model weights and the KV cache from GPU memory.

VariantWhat it measuresWho cares most
Output tokens per second per userDecode speed for one stream after the first tokenProduct and UX teams
Input tokens per secondPrompt processing speed during prefillTeams serving long documents or RAG
Aggregate output throughputTotal generated tokens across all concurrent requestsCapacity planners
Total throughputInput plus output tokens per secondCost and billing analysis
Tokens per second per GPUAggregate throughput normalized by hardwareInfrastructure procurement
Common variants of the tokens per second metric

Per-user speed is closely related to time per output token: 20 milliseconds per token equals 50 tokens per second. It is separate from time to first token, which captures queueing and prefill delay before streaming begins.

Why per-user speed and total throughput pull apart

Per-user speed and aggregate throughput trade against each other through batch size. A GPU decoding a single sequence wastes most of its compute, because it reads gigabytes of weights to produce one token. Batching many sequences reuses each weight read, so aggregate throughput rises sharply while each individual stream slows somewhat.

The table below shows the typical shape of this trade-off for a mid-sized model on one modern data center GPU. The figures are illustrative ranges, since real values depend on model size, precision, sequence length and serving software.

Concurrent requestsPer-user output speedAggregate output throughputMain bottleneck
1High, often 60 to 150 tokens/sEqual to per-user speedMemory bandwidth
8Slightly lowerSeveral times higherMemory bandwidth
64Noticeably lowerOften 10 to 30 times single-userKV cache capacity
256 or moreCan fall below reading speedNear the hardware ceilingCompute and scheduling
Illustrative effect of concurrency on throughput

Operators therefore set latency targets first and then maximize throughput under them, for example requiring at least 30 output tokens per second per user at the 95th percentile. This framing, sometimes called goodput, counts only tokens served within the agreed service level.

Techniques that raise token throughput

Most inference optimization work aims to raise tokens per second without breaking latency targets. Each technique attacks a specific bottleneck, which is why they are usually combined rather than chosen individually.

TechniqueMechanismMain effect
Continuous batchingAdds and removes requests at every decode stepHigher aggregate throughput
PagedAttentionStores the KV cache in fixed-size pagesMore concurrent sequences per GPU
QuantizationUses 8-bit or 4-bit weights and cachesLess memory traffic per token
Speculative decodingA draft model proposes tokens the main model verifiesHigher per-user speed
FlashAttentionFuses attention into memory-efficient kernelsFaster prefill on long contexts
Tensor parallelismSplits each layer across GPUsLower latency for large models
Disaggregated servingRuns prefill and decode on separate hardwareLess interference between phases
Optimization techniques and their effect on tokens per second

Serving engines such as vLLM implement several of these at once. Prompt caching also helps indirectly: reusing a cached prefix skips prefill work, which frees compute for decoding and shortens time to first token.

Measuring tokens per second correctly

A tokens per second figure is meaningful only together with its test conditions. Two numbers from different benchmarks are rarely comparable, and published vendor figures often reflect best-case settings.

  • State the tokenizer, because models with larger vocabularies produce fewer tokens for the same text and therefore look slower per token while being equally fast per word.
  • Report input and output lengths, since long prompts reduce decode speed through larger KV cache reads.
  • Separate time to first token from steady-state decode speed instead of averaging them together.
  • Report percentiles, such as p50 and p95, rather than a single mean.
  • Fix concurrency and hardware, including GPU type, count and numeric precision.

Comparisons across tokenizers are a frequent trap. Normalizing to words or characters per second gives a fairer view when models from different families are compared on the same workload.

Limitations and common misconceptions

Tokens per second is a necessary but never sufficient metric. A system can post excellent throughput while delivering a poor experience or poor answers, and several misconceptions persist among buyers and engineers.

MisconceptionRealityBetter practice
Higher is always better for usersAbove reading speed, gains are hard to notice in chatOptimize time to first token as well
Vendor peak equals production speedPeaks use short prompts and ideal batchingBenchmark with real traffic patterns
Throughput equals qualityAggressive quantization can lower accuracyEvaluate quality at each setting
Reasoning models are slowThey often generate many hidden tokensMeasure time to final answer
Misconceptions about tokens per second

For agentic and reasoning workloads, the relevant figure is often total time to complete a task, because a model may generate thousands of intermediate tokens before producing visible output. In those cases, high tokens per second reduces waiting, but the number of tokens required matters just as much.

In everyday language

Tokens per second tells you how fast an AI model writes. A token is a small piece of a word, so a model that produces 50 tokens per second writes roughly 35 to 40 words every second, faster than most people read.

There are two ways to look at the number. For one person, it is how quickly the answer appears on screen. For the company running the system, it is how much text all its machines produce together, much like a restaurant counting meals per hour rather than how fast one plate arrives. Serving more people at once usually raises the total but makes each answer a little slower.

Seen in the real world

An online retailer runs a customer support assistant on a cluster of eight GPUs. During a holiday sale, traffic triples and users start complaining that answers stall. Monitoring shows that aggregate throughput is high, but per-user output speed has fallen to about 12 tokens per second at the 95th percentile, slower than comfortable reading.

The infrastructure team enables continuous batching with PagedAttention, switches the model weights to 8-bit precision after confirming that answer quality is unchanged on its evaluation set, and adds prompt caching for the long shared system prompt. They also set a service target of at least 30 tokens per second per user. After the changes, the same eight GPUs serve roughly twice as many concurrent conversations while keeping p95 speed above 35 tokens per second, which postpones a planned hardware purchase.