Time to first token

Also known as: TTFT

On this page 7

Time to first token (TTFT) is the latency between the moment a client submits a request to a language model and the moment it receives the first output token of the response. In AI infrastructure, it is the primary metric for perceived responsiveness of streaming applications such as chat assistants, coding tools and voice agents.

TTFT is dominated by queueing and the prefill phase, in which the model processes the entire prompt before it can emit anything. It is therefore distinct from time per output token, which governs how fast the rest of the answer streams, and both metrics are reported together in serving benchmarks and service level objectives.

What makes up time to first token

Time to first token is the sum of several sequential delays, and only part of it is model computation. Measured from the client, it includes network transfer, request handling in the gateway, waiting in the scheduler queue, tokenization, the prefill forward pass and the return of the first token over the stream.

The following table breaks TTFT into its components and shows what typically controls each one. The ranges are indicative and vary strongly with model size, hardware and load.

ComponentWhat happensMain driverTypical share
Network round tripRequest and first chunk travel between client and serverGeographic distanceTens of milliseconds
Gateway and routingAuthentication, rate limiting, model routingProxy designA few milliseconds
Queueing delayRequest waits for a scheduler slotServer loadZero to several seconds
TokenizationText is converted to token IDsPrompt lengthUsually negligible
Prefill computationAll prompt tokens pass through the modelPrompt length and model sizeOften the largest share
First decode stepThe first output token is sampledModel sizeOne decode iteration
Components of time to first token

Because of this structure, a TTFT figure is meaningful only when the measurement point is stated. Server-side TTFT excludes the network and is useful for capacity planning, while client-side TTFT reflects what users actually experience.

Why the prefill phase dominates

The prefill phase processes every prompt token in parallel to build the KV cache, the stored attention keys and values that later decoding reuses. Prefill is compute-bound: its cost grows roughly linearly with prompt length for the feed-forward layers and quadratically for attention, so a prompt of 32,000 tokens can take tens of times longer to prefill than one of 1,000 tokens.

Decoding, by contrast, generates one token per step and is memory-bandwidth-bound, because each step must read the model weights and the growing cache. This asymmetry explains why long-context workloads such as retrieval-augmented generation and document analysis suffer mainly from high TTFT, while long answers suffer mainly from time per output token.

The next table compares TTFT with the related latency and throughput metrics that serving teams track.

MetricWhat it measuresDominant phaseUser perception
Time to first tokenDelay until the first token arrivesQueueing and prefillHow quickly the reply starts
Time per output tokenAverage gap between later tokensDecodeHow smoothly text streams
End-to-end latencyTTFT plus generation of all tokensBothTotal wait for a complete answer
Tokens per secondGeneration rate per request or per serverDecodeReading speed or capacity
GoodputRequests served within latency targetsBothReliability under load
Time to first token compared with related serving metrics

Typical targets and how they differ by application

Acceptable TTFT depends on how the output is consumed. Interactive interfaces need the first words almost immediately, whereas offline jobs care only about throughput and cost per token.

WorkloadIndicative targetReason
Voice agentsBelow about 300 to 500 msPauses longer than natural turn gaps feel broken
Chat assistantsBelow about 1 sUsers expect the reply to start promptly
Code completion in editorsBelow a few hundred msSuggestions must keep pace with typing
Long-document question answeringSeveral seconds toleratedUsers accept delay for large context
Batch inferenceNot a primary targetThroughput and cost matter more
Indicative time to first token targets by workload

Service level objectives are usually expressed as percentiles, such as p50, p95 and p99, rather than averages. A median of 400 ms with a p99 of 6 seconds indicates queueing problems that an average would hide, so tail latency deserves explicit monitoring in LLM observability dashboards.

Techniques that reduce time to first token

Reducing TTFT means either shortening the wait in the queue or shortening prefill. Most production inference servers, including vLLM and comparable engines, combine several of the following techniques.

TechniqueHow it lowers TTFTTrade-off
Prompt cachingReuses the KV cache of a shared prompt prefixBenefits only repeated prefixes
Chunked prefillSplits long prompts so short requests are not blockedSlightly higher per-token decode latency
Disaggregated servingRuns prefill and decode on separate GPU poolsMore complex orchestration and cache transfer
Tensor parallelismSpreads prefill computation across GPUsInterconnect overhead and higher cost
FlashAttention kernelsComputes attention with less memory trafficHardware and kernel support required
Admission control and autoscalingKeeps queues short under burstsRejected or delayed requests at peak
Optimization techniques and their effect on time to first token

Prompt caching often brings the largest gain for assistants with long system prompts or tool definitions, because a cached prefix of several thousand tokens can cut prefill time by a large fraction of the total. Continuous batching improves throughput but can raise TTFT if the scheduler admits new requests too slowly, so scheduler policy is itself a tuning parameter.

Measurement pitfalls and common misconceptions

TTFT is easy to misread because benchmark conditions rarely match production. The points below summarize errors that frequently distort reported figures.

  • Unloaded benchmarks report TTFT at concurrency of one, which hides queueing delay that dominates under real traffic.
  • Fixed short prompts understate TTFT for retrieval-heavy applications whose prompts are many times longer.
  • Warm caches in repeated tests make prompt caching look universally effective.
  • Empty first chunks sent by some APIs, such as role headers or whitespace, can make TTFT appear lower than the arrival of meaningful text.
  • Reasoning models may generate hidden thinking tokens before visible output, so visible TTFT can be far higher than raw first-token latency.
  • Optimizing TTFT alone can degrade throughput and cost, because smaller batches waste GPU capacity.

A reliable evaluation therefore reports TTFT percentiles together with prompt length distribution, concurrency and hardware, and compares systems under the same request mix.

In plain words

Time to first token is how long you wait before an AI assistant starts answering. Think of ordering at a restaurant: the waiter first has to read your whole order and pass it to the kitchen before the first dish can arrive. A long, complicated order takes longer to get started.

In the same way, an AI model must read your entire question, plus any documents attached to it, before it writes the first word. After that, the rest of the answer flows at its own pace. A short wait for the first word makes the system feel fast, even if the full answer takes a while.

A real-world example

A company runs an internal support assistant on its own GPU cluster. Every request carries a system prompt and tool definitions of about 6,000 tokens plus retrieved policy documents. During the morning peak, monitoring shows a median TTFT of 1.8 seconds and a p95 of 7 seconds, and employees complain that the assistant feels frozen.

The infrastructure team enables prefix caching for the shared system prompt, turns on chunked prefill so long document requests no longer block short questions, and adds one more GPU replica during business hours. After the change, median TTFT falls to about 0.5 seconds and p95 to about 1.6 seconds, while time per output token stays almost unchanged. Complaints stop, and the team adds a p95 TTFT alert at two seconds to catch future regressions.