Time to first token (TTFT) is the latency between the moment a client submits a request to a language model and the moment it receives the first output token of the response. In AI infrastructure, it is the primary metric for perceived responsiveness of streaming applications such as chat assistants, coding tools and voice agents.
TTFT is dominated by queueing and the prefill phase, in which the model processes the entire prompt before it can emit anything. It is therefore distinct from time per output token, which governs how fast the rest of the answer streams, and both metrics are reported together in serving benchmarks and service level objectives.
What makes up time to first token
Time to first token is the sum of several sequential delays, and only part of it is model computation. Measured from the client, it includes network transfer, request handling in the gateway, waiting in the scheduler queue, tokenization, the prefill forward pass and the return of the first token over the stream.
The following table breaks TTFT into its components and shows what typically controls each one. The ranges are indicative and vary strongly with model size, hardware and load.
| Component | What happens | Main driver | Typical share |
|---|---|---|---|
| Network round trip | Request and first chunk travel between client and server | Geographic distance | Tens of milliseconds |
| Gateway and routing | Authentication, rate limiting, model routing | Proxy design | A few milliseconds |
| Queueing delay | Request waits for a scheduler slot | Server load | Zero to several seconds |
| Tokenization | Text is converted to token IDs | Prompt length | Usually negligible |
| Prefill computation | All prompt tokens pass through the model | Prompt length and model size | Often the largest share |
| First decode step | The first output token is sampled | Model size | One decode iteration |
Because of this structure, a TTFT figure is meaningful only when the measurement point is stated. Server-side TTFT excludes the network and is useful for capacity planning, while client-side TTFT reflects what users actually experience.
Why the prefill phase dominates
The prefill phase processes every prompt token in parallel to build the KV cache, the stored attention keys and values that later decoding reuses. Prefill is compute-bound: its cost grows roughly linearly with prompt length for the feed-forward layers and quadratically for attention, so a prompt of 32,000 tokens can take tens of times longer to prefill than one of 1,000 tokens.
Decoding, by contrast, generates one token per step and is memory-bandwidth-bound, because each step must read the model weights and the growing cache. This asymmetry explains why long-context workloads such as retrieval-augmented generation and document analysis suffer mainly from high TTFT, while long answers suffer mainly from time per output token.
The next table compares TTFT with the related latency and throughput metrics that serving teams track.
| Metric | What it measures | Dominant phase | User perception |
|---|---|---|---|
| Time to first token | Delay until the first token arrives | Queueing and prefill | How quickly the reply starts |
| Time per output token | Average gap between later tokens | Decode | How smoothly text streams |
| End-to-end latency | TTFT plus generation of all tokens | Both | Total wait for a complete answer |
| Tokens per second | Generation rate per request or per server | Decode | Reading speed or capacity |
| Goodput | Requests served within latency targets | Both | Reliability under load |
Typical targets and how they differ by application
Acceptable TTFT depends on how the output is consumed. Interactive interfaces need the first words almost immediately, whereas offline jobs care only about throughput and cost per token.
| Workload | Indicative target | Reason |
|---|---|---|
| Voice agents | Below about 300 to 500 ms | Pauses longer than natural turn gaps feel broken |
| Chat assistants | Below about 1 s | Users expect the reply to start promptly |
| Code completion in editors | Below a few hundred ms | Suggestions must keep pace with typing |
| Long-document question answering | Several seconds tolerated | Users accept delay for large context |
| Batch inference | Not a primary target | Throughput and cost matter more |
Service level objectives are usually expressed as percentiles, such as p50, p95 and p99, rather than averages. A median of 400 ms with a p99 of 6 seconds indicates queueing problems that an average would hide, so tail latency deserves explicit monitoring in LLM observability dashboards.
Techniques that reduce time to first token
Reducing TTFT means either shortening the wait in the queue or shortening prefill. Most production inference servers, including vLLM and comparable engines, combine several of the following techniques.
| Technique | How it lowers TTFT | Trade-off |
|---|---|---|
| Prompt caching | Reuses the KV cache of a shared prompt prefix | Benefits only repeated prefixes |
| Chunked prefill | Splits long prompts so short requests are not blocked | Slightly higher per-token decode latency |
| Disaggregated serving | Runs prefill and decode on separate GPU pools | More complex orchestration and cache transfer |
| Tensor parallelism | Spreads prefill computation across GPUs | Interconnect overhead and higher cost |
| FlashAttention kernels | Computes attention with less memory traffic | Hardware and kernel support required |
| Admission control and autoscaling | Keeps queues short under bursts | Rejected or delayed requests at peak |
Prompt caching often brings the largest gain for assistants with long system prompts or tool definitions, because a cached prefix of several thousand tokens can cut prefill time by a large fraction of the total. Continuous batching improves throughput but can raise TTFT if the scheduler admits new requests too slowly, so scheduler policy is itself a tuning parameter.
Measurement pitfalls and common misconceptions
TTFT is easy to misread because benchmark conditions rarely match production. The points below summarize errors that frequently distort reported figures.
- Unloaded benchmarks report TTFT at concurrency of one, which hides queueing delay that dominates under real traffic.
- Fixed short prompts understate TTFT for retrieval-heavy applications whose prompts are many times longer.
- Warm caches in repeated tests make prompt caching look universally effective.
- Empty first chunks sent by some APIs, such as role headers or whitespace, can make TTFT appear lower than the arrival of meaningful text.
- Reasoning models may generate hidden thinking tokens before visible output, so visible TTFT can be far higher than raw first-token latency.
- Optimizing TTFT alone can degrade throughput and cost, because smaller batches waste GPU capacity.
A reliable evaluation therefore reports TTFT percentiles together with prompt length distribution, concurrency and hardware, and compares systems under the same request mix.
In plain words
Time to first token is how long you wait before an AI assistant starts answering. Think of ordering at a restaurant: the waiter first has to read your whole order and pass it to the kitchen before the first dish can arrive. A long, complicated order takes longer to get started.
In the same way, an AI model must read your entire question, plus any documents attached to it, before it writes the first word. After that, the rest of the answer flows at its own pace. A short wait for the first word makes the system feel fast, even if the full answer takes a while.
A real-world example
A company runs an internal support assistant on its own GPU cluster. Every request carries a system prompt and tool definitions of about 6,000 tokens plus retrieved policy documents. During the morning peak, monitoring shows a median TTFT of 1.8 seconds and a p95 of 7 seconds, and employees complain that the assistant feels frozen.
The infrastructure team enables prefix caching for the shared system prompt, turns on chunked prefill so long document requests no longer block short questions, and adds one more GPU replica during business hours. After the change, median TTFT falls to about 0.5 seconds and p95 to about 1.6 seconds, while time per output token stays almost unchanged. Complaints stop, and the team adds a p95 TTFT alert at two seconds to catch future regressions.