Test-time compute is the amount of computation a trained model spends at the moment it answers a request, as opposed to the computation spent on training it. In language models, the term covers every technique that deliberately uses more inference work per query to get a better answer: generating long reasoning traces, sampling many candidate answers, verifying them or searching over partial solutions.
The idea is also called inference-time compute or test-time scaling, because accuracy on hard tasks often improves predictably as this budget grows. It shifts part of the cost of capability from a one-time training run to every individual request, which changes how models are designed, priced and deployed.
How test-time compute improves answers
Test-time compute improves answers by giving the model more steps to work through a problem before committing to a final output. An autoregressive model produces one token per forward pass, so a short answer allows only a limited amount of computation; a longer chain of intermediate tokens lets the model decompose the task, check partial results and correct errors.
There are two broad levers. Sequential scaling extends a single line of reasoning, as reasoning models do when they emit hidden or visible reasoning tokens. Parallel scaling produces several independent attempts and then selects or combines them, for example through majority voting or a verifier model that scores each candidate.
The following table summarizes the main mechanisms and what each one actually spends compute on.
| Mechanism | What extra compute buys | Typical cost driver |
|---|---|---|
| Extended reasoning | Longer step-by-step derivation in one pass | Number of reasoning tokens |
| Self-consistency | Several samples, most frequent answer wins | Number of samples |
| Best-of-N with a verifier | Candidates ranked by a reward or checker | Samples plus scoring calls |
| Tree or beam search | Exploration of alternative partial solutions | Branching factor and depth |
| Iterative self-refinement | Draft, critique and revise cycles | Number of revision rounds |
Training compute versus test-time compute
Training compute and test-time compute are two complementary budgets for capability. Training compute is spent once to set the model parameters, while test-time compute is spent on each query and can be adjusted per request. Research on scaling has shown that for some reasoning tasks, a smaller model with a generous inference budget can match a larger model answering directly, although this exchange rate depends heavily on the task.
The table below contrasts the two budgets along the dimensions that matter for engineering decisions.
| Dimension | Training compute | Test-time compute |
|---|---|---|
| When it is spent | Once, before deployment | On every request |
| Who pays | Model developer | Operator or end user per call |
| Adjustability | Fixed after training | Tunable per query or task |
| Main effect | Broad knowledge and skills | Depth of reasoning on one problem |
| Scaling limit | Data, hardware and budget | Latency, cost and verifier quality |
The two budgets interact. Models trained with reinforcement learning on reasoning traces learn to use additional tokens productively, whereas a base model given the same token budget often just rambles. In practice, useful test-time scaling must be trained in rather than simply switched on.
Controlling the inference budget
Operators control test-time compute through explicit budget parameters and through the choice of strategy. Many reasoning-capable models expose a thinking budget or an effort setting that caps the number of reasoning tokens, while the max tokens limit bounds the total output. Parallel methods are controlled by the number of samples and the sampling temperature, which must be high enough to produce diverse candidates.
Choosing the budget is a decision about the value of accuracy against latency and cost. The following decision table shows reasonable starting points.
| Situation | Recommended approach | Reason |
|---|---|---|
| Simple lookup or rewriting | Minimal or no extra reasoning | Extra tokens add cost without accuracy |
| Multi-step math or logic | Extended reasoning with a high budget | Errors compound without intermediate steps |
| Code with available tests | Best-of-N with test execution | Tests act as a reliable verifier |
| Answers with one exact value | Self-consistency voting | Agreement signals likely correctness |
| Interactive chat with tight latency | Small or adaptive budget | Users notice delays of several seconds |
Adaptive allocation is usually better than a fixed budget: easy prompts receive little extra compute and hard prompts receive more. Some systems estimate difficulty first and then route the query, which keeps average cost low without sacrificing accuracy on difficult cases.
Limitations and common misconceptions
Test-time compute has diminishing returns, and gains typically follow a roughly logarithmic curve: each doubling of the budget yields a smaller improvement. Beyond a task-specific point, longer reasoning can even reduce accuracy, because the model overthinks, second-guesses correct steps or drifts away from the question.
Parallel methods are only as good as their selection mechanism. Majority voting fails when the model is consistently wrong, and a learned reward model can be exploited by candidates that look convincing but are incorrect. Verification is strongest where an objective checker exists, such as unit tests or exact numeric answers.
| Mistake | Consequence | Correction |
|---|---|---|
| Maximum budget for every query | High cost and slow responses | Allocate budget by task difficulty |
| Treating reasoning text as an explanation | False confidence in the rationale | Evaluate final answers, not narratives |
| Voting on open-ended text | No meaningful agreement signal | Use a verifier or rubric instead |
| Ignoring context window limits | Truncated reasoning or lost instructions | Reserve space for reasoning tokens |
A further caution concerns chain-of-thought faithfulness: the visible reasoning does not necessarily reflect the internal computation that produced the answer. Extra compute can raise benchmark scores while the written rationale remains an unreliable audit trail.
Why test-time compute matters for deployment
Test-time compute changes the economics of serving models. Because cost scales with tokens generated, a reasoning-heavy request can consume many times the compute of a direct answer, which affects pricing, hardware capacity planning and energy use. Providers therefore often bill reasoning tokens separately or show them in usage reports.
- Latency budgets in user-facing products frequently limit how much reasoning is practical.
- Batch and offline workloads can afford much larger budgets because nobody waits for the reply.
- Evaluation results are only comparable when the inference budget is reported alongside the score.
A non-technical view
Test-time compute is the amount of thinking time a model gets when it answers a question. A student can answer a hard exam question quickly from memory, or take the time to write out each step, check the arithmetic and try a second approach. The second way usually gives a better answer, but it takes longer.
AI systems work the same way. Giving a model more room to reason, or letting it try several answers and pick the best, often makes it more accurate on hard problems. The price is that each answer becomes slower and more expensive, so the extra effort is worth it only when the question is genuinely difficult.
A concrete example
A software team uses a language model to fix failing functions in a large code repository. With a single direct answer per bug, the model produces a passing fix for roughly one bug in three, and engineers must repair the rest by hand.
The team then changes the pipeline to generate eight candidate patches per bug with a moderate reasoning budget and to run the existing unit tests on each candidate. Only patches that pass all tests are proposed for human review. The share of bugs with a usable fix rises substantially, while inference cost per bug grows about eightfold.
Because an engineer hour costs far more than the additional model calls, the team keeps the setup for overnight batch runs and uses a small single-answer budget for interactive coding assistance, where developers expect a reply within a few seconds.