Test-time compute

Also known as: inference-time compute · test-time scaling

On this page 7

Test-time compute is the amount of computation a trained model spends at the moment it answers a request, as opposed to the computation spent on training it. In language models, the term covers every technique that deliberately uses more inference work per query to get a better answer: generating long reasoning traces, sampling many candidate answers, verifying them or searching over partial solutions.

The idea is also called inference-time compute or test-time scaling, because accuracy on hard tasks often improves predictably as this budget grows. It shifts part of the cost of capability from a one-time training run to every individual request, which changes how models are designed, priced and deployed.

How test-time compute improves answers

Test-time compute improves answers by giving the model more steps to work through a problem before committing to a final output. An autoregressive model produces one token per forward pass, so a short answer allows only a limited amount of computation; a longer chain of intermediate tokens lets the model decompose the task, check partial results and correct errors.

There are two broad levers. Sequential scaling extends a single line of reasoning, as reasoning models do when they emit hidden or visible reasoning tokens. Parallel scaling produces several independent attempts and then selects or combines them, for example through majority voting or a verifier model that scores each candidate.

The following table summarizes the main mechanisms and what each one actually spends compute on.

MechanismWhat extra compute buysTypical cost driver
Extended reasoningLonger step-by-step derivation in one passNumber of reasoning tokens
Self-consistencySeveral samples, most frequent answer winsNumber of samples
Best-of-N with a verifierCandidates ranked by a reward or checkerSamples plus scoring calls
Tree or beam searchExploration of alternative partial solutionsBranching factor and depth
Iterative self-refinementDraft, critique and revise cyclesNumber of revision rounds
Main test-time compute mechanisms

Training compute versus test-time compute

Training compute and test-time compute are two complementary budgets for capability. Training compute is spent once to set the model parameters, while test-time compute is spent on each query and can be adjusted per request. Research on scaling has shown that for some reasoning tasks, a smaller model with a generous inference budget can match a larger model answering directly, although this exchange rate depends heavily on the task.

The table below contrasts the two budgets along the dimensions that matter for engineering decisions.

DimensionTraining computeTest-time compute
When it is spentOnce, before deploymentOn every request
Who paysModel developerOperator or end user per call
AdjustabilityFixed after trainingTunable per query or task
Main effectBroad knowledge and skillsDepth of reasoning on one problem
Scaling limitData, hardware and budgetLatency, cost and verifier quality
Comparison of the two compute budgets

The two budgets interact. Models trained with reinforcement learning on reasoning traces learn to use additional tokens productively, whereas a base model given the same token budget often just rambles. In practice, useful test-time scaling must be trained in rather than simply switched on.

Controlling the inference budget

Operators control test-time compute through explicit budget parameters and through the choice of strategy. Many reasoning-capable models expose a thinking budget or an effort setting that caps the number of reasoning tokens, while the max tokens limit bounds the total output. Parallel methods are controlled by the number of samples and the sampling temperature, which must be high enough to produce diverse candidates.

Choosing the budget is a decision about the value of accuracy against latency and cost. The following decision table shows reasonable starting points.

SituationRecommended approachReason
Simple lookup or rewritingMinimal or no extra reasoningExtra tokens add cost without accuracy
Multi-step math or logicExtended reasoning with a high budgetErrors compound without intermediate steps
Code with available testsBest-of-N with test executionTests act as a reliable verifier
Answers with one exact valueSelf-consistency votingAgreement signals likely correctness
Interactive chat with tight latencySmall or adaptive budgetUsers notice delays of several seconds
Choosing a test-time compute strategy

Adaptive allocation is usually better than a fixed budget: easy prompts receive little extra compute and hard prompts receive more. Some systems estimate difficulty first and then route the query, which keeps average cost low without sacrificing accuracy on difficult cases.

Limitations and common misconceptions

Test-time compute has diminishing returns, and gains typically follow a roughly logarithmic curve: each doubling of the budget yields a smaller improvement. Beyond a task-specific point, longer reasoning can even reduce accuracy, because the model overthinks, second-guesses correct steps or drifts away from the question.

Parallel methods are only as good as their selection mechanism. Majority voting fails when the model is consistently wrong, and a learned reward model can be exploited by candidates that look convincing but are incorrect. Verification is strongest where an objective checker exists, such as unit tests or exact numeric answers.

MistakeConsequenceCorrection
Maximum budget for every queryHigh cost and slow responsesAllocate budget by task difficulty
Treating reasoning text as an explanationFalse confidence in the rationaleEvaluate final answers, not narratives
Voting on open-ended textNo meaningful agreement signalUse a verifier or rubric instead
Ignoring context window limitsTruncated reasoning or lost instructionsReserve space for reasoning tokens
Frequent mistakes and how to correct them

A further caution concerns chain-of-thought faithfulness: the visible reasoning does not necessarily reflect the internal computation that produced the answer. Extra compute can raise benchmark scores while the written rationale remains an unreliable audit trail.

Why test-time compute matters for deployment

Test-time compute changes the economics of serving models. Because cost scales with tokens generated, a reasoning-heavy request can consume many times the compute of a direct answer, which affects pricing, hardware capacity planning and energy use. Providers therefore often bill reasoning tokens separately or show them in usage reports.

  • Latency budgets in user-facing products frequently limit how much reasoning is practical.
  • Batch and offline workloads can afford much larger budgets because nobody waits for the reply.
  • Evaluation results are only comparable when the inference budget is reported alongside the score.

A non-technical view

Test-time compute is the amount of thinking time a model gets when it answers a question. A student can answer a hard exam question quickly from memory, or take the time to write out each step, check the arithmetic and try a second approach. The second way usually gives a better answer, but it takes longer.

AI systems work the same way. Giving a model more room to reason, or letting it try several answers and pick the best, often makes it more accurate on hard problems. The price is that each answer becomes slower and more expensive, so the extra effort is worth it only when the question is genuinely difficult.

A concrete example

A software team uses a language model to fix failing functions in a large code repository. With a single direct answer per bug, the model produces a passing fix for roughly one bug in three, and engineers must repair the rest by hand.

The team then changes the pipeline to generate eight candidate patches per bug with a moderate reasoning budget and to run the existing unit tests on each candidate. Only patches that pass all tests are proposed for human review. The share of bugs with a usable fix rises substantially, while inference cost per bug grows about eightfold.

Because an engineer hour costs far more than the additional model calls, the team keeps the setup for overnight batch runs and uses a small single-answer budget for interactive coding assistance, where developers expect a reply within a few seconds.