A thinking budget is the maximum number of reasoning tokens that a language model may generate in its internal deliberation before it writes the visible answer. In artificial intelligence practice, and especially among language models built for step-by-step reasoning, the thinking budget is the main dial for test-time compute: a larger budget lets the model explore, check and revise its work, while a smaller one keeps latency and cost under control.
The concept is also called a reasoning budget. It is usually exposed as an API parameter measured in tokens or as a coarse effort level, and it applies to the hidden or summarized reasoning trace rather than to the final response, which is limited separately by max tokens.
How a thinking budget controls reasoning
A thinking budget works by limiting the length of the reasoning phase that precedes the answer. A reasoning model first produces reasoning tokens, a chain-of-thought that may be hidden, summarized or shown, and only then emits the response. The serving system counts these tokens and, when the budget is reached, either forces a transition to the answer or relies on the model having learned to stop earlier.
Two enforcement styles exist. A hard cap truncates thinking at the limit and injects an end-of-thinking marker, so the model must answer with whatever partial analysis it has. A soft target is learned during training: the model is rewarded for solving tasks within a given budget, so it adapts its depth to the allowance and often uses fewer tokens than permitted on easy prompts.
The parameters below describe the controls that providers typically expose. Exact names and limits differ between vendors and change between model versions.
| Parameter | Typical form | What it affects |
|---|---|---|
| Token budget | Integer, often 1,024 to 32,000+ | Upper bound on reasoning tokens |
| Effort level | Low, medium or high | Provider-defined depth of reasoning |
| Thinking on or off | Boolean switch | Whether a reasoning phase runs at all |
| Max tokens | Integer | Combined or answer-only output limit |
| Summary visibility | Hidden, summarized or full | What the caller can inspect |
Why the thinking budget changes accuracy
Extra reasoning tokens improve results because the model can decompose a problem, test intermediate results and backtrack from errors. On mathematics, coding and multi-step planning tasks, accuracy usually rises roughly with the logarithm of the budget: each doubling helps, but by a smaller margin than the previous one.
The curve eventually flattens or even declines. Past a task-dependent point, longer deliberation can introduce overthinking, where the model second-guesses a correct answer, drifts into irrelevant branches or exhausts its context. The useful budget therefore depends on task difficulty, not on a universal rule.
| Budget range | Typical behavior | Suitable tasks |
|---|---|---|
| Zero or disabled | Direct answer, no deliberation | Lookup, formatting, short chat |
| Small (about 1,000 to 4,000) | Brief plan and quick check | Routine code edits, simple analysis |
| Medium (about 8,000 to 16,000) | Multi-step derivation with self-review | Debugging, structured reasoning |
| Large (32,000 and above) | Extended search over alternatives | Competition math, complex proofs |
These ranges are indicative only. Models differ in token efficiency, so one model may solve a problem in 2,000 tokens that another needs 10,000 to solve.
Thinking budget compared with related controls
The thinking budget is one of several generation limits, and confusing them is a frequent source of truncated or empty responses. It governs deliberation, while neighboring settings govern the answer, the overall context and the sampling process.
| Control | What it limits | Key difference |
|---|---|---|
| Thinking budget | Reasoning tokens before the answer | Targets deliberation depth |
| Max tokens | Output tokens in total or for the answer | May include thinking tokens |
| Context window | Input plus output tokens | Hard architectural ceiling |
| Temperature | Randomness of token choice | Does not change length |
| Prompted chain-of-thought | Nothing by itself | Visible reasoning requested in text |
On many platforms the thinking budget must be smaller than max tokens, because both draw on the same output allowance. Setting them equal can leave no room for the final answer, which produces a response that ends mid-thought or contains only the reasoning summary.
Choosing and tuning a thinking budget
Selecting a budget is an engineering trade-off between quality, response time and spend. Reasoning tokens are normally billed as output tokens, and output tokens are priced several times higher than input tokens, so a large budget can dominate the cost of a request even when the visible answer is short.
| Situation | Recommended setting | Reason |
|---|---|---|
| Interactive chat assistant | Low or adaptive budget | Users notice delays beyond a few seconds |
| Batch code review | Medium budget | Accuracy matters more than speed |
| Hard math or research task | High budget | Gains continue at larger budgets |
| High-volume classification | Thinking disabled | Little benefit, large cost |
| Agent tool loop | Moderate budget per step | Many calls multiply the cost |
The most reliable method is empirical evaluation: run a representative test set at several budgets, plot accuracy against cost and latency, and choose the knee of the curve. Teams often add routing, sending easy prompts to a small budget and escalating only uncertain cases.
Limitations and common mistakes
A thinking budget is an upper bound, not a guarantee of effort. Models trained with adaptive reasoning frequently stop well before the limit, and some providers treat the value as guidance rather than a strict cap. Conversely, a hard cut can interrupt a promising line of reasoning and degrade the answer more than a smaller planned budget would.
- Assuming more is always better: overthinking can lower accuracy on simple questions.
- Ignoring cost visibility: hidden reasoning is still billed, so dashboards must track it.
- Reading the trace as proof: chain-of-thought faithfulness is limited, so the trace may not reflect the real computation.
- Reusing one budget across models: token efficiency varies, so budgets need retuning after a model upgrade.
- Forgetting multi-turn growth: retained reasoning blocks can consume the context window in long sessions.
The simple explanation
Some AI models think before they answer. They write private notes, try ideas and check their work. A thinking budget is simply a limit on how long that private thinking may run.
It works like giving a student a set amount of scratch paper for an exam question. With one small sheet, the student answers fast but may skip a check. With a whole pad, the student can work carefully, but it takes longer and costs more. The right amount depends on how hard the question is, and more paper does not always mean a better answer.
How it is used in practice
A software team runs an automated assistant that reviews about 2,000 pull requests per week. In the first trial, the team used a budget of 32,000 reasoning tokens for every review. Quality was high, but the average review took more than a minute and reasoning tokens made up most of the bill.
The engineers then tested budgets of 2,000, 8,000 and 16,000 tokens on 300 past pull requests with known defects. Detection was almost identical at 8,000 and 32,000 tokens, while 2,000 tokens missed noticeably more bugs. They set 8,000 tokens as the default and allowed escalation to 24,000 only for changes touching security-sensitive files.
The result was roughly a two-thirds reduction in reasoning cost with no measurable loss in defect detection on their test set, and the median review time fell to about 20 seconds.