Max tokens is the generation parameter that sets the upper limit on output tokens a language model may produce in one response; when the model reaches that ceiling, decoding stops even if the answer is unfinished. In the vocabulary of language models and artificial intelligence services, it is also called the output token limit, and it governs response length, latency and cost more directly than any other request setting.
The parameter is a hard cap, not a target. A model does not plan its answer to fill the budget; it generates token by token and either ends naturally with an end-of-sequence token or a stop sequence, or is cut off when the counter reaches the limit.
How the max tokens limit stops generation
Max tokens works as a counter inside the decoding loop. An autoregressive model predicts one token, appends it to the sequence and repeats; after every step the inference server compares the number of generated tokens with the limit and halts generation when the two are equal.
Because the check is purely numerical, the cut can land in the middle of a sentence, a word or a JSON object. Most APIs report the reason a response ended, which lets an application distinguish a natural ending from a truncation. The table below lists the usual finish reasons and what each one means for the caller.
| Finish reason | What triggered it | Implication for the caller |
|---|---|---|
| End of sequence | Model emitted its end token | Answer is complete as the model intended |
| Stop sequence | A configured string appeared | Output ended at a known delimiter |
| Length limit | Counter reached max tokens | Output is probably truncated |
| Context exhausted | Prompt plus output filled the window | Shorten input or raise capacity |
Applications should always inspect the finish reason rather than assume a response is whole. A length-limited answer can look plausible while missing its conclusion, a closing bracket or the final step of a procedure.
Max tokens and the context window
Max tokens and the context window are related but distinct limits. The context window is the total number of tokens the model can attend to at once, covering the system prompt, the conversation, any retrieved documents and the generated output; max tokens restricts only the output portion of that budget.
The practical rule is that prompt tokens plus max tokens must fit inside the context window. If a model has a window of 128,000 tokens and the prompt already uses 120,000, a request for 16,000 output tokens is either rejected or silently reduced, depending on the provider. Many models also have a separate maximum output length that is much smaller than the window, often between 4,000 and 64,000 tokens.
| Limit | What it counts | Typical behavior when exceeded |
|---|---|---|
| Context window | Input and output tokens together | Request error or truncated input |
| Model output ceiling | Output tokens the model supports | Value is rejected or clamped |
| Max tokens setting | Output tokens for this request | Generation stops with a length reason |
| Thinking budget | Internal reasoning tokens | Reasoning ends early or answer is shortened |
Token counts depend on tokenization, so the same text consumes different budgets in different models. As a rough guide, one English word averages about 1.3 tokens with common byte-pair encoding vocabularies, while code, numbers and many non-Latin scripts consume noticeably more.
Max tokens in reasoning models
In reasoning models, max tokens often has to cover hidden reasoning tokens as well as the visible answer. A model that thinks through a problem before replying may spend thousands of tokens on intermediate steps, and if the limit counts those tokens, a small value can leave no room for the final answer at all.
Providers handle this differently. Some expose a dedicated parameter for completion tokens that includes reasoning, others pair max tokens with a separate thinking budget, and some require the thinking budget to be smaller than the overall limit. The safe practice is to read the provider documentation for the specific model and to leave generous headroom above the expected reasoning length.
Choosing a max tokens value
A good max tokens value is slightly above the longest legitimate answer for the task. Setting it too low truncates useful output, while setting it far too high wastes little on ordinary requests but removes a safety net against runaway generation, repetitive loops and unexpectedly high bills.
The following decision table gives starting points for common workloads; the ranges assume English text and should be validated against real response lengths in logs.
| Workload | Suggested range | Reason |
|---|---|---|
| Classification or extraction | 10 to 200 tokens | Answers are short and structured |
| Chat assistant reply | 500 to 2,000 tokens | Covers detailed answers with lists |
| Long-form drafting | 4,000 to 16,000 tokens | Articles and reports need space |
| Code generation | 2,000 to 8,000 tokens | Code is token-dense |
| Reasoning model task | 16,000 tokens or more | Hidden reasoning shares the budget |
- Measure before tuning: record output lengths and set the cap near a high percentile, such as the 99th.
- Instruct length in the prompt: max tokens cannot make an answer concise; only instructions and examples shape length.
- Combine with stop sequences when the output has a natural delimiter, so generation ends cleanly before the cap.
- Budget for cost: output tokens are usually priced higher than input tokens, so the cap bounds the worst-case charge per call.
Common mistakes with max tokens
Most problems with max tokens come from treating it as a style control instead of a safety limit. The table summarizes frequent symptoms, their cause and the usual fix.
| Symptom | Likely cause | Fix |
|---|---|---|
| Answer ends mid-sentence | Cap is below the needed length | Raise the limit or request a shorter format |
| Invalid JSON output | Truncation before closing brackets | Raise the limit and validate the finish reason |
| Empty visible answer | Reasoning consumed the whole budget | Increase the limit or lower the thinking budget |
| Request rejected | Prompt plus cap exceeds the window | Trim input or reduce the cap |
| Answers still too long | Expecting the cap to shorten style | Specify length in the instructions |
A further misconception is that a higher limit makes the model write more. In practice the model ignores the remaining budget, because it does not see the max tokens value during generation; only the prompt and the decoding strategy influence how long it chooses to write.
Stripped to the essentials
Max tokens is a word-count ceiling for the reply. A language model writes its answer one small piece at a time, and this setting tells the system to stop once a certain number of pieces has been written.
Think of a taxi meter with an automatic stop. The ride ends when the passenger arrives or when the meter hits the preset amount. If the limit is too low, the answer is cut off before it is finished. If it is sensible, the answer ends on its own. The limit then simply guards against runaway length and cost.
Where it is applied
A support team builds a ticket-triage service that sends each incoming customer email to a language model and asks for a JSON object with a category, a priority and a one-sentence summary. The engineers first set max tokens to 60, and about 4% of responses fail to parse because longer summaries are truncated before the closing brace.
After logging output lengths for a week, they find that the 99th percentile is 85 tokens, so they raise the cap to 150 and add a check that resends any request whose finish reason is the length limit. Parsing failures drop to nearly zero, while the cap still bounds the worst-case cost if the model ever begins to repeat itself.