Max tokens

Also known as: output token limit

On this page 7

Max tokens is the generation parameter that sets the upper limit on output tokens a language model may produce in one response; when the model reaches that ceiling, decoding stops even if the answer is unfinished. In the vocabulary of language models and artificial intelligence services, it is also called the output token limit, and it governs response length, latency and cost more directly than any other request setting.

The parameter is a hard cap, not a target. A model does not plan its answer to fill the budget; it generates token by token and either ends naturally with an end-of-sequence token or a stop sequence, or is cut off when the counter reaches the limit.

How the max tokens limit stops generation

Max tokens works as a counter inside the decoding loop. An autoregressive model predicts one token, appends it to the sequence and repeats; after every step the inference server compares the number of generated tokens with the limit and halts generation when the two are equal.

Because the check is purely numerical, the cut can land in the middle of a sentence, a word or a JSON object. Most APIs report the reason a response ended, which lets an application distinguish a natural ending from a truncation. The table below lists the usual finish reasons and what each one means for the caller.

Finish reasonWhat triggered itImplication for the caller
End of sequenceModel emitted its end tokenAnswer is complete as the model intended
Stop sequenceA configured string appearedOutput ended at a known delimiter
Length limitCounter reached max tokensOutput is probably truncated
Context exhaustedPrompt plus output filled the windowShorten input or raise capacity
Typical finish reasons returned with a response

Applications should always inspect the finish reason rather than assume a response is whole. A length-limited answer can look plausible while missing its conclusion, a closing bracket or the final step of a procedure.

Max tokens and the context window

Max tokens and the context window are related but distinct limits. The context window is the total number of tokens the model can attend to at once, covering the system prompt, the conversation, any retrieved documents and the generated output; max tokens restricts only the output portion of that budget.

The practical rule is that prompt tokens plus max tokens must fit inside the context window. If a model has a window of 128,000 tokens and the prompt already uses 120,000, a request for 16,000 output tokens is either rejected or silently reduced, depending on the provider. Many models also have a separate maximum output length that is much smaller than the window, often between 4,000 and 64,000 tokens.

LimitWhat it countsTypical behavior when exceeded
Context windowInput and output tokens togetherRequest error or truncated input
Model output ceilingOutput tokens the model supportsValue is rejected or clamped
Max tokens settingOutput tokens for this requestGeneration stops with a length reason
Thinking budgetInternal reasoning tokensReasoning ends early or answer is shortened
Limits that interact with max tokens

Token counts depend on tokenization, so the same text consumes different budgets in different models. As a rough guide, one English word averages about 1.3 tokens with common byte-pair encoding vocabularies, while code, numbers and many non-Latin scripts consume noticeably more.

Max tokens in reasoning models

In reasoning models, max tokens often has to cover hidden reasoning tokens as well as the visible answer. A model that thinks through a problem before replying may spend thousands of tokens on intermediate steps, and if the limit counts those tokens, a small value can leave no room for the final answer at all.

Providers handle this differently. Some expose a dedicated parameter for completion tokens that includes reasoning, others pair max tokens with a separate thinking budget, and some require the thinking budget to be smaller than the overall limit. The safe practice is to read the provider documentation for the specific model and to leave generous headroom above the expected reasoning length.

Choosing a max tokens value

A good max tokens value is slightly above the longest legitimate answer for the task. Setting it too low truncates useful output, while setting it far too high wastes little on ordinary requests but removes a safety net against runaway generation, repetitive loops and unexpectedly high bills.

The following decision table gives starting points for common workloads; the ranges assume English text and should be validated against real response lengths in logs.

WorkloadSuggested rangeReason
Classification or extraction10 to 200 tokensAnswers are short and structured
Chat assistant reply500 to 2,000 tokensCovers detailed answers with lists
Long-form drafting4,000 to 16,000 tokensArticles and reports need space
Code generation2,000 to 8,000 tokensCode is token-dense
Reasoning model task16,000 tokens or moreHidden reasoning shares the budget
Starting values by workload
  • Measure before tuning: record output lengths and set the cap near a high percentile, such as the 99th.
  • Instruct length in the prompt: max tokens cannot make an answer concise; only instructions and examples shape length.
  • Combine with stop sequences when the output has a natural delimiter, so generation ends cleanly before the cap.
  • Budget for cost: output tokens are usually priced higher than input tokens, so the cap bounds the worst-case charge per call.

Common mistakes with max tokens

Most problems with max tokens come from treating it as a style control instead of a safety limit. The table summarizes frequent symptoms, their cause and the usual fix.

SymptomLikely causeFix
Answer ends mid-sentenceCap is below the needed lengthRaise the limit or request a shorter format
Invalid JSON outputTruncation before closing bracketsRaise the limit and validate the finish reason
Empty visible answerReasoning consumed the whole budgetIncrease the limit or lower the thinking budget
Request rejectedPrompt plus cap exceeds the windowTrim input or reduce the cap
Answers still too longExpecting the cap to shorten styleSpecify length in the instructions
Symptoms, causes and fixes

A further misconception is that a higher limit makes the model write more. In practice the model ignores the remaining budget, because it does not see the max tokens value during generation; only the prompt and the decoding strategy influence how long it chooses to write.

Stripped to the essentials

Max tokens is a word-count ceiling for the reply. A language model writes its answer one small piece at a time, and this setting tells the system to stop once a certain number of pieces has been written.

Think of a taxi meter with an automatic stop. The ride ends when the passenger arrives or when the meter hits the preset amount. If the limit is too low, the answer is cut off before it is finished. If it is sensible, the answer ends on its own. The limit then simply guards against runaway length and cost.

Where it is applied

A support team builds a ticket-triage service that sends each incoming customer email to a language model and asks for a JSON object with a category, a priority and a one-sentence summary. The engineers first set max tokens to 60, and about 4% of responses fail to parse because longer summaries are truncated before the closing brace.

After logging output lengths for a week, they find that the 99th percentile is 85 tokens, so they raise the cap to 150 and add a check that resends any request whose finish reason is the length limit. Parsing failures drop to nearly zero, while the cap still bounds the worst-case cost if the model ever begins to repeat itself.