Stop sequence

On this page 7

A stop sequence is a string of characters that, once it appears in the generated output, makes the inference system halt text generation immediately. In language models, it is one of the main controls over where a response ends, working alongside the end-of-sequence token learned during training and the max tokens limit set by the caller.

Stop sequences matter wherever generated text feeds into software rather than a human reader. In applied artificial intelligence, they keep a model from inventing the next turn of a dialogue, from running past the end of a structured record, or from continuing a few-shot pattern indefinitely. They are cheap, deterministic and transparent, but they act on text, not on meaning, so they must be chosen with care.

How a stop sequence ends generation

A stop sequence ends generation by being matched against the decoded output after every new token. An autoregressive model produces one token at a time through next-token prediction; the serving layer decodes each token to text, appends it to a buffer and checks whether the buffer now ends with any configured stop string. When it does, decoding stops and no further tokens are sampled.

The match is usually performed on text rather than on token IDs, because one string can be tokenized in several ways. A stop string such as \nUser: may arrive as one token, as three tokens, or split across a boundary that also contains other characters. Serving engines therefore keep a small lookahead buffer and withhold the last few characters from a streaming client until they are sure those characters are not the beginning of a stop sequence.

Most interfaces exclude the stop sequence from the returned text and report a finish reason such as stop, while some return the matched string in a separate field. This distinction matters for parsers: code that expects a closing delimiter must add it back when the API removes it.

The table below shows the stages of a single generation step and the role the stop sequence plays in each.

StageWhat happensRole of the stop sequence
Forward passModel computes logits for the next tokenNone; the model is unaware of the stop list
SamplingA token is selected by greedy decoding or samplingNone, unless constrained decoding is used
DetokenizationThe token is converted to text and appendedPrepares the text that will be matched
Stop checkThe output tail is compared with each stop stringEnds generation on a match
Streaming releaseSafe characters are sent to the clientHolds back a possible partial match
Where the stop check sits in the decoding loop

Stop sequences compared with other termination controls

A stop sequence is one of three complementary ways to end a response, and each answers a different question. The end-of-sequence token reflects what the model learned about natural endings, the max tokens limit caps cost and latency, and the stop sequence encodes an application rule about where useful output ends.

The following comparison clarifies who controls each mechanism and how it typically fails.

MechanismWho sets itTypical finish reasonMain failure mode
End-of-sequence tokenModel training and chat templatestop or end_turnModel never emits it and rambles
Stop sequenceApplication developerstop or stop_sequenceFires too early on legitimate text
Max tokensApplication developerlength or max_tokensTruncates an answer mid-sentence
Constrained decodingGrammar or schemastop when the grammar closesGrammar too strict for the task
Termination mechanisms in language model inference

In chat-tuned models, the chat template already inserts special end-of-turn tokens, so explicit stop sequences are needed less often than with a base model. They remain essential for raw completion endpoints, for few-shot prompting and for any format the model was not trained to close on its own.

Common stop sequence patterns

Effective stop sequences are strings that should never appear inside a correct answer but almost always follow one. Choosing them means looking at the prompt format and asking what the model would write next if it kept going.

Use caseExample stop stringWhat it prevents
Dialogue transcript\nUser:The model writing the next user turn
Few-shot list\n\nInput:Generating extra invented examples
Single-line answer\nExplanations after a short value
Tagged output</answer>Text after the closing tag
Code completion\ndef or \nclass A second, unrequested function
Tool-call formatObservation:The model hallucinating a tool result
Typical stop strings and their purpose

The last pattern is central to agent loops built on the ReAct style: the model writes an action, the stop sequence halts it before it fabricates an observation, and the orchestrator runs the real tool and appends the genuine result. Without that stop, the model would simply invent plausible tool output, a form of hallucination that is hard to detect downstream.

Configuring stop sequences in practice

Stop sequences are passed as a list of strings in the request, commonly under a parameter named stop or stop_sequences. Providers impose a small maximum count, often between four and a few dozen strings, and sometimes a maximum length per string, so the list should contain only the strings that matter.

  • Match the exact whitespace of the prompt format; User: and \nUser: behave very differently.
  • Prefer distinctive delimiters such as ### or closing tags over common words that can occur naturally.
  • Always set max tokens as well, because a stop sequence that never appears offers no protection.
  • Check the finish reason in every response to distinguish a clean stop from truncation.
  • Test with streaming enabled, since partial-match buffering can change perceived latency.

Decoding parameters such as temperature, top-p sampling and repetition penalty do not change how stop sequences are evaluated, but they change how likely the model is to produce the stop string at the right moment. A higher temperature makes unexpected continuations, and therefore missed or premature stops, more frequent.

Limitations and common mistakes

The central limitation is that a stop sequence is a literal string match with no understanding of context. It cannot tell a delimiter from the same characters inside a quotation, a code block or a URL, and it cannot stop generation for a reason that is not visible in the text.

SymptomLikely causeFix
Empty or one-word outputStop string matches the first token producedUse a longer, more specific delimiter
Answer cut in the middleStop string occurs naturally in the contentChoose a string that cannot appear in valid output
Model keeps goingWhitespace or casing differs from the stop stringCopy the delimiter exactly from the prompt
Parser fails on outputAPI removed the closing delimiterRe-append the delimiter before parsing
Partial delimiter shown to userClient streams without a lookahead bufferBuffer the stream tail before display
Symptoms, causes and fixes

Stop sequences are also not a safety mechanism. They cannot reliably block harmful content, because the model can phrase the same idea in endless ways. For structural guarantees, such as valid JSON, constrained decoding or schema-guided generation is more robust; stop sequences are best treated as a lightweight boundary marker.

Stripped to the essentials

A language model writes text one small piece at a time and does not always know when to quit. A stop sequence is a word or symbol you choose in advance, and the moment the model writes it, the system cuts the model off.

It works like a finish line painted on a running track. The runner does not need to understand the race; once they cross the line, the race is over. The key point is that the line must be placed where the useful part really ends, or the answer gets cut short or runs too long.

Where it is applied

A support team builds an assistant that answers product questions by calling an internal search tool. The prompt uses a simple format in which the model writes Thought:, then Action: with a search query, and the orchestrator later appends Observation: with real search results. The developers set Observation: as a stop sequence and limit each step to 256 max tokens.

Before the stop sequence was added, testing showed the model often writing its own fake observation, complete with invented product specifications, and then answering from that fiction. After the change, every step ends cleanly right before the observation, the orchestrator inserts the genuine search result, and the model continues from verified data. The team also logs the finish reason for each step, which reveals that about one step in fifty ends on the token limit instead, flagging prompts that need shorter reasoning.