Response prefilling, also called assistant prefill, is a technique in which the application writes the opening tokens of the model’s reply and the language model generates only the continuation. Prompting with a prefill turns an open request into a constrained completion: the model cannot choose how to begin, so it follows the format, language or voice that the first characters establish.
The technique exploits the fact that a chat model is still a next-token predictor. Once the reply already starts with an opening brace, an XML tag or a phrase in character, the most probable continuation is the one consistent with that start. Prefilling is a small, cheap control that often replaces long format instructions in the system prompt.
How response prefilling works inside a chat request
In a chat-style API, a request normally ends with a user message, and the model writes the whole assistant turn. With prefilling, the request ends with an assistant message that contains partial content. The model treats that text as already written and continues from its last token, without restating it.
The chat template makes this possible. The server renders the conversation into one token sequence, places the role marker for the assistant turn, appends the prefill and omits the end-of-turn marker. Generation therefore resumes inside the assistant turn instead of opening a new one.
| Element | Standard request | Prefilled request |
|---|---|---|
| Last message role | User | Assistant |
| Start of the answer | Chosen by the model | Fixed by the application |
| Returned text | Complete answer | Continuation only, without the prefill |
| Client work | Use output directly | Concatenate prefill and continuation |
A frequent integration error is forgetting the last row: the API returns only the generated continuation, so parsing a JSON prefill such as { fails unless the client joins the two parts before decoding.
Common prefill patterns and what they control
Most prefills are short, between one character and one sentence. Longer prefills give stronger control but also put words in the model’s mouth that it might not endorse, which can reduce answer quality.
| Prefill | Purpose | Effect on output |
|---|---|---|
| Opening brace | Force JSON output | Skips preamble such as an introductory sentence |
| Opening XML tag | Wrap the answer in a known tag | Enables parsing with a matching stop sequence |
| Bracketed character name | Keep a persona in role-play | Reduces drift back to a neutral assistant voice |
| First words in a target language | Fix the reply language | Prevents switching to the language of the source text |
| Option label such as Answer: | Constrain classification | Produces a short label instead of an explanation |
Prefills combine well with stop sequences. A prefill of an opening tag and a stop sequence on the closing tag produce a clean, bounded extraction that needs no post-processing beyond trimming.
Response prefilling compared with related prompting techniques
Prefilling is one of several ways to shape output, and it differs from neighbors such as an output format instruction or structured output modes in where the constraint lives. Instructions ask the model to comply, prefills place the model inside the desired format, and constrained decoding enforces a grammar at the token level.
| Technique | Mechanism | Strength of guarantee |
|---|---|---|
| Output format instruction | Natural-language request in the prompt | Weak, depends on instruction following |
| Few-shot examples | Demonstrated answers in the context | Moderate, imitation of the shown format |
| Response prefilling | Fixed first tokens of the reply | Strong for the opening, weaker for the end |
| Structured output or JSON mode | Schema-constrained decoding | Strongest, syntactically valid by design |
Because prefilling controls only the start, it is best paired with a clear format instruction and validation in code. When a provider offers schema-constrained decoding, that mode is usually the more reliable choice for strict JSON.
Provider support and configuration details
Support for prefilling varies by provider and by model version. Some hosted APIs accept a trailing assistant message directly, others ignore it or reject it, and open-weight inference servers usually expose an explicit option to continue the final message.
- Hosted APIs that accept assistant prefill treat a final assistant message as the start of the reply; some reject a prefill that ends with trailing whitespace, because the tokenizer would split the next word incorrectly.
- Open-weight servers such as vLLM or llama.cpp support continuation through a flag that suppresses the generation prompt and the end-of-turn token.
- Reasoning modes often conflict with prefilling: when a model must first produce hidden thinking, a prefilled visible answer is not supported or is ignored by several providers.
- Newer model versions may restrict prefilling, so behavior must be verified per model rather than assumed across a family.
A practical configuration rule is to keep the prefill free of trailing spaces, to end it at a natural token boundary, and to test it with the exact model identifier used in production.
Risks, failure modes and safety considerations
Prefilling is powerful precisely because it bypasses the model’s own choice of opening, and that power creates both quality and safety risks. Experienced teams monitor the following symptoms.
| Symptom | Likely cause | Fix |
|---|---|---|
| Broken first word | Prefill ends with a space or mid-token | End the prefill without trailing whitespace |
| Duplicated opening | Client prints prefill and model repeats it | Shorten prefill and concatenate once |
| Invalid JSON at the end | Prefill fixes only the start | Add schema validation and a stop sequence |
| Lower answer quality | Overly long or leading prefill | Reduce prefill to the minimal format cue |
The safety concern is the prefill attack: if end users can control the assistant turn, they can start a reply with an apparent agreement to a harmful request and push the model past its refusal. Applications should therefore never expose raw prefill control to untrusted users, and providers train models to recover from such forced openings.
In plain words
Response prefilling means writing the first few words of the answer yourself and letting the AI finish it. It works like starting a sentence for someone: if you say “The three main reasons are”, the other person will almost certainly list three reasons.
Because the AI always continues from where the text stops, a good start steers the whole answer. Starting with a curly bracket gets data in a computer-friendly format, and starting in French keeps the answer in French. The trick controls the beginning very well, but the rest still needs checking.
A real-world example
An online retailer uses a language model to extract product attributes, such as color, size and material, from about 40,000 supplier descriptions per day. With only a format instruction, roughly one response in fifty started with a sentence like “Here is the extracted data”, which broke the JSON parser and sent the record to manual review.
The engineering team added a prefill containing a single opening brace, joined it with the returned continuation in code and validated the result against a schema. Failed records were retried once with a lower temperature.
After the change, preamble errors disappeared completely and the share of records needing manual review fell to a few per thousand, mostly caused by genuinely ambiguous descriptions rather than formatting problems.