Response prefilling

Also known as: assistant prefill

On this page 7

Response prefilling, also called assistant prefill, is a technique in which the application writes the opening tokens of the model’s reply and the language model generates only the continuation. Prompting with a prefill turns an open request into a constrained completion: the model cannot choose how to begin, so it follows the format, language or voice that the first characters establish.

The technique exploits the fact that a chat model is still a next-token predictor. Once the reply already starts with an opening brace, an XML tag or a phrase in character, the most probable continuation is the one consistent with that start. Prefilling is a small, cheap control that often replaces long format instructions in the system prompt.

How response prefilling works inside a chat request

In a chat-style API, a request normally ends with a user message, and the model writes the whole assistant turn. With prefilling, the request ends with an assistant message that contains partial content. The model treats that text as already written and continues from its last token, without restating it.

The chat template makes this possible. The server renders the conversation into one token sequence, places the role marker for the assistant turn, appends the prefill and omits the end-of-turn marker. Generation therefore resumes inside the assistant turn instead of opening a new one.

ElementStandard requestPrefilled request
Last message roleUserAssistant
Start of the answerChosen by the modelFixed by the application
Returned textComplete answerContinuation only, without the prefill
Client workUse output directlyConcatenate prefill and continuation
Request structure with and without a prefill

A frequent integration error is forgetting the last row: the API returns only the generated continuation, so parsing a JSON prefill such as { fails unless the client joins the two parts before decoding.

Common prefill patterns and what they control

Most prefills are short, between one character and one sentence. Longer prefills give stronger control but also put words in the model’s mouth that it might not endorse, which can reduce answer quality.

PrefillPurposeEffect on output
Opening braceForce JSON outputSkips preamble such as an introductory sentence
Opening XML tagWrap the answer in a known tagEnables parsing with a matching stop sequence
Bracketed character nameKeep a persona in role-playReduces drift back to a neutral assistant voice
First words in a target languageFix the reply languagePrevents switching to the language of the source text
Option label such as Answer:Constrain classificationProduces a short label instead of an explanation
Typical prefill patterns

Prefills combine well with stop sequences. A prefill of an opening tag and a stop sequence on the closing tag produce a clean, bounded extraction that needs no post-processing beyond trimming.

Prefilling is one of several ways to shape output, and it differs from neighbors such as an output format instruction or structured output modes in where the constraint lives. Instructions ask the model to comply, prefills place the model inside the desired format, and constrained decoding enforces a grammar at the token level.

TechniqueMechanismStrength of guarantee
Output format instructionNatural-language request in the promptWeak, depends on instruction following
Few-shot examplesDemonstrated answers in the contextModerate, imitation of the shown format
Response prefillingFixed first tokens of the replyStrong for the opening, weaker for the end
Structured output or JSON modeSchema-constrained decodingStrongest, syntactically valid by design
Ways to control the form of a model answer

Because prefilling controls only the start, it is best paired with a clear format instruction and validation in code. When a provider offers schema-constrained decoding, that mode is usually the more reliable choice for strict JSON.

Provider support and configuration details

Support for prefilling varies by provider and by model version. Some hosted APIs accept a trailing assistant message directly, others ignore it or reject it, and open-weight inference servers usually expose an explicit option to continue the final message.

  • Hosted APIs that accept assistant prefill treat a final assistant message as the start of the reply; some reject a prefill that ends with trailing whitespace, because the tokenizer would split the next word incorrectly.
  • Open-weight servers such as vLLM or llama.cpp support continuation through a flag that suppresses the generation prompt and the end-of-turn token.
  • Reasoning modes often conflict with prefilling: when a model must first produce hidden thinking, a prefilled visible answer is not supported or is ignored by several providers.
  • Newer model versions may restrict prefilling, so behavior must be verified per model rather than assumed across a family.

A practical configuration rule is to keep the prefill free of trailing spaces, to end it at a natural token boundary, and to test it with the exact model identifier used in production.

Risks, failure modes and safety considerations

Prefilling is powerful precisely because it bypasses the model’s own choice of opening, and that power creates both quality and safety risks. Experienced teams monitor the following symptoms.

SymptomLikely causeFix
Broken first wordPrefill ends with a space or mid-tokenEnd the prefill without trailing whitespace
Duplicated openingClient prints prefill and model repeats itShorten prefill and concatenate once
Invalid JSON at the endPrefill fixes only the startAdd schema validation and a stop sequence
Lower answer qualityOverly long or leading prefillReduce prefill to the minimal format cue
Prefill problems and their fixes

The safety concern is the prefill attack: if end users can control the assistant turn, they can start a reply with an apparent agreement to a harmful request and push the model past its refusal. Applications should therefore never expose raw prefill control to untrusted users, and providers train models to recover from such forced openings.

In plain words

Response prefilling means writing the first few words of the answer yourself and letting the AI finish it. It works like starting a sentence for someone: if you say “The three main reasons are”, the other person will almost certainly list three reasons.

Because the AI always continues from where the text stops, a good start steers the whole answer. Starting with a curly bracket gets data in a computer-friendly format, and starting in French keeps the answer in French. The trick controls the beginning very well, but the rest still needs checking.

A real-world example

An online retailer uses a language model to extract product attributes, such as color, size and material, from about 40,000 supplier descriptions per day. With only a format instruction, roughly one response in fifty started with a sentence like “Here is the extracted data”, which broke the JSON parser and sent the record to manual review.

The engineering team added a prefill containing a single opening brace, joined it with the returned continuation in code and validated the result against a schema. Failed records were retried once with a lower temperature.

After the change, preamble errors disappeared completely and the share of records needing manual review fell to a few per thousand, mostly caused by genuinely ambiguous descriptions rather than formatting problems.