Chat template

On this page 7

A chat template is the formatting rule that converts a structured conversation, a list of messages with roles such as system, user and assistant, into the single flat token sequence that a language model actually processes. In language models built for dialogue, the template inserts role markers, special tokens and turn delimiters so that the model can tell who is speaking, where each turn ends and where its own answer should begin.

The template matters because a chat model is still a next-token predictor: it sees only one long string of tokens. During instruction tuning, the model learned one specific layout, and at inference time the prompt must reproduce that layout exactly. A mismatched template rarely causes an error message; instead it produces quietly degraded output, such as ignored instructions, runaway generation or the model continuing the user turn.

How a chat template serializes a conversation

A chat template works as a deterministic serialization step between the application and the tokenizer. The application sends a list of messages, each with a role and content field; the template wraps every message in opening and closing markers, concatenates the turns and, when requested, appends the generation prompt, the header that tells the model the assistant is now speaking.

In the Hugging Face ecosystem, the template is stored as a Jinja template string in the tokenizer configuration and applied with apply_chat_template. Other runtimes embed an equivalent template in the model file, for example in GGUF metadata, or hard-code it in the serving engine. The logic is the same everywhere: messages in, one formatted prompt out.

The following table lists the components that a typical chat template inserts and the role each one plays.

ComponentPurposeTypical form
Beginning-of-sequence tokenMarks the start of the whole inputA reserved token such as BOS
Role headerNames the speaker of the turnsystem, user, assistant or tool
Turn delimiterSeparates one message from the nextA reserved end-of-turn token
Generation promptOpens an empty assistant turnAssistant header without content
End-of-sequence tokenSignals that the answer is completeModel-specific stop token
Components of a typical chat template

Because these markers are usually reserved special tokens rather than ordinary text, they occupy a single token ID each and cannot be produced accidentally by user input when the tokenizer is configured correctly. This separation is also a basic defense against role spoofing, where a user types text that imitates a system header.

Common chat template formats

No universal standard exists, so each model family defines its own format, and the format is fixed by the data used during fine-tuning. Several layouts have become widespread and are often reused by open-weight models.

FormatTurn markersSystem prompt handling
ChatMLim_start and im_end tokens with a role nameNative system role
Llama 3 styleHeader tokens around the role, eot_id at turn endNative system role
Mistral instruct style[INST] and [/INST] around user textOften merged into the first user turn
Gemma styleStart and end of turn tokens with user or modelOften merged into the first user turn
Plain text styleLabels such as User: and Assistant:Free text at the top
Widely used chat template families

The differences look cosmetic, but they are not interchangeable. A model tuned on one layout assigns meaning to its specific token IDs; feeding it another layout turns those markers into unfamiliar ordinary text, and the model must guess the structure.

Chat templates in training and inference

The same chat template is used twice in the life of a model: once to format supervised fine-tuning data and again to format live prompts. Consistency between the two is the entire point of the template.

During supervised fine-tuning, the template also defines which tokens contribute to the loss. A common practice is loss masking, which computes the training loss only on assistant tokens, so the model learns to answer rather than to imitate user messages. At inference time, the template appends the generation prompt and the serving engine stops decoding at the end-of-turn token, which acts as the natural stop sequence.

The table below contrasts how the template is used in each phase.

PhaseTemplate inputKey setting
Fine-tuningComplete conversations including answersLoss mask on assistant turns
InferenceConversation up to the latest user turnGeneration prompt enabled
Tool callingMessages plus tool schemas and resultsTool role and call syntax
EvaluationBenchmark prompts in chat formSame template as deployment
Role of the chat template by phase

Modern templates often carry more than roles. They may render tool definitions, format function call results, insert a default system prompt, or wrap reasoning tokens of a reasoning model in dedicated tags and strip them from earlier turns to save context window space.

A chat template is formatting infrastructure, not content, and it is easily confused with neighboring ideas in prompt design.

  • A system prompt is the content of the system message; the chat template decides where and how that content is placed.
  • A prompt template in application code fills variables into task text; the chat template operates one level lower, on roles and tokens.
  • Tokenization splits text into tokens; the chat template runs first and produces the text, and often the special tokens, that the tokenizer encodes.
  • A base model usually has no chat template at all, because it was trained on raw text rather than on role-structured dialogue.

Hosted chat APIs hide the template completely: the client sends JSON messages and the provider applies its internal format on the server. Developers meet chat templates directly when they run open-weight models locally, fine-tune them or call a raw completion endpoint.

Common mistakes with chat templates

Most template problems are silent failures: the model still responds, but quality, safety behavior or stopping behavior degrades. The following table lists frequent symptoms and their usual causes.

SymptomLikely causeFix
Endless or rambling outputEnd-of-turn token not set as a stop tokenConfigure the correct stop token IDs
Model continues the user turnGeneration prompt missingEnable the generation prompt flag
Ignored system instructionsSystem role not supported by the formatMerge the system text as the template expects
Duplicated start tokenTemplate and tokenizer both add BOSDisable one of the two insertions
Weaker answers after fine-tuningTraining and inference templates differUse one template for both phases
Chat template errors and fixes

A reliable habit is to render the formatted prompt as text and inspect it before deployment. Small details, such as a missing newline after a role header or an extra space before the answer, can shift token boundaries and measurably change model behavior.

A plain-language summary

A chat model does not really see a conversation. It sees one long line of text. The chat template is the set of rules that turns the back-and-forth of a chat into that single line, adding hidden labels such as who is talking and where each message ends.

Think of it as the form a clerk expects to receive. If every box is filled in the usual place, the clerk works quickly and correctly. If the same information arrives on the wrong form, the clerk may still try, but mistakes become likely. Using the template the model learned with keeps its answers reliable.

An example from practice

A small software team runs an open-weight 8-billion-parameter model on its own server to answer internal support questions. The first prototype sends prompts as plain text with labels such as User: and Assistant:. Answers are vague, the model sometimes writes a fake follow-up question from the user, and replies often run until the 512-token limit.

An engineer prints the prompt produced by the model’s own apply_chat_template function and compares it with the prototype. The official format uses reserved header tokens and an end-of-turn token that the prototype never sent. After switching to the tokenizer’s template, enabling the generation prompt and registering the end-of-turn token as a stop token, replies end cleanly, the system instructions are followed, and the team reports noticeably shorter and more accurate answers without changing the model or its weights.