A chat template is the formatting rule that converts a structured conversation, a list of messages with roles such as system, user and assistant, into the single flat token sequence that a language model actually processes. In language models built for dialogue, the template inserts role markers, special tokens and turn delimiters so that the model can tell who is speaking, where each turn ends and where its own answer should begin.
The template matters because a chat model is still a next-token predictor: it sees only one long string of tokens. During instruction tuning, the model learned one specific layout, and at inference time the prompt must reproduce that layout exactly. A mismatched template rarely causes an error message; instead it produces quietly degraded output, such as ignored instructions, runaway generation or the model continuing the user turn.
How a chat template serializes a conversation
A chat template works as a deterministic serialization step between the application and the tokenizer. The application sends a list of messages, each with a role and content field; the template wraps every message in opening and closing markers, concatenates the turns and, when requested, appends the generation prompt, the header that tells the model the assistant is now speaking.
In the Hugging Face ecosystem, the template is stored as a Jinja template string in the tokenizer configuration and applied with apply_chat_template. Other runtimes embed an equivalent template in the model file, for example in GGUF metadata, or hard-code it in the serving engine. The logic is the same everywhere: messages in, one formatted prompt out.
The following table lists the components that a typical chat template inserts and the role each one plays.
| Component | Purpose | Typical form |
|---|---|---|
| Beginning-of-sequence token | Marks the start of the whole input | A reserved token such as BOS |
| Role header | Names the speaker of the turn | system, user, assistant or tool |
| Turn delimiter | Separates one message from the next | A reserved end-of-turn token |
| Generation prompt | Opens an empty assistant turn | Assistant header without content |
| End-of-sequence token | Signals that the answer is complete | Model-specific stop token |
Because these markers are usually reserved special tokens rather than ordinary text, they occupy a single token ID each and cannot be produced accidentally by user input when the tokenizer is configured correctly. This separation is also a basic defense against role spoofing, where a user types text that imitates a system header.
Common chat template formats
No universal standard exists, so each model family defines its own format, and the format is fixed by the data used during fine-tuning. Several layouts have become widespread and are often reused by open-weight models.
| Format | Turn markers | System prompt handling |
|---|---|---|
| ChatML | im_start and im_end tokens with a role name | Native system role |
| Llama 3 style | Header tokens around the role, eot_id at turn end | Native system role |
| Mistral instruct style | [INST] and [/INST] around user text | Often merged into the first user turn |
| Gemma style | Start and end of turn tokens with user or model | Often merged into the first user turn |
| Plain text style | Labels such as User: and Assistant: | Free text at the top |
The differences look cosmetic, but they are not interchangeable. A model tuned on one layout assigns meaning to its specific token IDs; feeding it another layout turns those markers into unfamiliar ordinary text, and the model must guess the structure.
Chat templates in training and inference
The same chat template is used twice in the life of a model: once to format supervised fine-tuning data and again to format live prompts. Consistency between the two is the entire point of the template.
During supervised fine-tuning, the template also defines which tokens contribute to the loss. A common practice is loss masking, which computes the training loss only on assistant tokens, so the model learns to answer rather than to imitate user messages. At inference time, the template appends the generation prompt and the serving engine stops decoding at the end-of-turn token, which acts as the natural stop sequence.
The table below contrasts how the template is used in each phase.
| Phase | Template input | Key setting |
|---|---|---|
| Fine-tuning | Complete conversations including answers | Loss mask on assistant turns |
| Inference | Conversation up to the latest user turn | Generation prompt enabled |
| Tool calling | Messages plus tool schemas and results | Tool role and call syntax |
| Evaluation | Benchmark prompts in chat form | Same template as deployment |
Modern templates often carry more than roles. They may render tool definitions, format function call results, insert a default system prompt, or wrap reasoning tokens of a reasoning model in dedicated tags and strip them from earlier turns to save context window space.
How a chat template differs from related concepts
A chat template is formatting infrastructure, not content, and it is easily confused with neighboring ideas in prompt design.
- A system prompt is the content of the system message; the chat template decides where and how that content is placed.
- A prompt template in application code fills variables into task text; the chat template operates one level lower, on roles and tokens.
- Tokenization splits text into tokens; the chat template runs first and produces the text, and often the special tokens, that the tokenizer encodes.
- A base model usually has no chat template at all, because it was trained on raw text rather than on role-structured dialogue.
Hosted chat APIs hide the template completely: the client sends JSON messages and the provider applies its internal format on the server. Developers meet chat templates directly when they run open-weight models locally, fine-tune them or call a raw completion endpoint.
Common mistakes with chat templates
Most template problems are silent failures: the model still responds, but quality, safety behavior or stopping behavior degrades. The following table lists frequent symptoms and their usual causes.
| Symptom | Likely cause | Fix |
|---|---|---|
| Endless or rambling output | End-of-turn token not set as a stop token | Configure the correct stop token IDs |
| Model continues the user turn | Generation prompt missing | Enable the generation prompt flag |
| Ignored system instructions | System role not supported by the format | Merge the system text as the template expects |
| Duplicated start token | Template and tokenizer both add BOS | Disable one of the two insertions |
| Weaker answers after fine-tuning | Training and inference templates differ | Use one template for both phases |
A reliable habit is to render the formatted prompt as text and inspect it before deployment. Small details, such as a missing newline after a role header or an extra space before the answer, can shift token boundaries and measurably change model behavior.
A plain-language summary
A chat model does not really see a conversation. It sees one long line of text. The chat template is the set of rules that turns the back-and-forth of a chat into that single line, adding hidden labels such as who is talking and where each message ends.
Think of it as the form a clerk expects to receive. If every box is filled in the usual place, the clerk works quickly and correctly. If the same information arrives on the wrong form, the clerk may still try, but mistakes become likely. Using the template the model learned with keeps its answers reliable.
An example from practice
A small software team runs an open-weight 8-billion-parameter model on its own server to answer internal support questions. The first prototype sends prompts as plain text with labels such as User: and Assistant:. Answers are vague, the model sometimes writes a fake follow-up question from the user, and replies often run until the 512-token limit.
An engineer prints the prompt produced by the model’s own apply_chat_template function and compares it with the prototype. The official format uses reserved header tokens and an end-of-turn token that the prototype never sent. After switching to the tokenizer’s template, enabling the generation prompt and registering the end-of-turn token as a stop token, replies end cleanly, the system instructions are followed, and the team reports noticeably shorter and more accurate answers without changing the model or its weights.