Instruction tuning

Also known as: instruction-tuned model · instruct model

On this page 7

Instruction tuning is a supervised training stage in which a pretrained language model is further trained on examples of natural-language instructions paired with desired responses, so that it learns to follow requests instead of merely continuing text. In artificial intelligence, it is the step that turns a raw next-token predictor into an instruction-tuned model, often called an instruct model, that answers questions, summarizes documents and performs tasks on demand.

In the Training topic, instruction tuning sits between pretraining and preference optimization such as RLHF or DPO. It is usually implemented as supervised fine-tuning on a curated instruction dataset, and its main contribution is behavioral alignment of format and intent rather than new factual knowledge.

How instruction tuning reshapes a pretrained model

Instruction tuning works by continuing gradient-based training on sequences that contain an instruction, optional input and a target response. The model receives the full sequence, but the loss is usually computed only on the response tokens, so the model learns to produce answers rather than to imitate prompts.

A pretrained model already contains broad linguistic and world knowledge, yet it treats a question as text to be continued and may answer with another question. After instruction tuning, the same weights map an imperative request to a helpful completion. Research on multitask instruction datasets showed that this training also improves zero-shot generalization to unseen tasks, because the model learns the general pattern of reading a task description and acting on it.

The following table lists the main components of a typical instruction tuning pipeline and the role each one plays.

ComponentTypical formRole in training
Base modelDecoder-only transformer after pretrainingSupplies knowledge and language ability
Instruction datasetThousands to millions of prompt and response pairsDefines target behavior and output style
Chat templateRole markers for system, user and assistant turnsSeparates speakers and marks where answers begin
Loss maskingCross-entropy on response tokens onlyPrevents learning to generate user prompts
Optimizer settingsAdamW, low learning rate, 1 to 3 epochsLimits drift from pretrained weights
Components of an instruction tuning pipeline

Sources and types of instruction data

The quality of an instruct model depends more on the diversity and quality of instruction data than on raw volume. Studies with small curated sets of around one thousand carefully written examples have produced strong conversational behavior, which supports the view that instruction tuning mainly surfaces abilities learned during pretraining.

Datasets come from several sources, each with characteristic strengths and risks, as summarized below.

Data sourceStrengthMain risk
Templated academic tasksLarge scale and wide task coverageShort, rigid answers and repetitive phrasing
Human-written demonstrationsHigh quality and realistic requestsExpensive and slow to collect
Synthetic data from stronger modelsCheap, fast and easy to scaleInherited errors and license restrictions
Self-Instruct bootstrappingExpands a small seed set automaticallyLow diversity and repeated mistakes
Real user conversationsMatches actual usage patternsPrivacy issues and noisy targets
Common sources of instruction data

Practitioners typically combine sources in a deliberate data mixture, deduplicate near-identical prompts, filter low-quality responses and balance task categories such as coding, reasoning, writing, extraction and safety refusals. Multi-turn dialogues are included when the model must handle follow-up questions.

Instruction tuning is a specific application of fine-tuning, defined by its data rather than by its algorithm. The terms are often used loosely, so the distinctions below help when reading model documentation or planning a training run.

MethodTraining signalPrimary goal
PretrainingNext-token prediction on raw textBroad knowledge and language modeling
Instruction tuningDemonstrations of instruction and responseFollowing requests in a usable format
Supervised fine-tuningAny labeled target outputsAdapting to a task or domain
RLHFReward model trained on human preferencesPreferring helpful and harmless answers
DPOPairs of preferred and rejected answersPreference alignment without a reward model
Continued pretrainingUnlabeled domain textAdding domain vocabulary and knowledge
Instruction tuning and neighboring methods

In a modern post-training pipeline, instruction tuning usually comes first because preference methods need a model that already produces reasonable answers to compare. Instruction tuning teaches what a good answer looks like, while preference optimization teaches which of several plausible answers is better.

Configuring an instruction tuning run

A practical instruction tuning run requires choices about scale, method and hyperparameters, and conservative settings usually work best because the base model already holds most of the needed capability. The decision table below summarizes common situations.

SituationRecommended approachReason
Limited GPU memoryLoRA or QLoRA adaptersTrains a small fraction of parameters
Need for a general assistantFull fine-tuning on diverse dataMaximizes breadth of behavior
Narrow enterprise taskSmall curated set in the house styleQuality matters more than volume
Signs of forgettingLower learning rate and mixed general dataPreserves pretrained abilities
Output format driftingConsistent chat template in training and inferenceMismatched templates degrade answers
Choosing an instruction tuning setup

Typical full fine-tuning learning rates fall roughly between 1e-6 and 2e-5, while adapter methods tolerate higher values. Training rarely exceeds three epochs, since repeated passes over a small dataset cause the model to memorize phrasing. Evaluation should combine held-out instructions, instruction-following benchmarks such as IFEval and human review.

Limitations and common mistakes

Instruction tuning has well-known failure modes, and most of them trace back to data problems rather than algorithms. The most frequent issues are listed here.

  • Knowledge is not added reliably: training on facts the base model does not know can increase hallucination, because the model learns to answer confidently beyond its knowledge.
  • Catastrophic forgetting appears when the learning rate is too high or the data too narrow, reducing reasoning or multilingual skills.
  • Style imitation without substance occurs when synthetic data from a stronger model teaches tone but not underlying ability.
  • Benchmark contamination inflates scores when evaluation questions leak into the instruction dataset.
  • Over-refusal results from an excess of safety examples, making the model decline harmless requests.

A careful team therefore treats instruction data as a product: it is versioned, audited and tested, and every change in the mixture is checked against both capability and safety evaluations.

Understanding it simply

A freshly pretrained language model is like a well-read person who has never held a job. It knows a great deal, but when asked something, it may ramble or reply in an odd way.

Instruction tuning is the on-the-job training. The model is shown many examples of a request followed by a good answer, and it gradually learns that a question deserves a direct, useful reply. It learns manners and format, not new facts, which is why the quality of the examples matters so much.

Applied in practice

A legal technology team wants an assistant that summarizes contracts into a fixed structure of parties, obligations, deadlines and risks. The team starts from an open-weight base model with about 8 billion parameters, which produces fluent text but ignores the requested structure.

The team writes about 2,000 instruction and response pairs with input from two contract lawyers, adds 3,000 general instructions to prevent forgetting, and trains LoRA adapters for two epochs on a single GPU. Loss is masked to the responses, and the chat template matches the one used in production.

On a held-out set of 200 contracts, the share of summaries that follow the required structure rises from under 30 percent to above 90 percent, and lawyers rate the tuned model as clearly more usable, while general question answering remains close to the base model.