Supervised fine-tuning

Also known as: SFT

On this page 7

Supervised fine-tuning (SFT) is the training stage in which a pretrained model is further trained on labeled input and output pairs, so that it learns to produce the desired response for a given input. In artificial intelligence, and especially in the training of large language models, SFT is usually the first step of post-training: it turns a model that only predicts the next token of raw text into one that follows instructions, answers in a chosen format and adopts a consistent style.

The method is called supervised because every example carries an explicit target answer written or approved by people, or generated and filtered by another model. The model is not rewarded or ranked; it is simply taught to imitate demonstrations, which makes SFT predictable, cheap relative to pretraining and the foundation on which preference methods such as RLHF and DPO are built.

How supervised fine-tuning trains a model

Supervised fine-tuning works by minimizing the cross-entropy loss on target tokens while starting from the weights of an already pretrained model. Each training example contains a prompt, for example a user question, and a completion, for example the ideal answer. The two are concatenated into one sequence and passed through the network with teacher forcing, so the model sees the correct previous tokens at every position.

A key detail is loss masking: the loss is usually computed only on the completion tokens, not on the prompt. The model therefore learns to produce good answers rather than to reproduce questions. Gradients flow back through the network and an optimizer, typically the Adam optimizer with weight decay, updates the parameters with a learning rate far smaller than in pretraining.

The process runs for a small number of passes over the data, commonly one to five epochs. More passes on a small dataset quickly lead to memorization, because the model already has enormous capacity and the dataset is tiny compared with its pretraining corpus.

The following table lists the main hyperparameters of a typical SFT run for a language model and what each one controls.

ParameterCommon rangeWhat it affects
Learning rate1e-6 to 2e-5 (full); 1e-4 to 3e-4 (LoRA)Speed of adaptation and risk of forgetting
Epochs1 to 5Fit to the data versus memorization
Batch size32 to 512 sequencesGradient noise and training stability
Sequence length2,048 to 32,768 tokensLongest example the model can learn from
Warmup1 to 5 percent of stepsStability of the first updates
Loss maskingCompletion tokens onlyWhether prompts are learned or ignored
Typical SFT hyperparameters for language models

The SFT dataset and its quality

The dataset is the decisive factor in supervised fine-tuning, because the model will imitate whatever the demonstrations show, including their mistakes. Experience across many projects shows that a few thousand carefully curated examples often outperform hundreds of thousands of noisy ones, a finding sometimes summarized as quality over quantity.

Data comes from several sources, each with its own strengths and weaknesses, as the table below summarizes.

SourceStrengthMain weakness
Human-written demonstrationsHigh quality and control of styleSlow and expensive to produce
Converted public datasetsLarge volume and task varietyRigid templates and short answers
Synthetic data from a stronger modelCheap and scalableInherits errors and limits of the source model
Rejection-sampled outputsVerified correctness for math or codeRequires a reliable checker or reward model
Production logs with correctionsMatches real user requestsPrivacy review and consent are required
Common sources of SFT data

Good SFT datasets share a few properties: diverse tasks and phrasings, answers that are factually correct and consistently formatted, balanced lengths and the removal of near-duplicates. Data labeling guidelines, inter-annotator review and automatic filters are standard tools for keeping quality high. Methods such as Self-Instruct and rejection sampling fine-tuning are ways of producing SFT data at scale.

Supervised fine-tuning differs from neighboring methods mainly in what signal the model learns from. SFT learns from correct examples, whereas preference methods learn from comparisons between better and worse answers, and pretraining learns from unlabeled text.

MethodTraining signalTypical role
PretrainingNext-token prediction on raw textBuilds general knowledge and language ability
Supervised fine-tuningTarget answers for given inputsTeaches format, tasks and instruction following
Instruction tuningSFT on instruction and response pairsA common special case of SFT
RLHFReward model trained on human preferencesRefines helpfulness and harmlessness after SFT
DPOPairs of preferred and rejected answersSimpler preference optimization without a reward model
DistillationOutputs or probabilities of a teacher modelTransfers ability into a smaller student
How SFT relates to other training stages

The boundaries overlap. Instruction tuning is SFT applied to instruction data, and distillation becomes SFT when the student learns from the teacher’s generated text rather than its probability distribution. SFT also differs from continued pretraining, which adds domain text without targets and changes knowledge more than behavior.

Full fine-tuning versus parameter-efficient SFT

Supervised fine-tuning can update all model weights or only a small added set of parameters. Full fine-tuning gives the most capacity to change behavior but needs memory for the weights, gradients and optimizer states, often three to four times the model size in mixed-precision training.

Parameter-efficient methods reduce this cost sharply. LoRA freezes the original weights and trains small low-rank matrices, while QLoRA additionally keeps the frozen base in 4-bit quantization, which allows a model with tens of billions of parameters to be fine-tuned on a single high-memory GPU.

SituationRecommended approachReason
Limited GPU memoryQLoRALowest memory footprint with modest quality loss
Many task-specific variantsLoRA adaptersSmall adapters can be swapped on one base model
Deep change of behavior or languageFull fine-tuningAll parameters can adapt
Small dataset under 1,000 examplesLoRA with few epochsLimits overfitting and forgetting
Choosing an SFT approach

Limitations and common mistakes in SFT

Supervised fine-tuning mostly shapes behavior rather than adding knowledge. Facts that the pretrained model never saw are learned poorly from a few examples, and training on unfamiliar facts can even increase hallucination, because the model learns to answer confidently beyond what it knows.

  • Catastrophic forgetting: a high learning rate or too many epochs erode general abilities learned in pretraining.
  • Overfitting to style: the model copies surface patterns such as fixed openings or a single answer length.
  • Exposure bias: with teacher forcing the model never sees its own errors during training, so mistakes can compound at inference.
  • Imitation ceiling: SFT cannot exceed the quality of its demonstrations, which is one reason preference training follows it.
  • Contaminated evaluation: benchmark questions that leak into SFT data inflate reported scores.

Practitioners counter these risks with held-out validation sets, early stopping, mixing some general data into the domain data and evaluating both the target task and broad capabilities after each run.

Understanding it simply

A pretrained language model has read an enormous amount of text, but it does not yet know how to act as a helpful assistant. Supervised fine-tuning is like giving a well-read new employee a binder of model answers: here is a question, here is exactly how we reply.

The model studies these examples and adjusts itself to answer in the same way. It does not learn the world from scratch; it learns how to use what it already knows in the format and tone people want. The better the examples, the better the result.

Applied in practice

A hospital network wants an assistant that turns clinicians’ dictated notes into structured discharge summaries. A general open-weight model with 8 billion parameters writes fluent text but ignores the required headings and sometimes invents medication details.

The team assembles 4,000 de-identified note and summary pairs, each reviewed by a clinical documentation specialist. They run SFT with LoRA at a learning rate of 2e-4 for three epochs, masking the loss on the notes, and keep 400 pairs as a validation set. Training takes about four hours on one GPU.

After fine-tuning, the share of summaries with every required section rises from roughly 60 to over 95 percent, and reviewers flag fewer unsupported medication entries. The team still keeps a human sign-off step, because SFT improved format and faithfulness but cannot guarantee factual accuracy.