Rejection sampling fine-tuning

On this page 7

Rejection sampling fine-tuning is a post-training method in which a language model generates several candidate responses per prompt, an external scorer such as a reward model or automatic verifier accepts only the best or the correct ones, and the model is then fine-tuned with ordinary supervised learning on the accepted samples. In artificial intelligence training, it sits between supervised fine-tuning and reinforcement learning: it uses a reward signal to select data, yet it updates the weights with a simple cross-entropy loss instead of a policy gradient.

The method appears under several names, including rejection sampling, RFT, best-of-n fine-tuning, expert iteration and, in reasoning research, self-taught reasoning. All variants share one idea: filter the model’s own outputs, then imitate the filtered set.

How rejection sampling fine-tuning works

Rejection sampling fine-tuning works as a generate, score, filter and train loop. A starting model, usually already instruction-tuned, produces k completions for each prompt at a nonzero sampling temperature, so the candidates differ in content and quality.

Each candidate then receives a score. For tasks with checkable answers, such as arithmetic or code, a programmatic verifier compares the final answer with a reference or runs unit tests. For open-ended tasks, a learned reward model assigns a scalar preference score. The pipeline keeps the top-scoring sample per prompt, or every sample above a threshold, and discards the rest.

The following table summarizes the stages of one iteration and what each stage controls.

StageWhat happensMain control
SamplingModel generates k candidates per promptNumber of samples, temperature
ScoringVerifier or reward model rates each candidateScorer quality and calibration
FilteringBest or passing candidates are keptTop-1, top-m or score threshold
TrainingSupervised fine-tuning on kept samplesLearning rate, epochs, data mixture
IterationUpdated model samples againNumber of rounds, fresh prompts
Stages of one rejection sampling iteration

Because the final step is plain supervised training, the procedure inherits the stability and simplicity of SFT. There is no value network, no clipping objective and no online rollout during the gradient step, which makes it attractive for teams without large reinforcement learning infrastructure.

Key parameters and their effects

The quality of rejection sampling fine-tuning depends on a small set of parameters that trade compute against data quality and diversity. Typical published settings use between 4 and 64 samples per prompt, although values in the hundreds appear for hard reasoning problems.

ParameterTypical rangeEffect of increasing it
Samples per prompt (k)4 to 64Higher chance of a correct or excellent sample, linear growth in cost
Sampling temperature0.6 to 1.0More diverse candidates, more low-quality ones
Acceptance thresholdTop-1 to top-25%Stricter filter gives cleaner data but fewer examples
Distinct samples kept1 to several per promptMore reasoning paths, risk of near-duplicates
Number of rounds1 to 5Gradual improvement, rising risk of collapse
Parameters that shape the accepted dataset

The expected gain from sampling more grows roughly with the logarithm of k, so doubling the sample count yields diminishing returns. Keeping several distinct correct solutions per prompt, after removing duplicates, often helps more than raising k, because diversity of reasoning paths improves generalization.

Comparison with SFT, RLHF and DPO

Rejection sampling fine-tuning differs from neighboring post-training methods mainly in where the training data comes from and how the reward signal enters the update. The table contrasts it with the most common alternatives in the Training topic.

MethodTraining data sourceHow reward is usedRelative complexity
Supervised fine-tuningHuman-written demonstrationsNot usedLow
Rejection sampling fine-tuningModel’s own filtered outputsSelects which samples to keepLow to moderate
RLHF with PPOOnline model rolloutsDrives a policy gradientHigh
DPOPairs of preferred and rejected responsesImplicit, through preference pairsModerate
DistillationOutputs of a larger teacher modelUsually not usedLow
Rejection sampling compared with related post-training methods

A useful mental model treats rejection sampling as offline reinforcement learning with a binary filter. It learns only from positive examples, which is why it is sample-inefficient compared with PPO but far more robust. DPO, by contrast, also learns from the rejected samples. Many pipelines therefore combine the two: rejection sampling first lifts the model’s baseline, and preference optimization then refines it.

Where rejection sampling fine-tuning is used

Rejection sampling fine-tuning is most effective when a reliable scorer exists and the base model already solves the task some of the time. If the model never produces a correct answer, there is nothing to keep, and the method stalls.

  • Mathematical reasoning, where final answers can be checked exactly against references.
  • Code generation, where unit tests act as an execution-based verifier.
  • Chat alignment rounds, where a reward model selects the most helpful and harmless response, as in several open-weight model releases.
  • Tool use and agent traces, where a successful end state confirms that a trajectory worked.
  • Bootstrapping chain-of-thought, where only rationales that lead to correct answers are kept.

In practice, teams often run two to four alternating rounds of sampling and training, refreshing prompts each round and mixing in a portion of the original human-written SFT data to anchor style and reduce drift.

Limitations, risks and common mistakes

The method amplifies whatever the scorer rewards, so scorer errors become training data. A reward model with blind spots invites reward hacking, and an answer-only verifier accepts right answers reached by flawed reasoning.

SymptomLikely causeRemedy
Shrinking output diversityRepeated self-imitation over many roundsKeep multiple distinct samples, limit rounds
Verbose or formulaic answersReward model favors length or styleLength-normalize scores, audit the reward model
No gains on hard promptsModel rarely solves them, so none are keptIncrease k, add hints or teacher samples
Correct answers, wrong reasoningVerifier checks only final answersAdd process checks or a process reward model
Loss of general skillsNarrow accepted datasetMix in general SFT data, watch for catastrophic forgetting
Typical failure modes and remedies

Another frequent mistake is evaluating on the same prompts used for sampling. Because the model has seen its own best answers to those prompts, scores rise without real generalization, so a held-out evaluation set is essential.

Explained without jargon

Imagine a student who answers each homework question several times, then a teacher marks the answers and the student studies only the ones marked correct. Next week, the student starts from a better level and repeats the exercise.

Rejection sampling fine-tuning does the same with an AI model. The model writes many attempts, a checker keeps the good ones, and the model learns from its own best work. The approach is simple and stable, but it only works if the checker is trustworthy and the model can already get things right at least occasionally.

A typical use case

A research team wants a 7-billion-parameter model to solve grade-school math word problems more reliably. The instruction-tuned model answers about 40% of 8,000 training problems correctly with a single attempt. The team samples 16 solutions per problem at temperature 0.8 and checks each final number against the reference answer.

At least one correct solution appears for roughly 75% of the problems. After removing duplicates, the team keeps up to four distinct correct solutions per problem, about 20,000 examples in total, and fine-tunes the model for two epochs together with a slice of the original instruction data.

On a held-out test set, single-attempt accuracy rises by several percentage points. A second round on the improved model adds a smaller gain, and the team stops when output diversity begins to fall, a sign that further rounds would bring diminishing returns.