Rejection sampling fine-tuning is a post-training method in which a language model generates several candidate responses per prompt, an external scorer such as a reward model or automatic verifier accepts only the best or the correct ones, and the model is then fine-tuned with ordinary supervised learning on the accepted samples. In artificial intelligence training, it sits between supervised fine-tuning and reinforcement learning: it uses a reward signal to select data, yet it updates the weights with a simple cross-entropy loss instead of a policy gradient.
The method appears under several names, including rejection sampling, RFT, best-of-n fine-tuning, expert iteration and, in reasoning research, self-taught reasoning. All variants share one idea: filter the model’s own outputs, then imitate the filtered set.
How rejection sampling fine-tuning works
Rejection sampling fine-tuning works as a generate, score, filter and train loop. A starting model, usually already instruction-tuned, produces k completions for each prompt at a nonzero sampling temperature, so the candidates differ in content and quality.
Each candidate then receives a score. For tasks with checkable answers, such as arithmetic or code, a programmatic verifier compares the final answer with a reference or runs unit tests. For open-ended tasks, a learned reward model assigns a scalar preference score. The pipeline keeps the top-scoring sample per prompt, or every sample above a threshold, and discards the rest.
The following table summarizes the stages of one iteration and what each stage controls.
| Stage | What happens | Main control |
|---|---|---|
| Sampling | Model generates k candidates per prompt | Number of samples, temperature |
| Scoring | Verifier or reward model rates each candidate | Scorer quality and calibration |
| Filtering | Best or passing candidates are kept | Top-1, top-m or score threshold |
| Training | Supervised fine-tuning on kept samples | Learning rate, epochs, data mixture |
| Iteration | Updated model samples again | Number of rounds, fresh prompts |
Because the final step is plain supervised training, the procedure inherits the stability and simplicity of SFT. There is no value network, no clipping objective and no online rollout during the gradient step, which makes it attractive for teams without large reinforcement learning infrastructure.
Key parameters and their effects
The quality of rejection sampling fine-tuning depends on a small set of parameters that trade compute against data quality and diversity. Typical published settings use between 4 and 64 samples per prompt, although values in the hundreds appear for hard reasoning problems.
| Parameter | Typical range | Effect of increasing it |
|---|---|---|
| Samples per prompt (k) | 4 to 64 | Higher chance of a correct or excellent sample, linear growth in cost |
| Sampling temperature | 0.6 to 1.0 | More diverse candidates, more low-quality ones |
| Acceptance threshold | Top-1 to top-25% | Stricter filter gives cleaner data but fewer examples |
| Distinct samples kept | 1 to several per prompt | More reasoning paths, risk of near-duplicates |
| Number of rounds | 1 to 5 | Gradual improvement, rising risk of collapse |
The expected gain from sampling more grows roughly with the logarithm of k, so doubling the sample count yields diminishing returns. Keeping several distinct correct solutions per prompt, after removing duplicates, often helps more than raising k, because diversity of reasoning paths improves generalization.
Comparison with SFT, RLHF and DPO
Rejection sampling fine-tuning differs from neighboring post-training methods mainly in where the training data comes from and how the reward signal enters the update. The table contrasts it with the most common alternatives in the Training topic.
| Method | Training data source | How reward is used | Relative complexity |
|---|---|---|---|
| Supervised fine-tuning | Human-written demonstrations | Not used | Low |
| Rejection sampling fine-tuning | Model’s own filtered outputs | Selects which samples to keep | Low to moderate |
| RLHF with PPO | Online model rollouts | Drives a policy gradient | High |
| DPO | Pairs of preferred and rejected responses | Implicit, through preference pairs | Moderate |
| Distillation | Outputs of a larger teacher model | Usually not used | Low |
A useful mental model treats rejection sampling as offline reinforcement learning with a binary filter. It learns only from positive examples, which is why it is sample-inefficient compared with PPO but far more robust. DPO, by contrast, also learns from the rejected samples. Many pipelines therefore combine the two: rejection sampling first lifts the model’s baseline, and preference optimization then refines it.
Where rejection sampling fine-tuning is used
Rejection sampling fine-tuning is most effective when a reliable scorer exists and the base model already solves the task some of the time. If the model never produces a correct answer, there is nothing to keep, and the method stalls.
- Mathematical reasoning, where final answers can be checked exactly against references.
- Code generation, where unit tests act as an execution-based verifier.
- Chat alignment rounds, where a reward model selects the most helpful and harmless response, as in several open-weight model releases.
- Tool use and agent traces, where a successful end state confirms that a trajectory worked.
- Bootstrapping chain-of-thought, where only rationales that lead to correct answers are kept.
In practice, teams often run two to four alternating rounds of sampling and training, refreshing prompts each round and mixing in a portion of the original human-written SFT data to anchor style and reduce drift.
Limitations, risks and common mistakes
The method amplifies whatever the scorer rewards, so scorer errors become training data. A reward model with blind spots invites reward hacking, and an answer-only verifier accepts right answers reached by flawed reasoning.
| Symptom | Likely cause | Remedy |
|---|---|---|
| Shrinking output diversity | Repeated self-imitation over many rounds | Keep multiple distinct samples, limit rounds |
| Verbose or formulaic answers | Reward model favors length or style | Length-normalize scores, audit the reward model |
| No gains on hard prompts | Model rarely solves them, so none are kept | Increase k, add hints or teacher samples |
| Correct answers, wrong reasoning | Verifier checks only final answers | Add process checks or a process reward model |
| Loss of general skills | Narrow accepted dataset | Mix in general SFT data, watch for catastrophic forgetting |
Another frequent mistake is evaluating on the same prompts used for sampling. Because the model has seen its own best answers to those prompts, scores rise without real generalization, so a held-out evaluation set is essential.
Explained without jargon
Imagine a student who answers each homework question several times, then a teacher marks the answers and the student studies only the ones marked correct. Next week, the student starts from a better level and repeats the exercise.
Rejection sampling fine-tuning does the same with an AI model. The model writes many attempts, a checker keeps the good ones, and the model learns from its own best work. The approach is simple and stable, but it only works if the checker is trustworthy and the model can already get things right at least occasionally.
A typical use case
A research team wants a 7-billion-parameter model to solve grade-school math word problems more reliably. The instruction-tuned model answers about 40% of 8,000 training problems correctly with a single attempt. The team samples 16 solutions per problem at temperature 0.8 and checks each final number against the reference answer.
At least one correct solution appears for roughly 75% of the problems. After removing duplicates, the team keeps up to four distinct correct solutions per problem, about 20,000 examples in total, and fine-tunes the model for two epochs together with a slice of the original instruction data.
On a held-out test set, single-attempt accuracy rises by several percentage points. A second round on the improved model adds a smaller gain, and the team stops when output diversity begins to fall, a sign that further rounds would bring diminishing returns.