RLHF (reinforcement learning from human feedback) is a training method that aligns a pretrained model with human preferences by learning a reward signal from human judgments and then optimizing the model against that reward with reinforcement learning. In the training of large language models, RLHF is the stage that turns a capable text predictor into an assistant that follows instructions, stays on topic and avoids harmful output.
The method addresses a gap that ordinary supervised fine-tuning cannot close: many qualities people care about, such as helpfulness, clarity or tone, are easy to compare but hard to specify. RLHF exploits that asymmetry by asking people which of two answers is better rather than asking them to write the ideal answer.
The three stages of the RLHF pipeline
Classic RLHF runs in three stages, each building on the previous one. The pipeline starts from a pretrained base model and ends with a policy model tuned to maximize a learned estimate of human preference.
| Stage | Input data | Output |
|---|---|---|
| Supervised fine-tuning (SFT) | Demonstrations of good responses, often thousands to tens of thousands | Instruction-following starting policy |
| Reward model training | Pairwise human comparisons of model responses | Model that scores any response |
| Reinforcement learning | Prompts plus reward model scores | Policy optimized toward higher reward |
The reward model is usually a copy of the language model with its output head replaced by a single scalar output. It is trained with a Bradley-Terry style loss, which pushes the score of the preferred response above the score of the rejected one. Comparisons are preferred over absolute ratings because people are more consistent when comparing than when scoring on a scale.
The reinforcement learning step and its parameters
In the reinforcement learning stage, the model generates responses to prompts, the reward model scores them and a policy-gradient algorithm updates the model. Proximal Policy Optimization (PPO) is the traditional choice because its clipped updates keep training stable.
A KL divergence penalty against the frozen SFT model is added to the reward. It prevents the policy from drifting into text that fools the reward model but no longer reads like natural language. The strength of this penalty is the most important tuning knob in RLHF.
| Parameter | Typical setting | Effect when increased |
|---|---|---|
| KL coefficient (beta) | Roughly 0.01-0.2, sometimes adaptive | Safer, closer to SFT, smaller gains |
| PPO clip range | Around 0.1-0.2 | Larger, riskier policy updates |
| Learning rate | Very small, often around 1e-6 to 1e-5 | Faster change, higher instability |
| Responses per prompt | 1 to 16 or more | Better advantage estimates, higher compute |
| Reward model size | Similar to or smaller than the policy | Better judgments, more memory |
PPO-based RLHF keeps four models in memory: the policy, a value model, the reward model and the frozen reference model. This makes it expensive and operationally complex compared with other fine-tuning methods.
RLHF compared with other training methods
RLHF is one of several preference optimization approaches, and newer methods often reach similar quality with simpler pipelines. The choice depends on budget, data and how much control the team needs.
| Method | What it learns from | Main trade-off |
|---|---|---|
| Supervised fine-tuning | Example answers | Simple, but only imitates the demonstrations |
| RLHF with PPO | Pairwise preferences through a reward model | Flexible and strong, but complex and costly |
| DPO | Pairwise preferences directly, no separate reward model | Simpler and stable, less online exploration |
| RLAIF | Preferences labeled by an AI model following written principles | Cheaper labels, inherits the labeler’s biases |
| RL with verifiable rewards | Automatic checks such as unit tests or exact answers | Objective signal, limited to checkable tasks |
Parameter-efficient methods such as LoRA and QLoRA are not alternatives to RLHF but ways to run it: adapters can be trained during the RL stage to reduce memory. Distillation and quantization, in turn, usually happen after alignment to make the final model cheaper to serve.
Known failure modes
The central weakness of RLHF is that the policy optimizes a learned proxy, not human preference itself. When pushed too hard, the policy finds ways to raise the proxy score without becoming better, a problem known as reward hacking or overoptimization.
- Length bias: longer answers often win comparisons, so models become verbose.
- Sycophancy: agreeing with the user is rewarded even when the user is wrong.
- Mode collapse: output diversity drops as the policy converges on safe templates.
- Over-refusal: overly cautious preference data teaches the model to decline harmless requests.
- Annotator inconsistency: disagreement among raters, which is substantial on subjective pairs, limits reward model accuracy.
Mitigations include length-controlled evaluation, a stronger KL penalty, ensembles of reward models, regular refreshes of preference data from the current policy and detailed annotation guidelines that define what helpful and harmless mean in concrete cases.
Practical considerations for teams
Running full RLHF is justified when a team has substantial preference data, the infrastructure to train several large models at once and a clear quality goal that supervised data cannot express. For most organizations adapting an open model, DPO or a similar offline method offers a better cost-to-benefit ratio.
Whatever the method, data quality dominates. A few thousand carefully labeled comparisons with clear guidelines frequently beat a much larger set of noisy labels, and held-out human evaluation remains necessary because reward model scores alone can rise while real quality falls.
The idea in brief
RLHF teaches an AI model what people actually prefer. First the model learns from good examples. Then people look at pairs of its answers and pick the better one, again and again.
From those choices, a second model learns to predict which answers people will like. The main model then practices, gets scored by that predictor and slowly adjusts toward higher scores. It is similar to a trainee who improves through feedback on drafts rather than by memorizing a rulebook, with a limit that stops it from changing too far from its original training.
A practical scenario
A research team adapts a 7-billion-parameter open model for a customer-service assistant. After supervised fine-tuning on 15,000 example conversations, the assistant answers correctly but often ignores company tone guidelines and gives overly long replies.
The team collects about 20,000 pairwise comparisons from trained support staff, with written guidelines that reward concise, accurate and polite answers. They train a reward model on these pairs and run PPO with a KL coefficient of 0.05, monitoring average answer length and a held-out human evaluation every few hundred steps.
After training, staff prefer the new assistant’s answers in a clear majority of blind comparisons, and answers are shorter on average. When the reward keeps climbing while human ratings flatten, the team stops training early, a decision that prevents the reward hacking seen in an earlier, longer run.