Preference data is a collection of records in which a human or a model judges which of several candidate responses is better for the same prompt, typically stored as a prompt, a chosen response and a rejected response. In artificial intelligence, a preference dataset is the main input for alignment training after pretraining and supervised fine-tuning, because it teaches a language model what people consider helpful, honest and safe rather than only what text is likely.
Preference data matters because many qualities of a good answer are easier to compare than to define. An annotator may struggle to write the ideal reply to a medical question, yet can reliably say which of two drafts is clearer and more accurate. Training methods such as RLHF, DPO and RLAIF turn these relative judgments into a learning signal.
How preference data is structured
A preference record pairs one prompt with two or more completions and a label that orders them. The most common form is the pairwise comparison, where the label simply marks one completion as preferred. Richer formats add rankings over four to nine responses, numeric ratings on a Likert scale, or written rationales explaining the decision.
The table below lists the fields that a typical preference record contains and why each one is kept.
| Field | Content | Purpose |
|---|---|---|
| Prompt | User instruction or multi-turn conversation | Defines the context the responses answer |
| Chosen response | Completion judged better | Positive training target |
| Rejected response | Completion judged worse | Negative contrast for the loss |
| Preference strength | Margin such as slightly or much better | Weights or filters uncertain pairs |
| Annotator metadata | Rater ID, guideline version, time spent | Supports quality control and audits |
| Rationale | Short written justification | Explains the label and trains critique models |
Rankings over more than two responses are often expanded into all pairwise combinations: a ranking of four responses yields six pairs. This multiplies the training signal per annotation, but the derived pairs are statistically correlated, so they should not be treated as independent samples when estimating label quality.
Sources of preference labels
Preference labels come from human annotators, AI judges or implicit user behavior, and each source trades cost against fidelity. Professional annotators following detailed guidelines produce the most trustworthy labels, while AI feedback scales cheaply but inherits the biases of the judging model.
The following comparison summarizes the main label sources used in practice.
| Source | Typical cost per comparison | Strength | Main weakness |
|---|---|---|---|
| Expert annotators | High, often several dollars | Domain accuracy and nuance | Slow, small volume |
| Crowd annotators | Low to moderate | Volume and diversity | Noisy, prone to shortcuts |
| AI feedback (RLAIF) | Fractions of a cent | Scale and consistency | Inherits judge model bias |
| Implicit user signals | Near zero | Reflects real usage | Confounded by position and habit |
| Constitution-guided AI labels | Low | Explicit written principles | Only as good as the principles |
Many production pipelines mix these sources. A common pattern is to collect a smaller human-labeled core set for high-stakes behaviors such as safety refusals and to label routine helpfulness comparisons with an AI judge calibrated against humans.
How training methods consume preference data
Preference data feeds alignment either indirectly through a reward model or directly through a preference loss. In classic RLHF, a reward model is first trained to score responses so that chosen completions receive higher scores, usually with the Bradley-Terry objective; a policy is then optimized against that reward with an algorithm such as PPO. Direct Preference Optimization (DPO) skips the separate reward model and adjusts the policy so that the log-probability ratio of chosen to rejected responses increases relative to a frozen reference model.
| Method | Uses a reward model | Data requirement | Typical trade-off |
|---|---|---|---|
| RLHF with PPO | Yes | Pairwise or ranked comparisons | Flexible but complex and unstable |
| DPO | No | Pairwise chosen and rejected | Simple, sensitive to data quality |
| KTO | No | Single responses marked good or bad | Works with unpaired labels |
| Rejection sampling fine-tuning | Yes, for ranking | Many samples per prompt | Stable, less expressive |
| Online iterative DPO | Optional | Fresh pairs from the current policy | Reduces distribution mismatch |
A key distinction is between offline and on-policy data. Offline preference data is collected once from earlier models, so the trained policy may drift into regions where the data says little. On-policy data is regenerated from the current model during training and usually gives stronger alignment gains at the cost of repeated labeling.
What makes preference data high quality
High-quality preference data combines clear guidelines, informative pairs and measured agreement. Guidelines must define the criteria, such as helpfulness, harmlessness and factual accuracy, and state how to resolve conflicts between them. Without that, annotators silently apply personal standards and the dataset encodes a mixture of incompatible preferences.
- Informative pairs: comparisons between two nearly identical responses or between an excellent and an absurd one teach little; the most useful pairs differ in one meaningful aspect.
- Inter-annotator agreement: agreement rates between independent raters on pairwise tasks often fall in the 60 to 80 percent range, which sets a practical ceiling on reward model accuracy.
- Prompt coverage: the prompts should match the real distribution of tasks, languages and difficulty levels the model will face.
- Balanced length and style: if chosen responses are systematically longer, the model learns length as a proxy for quality.
Quality teams monitor these properties continuously, because a few thousand systematically mislabeled pairs can shift model behavior noticeably, especially with direct methods like DPO that have no reward model to smooth out noise.
Limitations and failure modes
Preference data captures what raters prefer, not what is true or beneficial, and that gap produces several well-documented failure modes. The most cited is reward hacking, where the policy exploits regularities in the labels rather than improving the underlying quality.
| Symptom | Likely cause in the data | Mitigation |
|---|---|---|
| Verbose answers | Longer responses chosen more often | Length-controlled pairs and length penalties |
| Sycophancy | Raters reward agreement with their views | Guidelines that reward correction of false premises |
| Confident hallucinations | Fluent but wrong answers preferred | Expert review and fact-check labels |
| Over-refusal | Cautious responses chosen by default | Separate helpfulness and harm criteria |
| Narrow cultural defaults | Homogeneous rater pool | Diverse annotators and per-group analysis |
Another limitation is aggregation. A single binary label hides genuine disagreement among people with different values, so the resulting model reflects an averaged preference that may suit no group well. Research into pluralistic alignment explores keeping annotator identity and disagreement in the data instead of collapsing it.
The short version
Preference data is a long list of small taste tests. Someone asks a question, the model writes two answers, and a person or another model picks the better one. After many thousands of such picks, the model learns which kinds of answers people tend to like.
It works because judging is easier than writing. Most people cannot write a perfect essay, but they can usually tell which of two essays is better. The catch is that the model learns the judges’ habits too, so careless or biased judges produce a careless or biased model.
Where it makes a difference
A team building a customer support assistant for a software company finds that its fine-tuned model answers correctly but often buries the solution under long apologies. The team samples four responses for each of 12,000 real support questions, and trained support agents rank them using a guideline that rewards a direct fix first, accurate steps and a polite but brief tone.
The rankings are expanded into pairs, and the team checks that chosen and rejected responses have similar average length so the model does not simply learn to be short. After one round of DPO, internal reviewers prefer the new model in about 70 percent of blind comparisons, and the average reply shrinks by roughly a third without losing correct troubleshooting steps.