Preference data

Also known as: preference dataset

On this page 7

Preference data is a collection of records in which a human or a model judges which of several candidate responses is better for the same prompt, typically stored as a prompt, a chosen response and a rejected response. In artificial intelligence, a preference dataset is the main input for alignment training after pretraining and supervised fine-tuning, because it teaches a language model what people consider helpful, honest and safe rather than only what text is likely.

Preference data matters because many qualities of a good answer are easier to compare than to define. An annotator may struggle to write the ideal reply to a medical question, yet can reliably say which of two drafts is clearer and more accurate. Training methods such as RLHF, DPO and RLAIF turn these relative judgments into a learning signal.

How preference data is structured

A preference record pairs one prompt with two or more completions and a label that orders them. The most common form is the pairwise comparison, where the label simply marks one completion as preferred. Richer formats add rankings over four to nine responses, numeric ratings on a Likert scale, or written rationales explaining the decision.

The table below lists the fields that a typical preference record contains and why each one is kept.

FieldContentPurpose
PromptUser instruction or multi-turn conversationDefines the context the responses answer
Chosen responseCompletion judged betterPositive training target
Rejected responseCompletion judged worseNegative contrast for the loss
Preference strengthMargin such as slightly or much betterWeights or filters uncertain pairs
Annotator metadataRater ID, guideline version, time spentSupports quality control and audits
RationaleShort written justificationExplains the label and trains critique models
Fields in a typical preference record

Rankings over more than two responses are often expanded into all pairwise combinations: a ranking of four responses yields six pairs. This multiplies the training signal per annotation, but the derived pairs are statistically correlated, so they should not be treated as independent samples when estimating label quality.

Sources of preference labels

Preference labels come from human annotators, AI judges or implicit user behavior, and each source trades cost against fidelity. Professional annotators following detailed guidelines produce the most trustworthy labels, while AI feedback scales cheaply but inherits the biases of the judging model.

The following comparison summarizes the main label sources used in practice.

SourceTypical cost per comparisonStrengthMain weakness
Expert annotatorsHigh, often several dollarsDomain accuracy and nuanceSlow, small volume
Crowd annotatorsLow to moderateVolume and diversityNoisy, prone to shortcuts
AI feedback (RLAIF)Fractions of a centScale and consistencyInherits judge model bias
Implicit user signalsNear zeroReflects real usageConfounded by position and habit
Constitution-guided AI labelsLowExplicit written principlesOnly as good as the principles
Common sources of preference labels

Many production pipelines mix these sources. A common pattern is to collect a smaller human-labeled core set for high-stakes behaviors such as safety refusals and to label routine helpfulness comparisons with an AI judge calibrated against humans.

How training methods consume preference data

Preference data feeds alignment either indirectly through a reward model or directly through a preference loss. In classic RLHF, a reward model is first trained to score responses so that chosen completions receive higher scores, usually with the Bradley-Terry objective; a policy is then optimized against that reward with an algorithm such as PPO. Direct Preference Optimization (DPO) skips the separate reward model and adjusts the policy so that the log-probability ratio of chosen to rejected responses increases relative to a frozen reference model.

MethodUses a reward modelData requirementTypical trade-off
RLHF with PPOYesPairwise or ranked comparisonsFlexible but complex and unstable
DPONoPairwise chosen and rejectedSimple, sensitive to data quality
KTONoSingle responses marked good or badWorks with unpaired labels
Rejection sampling fine-tuningYes, for rankingMany samples per promptStable, less expressive
Online iterative DPOOptionalFresh pairs from the current policyReduces distribution mismatch
Training methods that use preference data

A key distinction is between offline and on-policy data. Offline preference data is collected once from earlier models, so the trained policy may drift into regions where the data says little. On-policy data is regenerated from the current model during training and usually gives stronger alignment gains at the cost of repeated labeling.

What makes preference data high quality

High-quality preference data combines clear guidelines, informative pairs and measured agreement. Guidelines must define the criteria, such as helpfulness, harmlessness and factual accuracy, and state how to resolve conflicts between them. Without that, annotators silently apply personal standards and the dataset encodes a mixture of incompatible preferences.

  • Informative pairs: comparisons between two nearly identical responses or between an excellent and an absurd one teach little; the most useful pairs differ in one meaningful aspect.
  • Inter-annotator agreement: agreement rates between independent raters on pairwise tasks often fall in the 60 to 80 percent range, which sets a practical ceiling on reward model accuracy.
  • Prompt coverage: the prompts should match the real distribution of tasks, languages and difficulty levels the model will face.
  • Balanced length and style: if chosen responses are systematically longer, the model learns length as a proxy for quality.

Quality teams monitor these properties continuously, because a few thousand systematically mislabeled pairs can shift model behavior noticeably, especially with direct methods like DPO that have no reward model to smooth out noise.

Limitations and failure modes

Preference data captures what raters prefer, not what is true or beneficial, and that gap produces several well-documented failure modes. The most cited is reward hacking, where the policy exploits regularities in the labels rather than improving the underlying quality.

SymptomLikely cause in the dataMitigation
Verbose answersLonger responses chosen more oftenLength-controlled pairs and length penalties
SycophancyRaters reward agreement with their viewsGuidelines that reward correction of false premises
Confident hallucinationsFluent but wrong answers preferredExpert review and fact-check labels
Over-refusalCautious responses chosen by defaultSeparate helpfulness and harm criteria
Narrow cultural defaultsHomogeneous rater poolDiverse annotators and per-group analysis
Symptoms, causes and mitigations of preference data problems

Another limitation is aggregation. A single binary label hides genuine disagreement among people with different values, so the resulting model reflects an averaged preference that may suit no group well. Research into pluralistic alignment explores keeping annotator identity and disagreement in the data instead of collapsing it.

The short version

Preference data is a long list of small taste tests. Someone asks a question, the model writes two answers, and a person or another model picks the better one. After many thousands of such picks, the model learns which kinds of answers people tend to like.

It works because judging is easier than writing. Most people cannot write a perfect essay, but they can usually tell which of two essays is better. The catch is that the model learns the judges’ habits too, so careless or biased judges produce a careless or biased model.

Where it makes a difference

A team building a customer support assistant for a software company finds that its fine-tuned model answers correctly but often buries the solution under long apologies. The team samples four responses for each of 12,000 real support questions, and trained support agents rank them using a guideline that rewards a direct fix first, accurate steps and a polite but brief tone.

The rankings are expanded into pairs, and the team checks that chosen and rejected responses have similar average length so the model does not simply learn to be short. After one round of DPO, internal reviewers prefer the new model in about 70 percent of blind comparisons, and the average reply shrinks by roughly a third without losing correct troubleshooting steps.