Inter-annotator agreement is the degree to which independent human annotators assign the same labels to the same items, measured with statistics that correct for agreement expected by chance. In model evaluation for artificial intelligence, it quantifies how reliable the human judgments behind training data, evaluation datasets and human evaluation studies really are.
Agreement matters because a model cannot be measured more precisely than its reference labels allow. When annotators disagree often, accuracy and F1 scores computed against their labels carry hidden noise, and differences between models may reflect annotation inconsistency rather than real capability. Agreement therefore sets a practical ceiling on meaningful evaluation.
How inter-annotator agreement is measured
Inter-annotator agreement is measured by comparing the labels that two or more annotators give to an overlapping sample of items. The simplest statistic is raw percent agreement, the share of items on which annotators chose the same label. It is easy to read but misleading, because annotators agree partly by chance, especially when one label dominates.
Chance-corrected coefficients solve this by comparing observed agreement with the agreement expected if annotators labeled at random according to their label frequencies. Most share the form (observed minus expected) divided by (one minus expected), producing a value where 1 means perfect agreement, 0 means chance level and negative values mean systematic disagreement.
The table below lists the coefficients most often used in machine learning annotation projects and when each one applies.
| Coefficient | Annotators | Data type | Key property |
|---|---|---|---|
| Percent agreement | Two or more | Any categorical | No chance correction |
| Cohen’s kappa | Exactly two | Nominal | Uses each rater’s own label distribution |
| Fleiss’ kappa | Fixed number, any raters | Nominal | Generalizes kappa to many raters |
| Krippendorff’s alpha | Any number | Nominal, ordinal, interval, ratio | Handles missing labels |
| Weighted kappa | Exactly two | Ordinal | Partial credit for near misses |
| Intraclass correlation | Two or more | Continuous scores | Based on variance components |
Interpreting agreement values
Agreement values are interpreted against the difficulty of the task, not against a universal standard. A widely cited scale from medical statistics labels kappa values above 0.8 as almost perfect, but these bands were proposed as rough conventions, not validated thresholds, and should not be applied mechanically.
The following table gives typical ranges seen in practice and how teams usually treat them. The actions are guidelines, and task subjectivity shifts every boundary.
| Value range | Common reading | Typical action |
|---|---|---|
| Above 0.80 | Strong agreement | Labels suitable as gold standard |
| 0.67 to 0.80 | Acceptable for many tasks | Use with stated caution |
| 0.40 to 0.67 | Moderate agreement | Revise guidelines, retrain annotators |
| 0.20 to 0.40 | Weak agreement | Redefine labels or task |
| Below 0.20 | Near chance | Treat labels as unreliable |
Content analysis traditions often require Krippendorff’s alpha of at least 0.667 for tentative conclusions and 0.8 for firm ones. For inherently subjective judgments, such as rating the helpfulness of chatbot answers, agreement of 0.5 to 0.7 is common even among careful experts, and that level reflects genuine ambiguity rather than annotator failure.
Why agreement matters for AI evaluation
Agreement determines how much trust a team can place in human-labeled benchmarks and in human evaluation of generative models. If annotators agree on only 70 percent of items, a model reporting 95 percent accuracy against one annotator’s labels is partly fitting annotator idiosyncrasies, and the true ceiling is unclear.
Agreement also serves as a reference point for automated judges. When an LLM-as-a-judge setup agrees with humans about as often as humans agree with each other, it is commonly considered a reasonable substitute. The table below shows where agreement analysis enters the evaluation workflow.
| Evaluation activity | Role of agreement | Risk if ignored |
|---|---|---|
| Benchmark construction | Validates gold labels | Noisy labels cap measurable accuracy |
| Pairwise preference ratings | Shows how consistent win-rate judgments are | Leaderboard gaps within noise |
| LLM-as-a-judge validation | Human-human agreement sets the target | Overtrusting automated scores |
| Preference data for RLHF | Estimates label noise in reward data | Reward model learns inconsistency |
| Error analysis | Separates model errors from ambiguous items | Misdiagnosed failure modes |
Improving agreement in annotation projects
Agreement improves most through clear annotation guidelines and iterative calibration, not through simply hiring more annotators. A standard process runs a pilot round, measures agreement, discusses disagreements, revises the guidelines and repeats until the coefficient stabilizes.
- Define labels with examples: include borderline cases and explicit decision rules for each category.
- Double-annotate a sample: overlap of 10 to 20 percent of items is a common compromise between cost and confidence.
- Run calibration sessions: discuss disagreements to surface ambiguous wording in the guidelines.
- Adjudicate conflicts: a senior annotator or majority vote resolves disagreements for the final gold label.
- Track agreement over time: annotator drift can lower consistency in long projects.
Some disagreement should be preserved rather than eliminated. Research on perspectivist annotation argues that, for tasks such as toxicity or offensiveness, the distribution of annotator opinions is itself informative, and collapsing it into one label hides legitimate differences in judgment.
Limitations and common mistakes
Agreement statistics have known quirks that can mislead teams who report a single number without context. The most frequent problem is the kappa paradox: when one label is very rare, kappa can be low even when raw agreement is above 95 percent, because expected chance agreement is already high.
- Reporting percent agreement alone overstates reliability on imbalanced label sets.
- Using Cohen’s kappa for many raters by averaging pairs instead of choosing Fleiss’ kappa or Krippendorff’s alpha.
- Ignoring ordinal structure, which treats a one-point difference on a five-point scale as a full disagreement.
- Measuring agreement on easy items only, which inflates the coefficient compared with the full dataset.
- Equating high agreement with correctness: annotators can agree consistently and still be wrong if they share a bias.
Reporting should state the coefficient used, the number of annotators, the overlap sample size and, ideally, a confidence interval, so that readers can judge how stable the estimate is.
In everyday language
Picture three teachers grading the same essays without talking to each other. If they almost always give the same grade, the grading is trustworthy. If their grades are all over the place, nobody can say which grade is right.
Inter-annotator agreement checks exactly this for the people who label data used to test AI systems. It also subtracts the matches that would happen by pure luck. High agreement means the labels are solid; low agreement means the questions or instructions are unclear, or that the task itself is a matter of opinion.
Seen in the real world
A company building a customer service chatbot wants to compare two model versions using human ratings. Three trained reviewers rate 400 answers on a five-point helpfulness scale, with every answer rated by all three. The first pilot yields a Krippendorff’s alpha of 0.48, too low to support conclusions about which model is better.
The team reviews the disagreements and finds that reviewers treat polite but incomplete answers very differently. They add a rule that completeness outweighs tone, include ten annotated borderline examples and hold one calibration meeting. The second round on a fresh sample reaches an alpha of 0.71. The team then validates an automated judge against the adjudicated labels and accepts it only after its agreement with humans matches the human-human level.