LLM evaluation

Also known as: LLM evals · AI evals

On this page 7

LLM evaluation, often shortened to LLM evals or AI evals, is the systematic measurement of how well a large language model performs on defined tasks, using test datasets, scoring methods and human or automated judgment. In model evaluation for artificial intelligence, it answers three practical questions: what the model can do, how reliably it does it, and whether it is safe for a given use.

Unlike classic classifier evaluation, where a single accuracy figure on a held-out set is often meaningful, evaluating a language model is difficult because outputs are open-ended text, many answers can be correct, and the same model behaves differently depending on the prompt. Good LLM evaluation therefore combines several methods and treats every single score with caution.

Levels of LLM evaluation

LLM evaluation operates at two distinct levels: evaluating the model itself and evaluating a complete application built on it. Model developers focus on general capability, while product teams care about task-specific quality inside their own prompts, retrieval pipeline and tools.

The table below separates the main levels and what each one measures.

LevelTypical questionTypical method
Capability evaluationHow strong is the base or chat modelPublic benchmarks such as MMLU or GPQA
Application evaluationDoes the product answer users correctlyCustom evaluation dataset with graders
Safety evaluationDoes the model refuse or leak harmful contentRed teaming and adversarial prompt sets
Online evaluationDo real users prefer the new versionA/B tests, win rate and feedback signals
Regression evaluationDid a change break earlier behaviorFixed test suite run on every release
Levels at which language models are evaluated

Mature teams run these levels as a pipeline: offline evals gate a release, and online measurements confirm that offline gains translate into real user value.

How LLM outputs are scored

The scoring method determines what an evaluation actually measures, so it deserves as much design effort as the test items. Methods range from strict string comparison to model-based judgment, and each trades cost, scalability and validity differently.

MethodStrengthMain weakness
Exact matchCheap and fully reproduciblePenalizes correct answers phrased differently
Programmatic checksObjective for code, math and formatsOnly works where correctness is verifiable
Overlap metricsFast reference comparison with ROUGE or BLEUWeak correlation with meaning and quality
LLM-as-a-judgeScales nuanced grading to thousands of itemsPosition, length and self-preference bias
Human evaluationHighest validity for subjective qualitySlow, costly, needs inter-annotator agreement
Scoring methods for language model outputs

For code, functional tests and the pass@k metric used by HumanEval and SWE-bench are preferred because they check behavior rather than surface text. For open-ended answers, a written rubric with explicit criteria makes both human and model graders far more consistent.

When a model grades another model, the judge should be validated against human labels on a sample. Without that calibration step, an apparently precise score may simply reflect the preferences of the judge model.

Benchmarks and their limits

Public benchmarks provide a shared yardstick for comparing models, but they measure narrow skills under fixed conditions. Results depend heavily on the prompt format, number of examples, decoding settings and answer extraction, so two reports of the same benchmark are often not comparable.

  • Benchmark contamination: test items leak into training data, inflating scores without real capability gains.
  • Benchmark saturation: top models approach the ceiling, so remaining differences are mostly noise; this drove the creation of harder sets such as Humanity’s Last Exam and FrontierMath.
  • Construct validity: a high multiple-choice score does not prove the model can apply the knowledge in free-form work.
  • Statistical noise: on a set of a few hundred items, differences of one or two points often fall within the confidence interval.

For these reasons, practitioners treat leaderboards such as LMArena as one signal among several and rely on private, task-specific evaluation sets for decisions that matter.

Building an application evaluation set

An application evaluation set is a curated collection of realistic inputs with expected outputs or grading criteria, drawn from the actual use case. It is the most valuable asset in LLM evaluation because it reflects the distribution of real requests, including rare and difficult ones.

The following decision table shows how to match the evaluation approach to the type of task.

Task typeRecommended approachReason
Structured extractionExact match on fields plus schema validationCorrect values are unambiguous
Retrieval-augmented answersGroundedness and correctness rubricsHallucination rate matters most
Code generationUnit tests in a sandboxExecution proves functional correctness
Open-ended writingPairwise comparison with a calibrated judgeRelative preference is easier to judge
Agent workflowsEnd-to-end task success and step tracesErrors compound across steps
Choosing an evaluation approach by task

A useful set starts small, often 50 to 200 carefully chosen cases, and grows through error analysis: every production failure that is understood becomes a new test item. An evaluation harness then runs the full set automatically on each prompt, model or retrieval change.

Common mistakes in LLM evaluation

The most frequent error is trusting a single aggregate number. Averages hide failures on important subgroups, and a model that improves overall may regress badly on the cases a business cares about most.

PitfallConsequenceRemedy
Evaluating with one promptScores reflect prompt luckTest several prompt variants
Ignoring varianceRandom changes look like progressRepeat runs and report intervals
Unvalidated judge modelSystematic grading biasCompare with human labels
Tuning on the test setOverfitting to the evaluationKeep a separate held-out set
Only offline metricsGains that users never noticeConfirm with online measurement
Evaluation pitfalls and remedies

Evaluation should be treated as continuous engineering work, versioned alongside prompts and models, rather than a one-time report produced before launch.

Put simply

LLM evaluation is testing an AI chatbot before trusting it. Like a driving test, it checks specific skills under known conditions and gives a score.

The difficulty is that language has many correct answers, and a model can pass a famous exam while still failing the questions real people ask. That is why careful teams write their own tests from real cases, check them with people and automated graders, and rerun them every time something changes.

In practice

An insurance company builds an assistant that answers policy questions from internal documents. Before switching to a newer model, the team runs its evaluation set of 300 real customer questions, each with a reference answer and a rubric covering correctness, citation of the right policy clause and appropriate refusal when information is missing.

A judge model, previously checked against 100 human-graded answers with high agreement, scores both versions. The new model improves overall correctness from 82 to 88 percent, but error analysis reveals that it invents coverage limits on 4 percent of questions about rare policy types, a failure the old model never made. The team adds a retrieval fix and stricter grounding instructions, reruns the suite, and ships only after the hallucination rate returns to zero on that subgroup.