LLM evaluation, often shortened to LLM evals or AI evals, is the systematic measurement of how well a large language model performs on defined tasks, using test datasets, scoring methods and human or automated judgment. In model evaluation for artificial intelligence, it answers three practical questions: what the model can do, how reliably it does it, and whether it is safe for a given use.
Unlike classic classifier evaluation, where a single accuracy figure on a held-out set is often meaningful, evaluating a language model is difficult because outputs are open-ended text, many answers can be correct, and the same model behaves differently depending on the prompt. Good LLM evaluation therefore combines several methods and treats every single score with caution.
Levels of LLM evaluation
LLM evaluation operates at two distinct levels: evaluating the model itself and evaluating a complete application built on it. Model developers focus on general capability, while product teams care about task-specific quality inside their own prompts, retrieval pipeline and tools.
The table below separates the main levels and what each one measures.
| Level | Typical question | Typical method |
|---|---|---|
| Capability evaluation | How strong is the base or chat model | Public benchmarks such as MMLU or GPQA |
| Application evaluation | Does the product answer users correctly | Custom evaluation dataset with graders |
| Safety evaluation | Does the model refuse or leak harmful content | Red teaming and adversarial prompt sets |
| Online evaluation | Do real users prefer the new version | A/B tests, win rate and feedback signals |
| Regression evaluation | Did a change break earlier behavior | Fixed test suite run on every release |
Mature teams run these levels as a pipeline: offline evals gate a release, and online measurements confirm that offline gains translate into real user value.
How LLM outputs are scored
The scoring method determines what an evaluation actually measures, so it deserves as much design effort as the test items. Methods range from strict string comparison to model-based judgment, and each trades cost, scalability and validity differently.
| Method | Strength | Main weakness |
|---|---|---|
| Exact match | Cheap and fully reproducible | Penalizes correct answers phrased differently |
| Programmatic checks | Objective for code, math and formats | Only works where correctness is verifiable |
| Overlap metrics | Fast reference comparison with ROUGE or BLEU | Weak correlation with meaning and quality |
| LLM-as-a-judge | Scales nuanced grading to thousands of items | Position, length and self-preference bias |
| Human evaluation | Highest validity for subjective quality | Slow, costly, needs inter-annotator agreement |
For code, functional tests and the pass@k metric used by HumanEval and SWE-bench are preferred because they check behavior rather than surface text. For open-ended answers, a written rubric with explicit criteria makes both human and model graders far more consistent.
When a model grades another model, the judge should be validated against human labels on a sample. Without that calibration step, an apparently precise score may simply reflect the preferences of the judge model.
Benchmarks and their limits
Public benchmarks provide a shared yardstick for comparing models, but they measure narrow skills under fixed conditions. Results depend heavily on the prompt format, number of examples, decoding settings and answer extraction, so two reports of the same benchmark are often not comparable.
- Benchmark contamination: test items leak into training data, inflating scores without real capability gains.
- Benchmark saturation: top models approach the ceiling, so remaining differences are mostly noise; this drove the creation of harder sets such as Humanity’s Last Exam and FrontierMath.
- Construct validity: a high multiple-choice score does not prove the model can apply the knowledge in free-form work.
- Statistical noise: on a set of a few hundred items, differences of one or two points often fall within the confidence interval.
For these reasons, practitioners treat leaderboards such as LMArena as one signal among several and rely on private, task-specific evaluation sets for decisions that matter.
Building an application evaluation set
An application evaluation set is a curated collection of realistic inputs with expected outputs or grading criteria, drawn from the actual use case. It is the most valuable asset in LLM evaluation because it reflects the distribution of real requests, including rare and difficult ones.
The following decision table shows how to match the evaluation approach to the type of task.
| Task type | Recommended approach | Reason |
|---|---|---|
| Structured extraction | Exact match on fields plus schema validation | Correct values are unambiguous |
| Retrieval-augmented answers | Groundedness and correctness rubrics | Hallucination rate matters most |
| Code generation | Unit tests in a sandbox | Execution proves functional correctness |
| Open-ended writing | Pairwise comparison with a calibrated judge | Relative preference is easier to judge |
| Agent workflows | End-to-end task success and step traces | Errors compound across steps |
A useful set starts small, often 50 to 200 carefully chosen cases, and grows through error analysis: every production failure that is understood becomes a new test item. An evaluation harness then runs the full set automatically on each prompt, model or retrieval change.
Common mistakes in LLM evaluation
The most frequent error is trusting a single aggregate number. Averages hide failures on important subgroups, and a model that improves overall may regress badly on the cases a business cares about most.
| Pitfall | Consequence | Remedy |
|---|---|---|
| Evaluating with one prompt | Scores reflect prompt luck | Test several prompt variants |
| Ignoring variance | Random changes look like progress | Repeat runs and report intervals |
| Unvalidated judge model | Systematic grading bias | Compare with human labels |
| Tuning on the test set | Overfitting to the evaluation | Keep a separate held-out set |
| Only offline metrics | Gains that users never notice | Confirm with online measurement |
Evaluation should be treated as continuous engineering work, versioned alongside prompts and models, rather than a one-time report produced before launch.
Put simply
LLM evaluation is testing an AI chatbot before trusting it. Like a driving test, it checks specific skills under known conditions and gives a score.
The difficulty is that language has many correct answers, and a model can pass a famous exam while still failing the questions real people ask. That is why careful teams write their own tests from real cases, check them with people and automated graders, and rerun them every time something changes.
In practice
An insurance company builds an assistant that answers policy questions from internal documents. Before switching to a newer model, the team runs its evaluation set of 300 real customer questions, each with a reference answer and a rubric covering correctness, citation of the right policy clause and appropriate refusal when information is missing.
A judge model, previously checked against 100 human-graded answers with high agreement, scores both versions. The new model improves overall correctness from 82 to 88 percent, but error analysis reveals that it invents coverage limits on 4 percent of questions about rare policy types, a failure the old model never made. The team adds a retrieval fix and stricter grounding instructions, reruns the suite, and ships only after the hallucination rate returns to zero on that subgroup.