HumanEval is a code generation benchmark of 164 hand-written Python programming problems, each graded by executing the generated code against hidden unit tests. In model evaluation it is the classic test of functional correctness: a model reads a function signature and a docstring, writes the function body, and earns credit only if every test passes.
Released in 2021 together with the paper that introduced the Codex models, HumanEval made the pass@k metric the standard way to report coding ability in artificial intelligence. Its small size and clear grading made it widely used, and the same properties now make it nearly saturated and vulnerable to contamination, so it is best read alongside harder and newer benchmarks.
How HumanEval tests code generation
HumanEval tests a model by giving it a prompt made of a function header and a docstring, sometimes with input-output examples, and asking it to complete the function. The completion is concatenated with the prompt, executed in a sandboxed Python environment, and checked against a test function that calls it with fixed inputs.
The problems were written by hand rather than scraped from public repositories, which was meant to reduce the chance that models had memorized them. They cover string manipulation, simple algorithms, arithmetic and list processing, roughly at the level of an easy technical interview question. Each problem has on average about 7.7 unit tests, which is enough to catch many errors but not all of them.
The table below describes the components of a single HumanEval task.
| Component | Content | Role in grading |
|---|---|---|
| Task ID | Identifier such as HumanEval/0 | Tracks results per problem |
| Prompt | Imports, signature and docstring | The only input the model sees |
| Canonical solution | Reference implementation | Validates the tests, not shown to the model |
| Test function | Assertions on fixed inputs | Decides pass or fail |
| Entry point | Name of the function to call | Links the completion to the tests |
The pass@k metric behind HumanEval scores
HumanEval results are reported as pass@k, the probability that at least one of k sampled completions passes all tests. Pass@1 approximates the chance that a single answer is correct, while pass@10 and pass@100 show how often the correct solution exists somewhere in the model’s samples.
Computing pass@k naively from exactly k samples gives a high-variance estimate. The benchmark therefore uses an unbiased estimator: generate n samples per problem with n at least k, count the c correct ones, and compute 1 - C(n-c, k) / C(n, k), averaged over all 164 problems. Typical settings use n = 200 for pass@100 and a low sampling temperature, around 0.2, for pass@1.
The following table shows how the main reporting settings differ.
| Setting | Sampling | What it reveals |
|---|---|---|
| Pass@1 greedy | One deterministic completion | Everyday single-answer accuracy |
| Pass@1 sampled | Many samples, low temperature | Smoother estimate of single-answer accuracy |
| Pass@10 | Higher temperature, about 0.6-0.8 | Benefit of a few retries |
| Pass@100 | High temperature, 200 samples | Coverage of the solution space |
Scores are only comparable when the sampling settings, prompt format and post-processing match. A model reported with greedy decoding and one reported with the best of several prompt templates can differ by several points without any real difference in ability.
HumanEval compared with other coding benchmarks
HumanEval measures short, self-contained function synthesis, which is only one part of programming skill. Later benchmarks extended it with stronger tests, more languages or realistic software engineering tasks, and the choice among them depends on what the evaluation needs to show.
| Benchmark | Scope | Grading | Main difference |
|---|---|---|---|
| HumanEval | 164 Python functions | Hidden unit tests | Original short-function baseline |
| HumanEval+ | Same 164 problems | About 80 times more tests | Catches solutions that pass weak tests |
| MultiPL-E | HumanEval translated to many languages | Language-specific tests | Measures cross-language ability |
| MBPP | About 1,000 basic Python tasks | Three tests per task | Larger, simpler, crowd-written |
| SWE-bench | Real GitHub issues | Repository test suites | Multi-file, realistic engineering work |
Moving from HumanEval to HumanEval+ often lowers pass@1 by several percentage points, which shows how many accepted solutions were in fact incorrect on edge cases. SWE-bench, by contrast, measures navigation of large codebases and is far from saturated, so a high HumanEval score says little about performance on it.
Interpreting HumanEval results responsibly
A HumanEval score is meaningful only together with its evaluation conditions and the model’s era. The original 12-billion-parameter Codex model reached about 29 percent pass@1, while current frontier models report above 90 percent, which leaves little headroom to separate strong systems.
The decision table below lists typical evaluation goals and whether HumanEval fits them.
| Evaluation goal | Is HumanEval suitable | Better complement |
|---|---|---|
| Quick regression check during training | Yes, fast and cheap | HumanEval+ for stricter tests |
| Comparing small or older models | Yes, scores still spread out | MBPP for more problems |
| Ranking frontier models | Weakly, scores are saturated | SWE-bench or newer contests |
| Non-Python coding ability | No, Python only | MultiPL-E |
| Agentic software engineering | No, single function scope | Repository-level benchmarks |
Because the problems have been public since 2021, benchmark contamination is a serious concern: solutions appear in many repositories and tutorials that may enter training data. Teams that rely on HumanEval increasingly check for verbatim overlap with training corpora or use rewritten variants to confirm that a high score reflects skill rather than memory.
Known weaknesses of HumanEval
The main weaknesses of HumanEval stem from its small size and limited test coverage. With 164 problems, one additional solved task moves the score by about 0.6 percentage points, so differences of one or two points are often within noise.
- Weak tests: some incorrect solutions pass because edge cases are not tested.
- Saturation: top models solve nearly every problem, so the benchmark no longer discriminates.
- Contamination: public solutions may have been seen during pretraining.
- Narrow scope: no debugging, refactoring, documentation reading or multi-file reasoning.
- Execution risk: running untrusted generated code requires strict sandboxing.
Explained without jargon
HumanEval is a coding quiz for AI models. It contains 164 small programming exercises, each describing a function in plain words. The model writes the code, and a computer runs it through a set of checks.
If every check passes, the exercise counts as solved. The final score is the share of exercises solved. It is a bit like a driving test on a quiet road: passing proves basic skill, but it does not prove the driver can handle a busy city.
A typical use case
A research team fine-tunes a 7-billion-parameter open model on a mix of Python code and explanations. After every training checkpoint they run HumanEval and HumanEval+ in an isolated container with no network access, sampling 20 completions per problem at temperature 0.2 to estimate pass@1.
The base model scores 31 percent pass@1 on HumanEval and the fine-tuned checkpoint reaches 48 percent, while HumanEval+ rises from 27 to 41 percent, which confirms that the gain is not an artifact of weak tests. Before announcing the result, the team searches its training data for HumanEval docstrings and removes eleven near-duplicates, then reruns the evaluation and sees the score fall by only one point, so the improvement holds up.