HumanEval

On this page 7

HumanEval is a code generation benchmark of 164 hand-written Python programming problems, each graded by executing the generated code against hidden unit tests. In model evaluation it is the classic test of functional correctness: a model reads a function signature and a docstring, writes the function body, and earns credit only if every test passes.

Released in 2021 together with the paper that introduced the Codex models, HumanEval made the pass@k metric the standard way to report coding ability in artificial intelligence. Its small size and clear grading made it widely used, and the same properties now make it nearly saturated and vulnerable to contamination, so it is best read alongside harder and newer benchmarks.

How HumanEval tests code generation

HumanEval tests a model by giving it a prompt made of a function header and a docstring, sometimes with input-output examples, and asking it to complete the function. The completion is concatenated with the prompt, executed in a sandboxed Python environment, and checked against a test function that calls it with fixed inputs.

The problems were written by hand rather than scraped from public repositories, which was meant to reduce the chance that models had memorized them. They cover string manipulation, simple algorithms, arithmetic and list processing, roughly at the level of an easy technical interview question. Each problem has on average about 7.7 unit tests, which is enough to catch many errors but not all of them.

The table below describes the components of a single HumanEval task.

ComponentContentRole in grading
Task IDIdentifier such as HumanEval/0Tracks results per problem
PromptImports, signature and docstringThe only input the model sees
Canonical solutionReference implementationValidates the tests, not shown to the model
Test functionAssertions on fixed inputsDecides pass or fail
Entry pointName of the function to callLinks the completion to the tests
Components of one HumanEval task

The pass@k metric behind HumanEval scores

HumanEval results are reported as pass@k, the probability that at least one of k sampled completions passes all tests. Pass@1 approximates the chance that a single answer is correct, while pass@10 and pass@100 show how often the correct solution exists somewhere in the model’s samples.

Computing pass@k naively from exactly k samples gives a high-variance estimate. The benchmark therefore uses an unbiased estimator: generate n samples per problem with n at least k, count the c correct ones, and compute 1 - C(n-c, k) / C(n, k), averaged over all 164 problems. Typical settings use n = 200 for pass@100 and a low sampling temperature, around 0.2, for pass@1.

The following table shows how the main reporting settings differ.

SettingSamplingWhat it reveals
Pass@1 greedyOne deterministic completionEveryday single-answer accuracy
Pass@1 sampledMany samples, low temperatureSmoother estimate of single-answer accuracy
Pass@10Higher temperature, about 0.6-0.8Benefit of a few retries
Pass@100High temperature, 200 samplesCoverage of the solution space
Common HumanEval reporting settings

Scores are only comparable when the sampling settings, prompt format and post-processing match. A model reported with greedy decoding and one reported with the best of several prompt templates can differ by several points without any real difference in ability.

HumanEval compared with other coding benchmarks

HumanEval measures short, self-contained function synthesis, which is only one part of programming skill. Later benchmarks extended it with stronger tests, more languages or realistic software engineering tasks, and the choice among them depends on what the evaluation needs to show.

BenchmarkScopeGradingMain difference
HumanEval164 Python functionsHidden unit testsOriginal short-function baseline
HumanEval+Same 164 problemsAbout 80 times more testsCatches solutions that pass weak tests
MultiPL-EHumanEval translated to many languagesLanguage-specific testsMeasures cross-language ability
MBPPAbout 1,000 basic Python tasksThree tests per taskLarger, simpler, crowd-written
SWE-benchReal GitHub issuesRepository test suitesMulti-file, realistic engineering work
HumanEval and related code benchmarks

Moving from HumanEval to HumanEval+ often lowers pass@1 by several percentage points, which shows how many accepted solutions were in fact incorrect on edge cases. SWE-bench, by contrast, measures navigation of large codebases and is far from saturated, so a high HumanEval score says little about performance on it.

Interpreting HumanEval results responsibly

A HumanEval score is meaningful only together with its evaluation conditions and the model’s era. The original 12-billion-parameter Codex model reached about 29 percent pass@1, while current frontier models report above 90 percent, which leaves little headroom to separate strong systems.

The decision table below lists typical evaluation goals and whether HumanEval fits them.

Evaluation goalIs HumanEval suitableBetter complement
Quick regression check during trainingYes, fast and cheapHumanEval+ for stricter tests
Comparing small or older modelsYes, scores still spread outMBPP for more problems
Ranking frontier modelsWeakly, scores are saturatedSWE-bench or newer contests
Non-Python coding abilityNo, Python onlyMultiPL-E
Agentic software engineeringNo, single function scopeRepository-level benchmarks
When HumanEval is a suitable measure

Because the problems have been public since 2021, benchmark contamination is a serious concern: solutions appear in many repositories and tutorials that may enter training data. Teams that rely on HumanEval increasingly check for verbatim overlap with training corpora or use rewritten variants to confirm that a high score reflects skill rather than memory.

Known weaknesses of HumanEval

The main weaknesses of HumanEval stem from its small size and limited test coverage. With 164 problems, one additional solved task moves the score by about 0.6 percentage points, so differences of one or two points are often within noise.

  • Weak tests: some incorrect solutions pass because edge cases are not tested.
  • Saturation: top models solve nearly every problem, so the benchmark no longer discriminates.
  • Contamination: public solutions may have been seen during pretraining.
  • Narrow scope: no debugging, refactoring, documentation reading or multi-file reasoning.
  • Execution risk: running untrusted generated code requires strict sandboxing.

Explained without jargon

HumanEval is a coding quiz for AI models. It contains 164 small programming exercises, each describing a function in plain words. The model writes the code, and a computer runs it through a set of checks.

If every check passes, the exercise counts as solved. The final score is the share of exercises solved. It is a bit like a driving test on a quiet road: passing proves basic skill, but it does not prove the driver can handle a busy city.

A typical use case

A research team fine-tunes a 7-billion-parameter open model on a mix of Python code and explanations. After every training checkpoint they run HumanEval and HumanEval+ in an isolated container with no network access, sampling 20 completions per problem at temperature 0.2 to estimate pass@1.

The base model scores 31 percent pass@1 on HumanEval and the fine-tuned checkpoint reaches 48 percent, while HumanEval+ rises from 27 to 41 percent, which confirms that the gain is not an artifact of weak tests. Before announcing the result, the team searches its training data for HumanEval docstrings and removes eleven near-duplicates, then reruns the evaluation and sees the score fall by only one point, so the improvement holds up.