pass@k

Also known as: pass at k

On this page 7

pass@k (read as “pass at k”) is a model evaluation metric that estimates the probability that at least one of k sampled outputs is correct, where correctness is judged by an automatic check such as unit tests. It became the standard way to score code generation in artificial intelligence after its use with the HumanEval benchmark, and it is now applied to mathematical reasoning, agent tasks and any setting with a verifiable answer.

The metric captures a practical question: if a developer or an automated system can draw several attempts and keep the one that works, how often will the model succeed? It therefore measures coverage of the solution space rather than the quality of a single answer, which makes it useful but also easy to misread.

How pass@k is calculated

The naive approach generates exactly k samples per problem and records whether any passes. That estimate has high variance, so the accepted method generates n samples per problem, with n at least k, counts the number c that pass, and computes an unbiased estimator: pass@k equals 1 minus the ratio of the binomial coefficients C(n minus c, k) and C(n, k). The benchmark score is the average of this value over all problems.

Intuitively, the formula asks how likely it is that a random subset of k samples drawn from the n generated ones contains no correct solution, and subtracts that from one. Implementations compute it as a numerically stable product rather than with large factorials, because binomial coefficients overflow quickly.

The table below lists the quantities involved and their typical values.

SymbolMeaningTypical values
kAttempts the user is allowed1, 5, 10, 100
nSamples generated per problem10 to 200, at least k
cSamples that pass all tests0 to n
TSampling temperatureNear 0 for pass@1, 0.6 to 0.8 for large k
Quantities in the pass@k estimator

Interpreting different values of k

pass@1 is the closest proxy for single-shot reliability: it equals the expected fraction of problems solved when only one answer is used. Higher values of k describe how well the model can solve a task when a selection mechanism exists, such as running tests and keeping the first passing program.

Because pass@k is monotonically non-decreasing in k, scores rise as k grows, and the gap between pass@1 and pass@100 indicates how much latent capability a better ranking or verification step could unlock. A model with a large gap is diverse but unreliable; a small gap suggests consistent but less exploratory behavior.

SettingQuestion answeredTypical use
pass@1How often is the first answer rightLeaderboards, production assistants
pass@10Can a short list contain a solutionInteractive tools that offer alternatives
pass@100Does the model know any path to the answerResearch on sampling and search
pass@k vs pass^kIs the model consistently right in all k trialsAgent reliability testing
What each common setting of k tells an evaluator

pass@k belongs to a family of execution-based metrics that judge outputs by behavior rather than by text similarity. This distinguishes it sharply from BLEU score, ROUGE or BERTScore, which reward surface overlap and correlate poorly with whether code actually runs.

MetricWhat it checksMain weakness
pass@kAny of k samples passes testsIgnores how the right sample is found
Exact matchOutput equals a reference stringRejects valid alternative solutions
BLEU scoreN-gram overlap with a referenceWeakly linked to functional correctness
Top-k accuracyTrue label among k ranked predictionsNeeds a fixed label set
Majority voting accuracyMost frequent answer is correctRequires comparable final answers
pass@k compared with neighboring evaluation measures

pass@k resembles top-k accuracy in spirit, but top-k ranks predictions from a single forward pass, while pass@k draws independent stochastic samples. Reasoning benchmarks often report majority voting (also called cons@k) alongside pass@k, because voting reflects what a system can achieve without an oracle verifier.

Running a reliable pass@k evaluation

A trustworthy result depends on the evaluation harness as much as on the model. Generated code must run in an isolated sandbox with time and memory limits, the tests must be strong enough to reject wrong programs, and the sampling settings must be reported.

  • Generate n well above k, for example n equal to 200 when reporting pass@100, to keep variance low.
  • Report the sampling temperature and top-p for each k, since optimal temperature rises with k.
  • Use the unbiased estimator, not the fraction of runs in which one of exactly k samples passed.
  • Check for benchmark contamination, because memorized solutions inflate every value of k.

The table below lists frequent failure patterns and their fixes.

SymptomLikely causeFix
Scores too highWeak or few unit testsAdd hidden tests and edge cases
Large run-to-run varianceToo few samples per problemIncrease n and report confidence intervals
Unfair comparisonDifferent temperatures or promptsFix decoding settings across models
Timeouts counted as passesFaulty harness logicTreat any error or timeout as failure
Common problems in pass@k measurement

Limitations and common misreadings

The most serious misreading treats pass@k at large k as user-facing accuracy. Without a reliable way to pick the correct sample, a user experiences something close to pass@1, so high pass@100 numbers can overstate real usefulness.

pass@k also says nothing about code quality, security, readability or efficiency; a program that passes tests may still be slow or vulnerable. Finally, it rewards diversity, so a model tuned for high pass@1 through reinforcement learning may show a lower pass@k at large k than its base model, a pattern that research on reasoning models has repeatedly observed and debated.

The idea in brief

pass@k answers a simple question: if an AI gets k tries at a programming puzzle, how likely is at least one try to work? It is like giving a student several attempts at a lock combination and counting whether any attempt opens it.

pass@1 tells you how often the first answer is right, which matters most in everyday use. Larger values of k show what the model could do if someone or something reliably picked the best attempt.

A practical scenario

An engineering team compares two code models before choosing one for an internal assistant. They run both on a set of 164 Python problems with hidden unit tests, generating 200 samples per problem in a sandbox with a ten-second limit per test.

Model A scores pass@1 of 62 percent and pass@10 of 81 percent. Model B scores pass@1 of 55 percent but pass@10 of 88 percent. Because the assistant shows developers only one suggestion, the team selects model A for interactive use.

For a separate batch job that can run tests automatically and keep the first passing program, the team uses model B with ten attempts per task, where its higher pass@10 translates directly into more solved tickets.