pass@k (read as “pass at k”) is a model evaluation metric that estimates the probability that at least one of k sampled outputs is correct, where correctness is judged by an automatic check such as unit tests. It became the standard way to score code generation in artificial intelligence after its use with the HumanEval benchmark, and it is now applied to mathematical reasoning, agent tasks and any setting with a verifiable answer.
The metric captures a practical question: if a developer or an automated system can draw several attempts and keep the one that works, how often will the model succeed? It therefore measures coverage of the solution space rather than the quality of a single answer, which makes it useful but also easy to misread.
How pass@k is calculated
The naive approach generates exactly k samples per problem and records whether any passes. That estimate has high variance, so the accepted method generates n samples per problem, with n at least k, counts the number c that pass, and computes an unbiased estimator: pass@k equals 1 minus the ratio of the binomial coefficients C(n minus c, k) and C(n, k). The benchmark score is the average of this value over all problems.
Intuitively, the formula asks how likely it is that a random subset of k samples drawn from the n generated ones contains no correct solution, and subtracts that from one. Implementations compute it as a numerically stable product rather than with large factorials, because binomial coefficients overflow quickly.
The table below lists the quantities involved and their typical values.
| Symbol | Meaning | Typical values |
|---|---|---|
| k | Attempts the user is allowed | 1, 5, 10, 100 |
| n | Samples generated per problem | 10 to 200, at least k |
| c | Samples that pass all tests | 0 to n |
| T | Sampling temperature | Near 0 for pass@1, 0.6 to 0.8 for large k |
Interpreting different values of k
pass@1 is the closest proxy for single-shot reliability: it equals the expected fraction of problems solved when only one answer is used. Higher values of k describe how well the model can solve a task when a selection mechanism exists, such as running tests and keeping the first passing program.
Because pass@k is monotonically non-decreasing in k, scores rise as k grows, and the gap between pass@1 and pass@100 indicates how much latent capability a better ranking or verification step could unlock. A model with a large gap is diverse but unreliable; a small gap suggests consistent but less exploratory behavior.
| Setting | Question answered | Typical use |
|---|---|---|
| pass@1 | How often is the first answer right | Leaderboards, production assistants |
| pass@10 | Can a short list contain a solution | Interactive tools that offer alternatives |
| pass@100 | Does the model know any path to the answer | Research on sampling and search |
| pass@k vs pass^k | Is the model consistently right in all k trials | Agent reliability testing |
How pass@k compares with related metrics
pass@k belongs to a family of execution-based metrics that judge outputs by behavior rather than by text similarity. This distinguishes it sharply from BLEU score, ROUGE or BERTScore, which reward surface overlap and correlate poorly with whether code actually runs.
| Metric | What it checks | Main weakness |
|---|---|---|
| pass@k | Any of k samples passes tests | Ignores how the right sample is found |
| Exact match | Output equals a reference string | Rejects valid alternative solutions |
| BLEU score | N-gram overlap with a reference | Weakly linked to functional correctness |
| Top-k accuracy | True label among k ranked predictions | Needs a fixed label set |
| Majority voting accuracy | Most frequent answer is correct | Requires comparable final answers |
pass@k resembles top-k accuracy in spirit, but top-k ranks predictions from a single forward pass, while pass@k draws independent stochastic samples. Reasoning benchmarks often report majority voting (also called cons@k) alongside pass@k, because voting reflects what a system can achieve without an oracle verifier.
Running a reliable pass@k evaluation
A trustworthy result depends on the evaluation harness as much as on the model. Generated code must run in an isolated sandbox with time and memory limits, the tests must be strong enough to reject wrong programs, and the sampling settings must be reported.
- Generate n well above k, for example n equal to 200 when reporting pass@100, to keep variance low.
- Report the sampling temperature and top-p for each k, since optimal temperature rises with k.
- Use the unbiased estimator, not the fraction of runs in which one of exactly k samples passed.
- Check for benchmark contamination, because memorized solutions inflate every value of k.
The table below lists frequent failure patterns and their fixes.
| Symptom | Likely cause | Fix |
|---|---|---|
| Scores too high | Weak or few unit tests | Add hidden tests and edge cases |
| Large run-to-run variance | Too few samples per problem | Increase n and report confidence intervals |
| Unfair comparison | Different temperatures or prompts | Fix decoding settings across models |
| Timeouts counted as passes | Faulty harness logic | Treat any error or timeout as failure |
Limitations and common misreadings
The most serious misreading treats pass@k at large k as user-facing accuracy. Without a reliable way to pick the correct sample, a user experiences something close to pass@1, so high pass@100 numbers can overstate real usefulness.
pass@k also says nothing about code quality, security, readability or efficiency; a program that passes tests may still be slow or vulnerable. Finally, it rewards diversity, so a model tuned for high pass@1 through reinforcement learning may show a lower pass@k at large k than its base model, a pattern that research on reasoning models has repeatedly observed and debated.
The idea in brief
pass@k answers a simple question: if an AI gets k tries at a programming puzzle, how likely is at least one try to work? It is like giving a student several attempts at a lock combination and counting whether any attempt opens it.
pass@1 tells you how often the first answer is right, which matters most in everyday use. Larger values of k show what the model could do if someone or something reliably picked the best attempt.
A practical scenario
An engineering team compares two code models before choosing one for an internal assistant. They run both on a set of 164 Python problems with hidden unit tests, generating 200 samples per problem in a sandbox with a ten-second limit per test.
Model A scores pass@1 of 62 percent and pass@10 of 81 percent. Model B scores pass@1 of 55 percent but pass@10 of 88 percent. Because the assistant shows developers only one suggestion, the team selects model A for interactive use.
For a separate batch job that can run tests automatically and keep the first passing program, the team uses model B with ten attempts per task, where its higher pass@10 translates directly into more solved tickets.