SWE-bench

On this page 7

SWE-bench is a software engineering benchmark that measures whether a language model or AI agent can resolve real GitHub issues by producing a code patch that makes the repository’s own tests pass. In model evaluation, it is one of the most cited measures of practical coding ability, because each task comes from an actual bug report or feature request in a widely used open-source project rather than from a short, self-contained puzzle.

The original dataset, released in 2023, contains 2,294 task instances drawn from 12 popular Python repositories, including projects for web development, scientific computing and data analysis. The headline metric is the percentage of issues resolved, judged automatically by running tests inside a reproducible environment.

How a SWE-bench task is built and scored

Each SWE-bench task pairs an issue description with a repository snapshot taken just before the human fix was merged. The system under test receives the issue text and the codebase, and it must return a patch in diff format. No access to the original pull request or its tests is given.

Scoring relies on two groups of tests taken from the merged pull request. The table below summarizes the components of a single task instance.

ComponentContentRole in evaluation
Problem statementText of the GitHub issueOnly task description given to the model
Base commitRepository state before the fixStarting point for the patch
Gold patchHuman fix from the pull requestReference solution, hidden from the model
FAIL_TO_PASS testsTests that fail before and pass after the fixProve the issue is resolved
PASS_TO_PASS testsTests that passed before the fixProve nothing else broke
Components of a SWE-bench task instance

A task counts as resolved only when every FAIL_TO_PASS test passes and every PASS_TO_PASS test still passes. This binary criterion makes the benchmark strict: a patch that fixes the bug but breaks an unrelated function scores zero. Evaluation runs in containerized environments, typically Docker images, so that dependency versions match the historical snapshot.

SWE-bench variants and subsets

Several curated subsets and extensions of SWE-bench exist because the full dataset is expensive to run and contains some flawed tasks. Results are only comparable when they refer to the same variant.

VariantSizeDistinguishing feature
SWE-bench (full)2,294 tasksOriginal collection from 12 Python repositories
SWE-bench Lite300 tasksCheaper subset with simpler, self-contained issues
SWE-bench Verified500 tasksHuman-screened for clear issues and fair tests
SWE-bench MultimodalSeveral hundred tasksJavaScript issues that include screenshots or images
SWE-bench MultilingualSeveral hundred tasksIssues in programming languages beyond Python
Main SWE-bench variants

SWE-bench Verified became the most widely reported version after professional developers reviewed samples and removed tasks with ambiguous issue descriptions or tests that demanded specific, unstated details. Its creation showed that a meaningful share of the original tasks were close to impossible to solve from the issue text alone, which had depressed early scores.

What SWE-bench measures and how it compares

SWE-bench measures repository-level problem solving: locating the relevant files, understanding existing abstractions, editing several places consistently and avoiding regressions. That differs sharply from function-level coding benchmarks.

BenchmarkTask scopeContext neededTypical metric
SWE-benchFix a real issue in a full repositoryThousands of filesPercentage resolved
HumanEvalWrite one function from a docstringA single promptpass@k
Competitive programming setsSolve an algorithmic puzzleProblem statement onlyAccepted solutions
Terminal and agent tasksComplete multi-step shell workLive environmentTask success rate
SWE-bench compared with other coding evaluations

Because the benchmark rewards navigation and tool use, scores depend heavily on the agent scaffold around the model: the retrieval method, the editing tools, the number of attempts and whether the agent can run tests. Early systems that only retrieved files and generated a patch resolved only a few percent of tasks, while agentic systems that browse, edit and test iteratively now resolve a large majority of SWE-bench Verified tasks.

Reading and reporting SWE-bench results

A SWE-bench score is meaningful only together with the variant, the scaffold and the attempt budget. Two identical models can differ by many percentage points when one uses a stronger agent loop or selects the best of several candidate patches.

FactorWhy it mattersWhat to look for
Dataset variantDifficulty differs between subsetsFull, Lite or Verified
ScaffoldTools change achievable resultsAgent framework and available actions
Attempts per taskMultiple tries inflate successSingle attempt or best-of-n
Test accessRunning hidden tests leaks the answerConfirmation that gold tests stayed hidden
Cost and stepsHigh scores can need heavy computeTokens or dollars per resolved task
Factors to check before comparing scores

Responsible reports state these settings openly and, where possible, publish trajectories and patches so others can audit them. Leaderboards maintained by the benchmark authors require submissions to follow fixed rules, which makes their entries more comparable than vendor claims in isolation.

Limitations and known weaknesses

The most serious limitation is benchmark contamination. The repositories, issues and fixes are public on GitHub, so models trained on web-scale code may have seen the gold patches. High scores therefore do not prove generalization to private codebases.

  • Narrow repository coverage: a few large Python projects dominate the original set, so results may not transfer to other ecosystems.
  • Weak tests: some test suites accept incorrect patches, while others reject valid alternative fixes.
  • Benchmark saturation: as top scores approach the ceiling, small differences say little about real capability.
  • Missing dimensions: code quality, readability, security and maintainability are not scored.

These weaknesses explain why practitioners treat SWE-bench as one strong signal among several, combining it with newer, uncontaminated tasks and with evaluations on their own internal code.

A non-technical view

Imagine hiring a mechanic and testing them on real cars that came into a garage with real complaints. The mechanic reads the customer’s note, opens the hood and fixes the problem. Afterward, an inspector checks that the original fault is gone and that nothing else in the car was broken during the repair.

SWE-bench does the same for AI coding systems. It hands them genuine bug reports from public software projects and checks the repairs with automatic tests. The share of repairs that pass is the score, and it shows how well an AI handles realistic programming work.

A concrete example

An engineering team at a mid-sized software company wants to decide which AI coding assistant to roll out to about 200 developers. Vendor marketing quotes different numbers, so the team runs SWE-bench Verified itself with an identical agent scaffold, a single attempt per task and a fixed limit of 50 tool calls per issue.

Assistant A resolves 62% of the 500 tasks and Assistant B resolves 55%, but Assistant A uses roughly twice as many tokens per resolved issue. The team then runs both tools on 40 recently closed bugs from its own private repositories, which no model could have seen during training. There, the gap narrows to four percentage points. The team selects Assistant B for daily use because of its lower cost per resolved task, and keeps the internal test set for future comparisons.