SWE-bench is a software engineering benchmark that measures whether a language model or AI agent can resolve real GitHub issues by producing a code patch that makes the repository’s own tests pass. In model evaluation, it is one of the most cited measures of practical coding ability, because each task comes from an actual bug report or feature request in a widely used open-source project rather than from a short, self-contained puzzle.
The original dataset, released in 2023, contains 2,294 task instances drawn from 12 popular Python repositories, including projects for web development, scientific computing and data analysis. The headline metric is the percentage of issues resolved, judged automatically by running tests inside a reproducible environment.
How a SWE-bench task is built and scored
Each SWE-bench task pairs an issue description with a repository snapshot taken just before the human fix was merged. The system under test receives the issue text and the codebase, and it must return a patch in diff format. No access to the original pull request or its tests is given.
Scoring relies on two groups of tests taken from the merged pull request. The table below summarizes the components of a single task instance.
| Component | Content | Role in evaluation |
|---|---|---|
| Problem statement | Text of the GitHub issue | Only task description given to the model |
| Base commit | Repository state before the fix | Starting point for the patch |
| Gold patch | Human fix from the pull request | Reference solution, hidden from the model |
| FAIL_TO_PASS tests | Tests that fail before and pass after the fix | Prove the issue is resolved |
| PASS_TO_PASS tests | Tests that passed before the fix | Prove nothing else broke |
A task counts as resolved only when every FAIL_TO_PASS test passes and every PASS_TO_PASS test still passes. This binary criterion makes the benchmark strict: a patch that fixes the bug but breaks an unrelated function scores zero. Evaluation runs in containerized environments, typically Docker images, so that dependency versions match the historical snapshot.
SWE-bench variants and subsets
Several curated subsets and extensions of SWE-bench exist because the full dataset is expensive to run and contains some flawed tasks. Results are only comparable when they refer to the same variant.
| Variant | Size | Distinguishing feature |
|---|---|---|
| SWE-bench (full) | 2,294 tasks | Original collection from 12 Python repositories |
| SWE-bench Lite | 300 tasks | Cheaper subset with simpler, self-contained issues |
| SWE-bench Verified | 500 tasks | Human-screened for clear issues and fair tests |
| SWE-bench Multimodal | Several hundred tasks | JavaScript issues that include screenshots or images |
| SWE-bench Multilingual | Several hundred tasks | Issues in programming languages beyond Python |
SWE-bench Verified became the most widely reported version after professional developers reviewed samples and removed tasks with ambiguous issue descriptions or tests that demanded specific, unstated details. Its creation showed that a meaningful share of the original tasks were close to impossible to solve from the issue text alone, which had depressed early scores.
What SWE-bench measures and how it compares
SWE-bench measures repository-level problem solving: locating the relevant files, understanding existing abstractions, editing several places consistently and avoiding regressions. That differs sharply from function-level coding benchmarks.
| Benchmark | Task scope | Context needed | Typical metric |
|---|---|---|---|
| SWE-bench | Fix a real issue in a full repository | Thousands of files | Percentage resolved |
| HumanEval | Write one function from a docstring | A single prompt | pass@k |
| Competitive programming sets | Solve an algorithmic puzzle | Problem statement only | Accepted solutions |
| Terminal and agent tasks | Complete multi-step shell work | Live environment | Task success rate |
Because the benchmark rewards navigation and tool use, scores depend heavily on the agent scaffold around the model: the retrieval method, the editing tools, the number of attempts and whether the agent can run tests. Early systems that only retrieved files and generated a patch resolved only a few percent of tasks, while agentic systems that browse, edit and test iteratively now resolve a large majority of SWE-bench Verified tasks.
Reading and reporting SWE-bench results
A SWE-bench score is meaningful only together with the variant, the scaffold and the attempt budget. Two identical models can differ by many percentage points when one uses a stronger agent loop or selects the best of several candidate patches.
| Factor | Why it matters | What to look for |
|---|---|---|
| Dataset variant | Difficulty differs between subsets | Full, Lite or Verified |
| Scaffold | Tools change achievable results | Agent framework and available actions |
| Attempts per task | Multiple tries inflate success | Single attempt or best-of-n |
| Test access | Running hidden tests leaks the answer | Confirmation that gold tests stayed hidden |
| Cost and steps | High scores can need heavy compute | Tokens or dollars per resolved task |
Responsible reports state these settings openly and, where possible, publish trajectories and patches so others can audit them. Leaderboards maintained by the benchmark authors require submissions to follow fixed rules, which makes their entries more comparable than vendor claims in isolation.
Limitations and known weaknesses
The most serious limitation is benchmark contamination. The repositories, issues and fixes are public on GitHub, so models trained on web-scale code may have seen the gold patches. High scores therefore do not prove generalization to private codebases.
- Narrow repository coverage: a few large Python projects dominate the original set, so results may not transfer to other ecosystems.
- Weak tests: some test suites accept incorrect patches, while others reject valid alternative fixes.
- Benchmark saturation: as top scores approach the ceiling, small differences say little about real capability.
- Missing dimensions: code quality, readability, security and maintainability are not scored.
These weaknesses explain why practitioners treat SWE-bench as one strong signal among several, combining it with newer, uncontaminated tasks and with evaluations on their own internal code.
A non-technical view
Imagine hiring a mechanic and testing them on real cars that came into a garage with real complaints. The mechanic reads the customer’s note, opens the hood and fixes the problem. Afterward, an inspector checks that the original fault is gone and that nothing else in the car was broken during the repair.
SWE-bench does the same for AI coding systems. It hands them genuine bug reports from public software projects and checks the repairs with automatic tests. The share of repairs that pass is the score, and it shows how well an AI handles realistic programming work.
A concrete example
An engineering team at a mid-sized software company wants to decide which AI coding assistant to roll out to about 200 developers. Vendor marketing quotes different numbers, so the team runs SWE-bench Verified itself with an identical agent scaffold, a single attempt per task and a fixed limit of 50 tool calls per issue.
Assistant A resolves 62% of the 500 tasks and Assistant B resolves 55%, but Assistant A uses roughly twice as many tokens per resolved issue. The team then runs both tools on 40 recently closed bugs from its own private repositories, which no model could have seen during training. There, the gap narrows to four percentage points. The team selects Assistant B for daily use because of its lower cost per resolved task, and keeps the internal test set for future comparisons.