How SWE-bench Was Built, and How It Broke

This post was originally written in Chinese and translated to English by Claude Fable 5. The original version is here.

In February this year, OpenAI published an engineering blog announcing they would no longer report SWE-bench Verified scores. Their recommendation: Verified is no longer suitable for evaluating frontier models, use SWE-bench Pro instead.

Less than five months later, last week, I saw on X that OpenAI had published another blog. They had audited SWE-bench Pro's data, estimated that roughly 30% of the tasks have serious problems, and walked back their earlier recommendation.

I remember when SWE-bench first came out in 2023, the strongest models solved less than 2% of it, and everyone figured it would last years. Then people discovered the original dataset had too many broken tasks, so OpenAI manually filtered it into Verified. Once Verified was nearly saturated, Scale AI released the harder Pro. And now Pro has gone from "recommended" to "roughly a third of the tasks may be broken" in five months.

The same task production method, patched and repatched across three generations. The problem isn't just that models improve too fast and saturate old questions. It's more fundamental: how exactly do we convert a piece of real software engineering work into an exam that can be graded automatically?

Over the past few years, whenever a coding model launches, we've gotten used to looking straight at the SWE-bench percentage. 70%, 75%, 80%, as if a higher score means the model is closer to being a software engineer who can work independently. But before comparing those numbers, it's worth looking at how this benchmark was built in the first place.

How SWE-bench Makes Tasks: Reverse-Engineering Exams from GitHub History

The original SWE-bench was released in 2023. Its biggest contribution was pushing code generation from HumanEval-style "complete a function" to issue resolution in real repositories.

The researchers first scraped about 90,000 pull requests from 12 major Python repositories, then filtered layer by layer:

1. The PR is merged and explicitly resolves a GitHub issue.
2. The PR modifies test-related files, suggesting the author likely added tests for the change.
3. The repository installs and runs successfully at the base commit before the PR.
4. When the tests added or modified in the PR are applied to the old code, at least one test fails.
5. After applying the PR's implementation code, those tests all pass, and previously passing regression tests don't break.

After filtering 90,000 PRs, 2,294 tasks remained. Each task has four parts:

• Codebase: the repository snapshot before the PR was merged.
• Problem statement: the linked issue's title, body, and relevant discussion before the PR started.
• Gold patch: the non-test code changes from the original PR, a known-working reference answer.
• Test patch: the test changes from the original PR, serving as a verifier the agent never sees.

During evaluation, the agent only gets the problem statement and the old codebase. It can search the code, modify files, run existing tests, and finally submit a patch. The benchmark applies the hidden test patch, and if all FAIL_TO_PASS and PASS_TO_PASS tests pass, the task counts as solved.

In 2023, this design was genuinely good. Real codebases, real issues, solutions written by real developers, and an automatically executable grader. It also scales: GitHub history already contains ready-made problems, answers, and tests, so you don't need to organize hundreds of engineers to write tasks from scratch.

But the cleverest part of SWE-bench later became the part where it broke.

Problem One: Issues, PRs, and Tests Were Never Designed as an Exam

SWE-bench implicitly assumes two equations:

Issue description = complete requirements

The original PR's tests = complete, implementation-agnostic acceptance criteria

Real software development doesn't work that way.

A GitHub issue is just a slice of a team's collaboration. Huge amounts of context live in maintainers' heads, in repository conventions, in other issues, even in offline discussions. The issue author only needs people familiar with the project to know what they're talking about. They never intended to write a PRD that a stranger could pick up and fully implement.

Same with the tests in a PR. They exist to prove that this one implementation in front of them is safe to merge, and incidentally to prevent this bug from recurring. The original author has already chosen the function names, module boundaries, and control flow, so the tests naturally get written around those choices.

But once you put them in a benchmark, the tests' role changes: they're no longer verifying one known implementation, they're supposed to serve as a universal grader for all unknown implementations. The distance between those two jobs is much larger than it looks.

OpenAI later categorized the common failure modes:

• Overly strict tests: tests depend on helper names, function signatures, or internal structures never mentioned in the prompt, rejecting functionally correct alternative implementations.
• Underspecified prompts: hidden tests check behavior the task never specified and that can't be reasonably inferred from the repository.
• Low-coverage tests: tests only cover the paths the original PR cared about, so an agent can pass with a stub or a partial implementation.
• Misleading prompts: the task's requirements outright contradict the hidden tests.

A very typical overly strict case: the original author extracted a private helper along the way while writing the fix, and the tests import that helper directly. An agent writes the same logic inline at the call site, with completely correct external behavior, but fails to compile because it didn't guess that function name.

The reverse happens too. The original PR's tests only cover a small part of a new interface, so an agent scaffolds the interface, leaves most of the logic as a no-op, and still gets a pass.

So the benchmark wants to measure "did you implement the requirements correctly," but the actual grading mixes in "did you reproduce the shape of the original author's implementation."

Verified and Pro Both Failed to Fix It

SWE-bench Verified was aimed squarely at these data quality problems.

In 2024, OpenAI hired 93 software engineers with Python experience to manually review 1,699 tasks randomly sampled from the original SWE-bench. Each task was independently annotated by three people, checking whether the problem statement was complete and whether the FAIL_TO_PASS tests would reject reasonable answers. If even one of the three flagged a serious problem, the task was filtered out. 500 tasks survived, forming SWE-bench Verified.

This genuinely made the benchmark much better. But as frontier models' pass rates approached 80%, the remaining failures started to look increasingly strange.

In February this year, OpenAI picked 138 tasks that o3 frequently failed across 64 independent runs and audited them more deeply, with at least six experienced engineers reviewing each task independently. The result: 59.4% of the audited tasks had substantive test design or problem description issues. 35.5% had overly narrow tests, and 18.8% had tests checking extra functionality the prompt never asked for. OpenAI stopped reporting SWE-bench Verified as a result.

Read that number carefully. 59.4% does not mean six in ten of Verified's 500 tasks are broken. OpenAI deliberately selected the tasks models frequently fail, so the problem rate was always going to be higher. What it really shows: once models can solve most of the normal tasks, a substantial share of the remaining "difficulty" on the leaderboard is actually measurement error.

SWE-bench Pro tried to raise the bar again. It expanded to 41 repositories and 1,865 tasks, added JavaScript, TypeScript, and Go, and included held-out and commercial repositories. Tasks were longer, often required editing multiple files, and went through human augmentation and review. At launch it looked effective, with the strongest models passing only around 23%. But eight months later, the top score on the public split had climbed to 80.3%.

Which brings us back to the blog from the opening. In July this year, OpenAI audited the 731 public SWE-bench Pro tasks. An agent-assisted pipeline flagged 200 tasks with serious problems, 27.4%. A separate group of experienced software engineers independently reviewed the tasks and flagged 249, 34.1%. OpenAI's final estimate: roughly 30% of SWE-bench Pro's public tasks are broken.

Pro's problem isn't that it failed to add difficulty, or that it skipped human review entirely. Its problem is that the underlying production method never changed: tasks were still primarily extracted from historical feature changes, issues, and PRs, then materials that had served one specific development process were reworked into prompts and tests.

If the prompt, solution, and verifier were never designed together as one evaluation artifact, later human review can only try to patch the gaps between them. The gaps are very hard to patch completely.

Problem Two: Data Contamination

This production method has a second, more obvious problem: the questions and answers were in the models' training data all along.

SWE-bench's issues, PR discussions, tests, and gold patches all sit publicly on GitHub. And public GitHub repositories happen to be one of the most important training data sources for every frontier model today.

OpenAI ran dedicated contamination probes against GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash. Every frontier model tested could recite the original fix or reproduce task-specific information verbatim on some tasks. Given nothing but a task ID, Gemini could output the specific regex change and gold patch for the Django username validator task. While solving a task that was nearly impossible to infer from the prompt, GPT-5.2 spontaneously mentioned the edit_only parameter from the corresponding Django release note.

A model doesn't need to have memorized the entire patch to gain an edge. If it has seen the relevant release note, PR discussion, function names, or the general approach to the fix, it's no longer facing a genuinely new problem.

Contamination and verifier error are two independent problems. Verifier error grades correct answers as wrong, or incomplete answers as right. Contamination lets models that have seen the answers systematically score higher. And tasks that are too short, drawn from too few repositories, create ceiling effects that squeeze models of different capability into a narrow score range. All of these distort the leaderboard, but in different directions. And a model's ability to memorize training data is not the same as its ability to solve problems in an unfamiliar codebase.

Both problems with the SWE-bench approach are now on the table: verifiers don't match the requirements, and the questions and answers leaked long ago. Let's look at how the new generation of benchmarks responds to each.

New Benchmark 1: DeepSWE, Designing Prompt, Solution, and Verifier Together

DeepSWE, released in May this year, looks on the surface like just a harder SWE benchmark: 113 tasks covering 91 active open-source repositories and five languages. The average prompt is only half as long as SWE-bench Pro's, but the reference solutions are 668 lines longer on average, about 5.5 times Pro's. Agents have to explore the codebase themselves, find where to make changes, and decide on an implementation strategy, rather than coding against a detailed todo list.

But I think its most important change isn't longer tasks. It's that it no longer reverse-engineers exam questions from already-merged PRs.

DeepSWE's tasks are written by engineers from scratch. Some are inspired by unresolved GitHub issues, but the reference solutions are entirely new and never get merged back into the upstream repository. For each task, the prompt, reference solution, and verifier are designed together: the prompt describes the external behavior to implement; the reference solution proves the task is actually solvable but plays no role in final grading; the verifier checks behavior through public APIs and observable output, without depending on private helpers or the internal structure the author happened to choose.

Every verifier goes through dedicated QA before entering the pool. Reviewers check whether the prompt and tests correspond one-to-one, whether multiple reasonable implementations would be accepted, whether the task is something a maintainer would actually hand to an agent, and whether the environment and dependencies are stable. Multiple frontier agents also attempt each task in advance. The rollouts that almost solve a task are especially valuable, because they expose the verifier's misjudgments at the boundaries.

DeepSWE ran its own cross-benchmark audit: randomly sample 30 tasks each from DeepSWE and SWE-bench Pro, run 10 agent configurations three times each, then have an LLM analyzer independently judge whether each patch truly satisfies the requirements. By that judgment, Pro's verifiers rejected about 24% of correct answers and accepted about 8% of incorrect ones. DeepSWE's corresponding rates were 1.1% and 0.3%.

The direction of this result is convincing, though the exact numbers deserve a discount. The analyzer is itself an LLM judge without fully independent human calibration, and the auditor is DeepSWE's own publisher. To their credit, they at least published the tasks, trajectories, and analysis, so others can keep verifying.

DeepSWE v1.1 added another layer of isolation. The agent works and commits in its own container, and the evaluation system only extracts the committed diff and runs it in a completely fresh verifier container. Future git history is deleted, and test results are recorded item by item in a structured format. This way an agent can't cheat the grader by modifying the test runner, deleting tests, or digging future commits out of git log.

This process is obviously more expensive. It gives up SWE-bench's scale advantage of automatically mining tasks from GitHub history. But what it measures is also cleaner: not whether an agent can guess how the original author wrote it, but whether it can implement the specified behavior.

New Benchmark 2: CursorBench, Matching the Task Distribution to Real Usage

DeepSWE mainly fixes validity within a single task: the prompt, solution, and verifier must actually align. CursorBench cares about something else: whether the tasks, taken together, represent the work developers actually hand to coding agents.

Cursor uses Cursor Blame to trace already-committed code back to the agent request that produced it, obtaining naturally paired data from internal codebases and other controlled sources: what the developer said at the time, and what code they ultimately accepted.

This lets CursorBench's prompts keep the brevity and ambiguity of real usage, without being rewritten into exhaustively detailed specifications. The tasks also go beyond bug fixes, covering edit, refactor, bug finding, planning, code review, instruction following, and advanced tool use.

Cursor also doesn't treat the offline benchmark as final truth. They run controlled experiments on live traffic, observing multiple signals of interaction and output quality, checking whether agents that score higher offline actually deliver better results for developers. The tasks themselves get refreshed every few months, evolving with how coding agents are actually used.

This resonates with me personally. I previously worked on code migration benchmark generation, and the most immediate problem was that repositories randomly collected from GitHub included lots of toy projects whose distribution looked nothing like real customer workloads. Every individual test running correctly doesn't mean the aggregated success rate can predict production.

CursorBench's cost is auditability. The tasks and graders are mostly private, so outsiders can hardly judge independently whether it favors Cursor's own harness or particular models. Private data also doesn't guarantee zero contamination. Cursor's current leaderboard carries a specific note: Grok 4.5's training data accidentally included an earlier snapshot of the Cursor codebase, giving it an advantage on CursorBench that can't be precisely quantified.

Putting the three approaches side by side, they're actually optimizing for three different problems:

• SWE-bench pursues scalability, reproducibility, and automatic grading.
• DeepSWE focuses on task-verifier alignment and generalization to new problems.
• CursorBench focuses on aligning the benchmark distribution with real product usage.

No single design gets full marks on everything. Public benchmarks are easy to audit but will eventually end up in training sets. Private benchmarks resist contamination but struggle to earn external trust. Writing tasks from scratch by hand makes verifiers cleaner, but it's expensive and the sample sizes are small. Sampling directly from production is closest to real usage, but it's hard to make tasks reproducible, and product harness capability can get mixed into model capability.

Harder Doesn't Mean Better

When a new benchmark knocks model scores from 80% back down to 40%, it's easy to assume the new ruler is more accurate.

But spreading out the scores proves nothing by itself. Make the tasks obscure enough, make the environment fragile enough, or have hidden tests check more details never written into the prompt, and you can drive every model's score down just the same. What actually matters: what exactly is being differentiated?

If a benchmark wants to measure autonomous software engineering capability, it needs to answer several things at once:

1. Task validity: are these the software engineering tasks we actually care about, or just the ones that are easy to collect?
2. Verifier validity: does passing mean the requirements were truly met, and does failing truly reflect a capability gap?
3. Contamination control: has the model seen the problem, the answer, or side-channel information sufficient to reconstruct the answer during training?
4. Harness clarity: is the score measuring the base model, the model plus a standardized scaffold, or the complete product?
5. Temporal validity: will the task distribution and difficulty go stale as agent capability and developer usage evolve?

SWE-bench wasn't a bad benchmark back then. It defined the basic task format for evaluating coding agents for years afterward. It's just that once models went from 2% to 80%, the data noise that could once be ignored became the dominant signal on the leaderboard.

That's my biggest reflection after reading all this material: a benchmark is not a set of questions you publish once and never touch again. It's more like a continuously operating measurement system. Someone produces new tasks, someone reviews prompts and verifiers, someone checks for contamination and reward hacking, someone calibrates offline results against real usage. Models are changing, agent harnesses are changing, and the work developers hand to agents is changing, so the system has to keep changing with them.

The truly reliable coding evals of the future probably won't be a static dataset everyone grinds for three years, but an ongoing process of writing, rotating, auditing, and calibrating tasks.

References:

• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• Introducing SWE-bench Verified
• Why SWE-bench Verified no longer measures frontier coding capabilities
• Separating signal from noise in coding evaluations
• SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
• DeepSWE
• DeepSWE v1.1
• How we compare model quality in Cursor
• SWE-Bench Pro 饱和之后，有人做了一把新尺子