Best AI for coding: 97% on one benchmark, 61.5% on a harder one
If you want one name, the honest answer as of August 2026 is that the best AI for coding depends on which repository you test it against, and the gap between repository sets is wider than the gap between models. On SWE-bench Verified, a 500-task set of real GitHub issues run in isolated Docker containers, Vals reports Claude Opus 5 at 97.00%, with seven of the 86 models it has evaluated sitting at 95% or better. Move to SWE-bench Pro, which holds a different set of repositories constant, and the best public score is 61.50%. That is 35 points between the top of one board and the top of the other, by our subtraction, on the same kind of work.
So the useful question isn’t which model wins. It’s which constant a benchmark holds, and whether that constant looks anything like your codebase. Every figure below comes from a leaderboard, paper or pricing page we opened in August 2026, and in each case we say what the test held fixed. We didn’t run a private head-to-head, because the published tests already hold a repository constant across models, which is the only thing a private test would be doing.
What “the same repository” buys you, and what it hides
SWE-bench was built on that idea. The original paper assembled 2,294 problems from real GitHub issues and their pull requests across 12 popular Python repositories. Every model gets the same codebase and the same issue text, and the test is whether the project’s own tests pass afterwards. When that paper landed, the best model was Claude 2, and it resolved 1.96% of the issues.
That number is worth sitting with, because it dates the whole exercise. A benchmark that separated nothing but degrees of failure now has seven models bunched above 95%. We looked at that pattern in why benchmarks saturate within months, and coding is the clearest case of it anywhere.
The catch is what a fixed public repository set does to contamination. Those 12 Python projects are popular and heavily discussed, so a model trained after 2023 has very likely read them, and read the patches that closed the issues. A high score can mean the model solved the problem, or that it recalled the fix. Nobody outside the labs can tell those apart from a score alone.
SWE-bench Verified: 500 problems and a crowded ceiling
Verified is the cut-down set that most vendors quote. Vals says it runs the models under a harness of its own, and that harness is deliberately thin: the model gets one tool, bash, and must navigate, search and edit using standard command line tools. Its board was last updated on 19 August 2026.
| Model | SWE-bench Verified accuracy |
|---|---|
| Claude Opus 5 | 97.00% |
| DeepSeek V4 Pro 0813 | 96.40% |
| Kimi K3 | 93.40% |
| Claude Opus 4.8 | 88.60% |
| Grok 4.5 | 86.60% |
Vals’ own note on the board is the more useful line: seven of 86 models reach 95% or better, so little room is left to separate frontier models. Read as a buying decision, that says the top tier is close to a coin flip and the real differences sit below it. Claude Opus 4.8 at 88.60% trails the leader by 8.4 points, which works out at roughly one extra failed problem in twelve on a 500-task set. That is our arithmetic, not the board’s.
SWE-bench Pro: the scores on code the models have not read
SWE-bench Pro was built to break the contamination assumption. The paper behind it, titled “SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?”, describes 1,865 problems across 41 actively maintained repositories, split into a public set of 11 repos, a held-out set of 12, and 18 proprietary commercial codebases. The authors say every task is human-verified and augmented with enough context to be resolvable, so the difficulty isn’t meant to come from missing information.
The public split is 731 instances from GPL-licensed projects, and the leaderboard says that choice is deliberate: strong copyleft licences reduce contamination risk, because models are unlikely to have encountered the code in training. A further 276 instances come from startup codebases and 858 are held back entirely.
| Model | SWE-bench Pro public score |
|---|---|
| Muse Spark 1.1 | 61.50 ± 3.10 |
| gpt-5.4 (xHigh) | 59.10 ± 3.56 |
| Muse Spark | 55.00 ± 3.60 |
| claude-opus-4-6 (thinking) | 51.90 ± 3.61 |
| gemini-3.1-pro (thinking) | 46.10 ± 3.60 |
| claude-4-5-Sonnet | 43.60 ± 3.60 |
| gpt-5.2-codex | 41.04 ± 3.57 |
| qwen3-coder-480b-a35b | 38.70 ± 3.55 |
| gemini-3-flash | 34.63 ± 3.55 |
| gpt-oss-120b | 16.20 ± 2.67 |
| glm-4.6 | 9.67 ± 2.15 |
| llama4-maverick-17b-instruct | 5.24 ± 1.24 |
| codestral-2405 | 1.51 ± 1.51 |
Two things stand out. The first is that the lead isn’t clean: Muse Spark 1.1 posts 61.50 with an interval of plus or minus 3.10, and gpt-5.4 at xHigh effort posts 59.10 plus or minus 3.56, so the two overlap. Anyone quoting a single winner from this board is quoting noise.
The second is that the bottom is nothing like the top. On the same 731 problems, codestral-2405 resolves 1.51% and glm-4.6 resolves 9.67%, while llama4-maverick-17b-instruct manages 5.24%. Those are the losers on this test, and the distance between them and the leaders is the part of the field that the Verified board no longer shows you at all.
Seven models clear 95% on the repositories they were probably trained on. On GPL code picked because they probably were not, the best score is 61.50%.
Vals SWE-bench Verified and the SWE-bench Pro public leaderboard, August 2026
Terminal-Bench: the harness gets scored too
Terminal-Bench holds a terminal constant rather than a repository, so it covers the shell work that sits around an issue fix. Its 2.1 leaderboard takes submissions run through a common runner, and it lists an agent paired with a model rather than a model on its own. Reading it as a model ranking gets you the wrong answer, because several models appear twice.
| Model | Agent | Accuracy | Agent | Accuracy | Spread |
|---|---|---|---|---|---|
| GPT-5.5 | Codex | 83.1% | Terminus 2 | 78.0% | 5.1 |
| Fable 5 | Claude Code | 83.8% | Terminus 2 | 80.4% | 3.4 |
| Opus 4.7 | Claude Code | 68.9% | Terminus 2 | 66.1% | 2.8 |
| Gemini 3.1 Pro | Gemini CLI | 65.8% | Terminus 2 | 65.6% | 0.2 |
That spread is the finding. Swapping GPT-5.5 from Codex to Terminus 2 costs 5.1 points, which is larger than the entire gap between first and second place on the board. Claude Code with Fable 5 leads at 83.8% and Codex with GPT-5.5 follows at 83.1%, so 0.7 points separate the top two. The tool you wrap around the model moves the number more than the choice of model does at that end of the table.
The bottom of that board names a loser too. Claude Code running GLM-5.1 sits 17th of 17 at 58.7%, which is 25.1 points behind the same agent running Fable 5. If you want the mechanics of why the wrapper matters this much, we walked through the stages in how coding agents actually work.
The one test run on repositories the developers already knew
Every benchmark above holds a repository constant that is new to the person reading the score. METR ran the opposite experiment. In its randomised controlled trial, 16 experienced developers worked on 246 issues in their own large open-source repositories, averaging more than 22,000 stars and a million lines of code. Each issue was randomly assigned to allow AI tools or forbid them.
When AI was allowed, the developers took 19% longer. They had forecast a 24% speed-up beforehand, and after the fact they still believed AI had sped them up by 20%. So the measurement and the perception pointed in opposite directions, which is the single most awkward result in this whole area.
The counter-case is real and METR states it plainly: 16 developers is a small sample, and the authors do not claim their developers or repositories represent most software work. Deep familiarity with a mature codebase is exactly the condition where a model’s suggestions have least to add. Even so, that condition describes a lot of paid engineering, and no leaderboard measures it.
What it costs, and the number almost nobody publishes
Prices for the coding tools cluster tightly at the entry tier and then fan out. Anthropic’s pricing page lists Claude Pro at $20 per month billed monthly or $17 on an annual plan, Max 5x from $100 per month, and Team seats at $25 monthly or $20 annually, with Claude Code included on every tier. GitHub’s plan documentation lists Copilot Pro at $10 per month with 1,500 AI credits, Pro+ at $39 with 7,000, and Max at $100 with 20,000. Cursor lists a free Hobby tier, Pro at $20 per month and Teams Standard at $40 per user, with on-demand usage billed in arrears once the included allowance runs out.
| Plan | Monthly price | Included allowance as stated |
|---|---|---|
| GitHub Copilot Pro | $10 | 1,500 AI credits (1,000 base, 500 flex) |
| Cursor Pro | $20 | Extended agent limits, then on-demand billing |
| Claude Pro | $20 ($17 annual) | Claude Code included |
| GitHub Copilot Business | $19 per seat | 1,900 credits per user |
| GitHub Copilot Pro+ | $39 | 7,000 credits (3,900 base, 3,100 flex) |
| Cursor Teams Standard | $40 per user | Included usage, then billed in arrears |
| Claude Max 5x | from $100 | 5x Pro usage |
| GitHub Copilot Max | $100 | 20,000 credits (10,000 base, 10,000 flex) |
None of those figures tells you the cost of finishing a task, because credits, agent limits and usage multipliers are all vendor-defined units. The one public coding board we found that prints real money is aider’s polyglot leaderboard, which runs 225 Exercism exercises across C++, Go, Java, JavaScript, Python and Rust and publishes the total dollar cost of each run.
On that board o3-pro at high effort scores 84.9% for $146.32, while gpt-5 at low effort scores 81.3% for $10.37. So 3.6 points of accuracy cost about 14 times the money, which is our division of the two published totals. Those are older model versions than the ones on the other boards here, so treat the ratio as the finding rather than the specific models. We worked the same trade-off from the billing side in our piece on whether to buy a seat or run a GPU.
What we could not verify, and what would change the answer
Three limits are worth stating outright. Only one of these boards describes running the models itself, which is Vals, while Terminal-Bench and SWE-bench Pro publish results submitted through a shared runner. Submitted numbers are reproducible in principle and self-selected in practice, because a bad run doesn’t get sent in.
The SWE-bench Pro board also mixes conditions inside one table. Rows marked with an asterisk were run under the mini-swe-agent harness, and greyed-out results used capped cost with a 50-turn limit while the rest ran uncapped with 250 turns. A model given five times the turns is not being asked the same question, so the column doesn’t rank cleanly top to bottom.
And all of this is Python-shaped. A 2026 paper on unfamiliar languages tested six coding agents on four esoteric languages, and found the strongest of them, Claude Opus 4.6 and GPT-5.4 at xhigh effort, often avoided writing the target language at all. They wrote Python that generated Brainfuck and Befunge-98 instead.
The authors read that as evidence the agents build a working model of a language from tool feedback rather than knowing it, which is encouraging for adaptability and unhelpful if you want a direct answer about your own stack. For a wider view of what earns a slot in a working setup, we covered that in AI tools for developers.
What would change the conclusion is a benchmark that holds a private repository constant and publishes the cost per resolved issue alongside the score. SWE-bench Pro’s commercial split is the closest thing to it, and 18 of its 41 repositories are proprietary, but those results aren’t public. Until something like that exists, the defensible reading is narrow: on public Python the frontier is saturated and the model barely matters, on unfamiliar code the field spreads across 60 points, and on a codebase you already know well the only controlled trial anyone has run found the tools made experienced developers slower.
Get the daily rundown
One email each weekday with the AI news that matters, every claim linked to its primary source.
Free, one email each weekday, unsubscribe in one click. We never sell or share your address.
