Updated April 25, 2026: Claude Opus 4.7 now leads SWE-bench Verified at 87.6% and SWE-bench Pro at 64.3%. GPT-5.5 pushes Terminal-Bench 2.0 to 82.7%. MiniMax M2.7 hits 57.0% on Terminal-Bench 2 — open source, publicly available weights. The Stanford AI Index 2026 confirms SWE-bench Verified is approaching 100% of the human baseline. The plateau is gone. The benchmark is cooked. And that changes the analysis below in ways worth reading. The new section is at the bottom.
Three frontier models walk into SWE-bench Verified. Claude Opus 4.6 scores 80.8%. Gemini 3.1 Pro, released February 19, scores 80.6%. GPT-5.3-Codex scores 80.0%.
The variance is smaller than the margin of error on most software benchmarks. Statistically, these three models are identical on the metric the industry has been using as its primary yardstick for coding capability.
This is not a success story for the benchmark. It’s a failure mode.
What SWE-bench Actually Measures#
SWE-bench Verified presents models with real GitHub issues from Python open-source repositories. The model must write a patch that passes the associated test suite. It’s a respectable benchmark — grounded in real-world bugs, not synthetic puzzles — and it drove meaningful progress for two years.
But it measures one narrow thing: isolated bug-fixing in a well-structured Python codebase with existing tests. A model that scores 80% on SWE-bench can:
- Read a traceback and identify the affected function
- Generate a patch that passes the existing test suite
- Handle standard Python idioms across popular libraries
What it cannot reveal:
- Whether the model can maintain coherent intent across 50+ files
- Whether it can design and implement a feature from a spec, not just fix a known bug
- Whether it can orchestrate tools, manage state, and recover from errors in a multi-step agent loop
- Whether it can interact with a running UI, a database, or a cloud API
- Whether it can work autonomously for 2 hours without going off the rails
The tasks that matter most for real-world agentic coding are nowhere in SWE-bench’s test suite.
The Price Compression Problem#
The benchmark convergence arrives alongside a pricing story that’s harder to dismiss.
Gemini 3.1 Pro costs $2 per million input tokens and $12 per million output tokens. Claude Opus 4.6 costs $15 input and $75 output. For teams building agentic pipelines that process millions of tokens per day, Gemini is 6–8x cheaper for nominally equivalent SWE-bench performance.
Then there’s MiniMax M2.5, which benchmarks at 80.2% SWE-bench Verified at $0.30 input / $1.20 output. The price-per-point at the frontier is collapsing.
This matters because it forces a sharper question: if you can’t distinguish these models on the benchmark everyone uses, and one costs 25x less, why are you paying for the expensive one?
The answer cannot be “better benchmark numbers.” It has to be something the benchmark doesn’t measure.
Terminal-Bench 2.0: A Different Story#
The more interesting results come from Terminal-Bench 2.0, which evaluates models on realistic terminal-based agentic tasks — not isolated patches, but multi-step autonomous workflows.
The rankings shift noticeably:
| Model | Terminal-Bench 2.0 |
|---|---|
| GPT-5.4 | 75.1% |
| Gemini 3.1 Pro | 68.5% |
| Claude Opus 4.6 | 65.4% |
Claude drops behind on this benchmark — which is worth being honest about. But Terminal-Bench 2.0 still doesn’t capture tool orchestration quality, computer use fidelity, long-context coherence at 1M tokens, or the behavioral properties that matter when an agent is running unsupervised for hours.
What these numbers show is that the relative rankings change depending on what you measure. A model that performs identically on SWE-bench may behave very differently under agentic conditions. The flat-line at 80% is an artifact of the benchmark reaching its ceiling, not a statement about capability parity.
What Actually Differentiates Frontier Models in 2026#
For teams making actual architecture decisions, here is where the real differences live:
Tool orchestration. Claude consistently performs better on complex multi-tool chained tasks — reasoning across API calls, file edits, terminal output, and web fetches in a single coherent session. This is a function of how Anthropic has trained Claude’s tool use behavior and how Claude Code’s MCP-native architecture structures context.
Computer use fidelity. Claude Opus 4.6 and Sonnet 4.6 are the only frontier models with production-grade computer use deployed in a major coding tool (Claude Code, launched March 23). Gemini’s computer use is available in Gemini Code Assist but tightly constrained to IDE surfaces. This is a meaningful gap for any task requiring visual context — rendered UIs, desktop apps, or systems with no programmatic API.
Long-context coherence. Claude Code now defaults to 1M context for Max, Team, and Enterprise accounts. What matters is not just the window size but whether the model maintains coherent intent across it. Independent testing consistently shows Claude with stronger recall and reasoning across very long contexts.
Behavioral safety under autonomy. This is the hardest to benchmark and the most important for unsupervised agentic use. How does the model behave when it hits an unexpected state at step 47 of a 60-step task? Does it attempt a risky recovery or pause and report? Claude’s Constitutional AI training and Anthropic’s alignment investment show up here — not in benchmark scores, but in real production deployments where teams report fewer catastrophic failures in long-running agent sessions.
Ecosystem integration. Claude Code’s MCP-native architecture means every tool in the 5,800+ MCP server ecosystem is available natively. Gemini Code Assist runs in VS Code and JetBrains. The surface area of what Claude can touch is structurally larger.
The Benchmark Gap Anthropic Should Worry About#
One number worth sitting with: on Terminal-Bench 2.0, GPT-5.4 leads at 75.1% and Gemini 3.1 Pro is at 68.5%, while Claude Opus 4.6 is at 65.4%. OpenAI’s lead on that benchmark is real, and it suggests that for raw agentic terminal task performance, Claude is not automatically first.
The counterargument is that Terminal-Bench 2.0 still measures isolated agentic tasks, not integrated systems. Claude Code’s architecture, tooling, and the full MCP ecosystem mean the system — agent plus tools plus infrastructure — outperforms what the raw model score suggests. But that’s a fragile argument if OpenAI or Google close the integration gap.
Anthropic’s moat is not the model number. It’s the investment in safe, predictable autonomous behavior at scale — the kind that enterprise customers need before they’ll give an agent access to production systems. That’s harder to benchmark and harder to copy.
What Should Actually Replace SWE-bench#
The field needs a benchmark that reflects how models are actually being used in 2026:
- Multi-session, long-horizon tasks (days, not minutes)
- Tool-use complexity (orchestrating 10+ MCP servers simultaneously)
- Autonomous recovery from unexpected states
- Security and safety behavior under adversarial inputs
- Real business outcomes (feature shipped, incident resolved, PR merged)
Until that benchmark exists, the 80% cluster is a ceiling artifact, not a verdict. The models that matter are the ones that do well on the hard stuff that nobody has figured out how to measure yet.
Claude is betting that hard stuff is agentic autonomy at scale, with safety guarantees, in complex multi-tool environments. The March 23 Computer Use launch, the Agent Teams architecture, and the 1M context window are all part of that bet.
The SWE-bench score is a floor, not a ceiling. Everything interesting is above it.
April 2026 Update: The Plateau Is Gone — and So Is the Benchmark#
Three weeks after this piece was published, the plateau broke.
Claude Opus 4.7 launched April 14 with 87.6% on SWE-bench Verified and 64.3% on SWE-bench Pro. GPT-5.5, released April 23, scored 82.7% on Terminal-Bench 2.0. MiniMax M2.7, open-sourced April 12, hit 56.22% on SWE-Pro and 57.0% on Terminal-Bench 2.
The updated Terminal-Bench 2.0 picture:
| Model | Terminal-Bench 2.0 | SWE-bench Pro |
|---|---|---|
| GPT-5.5 | 82.7% | 58.6% |
| GPT-5.4 | 75.1% | — |
| Gemini 3.1 Pro | 68.5% | — |
| Claude Opus 4.6 | 65.4% | — |
| Claude Opus 4.7 | — | 64.3% |
| MiniMax M2.7 (open) | 57.0% | 56.22% |
Two things changed that shift the analysis:
SWE-bench Verified is now a dead benchmark. The Stanford AI Index 2026 (published April 2026) documents SWE-bench Verified approaching 100% of the human expert baseline — up from roughly 60% just one year earlier. A benchmark that closes from 60% to near-100% in twelve months is not measuring general capability. It’s measuring training data coverage and scaffold optimization. OpenAI quietly retired it as a primary metric after acknowledging contamination and flawed test design in approximately 59% of the evaluation set (see SWE-bench Pro vs. Verified: The Benchmark That Lied for the full analysis). SWE-bench Pro, which uses harder, curated GitHub issues with stricter validation, is now the meaningful number.
The open-source gap closed dramatically. When this article was written, there was no credible open-source competitor to frontier models on hard coding benchmarks. MiniMax M2.7 at 56.22% SWE-Pro — available on Hugging Face and Ollama, fully self-hostable — is eight points behind Claude Opus 4.7 on the same benchmark. That gap used to be 20+ points. The self-improvement technique M2.7 used (100 autonomous rounds of scaffold optimization) suggests the gap will keep closing.
The core argument of this piece stands: benchmark scores are a floor, not a verdict. What actually differentiates frontier models in production — tool orchestration, computer use fidelity, long-context coherence, behavioral safety under autonomy — is not captured by SWE-bench in any version. That argument is now more urgent, not less, as the closed-source models approach the SWE-Pro ceiling and open-source models close from below.
The question is no longer whether your model can solve an isolated Python bug. It’s whether your agent can ship a feature end-to-end without supervision and without doing something catastrophic when it hits an unexpected state at step 47. No benchmark measures that. Production deployments do.
Sources
- Gemini 3.1 Pro — Google Blog
- Gemini 3.1 Pro Complete Guide: Benchmarks, Pricing — NxCode
- Best AI for Coding 2026: Every Model Ranked — MorphLLM
- New AI Model Releases March 2026 — RenovateQR
- SWE-bench Verified leaderboard — swebench.com
- Stanford AI Index 2026 — HAI Stanford
- MiniMax M2.7 Official Announcement — MiniMax
- OpenAI GPT-5.5 Announcement — OpenAI
- Claude Opus 4.7 Release — Anthropic