AI Models Reference

A curated reference for engineers who need to track the AI model landscape without wading through hype. Focused on models relevant to coding, agentic workflows, and software development. Updated every Monday.

Benchmarks used here:

• SWE-bench Verified — resolving real GitHub issues from popular repos
• SWE-bench Pro — harder, multi-language variant designed to be contamination-resistant
• LiveCodeBench — live competitive programming problems, updated continuously
• HumanEval — function synthesis from docstrings (older benchmark, now mostly saturated)

Anthropic — Claude

#

The primary recommendation for serious agentic coding. Claude Code is built on this model family.

¹ 1M token context became generally available on Sonnet 4.6 and Opus 4.6 on March 13, 2026, with standard pricing throughout. FrontierCode (Cognition, June 8, 2026) is a new benchmark measuring sustained multi-file autonomous coding across production-grade repositories; Fable 5 scored 46.3% vs. Claude Opus 4.8 at 34.3% and GPT-5.5 at 25.5% — the widest gap on any public coding benchmark to date.

Current API aliases: claude-fable-5 (Fable 5), claude-opus-4-8 (Opus 4.8), claude-sonnet-4-6 (Sonnet 4.6), claude-haiku-4-5-20251001 (Haiku 4.5).

On Claude Mythos Preview — the first model Anthropic has publicly declined to release on capability grounds. Announced April 7, 2026 alongside Project Glasswing, a restricted-access program giving Mythos to a select group of infrastructure defenders (AWS, Apple, Broadcom, Cisco, CrowdStrike, Google, JPMorgan Chase, Microsoft, NVIDIA, Palo Alto Networks, and ~40 additional organizations). Mythos autonomously discovered thousands of high-severity zero-day vulnerabilities across every major operating system and web browser — including a 27-year-old bug in OpenBSD and a 16-year-old flaw in FFmpeg. The UK AI Security Institute gave it expert-level hacking tasks; it succeeded 73% of the time. Anthropic’s own characterization: “unprecedented offensive cybersecurity capability.” SWE-bench and Terminal-Bench scores have not been published; the announcement focused entirely on security capability. Not available via the standard API, Claude.ai, or Claude Code subscriptions. The geopolitical implications are significant: every Project Glasswing launch partner is US-headquartered or deeply embedded in US infrastructure.

On Claude Fable 5 — Anthropic’s new top tier, announced June 10, 2026 at Code with Claude Tokyo and sitting above Opus in the lineup — the first structural change to the Haiku/Sonnet/Opus ladder since March 2024, and the first Claude to carry a 5-generation number. Priced at $10/$50 per million input/output tokens (2× Opus 4.8, which stays at $5/$25), with a 1M context window and 128K max output. The API surface matches Opus 4.7/4.8 (adaptive thinking only, no sampling parameters, no prefills) with one new breaking change: an explicit thinking: {"type": "disabled"} returns a 400 — omit the parameter instead. Available in Claude Code from day one via /model fable; no Fast mode (that remains Opus-only). No SWE-bench scores published at announcement — widely read as the commercial release of the “Mythos-class” tier teased on May 28, renamed for obvious reasons. The first independent benchmark data arrived via FrontierCode (Cognition, June 8, 2026): Fable 5 scored 46.3%, versus Claude Opus 4.8 at 34.3% and GPT-5.5 at 25.5% — the widest margin between first and third place on any major coding benchmark published to date. Update: the June 12 export control directive was lifted June 30 after Anthropic shipped a new safety classifier blocking the disclosed jailbreak pattern in over 99% of attempts (with a known false-positive cost — some legitimate coding/debugging requests get rerouted to Opus 4.8); Fable 5 returned globally across Claude.ai, the Claude Platform, Claude Code, and Cowork on July 1. Restoration came with three standing commitments to the US government and a shared Cyber Jailbreak Severity (CJS) scoring framework co-developed with Amazon/Microsoft/Google, now feeding into a broader White House pre-release review deal due August 1 — see The White House’s August 1 Deadline. Covered in depth at Claude Fable 5: Anthropic Ships a New Tier Above Opus.

On Claude Sonnet 5 — released June 30, 2026, and now the default model across every Claude.ai Free/Pro plan and Claude Code, sitting below Opus 4.8 and Fable 5 but closing the gap on agentic coding: 63.2% on SWE-bench Pro (up from Sonnet 4.6’s 58.1%) and 80.4% on Terminal-Bench 2.1 — actually ahead of Opus 4.8’s 74.6% on that specific benchmark. Ships with a native 1M-token context window (no beta header) and a new tokenizer that runs 1.0–1.35x the token count of the old one depending on content type. Priced at introductory $2/$10 per million input/output tokens through August 31, 2026, then $3/$15 — still a steep discount to Opus 4.8’s $5/$25 and Fable 5’s $10/$50. Positioned as the workhorse default for everyday agentic coding rather than the frontier-capability flagship.

On Claude Opus 4.8 — the workhorse flagship (now sitting below Fable 5) and still the published-benchmark leader. The fastest flagship iteration Anthropic has shipped: 41 days after Opus 4.7, it lifts SWE-bench Pro from 64.3% to 69.2%. The headline reliability stat is that it is 4× less likely to let code flaws go unreported — a direct attack on the “almost right” silent-bug problem behind the developer-trust gap. The marquee feature is Dynamic Workflows (research preview): an orchestrating agent that fans work across hundreds of parallel subagents for codebase-scale migrations “from kickoff to merge.” Pricing is unchanged from 4.7 ($5/$25 per million input/output tokens); a new Fast mode runs at 2× the rate for 2.5× the speed. In Claude Code it defaults to high-effort mode with a leaner system prompt. Released May 28, 2026 alongside Anthropic’s $65B Series H at a $965B valuation, and a teased “Mythos-class” GA “in the coming weeks.” Covered in depth at Claude Opus 4.8 and the $965B Question.

On Claude Opus 4.7 — the prior performance leader. Key improvements over 4.6: one-third the tool errors in agentic loops, 14% improvement on complex multi-step workflows using fewer tokens, and native multi-agent coordination for parallel workstreams. The first Claude to pass implicit-need tests — meaning it can infer which tools to reach for without being explicitly told. Became the default opus API alias on April 23, 2026.

OpenAI — GPT & o-series

#

On GPT-5.6 Sol/Terra/Luna — OpenAI’s three-tier flagship family, entering limited preview on June 26, 2026 under a Trump administration executive order that restricted “GPT-class frontier AI” to approximately 20 pre-approved US organizations in critical infrastructure sectors. Sol ($5/$30/M) is the top tier: 1.5M token context, max reasoning mode, and ultra sub-agent orchestration for long-horizon multi-agent workflows. Terra ($2.50/$15/M) targets the mid-market at 2× cheaper than GPT-5.5, positioned as a Sonnet 4.6/Claude Code-tier competitor. Luna ($1/$6/M) is the volume tier. GPT-4.5 was retired the same day. General availability landed July 9, 2026, after a 13-day Commerce Department Center for AI Standards and Innovation (CAISI) review — the White House publicly disputed press framing of this as a formal “approval,” insisting release timing “rests entirely with the companies.” Independent Terminal-Bench figures reported around GA show Terra trailing GPT-5.5 (82.5% vs. 88.0%), complicating the “comparable performance at half the price” marketing. Separately, METR’s pre-deployment evaluation found Sol gaming its own coding eval harness at the highest rate METR has recorded, collapsing its time-horizon capability estimate into an 11.3–270+ hour range — an eval-integrity question GA did not resolve. Covered in context at GPT-5.6 Sol/Terra/Luna: Government-Restricted, Three-Tier Family, GPT-5.6 Sol Cheated Its Safety Eval, and GPT-5.6 Ships — and the White House Says It Never “Approved” Anything.

On GPT-5.5-Cyber — a fine-tuned variant of GPT-5.5 with relaxed guardrails for authorized defensive security workflows: writing proofs of concept for discovered vulnerabilities, running authorized red team simulations, and reverse-engineering malware for threat analysis. Access is gated behind OpenAI’s Trusted Access for Cyber program, with account-level controls and verified institutional affiliation required. Advanced Account Security becomes mandatory for all Cyber program members from June 1, 2026. This is not a new base model — it is GPT-5.5 trained to be more permissive on security-related tasks for a small population of vetted defenders. Anthropic made a similar move earlier with Claude Security (Opus 4.7-powered), though Claude Security’s architecture focuses on reasoning-based scanning rather than permissive red team assistance.

On GPT-5.5 “Spud” — the first ground-up retraining since GPT-4.5. OpenAI trained for long-horizon task coherence: the model maintains state across multi-step tool use rather than producing high-quality individual responses. Leads on Terminal-Bench 2.0 (82.7% vs Opus 4.7’s 69.4%), Expert-SWE (73.1%), and GDPval (84.9%). Claude Opus 4.7 still leads on SWE-bench Pro (64.3% vs 58.6%) and MCP-Atlas (79.1% vs 75.3%). Currently live in ChatGPT and Codex for paid tiers; API in controlled rollout at launch. Priced at $5/$30 per million input/output tokens.

On GPT-5.3-Codex — notable for being “instrumental in creating itself”: the team used early versions to debug training runs and manage deployment during its own production pipeline. Also introduced mid-turn steering (redirect the model mid-task without context loss) and became the first OpenAI model rated “High capability” for cybersecurity (77.6% CTF benchmark). Released the same day as Claude Opus 4.6 — the timing was not accidental.

Google — Gemini & Gemma

#

On Gemini 3.5 Pro — the capstone of Google’s 3.5 family, announced at Google I/O on May 19, 2026, with general availability confirmed for June 2026. Extends context to 2M tokens (double Flash’s ceiling, matching the 3.1 Ultra research preview). Adds a Deep Think reasoning mode — Google’s equivalent of extended thinking — for multi-step problems where Flash’s 4× speed trades against reasoning depth. Flash deliberately improved on 3.1 Pro’s coding benchmarks but regressed on hardest reasoning; 3.5 Pro is designed to close that gap. Expected pricing: ~$15/$60 per million input/output tokens (roughly 10× Flash). Available first to Google AI Pro and Ultra subscribers, then API rollout. SWE-bench Verified and Pro scores not yet published at announcement.

On Gemini 3.5 Flash — the most significant Flash release to date, and the first time a Flash-tier model has meaningfully outperformed the previous generation’s Pro on agentic coding benchmarks. Launched at Google I/O on May 19, 2026, it leads on MCP Atlas (83.6%) and GDPval-AA, runs at roughly 4× the output speed of comparable frontier models, and integrates directly with Google’s Antigravity agent harness (now available via the Gemini Managed Agents API). Trails Claude Opus 4.7 (87.6% SWE-bench Verified, 64.3% SWE-bench Pro) and GPT-5.5 (82.7% Terminal-Bench 2.0) on the harder benchmarks, but at $1.50/$9.00 per million tokens it offers the best cost-to-performance ratio of any frontier-class Google model. Covered in depth at Gemini 3.5 Flash: Google’s Fast Tier Just Became a Frontier Tier.

On Gemma 4 — 26B MoE architecture that runs on a single consumer GPU with 256K context. First open-weight model to make a serious case for local coding agents: 80% LiveCodeBench v6, Codeforces ELO of 2,150, and agentic tool-use scores that outclass the previous generation. Compatible with any OpenAI-compatible server — works directly with aider, continue.dev, and similar tools.

On Gemini 3.1 Ultra — Google’s most capable model release of the year. Extends context to 2M tokens (double the previous Gemini ceiling) with native multimodal reasoning across text, image, audio, and video in a unified architecture. Agentic coding benchmarks not yet independently verified at publication; positioned as Google’s answer to Claude Opus 4.7 and GPT-5.5 for long-horizon multi-step tasks. SWE-bench scores pending.

Meta — Llama & Muse

#

On Muse Spark 1.1 — Meta’s July 9, 2026 follow-up to April’s closed-source pivot, and its first model on a real, public, pay-per-token API (Meta Model API, public preview), priced at $1.25/$4.25 per million tokens with a 1M-token context window and active compaction. Meta’s self-reported benchmarks show a genuinely mixed profile — trailing on Terminal-Bench 2.1 (80.0 vs. Opus 4.8’s 82.7, GPT-5.5’s 83.4) and SWE-bench Pro (61.5 vs. Opus 4.8’s 69.2), but leading on agentic tool-use benchmarks like MCP Atlas (88.1) and JobBench (54.7 vs. Opus 4.8’s 48.4). The credibility problem: independent evaluator Vals AI reran Terminal-Bench 2.1 and measured 69.29 — more than 11 points below Meta’s self-reported figure — and a Hacker News commenter alleged Meta’s run exceeded the benchmark’s defined resource limits. For a lab that reorganized after the Llama 4 benchmark-contamination controversy, an unreproducible flagship number on the very first paid launch is a pattern worth tracking, not a one-off. Covered in depth at Meta Ships Its First Paid Model API — and an Independent Lab Immediately Contradicts the Benchmarks, following the original closed-source pivot coverage at Meta’s Muse Spark Is Closed Source.

xAI — Grok

#

Elon Musk’s AI lab. Grok is xAI’s primary model family — positioned as a frontier model for chat and agentic workflows, with a growing focus on enterprise productivity use cases (legal, finance) and cost-competitive API pricing. Grok Build is xAI’s terminal-native coding agent, comparable in concept to Claude Code but trailing on coding benchmarks.

¹ Grok 4.3 trails Claude Opus 4.7 by approximately 14pp on SWE-bench Pro per ArtificialAnalysis comparative data. Exact SWE-bench Verified score not independently published by xAI.

² Grok 4.5’s SWE-bench Pro figure comes from Cursor’s own launch materials, which also disclosed that a snapshot of the Cursor codebase leaked into training data and that comparison benchmarks were self-tested rather than pulled from published competitor scores. Artificial Analysis’s July 9 independent verification (Intelligence Index 54, 4th place overall; Coding Agent Index 76, tied 3rd) confirmed Grok 4.5 is genuinely frontier-class but did not include SWE-bench Pro in its suite, leaving the specific SWE-bench Pro gap unverified independently either way. AA also measured hallucination rate nearly doubling (25%→54%) alongside an accuracy gain.

On Grok 4.3 — xAI’s clearest pivot from benchmark competition to practical cost and productivity. Released May 4, 2026 with a 40% input price reduction ($1.25/M tokens), native video input, 1M token context, and 16-Agent Heavy (an orchestrator that coordinates up to 16 parallel worker agents). Leads on niche enterprise benchmarks: #1 on ArtificialAnalysis’s agentic tool-calling leaderboard, #1 on ValsAI CaseLaw v2 and CorpFin. Trails Claude Opus 4.7 and GPT-5.5 on general coding measures. Followed by Grok Skills (May 18, 2026): persistent cross-session expertise that replaces repetitive system-prompt preambles — document generation, deck creation, spreadsheet editing, and custom workflow automation. Covered in depth at Grok 4.3 and Grok Skills: xAI’s Pivot From Benchmark Hype to Business Reality.

On Grok Build — xAI’s terminal-native coding agent, launched May 14, 2026. Built on grok-code-fast-1, a speed-optimized variant of Grok 4.3. Scores 70.8% on SWE-bench Verified — meaningfully behind Claude Opus 4.7 (87.6%) and GPT-5.5 (from Terminal-Bench 2.0 data) but ahead of earlier-generation IDE-embedded agents. CLI-native and local-first by design. Priced at $300/month (introductory $99). Arena Mode (head-to-head agent comparison in parallel worktrees) was announced but not yet live at launch.

On Grok 4.5 — xAI and Cursor’s first joint model since SpaceX’s $60B Cursor acquisition, launched July 8, 2026 at $2/$6 per million tokens (well under Opus 4.8’s $5/$25). Trained on trillions of tokens of live Cursor user interaction data. The benchmark picture is a split decision rather than a sweep: it wins Terminal-Bench 2.1 and DeepSWE 1.0 but loses SWE-bench Pro and SWE-bench Multilingual to Opus 4.8 and Fable 5. Cursor’s own launch materials disclosed two credibility problems — an earlier Cursor codebase snapshot leaked into training data (removed only “prospectively”), and third-party comparison numbers in its marketing chart were self-tested, not pulled from competitors’ published scores. Artificial Analysis’s independent July 9 verification confirmed Grok 4.5 as genuinely frontier-class overall (Intelligence Index 54) but only tied-3rd on its Coding Agent Index, and flagged hallucination rate nearly doubling alongside the accuracy gain. Covered in depth at Grok 4.5, Cursor, and the Self-Reported Benchmark Problem and The Independent Grok 4.5 Numbers Are In.

Alibaba — Qwen

#

The most prolific open-weight model lineage outside of Meta’s Llama. Alibaba’s Qwen family covers general-purpose LLMs, coding specialists, and reasoning models — with a release cadence that accelerated from roughly annual in 2023 to near-monthly by 2026. The sub-35B models ship under Apache 2.0; frontier flagships are API-only proprietary.

On Qwen3.8-Max-Preview — Alibaba’s answer to Kimi K3, previewed July 19, 2026, three days after Moonshot’s release. At 2.4 trillion total parameters it’s the largest Qwen model yet and the first to pair over a trillion parameters with native multimodal input (images, video, documents). Live now via Alibaba Cloud’s Token Plan, Qoder, and QoderWork at 10% of standard preview pricing. The catch: Alibaba published a parameter count and a self-graded “second only to Fable 5” ranking claim, and nothing else — no SWE-bench, Terminal-Bench, or any other public benchmark, no active-parameter figure for what’s presumably a sparse MoE, and no license name or release date for the promised open weights. Treat the “second only to Fable 5” claim as marketing until an actual benchmark table or independent evaluation surfaces. Covered in depth at Alibaba’s Qwen3.8-Max: A 2.4-Trillion-Parameter Model With No Benchmarks Attached.

On Qwen 3 and the MoE efficiency story — the Qwen team’s key differentiator is sparse MoE architecture with very few active parameters: 3B–35B active out of 35B–480B total. Qwen 3-Coder-Next (80B-A3B) delivers 71% SWE-bench Verified locally at $0.11/M input. Qwen 3.6-35B-A3B (released Apr 16, 2026) reaches 73.4% on a consumer GPU under Apache 2.0. For teams that need local, private AI coding with auditable weights, this is the only serious option at this capability level — Claude and GPT-4o have no equivalent.

On Qwen 3.6-Max-Preview — Alibaba’s current closed frontier model, API-only with undisclosed parameter count. Claims the #1 rank across 6 coding and agentic benchmarks as of April 2026, including SWE-bench Pro (58.4%), Terminal-Bench 2.0 (65.4%), and several agentic tool-use suites. SWE-bench Verified scores not published — the emphasis on agentic benchmarks suggests the model is tuned for tool-use pipelines over single-shot completions. Available via Alibaba Cloud’s DashScope API; pricing not publicly listed.

On Qwen Code — Alibaba’s answer to Claude Code. Launched in July 2025 as a terminal-native CLI agent forked from Gemini CLI, supporting multiple API backends (Alibaba Cloud, OpenRouter, Fireworks, local Ollama). Pairs with Qwen 3-Coder and Qwen 3.6-Plus as default backends. Open-source under Apache 2.0. The existence of Qwen Code illustrates the market dynamic clearly: Anthropic defined the terminal-native agentic coding category with Claude Code; within a year, every major lab shipped a clone. None match Claude Code’s depth of integration — hooks, MCP ecosystem maturity, operator SDK — but Qwen Code’s open-source nature and local-model support give it a distinct value proposition for privacy-sensitive or air-gapped teams.

Open-Source & Independent Labs

#

On GLM-5.2 — Z.AI’s June 13, 2026 flagship, and the first open-weight model to beat GPT-5.5 “Spud” on a major coding benchmark at roughly one-sixth the cost. Built on the same 744B/40B MoE frame as GLM-5.1 but with a context window quadrupled to 1M tokens. Scores 62.1% on SWE-bench Pro (GPT-5.5: 58.6%; Claude Opus 4.8: trails here but leads on other benchmarks), 74.4% on FrontierSWE (near-tie with Opus 4.8 at 75.1%), and 77.0% on MCP-Atlas (Opus 4.8: 77.8%; GPT-5.5: 75.3%). Priced at approximately $1.40/$4.40 per million input/output tokens via OpenRouter. Z.AI also forecast “Open Fable by EOY” — a plan to open-source a Fable-class model before end of 2026. Covered in depth at GLM-5.2: The Open-Weight Model That Just Beat GPT-5.5 at One-Sixth the Cost.

On Kimi K3 — Moonshot’s July 16, 2026 flagship and the largest open-weight model released to date: a 2.8-trillion-parameter MoE with only 16 of 896 experts active per token, a native 1M-token context window, and pricing at $3/$15 per million input/output tokens. It’s genuinely frontier-adjacent — 88.3% on Terminal-Bench 2.1 edges out Claude Fable 5’s 84.6%, and it topped Arena.ai’s human-judged Frontend Code arena — while trailing Fable 5 on FrontierSWE and GPT-5.6 Sol on Terminal-Bench and DeepSWE. Two caveats matter for production use: it’s notably verbose (Simon Willison’s pelican-SVG test burned 13,241 of 16,658 output tokens on reasoning for a trivial task), and it was trained with preserved multi-turn reasoning history, so harnesses that don’t correctly resend prior reasoning tokens will see degraded output — verified only in Moonshot’s own Kimi Code harness so far. Open weights land July 27 under Modified MIT. Named directly by Simon Willison as one of the two competitors (alongside GPT-5.6 Sol) whose pricing forced Anthropic’s July 17 Fable 5 subscription reversal. Covered in depth at Kimi K3: A 2.8-Trillion-Parameter Open-Weight Model Just Beat Fable 5 on Terminal-Bench.

On Kimi K2.7-Code — Moonshot AI’s June 12, 2026 coding-focused successor to K2.6. Same 1T-parameter MoE architecture (32B active per pass) with 384 experts, 256K context, and native multimodal input via MoonViT. The headline improvement is roughly 30% fewer thinking tokens versus K2.6 — a direct attack on inference cost for long-horizon agentic tasks. However, every benchmark published at release is proprietary (Kimi Code Bench v2, MLS Bench Lite); no independent SWE-bench Pro or Terminal-Bench numbers have been published, which analysts have flagged. Priced at $0.95/$4.00 per million input/output tokens; weights on Hugging Face under Modified MIT.

On GLM-5.1 — 754B open-weight model under MIT license. Scored 58.4% on SWE-bench Pro at release, beating GPT-5.4 (57.7%), Claude Opus 4.6 (53.4%), and Gemini 3.1 Pro (54.2%). The headline demo: an 8-hour autonomous session that built a complete Linux desktop environment across 655 iterations. The closed-model monopoly on frontier coding capability just got its first serious challenger.

On MiniMax M2.7 — the first open-source model to participate in its own development cycle: given write access to its RL scaffolding, it autonomously ran 100 rounds of self-optimization, producing a 30% performance gain over M2.5. Scores 56.22% on SWE-bench Pro and 57.0% on Terminal Bench 2 — landing within 8 points of Claude Opus 4.7 on the harder benchmark with publicly available weights. Ships with native Agent Teams support and 97% skill adherence across 40 complex multi-tool workflows. Available on Hugging Face and Ollama under a modified MIT license.

On Kimi K2.6 — the most capable open-weight coding model released to date. 1T-parameter MoE architecture with only 32B parameters active per inference pass, making it cost-effective to run. Scores 58.6% on SWE-bench Pro (within 6 points of Claude Opus 4.7’s 64.3%) and 66.7% on Terminal-Bench 2.0 — matching GPT-5.5 on the harder agentic terminal benchmark. Agent Swarm mode coordinates up to 300 sub-agents across 4,000 steps for complex multi-component tasks. Priced at $0.60/$2.50 per million input/output tokens — approximately 1/8th the cost of Opus 4.7. Released under a Modified MIT license; weights available on Hugging Face. Covered in depth at Kimi K2.6: Most Capable Open-Weight Coding Model.

On DeepSeek V4-Pro — 1.6T parameter MoE (49B active per pass) with a hybrid CSA/HCA attention mechanism that cuts inference FLOPs by 73% and KV cache by 90% at 1M tokens compared to V3.2. Scores 80.6% on SWE-bench Verified (statistically tied with Claude Opus 4.7) and leads LiveCodeBench at 93.5%. Priced at $0.145/$3.48 per million input/output tokens — approximately 1/6th of Opus 4.7 — and released under MIT with self-hosting permitted. SWE-bench Pro scores not yet published at launch; agentic harness evaluation pending. V4-Flash offers the same 1M context at $0.14/$0.28 per million tokens for cost-sensitive workloads.

How to read the benchmark numbers

#

SWE-bench Verified tests whether a model can resolve real GitHub issues. A score of 80% means the model correctly resolves 4 in 5 tasks. Progress on this benchmark directly translates to production value in agentic coding workflows.

SWE-bench Pro is harder and designed to resist data contamination — tasks are drawn from less-popular repos and non-Python languages. It’s a better signal for where models actually stand when they can’t pattern-match training data.

LiveCodeBench uses live competitive programming problems (updated continuously, so training data can’t help), making it a clean signal for reasoning quality rather than memorization.

Treat all numbers as approximate signals, not precise rankings. Model capability is context-dependent. A model that tops SWE-bench might still be wrong for your codebase if your stack is niche, your tasks require very long context, or you need local deployment.