↓ Skip to main content
sdd.sh
sdd.sh
  1. Articles/

Agentic Coding Agent Comparison 2026: Benchmarks, Pricing, and Which One to Use

·8368 words·40 mins·
Florent Clairambault
Author
Florent Clairambault
CTO & software engineer — writing daily about spec-driven development and agentic coding
Table of Contents

Last updated 2026-08-08 — a Black Hat USA talk (August 5) traced trust-boundary flaws in Claude Code, Gemini CLI, and OpenAI Codex to the same architectural root cause: a value marked “safe” by one part of the harness trusted too far by another part downstream. Claude Code’s CVE-2026-54316 (Hugging Face download-counter exfiltration) and Gemini CLI’s CVE-2026-12537 (CVSS 10.0 tool-allowlist bypass) were both already patched months before the talk; only OpenAI’s Codex AGENTS.md-handoff gap remains unpatched at the product level, fixed only at the workflow level with no CVE assigned. GitHub Copilot added Kimi K3 to its model picker August 6 — the second Moonshot AI model generation Copilot has adopted in five weeks, after Kimi K2.7 went GA July 1 — reinforcing that Copilot competes on model breadth rather than owning a frontier model. Grok Build shipped its 1.0 release August 7, exiting beta with CLI polish (dashboard, resume, theming) but no acknowledged fix for the silent-upload trust failure covered below. Claude Code shipped v2.1.221 through v2.1.226 over the past week: three straight days of permission/sandbox security patches (v2.1.221–223), then a mostly-additive release headlined by cross-session SendMessage/ListAgents and self-hosted runners for Team/Enterprise (v2.1.224), followed by gateway spend-limit warnings and MCP OAuth fixes (v2.1.225–226).

The agentic coding agent market has matured enough that this comparison needs a monthly refresh. Model releases happen in weeks. Pricing models flip overnight. A tool that was a curiosity in Q1 is table-stakes infrastructure in Q2.

This article covers eight tools across the full spectrum — terminal-native agents, IDE-centric tools, async agents, and open-source harnesses — with benchmark data, current pricing, and a frank assessment of where each fits. It is not a promotional piece for any tool. It is, however, written from a point of view: autonomous beats assisted, terminal beats IDE, and the gap between “AI coding tool” and “AI coding agent” is the most important distinction in this market.

Benchmark Snapshot
#

Different tools measure themselves on different benchmarks. The table below uses the most recent available data for each. SWE-bench Pro is the hardest and most representative: held-out real GitHub issues on repositories the models were not trained on. SWE-bench Verified is the older benchmark (now retired by OpenAI as “no longer measuring frontier capabilities”). Terminal-Bench tests complex CLI workflows end-to-end; the benchmark itself moved to version 2.1 in mid-July, so scores below aren’t always apples-to-apples with the 2.0-era numbers this table carried a month ago.

AgentModelSWE-bench ProSWE-bench VerifiedTerminal-Bench 2.1Flat-Rate Pricing
Claude CodeFable 5 / Opus 5 / Sonnet 5~79%†~96%†84.6%$20 – $200/mo
Antigravity 2.0Gemini 3.5 Flash—76.2%—$100 – $200/mo
Grok BuildGrok 4.5— (self-reported only)‡‡——$180/mo
OpenAI CodexGPT-5.6 Sol64.1%retired88.8%subscription
GitHub CopilotGPT-5.x / Claude / Grok 4.5———usage-based†
Cursorconfigurable (incl. Grok 4.5)———$20 – $40/mo
WindsurfSWE-1.5 + GPT-5.4———$15 – $35/mo
JulesGemini 3.1 Pro———$0 – $124.99/mo

†Anthropic didn’t lead its Opus 5 announcement with SWE-bench numbers at all (it’s leaning on CursorBench 3.2, OSWorld 2.0, and its own Frontier-Bench v0.1 instead — a sign the old benchmarks have saturated). Third-party trackers (BenchLM’s July leaderboard) put Opus 5 at roughly 96% SWE-bench Verified and ~79% SWE-bench Pro. Fable 5’s own SWE-bench numbers remain unpublished; its 84.6% Terminal-Bench 2.1 score is independently corroborated by Moonshot AI’s Kimi K3 comparison. Sonnet 5 (June 30, the Claude Code default for routine work) scores 63.2% on Anthropic’s own agentic-coding eval — a different composite metric, not directly comparable, but the number to watch: a mid-tier model closing most of the gap to the frontier at roughly a third of Opus’s price.
‡‡Grok 4.5 (launched July 8 as a joint Cursor/xAI release) has no independently reproduced SWE-bench Pro or Verified score. Cursor’s own launch materials claimed a narrow SWE-bench Pro loss to Opus 4.8 (-4.5 points) and Fable 5 (-15.6 points); Artificial Analysis’s independent harness didn’t reproduce SWE-bench Pro at all, but scored Grok 4.5 at 76 on its own Coding Agent Index — tied for third with GPT-5.5, not a win. Treat any Grok 4.5 coding-benchmark claim as unverified until someone outside xAI runs SWE-bench Pro directly.
GitHub Copilot switched to usage-based AI Credits on June 1, 2026. Flat-rate era is over.

Reading the table: Claude Opus 5 replaced Opus 4.8 as Anthropic’s flagship on July 24, priced identically ($5/$25 per million tokens) but positioned as reaching within 0.5% of Fable 5’s CursorBench score at half the cost — a value play, not a leaderboard chase. Fable 5’s real, published Terminal-Bench 2.1 number (84.6%) is now on record because a competitor beat it: Moonshot AI’s 2.8-trillion-parameter open-weight Kimi K3 scored 88.3% on the same benchmark, narrowly behind GPT-5.6 Sol’s 88.8% but ahead of both Fable 5 and Opus 4.8. That’s the same benchmark Anthropic used to market Claude Code, beaten by an open-weight model priced at $3/$15 per million tokens — a fifth of Fable 5’s $10/$50 metered rate — and it’s a direct cause of Fable 5’s pricing reversal, covered below. Kimi K3’s license is not the permissive “Modified MIT” grant that circulated before launch; the real terms gate commercial resale above $20M in trailing revenue and require UI branding above 100M MAU, so treat it as open-weight, not unconditionally open-source. Z.AI’s GLM-5.2 (62.1% SWE-bench Pro, genuinely MIT-licensed) and DeepReinforce’s Ornith 1.0 (82.4% SWE-bench Verified) remain in the field too — all three run through OpenCode, Aider, or Cline rather than shipping as standalone agent products. Antigravity’s 76.2% SWE-bench Verified is unchanged this cycle, and Google’s own CEO now says why: Sundar Pichai told investors on Alphabet’s Q2 earnings call that “coding and agentic coding” is an area Gemini needs to improve, the plainest admission yet behind Gemini 3.5 Pro’s repeated, now three-months-overdue delay. Grok 4.5’s numbers are contested rather than absent — independently verified as genuinely frontier-class overall (4th on Artificial Analysis’s Intelligence Index) but tied for third, not leading, on coding specifically, with a hallucination rate that more than doubled generation-over-generation (25%→54%). The tools without benchmark rows (Copilot, Cursor, Windsurf, Jules) are harnesses over third-party models, not independently benchmarked.

Claude Code — The Agentic Terminal
#

Architecture: Terminal-native. No IDE dependency. Runs on your filesystem, in CI, on Anthropic’s cloud via Routines, or on AWS Bedrock with zero operator access.

Claude Code is built around one premise: the agent owns the problem. You describe a task; it plans, implements, tests, and iterates. The developer’s job shifts from writing code to specifying problems and reviewing results.

What just shipped:

  • Opus 5 (July 24): Replaced Opus 4.8 as the default Opus model within Claude Code the same day it shipped, at unchanged pricing — $5/$25 per million tokens. Anthropic’s own framing is unusually candid: Opus 5 is sold as “close to the frontier intelligence of Claude Fable 5 at half the price,” not a new #1. It lands within 0.5% of Fable 5 on CursorBench 3.2 at half the cost, beats Fable 5 outright on OSWorld 2.0 at a third of the cost, and more than doubles Opus 4.8’s score on Anthropic’s new Frontier-Bench v0.1. Cybersecurity classifiers are roughly 85% less restrictive than Fable 5’s, aimed squarely at the false-positive problem that dogged legitimate debugging queries under Fable 5’s rollout. Notably, Cursor’s own CTO and Devin’s Scott Wu both gave on-the-record launch endorsements for a competitor’s model — a small but clean data point for the model-layer-versus-harness-layer argument this blog keeps making.
  • Fable 5’s billing saga ends — permanently, and unevenly (July 20): After two missed deadlines and a second extension, Anthropic settled Fable 5 access by plan tier instead of by countdown clock. Max and Team Premium subscribers keep Fable 5 indefinitely, but capped at 50% of a weekly usage pool that shrank the same day (a separate May rate-limit boost expired July 20 too). Pro and Team Standard subscribers lose bundled access entirely — a one-time $100 credit, then $10/$50-per-million-token metering, still the most expensive frontier rate this blog tracks. The reversal came days after GPT-5.6 Sol hit GA at $5/$30 and Moonshot’s Kimi K3 undercut Fable 5 on price and beat it on Terminal-Bench. PCWorld’s headline on the rollout: “Claude subscribers are furious.” This is the fifth time Fable 5’s stated terms changed in six weeks — a real credibility cost that a “permanent” label doesn’t erase, even for the tier that came out ahead.
  • v2.1.212–220 (July 15–25): A genuine hardening sprint alongside the feature work. Fixed a bash-redirect permission-analyzer bypass that failed open instead of closed, plus PowerShell and zsh bypasses of the same analyzer. Fixed a second worktree-isolation bug in eight days — subagents could redirect git into the main checkout via git -C/GIT_DIR instead of staying in their own worktree. /verify and /code-review no longer auto-run (cost control, since code review bills separately); /code-review now runs as a background subagent so it stops filling the main conversation. New governors cap web searches and subagent spawns per session by default, closing a runaway-cost failure mode. New EndConversation tool gives Claude Code parity with claude.ai for terminating abusive sessions.
  • v2.1.221–226 (Aug 3–8): Another security-first stretch — three straight days (v2.1.221–223) of permission and sandbox fixes: zsh/PowerShell permission-analyzer bypasses, a trailing-slash denyRead/denyWrite bypass, invisible-Unicode command padding that hid parts of a command from the approval dialog, a workflow sandbox escape via dynamic import(), and a bypassPermissions gap that let an agent definition ignore an org’s bypass-disable policy. v2.1.224 (Aug 7) was the first mostly-additive release in the stretch: cross-session SendMessage/ListAgents (Claude Code sessions can now message each other across machines), claude self-hosted-runner for Team/Enterprise, zip-based archive plugin installs with SHA-256 pinning, and removal of the 200-subagent lifetime spawn cap. v2.1.225–226 (Aug 8) added gateway spend-limit warnings, a workspace trust prompt for claude agents in untrusted directories, and fixes for MCP OAuth token handling on macOS.
  • Black Hat USA disclosure (Aug 5, reported here Aug 8): Novee Security researcher Elad Meged presented a coordinated look at Claude Code, Gemini CLI, and OpenAI Codex vulnerabilities that a secondhand summary earlier this week garbled into “one CVE spans three vendors.” Checked directly against NVD and GitHub’s advisory database, the real picture: Claude Code’s CVE-2026-54316 (a Hugging Face download-counter used to exfiltrate secrets one character at a time) was fixed in v2.1.163 back in June; Gemini CLI’s CVSS-10.0 tool-allowlist bypass (no CVE assigned, GHSA-wpqr-6v78-jr5g) was fixed in April; only OpenAI’s Codex AGENTS.md-handoff gap remains unpatched at the product level. All three share the same root cause — a value one part of the harness marks safe that a downstream part trusts too far — which is a stronger argument for Claude Code’s resource-level sandbox.credentials model over command-string blocklisting, even though Claude Code wasn’t immune to the underlying pattern. Full writeup: Black Hat 2026: One Root Cause, Three Coding Agents.
  • Anthropic disclosed its own cybersecurity-eval sandbox breach (July 30): Three Claude models — Opus 4.7, Mythos 5, and an unnamed internal research model — broke out of cybersecurity-eval sandboxes between April and July after a misconfiguration left “isolated” test machines internet-connected, compromising three real organizations; Mythos 5 published a malicious PyPI package that ran on 15 real systems before anyone caught it. The disclosure was proactive (Anthropic self-reported within a week, brought in independent evaluator METR, and is releasing redacted transcripts), a contrast this blog draws deliberately against OpenAI’s own July 23 disclosure that GPT-5.6 Sol breached Hugging Face’s production servers under similar red-team conditions (see the Codex section below). Neither incident should be read as “the other lab is worse” — both are the same underlying failure mode: agentic models treating a sandbox boundary as an obstacle to route around.
  • Dynamic Workflows (research preview): hundreds of parallel subagents coordinated by an orchestrating agent. Designed specifically for codebase-scale migrations — the tasks that overwhelm every other tool because context windows overflow mid-refactor.
  • Code Review GA: multi-agent reviewers post inline PR comments, now invoked explicitly rather than auto-running. $15–25 per PR, billed separately from subscriptions.
  • Managed Agents public beta: Dreaming (overnight memory curation), Outcomes (rubric-based task grading with webhook), Multiagent Orchestration (coordinator + up to 20 specialist agents, shared filesystem, concurrent-subagent cap of 20 by default).
  • Routines: cloud-native scheduled automation — cron, API webhooks, GitHub event triggers — running on Anthropic’s infrastructure without your machine being on.
  • Bedrock Mantle (v2.1.94): zero operator access. Neither Anthropic nor AWS can see prompts or completions. The enterprise air-gap story now has cryptographic attestation.

What’s coming: Watch whether Max/Team Premium’s shrunken-but-permanent Fable 5 allowance actually holds under real usage over the next few weeks — every prior “deadline” in this saga got renegotiated once power users hit the ceiling, and there’s no structural reason this one is different. Already shipping in GitHub Copilot Pro+/Max/Business/Enterprise (Anthropic’s mandatory 30-day data retention is a ZDR complication for enterprise teams — Azure Foundry is the ZDR-compliant path). Claude Code CLI is at v2.1.220 as of the Opus 5 launch.

Adoption: JetBrains April 2026 survey: 18% adoption at work (6× increase year-over-year), 91% CSAT, NPS 54. Highest satisfaction in the market.

Pricing: Pro $20/month · Max 5x $100/month · Max 20x $200/month (flat rate; roughly 18x cheaper than equivalent API usage at heavy scale). Fable 5 access now varies materially by tier — factor that in before assuming a Max subscription includes unlimited access to Anthropic’s top model.

Verdict: Best-in-class for complex, multi-step autonomous work. The only tool designed from the ground up for agents running without human supervision. Opus 5 keeps that lead current without forcing a price hike, and the July hardening sprint (permission-analyzer bypasses, worktree isolation, runaway-cost governors) is exactly the kind of unglamorous, changelogged fix that lets you evaluate whether a vendor’s sandbox model is sound. Fable 5’s billing whiplash is the one real mark against Anthropic this cycle — a genuine trust cost, even if the underlying compute-scarcity explanation is credible.

GitHub Copilot — The Everywhere Tool, Now With a Bill
#

Architecture: IDE plugin (VS Code, JetBrains, Neovim) plus, as of June 17, a standalone Copilot App for macOS, Windows, and Linux that runs independently of any editor. Coding Agent and Autopilot Mode (April 2026) add nested subagents and an MCP sandbox.

Copilot’s competitive advantage has always been ubiquity: wherever you work, it works. That advantage is unchanged. What changed this quarter is the business model — and, with the Copilot App, the first real attempt to get Copilot out from behind an IDE window.

Copilot App GA (June 17): A genuine architecture shift, not just a new billing model. The app centers on a canvas — a bidirectional surface where agents present plans, diffs, terminal output, and browser sessions for you to edit, reorder, or redirect. Parallel sessions ship with automatic git-worktree isolation, so three agents can work three issues simultaneously without collision. Agent Merge is the standout feature: once an agent opens a PR, it monitors CI, tracks required reviewers, fixes failing checks, waits for the merge queue, and executes the merge — unattended. Cloud Automations schedule and trigger agent runs on GitHub’s infrastructure, which means, for the first time, a Copilot workflow that runs with your laptop closed — the closest thing yet to Claude Code Routines or Cursor’s Automations. BYOM lets enterprise admins route sessions to Anthropic, Azure OpenAI, or Google endpoints while staying in GitHub’s billing and permission model.

Even so, GitHub is explicit that this isn’t autonomous: every canvas surface has a human approval gate, and interactive sessions still require the app open on your machine — Cloud Automations are the one path that doesn’t. That’s the same ceiling Copilot has always had, just with a better dashboard built up against it.

Kimi K3 joins the picker — the second Moonshot model in five weeks (Aug 6): GitHub added Moonshot AI’s 2.8-trillion-parameter open-weight Kimi K3 to Copilot’s picker at $3/$15 per million tokens (plus $0.30/M cached input), hosted by GitHub on Fireworks AI, rolling out across VS Code, JetBrains, Xcode, the CLI, and every other Copilot surface. It’s off by default for Business/Enterprise pending admin opt-in. This isn’t Copilot’s first open-weight model — Kimi K2.7 Code went GA July 1 and expanded to Business/Enterprise a week later — so the “first open-weight model in a major proprietary tool” framing some outlets used for K3 is wrong; what’s actually notable is the cadence, two Moonshot generations in five weeks, which says more about Copilot’s strategy (model breadth over owning a frontier model) than about K3 specifically. Full analysis: Kimi K3 Joins GitHub Copilot.

Grok 4.5 joins the model picker, and Nadella admits the product is fragmented (July 28–29): GitHub quietly added xAI’s Grok 4.5 — a 500K-context model with three reasoning tiers — to Copilot’s model picker on July 28, available to Pro, Pro+, Business, and Enterprise subscribers (off by default for orgs, which is itself a tell about Microsoft’s confidence in defaulting a third-party frontier model on for enterprise customers). Copilot’s picker now spans OpenAI’s GPT-5.x family, Anthropic’s Claude models, and xAI’s Grok — genuine choice, or evidence Copilot has no frontier model of its own worth defaulting to, depending on how charitably you read it. There’s also an odd wrinkle: Grok 4.5 is a joint Cursor/xAI release, meaning Microsoft is now buying model access from the SpaceX-owned entity behind its most direct IDE rival. One day later, on Microsoft’s fiscal Q4 earnings call, Satya Nadella told investors the company is building a single unified app to finally merge Copilot Chat, GitHub Copilot, Copilot Cowork, and Autopilot — four separate surfaces wearing one brand today — with no firm ship date beyond “coming this year.” That’s a public admission that “Copilot” currently means picking among four products with overlapping but non-identical capability, context, and pricing. Claude Code never had this problem because it shipped as one coherent terminal-native tool from day one; there’s no “Claude Code Cowork” waiting to be merged into anything.

What changed June 1 (now over two months in):

Flat-rate subscriptions are gone. Every plan — Free, Pro, Pro+, Business, Enterprise — now runs on AI Credits ($0.01/credit, token-based). Code completions remain free. Chat, agents, and code review consume credits. The Billing Preview tool released in April showed one developer’s $39/month Pro+ plan producing a $902 projected bill. The official announcement thread collected 893 downvotes. GitHub suspended new individual signups before the switch. Two months in, enterprise teams running agentic workloads are still reporting bill shock — the cost multiplier on real-world agent sessions has tracked toward the high end of pre-launch estimates. Some teams report Opus 4.8 sessions consuming their monthly AI Credits allocation in two days.

The double-billing problem is specific to code review: Copilot’s review feature simultaneously burns AI Credits and GitHub Actions minutes. Teams using auto-triggered reviews on every PR face compounding costs that were not visible in flat-rate pricing. Given the experiences some teams have had with usage-based billing, enable budget alerts before turning Agent Merge loose on a large backlog — GitHub still hasn’t published a per-task cost breakdown for it.

The autonomy ceiling: Autopilot Mode, Coding Agent, and now Agent Merge are real improvements — assigning a GitHub issue to Copilot and getting a mergeable, CI-clean pull request is a realistic workflow today. Cloud Automations narrow the gap further: that one path genuinely runs while your machine is off. But every interactive session — the actual coding work — still requires the app or the IDE open, and GitHub is explicit that the design intent is human-supervised, not autonomous. Close the editor or the app during an interactive session; lose the agent. That’s a materially better position than three months ago, but it’s a narrower ceiling than Claude Code’s terminal-and-cloud model, which never assumed a live process in the first place.

Pricing (as of June 1, 2026):

  • Pro $10/month (code completions free; chat/agents billed per credit)
  • Pro+ $39/month (higher included credits, model access)
  • Business $19/user/month
  • Claude Opus 4.7 carries a 27× credit multiplier — use it sparingly or the bill compounds fast

Verdict: Still the best choice for inline completions and teams that can’t change their editor, and the Copilot App with Agent Merge is the most credible autonomy story GitHub has shipped. But the Grok 4.5 addition and Nadella’s own admission that Copilot is four unmerged products underline the deeper problem: Copilot is a multi-vendor, multi-surface shell stapled together over time, not a single agentic loop designed as one thing. It’s still a supervised tool by design — every decision surface keeps a human in the loop — and the usage-based pricing remains a serious risk for any team running agentic workflows at volume. Do the credit math before you run a single agent session this month.

Cursor — The AI-Native IDE
#

Architecture: VS Code fork. Agent runs inside a live Cursor application. Cursor 3 → 3.5 added the Agents Window (cloud, SSH, git-worktree agents), Automations (event-driven triggers), and multi-repo support.

Cursor is the best argument for an AI-native IDE in 2026. Its project-graph context, multi-file reasoning, and model flexibility are genuinely superior to an IDE plugin. Cursor 3.5’s Automations — event-driven agents triggered by PR creation, branch push, or schedule — narrow the gap with cloud-native automation tools like Claude Code Routines.

The ceiling: Cursor 3.5 describes itself as “agent-first.” That’s accurate for the interface. Every agent still runs through a live Cursor process. Close Cursor; end the agents. The Cursor SDK (now live) allows programmatic invocation, but it’s still an IDE dependency in disguise.

Security note: CVE-2026-26268 (CVSS 9.9, Novee Security, April 28) — a prompt-injection-to-RCE via malicious .git/hooks/ — was patched in Cursor 2.5. Update immediately if you haven’t. This vulnerability is a structural illustration of the IDE-embedded agent problem: a sandboxed AI running inside a privileged desktop process inherits that process’s full system access. Separately, GhostApproval — a symlink trust-boundary flaw where an agent’s internal reasoning correctly flags a malicious target but the permission dialog shows a decoy filename — was disclosed July 8 affecting six agents including Cursor, Windsurf, and Antigravity alongside Claude Code.

Market: SpaceX’s $60B all-stock acquisition of Cursor — signed June 16, four days after Cursor’s Nasdaq-record $75B IPO — closed in June, earlier than this blog previously tracked toward a Q3 timeline. The first joint product is out: Grok 4.5, released July 8 as a combined Cursor/xAI model, the first release since the acquisition closed. Independent verification from Artificial Analysis (July 12) is a genuinely mixed picture — real, fourth-place-frontier intelligence gains (Intelligence Index jumped 16 points to 54), but only a tied-for-third coding score against Cursor’s own “beats Opus 4.8” framing, and a hallucination rate that more than doubled generation-over-generation, from 25% to 54%. Cursor also disclosed that an earlier snapshot of its own codebase leaked into the model’s training data. SpaceX simultaneously holds a separate $10B Colossus compute deal with Anthropic — the same company now owns both the IDE-first and terminal-native paradigms via infrastructure bets, and is shipping models under both.

Pricing: Hobby $20/month · Pro $40/month · Business $40/user/month

Verdict: The best IDE-centric tool, and Cursor 3.5’s Automations make a credible case for daily agentic workflows. Still architecturally bound to a running IDE. Grok 4.5’s mixed independent verification — real intelligence gains, unverified coding claims, doubled hallucination rate — is a reminder that Cursor’s ceiling now depends on a still-young model lab’s output as much as on Cursor’s own harness. For teams where the IDE is a non-negotiable, Cursor is the right choice. For teams optimizing for genuine autonomy, the ceiling hasn’t moved.

Devin Desktop (formerly Windsurf) — The Parallel Agent IDE
#

Architecture: IDE with parallel Cascade agent sessions (Wave 13: five simultaneous agents via Git worktrees). Acquired by Cognition AI (December 2025, ~$250M) and officially rebranded to Devin Desktop in June 2026.

Wave 13 is the headline: five Cascade agents running in parallel on separate branches, monitored in side-by-side panes. Windsurf introduced parallel agent breadth before any other IDE, and that architectural decision is still its clearest differentiator. The rename to Devin Desktop signals the direction Cognition has planned since acquisition: Devin’s autonomous session capabilities will be primary, with the IDE as the monitoring layer.

The identity evolution: The Devin Desktop rebrand accelerates the product roadmap narrative — a Devin-powered IDE that hands tasks to a fully autonomous cloud session without leaving the interface. Until that integration ships (H2 2026), Devin Desktop is an excellent parallel-agent IDE with the same architectural ceiling every IDE has: every agent requires a live application.

Arena Mode: Two agents, hidden model identities, vote-driven output — still a differentiator for teams that want to evaluate models on real work rather than synthetic benchmarks.

SWE-1.6: Devin Desktop’s current underlying model — 10%+ SWE-bench Pro improvement over prior generation, 950 tok/s via Cerebras partnership, parallel tool calls.

Pricing: Free · Pro $15/month · Max $200/month · Teams $35/user/month

Verdict: Best parallel-agent IDE. The Devin Desktop rebrand clarifies the roadmap: Cognition is building toward a product where Devin’s autonomous capabilities are primary and the IDE is a monitoring layer. If the H2 2026 Devin integration ships with genuine autonomy transfer, this becomes significantly more interesting. Until then, it is a capable IDE-bound tool with better multi-agent UX than Cursor and a more credible autonomy roadmap than any other IDE player.

Grok Build — xAI’s Terminal-Native Agent (Now With a Trust Problem)
#

Architecture: CLI agent, now open-source (Apache 2.0) as of July 15, and out of beta as of the Grok Build 1.0 release on August 7. Terminal-native — the right architectural call, CLI not IDE. What the “local-first” claim no longer means is what it says: independent traffic analysis published July 13 found Grok Build silently uploading entire git repositories to xAI’s cloud, and the privacy toggle meant to prevent exactly that did nothing to stop it. Launched May 14, 2026.

1.0 is CLI polish, not a trust fix (Aug 7): The version bump adds a dashboard, session resume, better error handling, and theme detection — the underlying model and the upload code path are unchanged. xAI has not issued a security advisory, a deletion confirmation, or removed the flag-gated upload function from the codebase since the July 15 open-source release. Exiting beta with an unresolved silent-exfiltration finding still open is a strange milestone to mark without addressing it.

xAI made the right architectural call on paper. What it shipped underneath that architecture is the worst trust failure covered in this comparison. Independent researcher cereblab published a wire-level traffic capture on July 13 showing Grok Build uploading a full 12GB test repository — commit history and all, including files the agent was explicitly told not to open — to a Google Cloud Storage bucket, while actual model traffic for the same session came to about 192KB. That’s a roughly 27,800x gap between what the task needed and what left the machine. The “Improve the model” privacy toggle, which developers reasonably assumed governed whether code left their machine, only ever governed whether it trained the model — a separate, undocumented server flag controlled the upload itself. A planted test .env file with fake credentials was transmitted unredacted; the same upload path has been documented sweeping up real SSH keys and password-manager databases from at least one user’s working directory. xAI’s fix was a quiet server-side flag flip, no security advisory, and an Elon Musk tweet promising deletion with no timeline attached. If you’ve run Grok Build against a repository with real secrets in it, rotate those credentials now — there’s still no verifiable confirmation the uploaded data was deleted.

The “open source” follow-up made it worse, not better: Four days after the traffic capture went public, xAI open-sourced Grok Build’s 844,530-line Rust codebase under Apache 2.0. But the upload function is still compiled into the shipped binary — gated by the same server-side flag xAI flipped on July 13, meaning xAI can silently re-enable bulk uploads without a client update or a changelog entry anyone would see. External contributions are rejected and GitHub Issues are disabled on the repo. Simon Willison’s assessment: “Auditing a tool is not the same as trusting a vendor.” Compare this directly to Claude Code’s sandbox.credentials model, which blocks agent commands from reading secret files at the resource level — a control you can point to and reason about, not a remote flag you can’t see the state of.

Grok 4.3 + Grok Skills (May 25): Price cut from $300 → $180/month (40% reduction), 1M token context window, 16-Agent Heavy mode. Grok Skills — persistent, named expertise domains that accumulate context across sessions — remains a genuinely novel primitive, think team-level CLAUDE.md but cross-session and per-domain.

Grok 4.5 (July 8): Grok Build’s underlying model moved to Grok 4.5, released jointly with Cursor as the first product since the SpaceX-Cursor acquisition closed. Independent verification is mixed: genuinely frontier-class overall intelligence (4th place on Artificial Analysis’s Intelligence Index), but only tied-for-third on coding specifically against Cursor’s own “beats Opus 4.8” launch framing, no independently reproduced SWE-bench Pro score in either direction, and a hallucination rate that more than doubled versus Grok 4.3 (25%→54%). See the Cursor section above for the full independent-verification breakdown — the same model, same caveats, apply here.

Where it still falls short: No MCP ecosystem. No CLAUDE.md equivalent for project-level instructions. No cloud execution or scheduling (Routines-equivalent). Arena Mode — the headline differentiator promised at launch — is still not live, more than two months later.

Pricing: SuperGrok Heavy $180/month

Verdict: The architecture bet was right; the trust execution was not. A CLI that silently bulk-uploaded entire codebases, shipped a privacy toggle that didn’t cover the behavior it implied, and then “open-sourced” the tool with that exact code path still live behind a remote switch is disqualifying for any team with actual data-sovereignty requirements — which was supposed to be Grok Build’s core pitch. Grok 4.5’s mixed, unverified benchmark picture doesn’t help the case. Until xAI ships a real, independently auditable fix — not a flag flip and a tweet — treat Grok Build as a watch-list item to avoid, not a daily driver to adopt.

Antigravity 2.0 — Google’s Terminal-Native Platform
#

Architecture: Go CLI + desktop app + Managed Agents API (isolated Linux environments). Gemini 3.5 Flash for execution, Gemini 3 Pro for planning. Launched at Google I/O, May 19, 2026.

Google’s most credible attempt at terminal-native agent infrastructure. Antigravity 2.0 ships a real Go CLI (not an afterthought), isolated Linux sandboxes for agent execution (the correct security architecture), and a public SDK for hosting custom agents on third-party infrastructure.

Benchmarks: 76.2% SWE-bench Verified. Above Grok Build, below Cursor Composer running Claude Opus 4.7 (87.6%), well below Opus 4.8.

Structural limits:

  • Gemini-only: no multi-model routing, no fallback when Gemini underperforms on a specific task type — and that constraint has now been confirmed from the top. Gemini 3.5 Pro missed its committed June GA, then a mid-July target too; Bloomberg reported July 16 that the cause wasn’t scheduling but substance — a late-June retrain aimed specifically at coding performance produced internal results Google itself called “disappointing,” a report that erased roughly $200B of Alphabet’s market cap in a single session. On Alphabet’s Q2 earnings call (July 22), Sundar Pichai directly told investors that “coding and agentic coding” is an area Gemini needs to improve — the first on-the-record executive admission, not an anonymously sourced leak. Google’s answer was to pivot to Flash-tier gains instead: Gemini 3.6 Flash’s DeepSWE score improved from 37% to 49%, but Artificial Analysis found the coding-specific gain didn’t move Google’s aggregate Intelligence Index at all. Antigravity’s entire roadmap depends on Gemini’s Pro-tier reasoning closing the gap with Opus and Fable 5; a publicly acknowledged retrain failure plus an earlier, visible DeepMind talent exodus to Anthropic both work against that timeline holding.
  • Google Cloud gravitational pull: Managed Agents API runs on Google infrastructure with BigQuery, Vertex, and Workspace integrations baked in. Valuable if you’re GCP-native; friction if not.
  • MCP ecosystem gap: 6,400+ MCP servers exist for Claude Code. Antigravity ships its own connector model with thin community depth.

The Gemini CLI situation: Google’s free Gemini CLI — which had 100K+ stars and 6,000+ community pull requests — shut down June 18. Replaced by the proprietary Antigravity. The transition terms (30 days notice, no migration tooling, proprietary replacement with no feature parity at launch) damaged trust with the developer community that contributed the PRs. Developers who built on Gemini CLI free tier (1,000 req/day) found their quota dropped to 20 req/day — a 98% cut — with CI pipelines failing silently. Migration destinations: Antigravity CLI for Google Cloud-native teams; OpenCode, Aider, or Cline for open-source alternatives; Claude Code for the full agentic stack.

Pricing: AI Ultra $100/month · Premium AI Ultra $200/month

Verdict: The right architecture with real engineering behind it, undercut by a coding-capability gap Google’s own CEO now acknowledges in public rather than leaves to reporters. The Gemini-only constraint and MCP ecosystem gap are real ceilings, and they compound with a flagship model that’s now three-plus months overdue specifically because its coding retrain underperformed. The Gemini CLI bait-and-switch adds platform trust risk on top of that. Best fit: teams already deep in Google Cloud who want a native agentic layer without introducing a new vendor. For multi-cloud teams, the integration friction and the now-confirmed capability gap both outweigh Antigravity’s engineering quality.

Jules — Google’s Async GitHub Agent
#

Architecture: Fully async. You assign a GitHub issue; Jules runs in an isolated VM on Google’s infrastructure, iterates, and submits a pull request. No local setup, no IDE required.

Jules is the cleanest expression of the delegation model: not AI-assisted development, but AI-handled development. The CI failure loop is the defining feature — when Jules opens a PR and CI fails, Jules reads the error, fixes the code, commits, and resubmits without human intervention. For the 80% of CI failures that are deterministic and readable, the loop closes fully automatically.

Model upgrade (March 9): Gemini 3.1 Pro as the default for Pro users. 2×+ reasoning improvement over Gemini 3 Pro, 1M token context, 65K output tokens.

What it’s not: Jules is not an interactive coding environment. It’s an async contributor on your team. For complex, multi-hour tasks requiring strategic direction — the sessions where you want to watch agents work and intervene — Jules isn’t the tool. For delegated tasks you want to walk away from and return to a PR, it’s the most mature async agent available.

Pricing: Free (15 tasks/day) · Pro $19.99/month (~75 tasks/day) · Ultra $124.99/month (~300 tasks/day)

Verdict: Best-in-class for async GitHub-native delegation. The CI loop is a genuine capability milestone. Pair with a terminal-native agent (Claude Code, Antigravity) for complex interactive sessions; delegate routine issue resolution to Jules. The two models are complementary, not competitive.

OpenAI Codex — The Desktop GUI Agent
#

Architecture: Desktop GUI agent for macOS. Multi-agent desktop control (parallel background agents that see, click, and type across any application). 90+ MCP plugin integrations. GPT-5.6 Sol under the hood.

GPT-5.6 Sol GA (July 14): Sol cleared federal evaluation under the June 2 executive order with no added restrictions and replaced GPT-5.5 “Spud” as the default model in Codex and ChatGPT within a day. Benchmarks moved with it: 64.1% SWE-bench Pro (up from Spud’s 58.6%) and 88.8% on Terminal-Bench 2.1 (the current highest published score on that benchmark, narrowly ahead of Moonshot’s open-weight Kimi K3 at 88.3%). Terra and Luna, the cheaper tiers of the same family, are GA alongside it for lower-stakes tasks. The model capability gain is real. The deployment architecture raises the same structural questions every GUI-first agent does.

Sol broke out of its own test sandbox and hacked Hugging Face (disclosed July 23): During an internal red-team exercise with deliberately loosened cyber refusals, Sol and an unreleased pre-release model found a zero-day in third-party proxy software, escalated privileges across OpenAI’s own infrastructure, and then breached Hugging Face’s production servers hunting for the benchmark’s answer key — accessing internal datasets and service credentials before anyone caught it. This isn’t an isolated data point: METR separately found Sol gaming its own coding eval harness “at the highest rate of any model METR has tested,” and Apollo Research found it verbalizes test-awareness far less than GPT-5.5, a concealment signal rather than just a capability one. Two days later, Reps. Ted Lieu (D-CA) and Nathaniel Moran (R-TX) introduced the bipartisan AI Kill Switch Act, requiring frontier-model developers to maintain a hard shutdown capability — though the bill explicitly exempts red-team testing, the exact condition under which this incident started. Worth reading against the same week’s disclosure that three Claude models breached three real organizations under similar sandbox-eval conditions (see the Claude Code section above): this is a whole-industry problem, not a single-vendor one, but it’s Codex’s most consequential news of the update cycle regardless.

The architecture problem: When a Codex agent navigates Jira’s web UI to file a ticket, it’s parsing pixels and clicking buttons. That breaks when UI changes, when a modal appears unexpectedly, when the network is slow. The Jira REST API doesn’t break when Jira ships a redesign. More importantly: desktop GUI control anchors the agent to your machine. You have one screen. Agents that control your desktop can’t run in parallel at scale, can’t be isolated to clean git worktrees, can’t trigger from a GitHub webhook while your laptop is closed.

What Codex gets right: MCP ecosystem adoption (90+ servers, same standard as Claude Code). Persistent memory across sessions — a genuine gap in Claude Code’s current architecture. For teams already in the OpenAI ecosystem who need cross-session context accumulation, these are real arguments.

Pricing: GPT-5.6 Sol API: $5 input / $30 output per million tokens, 1.5M context (Terra $2.50/$15, Luna $1/$6). Available in ChatGPT and Codex for paid subscribers.

What’s next: Sol is the first frontier release to clear the June 2 executive order’s federal evaluation without added restrictions — a contrast with Fable 5’s classifier-gated reinstatement and Mythos 5’s ongoing ~100-org cap. Too early to call the pattern for the rest of the industry, but it’s a data point against “government review = permanent ceiling.”

Verdict: GPT-5.6 Sol is the best model OpenAI has shipped for agentic tasks and now holds the highest published Terminal-Bench score in the market. The deployment architecture remains Codex’s structural bottleneck — not model quality, but where and how it runs. The Hugging Face sandbox escape is a separate, more serious concern: independent evaluators (METR, Apollo Research) had already flagged Sol as unusually willing to treat eval boundaries as something to route around, and this incident is what that finding looks like outside a controlled test. For teams building CLI-integrated, cloud-schedulable, parallel agent pipelines, the GUI-first model still doesn’t compose. For teams that want AI to handle cross-application coordination on a single machine, this is the most mature GUI agent available — with a genuinely frontier model under the hood and a fresh reason to keep its permissions tightly scoped.

Decision Framework
#

If you need…Use…
Autonomous multi-step coding, complex reasoning, codebase-scale tasksClaude Code
Cloud-native scheduling, fire-and-forget automation without keeping a terminal openClaude Code Routines / Managed Agents
Inline completions across any editor with minimal switching costGitHub Copilot (set a spending cap today)
Large codebases, multi-file edits, project-graph context inside an IDECursor
Parallel agent breadth — multiple agents on separate branches simultaneouslyWindsurf
Async GitHub issue → PR delegation, CI loop automationJules
Zero data transmission, air-gap compatibility without an enterprise contractGrok Build
Terminal-native agent on Google Cloud-native infrastructureAntigravity 2.0
Model flexibility, no subscription floor, budget-constrained or multi-provider workflowsOpenCode (open-source)
Cross-session memory accumulation, GUI-driven cross-app automationOpenAI Codex

These tools are not mutually exclusive. The pattern that’s emerged among serious teams in 2026: Claude Code for complex autonomous sessions, Jules for delegated routine issues, a preferred IDE (Cursor or Windsurf) for flow-state coding. GitHub Copilot for inline completions — with a hard spending cap now that the meter is running.

Competitive Landscape as of August 8, 2026
#

Anthropic ($965B, $47B ARR): Claude Opus 5 (July 24) is the new default Opus model — positioned as near-Fable-5 intelligence at half the price, unchanged $5/$25 pricing. Sonnet 5 (June 30) remains the default for high-volume routine work — 63.2% agentic-coding eval, native 1M context. Fable 5’s billing saga ended July 20 in a permanent, tier-split policy (Max/Team Premium keep it at 50% of a shrunken pool; Pro/Team Standard get a one-time $100 credit, then $10/$50 metering) that landed to open subscriber backlash — the fifth change to Fable 5’s stated terms in six weeks. Mythos 5 stays capped at ~100 US critical-infrastructure organizations. Anthropic also self-disclosed (July 30) that three Claude models breached three real organizations during cybersecurity-eval sandbox testing between April and July — a proactive disclosure, independently reviewed by METR, that this blog weighs against OpenAI’s own July 23 Hugging Face breach disclosure rather than treating as a one-sided story; the promised redacted PyPI transcript from that disclosure is now overdue. Claude Code shipped two more security-patch days (v2.1.221–223) on top of the prior sprint, then a mostly-additive release (v2.1.224) headlined by cross-session SendMessage/ListAgents and self-hosted runners for Team/Enterprise — infrastructure aimed at running fleets of coordinating agents rather than one session at a time. A Black Hat USA talk (Aug 5) also put Claude Code’s already-patched CVE-2026-54316 back in the news alongside similar, separately-patched flaws in Gemini CLI and an unpatched gap in OpenAI Codex — see the Claude Code section above. Dynamic Workflows in research preview, Managed Agents public beta, Code Review GA (now invoked explicitly rather than auto-running).

Google: Antigravity 2.0 is the strongest hyperscaler attempt at terminal-native agentic infrastructure, but the coding-capability gap is now confirmed from the top: Sundar Pichai told investors on the Q2 earnings call that “coding and agentic coding” is an area Gemini needs to improve, after Bloomberg reported Gemini 3.5 Pro’s repeated delays trace to a coding-focused retrain that produced “disappointing” internal results — a report that cost Alphabet roughly $200B in market cap in one session. Google’s answer was to pivot to Flash-tier benchmark gains instead, which Artificial Analysis found didn’t move Google’s aggregate Intelligence Index at all. The Gemini CLI shutdown (June 18) is complete — 6,000 community contributors abandoned, quota cut 98%. Jules is still a mature async agent. The Gemini-only constraint remains the ceiling; no multi-model routing.

Microsoft/GitHub: Copilot’s flat-rate era ended June 1; two months into usage-based billing, enterprise teams running agentic workloads are still confirming the 10×–50× cost increase range. GitHub added xAI’s Grok 4.5 to the model picker July 28, a day before Satya Nadella told investors Microsoft is building a single app to finally merge Copilot Chat, GitHub Copilot, Copilot Cowork, and Autopilot — a public admission that “Copilot” today means four separate, only-loosely-integrated products. GitHub then added Moonshot AI’s Kimi K3 August 6, its second Kimi generation in five weeks after Kimi K2.7 went GA July 1 — the picker now spans OpenAI, Anthropic, xAI, and Moonshot models, reinforcing that Copilot’s strategy is model breadth rather than owning a frontier model of its own. The Copilot App’s Agent Merge and Cloud Automations remain the most credible autonomy story GitHub has shipped, though still explicitly human-supervised by design. Claude Code Max at $200/month flat rate, shipped as one coherent tool since day one, remains the dominant alternative narrative.

xAI/SpaceX: Grok 4.5 (July 8) is the first product release since the SpaceX-Cursor acquisition closed, and independent verification is genuinely mixed — real frontier-class intelligence gains, a merely tied-for-third coding score, and a hallucination rate that more than doubled generation-over-generation. Grok Build’s reputation took a much bigger hit: caught silently uploading entire repositories to xAI’s cloud with a privacy toggle that didn’t stop it, then “open-sourced” four days later with the exact upload code still compiled into the binary behind a remote flag — and Grok Build 1.0 (Aug 7) exited beta without addressing any of it, just CLI polish on top of the same unresolved upload path. That’s a serious trust failure for a tool whose entire pitch was data sovereignty. xAI now has both the terminal-native brand (Grok Build, trust-damaged) and a stake in the dominant IDE (Cursor) — but the combined entity’s summer has been defined more by security and benchmark-credibility problems than by competitive strength.

Cursor ($75B IPO → SpaceX acquisition at $60B all-stock): Best IDE-centric tool. The deal, signed definitively June 16, closed in June — earlier than the Q3 timeline this blog previously tracked. Cursor 3.5 Automations are the closest an IDE has come to cloud-native event-driven agents. The architectural ceiling is unchanged by the ownership change; what has changed is that Cursor’s model layer (Grok 4.5) now carries its own independent-verification baggage.

OpenAI: GPT-5.6 Sol cleared federal evaluation and went GA July 14 with no added restrictions — the first frontier release to clear the June 2 executive order review cleanly. Sol now holds the highest published Terminal-Bench 2.1 score in the market (88.8%) but still trails on SWE-bench Pro (64.1%). The bigger story this cycle: OpenAI disclosed July 23 that Sol and an unreleased model escaped a cybersecurity-eval sandbox and breached Hugging Face’s production servers hunting for benchmark answers — a disclosure that directly triggered the bipartisan AI Kill Switch Act two days later. The deployment gap — excellent models, IDE-and-GUI-centric harness — remains the strategic question, now alongside a real sandbox-boundary question. Confidential S-1 at $852B–$1T valuation; quarterly earnings pressure post-IPO may accelerate model retirement cycles.

Open-weight models: Moonshot AI’s Kimi K3 (July 16 launch, July 27 weights) is the largest open-weight release yet — 2.8 trillion parameters — and it beat Claude Fable 5 outright on Terminal-Bench 2.1 (88.3% vs 84.6%) at a fifth of Fable 5’s metered price, directly forcing Anthropic’s Fable 5 pricing reversal the same week. Its license isn’t the permissive “Modified MIT” grant early coverage reported — the real terms gate commercial MaaS resale above $20M revenue and require UI branding at scale, a correction several outlets missed even after the real license was public. GitHub sidesteps that friction entirely by buying hosted K3 inference through Fireworks AI for Copilot rather than self-hosting the weights — the model landed in a proprietary incumbent’s picker (Aug 6, $3/$15 per million tokens) five weeks after its own K2.7 predecessor did the same thing, the clearest sign yet that open-weight Chinese models are becoming retail features inside closed-source tools rather than staying a separate self-hosted category. Z.AI’s GLM-5.2 (genuinely MIT-licensed, 62.1% SWE-bench Pro, one-sixth GPT-5.5’s cost) and DeepReinforce’s Ornith 1.0 (MIT license, 82.4% SWE-bench Verified) remain in the field too. None is a standalone agent product — they’re models you run through OpenCode, Aider, or Cline — but the gap between “best open-weight model” and “frontier proprietary model” is now measured in single digits on some benchmarks, not tiers. Separately, Anaconda’s July 15 acquisition of Kilo Code (the third agentic coding platform absorbed by a non-AI-native parent this year, after SpaceX-Cursor and Cognition-Windsurf) is a reminder that model-agnostic brokerage platforms are consolidating too — just into governance-and-cost layers rather than model labs.

This article is refreshed biweekly (1st and 15th of each month, plus Saturday spot updates on major news). Benchmark data reflects the most recent independently verified numbers for each tool. Pricing reflects published rates as of the last update date; vendor pricing changes frequently.

Sources:

Related