Two days after this blog flagged that Cursor’s own Grok 4.5 launch numbers were self-reported and self-tested — including the SWE-bench Pro comparison used to claim “Opus-class” performance — the independent verification actually showed up. Artificial Analysis, the benchmarking outfit that runs every major model through a common evaluation harness rather than trusting vendor press releases, published its own Grok 4.5 analysis on July 9. The results aren’t a debunking. They’re something more useful: a genuinely mixed picture that confirms part of Cursor’s pitch, leaves part of it unverified, and surfaces a real cost nobody put in the launch announcement.
The Part That Holds Up: Grok 4.5 Is Actually Frontier-Class#
On Artificial Analysis’s Intelligence Index — a composite score built from the org’s own evaluation suite, not vendor-submitted numbers — Grok 4.5 scores 54, a 16-point jump over Grok 4.3’s 38. That’s the largest single-generation gain Artificial Analysis has recorded for a SpaceXAI (formerly xAI) model. It lands Grok 4.5 in fourth place among frontier models, behind Claude Fable 5, GPT-5.5, and Claude Opus 4.8, and ahead of every open-weight model and every Google Gemini model currently tracked. On the GDPval-AA v2 agentic evaluation, Grok 4.5 posts an Elo of 1543, sitting between Opus 4.8 (1600) and GLM-5.2 (1513).
That’s a real result, independently arrived at, and it validates the core of Cursor’s positioning even if the marketing chart overstated the specifics: Grok 4.5 is not a cheap also-ran. It’s a legitimate fourth seat at the frontier table, produced by a model lab that didn’t exist as a serious contender in this tier a year ago.
The Part That’s Still Unverified: Coding, Specifically#
Here’s where it gets more complicated than either “Grok 4.5 beats Opus 4.8” or “Grok 4.5 loses badly” — the two framings currently circulating. Artificial Analysis’s Coding Agent Index, built from DeepSWE, Terminal-Bench v2, and SWE-Atlas QnA, scores Grok 4.5 at 76 — tied for third place with GPT-5.5, not leading. That’s a more modest result than the Terminal-Bench 2.1 win Cursor’s launch blog led with, and notably, Artificial Analysis’s coding suite doesn’t include SWE-bench Pro at all — the exact benchmark where Cursor’s own self-reported numbers showed Grok 4.5 trailing Opus 4.8 by 4.5 points and Claude Fable 5 by 15.6 points.
That matters because it means the single most consequential number from the July 8 launch — the SWE-bench Pro gap this blog flagged as the real story buried under the “beats Opus 4.8” headline — still hasn’t been independently reproduced by anyone. Not confirmed, not debunked. Cursor’s self-graded loss on that benchmark remains exactly as uncertain as it was three days ago, just with a different, independently-measured coding score (a tied-for-third finish) sitting next to it as context. Teams evaluating Grok 4.5 for agentic coding work should treat both numbers as provisional until someone runs SWE-bench Pro itself outside of Cursor’s own infrastructure.
The Part Nobody Highlighted: Hallucinations Nearly Doubled#
The finding that got the least attention in Cursor’s own materials is the one Artificial Analysis’s AA-Omniscience knowledge-and-hallucination benchmark surfaced clearly. Grok 4.5’s raw accuracy improved substantially, from 35% to 52% versus Grok 4.3. But its hallucination rate — the rate at which the model states something false with confidence, as opposed to declining to answer — rose from 25% to 54%, more than double the prior generation. The net Omniscience Index score still improved, from 18 to 26, because the benchmark rewards correct answers and penalizes wrong ones without penalizing refusals, so the accuracy gain outweighs the hallucination cost on paper. But “knows more, and is more confident when it’s wrong” is a materially different risk profile for a coding agent than “knows more, full stop” — a hallucinating agent that’s more assertive about incorrect code, incorrect API usage, or fabricated library behavior is arguably more dangerous in production than a cautious one that hedges.
This is a pattern Artificial Analysis notes is common across model generations broadly, not unique to xAI — bigger, more capable models tend to also be more confidently wrong. But it’s a genuinely new data point that didn’t ship with Cursor’s own announcement, and it’s exactly the kind of thing an independent evaluator exists to catch.
The Efficiency Claim Actually Checks Out — With a Caveat#
Cursor’s other headline number was token efficiency: xAI claimed Grok 4.5 resolves SWE-bench Pro tasks using roughly 15,954 output tokens on average, against 67,020 for Opus 4.8 — a 4.2x gap. Artificial Analysis didn’t reproduce that specific SWE-bench Pro number (see above), but it did measure token usage on its own Coding Agent Index tasks, finding Grok 4.5 uses roughly 1.9 million tokens per task against Fable 5’s 7.2 million — a 60% reduction, independently observed, on a different benchmark suite. Directionally consistent with xAI’s claim, even if the exact multiplier and comparison model differ. At $2.59 per task on the Coding Agent Index and $0.31 per task on the Intelligence Index, Grok 4.5 is cheap in a way that’s now been measured by someone other than the company selling it.
What This Actually Changes#
Nothing here reverses the skepticism this blog flagged on July 10 — if anything, it sharpens it. The self-reported SWE-bench Pro loss is still unverified either direction. The “beats Opus 4.8” framing is now more precisely falsifiable: true on Intelligence Index-adjacent measures where Grok 4.5 is genuinely strong, not true on the coding-specific composite where it’s tied for third, and materially incomplete without the hallucination-rate cost that Cursor’s own announcement never mentioned. For a company whose entire pitch is that it trained a coding model on live IDE telemetry no one else has access to, the fact that an outside evaluator’s own coding suite puts it in a three-way tie rather than a clear win is the more important number — and the one worth watching if xAI or Cursor publishes a rebuttal.
The larger lesson holds regardless of how the SWE-bench Pro question eventually resolves: independent, common-harness evaluation caught something (the hallucination spike) that self-reported vendor benchmarks structurally can’t surface, because no lab benchmarks itself on “how often does my model confidently lie.” That’s the argument for treating Artificial Analysis, METR, and similar third parties as load-bearing infrastructure for this entire market, not optional color commentary on launch day.
Sources:
- Artificial Analysis — Grok 4.5 Brings SpaceXAI to the Intelligence Frontier
- Artificial Analysis — Grok 4.5 (high) Model Analysis
- Artificial Analysis — AA-Omniscience: Knowledge and Hallucination Benchmark
- OfficeChai — Grok 4.5 Places Behind Only Anthropic and OpenAI’s Top Models on Artificial Analysis Intelligence Index
- Tech Times — Grok 4.5 Cuts Coding-Agent Cost 80%: Near-Frontier Speed, Higher Hallucinations