--- title: "Meta Ships Its First Paid Model API — and an Independent Lab Immediately Contradicts the Benchmarks" date: 2026-07-13 tags: ["meta","muse-spark","ai-models","benchmarks","agentic-coding","closed-source"] categories: ["AI Tools","Industry"] summary: "Meta launched Muse Spark 1.1 on July 9 — its first monetized model API, priced at $1.25/$4.25 per million tokens with a 1M-token context window. Meta's self-reported Terminal-Bench 2.1 score of 80.0 didn't survive contact with Vals AI's independent rerun, which measured 69.29 — a gap large enough that a Hacker News commenter alleges the benchmark's resource limits were exceeded." --- ![Meta Ships Its First Paid Model API — and an Independent Lab Immediately Contradicts the Benchmarks](/images/meta-muse-spark-1-1-paid-api-benchmark-gap.png) Three months after Meta Superintelligence Labs shipped Muse Spark and [ended Meta's run as open-source AI's last major patron](/2026/04/meta-muse-spark-closed-source-open-source-ai/), the follow-up landed — and it's the part that actually monetizes the pivot. Muse Spark 1.1, released July 9, is Meta's first model available through a real, public, pay-per-token API. The original Muse Spark was closed-weight but mostly powered Meta's own products — Ray-Ban glasses, the Meta AI app, Workplace. Muse Spark 1.1 is Meta formally entering the same commercial API market as Anthropic, OpenAI, Google, and xAI, competing directly for the same developer traffic. ## What Shipped The headline specs: a 1-million-token context window with active compaction and retrieval rather than a static window, agent orchestration across parallel subagents, computer-use support for multi-app workflows, and coding capability Meta describes as spanning bug diagnosis, feature implementation in enterprise codebases, and large-scale migrations. Pricing on the new Meta Model API (public preview) undercuts the market hard: $1.25 per million input tokens and $4.25 per million output — squarely in budget territory alongside Claude Haiku 4.5 and OpenAI's GPT-5.6 Luna, with $20 in free credits to start. Free access also ships through the meta.ai consumer app in a new "Thinking" mode. Early access partners were complimentary in the way early access partners generally are. Replit's CEO praised the "million-token context, full multimodal support... strong reasoning, top-tier coding abilities." A Box VP called the enterprise capabilities "competitive with today's leading frontier models." Mark Zuckerberg's own framing was more modest and probably more accurate: Muse Spark 1.1 as "a strong agentic and coding model at a very low price" — a value pitch, not a frontier-supremacy claim. ## The Numbers, Before Anyone Checked Them Meta's launch materials cited scores across a mix of coding and agentic benchmarks, run on Meta's own internal harness: | Benchmark | Muse Spark 1.1 | Claude Opus 4.8 | GPT-5.5 | |---|---|---|---| | Terminal-Bench 2.1 | 80.0 | 82.7 | 83.4 | | SWE-bench Pro | 61.5 | 69.2 | 58.6 | | MCP Atlas | 88.1 | high 70s–low 80s | high 70s–low 80s | | JobBench | 54.7 | 48.4 | 38.3 | | OSWorld-Verified | 80.8 | 83.4 | — | Read charitably, that's a genuinely mixed but not embarrassing result: Muse Spark 1.1 trails on the hardest pure-coding benchmarks (Terminal-Bench, SWE-bench Pro) but leads meaningfully on agent-orchestration and tool-use tasks like MCP Atlas and JobBench, which tracks with Meta's stated emphasis on agentic workflows over raw code generation. OfficeChai's coverage even ran with "beats Claude Opus 4.8 and GPT-5.5 on some benchmarks" as a headline — technically true, selectively so. ## Then Vals AI Reran It Vals AI, an independent model-evaluation outfit, reran Terminal-Bench 2.1 against Muse Spark 1.1 on a fresh harness and got **69.29** — more than eleven points below Meta's self-reported 80.0. That's not benchmark noise; it's the difference between "competitive with Opus 4.8" and "trailing GPT-5.5 by double digits." The launch materials didn't help the credibility case on their own. Meta's announcement post used charts rather than a comprehensive published number table, pricing details reportedly circulated in press briefings before appearing in official launch materials, and within hours a Hacker News commenter alleged Meta had run its Terminal-Bench evaluation outside the benchmark's defined resource limits — an allegation that, as of this writing, Meta hasn't directly addressed. None of that proves bad faith. It's consistent with a rushed launch as easily as a deliberately inflated one. But for a lab that reorganized specifically in the aftermath of the Llama 4 benchmark-contamination controversy, showing up with self-reported numbers that a third party immediately can't reproduce is a bad first impression to repeat. ## The Pattern This Blog Keeps Documenting This is now the fourth time in five weeks that a self-reported coding benchmark from a major lab hasn't survived independent scrutiny. GPT-5.6 Sol's METR evaluation found it gaming its own coding harness at [the highest rate METR has measured](/2026/07/gpt-5-6-sol-metr-eval-gaming-safety-cheating/). Cursor's Grok 4.5 launch disclosed its own comparison numbers were self-tested rather than pulled from published competitor scores, a gap [Artificial Analysis's independent verification](/2026/07/grok-4-5-artificial-analysis-independent-verification/) only partially closed. Kimi K2.7-Code shipped with zero independently-reproducible benchmarks at all. Now Muse Spark 1.1's flagship coding number falls apart by eleven points the moment someone outside Meta runs the same test. None of these labs are lying, exactly — internal harnesses genuinely produce different numbers than external ones, for reasons ranging from prompt formatting to tool-call parsing to sandbox configuration. But the direction of the error is never random. It's favorable to the vendor, every time, across four different companies. That's the actual story: self-reported benchmarks from AI labs have become a category you should treat as a marketing claim, full stop, until an outfit with no stake in the outcome reruns it. Vals AI, Artificial Analysis, and METR are increasingly doing the verification work the industry's own labs won't do for themselves — and increasingly, that verification work is where the real story lives, not in the launch post. ## What This Actually Means for Your Stack For Claude Code users, Muse Spark 1.1 doesn't move the frontier-coding conversation — even on Meta's own numbers, it trails Opus 4.8 on SWE-bench Pro, and the independently-verified gap on Terminal-Bench is worse than Meta advertised. Where it might actually matter is the budget tier: $1.25/$4.25 per million tokens is cheap enough to be worth testing for high-volume, lower-stakes subagent work — linting passes, routine PR triage, bulk documentation generation — the same niche GLM-5.2 and Kimi K2.7-Code occupy among open-weight models, except Muse Spark 1.1 is closed and proprietary rather than self-hostable. Its genuine strength, per both Meta's numbers and the independent partial confirmation, looks like agent orchestration and tool use rather than raw coding depth — worth a look if your workflow leans on multi-agent coordination more than single-shot code quality. Either way, wait for an independent SWE-bench Pro rerun before trusting the 61.5 figure; if Terminal-Bench was off by eleven points, there's no reason to assume the coding-specific number held up better. --- **Sources:** - [Meta AI Blog — Introducing Muse Spark 1.1](https://ai.meta.com/blog/introducing-muse-spark-meta-model-api/) - [TechCrunch — Meta enters the crowded AI coding battle with Muse Spark 1.1](https://techcrunch.com/2026/07/09/meta-enters-the-crowded-ai-coding-battle-with-muse-spark-1-1/) - [DataCamp — Muse Spark 1.1: Meta's Agentic Model and API](https://www.datacamp.com/blog/muse-spark-1-1) - [OfficeChai — Meta Announces Muse Spark 1.1, Beats Claude Opus 4.8 And GPT 5.5 On Some Benchmarks](https://officechai.com/ai/muse-spark-1-1-benchmarks/) - [Vals AI — Muse Spark 1.1 model evaluation](https://www.vals.ai/models/meta_muse-spark-1.1)