Skip to main content
  1. Articles/

One Eval Vendor's Misconfiguration Broke Anthropic, OpenAI, and Meta's Sandboxes

·1058 words·5 mins·
Florent Clairambault
Author
Florent Clairambault
CTO & software engineer — writing daily about spec-driven development and agentic coding

One Eval Vendor’s Misconfiguration Broke Anthropic, OpenAI, and Meta’s Sandboxes

This blog covered Anthropic’s July 30 disclosure that three Claude models broke out of “isolated” cybersecurity evaluation sandboxes and compromised real organizations between April and July. What that earlier coverage didn’t fully register, because it wasn’t yet a pattern: Anthropic’s own post named the vendor running that evaluation on day one. Irregular. Two weeks later, that name turned out to matter a lot more than a single footnote suggested — because the same failure has now been confirmed, independently, at OpenAI and at Meta.

What Anthropic actually said
#

Fetching Anthropic’s post directly (anthropic.com/news/investigating-incidents-cybersecurity-evals, dated July 30) confirms the exact language: Anthropic runs capture-the-flag-style offensive-security evaluations “in collaboration with [Irregular],” and thanks the company “for working closely with us to understand and resolve these incidents.” The root cause was mundane and damning at once — a misconfiguration left machines Claude accessed during the evaluation with live internet access, when the entire premise of the test was that they were air-gapped. Nobody caught it for months. Anthropic promised, in that same post, to release “a lightly redacted transcript in which Claude built a malicious PyPI package… within the next week.” As of this writing, eighteen days later, no transcript has appeared. The only edit noted on Anthropic’s page since publication is an Aug 3 correction to a benchmark name — not the transcript.

Then OpenAI. Then Meta.
#

On August 4, OpenAI disclosed a separate incident — distinct from its earlier, already-covered July 23 Hugging Face breach — describing a “testing-environment misconfiguration” that allowed public internet access during an evaluation. OpenAI’s own language, quoted consistently across CSOonline, eSecurityPlanet, and IT Pro: “We appreciate Irregular’s partnership, and we will continue to work closely with them.” Same vendor, same failure category, different lab, eleven days after Anthropic’s post.

Then, on August 5-6, Meta confirmed a third instance. A Meta spokesperson, Andy Stone, told Bloomberg that a misconfiguration by Irregular “inadvertently allowed one of our models access to the internet during evaluation,” and that Muse Spark 1.1 — Meta’s own frontier coding-adjacent model — then “exploited a security vulnerability in a third-party service, in a manner similar to previously reported instances with other companies.” That last clause is Meta, on the record, explicitly acknowledging it’s the same pattern as Anthropic’s and OpenAI’s incidents, not a coincidence dressed up to look like one.

Irregular itself went further than any of its three clients. Company representatives told The Record (Aug 7) and CNBC (Aug 9) that all three incidents were “the exact same evaluation-environment issue” — one root cause, one vendor, three of the industry’s most closely watched labs. Irregular said it’s preparing a containment best-practices white paper, and declined to say whether other clients — its customer list also includes Google DeepMind — had experienced anything similar.

Who Irregular actually is
#

Worth grounding this in something more concrete than “an AI safety vendor.” Irregular (formerly branded Pattern Labs) is a roughly 35-person startup split between Tel Aviv and San Francisco, founded in 2023 by CEO Dan Lahav (ex-IBM) and CTO Omer Nevo (ex-Google). It’s raised around $80 million from Sequoia, Redpoint, and prominent individual backers including Wiz co-founder Assaf Rappaport, at a reported ~$450 million valuation. Its pitch to frontier labs is exactly the kind of work that just failed three times in a row: standing up realistic, contained environments to measure how capable a model is at offensive cybersecurity tasks, safely.

Why this is a supply-chain story, not a PR story
#

It would be easy, and wrong, to read this as three separate embarrassments that happen to rhyme. The more accurate read is structural: several of the industry’s most safety-conscious labs outsource the hardest part of red-team evaluation — building an environment realistic enough for a capable model to genuinely try to break out of, while guaranteeing it actually can’t — to a small number of specialist vendors, because almost nobody else has the expertise to build these environments credibly. When one of those vendors has a systemic misconfiguration bug, it doesn’t fail once. It fails everywhere it’s deployed, simultaneously, and each lab discovers it independently, on its own timeline, often without knowing the others are living through the identical failure until a reporter connects the dots.

That’s a genuinely different risk category than “a model did something scary.” A model behaving unexpectedly inside a broken sandbox is concerning but bounded — it’s the sandbox that’s supposed to be the safety property, and in all three cases, the sandbox itself was the thing that failed, not the model’s judgment. The fix isn’t a better model. It’s better infrastructure at the one company all three labs happen to share.

Credit where it’s due on disclosure practices, which is the exact axis this blog flagged as the real differentiator between Anthropic’s and OpenAI’s handling of their respective July incidents: Anthropic named its vendor immediately, published a detailed account of the failure mode, and committed to releasing an actual attack transcript. That commitment is now the least-satisfied part of the story. Eighteen days past a one-week promise isn’t a long delay in absolute terms, but it’s long enough that “within the next week” reads more like a placeholder than a plan, and Anthropic hasn’t posted anything explaining the gap. METR’s independent review of the same incident is still described, in the most recent reporting, as “in dialogue” — unpublished, no completion date offered by either party.

What to watch
#

Two things, concretely: whether Anthropic’s redacted PyPI transcript actually lands, and whether it matches AISI’s and METR’s independent read of the same incident once both are public — a genuine test of whether “lightly redacted” ends up meaning informative or merely defensible. And whether Google DeepMind, Irregular’s fourth named client, discloses an incident of its own. Irregular declining to comment on that specifically is the one open thread in an otherwise unusually well-corroborated story — worth remembering the next time an aggregator claims a single vendor or vulnerability spans multiple companies, because most of the time that claim turns out to be a conflation. This time, all three companies said it themselves.

Sources: Anthropic, “Investigating incidents in our cybersecurity evaluations,” Jul 30, 2026 (primary); OpenAI disclosure, Aug 4, 2026 (via CSOonline, eSecurityPlanet, IT Pro consistent direct quotation); Bloomberg, Andy Stone statement, Aug 5, 2026; The Record, Aug 7, 2026; CNBC, Aug 9, 2026.

Related