If you ship code with Claude Code, you are betting that the model behind it is what its maker says it is. Until now, the evidence for that bet came mostly from Anthropic itself. On October 15, Anthropic announced a change: it has started giving a team at METR, the nonprofit that evaluates frontier models, “employee-like access” to its models, safety practices and risk management.
It matters more than most model-release news this month. Here is what is actually known, what is not, and what developers should take from it.
What was announced#
According to coverage of Anthropic’s announcement, the arrangement looks like this:
- Who: six METR researchers, per Security Magazine’s report.
- What they can evaluate: Anthropic’s models, safety practices and risk management, not just final outputs.
- Publication: METR publishes its own analysis. Anthropic has no veto, but it may require redactions in “predefined categories.”
- Others in the pipeline: secondary coverage says Anthropic is also working with SecureBio and Accenture on evaluations and is in early talks with other groups, including Hugging Face. I could not confirm those names against a primary page this session, so treat them as reported, not verified.
- Tiers: the same coverage describes a tiered evaluator framework in which METR is the only group with employee-like access so far. Details of the tiers are secondhand.
The announcement sits on Anthropic’s news page with an Oct 15 date. I could not retrieve the full text, so the specifics above come from press coverage.
Why “employee-like” is the interesting part#
Most external evaluation of AI models has been a black-box exercise: call the API, run a test suite, write a report. Zach Stein-Perlman’s 2024 LessWrong post on external evaluators laid out the problem. Access was shallow, often limited to a safety-tuned API. Red-teaming ran about ten hours per person, while real model evaluation takes expert teams with something like ten thousand hours of accumulated test-suite work. Labs cited engineering burden, leak risk and unclear commitments. That post is two years old, but it is a useful baseline for why this announcement is a step up.
Employee-like access changes what an evaluator can see. An outside party who can inspect how a system reached an answer can find things a reviewer checking final outputs cannot. That is the claim of Darin Fredde of Ridge Security in Security Magazine’s piece, and it matches how engineering reviews work. Nobody audits a codebase by running the app and looking at the screen.
The critique worth taking seriously#
Jacob Krell of Suzu Labs is the sharpest critic quoted: “Six people cannot audit a frontier AI lab.” His argument is about volume. Employee-level access comes with employee-level volume of material, and six people also have to design their own evaluations. He points out that Anthropic’s own bank-examiner analogy cuts against it. Large banks host resident examiners, rotating staff and decades of examination process. Here, six embedded researchers review one company’s output. He calls it a pilot, and says the real test is whether Anthropic scales embedded headcount as capability grows.
I think that is the right frame. Access is necessary, not sufficient. As one analysis summarized in the coverage puts it, expanded access does not guarantee evaluators ask the right questions. That depends on expertise, independence, resources and, crucially, publication rights.
That last point is where the details matter. “No veto, but redactions in predefined categories” is only as strong as the categories. If they are narrow (say, security-sensitive exploit details or unreleased product names), the arrangement has teeth. If they stretch, a redaction can hollow out a finding. The categories were not specified in anything I could retrieve. Whoever reads METR’s first report should check the redaction log before the conclusions.
Competitors: unverified#
One aggregator claims Sam Altman called the idea “a great idea” and said OpenAI “will do the same.” I could not corroborate that, and I would not repeat it as fact. What I can say from the coverage is that no equivalent embedded-access arrangement at this scale has been announced by another lab. If OpenAI or Google DeepMind follow, the pattern becomes an industry norm. If they don’t, Anthropic has set a bar its rivals will be asked about.
What this means if you build on Claude#
Three practical takeaways, none of them dramatic:
- Treat it as a trust input, not a guarantee. A published METR analysis will be more useful than a system card alone, because the author is independent. Until it lands, nothing about model behavior has changed.
- Watch the first report. The things worth looking for are the scope METR actually had, what was redacted, and whether the findings include uncomfortable ones. A clean bill of health on day one would be less credible than a report that names real issues.
- Keep your own evals. No lab-level audit replaces a regression suite on your codebase. This is the same lesson as the silent effort-default change earlier this year: vendor behavior moves, and your tests are the only thing that notices.
The take#
Anthropic has spent years arguing that safety and capability are compatible. Letting outsiders look behind the curtain, with the right to publish, is the first move in a while that tests that claim instead of asserting it. It is also small: six people, one lab, a pilot. Credit it for what it is, and hold Anthropic to scaling it.
For a company that has just filed its S-1 and will soon answer to public-market investors, independent verification is also good business. Quarterly pressure and safety review will pull in different directions at some point. External evaluators with publication rights are one of the few mechanisms that could make that tension visible.
