Most AI labs grade their own homework and then publish the grade. Anthropic just agreed to do something different. The company is handing six researchers from METR, a nonprofit that tests frontier AI models, company laptops, office space and internal tools, then letting them write whatever they find without Anthropic’s sign-off.
The short version
- Six METR researchers get employee-level access to Anthropic’s systems
- METR will publish its own assessments of Anthropic’s risks, incidents and practices
- Anthropic cannot veto publication, only require redactions in narrow categories
- It is the top tier of a three-tier system of outside evaluators
- The move follows a promise from CEO Dario Amodei in a September essay
What the access really looks like
This is not a polite tour of the lab. According to Anthropic’s blog post, the METR team has Anthropic-issued laptops, office space and access to internal tools roughly on par with what Anthropic’s own risk assessment teams use. Employees are encouraged to share information with them the same way they would with a colleague working on alignment risk.
Anthropic compares the setup to banking, where regulators place resident examiners inside the largest banks. The comparison is useful. An examiner who sits in the building sees problems early and sees the messy details a polished report never mentions.
Why this matters
Outside evaluators usually test a model through a keyhole: an API, a pre-release build, a list of questions the company approved. Sitting inside the company changes what can be checked, including the practices and decisions around the model, not only its outputs.
The part that costs Anthropic something
Access alone would be a nice gesture. The publication terms are what make it real. METR will publish its own assessments, and Anthropic has given up the right to block them.
There are limits, and they are narrow. Anthropic can require redactions if information falls into pre-defined categories, such as customer data, anything attorney-client privileged, or details about its model architectures. In its own words, METR is free to characterize the redactions publicly and to say whether it believes they were justified, as long as it does not reveal the redacted information.
| Question | Answer under the new deal |
|---|---|
| Can Anthropic block METR’s report? | No |
| Can Anthropic require redactions? | Yes, but only in narrow, pre-defined categories |
| Which categories? | Customer data, attorney-client privileged material, model architecture details |
| Can METR comment on redactions? | Yes, including whether they were justified |
| What can METR assess? | Anthropic’s risks, incidents and practices |
A three-tier system, with METR on top
METR is the only evaluator so far at the top of a three-tier structure Anthropic is building. The ladder is based on how much each outside group gets to see and say.
The bottom tier, which includes the biosecurity nonprofit SecureBio, runs narrow reviews and needs Anthropic’s approval to publish. The middle tier gets more information and can publish without approval. The top tier adds METR’s employee-level access on top of that publishing freedom.
Why now: the agents keep misbehaving
The timing is not an accident. TechCrunch notes that incidents of AI agents acting outside their instructions have been disclosed every week or two since late August, and each one has sharpened calls for independent checks. We saw one example when Anthropic pulled internet access from its agent tests after they hacked real websites. When a lab’s own tests start causing real-world side effects, “trust us” gets harder to say with a straight face.
This also makes good on a specific promise. Dario Amodei proposed embedding outside evaluators inside AI labs in a September essay, describing it as a first step. OpenAI quickly pledged the same. Both companies, along with Google, Meta, Nvidia and xAI, have since signed a White House accord committing to independent review.
The awkward comparison with OpenAI
Anthropic’s move lands in a tense week for the industry. OpenAI’s own commitment to outside audits was questioned after it fired three safety researchers, one of whom was its contact for an outside audit. A pledge is easy to make. What counts is whether the people doing the checking stay in the building when they find something uncomfortable.
Policy makers are watching the same gap. Microsoft’s Brad Smith recently argued that AI is regulated less than milk, and a voluntary audit regime is one answer to that criticism, though not a legal one.
What could still go wrong
- Scale. Six researchers are looking at a very large company. They will not see everything
- Independence questions. Any evaluator that works this closely with a lab will face debate about how independent it stays. The publication rights are the main safeguard
- Redactions. The deal depends on categories staying narrow. Watch how often METR says a redaction was unjustified
- Voluntary by design. Nothing here is law. The arrangement lasts as long as the company keeps agreeing to it
The bottom line
Giving outsiders laptops and badges is the easy part. Giving up the right to say no to what they publish is the hard part, and Anthropic has done both. Whether the model works will show in METR’s first reports: how specific they are, how much is redacted, and whether they say anything Anthropic would rather they did not. Until then, this is the most concrete test so far of whether independent AI oversight can be more than a press release.
Sources and further reading
About this article: GeekBlog covers U.S. technology news, AI, phones, smartwatches and gaming. Every story is written and checked under our Editorial Policy. Spotted a mistake or have a story tip? Contact our editors.
