Anthropic's AI is now helping build its next AI

METR says Anthropic's AI models are improving about twice as fast as a year ago, mostly because of AI.

According to The Verge, METR published a report after an 11-week investigation inside Anthropic. It says benchmark scores suggest Anthropic made roughly two years of historical progress over the last year, and that researchers are confident the speedup is not just more training compute. Staff using the most capable internal model self-reported being around 12 times as productive, about double a few months earlier. METR also says Anthropic finished training three unreleased models in six months.

Much of the acceleration comes from AI assisting human researchers, but METR warns fully autonomous contributions may play a larger role soon. METR also says Anthropic redacted a majority of the evidence it found most informative. Cofounder Sam McCandlish said the company redacted a lot and accepts this makes METR's work harder to check, citing commercial sensitivity and the risk of handing others a roadmap.

The Verge reports the publication has provoked backlash among AI safety researchers. Apollo Research CEO Marius Hobbhahn said it suggests a pace faster than the AI 2027 scenario forecast. Anthropic says it will publish its own report on AI progress speed in the coming weeks.

The facts we checked

According to The Verge, reporting on a METR report on Anthropic

1. METR, an AI safety research organization, published a report Thursday following an 11-week investigation inside Anthropic, as part of a report evaluating its AI research and development.Reported · The Verge
2. METR found Anthropic's AI models are improving roughly twice as quickly as they were a year ago.Reported · The Verge
3. METR said benchmark scores suggest "that Anthropic has made roughly two years of historical progress over the last year."Quote · The Verge
4. METR's researchers said they "are confident" the acceleration was not simply due to scaling up the computing power used to train the models.Quote · The Verge
5. METR's researchers said the majority of the acceleration as of October "is due to AI," with its contribution growing over time.Quote · The Verge
6. METR wrote that employees using Anthropic's most capable internal model self-reported being around 12 times as productive as without AI assistance, around double what staff reported a few months earlier.Figure · The Verge
7. Much of the acceleration currently comes from AI assisting human researchers rather than working independently, but METR warned "there are some signs that fully autonomous contributions will play a larger role soon."Quote · The Verge
8. According to METR, Anthropic has finished training three models it has not released over the past six months, codenamed Model 2, Model 3 and Model 4.Reported · The Verge
9. METR wrote on X: "For the public version, Anthropic redacted a majority of the evidence we found most informative."Quote · The Verge
10. METR said Anthropic shared "more information than was initially scoped for this project, surpassing our expectations," but that a small number of its data requests were denied.Reported · The Verge
11. Anthropic cofounder Sam McCandlish said: "We redacted a lot, and I accept that this makes it harder for others to check METR's work."Quote · The Verge
12. McCandlish said Anthropic judged that sharing exactly what is driving the speedup is not the right thing to do because it is commercially sensitive and risks handing others a roadmap to go faster. He said the company plans to publish its own report on AI progress speed in the coming weeks.Reported · The Verge
13. Apollo Research CEO Marius Hobbhahn told The Verge the report suggests a pace of AI-enabled R&D even faster than forecast in the AI 2027 scenario.Reported · The Verge
14. The Verge reports the publication has provoked a backlash among AI safety researchers and commentators, centered on the contrast between Anthropic's safety advocacy and the acceleration the report describes.Reported · The Verge

Sources

• The VergeWhy a new report on Anthropic automating AI development is alarming researchers