Early Anthropic hire, former METR COO have found a way to rein in rogue AI agents; plus, health benefits platform Thatch reaches $1B valuation

This

is TechCrunch.

Innovation has always kept the world moving forward, but it also comes with risk. In a world that needs new energy solutions, AXA XL is facing into risk. As a leading global insurer, we partner with businesses breaking down barriers to accelerate the energy transition.

AXA XL, facing into risk for a future to be imagined. This advertisement doesn't constitute an offer or solicitation nor a description of any products or services of AXA XL.

A day after anthropic researcher Jacob Coxon quit his job over concerns that AI could kill us all by the end of the decade, TechCrunch's Julie Bort met with founders and brothers-in-law Rune Kvist and Rajiv Dettani, and they think they've got a solution that could save us all, or at least help prevent AI agents from going rogue inside enterprises. Kvist, an early Anthropic employee who is also married to Datani's sister, said, AI is getting smarter at an increasingly rapid rate. The surprising thing about AI is that it becomes harder to adopt and harder to control as AI gets smarter, not easier. Datani said, is the former COO of the AI safety research organization, Meter. And the pair launched a startup called Artificial Intelligence Underwriting Company that hopes to bring AI safety to enterprises and companies building AI models and agents. The startup names Cursor, Lovable, Harvey, and Eleven Labs as customers. On Tuesday, AIUC announced a $40 million Series A led by Ribbit Capital with participation from First Harmonic. It previously closed a $15 million seed round from Nat Friedman through the fund NFDG, along with Emergence, Terrain, and Anthropic co-founder Ben Mann, among others, bringing its total funding to $55 million. What caught the attention of this A-list group of investors is AIUC's attempt to apply a familiar cybersecurity model to a new set of AI risks. The company has built a third-party audit and certification layer for AI agents. Kvist said, Banks, hospitals, governments, and militaries no longer decline to deploy AI because a model isn't smart enough. They decline because they've made commitments to their own customers about what a system will and won't do. And nobody can currently guarantee that. Using the widely adopted cybersecurity standard SOC 2 as its muse, AIUC has developed a standard called AIUC1 and a testing service to validate agents against the standard. To build the standard, AIUC assembled a consortium of about 250 security and risk leaders, the buyers of agents. And Detani said, you know, these are the people who we meet with on a monthly basis. And the question we ask them is, when you're buying agents from someone, what would you look for? What are the questions you'd want to ask? And what would you want to see addressed? That feedback shapes the tests. The startup then runs an agent through a suite of some 5,000 tests to see how it behaves in scenarios involving jailbreaks, hallucinations, and data leaks. The results produce a roughly 100-page report detailing where an agent performs safely and reliably and where it does not. Interestingly, AIUC uses AI agents to run the tests and AI to analyze the data. Humans, however, verify the final audit,