VC10X Pulse - Will AI Kill Us All?

What if the biggest warning about AI isn't coming from politicians or activists? Or people who don't understand technology? What if it's coming from people who are actually building it? This week, a researcher at Anthropik, one of the world's most safety-focused AI companies, resigned. His name is Jacob Coxon. And his reason for leaving is terrifying. He says the world's leading AI labs are racing towards self-improving intelligence and that they may be gambling with our lives. And here's the part that should really make you uncomfortable. He doesn't think the labs can simply stop. Because even if Anthropic slows down, OpenAI won't. And if OpenAI slows down, someone else might not. So the question isn't just, is AI getting dangerous? The bigger question is, have we created a race where nobody can afford to lose? Jacob Coxon spent roughly three years doing pre-training research across OpenAI and Anthropic. And this week, he walked away from Anthropic. He publicly explained why on X. His argument was brutally simple. The frontier labs are moving towards increasingly powerful, potentially self-improving AI systems. while the safety mechanisms needed to control those systems aren't keeping pace. Coxon says the people building AI themselves seriously believe that advanced AI could potentially cause catastrophic harm within this decade and yet they keep building. That's the paradox at the heart of this entire industry. The people most worried about AI are often the same people with the most incentive to make it more powerful. But this didn't start with Coxon, and this is where the story gets interesting, because Jacob isn't the first researcher to design a major AI lab over safety concerns. Back in 2024, OpenAI's superalignment leadership suffered a major exodus. Ilya Sutskevar left, Jan Laike left, Daniel Kokotilo left, others followed. Laike famously argued that safety in processes had taken a backseat to product development. Coco Tayo later became one of the voices arguing that frontier AI companies need stronger protections for employees who want to warn the public about AI risks. So there is a pattern here, not necessarily proof that AI is about to destroy humanity, but certainly evidence that some of the people closest to the technology don't believe the current trajectory is comfortable. And that's a signal investors, founders and policymakers should pay attention to. The real problem is the AI arms race. Now let's forget the apocalypse for a moment. Let's think like a VC. Imagine you are running Anthropic. You believe increasingly capable AI could transform science, medicine, software, defense, and the global economy. Your competitor is OpenAI. Google is building, Meta is building, Chinese labs are building. Thousands of startups are building on top of all of them. Now imagine you discover that your next model could be significantly more capable than the previous generation. Do you stop because you're worried about what it might do or do you continue because if you don't build it someone else will? That's the game theory problem and it's brutal because safety becomes a coordination problem. If everyone agrees to slow down great but if you