Nick Bostrom: How Should We Navigate Superintelligence?

If we do develop this safely, then human life expectancy could go up hugely. If you imagine people having the same mortality rate as, like, a twenty-year-old, our life expectancy would maybe be, you know, twelve hundred, fourteen hundred years or so. On the other hand, if the risk of AI goes down too slow, then you also don't wanna wait because, like, we are just dying in the meantime, and you lose more expected life years from waiting than you gain from having the risk. So, so it requires a sort of a rightly balanced intermediate case to get sort of longer wait periods of many years.

[upbeat music] Nick Bostrom spent decades thinking about superintelligence before it became part of the mainstream AI debate. Now that increasingly capable systems are actually here, how has his thinking changed? In this episode, Nick joins Theo Jaffee and Sofia Puccini on MGS to revisit AI alignment, existential risk, and the debate over whether frontier development should slow down. Nick explains why he's sympathetic to pacing the frontier, preserving the ability to slow down when the risks warrant it, but is more skeptical of cause for a broad or indefinite pause. And he argues that any calculation about risk also has to account for what delaying advanced AI could cost, including potential breakthroughs in medicine and longevity. They also tackle some bigger questions, whether current AI systems might already have subjective experiences, how open source changes the balance between concentrated power and dangerous capabilities, and whether the upside of superintelligence could be far stranger and far greater than we currently have the imagination to understand.

All right. We are back. We are live with Nick Bostrom, who was a professor at Oxford, founding director of the Future of Humanity Institute. He's now founder and principal researcher of the Macrostrategy Research Initiative. You may know him as the author of "Superintelligence," of "Deep Utopia," of a lot of work that coined or framed the ideas that we take for granted today in the discourse, existential risk, the simulation hypothesis, the vulnerable world hypothesis, so much more. Nick, this has been in the works for a long time, so we're thrilled to have you on. Welcome to MGS.

I'm happy to be here.

So-

Yeah. So where should we start?

[chuckles] I guess we could start with what's your opinion on Trump calling AI superintelligence? How does it-

[chuckles]

What are your thoughts on this?

Uh, it's a, it's a strange time we're living in. Um, we'll see if it catches on. Traditionally, AI has been the larger the umbrella term, right? You could have a little simple expert system from the nineteen seventies, that would be AI. It's kind of a stretch to use the word superintelligence for all of that. Uh, in, in my terminology, superintelligence has been this thing farther down the road where you really have machines that radically outperform us in all cognitive domains. But, um, I mean, you could, I guess, look at some of the current achievements, you know, solving all these math problems and so forth. And, uh, I mean, it's not completely ridiculous to start calling s- at least some of these, uh, performances