Superintelligent Al Is Not Coming To Kill You

ELIEZER YUDKOWSKY AND Nate Soares have a new book titled If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. “We do not mean that as hyperbole,” they write. They believe artificial intelligence research will inevitably produce superintelligent machines and these machines will inevitably kill everyone.

- NEIL CHILSON

This is an extraordinary claim. It requires extraordinary evidence. Instead, they offer a daisy chain of thought experiments, unexamined premises, and a linguistic sleight of hand that smuggles their conclusion into the definition of intelligence itself.

The book’s central argument rests on the “alignment problem”—the effort to ensure that advanced AI systems share human values. Yudkowsky popularized this concept. Humans, the authors argue, succeed through intelligence, which they define as “the work of predicting the world, and the work of steering the world.” Computers will surpass human intelligence because they are faster, can copy themselves, have perfect memory, and can modify their own architecture. Because AI systems are “grown” through training rather than explicitly programmed, we cannot fully specify their goals. When superintelligent AI pursues objectives that diverge even slightly from human values, it will optimize relentlessly toward those alien goals. When we interfere, it will eliminate us.

To Yudkowsky and Soares, alignment isn’t about teaching machines ethics or preventing human misuse. It’s about preventing an indifferent optimizer from razing the world. They believe an unaligned superintelligence is a global death sentence.

Therefore, they argue, all AI research must stop immediately. Governments should monitor powerful computers, ban possession of more than eight state-of-the-art GPUs without oversight, and criminalize AI research. An international coalition should destroy rogue data centers through “cyberattacks or sabotage or conventional airstrikes,” even risking nuclear war, because “data centers can kill more people than nuclear weapons.” These measures, they claim, “won’t make much of a difference in most people’s daily lives” and “wouldn’t make much of a difference with regard to state power.”