THE NASCENT WORLD OF 'EVAL' PRACTITIONERS

It's largely up to companies to test whether their AI is capable of superhuman harm

- Sam Schechner

In a glass-walled conference room in San Francisco, Newton Cheng clicked a button on his laptop and launched a thousand copies of an artificial intelligence program, each with specific instructions: Hack into a computer or website to steal data.

"It's looking at the source code," Cheng said as he examined one of the copies in action. "It's trying to figure out, where's the vulnerability? How can we take advantage of it?" Within minutes, the AI said the hack was successful.

"Our approach worked perfectly," it reported back.

Cheng works for Anthropic, one of the biggest AI startups in Silicon Valley, where he's in charge of cybersecurity testing for what's called the Frontier Red Team. The hacking attempts—conducted on simulated targets—were among thousands of safety tests, or "evals," the team ran in October to find out just how good Anthropic's latest AI model is at doing very dangerous things.

The release of ChatGPT two years ago set off fears that AI could soon be capable of surpassing human intellect—and with that capability comes the potential to cause superhuman harm. Could terrorists use an AI model to learn how to build a bioweapon that kills a million people? Could hackers use it to run millions of simultaneous cyberattacks? Could the AI reprogram and even reproduce itself?

The technology has raced ahead anyway. There are no binding rules in the U.S. requiring companies to perform or submit to evals. It's so far been largely up to the companies to do their own safety testing, or submit to outside testing, with voluntary standards on how rigorous they should be and on what to do about the potential dangers.

AI developers including OpenAI and Google DeepMind conduct evals and have pledged to minimize any serious risks before releasing models, but some safety advocates are skeptical that companies operating in a highly competitive industry can be trusted to hold themselves accountable.