How do we know how smart AI really is?

Maths questions. Silly word puzzles. Counting the letter "r" in a sentence. Nicole Kobie reveals how we're trying to work out exactly how intelligent AI is

- Nicole Kobie

As AI becomes increasingly competent at answering emails, making funny pictures and solving complicated science problems that have long stumped us humans, it raises a question: how smart is it really? And we're not sure how to answer that yet.

The goal of companies such as OpenAI isn't to ease the lives of office workers - though that draws investors such as Microsoft, hence the sudden focus on productivity tools - but to build artificial general intelligence (AGI). This is defined in a multitude of ways, but OpenAI describes it as "highly autonomous systems that outperform humans at most economically valuable work".

Alongside AGI, we have ideas such as human-level AI, expert AI and superintelligent Al. All have slightly different definitions depending on who you listen to, but the point is to create a machine that can do what humans can, before moving well beyond what we can do. (There's also a side idea of whether AI is sentient, but that's a whole other problem.) Now, to be clear, we don't yet have AGI and we may never be technically capable of building it - even OpenAI CEO Sam Altman has said another breakthrough in AI is likely required before AGI could become possible.

Semantics and timelines aside, how do we know if AI is as smart as us? The Turing test is one long-running technique for rating machine intelligence, but it's now fallen by the wayside due to its limited focus on language and conversation. Academic exams are used as benchmarks, to see if AI can reason and apply knowledge like a college student. But perhaps we need new ways to quiz our future AI overlords and a few are in the works, including the dramatically named "Humanity's Last Exam".

Turing then and now

Alan Turing laid out the idea for what is now known as the Turing test in a 1950 paper, calling it the "imitation game".