What idioms teach us about AI and its evolution

Research into how AI interprets idioms into images and text reveals a lot about the technology - and us, finds Nicole Kobie

- Nicole Kobie

Counting the instances of the letter “r” in “strawberry”. Generating an image to show kicking the bucket, monkey business or another idiom - or no elephants in a room. These are all examples of viral pranks played on large language models (LLMs), and they reveal plenty about how these systems work, when they don’t and how they’re slowly getting better.

When it comes to idioms, LLMs perform much better when sticking with text than they do with images. Ask ChatGPT et al to explain a phrase in text, and you're likely to receive a detailed and even accurate answer.

But it’s still possible to trick Al by making up idioms. On social media, one suggestion was: “You can’t lick a badger twice.” It felt real enough to fool Google’s AI Overview (see “Make your own idiom”, p28), but ChatGPT 40 returns a more sensible result, saying it’s not a common phrase, suggesting a few potential meanings from similar idioms and asking for more information, which is as close as these LLMs get to saying they don’t know.

Understanding LLMs

Tom Pickard, a researcher at the University of Sheffield, studies how AI manages idioms, in part to help develop systems that are better at deciphering the weird figures of speech created by humans. It turns out that idioms are a great way to help people understand how LLMs work, and why they sometimes don’t.

That really comes down to the fact that these systems don’t “understand” our language. In fact, they rip it apart, reducing words and letters into tokens devoid of any meaning. “The tokens, the words are broken up in training, it doesn’t process text, it processes multidimensional vectors and numbers,” Pickard said at the Turing Institute’s AI conference.