'Hallucination' Flaws of AI limit benefits to economy

From helping consultants diagnose cancer to aiding teachers in drawing up lesson plans – and flooding social media with derivative slop – generative artificial intelligence is being adopted at breakneck speed.

- Heather Stewart

Yet a growing number of voices are starting to ask how much of an asset the technology can be to the UK's sluggish economy. Not least because there is no escaping a persistent flaw: large language models (LLMs) remain prone to casually making things up.

It's a phenomenon known as "hallucination". In a recent blogpost, the barrister Tahir Khan cited three cases in which lawyers had used LLMs to formulate legal filings or arguments – only to find they slipped in fictitious supreme court cases, and made up regulations or nonexistent laws.

"Hallucinated legal texts often appear stylistically legitimate, formatted with citations, statutes, and judicial opinions, creating an illusion of credibility that can mislead even experienced legal professionals," he warned.

The tech-sceptic journalist Ed Zitron argued in a blogpost that the tendency of ChatGPT (and other chatbots) to "assert something to be true, when it isn't" meant it was "a non-starter for most business customers, where (obviously) what you write has to be true".

University of Glasgow academics have said that because models are not set up to solve problems or reason, but to predict the most plausible-sounding sentence based on the reams of data they have hoovered up, a better word for their factual hiccups is not "hallucinations" but "bullshit".