‘Are you joking, mate?’ AI doesn’t get it

IN 2018, my Australian coworker asked me, “Hey, how are you going?”. My response - “I am taking a bus” — was met with a smirk.

- ADITYA JOSHI

I had recently moved to Australia. Despite studying English for more than 20 years, it took me a while to familiarise myself with the Australian variety of the language.

It turns out large language models powered by artificial intelligence (AI) such as ChatGPT experience a similar problem.

In new research, published in the Findings of the Association for Computational Linguistics 2025, my colleagues and | introduce a new tool for evaluating the ability of different large language models to detect sentiment and sarcasm in three varieties of English: Australian English, Indian English and British English.

The results show there is still a long way to go until the promised benefits of AI are enjoyed by all, no matter the type or variety of language they speak.

Large language models are often reported to achieve superlative performance on several standardised sets of tasks known as benchmarks.