Spotting text written by ChatGPT is still a struggle

Some signs may serve as tells but are unable to definitively confirm if AI was used in any writing.

- Roger J. Kreuz

People are now routinely using chatbots to write computer code, summarize articles and books, or solicit advice. But these chatbots are also employed to quickly generate text from scratch, with some users passing off the words as their own.

This has, not surprisingly, created headaches for teachers tasked with evaluating their students' written work. It has also created issues for people seeking advice on forums like Reddit, or consulting product reviews before making a purchase.

Over the past few years, researchers have been exploring whether it is even possible to distinguish human writing from artificial intelligence-generated text. But the best strategies to distinguish between the two may come from the chatbots themselves.

TOO GOOD TO BE HUMAN?

Several recent studies have highlighted just how difficult it is to determine whether text was generated by a human or a chatbot.

Research participants recruited for a 2021 online study, for example, were unable to distinguish between human- and ChatGPT-generated stories, news articles and recipes.

Language experts fare no better. In a 2023 study, editorial board members of top linguistics journals were unable to determine which article abstracts had been written by humans and which were generated by ChatGPT. And a 2024 study found that 94 per cent of undergraduate exams written by ChatGPT went undetected by graders at a British university. Clearly, humans are not very good at this.

A commonly held belief is that rare or unusual words can serve as "tells" regarding authorship, just as a poker player might somehow give away that they hold a winning hand.

Researchers have, in fact, documented a dramatic increase in relatively uncommon words, such as "delves" or "crucial", in articles published in scientific journals over the past couple of years. This suggests that unusual terms could serve as tells that generative AI has been used.