A Chatbot Dystopian Nightmare

The Trump administration sees an AI-driven federal workforce as more efficient. Instead, with chatbots unable to carry out critical tasks, it would be a diabolical mess

- ASMELASH TEKA HADGU AND TIMNIT GEBRU

IMAGINE CALLING the Social Security Administration and asking, “Where is my July payment?” only to have a chatbot respond, “Canceling all future payments.” Your check has just fallen victim to “hallucination,” a phenomenon in which an automatic speech-recognition system outputs text that bears little or no relation to the input.

Hallucination is one of the many issues that plague so-called generative artificial-intelligence systems such as OpenAI’s ChatGPT, xAI’s Grok, Anthropic’s Claude and Meta’s Llama. These pitfalls result from design flaws in the architecture of these systems that make them problematic. Yet these are the same types of generative AI tools that the Trump administration and its Department of Government Efficiency (DOGE) want to use to, in one official’s words, replace “the human workforce with machines.”

This proposition is terrifying. There is no “one weird trick” that removes experts and creates miracle machines capable of doing everything humans can do but better. The prospect of replacing federal workers who handle critical tasks—ones that could result in life-and-death scenarios for hundreds of millions of people—with automated systems that can’t even perform basic speech-to-text transcription without making up large swaths of text is catastrophic. If these automated systems can’t even reliably parrot back the exact information that is given to them, then their outputs will be riddled with errors, leading to inappropriate or even dangerous actions. Automated systems cannot be trusted to make decisions the way federal workers—actual people—can.