Limits of LLMs

WHEN AI SOUNDS CONFIDENT BUT GETS IT WRONG: THE HIDDEN EPISTEMOLOGY OF LLMs

- SIDDHARTH PAI Technology consultant and venture capitalist By invitation

ONE OF THE more amusing, and occasionally sobering, discoveries in the last year has been the gulf between what generative AI (GenAI) sounds like it knows and what it can actually compute.

This gap was thrown into sharp relief during an experiment where, on a lark, I recently pushed an AI model to produce astrological predictions. The model responded with elaborate charts, detailed planetary degrees, multi-decade timelines, and sweeping narratives of destiny. It sounded persuasive. It was also wrong.

This episode, trivial as it might seem, illuminates a deeper, and far more important, truth about today’s GenAI systems. They are not knowledge engines. They do not perform calculations. They cannot guarantee the factual accuracy of the very outputs they generate with such fluency and confidence. Instead, they operate on a form of linguistic probability: a statistical sense of what words are likely to appear next to other words. For instance, the word following “very” is far more likely to be the word “good” than the word “hippopotamus”.

Understanding this distinction between appearance and capability is crucial as enterprises experiment with GenAI for customer management, decision support, and even risk modelling. If we do not appreciate the epistemology—the “philosophy of knowledge”—behind these systems, we will be caught flatfooted when they fail in places we assumed they would excel.