Bridging language divide in GenAI

LLMs enable multi-language communication

- Keshav Murugesh

IN A RECENT breakthrough, an assistant professor and a PhD student of computer science at Stanford University have created an open-source Vietnamese large language model (LLM). In collaboration with Ho Chi Minh City University of Technology and Ontocord.ai, they have developed an extensive evaluation framework to boost accessibility, democratization, trust and engagement of this technology amongst users.

Approximately 1.35 billion people, representing 17% of the world's population, speak English. Yet, 46% of the datasets is in English, according to Common Crawl. Current LLM models are challenged by the disproportionate availability of language resources, leading to biases. LLMs are inadequately positioned to perform beyond particular types of English—thus retarding the progress of non-English speaking countries.