Reducing AI's Dependence ON POWERFUL GPUs

The quest is on, to reduce Al's over-dependence on power-hungry and expensive GPUs, to make it more affordable, accessible, and sustainable.

- JANANI G. VIKRAM

To the novice, it might seem as if artificial intelligence (AI) is a magical genie that gives you everything you ask for with the snap of a finger, but techies know that a lot of muscle power (read graphics processing units, or GPUs) goes into making it all work.

The tech infrastructure required for training and deploying AI systems is so vast that it may not even be sustainable in the future. Environmentalists are already very concerned about the widespread use of GPUs for AI, as it guzzles power, increases carbon dioxide emissions, and uses alarming amounts of water to cool data centres. Also, the volatile geopolitical landscape of today is putting GPUs out of reach of many countries, thereby causing an imbalance in the concentration of tech power.

Today, when we refer to GPUs in the AI context, it invariably means Nvidia’s GPUs, such as the A100, H100, and GB200. Tensor core accelerators that speed up matrix operations, and an effective parallel computing platform and programming model called CUDA (compute unified device architecture), have made Nvidia GPUs the prime choice for training large-scale AI models. The latest from their stables is the GB200 series of superchips, which bring together Blackwell GPUs and Grace CPUs into a core, enabling massive generative AI (GenAI) workloads like training and real-time inference of trillion-parameter large-language models (LLMs). Market studies estimate that Nvidia holds more than 90% of the discrete GPU market today.

Over-dependence on Nvidia GPUs

Despite Nvidia’s GPUs being technically superior, the industry and academia are working hard today to reduce the dependence of AI models on these powerful GPUs. One obvious reason is the export restrictions imposed by the US government. There is limited or no supply of H100, A100, and GB200 to many countries; and most others must be content with less capable cousins like the H20, or the GTX and RTX series GPUs.