When AI moves from chips to racks

AI performance is no longer just about faster chips. It is about how racks, power, networking, and orchestration work together. As agentic AI grows, infrastructure must become predictable, open, and built for scale from day one

- By Ashok Pandey

At the AI Summit, one theme stood out: performance is no longer decided at the chip level. It is defined at the rack level.

Archana Vemulapalli, Corporate Vice President of Global Commercial Sales at AMD, and Mahesh Balasubramanian, Senior Director, Data Center GPU Product Marketing at AMD, outlined how this shift is reshaping infrastructure thinking. The message was clear: AI scaling is no longer about stacking individual servers. The rack has become the unit of design.

For large frontier models, tightly integrated infrastructure drives efficiency. Multi-hundred-billion and even multi-trillion parameter models demand more than powerful accelerators. They require compute, memory, networking, power, and cooling engineered as one system.

The chip is still core. But it is no longer enough.

Why rack scale matters now

Rack-scale infrastructure is becoming critical in two major scenarios.

First, pre-training large models. At that scale, dividing workloads across loosely connected systems introduces communication overhead. Efficiency drops. Integrated racks reduce those inefficiencies.

Second, inference at scale. When inference serves a large number of users or AI agents, efficiency becomes a function of integration. Large-context inference and massive concurrency benefit from fully integrated rack-scale systems.

AI is also expanding beyond centralized training clusters. It is proliferating across:

• CPUs for traditional enterprise applications.

• GPUs for model training and high-performance inference.

• PCs for local and private AI use cases.

• Edge systems in telecom base stations and distributed environments.

• Field-Programmable Gate Arrays (FPGAs) for accelerating AI at the edge.

Yet, at the front end of AI development, especially large model training, rack-scale integration is becoming decisive.