Why Do We Need To Invest In Infrastructure Beneath Foundation Models

Foundation models or big AI systems like GPT‑4 (for generating and understanding text and images), BERT (for making sense of sentences) and Stable Diffusion (for crafting vivid visuals) all share something powerful: they're trained using self‑supervised learning on massive unlabeled data. Meaning they learn patterns by predicting parts of the data themselves. Because of this broad training, a single foundation model can adapt to many different tasks like writing, image creation, or analyzing code — with just a little extra fine-tuning.

The magic of these models isn’t just in the clever algorithms, it’s in the giant, high‑power machines behind the scenes. Imagine thousands of super‑fast GPUs working together, chatting over lightning‑fast connections, all fed by optimised systems that keep everything running smoothly. That’s the real muscle behind their smarts and without this high‑speed hardware backbone, their impressive abilities and flexibility wouldn’t be possible.

Now imagine this: Nvidia’s H100 GPUs, the powerhouse chips optimised for modern foundation models, each sip up to 700 watts. That’s more than the electricity an entire household might use at peak!

And multiply that by millions. At a typical yearly utilization rate of about 61%, each H100 runs through roughly 3.74 megawatt-hours (MWh) a year. If 3.5 million of these GPUs are deployed globally, they could consume a whopping 13,000 gigawatt-hours annually. That’s like powering entire nations — places such as Lithuania or Guatemala!

When these GPUs are deployed at scale, data centers effectively become "AI factories". Think massive racks drawing kilowatts, like 240 kW per rack, pushing facility demands into the realm of gigawatts. That leads to investments reaching half a trillion dollars by 2028, as operators chase power density and efficiency.

The bottom line is that while the headlines fixate on models running on H100s, the real backbone is the infrastructure. It’s the power-delivery systems, the cooling, the grid-level upgrades! And that’s what truly enables their scale, efficiency, and real-world impact.

Zooming into India, the hurdles mirror the global ones, but with local flavour. Cutting-edge compute like A100/H100 clusters often have to be imported, and data centers aren’t always built with AI efficiency in mind. There’s a shortage of big, clean datasets in India’s 21+ languages, which skews model performance toward urban or English-centric use cases.

On top of that, policy frameworks around open data and coordination between research bodies are still immature. And manpower? Skilled AI infrastructure engineers and researchers are few, with efforts split across silos.

Today’s AI isn’t about stacking on more parameters; it’s about working smarter! New-gen models like Mixture-of-Experts only activate the parts of the model needed for each task, optimising performance and speed. OpenAI's “o-series” models lean on internal chains of thought for complex reasoning using fewer parameters, while Liquid AI’s compact Foundation Models pack big-edge performance with low latency. But this smart behavior demands equally intelligent infrastructure hardware that can dynamically route, compute, handle high-bandwidth reasoning, and switch fast in real-world AI agents.

To keep pace, infrastructure must evolve from brute horsepower to nimble orchestration. Companies like Astera Labs (US) are building high-bandwidth, low-latency connectivity chips and smart fabric switches to solve AI data bottlenecks. In India, platforms like Neysa are offering GPU-as-a-service with managed H100 access, orchestration, cost efficiency, and streamlined MLOps for emerging AI players. In short: modern AI requires modern infrastructure, and the shift is already underway!

Soniya Sadhnani, Manager - Seed & Acceleration (IIMA Ventures) & Vipul Patel, Partner - Seed Investing (IIMA Ventures).

Disclaimer: The views expressed here are those of the author and do not necessarily represent the views of NDTV Profit or its editorial team.