More powerful AI accelerators don't always translate to faster models. IBM's Kaoutar El Maghraoui walks through the "memory wall," a fundamental bottleneck where AI workloads become constrained by memory bandwidth rather than capacity.
The video covers arithmetic intensity, the roofline model for performance prediction, and the difference between compute-bound and memory-bound workloads. It also explains why performance varies between the prefill and decode phases, and how techniques like continuous batching can improve efficiency by better utilizing available bandwidth.
Understanding these constraints is essential for anyone deploying large language models at scale or designing AI infrastructure.
This is an AI-written summary. The original reporting belongs to @IBMTechnology. How we write these · Spot a mistake?