What is quantization? How AI models get smaller and faster

Quantization is a technique that compresses large language models by reducing the precision of their parameters, typically from 32-bit floating-point to 8-bit or 4-bit integer formats. This shrinks model size and speeds up inference while using less GPU memory, though with some trade-off in accuracy.

The video walks through the math: a 70-billion-parameter model stored in standard format requires enormous memory, but quantization can cut that significantly. IBM explains different formats like FP16, FP8, INT8 and INT4, and compares two approaches—post-training quantization and quantization-aware training—to help developers choose the right compression strategy for their use case.