Y Combinator's Paper Club video examines —the phase where trained AI models serve requests—and argues it drives far more value than model training itself. For every unit of spent training, roughly six go to serving models annually, generating over $220 billion in market cap over the past decade.
The discussion covers the full inference stack: routing, caching, scheduling, kernels, memory, and hardware tradeoffs. Speakers from Together AI, Ollama, Luminal, and SGLang explore how move from request to output, why identical open models perform differently depending on who serves them, and how new compiler and caching approaches are cutting inference costs and latency.
Topics include the lifecycle of inference requests, token efficiency, model compilation using search rather than heuristics, and memory optimization for AI systems.
This is an AI-written summary. The original reporting belongs to Y Combinator. How we write these · Spot a mistake?