API-hosted vs. self-hosted deployment as a real architectural decision, inference optimization (quantization, KV-caching, batching), why naive serving doesn’t scale, latency vs. throughput as a tradeoff you have to choose, autoscaling variable traffic, and edge deployment as a distinct path.
Module 11 covered the hardware landscape a model actually runs on — GPUs, TPUs, NPUs, and the memory bandwidth that quietly bottlenecks all of them. This module is where that hardware becomes a real deployment decision: whether to call a model through a hosted API or run it yourself (the exact tradeoff this studio’s API vs. Self-Hosted Models concept explainer illustrates), which inference optimizations actually matter, and why a naive “call generate() in a loop” approach falls apart the moment real traffic arrives.
By the end of this module you should be able to explain why time-to-first-token, tokens-per-second, and requests-per-second genuinely compete with one another, what a production inference server actually adds over a hand-rolled loop, and why edge deployment is a distinct engineering decision rather than just self-hosting on smaller hardware. Project 4, the Production LLM Inference Service, is where this entire module gets applied to one real, worked system.