A worked case study in self-hosting an open-weight LLM behind a production API — a cost-crossover calculation against a per-token provider, quantization and KV-cache memory sizing, continuous batching, a concurrent-request throughput budget, and autoscaling on queue depth and p95 latency.
The project brief: a company wants to stop paying per token to an external provider and instead self-host an open-weight LLM behind its own API, for both data-privacy and unit-economics reasons, and needs that service to hold up under a real concurrent request volume within a defined latency budget. This module works through the self-hosted-vs-API decision the way a real infrastructure engineer would — not as a blanket preference, but as a breakeven calculation: given a GPU instance's hourly cost and the API's per-token price, what monthly request volume actually makes self-hosting the cheaper option.
From there the module covers fitting a model's weights and its KV cache into a target GPU's memory, the three optimizations that make self-hosting viable at volume — quantization, continuous batching, and KV caching — a worked latency/throughput budget that computes exactly how many concurrent requests one GPU instance can serve inside an SLA before a second instance is needed, and a monitoring plan built on queue depth and p95 latency rather than raw GPU utilization. The full worked numbers, complete tech-stack manifest, and finished design reasoning are part of the unlocked module below.