What the estimate includes
The model size is entered in decimal billions of stored parameters. Weight storage is parameters × bits per parameter ÷ 8; a separate metadata percentage allows for scales, zero points and other representation costs. Results use binary GiB (2³⁰ bytes). Quantization formats differ, so the metadata input is an allowance rather than a claim about a specific checkpoint.
KV cache follows KV heads, not every query head
For a uniform full-attention decoder, raw KV bytes = 2 × layers × KV heads × head dimension × cached tokens per sequence × simultaneous sequences × bytes per cache value. The factor two stores keys and values. Grouped-query attention can use fewer KV heads than query heads. Sliding-window layers, hybrid architectures, latent attention and backend padding need a model-specific calculation beyond this uniform estimate.
Multiple GPUs require a supported execution plan
The example assumes evenly tensor-sharded weights and requires query heads divisible by the GPU count. Cache sharding is assumed only when KV heads divide evenly across the selected GPUs; otherwise the estimate retains a full cache on each device. Runtime overhead is charged on every GPU. This does not validate the interconnect, partitioning support, kernel layout or communication buffers of a real serving engine.
Read the headroom as a planning result
Positive headroom means only that this assumed memory budget is below the entered usable capacity. It does not guarantee successful loading, acceptable latency or a particular token rate. CUDA graphs, allocator reservations, activations, batching strategy and other processes can change the actual requirement. Measure representative peak use on the intended engine before committing capacity.
Worked example
- Illustrative 32.5B model, 4-bit weights, 10% weight metadata and two-way tensor parallelism: raw weights are 15.134 GiB and metadata adds 1.513 GiB across the model.
- With 64 layers, 8 KV heads, head dimension 128, 8,192 tokens, 4 simultaneous sequences and FP16 KV, raw KV cache is exactly 8 GiB. Even KV-head sharding puts 4 GiB on each GPU.
- Each GPU therefore needs about 8.324 GiB for weights/metadata + 4 GiB KV + 4 GiB runtime allowance = 16.324 GiB. Against an entered 24 GiB usable capacity, planning headroom is about 7.676 GiB.
Questions and interpretation
Does this size training memory?
No. Gradients, optimizer states, saved activations and training parallelism are not included.
Can I use every GPU’s memory as one pool?
Only a supported partitioned execution plan can distribute tensors across devices. This calculator makes an explicit even tensor-parallel assumption and reports a per-device budget.
Why does context length change the result?
The full-attention KV cache retains key/value state for cached tokens. Larger contexts and more simultaneous sequences increase that term linearly in this estimate.
References and verification
The linked sources explain the background. The interactive calculations and scenarios are original teaching exercises with explicit assumptions, not certified equipment selection, measured performance or a real security assessment.