ENGINEERING WORKBENCH · INCLUDED WITH ALL-ACCESS

GPU Memory / VRAM Sizing Calculator

Inference memory includes model weights, quantization metadata, the key/value cache and runtime buffers. Context length and simultaneous sequences can exhaust VRAM even when the weights fit. This calculator separates those terms and reports an assumed per-GPU budget rather than adding every GPU’s memory into one automatic pool.

Published and reviewed · EngineersUniverse · Interactive workspace · All-Access ↓ · Worked example ↓

What the estimate includes

The model size is entered in decimal billions of stored parameters. Weight storage is parameters × bits per parameter ÷ 8; a separate metadata percentage allows for scales, zero points and other representation costs. Results use binary GiB (2³⁰ bytes). Quantization formats differ, so the metadata input is an allowance rather than a claim about a specific checkpoint.

KV cache follows KV heads, not every query head

For a uniform full-attention decoder, raw KV bytes = 2 × layers × KV heads × head dimension × cached tokens per sequence × simultaneous sequences × bytes per cache value. The factor two stores keys and values. Grouped-query attention can use fewer KV heads than query heads. Sliding-window layers, hybrid architectures, latent attention and backend padding need a model-specific calculation beyond this uniform estimate.

Multiple GPUs require a supported execution plan

The example assumes evenly tensor-sharded weights and requires query heads divisible by the GPU count. Cache sharding is assumed only when KV heads divide evenly across the selected GPUs; otherwise the estimate retains a full cache on each device. Runtime overhead is charged on every GPU. This does not validate the interconnect, partitioning support, kernel layout or communication buffers of a real serving engine.

Read the headroom as a planning result

Positive headroom means only that this assumed memory budget is below the entered usable capacity. It does not guarantee successful loading, acceptable latency or a particular token rate. CUDA graphs, allocator reservations, activations, batching strategy and other processes can change the actual requirement. Measure representative peak use on the intended engine before committing capacity.

Worked example

  1. Illustrative 32.5B model, 4-bit weights, 10% weight metadata and two-way tensor parallelism: raw weights are 15.134 GiB and metadata adds 1.513 GiB across the model.
  2. With 64 layers, 8 KV heads, head dimension 128, 8,192 tokens, 4 simultaneous sequences and FP16 KV, raw KV cache is exactly 8 GiB. Even KV-head sharding puts 4 GiB on each GPU.
  3. Each GPU therefore needs about 8.324 GiB for weights/metadata + 4 GiB KV + 4 GiB runtime allowance = 16.324 GiB. Against an entered 24 GiB usable capacity, planning headroom is about 7.676 GiB.

Questions and interpretation

Does this size training memory?

No. Gradients, optimizer states, saved activations and training parallelism are not included.

Can I use every GPU’s memory as one pool?

Only a supported partitioned execution plan can distribute tensors across devices. This calculator makes an explicit even tensor-parallel assumption and reports a per-device budget.

Why does context length change the result?

The full-attention KV cache retains key/value state for cached tokens. Larger contexts and more simultaneous sequences increase that term linearly in this estimate.

References and verification

The linked sources explain the background. The interactive calculations and scenarios are original teaching exercises with explicit assumptions, not certified equipment selection, measured performance or a real security assessment.

Continue the study

Explore the other workbenches