What is KV caching and why is it essential for LLM inference?

KV caching stores the Key and Value tensors computed for each input token during the prefill phase and reuses them during autoregressive token generation. Without KV caching, the attention computation would need to reprocess all previous tokens at each generation step, making the cost grow as O(n²) with sequence length. With KV caching, each new token requires only computing the attention of the new token against all cached Keys and Values, reducing the amortized cost to O(n). KV caching is considered mandatory for any production LLM deployment — the latency and compute cost without it would make real-time generation with long contexts completely infeasible. The tradeoff is memory: KV cache can consume tens of gigabytes for long contexts and large batch sizes.

What is the difference between GPTQ and AWQ quantization and which should I use?

GPTQ (Gradient-based PTQ) quantizes each layer's weights to minimize the layer's output error, using second-order information from the Hessian. It is effective, well-supported, and widely available for most open-source models. AWQ (Activation-Aware Weight Quantization) identifies the 1% of weights that most strongly influence activations and protects them from quantization, resulting in better accuracy on reasoning and factual tasks at the same bit width. For most production deployments, AWQ is the preferred choice — it generally produces better results than GPTQ at 4-bit precision, especially for instruction-following and reasoning tasks. GPTQ remains a valid option if AWQ-quantized versions of your target model are not available or if you need to quantize a custom fine-tuned model yourself.

When should I use vLLM vs TensorRT-LLM vs llama.cpp for LLM serving?

Use vLLM for most production deployments of open-source LLMs: it offers continuous batching, PagedAttention, OpenAI-compatible API, broad model support, and good throughput on NVIDIA and AMD GPUs. Use TensorRT-LLM when you need maximum throughput on NVIDIA GPUs and can tolerate the model compilation step (which takes minutes to hours) — it outperforms vLLM on raw throughput benchmarks through kernel fusion and hardware-specific optimizations. Use llama.cpp for CPU inference, development environments, edge deployments, or when running small quantized models on GPUs with limited VRAM — it is not suitable for high-throughput production serving but is invaluable for prototyping and local development without GPU access.

How do I estimate GPU memory requirements for deploying an LLM?

Estimate GPU memory as follows: model weights = parameters × bytes_per_parameter (2 for FP16/BF16, 1 for INT8, 0.5 for INT4). A 7B FP16 model needs 14 GB, a 70B FP16 model needs 140 GB. Add 20% for activations and framework overhead. Then add KV cache: 2 × layers × d_model × max_seq_len × batch_size × bytes_per_parameter. For a 7B LLaMA-3 model (32 layers, 4096 d_model, FP16, 8K context, batch size 8), KV cache is approximately 16 GB. Total: 14 + 2.8 + 16 = 33 GB — fits on an A100 40GB. Use AWQ 4-bit quantization to reduce the model weight footprint to ~3.5 GB if memory is tight.

What is speculative decoding and when does it improve LLM inference performance?

Speculative decoding uses a small, fast draft model to generate k candidate tokens simultaneously, which the large target model verifies in a single forward pass. Because GPU hardware can verify k+1 tokens in nearly the same time as generating 1 token (verification is parallelized), each accepted draft token costs very little additional compute. When the draft model's predictions are accepted with high probability (typically 70–90% on English text and code), speculative decoding reduces end-to-end generation latency by 2–4× without changing the target model's output distribution. It works best for domains where a well-chosen draft model closely matches the target model's distribution — it is less effective for highly creative tasks or domains where the draft model is weak. The tradeoff is doubled GPU memory usage (holding both the draft and target model) and added serving complexity.

AI & Automation·11 min read·June 19, 2026

⚙️ LLM Inference Optimization: Running AI Models in Production Without Breaking the Bank

A comprehensive technical guide to LLM inference optimization — covering KV cache mechanics, quantization methods (INT8, INT4, GPTQ, AWQ), speculative decoding, continuous batching, tensor parallelism, and comparing vLLM vs TensorRT-LLM vs llama.cpp for production deployments with cost estimation formulas and GPU memory sizing.

📖AI Roadmap for Electrical Engineers (Full Access)

View →

Why Inference Optimization Is the Critical Bottleneck in Production LLMs

Training and fine-tuning determine what a model knows and how it behaves. Inference optimization determines whether that model is economically viable in production. For most organizations deploying LLMs, inference costs dominate the total lifetime expense — often by a factor of 10 to 100 over training costs for a successful product. A model that produces excellent results in a research context can be completely infeasible to deploy if its inference cost per request is too high or its latency is too long for user experience requirements.

Topics covered

LLM inference optimizationKV cache mechanicsquantization INT8 INT4GPTQ quantizationAWQ quantizationspeculative decoding LLMcontinuous batching vLLMtensor parallelism LLMvLLM vs TensorRT-LLMllama.cpp inferenceLLM production deploymentGPU memory LLM sizingLLM cost optimizationtime to first tokenthroughput vs latency LLMFlashAttention optimizationLLM serving infrastructuretoken per second GPUinference cost estimationLLM deployment cost