Deep technical guide to the full AI compute stack — from GPU die architecture and HBM memory bandwidth to CUDA programming, inference engines, NVLink interconnects, cloud pricing, and the emerging ASIC landscape (Cerebras, Groq, TPUs).
NVIDIA H100 (Hopper, 2022): 80GB HBM2e, 3.35 TB/s memory bandwidth, 989 Tensor TFLOPS (FP16), 700W TDP. NVLink 4.0 at 900 GB/s for multi-GPU communication. H200 (2024): same die, upgraded to HBM3e with 4.8 TB/s bandwidth — bandwidth is often the bottleneck for large model inference. B200 (Blackwell, 2024): 192GB HBM3e, FP4 compute for 5× inference throughput vs H100. The key trend: each generation doubles memory bandwidth before doubling compute — because LLM inference is memory-bandwidth-bound, not compute-bound.
GPUs (NVIDIA, AMD): general-purpose programmable, best for model training and flexible inference, dominant market share. TPUs (Google): custom matrix multiply ASICs optimized for TensorFlow/JAX, excellent for Google-framework training at scale, not widely available outside Google Cloud. NPUs (Apple ANE, Qualcomm Hexagon, Intel NPU): edge inference on mobile/laptop, low power, optimized for specific model sizes. ASICs (Cerebras WSE-3: 4 trillion transistors, entire wafer-scale chip; Groq LPU: deterministic low-latency inference at fixed throughput): highly specialized, exceptional for specific workloads at the cost of programmability.
vLLM: uses PagedAttention for efficient KV cache memory management, enabling high-throughput serving with many concurrent users. Best for: production LLM APIs, multi-user serving. TensorRT-LLM (NVIDIA): aggressive quantization and kernel fusion for maximum NVIDIA GPU throughput. Best for: latency-critical single-user or small-batch inference on NVIDIA hardware. llama.cpp: CPU-friendly inference in C++, supports quantized models (Q4_K_M, Q8_0), runs LLaMA 70B on Mac Studio M2 Ultra. Best for: local inference, edge deployment, resource-constrained environments. Ollama wraps llama.cpp with a simple API for local use.
LLM inference is memory-bandwidth-bound during the autoregressive decoding phase (generating one token at a time). The model weights must be read from HBM for every token generated. A 70B parameter model in FP16 requires ~140GB of memory and ~140 GB/s bandwidth per token at minimum. On an H100 with 3.35 TB/s bandwidth, you can generate tokens much faster than on a GPU with less bandwidth but more raw TFLOPS. This is why H200 (4.8 TB/s HBM3e) improves inference throughput over H100 despite the same compute die — more bandwidth = faster token generation.
GPUs (Graphics Processing Units) are general-purpose parallel processors with thousands of cores optimized for floating-point matrix operations. They are programmable via CUDA (NVIDIA) or ROCm (AMD) and support any ML framework. TPUs (Tensor Processing Units) are custom ASICs designed by Google specifically for matrix multiply operations used in neural networks. TPUs use a systolic array architecture that is more efficient than GPU SIMD for the specific shapes of operations in transformer training, but they are only available on Google Cloud (TPU v5e, v5p) and work best with JAX or TensorFlow. GPUs are more flexible and widely available; TPUs can be 3–5× more cost-efficient for large-scale training on Google Cloud with compatible models.
Tensor cores are specialized hardware units inside NVIDIA GPUs (Volta and later) that perform fused multiply-add operations on small matrix tiles (4×4 or 8×8) in a single cycle. They dramatically accelerate the matrix multiplications that dominate transformer training and inference. Without tensor cores, a GPU performs one floating-point operation per cycle per core; tensor cores perform 64 or more FMA operations per cycle. H100 has 528 tensor cores, delivering 989 TFLOPS in FP16 with sparsity. Critical: tensor cores require matrix dimensions to be multiples of 8 (for FP16) or 16 (for FP8) — models and batch sizes not aligned to these dimensions miss significant performance.
During LLM token generation (autoregressive decoding), the GPU generates one token at a time. Each token requires reading all model weights from HBM memory. A 7B parameter model in FP16 = ~14GB of data read per token. An H100 with 3.35 TB/s bandwidth can read those 14GB approximately 240 times per second — so theoretical maximum is ~240 tokens/second before accounting for attention computation and other overheads. Doubling compute (TFLOPS) does not help if the bottleneck is memory bandwidth. This is why HBM memory (High Bandwidth Memory, stacked on the die via through-silicon vias) is so critical for LLM inference, and why the H200 upgrade (4.8 TB/s vs 3.35 TB/s) improves inference throughput more than a compute upgrade would.
LLaMA 3 70B in full FP16 precision requires approximately 140GB of VRAM (70 billion parameters × 2 bytes per FP16 parameter). This exceeds single-GPU capacity for most hardware. With 4-bit quantization (Q4_K_M via llama.cpp), the model requires ~40GB — runnable on two RTX 3090/4090 cards (48GB combined VRAM), or on Apple Silicon Mac Studio M2 Ultra (192GB unified memory, though much slower than GPU). With 8-bit quantization (Q8_0), approximately 75GB is needed. For production inference of 70B models at decent throughput, 2× H100 80GB (160GB total HBM) is the minimum GPU cluster. For local experimentation, Apple Mac Studio M2/M3 Ultra with 192GB unified memory and llama.cpp is the most practical single-machine solution.
NVLink is NVIDIA's high-bandwidth GPU-to-GPU interconnect, used in multi-GPU training and inference setups. NVLink 4.0 (H100) provides 900 GB/s bidirectional bandwidth between GPUs — compared to PCIe 5.0's 128 GB/s. NVLink matters when: (1) Training models too large for a single GPU (gradients must be communicated between GPUs via all-reduce operations — bandwidth is the bottleneck). (2) Running model-parallel inference across multiple GPUs. NVLink is available through NVSwitch fabrics in DGX H100 systems (8 GPUs, fully connected at 900 GB/s each). Without NVLink (using PCIe only), large model training efficiency drops 20–40%. For single-GPU inference (model fits in one GPU's VRAM), NVLink is irrelevant.