Technical deep-dive into how large language models work — from transformer architecture and self-attention math to scaling laws, LoRA fine-tuning, RLHF alignment, KV caching, quantization, and production deployment with vLLM and TensorRT-LLM.
The transformer, introduced in "Attention Is All You Need" (Vaswani et al., 2017), replaced recurrence with self-attention. Core components: Token Embeddings (map words to vectors), Positional Encodings (inject sequence position information), Multi-Head Self-Attention (each token attends to all others, capturing relationships), Feed-Forward Networks (per-position transformation), and Layer Normalization. The stack: decoder-only transformers (GPT family, LLaMA) generate text autoregressively; encoder-only (BERT) are used for classification and embeddings; encoder-decoder (T5, BART) for translation and summarization.
Self-attention computes three projections for each token: Query (Q), Key (K), and Value (V). Attention score = softmax(QKᵀ / √dₖ) × V. The √dₖ scaling prevents gradient vanishing as dimensionality grows. Multi-head attention runs H parallel attention heads (each with different Q/K/V projections), then concatenates and projects the results — allowing the model to attend to information from different representation subspaces. In GPT-4 style models, each layer has 96 attention heads with 128-dimensional head dimension. KV cache stores K and V tensors for already-generated tokens, so inference doesn't recompute attention for the full sequence on each new token.
Chinchilla scaling laws (Hoffmann et al., 2022): for optimal compute efficiency, train on 20× tokens per parameter. LLaMA 3 8B was trained on 15 trillion tokens (vs the ~160B recommended for compute-optimal). Open-source model landscape: LLaMA 3.1 (8B, 70B, 405B — Meta, best open weights), Mistral 7B/Mixtral 8×7B (sparse MoE, efficient inference), Qwen 2.5 (72B, strong multilingual and coding), Phi-3 (3.8B, Microsoft, outperforms models 3× its size on benchmarks), Gemma 2 (Google, 2B, 9B, 27B). Model size selection: 7–8B for on-device/edge, 70B for production quality, 405B+ for frontier tasks.
Full fine-tuning: updates all model weights — maximum quality, requires same memory as training from scratch (prohibitive for 70B+ models). LoRA (Low-Rank Adaptation): adds small trainable rank-decomposition matrices to attention layers; trains 0.1–1% of parameters; the original weights stay frozen. A 7B model can be LoRA fine-tuned on a single A100 80GB GPU. QLoRA (Quantized LoRA): combines 4-bit NF4 quantization of base weights with LoRA adapters — fine-tunes 70B models on 2× A100 80GB. Typical LoRA hyperparameters: rank r=8–64, alpha=2r, target modules=[q_proj, v_proj, k_proj, o_proj]. Libraries: Hugging Face PEFT (LoRA), Unsloth (2× faster LoRA training on NVIDIA and Apple Silicon).
N-gram models predict the next word based on the previous N-1 words using statistical co-occurrence counts. A trigram model (N=3) can only look back 2 words. They require no training in the deep learning sense, are computationally cheap, but fail to capture long-range dependencies. Transformers use self-attention to consider the entire input sequence simultaneously — each token can attend to any other token regardless of distance. A GPT-4 token can attend to tokens 128,000 positions earlier. Transformers also learn dense representations (embeddings) that capture semantic similarity, whereas n-grams treat each word as a discrete symbol. The transformer's ability to capture long-range context is the key reason it replaced n-grams and RNNs for language modeling.
RLHF (Reinforcement Learning from Human Feedback) is the training technique that converts a raw language model (trained to predict next tokens) into a helpful assistant. The three-step process: (1) Supervised Fine-Tuning (SFT) — fine-tune the base model on a dataset of prompt-response pairs that demonstrate helpful, harmless behavior. (2) Reward Model Training — human raters compare pairs of model responses and rank them; a separate reward model is trained to predict human preferences. (3) RL Optimization — use PPO (Proximal Policy Optimization) to update the SFT model to maximize the reward model score, while keeping the model close to the SFT baseline via a KL divergence penalty. The result: a model that produces responses humans rate as more helpful and less harmful. DPO (Direct Preference Optimization) is a more recent, simpler alternative that achieves similar results without the RL step.
Full fine-tuning updates all model parameters and can change any aspect of the model's behavior, but requires substantial GPU memory (roughly 16 bytes per parameter for mixed-precision training). A 7B model needs ~112GB of GPU memory for full fine-tuning. LoRA trains small adapter matrices (rank r=8–64) injected into the attention layers, reducing trainable parameters by 99%+ and GPU memory by 60–80%. Use full fine-tuning when: you need maximum domain adaptation quality, you have many GPUs available, or you're fine-tuning a small model (<3B parameters). Use LoRA when: you have limited GPU resources, you want fast experimentation (LoRA trains 3–5× faster), or you need to maintain multiple fine-tuned variants of the same base model (each LoRA adapter is ~10–100MB vs the full model's 14GB+). QLoRA extends LoRA to large models (70B+) by quantizing the frozen base weights to 4-bit.
vLLM: open-source Python library optimized for high-throughput multi-user LLM serving. Its key innovation is PagedAttention — manages KV cache as virtual pages (like OS virtual memory), enabling efficient batching of requests with different sequence lengths. Best for: serving LLM APIs with many concurrent users on NVIDIA GPUs. TensorRT-LLM (NVIDIA): compiles models to optimized CUDA kernels with int8/fp8 quantization and kernel fusion. Achieves the lowest latency on NVIDIA hardware for small batch sizes. Best for: latency-critical applications on NVIDIA GPUs. llama.cpp: C++ inference engine optimized for CPU and Apple Silicon (Metal GPU), supports GGUF quantized models (Q4, Q5, Q8). Best for: local inference on consumer hardware, edge deployment, or when NVIDIA GPUs are unavailable. Rule of thumb: use vLLM for cloud serving, TensorRT-LLM for latency-critical NVIDIA deployments, llama.cpp for local/edge.
Speculative decoding reduces LLM inference latency by using a small "draft" model to generate candidate tokens, then verifying them in parallel with the large "target" model. Standard autoregressive decoding generates one token at a time — each token requires a full forward pass through the large model. Speculative decoding: (1) Draft model (e.g., 7B) generates K candidate tokens quickly. (2) Target model (e.g., 70B) runs a single forward pass to evaluate all K candidates in parallel. (3) Accept all tokens that match the target model's distribution; resample from the first divergence. When draft and target models agree (common for high-probability tokens), you get K tokens for the compute cost of approximately 1 large model pass. Typical speedup: 2–3× on tasks where drafting is accurate (coding, factual Q&A), less on creative tasks. SpecInfer, Medusa, and EAGLE are implementations.