Question 1

What is the difference between a Transformer and an n-gram language model?

Accepted Answer

Both are language models that predict the next token given context, but they differ fundamentally in how they represent and generalize from training data. An n-gram model stores exact frequency counts of word sequences (bigrams, trigrams, etc.) and uses frequency ratios as probabilities. It cannot generalize beyond sequences seen during training. A Transformer LLM learns dense vector representations (embeddings) for every token, then uses layers of multi-head self-attention and feed-forward networks to model arbitrarily long dependencies. Self-attention allows every token to dynamically attend to every other token in the sequence — computing context-aware representations via the formula Attention(Q,K,V) = softmax(QKᵀ/√dₖ)V. The neural network generalizes: words with similar meanings end up with similar embedding vectors, and the attention mechanism can relate any two tokens regardless of their distance in the sequence. Transformers also scale efficiently with data and compute in ways n-gram models fundamentally cannot.

Question 2

What is RLHF and how does it work?

Accepted Answer

RLHF (Reinforcement Learning from Human Feedback) is the process that transforms a capable but instruction-following-naive base LLM into a helpful, cooperative assistant. It has three stages: (1) Supervised Fine-Tuning (SFT) — the base model is fine-tuned on instruction-response pairs, teaching it to answer questions rather than continue text. Loss is applied only to response tokens, not to the instruction/context. (2) Reward Model Training — human raters are shown a prompt and two model responses and rank them by quality. A separate model (the reward model) is trained to predict these scalar preference scores, serving as a proxy for human judgment. (3) Policy Optimization (PPO or DPO) — the LLM is optimized to maximize reward model scores while staying close to the SFT baseline via a KL penalty. This prevents "reward hacking" where the model learns to exploit the reward model rather than produce genuinely good outputs. DPO (Direct Preference Optimization) is a newer alternative that skips the reward model entirely, directly optimizing on pairwise preference data, and is now widely used (LLaMA 3, Mistral-Instruct). Crucially, alignment improves helpfulness and safety behaviors — but does not improve factual accuracy. Aligned models can still hallucinate confidently.

Question 3

What is the difference between LoRA and full fine-tuning?

Accepted Answer

Full fine-tuning updates all parameters in the model during training. For a 7B parameter model in BF16, this requires storing: 14 GB weights + 14 GB gradients + 28 GB Adam optimizer states = ~56 GB minimum, requiring multiple A100s. It offers maximum task performance but risks catastrophic forgetting (losing general capabilities) and is expensive to deploy since each fine-tuned task requires a separate 14 GB model copy. LoRA (Low-Rank Adaptation) keeps the base model frozen and trains only small low-rank decomposition matrices. Instead of updating weight matrix W (d×d), it learns ΔW = A×B where A ∈ Rᵈˣʳ and B ∈ Rʳˣᵈ with rank r ≪ d (typically r=8 or 16). Only A and B are trained — about 0.1–1% of total parameters. QLoRA extends this by quantizing the frozen base model to 4-bit, reducing its memory from 14 GB to 3.5 GB for 7B params, enabling fine-tuning a 70B model on a single GPU. LoRA adapters are also swappable — one base model can serve hundreds of tasks by loading different adapter weights. Engineering rule of thumb: LoRA or QLoRA solves ~80% of practical fine-tuning needs. Full fine-tuning is only justified when you have very large, high-quality datasets and need maximum performance on a specific task.

Question 4

How do I choose between vLLM, TensorRT-LLM, and llama.cpp for LLM inference?

Accepted Answer

The choice depends on your deployment target and requirements. vLLM is the best starting point for most teams running NVIDIA GPUs: it provides an OpenAI-compatible API server with PagedAttention memory management and continuous batching, achieving 3–5× higher throughput than naive HuggingFace inference at the same latency budget. It supports most popular open models and quantization formats. TensorRT-LLM is NVIDIA's own inference library optimized for maximum throughput and minimum latency on A100/H100 hardware. It uses kernel fusion and custom CUDA implementations to extract maximum GPU efficiency, often 20–40% faster than vLLM in throughput benchmarks at the cost of significantly more complex setup and limited model support. It is the right choice for high-volume production on NVIDIA hardware where engineering time is available. llama.cpp is the right choice for consumer hardware (MacBook, gaming GPU, CPU-only), development environments, or edge deployment. It runs quantized models in GGUF format on virtually any hardware, has minimal dependencies, and powers Ollama. It cannot match vLLM or TensorRT-LLM throughput. Key metrics to measure: Time to First Token (TTFT) for interactive use cases, tokens-per-second throughput for batch workloads, and cost-per-1K-tokens for budget planning.

Question 5

What is speculative decoding and how much does it speed up LLM inference?

Accepted Answer

Speculative decoding is an inference technique that uses a small "draft" model to generate multiple candidate tokens, which a larger "verifier" model then accepts or rejects in a single parallel forward pass. Standard autoregressive generation requires one full forward pass through the large model per output token — for a 70B model, this is the primary latency bottleneck. In speculative decoding: (1) A small model (e.g., 7B, ~10× faster) generates a batch of K candidate tokens (K=4–8 typically) in K sequential steps. (2) The large model runs ONE forward pass to verify all K candidates simultaneously — if the large model would have predicted the same token, it is accepted; if not, generation backtracks from that point. The speedup is proportional to the acceptance rate. When the draft model is well-matched to the verifier model (same family, good draft accuracy), acceptance rates of 60–80% are common, yielding 2–4× effective throughput improvement with mathematically identical output distribution — the output quality is exactly what the large model would have produced. Practical implementations: LLaMA uses LLaMA-7B as a draft for LLaMA-70B. Claude models use smaller distilled drafts. TensorRT-LLM and vLLM both support speculative decoding natively.