End-to-end engineering case study for building a domain-specific Large Language Model — from problem scoping and data pipeline design through transformer architecture, pretraining, fine-tuning, RAG, safety evaluation, and production serving.
The guide walks through each transformer component from first principles: token embeddings (|V|×d_model matrix), positional encoding (sinusoidal vs. learned vs. RoPE — LLaMA uses Rotary Position Embeddings for length generalization), multi-head self-attention with Grouped Query Attention (GQA reduces KV cache by 8–32×), SwiGLU feed-forward networks (3.5× d_model expansion in LLaMA 3), and RMSNorm for training stability. The self-attention formula Attention(Q,K,V)=softmax(QKᵀ/√dₖ)V runs in O(n²) time — FlashAttention-2 reduces memory to O(n) via SRAM-aware kernel fusion.
The fine-tuning chapter covers the full spectrum: full fine-tuning (all 7B params; 84GB GPU; catastrophic forgetting risk), LoRA (train rank-16 adapters on Q/K/V/O projections; 0.5% of params; single A100), and QLoRA (4-bit NF4 quantized base + LoRA; fits 70B on one A100 80GB). SFT loss is computed only on assistant tokens. Alignment uses DPO (Direct Preference Optimization) over explicit RLHF/PPO for most projects — requires only 5K–20K preference pairs, 2 models in memory, and standard gradient descent. Constitutional AI extends this with model self-critique loops.
RAG pipeline: chunk documents (256–512 tokens, semantic boundaries), embed with BGE-M3 or text-embedding-3, store in Qdrant or pgvector, retrieve top-K with BM25 + dense hybrid search, re-rank with cross-encoder, inject into context. Tool use follows the ReAct pattern (Reason → Act → Observe). Evaluation requires a golden eval set (200+ expert-labeled examples) built before training. Production serving uses vLLM with PagedAttention for 3–5× throughput vs. naive inference. INT4 quantization (AWQ) halves memory with ~3% benchmark degradation.
The appendices provide practical reference material: Appendix A compares open-weights models (LLaMA 3.1, Mistral, Qwen 2.5, Phi-3, Gemma 2) by params, context length, and license. Appendix B covers tokenizer APIs and chat template formats across model families. Appendix C includes a prompt template library (zero-shot, chain-of-thought, few-shot, RAG-grounded, JSON output). Appendix D–F provide evaluation checklists, deployment cost tables, and the full end-to-end build checklist. Appendix G is a 25-term glossary from attention through ZeRO.
Fine-tune an existing open-weights model (LLaMA 3, Mistral, Qwen) for almost all domain LLM projects. Training from scratch is only justified when your domain vocabulary is radically different from general text (e.g., genomics sequences) or your data cannot touch any third-party infrastructure. The practical test: fewer than 10B tokens of domain text + standard GPU cluster = fine-tune. A fine-tuned 7B model beats a from-scratch 1B model on most domain tasks at a fraction of the compute cost.
QLoRA (Quantized LoRA) quantizes the frozen base model to 4-bit NF4 format (reducing 7B params from 14GB to ~4.5GB), then trains small LoRA adapter matrices in full precision. This lets you fine-tune a 70B model on a single A100 80GB GPU — a task that otherwise requires 8+ GPUs for full fine-tuning. Quality loss vs. full fine-tuning is typically 2–5% on benchmarks. Start with QLoRA rank=16 using Unsloth (2× faster than HuggingFace PEFT), then increase rank if quality is insufficient before considering full fine-tuning.
Four main failure modes: (1) Semantic mismatch — query and document embeddings do not align despite relevant content; fix with BM25+dense hybrid retrieval. (2) Chunk boundary fragmentation — the answer spans two chunks; fix with parent-child chunking or larger chunks. (3) Lost-in-the-middle — LLMs attend poorly to middle context tokens; fix by ordering retrieved chunks by relevance descending. (4) Stale index — documents updated but not re-embedded; fix with document change detection and incremental re-indexing. Add a cross-encoder re-ranking step after initial retrieval for all production RAG systems.
Use LLM-as-judge: send model outputs to a capable judge model (GPT-4o or Claude 3.5) with a structured rubric scoring factual accuracy, completeness, clarity, and safety on a 1–5 scale. Output JSON scores for aggregation. Two important mitigations for judge bias: randomize response order in A/B comparisons (position bias favors the first response), and ask the judge to provide reasoning before scoring (reduces superficial judgments). Build a 200–500 example golden eval set with expert-written ideal answers before training begins — run every model version against it before deployment.