Both get called "shrinking the model," and both make it cheaper and faster to run — but one changes what the model fundamentally is, and the other only changes how its existing numbers are stored and computed.
Say a team needs to deploy a large model more cheaply, and two suggestions come back: "distill it" and "quantize it." Because both reduce compute, memory, and latency, it's tempting to treat them as interchangeable — two flavors of "model shrinking." They are not. Distillation trains a brand-new, genuinely smaller student model — fewer parameters, often a different architecture entirely — to mimic a larger teacher model's outputs, through a real training process. Quantization keeps the exact same architecture and the exact same parameter count, and instead reduces the numerical precision each weight and activation is stored and computed in — no new training run required, and no architectural change at all. That difference is exactly why the two are routinely combined rather than substituted for one another: distill a large model down to a smaller architecture, then quantize thatsmaller model's weights, and both efficiency gains stack.
Distillation operates on the model's structure — it produces a smaller network with fewer parameters, trained from scratch (or from a smaller initialization) to reproduce a larger teacher's behavior. Quantization operates on the model's number representation— it takes an existing set of weights and stores and computes them using fewer bits per value, without changing how many weights there are or how they're connected. Because one shrinks the network and the other shrinks the numbers inside an unchanged network, a common real-world pipeline is to do both in sequence: distill a large teacher into a smaller student architecture to cut parameter count, then quantize that student's weights down to 8-bit or 4-bit integers to cut memory and compute further still. Each step yields real, additive savings the other step doesn't touch.
False — they make different kinds of changes, and mixing them up leads to real engineering mistakes, like expecting a quantized model to have fewer parameters (it doesn't) or expecting a distilled model to load faster purely from a memory-format change (its savings come from having fewer weights, not smaller-format ones, unless it's also quantized). Distillation changes what the model fundamentally is — a new, smaller network with its own trained parameters, produced by an actual training process where a student learns to mimic a teacher. Quantization changes only how the model's existing numbers are stored and computed— same architecture, same parameter count, just fewer bits per value (typically no new training run at all, though quantization-aware training exists as a refinement that fine-tunes the model to tolerate lower precision better). Confusing the two means picking the wrong tool for the actual constraint you're trying to relieve — parameter count and architecture on one hand, numeric storage and arithmetic cost on the other.
Explains the fundamental distinction between model distillation (training an entirely new, smaller student model to mimic a larger teacher model's outputs — fewer parameters, often a genuinely different architecture, produced through a real training process) and quantization (keeping the exact same architecture and parameter count, but reducing the numerical precision each weight and activation is stored and computed in, typically without any new training run). Includes a side-by-side illustration of a teacher-to-student distillation training pipeline versus the same weight matrix re-encoded from 32-bit floating point down to 8-bit or 4-bit integers, plus a callout on why the two techniques are routinely combined in the same deployment pipeline rather than treated as substitutes for one another.
Both techniques get pitched as ways to make a large, expensive model cheaper and faster to run, and both genuinely deliver on that promise — smaller memory footprint, lower latency, cheaper hardware requirements. Because the *outcome* (a lighter, faster deployment) looks similar from the outside, it is easy to mentally file both under one bucket, "model shrinking," and treat them as interchangeable options to reach for depending on which is more familiar or convenient. That framing misses that they act on completely different parts of the model: one changes how many parameters exist and how they are arranged, the other changes only the numeric format those parameters are stored and computed in.
Distillation starts with a larger, already-trained teacher model and a separate, typically much smaller student model with fewer layers, fewer parameters, or a different architecture entirely. The student is trained — a real training process, with gradients computed and weights updated — using a loss function that measures how closely the student's outputs (often including the teacher's full probability distribution over possible outputs, not just its single top prediction, since that distribution carries richer information about how confident or uncertain the teacher was) match the teacher's. The teacher itself is not modified during this process; it stays fixed, acting purely as a source of training signal. What comes out the other end is a genuinely new model, with its own trained parameters, that happens to approximate the teacher's behavior at a fraction of its size.
Quantization takes an existing, already-trained model exactly as it is — same layers, same connections, same parameter count — and changes only how each weight (and often each intermediate activation value) is numerically represented and computed. A weight originally stored as a 32-bit floating-point number might be converted to an 8-bit or 4-bit integer, using a scaling scheme that maps the original range of values onto the smaller set of representable values. This is fundamentally a re-encoding, not a re-training: no gradients need to be computed and no new training data is required for basic post-training quantization. A more advanced variant, quantization-aware training, does run additional (usually lightweight) training so the model adapts to the precision loss and retains more accuracy — but that refinement still leaves the architecture and parameter count completely untouched; it never adds or removes a single parameter.
Because distillation reduces parameter count and quantization reduces per-parameter storage and compute cost, applying one does not use up or conflict with the savings available from the other. A common production pipeline runs both in sequence: distill a large teacher down into a smaller student architecture to cut the number of parameters that need to exist at all, then quantize that smaller student's weights down to low-bit integers to cut the cost of storing and computing each of the parameters that remain. The result compounds both kinds of savings — fewer weights, and each of those fewer weights cheaper to store and compute — which is exactly why real deployment pipelines for resource-constrained environments (mobile devices, edge hardware, high-throughput serving) very often use both together rather than picking just one.
No. Quantization changes only the numeric precision each existing parameter is stored and computed in — the count of parameters and the architecture connecting them stays exactly the same before and after quantization.
No. The teacher model stays fixed and unmodified throughout distillation — it only supplies the training signal (its outputs, and often its full output probability distribution) that the separate, smaller student model is trained to mimic.
Basic post-training quantization does not require any new training run — it is a direct numeric re-encoding of already-trained weights. A refinement called quantization-aware training does run additional, usually lightweight training so the model tolerates the lower precision better and retains more accuracy, but this is optional, not a requirement of quantization itself.
Yes, and this is a very common real-world pipeline: distill a large teacher into a smaller student architecture first (reducing parameter count), then quantize that smaller student's weights to lower precision (reducing per-parameter storage and compute cost). The two savings are additive because they act on different aspects of the model.
Not in the architectural sense. A quantized model has exactly the same number of parameters and the same architecture as the original — what shrinks is the storage size and compute cost per parameter, since each one now takes fewer bits. A distilled model is smaller in the architectural sense — it genuinely has fewer parameters and often a different structure.
They are not competing options solving the same problem — they relieve different constraints. Distillation is the right tool when the goal is a genuinely smaller network (fewer parameters, different architecture, possibly better suited to constrained hardware). Quantization is the right tool when the goal is to keep the same network but make its existing numbers cheaper to store and compute. Many deployments benefit from applying both.
Try our AI Engineering Studio
More calculators, simulators, and guides for this discipline.