Does quantization change how many parameters a model has?

No. Quantization changes only the numeric precision each existing parameter is stored and computed in — the count of parameters and the architecture connecting them stays exactly the same before and after quantization.

Does distillation require the teacher model to be modified?

No. The teacher model stays fixed and unmodified throughout distillation — it only supplies the training signal (its outputs, and often its full output probability distribution) that the separate, smaller student model is trained to mimic.

Does quantization always require a new training run?

Basic post-training quantization does not require any new training run — it is a direct numeric re-encoding of already-trained weights. A refinement called quantization-aware training does run additional, usually lightweight training so the model tolerates the lower precision better and retains more accuracy, but this is optional, not a requirement of quantization itself.

Can a model be both distilled and quantized?

Yes, and this is a very common real-world pipeline: distill a large teacher into a smaller student architecture first (reducing parameter count), then quantize that smaller student's weights to lower precision (reducing per-parameter storage and compute cost). The two savings are additive because they act on different aspects of the model.

If a model is quantized, is it a "smaller model" in the same sense a distilled model is?

Not in the architectural sense. A quantized model has exactly the same number of parameters and the same architecture as the original — what shrinks is the storage size and compute cost per parameter, since each one now takes fewer bits. A distilled model is smaller in the architectural sense — it genuinely has fewer parameters and often a different structure.

Which technique is better for reducing model size?

They are not competing options solving the same problem — they relieve different constraints. Distillation is the right tool when the goal is a genuinely smaller network (fewer parameters, different architecture, possibly better suited to constrained hardware). Quantization is the right tool when the goal is to keep the same network but make its existing numbers cheaper to store and compute. Many deployments benefit from applying both.

← AI Engineering Studio

Concept Explainer · AI Engineering

Model Distillation vs. Quantization

Both get called "shrinking the model," and both make it cheaper and faster to run — but one changes what the model fundamentally is, and the other only changes how its existing numbers are stored and computed.

Say a team needs to deploy a large model more cheaply, and two suggestions come back: "distill it" and "quantize it." Because both reduce compute, memory, and latency, it's tempting to treat them as interchangeable — two flavors of "model shrinking." They are not. Distillation trains a brand-new, genuinely smaller student model — fewer parameters, often a different architecture entirely — to mimic a larger teacher model's outputs, through a real training process. Quantization keeps the exact same architecture and the exact same parameter count, and instead reduces the numerical precision each weight and activation is stored and computed in — no new training run required, and no architectural change at all. That difference is exactly why the two are routinely combined rather than substituted for one another: distill a large model down to a smaller architecture, then quantize thatsmaller model's weights, and both efficiency gains stack.