The math behind why models underfit or overfit — and why you can't fix both by pushing a single dial in one direction.
Underfitting and overfitting look like two unrelated problems from the outside, but they share a single mathematical root: a model's total expected error on new data can be decomposed into two independent sources — bias and variance— plus a floor of noise no model can remove. The uncomfortable part is that bias and variance don't move independently. Turning the dial that reduces one almost always turns up the other. The entire practice of model selection, regularization, and cross-validation exists to find the one point on that dial where their sum is smallest — not the point where either one hits zero.
Bias is the error that comes from a model being too simple to represent the true pattern in the data — it makes strong, restrictive assumptions ("the relationship is a straight line") that are wrong regardless of how much training data you feed it. A high-bias model is consistently wrong in the same way every time, because the mistake lives in the model's shape, not in the particular sample it happened to see. High bias shows up as underfitting: poor performance on boththe training data and new data, because the model can't capture the real relationship even in the data it was fit on.
Variance is the error that comes from a model being too sensitive to the specific training sample it happened to see. A high-variance model is flexible enough to fit not just the true underlying pattern but also the noise and idiosyncrasies unique to one particular dataset — so if you retrained it on a different sample drawn from the exact same distribution, you'd get a noticeably different model. High variance shows up as overfitting: the model fits its own training data extremely well, but performs poorly on new data, because a meaningful part of what it "learned" doesn't generalize. (For the full mechanics of what underfitting and overfitting look like in practice — training-vs-validation error curves, early stopping — see the companion Overfitting vs. Underfitting explainer; this page is the theory underneath it.)
For a model predicting a numeric target, the expected squared error on a new point decomposes as:
Bias² measures how far the model's average prediction (averaged over many training sets) sits from the true value. Variance measures how much the model's prediction swings from one training set to the next. Irreducible noise is randomness inherent to the data-generating process itself — measurement error, unmeasured factors — that no model, however good, can ever predict away. Since noise is fixed, the only two terms an engineer can actually influence are bias and variance, and — as the curve above shows — they trade against each other as complexity changes. That's why regularization, pruning, ensembling, and cross-validation all exist: they're tools for sliding along this curve to the point that minimizes the sum, not for chasing either term to zero on its own.
Imagine training the same model many times, each time on a different random sample drawn from the same underlying data distribution, and plotting each trained model's prediction for one specific input as a dart thrown at a board — the bullseye is the true value. Where the darts land, and how tightly they cluster, separates bias from variance visually.
False, or at best dangerously incomplete. High accuracy specifically on the training data a model was fit to is much more indicative of low bias combined with potentially high variance — the model is flexible enough to fit the patterns in that one dataset, including its noise — than it is proof of a genuinely good, low-total-error model. A model can drive training accuracy arbitrarily high (in the limit, memorizing every row) while its variance quietly explodes, which is exactly the overfitting failure mode. True model quality means low total expected error on new data, which requires balancing bias andvariance together, and can only be evaluated using held-out validation or test data — or cross-validation — never using training-set accuracy alone. Training accuracy answers "did it fit this dataset?" Generalization answers "will it work on the next one?" Those are two very different questions, and only one of them is the one that matters in production.
Explains the two independent sources of a model's prediction error — bias (from a model too simple to capture the true pattern, causing underfitting) and variance (from a model too sensitive to its specific training sample, causing overfitting) — and why total expected error decomposes as bias squared plus variance plus irreducible noise, with model complexity trading one term against the other.
It is tempting to treat a model's error as one undifferentiated number to minimize, but that number is actually the sum of two very different, independently-caused problems. A model can have near-zero error on the data it was trained on and still be a poor model overall, because that low error can come from memorizing sample-specific noise (high variance) rather than from correctly capturing the underlying relationship (low bias). Conflating "fits its training data well" with "has low bias" — and further with "is a good model" — is the single most common misreading of this concept.
For a model f-hat estimating a true function f, the expected squared prediction error on a new point decomposes into three additive terms: Bias² — how far the model's average prediction (averaged across many hypothetical training sets drawn from the same distribution) sits from the true value; Variance — how much the model's prediction varies across those different training sets; and Irreducible noise — randomness in the data-generating process itself that no model can predict away, which sets a hard floor below which total error cannot fall. Because noise is fixed, bias and variance are the only two levers an engineer can pull, and they move in opposite directions as model complexity changes: more complexity (more parameters, deeper trees, higher polynomial degree) lets the model capture more true structure, lowering bias, but also gives it more freedom to fit noise specific to one training sample, raising variance.
The goal of model selection is never to drive bias or variance to zero individually — a model with zero bias (perfectly matching training data every time) typically has very high variance, and vice versa. The goal is to find the complexity level that minimizes their sum, which is what cross-validation, regularization (L1/L2 penalties, dropout, pruning), and ensembling (bagging averages away variance, boosting reduces bias) are all mechanically doing.
The dartboard visualization makes the distinction concrete: imagine retraining the same model architecture on many different random samples from the same underlying distribution, then plotting each trained model's prediction for one fixed test point as a dart. Low bias, low variance is a tight cluster of darts right on the bullseye — the ideal. High bias, low variance is a tight cluster that consistently lands off the bullseye in the same place — the model is stable but wrong, a signature of underfitting. Low bias, high variance is darts scattered widely around the bullseye but centered on it on average — the model is "right on average" but any single trained instance is unreliable, a signature of overfitting. High bias, high variance scatters widely and off-target — the worst of both problems at once.
They are closely linked but not identical. Bias and variance are the two mathematical error components; underfitting and overfitting are the observable symptoms. High bias causes underfitting (poor performance on both training and test data). High variance causes overfitting (great training performance, poor test performance). But a model's total error is always some mix of both terms — real models are rarely purely one or the other.
Sometimes, but not by adjusting model complexity alone — that dial trades one against the other. Getting more (clean) training data reduces variance without increasing bias, since more data makes it harder to fit sample-specific noise while the true pattern stays the true pattern. Better features or a fundamentally better model class can occasionally lower bias without a variance penalty. But within a fixed model class and fixed dataset, complexity is largely a one-dimensional tradeoff.
Bagging — training many high-variance models (like deep decision trees) on different bootstrap samples and averaging their predictions — cancels out a large portion of the variance term, because each model's sample-specific noise is somewhat independent and washes out in the average, while the shared true signal doesn't. Boosting works differently: it builds an ensemble sequentially, primarily targeting bias reduction by focusing each new model on the previous ensemble's errors.
Cross-validation doesn't reduce bias or variance directly — it estimates a model's expected error on unseen data using data the model wasn't fit on, which is exactly what's needed to find the complexity level that minimizes bias² + variance in practice. Without it, you can only observe training error, which reflects bias reasonably well but says almost nothing about variance.
A single good test-set score is strong evidence, but the more rigorous check is cross-validation across several different train/validation splits: if performance stays consistently good across splits, both bias and variance are likely low. If performance swings a lot between splits, variance is high even if any one split happened to look good.
Try our AI and Data Science Studio
More calculators, simulators, and guides for this discipline.