Why a model that's "too good" on the data it trained on is a red flag — not proof it learned anything.
It feels intuitive to judge a model by how well it fits the data you trained it on. It's also one of the most reliable ways to fool yourself. A model can score almost perfectly on its training set two very different ways: by genuinely learning the real underlying pattern, or by simply memorizing the specific rows, noise, and quirks of that exact dataset. Only one of those generalizes to new data. Telling them apart is the entire point of the underfitting/overfitting framework — and it's why no serious model is ever judged on training performance alone.
Underfitting happens when a model is too simple — or trained too little — to capture the real structure in the data. Fit a straight line to a curve and it will systematically miss the trend everywhere: not just on new data, but on the very data it was trained on. Overfittingis the opposite failure: a model complex enough (or trained long enough) to start memorizing the specific noise and idiosyncrasies of the training set rather than the general pattern underneath it. It can look outstanding on training data — sometimes near-perfect — while performing poorly on anything it hasn't seen, because what it learned was partly the dataset's quirks, not the phenomenon itself. The goal sits between the two: just enough capacity to capture the true pattern, without enough to also memorize the noise.
A model's training error can only ever tell you how well it fits the examples it already saw — and a sufficiently complex model, or one trained long enough, can always drive that number down, whether or not it has learned anything that generalizes. A held-out validation set breaks that loop: because the model never adjusted its parameters based on those examples, good performance there can't be explained by memorization. That's why the standard practice is to watch validation error as complexity or training time increases, and stop — via early stopping, regularization, or simply choosing a less complex model — right around the point where validation error stops improving and starts creeping back up, even though training error is still happily decreasing. That inflection point, not the training curve, is where the real signal ends and memorization begins.
Not necessarily — and taken to the extreme, it's often a warning sign rather than an achievement. Extremely low training error, especially when it comes from a very complex model or very long training, is exactly what memorization looks like from the outside: the model has become good at reproducing answers for the specific examples it saw, which is not the same thing as learning the underlying pattern those examples came from. The only way to actually judge whether a model has learned something genuinely generalizable is to evaluate it on data it never saw during training— a held-out validation or test set — not by how tightly it fits the data it was trained on. A training score alone can't distinguish "learned the pattern" from "memorized the noise;" only unseen data can.
Explains the two failure modes of model fitting — underfitting (too simple to capture the real pattern, so both training and test error stay high) and overfitting (complex or trained enough to memorize training-set noise, so training error looks great while test error stays high) — and why the ideal model complexity or stopping point is found by monitoring a held-out validation set, not training performance.
It is tempting to treat training accuracy as the scoreboard for a model, since it is the number that is easiest to watch improve during development. But training error only measures how well a model reproduces the examples it was directly optimized against. A model can drive that number arbitrarily low simply by having enough free parameters to memorize the training set outright, the same way a lookup table can "perfectly predict" every row it was built from without understanding anything about the underlying relationship. Low training error is consistent with genuine learning, but it is equally consistent with memorization — the number alone cannot tell the two apart.
Underfitting is a bias problem: the model's assumptions (e.g., "the relationship is a straight line") are too restrictive to represent the real pattern, so it performs poorly everywhere, training and test data alike. Overfitting is a variance problem: the model is flexible enough to fit the specific noise in one particular training sample, so its predictions swing based on which noisy sample it happened to see, and it fails to reproduce that noise-fitting success on a different sample of the same underlying process. Model capacity, training duration, and regularization are all levers for moving along this trade-off toward the point that minimizes error on data the model has not seen.
The standard approach is to hold out a validation set the model never trains on, then track both training and validation error as model complexity or training epochs increase. Training error decreases (or stays flat) throughout, since more capacity or more training time can only ever help the model fit what it has already seen. Validation error decreases at first, alongside training error, as the model captures more of the true pattern — but eventually turns upward again once additional capacity starts being spent on memorizing training-set-specific noise instead of general structure. The point where validation error is lowest, just before that upturn, is the target: stop training there (early stopping), or cap model complexity there, or add regularization strong enough to hold the model near it.
Model capacity (parameters relative to data) is the classic cause, but it is not the only one. Training a smaller model for too many epochs on too little data, or on data with a lot of noise relative to signal, can produce the same effect: the model eventually starts fitting quirks of that specific training sample rather than the general relationship, even without an enormous parameter count.
Not in the same region of the trade-off, but different parts of a model can behave differently on different subsets of the data — for example, a model might underfit a rare category with too few examples to learn from while overfitting a common category it has seen thousands of times. Evaluating performance by subgroup, not just in aggregate, catches this.
Repeatedly checking performance on the same held-out set to make decisions (which model, when to stop, which hyperparameters) lets that set quietly influence training by proxy, which can make its error an overly optimistic estimate of true generalization. That is why practice typically keeps a validation set for tuning decisions and a separate, untouched test set for the final, one-time performance check.
Often yes. More data makes it harder for a model to memorize specific noise, because the noise averages out across a larger and more varied sample while the true underlying pattern remains consistent. It does not help with underfitting, though — an underfit model is limited by its own simplicity, not by a shortage of examples to learn from.
Regularization (penalizing overly large or complex parameter values), dropout in neural networks, reducing model capacity, gathering more or more varied training data, data augmentation, and early stopping once validation error stops improving are all standard tools, often used in combination rather than alone.
Try our AI Engineering Studio
More calculators, simulators, and guides for this discipline.