Why training a model is just walking downhill on a loss landscape — and why that walk can get stuck.
"Training" a machine learning model sounds mysterious, but the core algorithm underneath almost all of it is mechanically simple: measure how wrong the model currently is, figure out which direction makes it less wrong, and take a step that way. Repeat thousands or millions of times. That algorithm is gradient descent, and nearly everything interesting about why training is fast, slow, stable, unstable, or stuck comes down to two things — how big a step you take, and what the shape of the landscape you're walking on actually looks like.
A model's loss (or cost) function measures how wrong its predictions currently are, as a function of its parameters. Gradient descent computes the gradient of that loss — the direction of steepest increase— at the model's current parameter values, then updates the parameters by stepping in exactly the opposite direction: steepest decrease. The size of each step is scaled by a hyperparameter called the learning rate. Compute the gradient, step downhill, repeat — until the loss stops meaningfully decreasing.
The gradient is a purely local measurement — it describes the slope at the current parameter values, not the shape of the entire landscape. That's exactly why it works so well on a smooth, bowl-shaped (convex) loss surface, and exactly why it can get trapped on the bumpy, non-convex surfaces typical of real neural networks: at a local minimum or a saddle point, the gradient is momentarily zero (or nearly so), and plain gradient descent has no built-in mechanism telling it a better solution exists somewhere else. Techniques like momentum (carrying velocity through flat regions), adaptive learning rates like Adam and RMSprop (adjusting step size per-parameter based on recent gradient history), and the inherent stochasticity of mini-batch SGD (different random batches nudge the path slightly differently each step) all exist specifically to help the optimizer escape these traps rather than stall in them permanently.
No — and this is especially true for the non-convex loss landscapes typical of real neural networks. Gradient descent only guarantees convergence to a minimum reachable from wherever it started, given its step sizes — not necessarily the global minimum, the single best solution across the entire parameter space. Depending on initialization, learning rate, and the exact shape of the landscape, two training runs on the same model can land in different local minima with different final loss values. This is precisely why practical training pipelines lean on momentum, adaptive learning-rate methods, and multiple random initializations or restarts: not to guarantee the global optimum, but to meaningfully improve the odds of escaping a poor local minimum rather than relying on plain gradient descent alone.
Explains gradient descent, the core iterative optimization algorithm behind model training — how the learning rate shapes convergence, and why non-convex loss landscapes let the algorithm get stuck in a local minimum or saddle point instead of finding the true global minimum.
Gradient descent is often described as "the model learning," which makes it sound smarter than it is. It is a purely local, mechanical procedure: compute the slope of the loss function at the current parameters, step in the opposite direction, repeat. It has no view of the overall shape of the loss landscape and no memory of alternatives it hasn't visited — which is exactly why its behavior depends so heavily on learning rate and starting point.
The learning rate scales how big each parameter update is. Too small, and the algorithm crawls toward the minimum, requiring an impractically large number of iterations to converge. Too large, and each step can overshoot the minimum entirely — causing the loss to oscillate back and forth around the minimum, or even diverge, getting worse with each step instead of better. Finding a workable learning rate (or using an adaptive method that adjusts it automatically) is one of the most consequential choices in training a model.
Most real neural network loss functions are non-convex — their landscape has many valleys, not just one. Plain gradient descent is only guaranteed to converge to a minimum reachable from its starting point, not the global minimum across the whole landscape. It can get stuck in a local minimum (lower than everywhere immediately nearby, but not the lowest point overall) or stall at a saddle point (where the gradient is near zero without being an actual minimum in every direction). Momentum, adaptive learning-rate methods like Adam and RMSprop, and the inherent noise of mini-batch stochastic gradient descent all exist specifically to help the optimizer escape these traps.
The gradient is a vector pointing in the direction of steepest increase of the loss function at the current parameter values. Gradient descent moves the parameters in the exact opposite direction — steepest decrease — because that's the direction that reduces loss fastest from where the model currently stands.
Often through experimentation, learning rate schedules (starting larger and decaying over training), or adaptive optimizers like Adam and RMSprop that adjust the effective step size per parameter automatically based on recent gradient history — reducing the need to hand-tune a single fixed value.
Not necessarily. In high-dimensional neural network loss landscapes, many local minima tend to have similarly good (low) loss values in practice, so getting stuck in "a" local minimum is often fine. But there is no guarantee of this in general, which is exactly why techniques to help escape poor local minima are standard practice.
At a local minimum, the loss increases in every direction you move from that point. At a saddle point, the gradient is also near zero, but the loss increases in some directions and decreases in others — it just is not a minimum in every direction. Saddle points are actually far more common than local minima in high-dimensional spaces and can slow down training even when they are not permanent traps.
Try our AI and Data Science Studio
More calculators, simulators, and guides for this discipline.