Both penalties shrink weights to fight overfitting. Only one of them can actually zero a weight out — and that difference comes straight from the shape of the penalty, not from a tuning knob.
Ask an engineer why Lasso (L1) produces sparse models while Ridge (L2) doesn't, and a common non-answer is "L1 penalizes harder." It doesn't — for small weights L1's penalty is actually gentler than L2's. The real reason is geometric: L1's penalty has corners sitting exactly on the axes, where one or more coefficients equal zero, and gradient-based optimization gets pulled into those corners. L2's penalty is a smooth circle with no corners anywhere, so it shrinks every weight toward zero without ever landing exactly on it. Same goal — control model complexity by penalizing large weights — completely different mechanism for getting there, and that mechanism is what decides whether you get automatic feature selection or not.
Both regularizers add a penalty term to the loss: L1 adds λ Σ|wᵢ|, L2 adds λ Σwᵢ². Framed as a constrained optimization, minimizing loss subject to a budget on the penalty, the constraint region for L1 is a diamond (an L1 ball) with corners sitting exactly on the coordinate axes, where one or more coordinates are zero. The constraint region for L2 is a sphere with no flat edges or corners at all. When you shrink the elliptical loss contours down until they just touch the constraint boundary, that first point of contact is the solution — and geometrically, an ellipse is far more likely to first touch a diamond precisely at one of its corners than a sphere is to touch it at a point where a coordinate happens to be exactly zero. That is the entire mechanism behind Lasso's automatic feature selection: it isn't that L1 "tries harder" to zero things out, it's that zero is a geometrically favored landing point only for the diamond.
No — increasing L2's λ shrinks every weight further toward zero, but because its gradient (2λw) itself shrinks proportionally as w gets small, the pull weakens right when a weight is closest to zero. The weight asymptotically approaches zero but essentially never lands on it exactly, no matter how large λ gets — you end up with many tiny, non-zero coefficients instead of a sparse model. L1's gradient (λ · sign(w)) stays constant in magnitude all the way down to zero, so once a weight is pushed to zero it gets clipped there by the optimizer (specifically via the soft-thresholding operator in coordinate descent, or the subgradient at 0 in other solvers) and stays exactly at zero. Turning up λ in L1 doesn't just shrink weights harder, it recruits more weights into that sparse zero set. Turning up λ in L2 shrinks everything more, but keeps essentially all features "in" the model at some small nonzero weight. This is also why Elastic Net exists: it mixes both penalties (αλΣ|wᵢ| + (1−α)λΣwᵢ²) to get some sparsity from the L1 term while retaining L2's better behavior with groups of correlated features, where pure Lasso tends to arbitrarily pick just one feature from the group and zero out the rest.
Explains why L1 (Lasso) regularization produces sparse models with exact zero coefficients while L2 (Ridge) only shrinks weights toward zero without reaching it — using the geometric constraint-region view (diamond vs. circle) and the gradient-near-zero view, plus why Elastic Net combines both.
Both L1 and L2 regularization are introduced as "penalties that discourage large weights," which makes it tempting to think of them as two strengths of the same dial. They are not interchangeable at any strength: L1 changes which coefficients end up exactly zero (a form of automatic, embedded feature selection), while L2 changes how much every coefficient shrinks without ever eliminating any of them. Confusing the two leads to real mistakes — for example, expecting a high-λ Ridge model to hand you a short list of "important" features the way Lasso would, when Ridge simply never zeroes anything out by design.
L1 regularization adds λΣ|wᵢ| to the loss function; L2 adds λΣwᵢ². Viewed as constrained optimization (minimize loss subject to a budget on the penalty term), the L1 constraint region is an L1-ball — a diamond in 2D, a cross-polytope in higher dimensions — with corners sitting exactly on the coordinate axes. The L2 constraint region is a smooth ball with no corners. The optimal solution is the point where the loss function's contours first touch the constraint boundary as it expands from the unconstrained optimum. Ellipses touching a diamond very often land on one of its corners (where a coordinate is exactly zero); ellipses touching a smooth sphere almost never land on a point where a coordinate is exactly zero. That single geometric fact is the entire origin of Lasso's sparsity.
Equivalently, look at each penalty's subgradient at w=0: L1's is a constant λ in magnitude regardless of how close w already is to zero, so coordinate descent's soft-thresholding step can push a small coefficient all the way to (and hold it at) zero. L2's gradient is 2λw, which shrinks toward zero as w does, so the pull weakens exactly when it would need to be strongest to finish the job — the coefficient approaches zero asymptotically but essentially never arrives there in finite iterations.
Use L1 (Lasso) when you suspect many features are irrelevant and want the model itself to perform feature selection — the resulting sparse coefficient vector is also easier to interpret and to deploy (fewer features to compute and monitor in production). Use L2 (Ridge) when you believe most features carry at least some signal and, critically, when features are correlated: L2 tends to spread weight evenly across a correlated group, while L1 tends to arbitrarily zero out all but one member of that group, which can make the selected feature set unstable across resamples of the training data. Elastic Net (a weighted mix of both penalties) is the common practical compromise — sparsity from the L1 term, stability under correlated features from the L2 term — and is frequently the default choice in production pipelines when neither pure Lasso nor pure Ridge behavior is clearly preferred. In deep learning, "weight decay" is effectively L2 regularization applied to network weights (with some important optimizer-specific caveats, e.g. how weight decay interacts with adaptive optimizers like Adam versus plain SGD), and L1 penalties are less commonly used directly on dense network weights since sparsity in that setting is usually pursued through other means (pruning, dropout, architecture choices).
Not necessarily at literally the same λ value — the two penalties aren't on the same numerical scale, so comparing them at equal λ isn't meaningful. But for any reasonable range of λ, L1 will produce exact zeros for some subset of coefficients, and L2 essentially never will (outside of degenerate cases). Sparsity is a structural property of L1, not a matter of tuning L2 hard enough.
No. Ridge keeps every feature in the model with a nonzero (if shrunk) coefficient. It reduces effective model complexity and variance without literally removing any feature — if you need dimensionality reduction or a shorter feature list, Lasso or an explicit feature-selection step is the better tool.
When two or more features are highly correlated, many different splits of weight between them achieve nearly identical loss. L1's corner-seeking geometry tends to arbitrarily pick one of them to keep and zero out the rest, and which one gets kept can be sensitive to small changes in the training data or even the random seed of the solver. L2 instead tends to spread the weight roughly evenly across the correlated group, which is more stable, and Elastic Net is often used specifically to get some sparsity while avoiding this instability.
Conceptually yes — classic weight decay directly implements an L2 penalty on the weights. But with adaptive optimizers like Adam, naively adding an L2 term to the loss interacts with the per-parameter adaptive learning rates in a way that doesn't behave like true weight decay; this is why "decoupled weight decay" (as in AdamW) was introduced to apply the decay separately from the gradient-based update, restoring the intended L2-like shrinkage effect.
Yes — this is exactly what Elastic Net does, combining both penalty terms with a mixing parameter (often called α or l1_ratio) that controls how much of the total regularization budget goes to each. It is a common default choice in practice because it captures some of Lasso's sparsity while retaining Ridge's better-behaved handling of correlated features.
Try our AI and Data Science Studio
More calculators, simulators, and guides for this discipline.
AI & ML Mastery (Full Access)
✨ Premium ContentA zoomable interactive reader — free preview, then unlock the full set.