If the world were perfect, we probably wouldn't need regularization at all. But real loss surfaces, past two dimensions, are rarely one clean slope down to the answer. They're full of bumps, shallow pits, and false valleys that look like the bottom but aren't. An optimizer rolling down that surface can fall into one of those pits early and never find its way out.
That's the deep-end version, further down this post. Here's the simplest possible one first, live: two weights, one smooth bowl, three ways of controlling where the ball is allowed to stop. That's basically the entire reason lasso regression can turn a weight off completely, and ridge regression can't. Drag the slider below.
0.75
No penalty
w₁, w₂—
distance to true weights—
L1 (lasso)
w₁, w₂—
distance to true weights—
L2 (ridge)
w₁, w₂—
distance to true weights—
loss contoursunconstrained fittrue weightsconstrained solution
Fig. 1 — same loss, three constraints. Drag the regularization amount and watch where L1 lands versus L2.
The shape of the constraint
Here's what you just dragged. It's the same regression problem in two dimensions, one weight on each axis. The rings are contours of the loss: every point on a ring gives the same squared error, and the dot at the center is the best fit possible with no restriction at all, wherever the data happens to pull it. It is not the origin, there's no reason it should be. The two axis lines only mark w₁ = 0 and w₂ = 0, so you can see a weight land exactly on one.
No penalty just picks that dot and stops. L1 and L2 both add a rule instead: the weights are only allowed to live inside some region around zero. The solution becomes wherever that region first touches a ring of loss, nothing more exotic than that.
The two regions differ in exactly one way that matters. A circle's edge is round everywhere, so as it shrinks, the touching point slides along smoothly, with no reason to prefer any particular spot. A diamond has corners sitting right on the axes, and shrinking it tends to pull the touching point onto one of those corners, because a corner reaches further out relative to its size than a flat edge does. Land on a corner, and the weight there is exactly zero. Not small. Zero.
A question that comes up almost every year: is it that the loss surface gets jagged, and regularization smooths it out?
Close, but the jaggedness is in the wrong place. The shapes above were always perfectly smooth, an ellipse, a diamond, a circle. Regularization strength never roughens them up. What actually gets jagged is the fitted function, which is exactly what's coming up next: give a flexible model too much freedom and too little data, and it swings its weights wherever it takes to chase every bump of noise. Penalize large weights, and that freedom disappears. A smoother fit falls out as a side effect, not as the goal.
Same idea, an actual fit
Geometry is convincing on paper, but it's worth watching it happen to a real curve. Below are ten points sampled from a sine wave with some noise added, fitted with a degree 9 polynomial: ten coefficients for ten points. That's exactly enough freedom to pass through every single point with zero error.
With no penalty, that's exactly what happens, and it isn't pretty in between the points. Nothing was ever asked to behave reasonably where there wasn't a data point holding it down, so it doesn't.
Turn the strength up and two different things happen. Ridge pulls every coefficient down a little, smoothly, and the curve calms down gradually. Lasso deletes coefficients outright, and once enough of the higher order terms are gone, what's left tracks the actual sine wave underneath the noise. Not because lasso knows about sine waves, but because a simpler model needed fewer of those ten knobs to describe this one well.
The bars under each chart are the coefficients themselves. Watch the gray ones show up under lasso as you turn the strength up. Those are exact zeros, not rounding.
0.32
No penalty
L1 (lasso)
L2 (ridge)
true functionobserved datafitted curve
Fig. 2 — same ten points, same polynomial, three penalties. Bars below each chart show the nine fitted coefficients.
Turn the same knob further. Sixteen points this time, fit with a degree fifteen polynomial: sixteen coefficients for sixteen points, still exactly enough freedom to hit every one of them exactly.
Six more parameters does not sound like much. It is. With no penalty, the fit still passes through every point, but between points it swings far harder than the degree nine version did: dozens of units near the edges of the data, and thousands once you step just outside it. Same mechanism as before, just with more room to misbehave.
Ridge and lasso handle it the same way they did a moment ago. Watch how much regularization it now takes to drag the curve back down to something reasonable.
0.32
No penalty
L1 (lasso)
L2 (ridge)
true functionobserved datafitted curve
Fig. 2b: sixteen points now, fit with a degree fifteen polynomial instead of nine. Same three penalties. Watch "None" swing clear off the top of the chart between points, something Fig. 2's milder version never quite does.
One more way to look at the same knob. Train the model on the same ten points as before, but this time also test it on ten different points it never got to see, sampled from that same sine wave. Two lines fall out of that: how wrong the model is on the points it trained on (dashed, only ever gets worse, since more regularization means trying less hard to match them), and how wrong it is on the new points (solid, gets better for a while, then worse again). The second line is the one you actually care about.
0.32
L1 (lasso)
L2 (ridge)
training errorheld-out errorbest on held-out datagap = overfitting
Fig. 2c — dashed: error on the training points, always gets worse. Solid: error on ten new points the model never saw, gets better then worse. The dot marks the regularization amount that does best on those new points, that's the number cross-validation would pick for you. The vertical segment connecting the two lines shows the gap at the slider's current setting, wide is overfitting, thin is underfitting.
Try this: push the L1 slider on the fit panel (Fig. 2) as high as it goes, then bring it down slowly, and note roughly where the curve stops looking like garbage. Now try Fig. 2c's own slider and see where its dot actually sits. That dot is what cross-validation finds: the strength that does best on data the model hasn't seen, instead of eyeballing a chart. See how close your eyeball estimate got.
Where that leaves us
No penalty: the fit is only as good as the data lets it be. With few points and a flexible model, that means it chases noise, and the price is paid off the sample points, where the curve is free to swing to extremes.
L2 (ridge): penalizes the sum of squared weights. Big weights cost disproportionately more than small ones, so it shrinks everything toward zero smoothly and roughly proportionally. Nothing lands on exactly zero, because the circular boundary has no corners to land on.
L1 (lasso): penalizes the sum of absolute weights. The diamond-shaped budget has corners sitting exactly on the axes, so the optimal point keeps landing there, some weights hit exactly zero. That's feature selection for free: a simpler model, not just a smaller one.
One more place this shows up: how the optimizer gets there
Regularization doesn't just decide which weights survive, it also changes the shape of the hole an optimizer is trying to fall into. Different optimizers take different paths down that hole, and they settle at different speeds. Want to see how? The optimizers post races six of them, SGD, Momentum, Nesterov, RMSProp, Adam, and Prodigy, on both simple and complex loss surfaces. Watching them actually run is more convincing than any static formula.