Regularization

The Problem We’re Solving

Overfitting is high variance: the model is too flexible, so it chases noise. We could just switch to a simpler model, but that throws away power we might need.

Regularization is the smarter move: keep the flexible model, but discourage it from using that flexibility unless it genuinely pays off.

Change What “Best” Means

Until now, “best model” meant exactly one thing: lowest fit error (lowest MSE). Regularization bolts a second term onto that goal:

$\text{new loss} = \text{fit error} + \lambda \times (\text{complexity penalty})$

Now the model is graded on two things at once: fit the data and stay simple. It may only spend a big, complicated weight if that weight earns its keep by improving the fit a lot. Otherwise the penalty makes it not worth it.

It becomes a tug-of-war:

• the fit term pulls the weights toward whatever matches the training data (noise and all),
• the penalty term pulls every weight back toward zero.

Why Shrinking Weights Means Smoothing

Here is the insight that makes the whole thing click. Wiggly curves need big weights. Those sharp swings up and down require large coefficients. A gentle, smooth curve needs only small ones.

So when the penalty drags the weights toward zero, it is literally forcing the curve to be smoother. Watch it happen as we turn the penalty up:

Punishing big weights is the same thing as demanding a calmer, simpler model.

Lambda: the Strength Knob

The $\lambda$ (lambda) out front sets how hard the penalty pulls:

• $\lambda = 0$ → no penalty at all. Straight back to overfitting.
• small $\lambda$ → gentle smoothing. Usually the sweet spot.
• large $\lambda$ → the penalty dominates, weights get crushed toward zero, and the model turns too simple. Now you’re underfitting.

Look closely at what that means: raising $\lambda$ trades variance for bias.

$\lambda$ is a dial straight onto the bias-variance tradeoff. Tuning it is choosing where you sit on the U-curve from the last note.

Two Flavors of Penalty

There are two classic ways to measure “complexity,” and they behave very differently:

• L2 (Ridge): penalty = the sum of the weights squared. It shrinks all the weights smoothly toward zero, but almost never to exactly zero. Everything just gets a little smaller.
• L1 (Lasso): penalty = the sum of the absolute values of the weights. It pushes some weights to exactly zero, switching those features off completely. The result is a sparse model that has quietly selected which features matter.

L2 says “everyone tighten your belt a little.” L1 says “cut the dead weight entirely.” L1 gives you feature selection for free.

Why This Reaches All the Way Up

This is not a one-off classical trick. The exact same idea runs straight up to frontier models:

• Weight decay, used to train essentially every large neural network, is L2 regularization.
• Dropout, which you’ll meet in deep learning, is another regularizer chasing the same goal.

Fighting variance by penalizing complexity is a theme you will meet again at every scale, from a wiggly little curve to a trillion-parameter model. Now that you can fit a model and stop it from overfitting, you have the complete classical picture. Next we start turning these ideas into the building block of deep learning: the neuron.