The Bias-Variance Tradeoff

Two Ways to Be Wrong

Last note we met underfitting and overfitting. Now we name what’s really going on underneath, because these two failures have opposite personalities:

• Bias is being too rigid. The model’s built-in assumptions are wrong, so it misses the same way every time, no matter what data you give it.
• Variance is being too twitchy. The model is so flexible it reshapes itself around whatever exact points it happened to train on, chasing noise.

Bias is being confidently wrong. Variance is being unable to sit still.

A Scale You Can Feel

Picture two broken bathroom scales.

• The high-bias scale always reads exactly 5 lbs too heavy. Step on it ten times, same wrong number every time. Consistent, but systematically off.
• The high-variance scale gives a different random number every time, sometimes 10 high, sometimes 8 low. On average it might be right, but no single reading can be trusted.

Back to our curves:

• A straight line on curved data is high bias: wrong the same way regardless of which houses you sample.
• A wild wiggly curve is high variance: hand it a slightly different sample and it produces a completely different shape.

The Dartboard

This is the picture to burn into memory. The bullseye is the truth, and each dart is the model trained on one sample of data.

Read it like this:

• Bias = how far the center of the cluster sits from the bullseye.
• Variance = how spread out the darts are.

Seeing Variance Directly

Variance is the slippery one, so let’s make it literal. Take a simple model and a complex model, and re-train each one several times on different random samples of data:

• The simple model barely budges. Every re-training gives almost the same line. Low variance, but it’s a straight line missing a curve, so it’s biased.
• The complex model thrashes. Each sample produces a wildly different wiggle. High variance, even though on average it’s near the truth.

Variance is how much your model changes when the training data changes. A model that reinvents itself for every sample hasn’t learned a pattern. It has learned an accident.

The Tradeoff (the painful part)

Here’s what makes it a tradeoff and not a to-do list. As you turn up model complexity:

• Bias goes down ↓ (more flexibility lets it bend to the true pattern)
• Variance goes up ↑ (more flexibility makes it cling to the noise)

You cannot drive both to zero. Crank complexity to kill bias, and variance balloons. Simplify to kill variance, and bias creeps back in. The best model lives at the balance point, which is exactly the bottom of the U-curve from last note, now taken apart into its pieces.

The Whole Error, in Three Pieces

This is the equation the whole note was building toward. Your expected error on new data splits cleanly:

$\text{Total error} = \text{bias}^2 + \text{variance} + \text{irreducible noise}$

That last term is the noise floor from your earlier question, the randomness baked into the data itself. It is why test error can never reach zero. Balancing the first two, while accepting the third, is machine learning in one line.

Which Lever to Pull

Diagnosing which problem you have tells you exactly what to do:

• Too much bias (underfitting: high train error AND high test error) → a more complex model, better features.
• Too much variance (overfitting: low train error but high test error) → more data, a simpler model, regularization, or averaging many models (ensembles, coming later).

One teaser for much later: at truly enormous scale, deep networks bend this classic story in a surprising way called “double descent,” where making the model even bigger can lower variance again. We’ll revisit it once we reach frontier models. For everything up to that point, the tradeoff rules.