One Tree Is Too Jumpy
A single decision tree has one nasty habit, from the last note: it is high-variance. Change just a handful of training points and the whole tree reshapes itself, and its prediction jumps all over the place.
Picture a tree trying to trace a trend in noisy data. It doesn’t draw a smooth line, it draws a jagged staircase, bending to touch every little bump of noise.
The tree is flexible, which is good, but unstable, which is bad. We want the flexibility without the wobble.
The fix is one of the oldest ideas around: ask a crowd.
The Wisdom of Crowds
Ask one person to guess how many jellybeans are in a jar and they’ll be way off. Ask a thousand people and average their guesses, and the average is often eerily close to the truth.
Why? Because the mistakes point in random directions. Some guess too high, some too low, and when you average them, the errors cancel each other out.
A random forest is exactly this: a crowd of decision trees. Grow hundreds of them, let each one make its jittery guess, and average the jitter away.
Watch the Jitter Disappear
Here is that averaging in action. One tree is a jagged mess. Add more, each fitting a slightly different version of the data, and average their predictions:
- 1 tree → a wild staircase, overfitting every point.
- A few trees → already calmer.
- The whole forest → a smooth curve that follows the real trend and ignores the noise.
Same flexibility. None of the wobble.
But the Trees Must Disagree
There’s a catch. If all thousand jellybean-guessers just copied one person, averaging would do nothing. A crowd only helps when its members are independent and diverse. A thousand identical trees is just one tree.
So a random forest deliberately forces every tree to be different, in two ways:
1. Bagging (bootstrap sampling): each tree trains on its own random resample of the data, drawn with replacement. Some points get picked twice, some get left out.
2. Random features: at each split, a tree may only look at a random subset of the features, never all of them. This stops every tree from leaning on the same dominant feature.
Together these guarantee hundreds of trees that overfit, but each in a different direction, which is precisely what makes the averaging work.
Predicting: A Show of Hands
To classify a new point, you drop it through every tree, collect their votes, and take the majority (or the average, for regression).
Any single tree might be wrong. The crowd, almost never.
Single Tree vs Forest
| One decision tree | Random forest | |
|---|---|---|
| Bias | low | low |
| Variance | high (jumpy) | low (averaged out) |
| Overfits? | easily | very hard to |
| Readable? | yes, print the rules | no, it’s hundreds of trees |
| Tuning | fussy | almost none |
This is the whole trick of bagging: it slashes variance while leaving bias untouched. You keep the tree’s power and delete its instability.
In Practice
Random forests are a workhorse. They’re robust, need barely any tuning, are almost impossible to overfit (more trees never hurts), and hand you feature importances for free. The only real cost is that you’ve traded one readable tree for a black box of many.
On tabular data, a random forest is the strong baseline everyone tries first.
Next comes the other way to combine trees, with the opposite philosophy. Instead of a crowd of independent voters, imagine a relay of specialists, where each new tree studies the mistakes of the ones before it and fixes them. That is boosting.