The Opposite Strategy
Random forests grew hundreds of trees in parallel, independently, then averaged them. Boosting does the exact opposite: it grows trees one at a time, in a chain, and every new tree has a single job, fix the mistakes of the ones before it.
Two philosophies, same building block:
- Random forest = a crowd of independent voters.
- Boosting = a relay of specialists, each cleaning up what the last one missed.
Weak Learners, Chained
Here’s the twist that makes it work. Boosting deliberately uses weak trees, often just stumps (a tree with a single split), each barely better than a coin flip on its own.
Alone, a stump is almost useless. But chain hundreds of them, each patching the previous errors, and together they become one of the most accurate models in machine learning.
Gradient Boosting: Chase the Leftover Error
The cleanest version, gradient boosting, works like this when predicting a number:
- Start with a dumb guess, say the average of all the answers.
- Measure the errors (residuals): how far off is each point?
- Train a small tree to predict those errors.
- Add its output (shrunk by a learning rate) to the guess. You’re now a bit closer.
- Measure the new, smaller errors, fit another tree to them, add, and repeat.
Watch the fit above: a flat line, then each tree bends it closer while the red error bars shrink. Every tree is one small step that reduces what’s left over.
That is literally gradient descent: each tree is a single step downhill on the error. Hence the name, gradient boosting.
AdaBoost: Turn Up the Volume on Mistakes
The original flavour, AdaBoost, does it a little differently. Instead of fitting the errors, it reweights the data: after each weak stump, it makes the misclassified points heavier, so the next stump is forced to focus on them.
The points it keeps getting wrong grow louder and louder until some tree finally handles them. The final answer is a weighted vote, where the more accurate stumps get more say.
The Catch: It Can Overfit
This is the key difference from random forests:
| Random forest (bagging) | Boosting | |
|---|---|---|
| Trees are built | in parallel, independent | in sequence, dependent |
| Mainly reduces | variance | bias |
| Add too many trees? | never hurts | eventually overfits the noise |
| Usual accuracy | strong | often the best |
Because every boosting tree chases the remaining errors, if you add too many it will eventually start fitting noise. So the number of trees and the learning rate are real knobs to tune, usually with early stopping. More power, more care.
The Champions of Tabular Data
Modern gradient-boosted trees, XGBoost, LightGBM, and CatBoost, are the reigning champions on structured, tabular data. If you have a spreadsheet of numbers and want the single best prediction, this is very often what wins, and it dominates Kaggle competitions.
Bagging made a shaky tree stable. Boosting makes a weak tree powerful. Between them, ensembles of trees rule everything that lives in a table.
That closes the classical supervised toolbox. There’s one job left before we leave it: learning how to tell whether a classifier is any good, which is trickier than it sounds. That’s evaluating a classifier, next.