A Flowchart of Questions
A decision tree is the most human classifier there is: it is just a flowchart of yes/no questions. Start at the top, ask a question about the data, follow the branch, ask another, until you reach an answer at the bottom.
To classify something new, you drop it in at the top and let it fall through the questions to a leaf, and the leaf is your prediction. That is the whole algorithm at prediction time, and anyone can read it out loud.
How It Picks the Questions
The cleverness is choosing which question to ask at each node. The tree tries every possible split (every feature, every threshold) and keeps the one that best separates the classes, the one that leaves the two resulting groups as pure as possible.
- Pure = almost all one class. All apples left, all oranges right. A great split.
- Mixed = half and half on each side. A useless split.
Purity is scored with Gini impurity or entropy. The tree greedily grabs whichever split cuts impurity the most, then repeats the exact same trick on each side, over and over.
Build the tree once by chasing purity downward. Then every prediction is just a walk down the branches.
Growing It, and the Overfitting Trap
You keep splitting until each leaf is pure. But there’s a trap: let a tree grow all the way and it will make a separate leaf for every single training point, perfect on the training set, useless on anything new. Textbook overfitting.
The fixes:
- Limit the depth of the tree.
- Require a minimum number of points in each leaf.
- Prune: grow it fully, then cut weak branches back.
Depth is the familiar bias-variance knob: a shallow tree underfits (high bias), a deep tree overfits (high variance).
What the Boundary Looks Like
Because every split is a cut on one feature (“income > 30k”), the boundaries are always axis-aligned: pure horizontal and vertical cuts that carve the space into rectangular boxes.
A tree tiles the plane with boxes, one prediction per box, very different from logistic regression’s slanted line or the SVM’s smooth curve. Watch the boxes multiply as depth grows: clean and blocky when shallow, shattered into tiny overfit patches when deep.
Why They’re Loved, and the One Weakness
Trees are a favourite because they are:
- Readable, you can literally print the rules and hand them to a human.
- Flexible with data, numbers and categories side by side, and no feature scaling needed.
- Interaction-aware, they catch “high income and low debt” automatically.
But there is one real flaw. A single tree is unstable, high variance: nudge a few training points and you can get a completely different tree.
That instability is not a dead end, it is the whole setup for what comes next. Combine many trees and their individual noise averages away. That idea, random forests and boosting, turns this shaky little classifier into the reigning champion of tabular data. We build it next.