Why “Accuracy” Lies
The obvious way to grade a classifier is accuracy: the fraction it gets right. It is also dangerously misleading.
A disease affects 1 in 1000 people. Here is a detector that is 99.9% accurate: it says “healthy” to everyone. It’s right 999 times out of 1000, and it catches zero sick people. Worthless, yet the score looks superb.
Whenever one class is rare, accuracy hides the truth. We need sharper tools, and they all start from one small table.
The Confusion Matrix
Every yes/no prediction falls into one of four boxes:
| Actually yes | Actually no | |
|---|---|---|
| Said yes | ✅ true positive | ⚠️ false positive (false alarm) |
| Said no | ❌ false negative (a miss) | ✅ true negative |
Every other metric is just arithmetic on these four counts. And crucially, the two errors are not equally bad:
- A false alarm (FP): a spam filter trashes a real email.
- A miss (FN): a cancer screen tells a sick patient they’re fine.
Which one you dread more depends entirely on the problem.
Precision and Recall
The clearest way to feel these two: imagine fishing with a net in a lake that holds fish (the positives) and rocks (the negatives). You cast, and you haul up a netful.
Precision asks: of everything in my net, how much is actually fish?
Recall asks: of all the fish in the lake, how many did I manage to catch?
In classifier terms:
- Precision is about your net’s purity. Low precision means it’s full of rocks, lots of false alarms.
- Recall is about your coverage. Low recall means most fish got away, lots of misses.
And here’s the rub, they pull against each other:
- Cast a huge net → you catch every fish (recall near 100%), but drag up piles of rock too (precision drops).
- Cast a tiny, careful net → everything in it is fish (precision near 100%), but you leave most of the lake untouched (recall drops).
So which do you chase? Follow the cost of the error:
| Problem | Chase | Because |
|---|---|---|
| Cancer screening | recall | never miss a sick patient; a false alarm just means one more test |
| Spam filter | precision | never bin a real email; a missed spam is only annoying |
(Want a single number? F1 is the harmonic mean of the two, high only when both are high.)
The Threshold Controls the Tradeoff
Most classifiers don’t output yes/no, they output a probability, and you pick a threshold to make the call. That one dial is the precision-recall tug of war:
- Lower the threshold → flag more → recall up, precision down.
- Raise it → flag fewer → precision up, recall down.
You are not stuck with the default of 0.5. You slide it to wherever the costs of your problem sit.
Judging Across All Thresholds: ROC and AUC
To grade a model independent of any one threshold, sweep the threshold across its whole range and trace the ROC curve: the true-positive rate against the false-positive rate.
A perfect model bows into the top-left corner; a coin flip rides the diagonal. The area underneath, the AUC, collapses all of that into one honest number:
AUC ranges from 0.5 (random) to 1.0 (perfect), and it doesn’t depend on where you set the threshold. It’s the go-to summary of a classifier’s raw quality.
The Takeaway
- Never trust accuracy alone, especially with rare classes.
- Read the confusion matrix to see which errors you’re making.
- Pick precision or recall based on which mistake actually costs you.
- Use AUC to compare models threshold-free.
That completes the classical supervised toolbox, from linear regression all the way to boosted trees, plus how to judge any of them. Next we drop the labels entirely and go hunting for structure on our own: unsupervised learning, beginning with clustering.