What is a Confusion Matrix?
A confusion matrix is a table that counts how many items of each true class a classification model assigned to each predicted class. It is also called an error matrix.
How to read it
In scikit-learn, each row is a true class and each column is a predicted class. The entry in row i and column j counts the observations of true class i that the model put in class j. Other texts swap the axes, so check the labels first.
The cells on the diagonal are correct predictions. The cells off the diagonal are errors, and each one shows which class the model confused with which.
The binary case
A model with two classes, positive and negative, gives four cells.
| Predicted positive | Predicted negative | |
|---|---|---|
| Actual positive | true positive (TP) | false negative (FN) |
| Actual negative | false positive (FP) | true negative (TN) |
A false negative is an actual positive that the model called negative. A false positive is an actual negative that the model called positive.
Metrics from the cells
- Accuracy = (TP + TN) / all observations.
- Precision = TP / (TP + FP).
- Recall = TP / (TP + FN).
- F1 = 2TP / (2TP + FP + FN).
Example from the scikit-learn documentation: 8 observations give TN = 2, FP = 1, FN = 2 and TP = 3. Accuracy is 5/8 = 0.625, precision is 3/4 = 0.75, recall is 3/5 = 0.6 and F1 is 6/9 = 0.67.
For more than two classes, compute precision and recall for each class. Balanced accuracy is the average of the recall of all classes. Cohen’s kappa and the Matthews correlation coefficient (MCC) also use the whole matrix. Kappa is between -1 and 1. The MCC has a maximum of +1, and with more than two classes its minimum is between -1 and 0.
Why not accuracy alone
On imbalanced data, accuracy can hide a model that finds nothing. The matrix shows the actual positives and where they went, so it exposes this at once.
Frequently Asked Questions
- Are the rows the true class or the predicted class?
- It depends on the source. In scikit-learn, each row is the true class and each column is the predicted class. Other texts swap the axes. Read the axis labels before you read the numbers.
- What is the difference between precision and recall?
- Precision is the share of positive predictions that are correct: TP / (TP + FP). Recall is the share of actual positives that the model finds: TP / (TP + FN). Recall is also called sensitivity.
- Why is accuracy not enough to judge a model?
- Accuracy can be high for a model that is useless. If 99 of 100 items are negative, a model that always predicts negative scores 99 percent. The matrix shows the problem, because every actual positive is a false negative.
- What does a normalized confusion matrix show?
- It shows proportions instead of counts. In scikit-learn, normalize='true' divides each row by its total, so the diagonal cell is the recall of that class. normalize='pred' divides each column, and normalize='all' divides by the number of observations.
References
- sklearn-model-evaluation
- scikit-learn User Guide, 3.4 Metrics and scoring: quantifying the quality of predictions scikit-learn
- sklearn-confusion-matrix-api
- sklearn.metrics.confusion_matrix scikit-learn
- google-ml-crash-course-metrics
- Machine Learning Crash Course: Classification, accuracy, precision and recall Google for Developers
- wikipedia-confusion-matrix
- Confusion matrix Wikipedia
This is a reference entry. TarmacView provides drone inspection services and doesn't sell the equipment or materials described in this glossary.