← All posts

Evaluating classification models with Accuracy, Precision and Recall

Originally published on Medium →

Context

Imagine building a classification algorithm that predicts whether it will rain. Over a 10-day period, we compare actual weather outcomes against our model’s predictions to evaluate performance.

Accuracy

Accuracy measures overall predictive correctness — how many predictions matched reality. In our rain prediction example, if we correctly predicted 8 out of 10 days, our accuracy is 80%.

The formal equation is:

Accuracy = (True Positives + True Negatives) / Total Predictions

The confusion matrix and how to understand it

The confusion matrix organizes classification results into a 2×2 table showing:

  • True Positive (TP): predicted positive, actually positive.
  • False Positive (FP): predicted positive, actually negative.
  • False Negative (FN): predicted negative, actually positive.
  • True Negative (TN): predicted negative, actually negative.

For our rain example: TP = 1, FP = 0, FN = 2, TN = 7.

Back to accuracy

Using the confusion matrix:

Accuracy = (1 + 7) / (1 + 7 + 0 + 2) = 8/10 = 80%

Precision

Precision measures how often positive predictions are correct. It answers: “Of all rainy predictions, how many were actually rainy?”

Precision = True Positives / (True Positives + False Positives)

For our example: 1 / (1 + 0) = 100%.

As Cassie Kozyrkov puts it: “A system with high precision might leave some good items out, but what it returns is of high quality.”

Use precision when you want high-quality results.

Recall

Recall measures what percentage of actual positives your model caught. It addresses: “Of all actually rainy days, how many did we identify?”

Recall = True Positives / (True Positives + False Negatives)

For our example: 1 / (1 + 2) = 33.33%.

Again, per Cassie Kozyrkov: “A system with high recall might give you a lot of duds, but it also returns most of the good items.”

Use recall when you cannot afford to miss relevant cases.

Bonus: the F1 score

The F1 score combines precision and recall into a single metric using their harmonic mean:

F1 Score = 2 × (Precision × Recall) / (Precision + Recall)

Use the F1 score when you want balanced performance across both metrics.

The End

With these four metrics, you can comprehensively evaluate classification model performance and make informed decisions about algorithm improvements.

May the code be with you!