Precision and recall

Image Source: Wikimedia Commons

Definition

Let us consider a experiment from $P$ positive instances and $N$ negative instances for some condition. The 4 possible outcomes can be formulated in a $2\times 2$ confusion matrix as follows:

Total Population	Predicted Positive (PP)	Predicted Negative (PN)
Positive (P)	True Positive (TP)	False Negative (FN)	Recall
Negative (N)	False Positive (FP)	True Negative (TN)
	Precision		F1 score

Precision and recall are then defined as follows:

$$\begin{array}{rl}\text{Precision}& =\frac{TP}{TP+FP}\\ \text{Recall}& =\frac{TP}{TP+FN}\end{array}$$

Intuition

Precision measures the ratio of the number of true positives ($TP$, i.e., instances that are actually positive and also predicted to be positive) over the number of instances that are predicted to be positive ($TP+FP$). We can interpret this as a measure of the model’s positive prediction:

What proportion of positive predictions was actually correct?

That is, precision focuses on the model’s quality of the predictions.

On the other hand, recall measures the ratio of the number of true positives ($TP$) over the number of instances that are actually positive ($TP+FN$). This can be interpreted as:

What proportion of actual positives was identified correctly?

That is, recall focuses on the dataset; the completeness of the predictions.

Limitations

However, precision and recall metrics alone are quite limiting as they are easy to ‘cheat’ for a better score.

Precision

Precision does not take into account negative predictions. This allows the following type of cheating:

1. Make 1 correct guess as positive.
2. All else are predicted as negative (resulting in all $FN$ or $TN$)
3. Then, $\text{Precision}=\frac{TP}{TP+FP}=\frac{1}{1+0}=1.$

Recall

1. Make every guess as positive.
2. Then, $\text{Recall}=\frac{TP}{TP+FN}=\frac{1}{1+0}=1.$

Notice that precision and recall complement each other. If precision is cheated to 1, then recall is 0; vice versa. The question arises then if we can design a system so that it optimises for both of the scores simultaneously in a single score. This is where F1 score comes into play.

References

• Wikipedia contributors. (2024, January 6). Precision and recall. Wikipedia. https://en.wikipedia.org/wiki/Precision_and_recall
• Classification: precision and recall. (n.d.). Google for Developers. https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall
• Scarlett’s Log. (2020, December 28). Precision, recall, & F1 score intuitively explained. YouTube. https://www.youtube.com/watch?v=8d3JbbSj-I8