F1 Score Intuition

One of the popular metrics to evaluate a binary classifier is F1 score and its variants. Technically, F1 score is defined as the harmonic mean of precision and recall. However, I often wondered what it means. The description failed to explain:

  1. Why only precision and recall?
  2. Why harmonic mean?
  3. Why its called F1?

Below is an attempt to explain the first two questions.

1. Why only precision and recall?

Often when dealing with a classification problem, we are interested in positive cases and our trained model’s ability to detect them. In other words, we are interested in true positives (TP). However, the number of true positives by itself is meaningless. For instance, if I say there are 50 TP; it doesn’t make any sense. We need a reference point to make true positive more meaningful. For instance, we can say that the model can detect 50 of 200 positive cases or 25% of actual positive cases.

However, there is a challenge. What should be the normalizing constant? There are two alternatives: one is the number of actual positives in our dataset, and another is the number of predicted positives. Using the number of actual positives as the normalizing constant gives true positive rate from the data perspective and is called as recall. Based on confusion matrix notations, it can be written as:

Recall = \frac{TP}{TP + FN}

In contrast, using the number of predicted positives as the normalizing constant gives true positive rate from the model perspective and termed as precision. It can be written as:

Precision = \frac{TP}{TP + FP}

2. Why harmonic mean?

A good metric usually has three qualities: it should be real-valued, bounded and monotonically increasing with the increasing quality of a model. However, above we have two different metrics to evaluate a model’s performance to detect positive cases. The challenge is then how do we combine the two to construct a single numerical metric. One solution is to use an average. However, there are many various kinds of averages such as “arithmetic,” “geometric” and “harmonic” mean. When dealing with ratios, as in our case, harmonic mean tends to make much more sense. Hence, F1 score is described as a harmonic mean of precision and recall and can be written as

F1 = (\frac{P^{-1} + R^{-1}}{2})^{-1} = 2\frac{PR}{P+R}

3. Why the metric is known as F1?

The name consists of two parts: the letter “F” and the number 1. I am not sure why the letter “F” is used to communicate the metric. I don’t also know how/why 1 is used but it make sense if we look at the general form of F_{\beta} score:

F_{\beta} = (1 + \beta)\frac{PR}{\beta P + R}

I still don’t understand how the general form of F score was derived. Something to still explore

Advertisements

2 thoughts on “F1 Score Intuition

  1. Hi Ritesh,
    Here is the derivation for F_{B}:

    \begin{equation}
    (1+\beta)\frac{PR}{\beta P + R}=\frac{(1+\beta)}{\frac{\beta P + R}{PR}}=\frac{(1+\beta)}{\frac{\beta}{R} + \frac{1}{P}}
    \end{equation}

    When $\beta=1$ (F1), we have the harmonic mean of precision and recall:
    \begin{equation}
    \frac{2}{\frac{1}{R} + \frac{1}{P}}=2\frac{PR}{P+R}
    \end{equation}

    Take a look at the slides 33-36 from the Modern Information Retrieval book about the E-measure which is related to F_{B}:
    http://grupoweb.upf.edu/mir2ed/pdf/slides_chap04.pdf
    Hope this is helpful.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s