Mean Average Precision (mAP) Explained: Everything You Need to Know

Check out:

1. V7 Image Annotation

2. V7 Video Annotation

3. V7 Dataset Management

4. V7 Automated-Annotation

5. 13 Best Image Annotation Tools

Now, let’s dive in!

What is Mean Average Precision (mAP)?

Mean Average Precision(mAP) is a metric used to evaluate object detection models such as Fast R-CNN, YOLO, Mask R-CNN, etc. The mean of average precision(AP) values are calculated over recall values from 0 to 1.

mAP formula is based on the following sub metrics:

• Confusion Matrix,

• Intersection over Union(IoU),

• Recall, 

• Precision
  
  

Let’s discuss each sub-metric and how it is interpreted.

Confusion Matrix

To create a confusion matrix, we need four attributes:

True Positives (TP):  The model predicted a label and matches correctly as per ground truth.

True Negatives (TN): The model does not predict the label and is not a part of the ground truth.

False Positives (FP): The model predicted a label, but it is not a part of the ground truth (Type I Error).

False Negatives (FN): The model does not predict a label, but it is part of the ground truth. (Type II Error).

Confusion matrix

Intersection over Union (IoU)

Intersection over Union indicates the overlap of the predicted bounding box coordinates to the ground truth box. Higher IoU indicates the predicted bounding box coordinates closely resembles the ground truth box coordinates.

Intersection over Union

Ground truth box vs predicted box

Precision

Precision measures how well you can find true positives(TP) out of all positive predictions. (TP+FP).

Precision formula

For instance, the precision is calculated using the IoU threshold in object detection tasks. 

In the image below, the cat on the left has 0.3 IoU (< IoU Threshold) w.r.t ground truth and is classified as false positive. In contrast, the cat on the right is classified as true positive because it has an IoU of 0.7 (> IoU Threshold) w.r.t ground truth.

Calculating IoU threshold

The precision value may vary based on the model's confidence threshold.

Recall

Recall measures how well you can find true positives(TP) out of all predictions(TP+FN).

Recall formula

How to correctly calculate mAP?

Average Precision is calculated as the weighted mean of precisions at each threshold; the weight is the increase in recall from the prior threshold.

Mean Average Precision is the average of AP of each class. However, the interpretation of AP and mAP varies in different contexts. For instance, in the evaluation document of the COCO object detection challenge, AP and mAP are the same.

Here is a summary of the steps to calculate the AP:

1. Generate the prediction scores using the model.

2. Convert the prediction scores to class labels.

3. Calculate the confusion matrix—TP, FP, TN, FN.

4. Calculate the precision and recall metrics.

5. Calculate the area under the precision-recall curve.

6. Measure the average precision.
   
   

The mAP is calculated by finding Average Precision(AP) for each class and then average over a number of classes.

The mAP incorporates the trade-off between precision and recall and considers both false positives (FP) and false negatives (FN). This property makes mAP a suitable metric for most detection applications.

Pro tip: Have a look at 27+ Most Popular Computer Vision Applications and Use Cases

Precision-Recall Curve breakdown

Precision-Recall curve is obtained by plotting the model's precision and recall values as a function of the model's confidence score threshold.

Precision is a measure of when ""your model predicts how often does it predicts correctly?"" It indicates how much we can rely on the model's positive predictions. 

Recall is a measure of ""has your model predicted every time that it should have predicted?"" It indicates any predictions that it should not have missed if the model is missing. 

Why do we need to use a precision-recall curve instead of precision and recall independently?

The paperspace article on mAP clearly articulates the tradeoff of using precision and recall as independent metrics as follows.

“When a model has high recall but low precision, then the model classifies most of the positive samples correctly but it has many false positives(i.e. classifies many Negative samples as Positive).“

“When a model has high precision but low recall, then the model is accurate when it classifies a sample as Positive but it may classify only some of the positive samples.”

The precision-recall curve encapsulates the tradeoff of both metrics and maximizes the effect of both metrics. It gives us a better idea of the overall accuracy of the model.

Based on the problem at hand, the model with an element of confidence score threshold can tradeoff precision for recall and vice versa. For instance, if you are dealing with a cancer tumor detection problem, avoiding false negatives is a higher priority than avoiding false positives.

We should avoid missing tumor detection at the cost of detecting more tumors with less accuracy. Lowering the confidence score threshold will encourage the model to output more predictions (high recall) at the expense of lowering correct predictions(lower precision).

The precision-recall is downward sloping because as the confidence score is decreased, more predictions are made (increasing recall), and fewer correct predictions are made (lowering precision). 

Consider a situation where you are supposed to guess all the countries in the world. 

You will confidently predict the names of a few countries (maybe 10 or 20) quickly with maximum precision. However, with each different guess, you will approach higher recall and lower your guesses' precision. If the precision-recall curve is upward sloping, then there is most likely an issue with the model's confidence score.

Over the years, AI researchers have tried to combine precision and recall into a single metric to compare models. There are a couple of metrics that are widely used:

• F1 Score—It finds the most optimal confidence score threshold where precision and recall give the highest F1 score. The F1 score calculates the balance between precision and recall. If the F1 score is high, precision and recall are high, and vice versa.
  
  

F1 score formula

• AUC (Area Under the Curve) covers the area underneath the precision-recall curve.
  
  

AUC (Area Under the Curve)

The Area Under Curve for precision-recall (PR-AUC) curve summarizes the PR values for different thresholds under a single metric. 

Different score metrics and their PR curves

The above image clearly shows how precision and recall values are incorporated in each metric: F1, Area Under Curve(AUC), and Average Precision(AP). The consideration of accuracy metric heavily depends on the type of problem.

AUC and AP are considered superior metrics compared to the F1 score because of the overall area coverage. For interpretability purposes, the researchers use AP as a standard metric.