Mean Average Precision Explanation

It has been difficult for me to find a simple explanation of Mean Average Precision (MAP) on the internet, so I figured I would take what I’ve discovered through multiple sources and attempt to boil this evaluation metric down through an easily understandable example.

MAP is a performance metric of Information Retrieval (IR). IR is the (automated) process of returning appropriate results to a user, given some form of user input. The most obvious example of IR is a Google search result. The user inputs a text string, and Google returns the webpages that best match the user’s input. Another example is a streaming movie service that recommends a selection of movies based on the user’s viewing history. For this post, we will use a fictional streaming movie example to illustrate how MAP evaluates an IR system’s performance.

Let’s pretend that John, a streaming movie company customer, is an avid fan of horror movies. This is obvious because the movies he has watched so far on the streaming service have included The Exorcist, The Blair Witch Project, and Poltergeist. The goal of the streaming movie company’s IR system is to make additional recommendations to John that he will most likely watch next. The data frame below shows the next 5 movies that John actually watched compared to the next 5 movies that were recommended to John.

     user    watched    recommended
1   John    Jaws          Jaws
2   John    Psycho      Taken
3   John    Alien         Fear
4   John    Tremors   Alien
5   John    Fear          Titanic

So how well did the IR system perform on recommending the next 5 movies that John should watch?

Average Precision (the “AP” portion of MAP) concerns itself with two things: (1) how many of the recommended movies were actually watched and (2) whether or not the recommendations that were actually watched are towards the top of the recommendation list.

The first thing we need to do is calculate the cumulative accuracy of the recommendations. i.e., for every correct recommendation that the IR system made, we sum the total accurate recommendations up to that point. This calculation does not care whether the accurate recommendations are in the same order as the actual movies watched.

     user    watched    recommended    cumul_acc
1   John   Jaws           Jaws                        1
2   John   Psycho       Taken                    0
3   John   Alien          Fear                       2
4   John   Tremors    Alien                     3
5   John   Fear           Titanic                  0

The next part of the calculation is to divide each row’s cumulative accuracy value by the row number.

    user    watched    recommended    cumul_acc    ratio
1  John   Jaws            Jaws                       1                      1/1
2  John   Psycho       Taken                    0                      0/2
3  John   Alien          Fear                       2                      2/3
4  John   Tremors    Alien                     3                      3/4
5  John   Fear           Titanic                  0                      0/5

We then take the average of the ratios, which calculates an AP@5 of .4833. You can see that positioning the accurate recommendations towards the top of all recommendations would result in a larger ratio. e.g., if Fear had been the 2nd recommendation instead of the 3rd recommendation, then its corresponding ratio would have equaled 1 (2/2) instead of 2/3.

If the IR system had accurately recommended every one of the next 5 movies watched (regardless if the recommendations were in the same order as the movies watched), then the AP@5 would have received a perfect score of 1.

Finally, the “M” portion of MAP comes into play by evaluating the average AP performance of an IR system on a set of recommendations. Let’s pretend that in addition to John’s AP@5 score of .4833, Jane had an AP@5 score of .6565 and Tom had an AP@5 score of .4449. The MAP@5 for this set would be (.4833 + .6565 + .4449)/3 = .5282

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: