How would you measure the success of the Netflix recommendation engine?

Question

How would you measure the success of the Netflix recommendation engine?

Asked at Dropbox over a year ago

10.7k views

Asked at

10.7k views

How to answer Metrics questions

Interview Guide

Top Metrics interview questions

How would you measure the success of Facebook Likes?40 answers | 61.3k views
What metrics would you use to measure the success of the Save feature at Facebook? 39 answers | 87.2k views
How would you measure the success of Gmail?23 answers | 17k views
See Metrics PM Interview Questions

Invite members
Invite by email

Add another

Lokesh Kumar R · Answer 1 · 2021-06-19T15:47:33+0000

That's a very interesting question to take. However, before jumping in I would like to share my understanding of Netflix and clarify doubts if any.

Netflix is a OTT platform that helps users watch their favourite TV shows and Movies from different languages. One can explore different genres of movies, TV shows, and documentaries through Search and the Reccommendation Engine. The Reccommendation engine feature helps users explore different things that they will like and in turn engages the user to the platform by making them watch things. It takes into many factors like Previously watched genres, Casts in prev watched including directors, Time watched, Language explored etc before suggesting a list.

The major competitors to Netflix is Amazon Prime, and other local players like Zee, MX Player, etc.

Clarifications/Assumptions:

I would like to measure the success of Recommendation engine of Netflix in the context of Indian Market. This is because, it is one of the growing market for Netflix and also a place where I'm experienced. Hope it is fine (Assuming Yes)

How are we defining the success? I would like you to suggest what definition makes more sense to us.

With the above grasp, I would like to first Define what is Success for us before moving to the metrics as defining success will help us understand what actions of user will be important and only then can we come up with reasonable metrics.

So, When I think about Netflix, it is one of the leader in the OTT space with more resources to watch in every genre and every language. Due to the above nature, Every other person knows Netflix nowadays. Thus, we can ignore Awarness right away. Due to high awareness, there is a lot of Word of mouth happening and thus in turn, acquiring users are also not much of an issue at the current stage of Netflix.

While, Due to higher presence of Competition, it is highly important for us to engage our users effectively at a large rate so as to save our users. So we need to focus on Engagement. By Engagement, I mean the engagement related to watching/finding movies and not other actions like creating multiple profiles, updating a profile, etc.. Before finalising it to be our definition of Success, we are at a stage where cash flow is of higher importance as only then, we can increase our Banner to further new productions and gets things under our umbrella.

So due to the above reason, I strongly Believe that Monetization and Engagement should be our Goal.

With the above definition of Success, I would like to next list out typical steps in a user's Journey while using Netflix.

User Actions Related to Recommendation System:

User decides to Binge, and opens Netflix
Clicks on the given recommendation/ Scrolls in the given recommendation
Hovers on a movie tile
Engages with the Recommended tile(like/unlike/adding to watchlist/playing)
Watches a movie/trailer
Closes

Of many steps that a user takes on to engage with the platform these are highly related to Recommendation system, and thus with the understanding, I would next look at metrics related to these action that help us reach closer to our goal.

Metrics:

# of searches per user per week (The lesser it is the more our recommendation system is doing good that users are directly coming here)
Like : Unlike Ratio in our Recommendation
# of movies added to watchlist per week from recommended ones
# of hovering before clicking on a movie in recommendation list
# of trailers watched from recommended movies per week
Time spent by 99 percentile of users in Recommendation before clicking a movie
# of clicks on the Next Button (Side Arrow) before a selection
% of movies watched after watching trailer from recommended list
Ratio of movie watched through Recommendation: Ratio of movie watched through Search
% of movies/shows watched with Recommendation score less than 70 (The number can be decided through some further discussion)
Conversion of users to Premium after 30 days trial
# of New Converted users per month
# of New trial users

I would like to prioritize top 3 metrics from the above list using the following criterias

Relevance to Goal
Impact to End User
Easiness of Measuring

With the above factors the Top 3 metrics are

% of movies/shows watched with Recommendation score less than 70
Ratio of movie watched through Recommendation: Ratio of movie watched through Search
Time spent by 99 percentile of users in Recommendation before clicking a movie

To summarize, The best success goal for Netflix at the current stage to measure how good the recommendation system is to see for the engagement. By Engagement, I mean the engagement with Finding & Watching films. The top 3 are % of movies/shows watched with Recommendation score less than 70; Ratio of movie watched through Recommendation: Ratio of movie watched through Search; Time spent by 99 percentile of users in Recommendation before clicking a movie.

amitbp · Answer 2 · 2020-09-01T23:28:54+0000

Feature Description -

Netflix recommends users movies and shows based on their interests, past views, user profile, popularity etc on the home page.

Goal

Online streaming industry has become extremely competitive in the past few years, Netflix wants to provide users with relevant content in order for them to spend more time on the platform and renew their subscription. I would say providing relevant content, user engagement/retention is the primary objective of this feature.

User Journey

User opens netflix, choses her profile, Netflix knows who has logged in and based on that Netfllix recommends shows or movies to the user. User either decides to watch a recommended movie or searches for a particular show or movie and decides to watch it.

Metrics

Based on the objective, below are the metrics I would like to measure -

Relevant Content

Avg # of shows watched after recommended
Avg. watch time for recommended shows.
Ratio of Avg. # shows watched after recommendation to Avg. # shows watched without recommendation
Avg. # of recommended shows rated up
Avg. # of recommended shows rated down
Avg time spent searching for a show

Engagement and Retention Metrics

Avg. Session Length
Avg. Session Frequency per day
DAU and MAU
Total # of hours viewed

Prioritization

From the above metrics I would primarily like to measure -

Avg time spent searching for a show - This metric shows how well is my recommendation feature working
DAU and MAU - This metric gives me my engagement metrics for the overall platform

Ruby Payal · Answer 3 · 2020-08-04T13:19:52+0000

Feature Description: Netflix recommendation helps user to get personal suggestions based on his/her interest areas and let the user know what's trending and what others are seeing most .

Goal: To - improve the user experience by reducing the time spent on search, also reduces the bounce rate that is probability of leaving the app

Business Goal: Increase the average time spent watching the content

Customer Journey:

Scenario 1 (when user is sure of his choices)- User opens the App either continue watching or searches for content, now the search could be done in following ways

In the search bar (if he knows exactly what he/she is looking for )
Or uses standard navigation bar and uses various filters according to his interest areas - to get the search results and then chooses among them

Scenario 2- When user is unsure

Then either he/she will try randomly and there could be a chance that user leaves if doesnt get desired content to watch
Or would go by what's recommended

Now in case of scenario 2 and at times in case of scenario1 too, recommendation plays an important role.

Following metrics could be used to measure the success of recommendation engines

How many people know ?

# % of subscribers who clicked at least once on the movie/series shown in recommendation

Usability ?

# What % of total movies/ series watched are based on recommendation per subscriber?

Engagement?

# % increase in Average user session of those who are using it

#% increase in no of movies/series completed?

User experience?

# Average reduction in time spent on search per user?

# Reduction in bounce rate that is the number of users who leaves the app altogether after searching for few minutes or seconds.

Abhinav Chitre · Answer 4 · 2020-06-18T13:58:51+0000

Netflix is an OTT platform which lets you stream TV shows and movies. You can explore new shows, movies, documentaries etc on Netflix and binge watch them. Netflix has its recommendation engine to recommend shows and movies to its users based on what kind of movies and shows they have watched, added to their list and liked. It takes into a variety of factors for eg. genres watched at different times of day, movies from specific directors , artists etc.

The main goal of this recommendation engine is to suggest new shows and movies to the users which they like so that they spent more time on Netflix binge watching shows and movies

The user journey for Netflix user is that the users opens Netflix —> Sees recommendations for shows and movies on the home screen — > clicks through to watch trailer and directly watches the show or adds to his list —> plays the content , watches it —> likes the show —> returns back to the platform to watch more episodes and more of the similar content —

Metrics to look at

CTR of home screen

CTR of Trending movies

CTR of different section on the home screen to help improve the recommendations

Number of New items added to my list from the recommendations per week

Number of new trailers watched from the recommendations per week

Time spent watching these trailers per week

%Conversion to actual watching of shows / movies after watching trailers per week

Average number of minutes for which recommended shows / movies were watched per week

We can look at the funnel conversion - were movies actually explored , added to the list down to actually watching them./

Minutes of content watched from the recommendations per week

New shows / movies watched from the recommendations per week

It should impact the overall goal of Netflix to make you binge watch on the back of great recommendations.

So we would look at :

Average time spent by user watching Netflix per day

Average Time spent on Netflix per session

To see if the recommendations were which would eventually bring those users back to the platform and also accurately figure out their interests : -

% of shows/movies liked from the recommended list

For retention

Average Time between Netflix sessions

DAU/MAU

Average time spent by user watching Netflix per day

To prioritize a few metrics which impact the goal the most, I would look at

CTR of home screen
DAU/MAU
funnel conversion from recommendations to actually watching the shows
New movies/shows watched by user per week

@bijan would like you to evaluate this answer. Struggling with metrics questions actually.