Facebook recently launched a new feed algorithm. As a result, the average session duration dropped by 20%. What would you do?

Clarification and assumption:

• Drop observed as a part of A/B test where we measured the new feed performance vs older

• New feed aims to provide better content recommendation and better UI

• Drop in US on all platforms

Product:

Lets understand the product/feature and value it provides to each stakeholder

Product Goal: Increase overall engagement

Newsfeed is the default landing page of FB and it provides users with personalized engaging content generating by friends or relevant sources

• Viewers: Feel engaged/entertained/connected

• Creators: Share life stories, knowledge etc and feel fulfiled, appreciated and heard

• FB: Getting more engagement and user data for more intelligent recommendations, which increases ad revenue

• Advertisers: They reach relevant target audience optimized for conversion

Hypothesis:

Lets generate a hypothesis and then look at some of the relevant metrics we need to look at to resolve the tradeoff

• Its possible that the new feed is providing more personalized content to user hence they dont have to scroll longer to find the same amount of information or the same level of engagement

• The new feed also might have some UI optimization for quickly viewing photo snippets or videos hence user save time

• The new feed might be providing more real estate to shorter content like stories etc which takes less time to view

My take here would be that although session duration might have dropped but there might be increased in # of sessions so the total time spent on the platform might not have dropped much. Also the content has become shorter (like stories) and its faster to navigate (new feed feature) so it takes less time to view the same amount of content. So, even the total time spent might have seen a small drop but the amount of content viewed might have gone up

Metric:

• Total time spent (D/W)

  • Session length

  • # of sessions per user (D/W)

• Total content viewed (D/W) per user

• Total interactions (Like/comment/share) per user

• Total content created per user

Resolve tradeoff:

• The new feed might decrease session length and total time spent slightly (say less than 2% which is acceptable as a guardrail metric). But I will specifically look for any decrease in content viewed, interactions or content creation. If there is any decrease in any of these metric then the new feed isnt achieveing the product or feature goal as it is decreasing the overall engagement. It might have a downstream effect of hampering the ecosystem.

• In other case if new feed decrease sssion length and total time spent (within limits) and increase all other metric then its is good choice and should be launched.

So there has been a drop in average session usage by 20% as soon as we launched a new feed algorithm. At such an incident, I would like to first of all check if there is a drop actually or is it just because of some reporting issue or measuring issue.

Assuming that there wasn't any reporting issue or measuing issue, Next I would like to go about thinking why was this launched, what was the goal we are trying to achieve through this and finally few information on the experiment.

Assuming that the goal was to help users find more meaningful content to interact with and thus improve the Engagement metrics and retention metrics like Average time spent, DAU/MAU. Let me also assume that this experiment was launched at America initially and the drop is pertaining to this region only.

Next, I would like to understand more about the drop, through the following questions:

Was the drop a more gradual ones or was it a sudden one?

Was the drop present across device types like mobile, desktop, or was present only at a particular type.

Finally I would  also want to check how are the related metrics doing like # of sessions per day, Average time spent overall. If these metrics are going up, then what we are facing is not a drop at all. So, I would want to check this as well.

Assuming the following to be answers to the above question:

The drop was a more gradual one.

The drop is common across devices.

All the related metrics have went down as well.

The actions that I would do, is to split probable causes as internal and external ones and then discuss diving deep into issues depending on the situations.

Since, the change has been a more gradual one, I would like to place my bet on External reasons like Bad PR, Government Regualtions and changes in User behaviour.

In the above cases, I would directly jump to change in User Behaviour as the experiment directly affects the behaviour of users.

To understand the user behaviour change, I would like to list the user Journey that one undergoes.

1. The user has a need, and thus opens FB
2. Lands on the News Feed page.
3. Scrolls down and interacts with different posts and ads present in the Feed
4. After spending some time, a user leaves the app.

So, in this I would like to check the how are the following metrics doing.

Opening rate of the application

# of posts that interacted by Top 90 percentile users before bouncing off

# of posts interacted by Top 90 percentile of users before seeing the following phrase, "You have caught up  everything"

Assuming that the 1st metric has went up while the rest have gone down significanlty, it could mean that the algorithm is overoptimizing the content to make sure that the contents are meaningful to the user depending upon the previous actions and thus, the # of posts that a user see is getting reduced. This will also mean that the user being shown to new genres to try will be low. Once these aspects gets reduced the user would naturally bounce off as he has nothing to interact with.

To confirm this hypothesis, I would like to check if the 2nd and 3rd metrics are equal to each other. Assuming a yes, We can conclude that the reason for the drop is that the new algorithm is over optimizing for the releavance to users that they are getting lower and lower posts to interact with.

Before finalizing on what to do, I would like to iterate what the Goal of FB has been. It was always to help users build a meaningful community and bring the world closer. 

This can be done only when users are shown different things relating to their interests and as well as new things so that we can learn more about users and as well as create more accurate and personalized ads.

Due to the above reasons, I would like to do the following:

Will interact with the Cross-functional team and try reducing the score that is reason behind the over-optimization. 

Next, I would like to implement the changes and look for how the metrics are doing through A/B Tests.

To summarize, I would check if the issue was actually an issue, then understand about the experiment, goal behind and finally understand about the drop. With all the information, I would next dive deep into the probable causes, check the metrics and  come up with a solution.