Question

If you were the PM for the Save Feature at Facebook, what metrics would you use to define the success of this feature?

Answer 1

I will solve this problem in these steps:

• I will clarify what the feature is.
• Then I will explain what our business goal is with this feature.
• Next I brain storm the metrics.
• I prioritise the metics
• At last I summarise. 

Clarify: Save Feature allows users to to Save Links, Pages, Posts, Locations, Movies, etc to view later. FB also reminds users about what items they have saved.

This feature affects users and marketer. Marketers do not want to be forgotten, so if they post something that attracts the attention of the user, they want the user to be able to find it again later, if they don't have immediate time to spend on it. For example, if there is a nice shoe advertised on FB and user likes it, but cannot check it now, or there is a discussion about a TV series that the user potentially finds interesting to watch later, the user can save it to check it out later.

User Goal: The benefit for user is that they do not need to copy paste or make screenshot of thing that they want to checkout later. They can have all these items in a categorised way (e.g. Movies, Pages) and can check them later.

So I as FB expect that Save Feature 

• Marketer Goal: increases revenue for marketers by increasing clicks and impressions
• BIZ Goal: increases user engagement
• BIZ Goal: Increase FB revenue by increasing CTR, and CPC and CPM. Because user might make a click that he would not have done otherwise if he could not save the post. So the goal is to increase CTR and consequently the revenue.

Metrics to measure the success:

1. Discoverability: %of users the have at least once Saved an item. --> User knows that he can Save items
2. Discoverability: %of users from (1) that have at least once opened Saved Page --> User knows where to find Saved items and knows how to work with it
3. Discoverability: %of users from (2) that at least once engage (metric 11, 12, 13) with at least one item in Saved Page
4. The avg amount of time it took user from Saving an item to opening it again
5. %of users that engage with an item in Saved page too late, so that the item is expired already (difficult to implement, how would the algorithm know if the page/link/... does not have the value it had at the time it was Saved?)
   1. Examples: it has been an offer for a discount/coupon that was only 1 week, or it has been a event would have been streamed live in an FB page
6. #of times a user clicks to Save an item vs. the #of items the user visits (% in 1 month)
7. #of times the user opens the Save page vs. the #of times the user visits FB (% in 1 month)
8. #of categories in Save page the user goes through to see what items are saved in each category
9. the amount of time user spends just looking at items in Save page
10. distribution of #items Saved in each category, e.g. Links, Movies, TV, Locations. ---> This can help us to clean our list of categories and remove the noise (i.e. categories that are barely saved) or add new categories based on what is Saved in what category, but our algorithm tells us the category does not match.
11. engagement: %of Saved items that the user opens from Saved page
12. engagement with the items that are opened from Saved page, i.e. 
    1. #likes
    2. #shares
    3. #clicking links
    4. #playing videos
    5. #comments
13. engagement: amount of time spent on a page, after opening it from Saved page.
14. %of Saved items that the user deletes without engaging with or opening them
15. %of Saved items that the user deletes after engaging with or opening them
16. retention: reaction to reminders from FB about items in Saved page, 
    1. #of times user clicks on reminder to see Saved items. 
    2. #of times he "engages" with items after getting the reminder.
17. retention: change in the # of items that user Saves every month .
18. Are we cannibalizing?  ---> Is there a decrease or increase in likes, shares, comments, #of posts after pre-launching this feature for test group?
19. User Segmentation: What is the user demographics for those who engage with this feature and those who do not
    1. Age
    2. Gender
    3. Mobile/desktop
    4. Location
20. User Segmentation: What type of users use this feature most often? Power users? Passives? Medium users?
    1. If we have a problem in Discoverability, can we encourage power users to write about this feature in their posts?
    2. Shall we begin the testing phase of the feature with power users to fine tune it and do A/B testing?
21. What features items have that are most Saved? 
    1. Item type (page, link, movie, location)
    2. What is in the item? 
       1. amount of text
       2. video
       3. image
    3. They have a lot of comments, shares, likes
    4. They are shared/liked/commented by a friend
       1. How close is the friend?
       2. what has been the type of the friend's reaction to this post? Like? Share? Comment?
       3. How many friends have reacted to it before I save it? --> Hypothese: It makes a difference for user if 2 people he knows have liked sth vs. 100 strangers.
    5. They have a high similarity to what user already interacts with: Hypothesis: We can use this information to suggest to users to Save an item, but needs to be implemented carefully to not to cannabalize active engagement of users with the hope of an engagement at a later time point. 
    6. They have a high similarity to what user sees in his News feed everyday, but does not react with --> Hypothese: User has been potentially interested in these items, but did not have the time to interact with them? We can also suggest users to Save items that he rarely interact with and see what users reactions is? Does he Save them? Does he go back to see them?
22. If we adjust News Feed of the user based on the items he has Saved, i.e. recommend new items in NF or give higher priority to items in NF that are highly similar to the Saved items, do we see an increase of engagement (likes, shares, comments, #of posts, clicking links, playing videos, time spent)?
23. Monetization: %Increase in CTR (#of clicks measured for click ads, the increase should be very low, but still since CTR is always very low, we can only increase it in very small steps) --> increase in revenue for Businesses
24. Monetization: %Increase in Impressions for Impression Ads 
25. Monetization: %of revenue for FB increase just based on clicks and impressions made through the funnel that includes Saved items

Prioritise:

Based on BIZ goals and user goals, I choose the following metrics:

1,2,3, 11, 12, 13, 21 (1-4), 22, 23, 24, 25

Summarise:

Save Feature is helping users to Save items to interact with them later. It expects to increase the revenue for marketers by helping them not to be forgotten by users. It also helps FB to increase engagement and finally revenue.

We brain stormed metrics to measure discoverability, engagement, retention and monetization effects for feature.

Answer 2

The first question I would ask is: what does “success” mean? Is it aimed at increasing user satisfaction, or usage/engagement/increased ad revenue? My assumption is that the answer is “yes” – at the core of a successful feature is the personal and social value for the users which ideally also generates profitability for Facebook and its advertisers. (A win/win/win).

A lot of telemetry can be measured, but in order to gain the most valuable insights, let’s start with the value for the users. Here are some reasons why they might want to make use of the save function:

-1- Ability to return to a certain post at a later time to consume or share/react/comment
-2- Ability to browse more content faster, while marking (=saving) a subset of interesting posts for more focused consumption/interaction
-3- Ability to collect posts related to certain themes/topics
-4- Ability to collect posts from certain friends or groups (what happens when a friend deletes a saved post from their wall??)

The assumption is that the save function is successful if it enables the above listed abilities. In the absence of actual user feedback, the combination of the following measurements are good proxies to assume that the save function is successful:

a. #saves per user per Facebook visit
b. #of returns to saved content per user
c. #interactions with saved content per user (or per saved piece of content)
d. #saves per amount of content scrolled through per user and FB visit

High numbers in a. plus any of b/c/d are a good indicator that a user is likely satisfied with the feature.

However, we also need to look at how many Facebook users (with regular activity on FB) have never used the save function. A high percentage would indicate a possible discoverability (awareness) issue.
If such an issue had been identified and addressed, then a decrease in the population which has never used the function is another measure of success.

If, furthermore, good analysis of what individual users tend to save helps surface more similar content in their newsfeed, we might be able to measure increased engagement with the newsfeed on content which had been better personalized based on previous save actions. Any such engagement can also be considered an (at least indirect) success metric for the save feature. Relevant measurements would be:
– #reactions/shares/comments/active consumption (e.g. play video) for content suggested based on a user’s saved list (even if it does not lead to added saves, which would be counted in a. above)

Other success metrics could be derived from non-instrumented data sources:
– actual user-feedback (e.g. one-question pulse queries on individual FB features)
– counterfactual testing (what if for a population of FB users the save function would be removed; how many complaints would we get?)
– Mining of any mention of the FB save feature in FB posts, comments, or FB-related user forums

The cost of deriving the last 3 suggested metrics is not likely justifying the result. Especially the first two suggestions have the potential to irritate the impacted users, while the ROI for the third is likely low.

I would therefore focus primarily on the readily available, yet relevant, logged usage telemetry described under a.-d. above, secondarily on the “indirect” success metrics, and furthermore measure if we can confirm any correlation between engagement with the save feature and increased # of visible and/or clicked ads. The latter measurement would be a metric related to the successful generation of financial profitability for the advertisers.

Comments

As far as #of returns to saved content per user are concerned i would bifurcate this into: % of users returning to view saved content organically (on their own) & % of users returning inorganically (i.e., reminded by facebook to view saved content). This would tell me: 1) how easy it is to find "Saved" posts (discover) and 2) Users intent to interact with saved posts later

Answer 3

Think there r about 5 categories of metrics to look at for most feature rollouts:
1) who is using it – persona / segment / type (novice, proficient, expert)
2) when – what do they do immediately before / after, etc.
3) usage – number of times in a session / week, duration between 1st-2nd, 2nd-3rd usages, etc.
4) impact – aarrr (short-term & long-term), funnel, etc.
5) cannibalization – did usage of some other feature decrease

Comments

Hi Roy
Thank you for submitting your answer. I Th ibk your answer is too short and needs to be more detailed. Have a look at the article I wrote about answering Metrics questions. https://productmanagementexercises.com/how-to-answer-a-metrics-question/
Good luck!

Answer 4

To my understanding, the way the Save feature works is – It let’s me save posts I want so I can view them later under my “Saved” label to the left of the Facebook News Feed.

I would track the following metrics to define success:
– Total saved posts per day
– Average/Median saved posts per user
– Number of saved posts revisited per day/week/month
– Number of webpage views for the “Saved” feature (and number of DAUs who also visited the saved webpage)
– Number of unsaved posts (the act of going to the saved webpage and clicking “unsave” to remove from the post from the list)
– I would also look at different posts to understand what users save overtime, do they save posts with/without videos? just regular posts? recommendation posts? posts with picture?
– amount of time spent on the saved webpage
– lastly, we can track the retention of the feature. The only tricky part here is to define what’s good retention. Unlike retention for the FB app (D1, D7, D14 etc…) as users who save posts might return to them even after a few weeks and that’s not necessarily a bad thing. Once we have a good benchmark of how retention for this feature would look like, we can align it to all players to help define its success

Comments

You might want to look at the scroll length. Do people scroll more per session because they are saving links that are hard to consume right away? Has time spent overall increased or decreased? What is the impact to videos watched? Because people are now saving – has the time spent watching videos and # of videos watched gone down? Are people saving and not coming back to view these videos?
Save also lets Facebook get more signals about what a user is interested in. So is the feed relevancy improving overall for users with more saves vs less saves? Has overall engagement with newsfeed gone up – more posts viewed, more active time spent, longer scroll length?

Answer 5

primary:
– # unique users that clicked ‘Save’ on post
– # unique users that clicked the “Saved” section
– # unique users that re-opened at least one saved post
– # items “saved”
– # times “Saved” section got clicked
– # times saved item re-opened

secondary:
– Overall DAU, WAU, MAU
– Stickiness (DAU/MAU)
– Average Mins per user spent on facebook before and after (platform, watching videos..)
– Monthly revenue from ads

Comments

Hi there
Thank you for submitting your answer. I Th ibk your answer is too short and needs to be more detailed. Have a look at the article I wrote about answering Metrics questions. https://productmanagementexercises.com/how-to-answer-a-metrics-question
Good luck!

Answer 6

Define & Clarify
I’d first want to clarify and define what the FB Save Feature is. The Save Feature allows users to save favorite posts from their news feed for later consumption. Users can save a post by clicking Save from the drop-down menu in the upper righthand corner of a post. Once a post has been saved, you can view it later by accessing the “Saved” page in the lefthand sidebar. The saved posts are organized by categories. 

At it’s core, the Save Feature solves for two main user problems: 

• Users don’t have time to read favorite posts in one session.
• Users can’t find favorite posts during a repeat user session.

Set Goal
The success of this features depends on what my goals are as a PM. There are two types of goals I’d would evaluate:

Value Added to User 

• Help users read their favorite posts in multiple sessions.
• Help users easily find their favorite posts during a repeat user session.

Value Added to Company

• Rev: Increase monetization opportunities
• Strategy: Diversify into the bookmark market.

I’d prioritize focusing on the user experience goals since it’s not wise to monetize features that don’t add value to users.

List User Actions
Before diving into metrics I'd first want to identify the user actions for the Save Feature because I can’t define the metrics without knowing what I should be measuring: 

• User scrolls through FB newsfeed and clicks on ‘Save’ for the posts he wants to view later.
• User returns for repeat FB session at a later time.
• User accesses the Saved page from the sidebar
• User scrolls through the Saved content under the All tab
• User clicks on the category tabs to find the saved content
• User clicks through to read the Saved content
• User likes the Saved post
• User comments on the Saved post
• User Shares the post
• Poster navigates back to the Saved page to view more bookmarked content

Identify Metrics
Now I’d translate the user actions to quantifiable metrics:

• Avg # Saved posts per user 
• Avg # page views of the Saved page per user
• % of users who viewed a Saved post after saving the post.
• Avg # likes of Saved posts per user
• Avg # comments of Saved posts per user
• Avg # Shares of Saved posts per user

Evaluate
To identify the most important metrics for success, I’d map the metrics back to my goals as a PM of this feature. 

1. Primary Goal: Help users read their favorite posts in multiple sessions

• MetricA: Avg # Saved posts per user session
• MetricB: Avg # views per Saved posts per user within a 90 day period.

2. Secondary Goal Help users easily find their favorite posts during a repeat user session.

• MetricC: Avg # page views of the Saved page per user

The most important metric I’d focus on is MetricB — Avg # of views of Saved posts per user within a 90 day period. MetricA and MetricC are means to get to MetricB. I.e. Users would save posts and list the Saved page in order to read their favorite content at a later time.  

Answer 7

Get clarity on business goal first:

- "save" feature helps users to locate interesting content, post, ads, information so they can revisit later (i.e. engagement)

- "save" feature help content generator (Ads & other content) to gain more visit, more interactions (comment, likes..) and potentially revenue (i.e. ads revenue, CTR, other interactions)

Key metrics

1. Engagement and interactions

- UV of "save" page

- # o people/A1 who uses this feature (defined as clicking save button + view item from saved page)

- Avg number of times people use this feature per day/per month

- Content saved in "save" page

- # of likes/shares/.. of saved content vs. un-saved content

2. Monetization

- CTR of saved content vs. un-saved content

- Revenue per saved content vs. un-saved content

- # of likes/shares/.. of saved content vs. un-saved content

Other metrics

Userability

- % of people who discover this feature (at least once)

- A/B test different CTR using different UI

Retention

- Persona of people who use this feature

- A1, A7, A30 of people who use this feature