15% off membership for Easter! Learn more. Close

You are the PM for Facebook, Instagram. You want to add new "Mute showing food" feature. How do you measure success?

Asked at Instagram
3.2k views
Answers (5)
crownAccess expert answers by becoming a member

You'll get access to over 3,000 product manager interview questions and answers

badge Platinum PM
You are the PM for Facebook, Instagram. You want to add new "Mute showing food" feature. How do you measure success?

 

1. Clarification:

I'd like to better understand what are we talking about: I'm assuming the intention of the feature is to allow users to block images and video (let's call it media) related to food - in their newsfeed? Yes.

Since "mute" sometimes refers to turning-off notifications only, I'll assume we're NOT talking about muting notifications, but about actually not showing the posts? Yes.

I'm assuming this is the first category we're introducing for people to be able to block, probably based on the number of such requests. Yes.

 

2. Define the goal / alignment to mission:

Let me think how that aligns to the mission - "give people the power to build community and bring the world closer together". This feature aligns to the mission by preventing the opposite from happening - if users are annoyed by a certain type of content, or find it irrelevant, pushing such content to them would contradict the mission, so we want to prevent that from happening. 

The goal, therefore, would be to address negative metrics, namely - help people curate their newsfeed to prevent churn. The freed up real estate would still be filled with other types of content, so we need to make sure that it's more relevant and interesting to the user, so another goal would be to increase engagement.

It's important to note that this feature is a little different from "positive" features, which we usually would want to grow the usage of. In this instance, we only want to grow the usage of it for certain cases.

 

3. User journey:

There are multiple players here: the end user who sees the food content, the users who create food content and advertisers who publish food-related ads.

I know that ads already have the feature to mark them irrelevant or annoying, so I'd like to focus on the end consumers who see food content and their journey:

a. open newsfeed

b. see food-related media

c. click on the options button, choose "mute showing food"

or

d. click on the post, from within the post, click the options button, choose "mute showing food"

e. user is back in his/her newsfeed, the post is gone (is my understanding correct? Yes)

f. next time user opens their newsfeed, there's no food-related media posts

g. if user wants to revert their decision, clicks on general Settings button, navigates to options related to posts, where it's possible to see and edit the muted categories (for now - just Food).

 

4. Map the metrics:

I'll map the user journey to appropriate metrics so that we have a comprehensive list and then I will prioritize the more relevant ones:
 

stagemetric
awareness1.1 % of users who received a food-related media post and clicked on the options button.
1.2 % of users who ... and went to the Settings -> Post settings menu and saw the categories muting option there.
1.3 share of users who chose one method over the other (in newsfeed, from within the post, from options) - to see which channel works best
activation 2.1 % of users who enabled the muting option
engagement We said we want to measure improvement in non-food related engagement, so all the standard metrics apply here:
For users who have blocked the food category:
3.1 time spent in newsfeed
3.2 time spent in FB/IG
3.3 # shares, comments, reactions per user who blocked. This would also address somewhat of a cannibalization here, since while the user might have not been happy seeing food-related content, they might have been actively engaging with it - even if to argue about it. Those conversations might have still sparked legitimate discussions.
retentionThis is the main part that we wanted to target:
4.1 DAU for people who received food-related media and blocked it as opposed to similar people (according to look-alike models) that received food-related media but didn't block it
4.2 MAU for same audience
Negative metricsWe want to ensure that by introducing the feature, we didn't harm the user experience
5.1 % of people who un-mute the food category from the settings out of those who muted within a week timeframe - would tell us it was either done by mistake, or the user is unhappy with the change.

5. Evaluation:

To evaluate the metrics, I'll use our defined goal and the complexity of the actual measurment to define a "north star" and primary and secondary metrics.

Since our main goal was to address churn, our North star metric will be 4.1 -  DAU for people who received food-related media and blocked it as opposed to similar people that received food-related media but didn't block it. Obviously, we don't want to just blindly encourage all users to use this feature, so it's important to maintain this balanced view of the two types of users.

The other main metrics would be related to engagement - 3.1 Time spent in newsfeed and 3.3 shares, comments and reactions - since, as we said, we want to also make sure the content is replaced by other content that's more relevant and interesting to the user.

Finally, the secondary metric to track would be 5.1 - % of un-muters, to keep an eye on potential bugs or popularity trends that would cause people to revert their previous decisions.

6. Summary:

In. order to address churn, we have introduced a mute-food feature, allowing people to block food media posts in the newsfeed and to track the success of it, we want to measure DAU of people who blocked vs DAU of look-alike people who didn't block it. Additionally, time spend and responses would let us know if the content is replaced by better content and finally % of un-muters would make we have addressed people's requests correctly.

Access expert answers by becoming a member
2 likes   |  
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs
badge Gold PM
You are the PM for Facebook, Instagram. You want to add new "Mute showing food" feature. How do you measure success?
 
 Clarify question:
  1. Is the “mute” feature based on the category of the content or based on the hashtags? : The mute feature is based on the category defined at the explore section

  2. Does this also apply to the stories? : Nope for now 

  3. Does this feature affects only the viewer’s feed or even discovery/search feature?: Feed and discovery 

  4. Is this feature specific to “showing food” or in general allows the user to mute a hashtag? : let’s say “mute showing food” feature

 
Feature Goal: Improve customer experience by improving content selection. This ties with FB Instagram's mission to connect people that view the world in interesting ways and share that expressions through images.
 
Feature Impact : Allows individuals to mute food category so it does not show up in their news feed. This way they can possibly spend more time in engaging with content that they are passionate about.
Success Metric: So I will want to see if the feature improves enagement and retention of user .
 
Feature Tradeoffs: 1- Reduce the content on the feed which limmits user engagement with other 
                               2- Discourages content creators to create food content if too many people start muting
                               3-  Negativel impacts ad revenue as audeince pool decreases.
Metrics to Track: effect on engagement and retention
# of ppl using the mute feature
% increase in time spent by using ppl DAU who use the feature
% increase of liking, commenting, exploring other categories of DAU who use the mute feature 
# of visitsi in app per day per user who use mute feature
# of visits in app per week per user who use mute future
 
Access expert answers by becoming a member
1 like   |  
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs
badge Platinum PM

Clarity

When you say 'mute showing food' features I assume you're referring to custom filters that lets you filter out 'topics' that you don't find interesting. 

Does this mean users can also set up filters for 'topics' they dont' find interesting outside of food? assuming that's the case

how does it work- does it work off hastag or image recognition, I'm assuming it the latter ( the scope of how that works is outside the scope of this answer)

Does this work on stories or feed or both? I'm assuming both 

Is it for a period of time, similar to how mute on a contact works on whatsapp? yes

Why would we do this?

Users may have personal preferences that are not reflected in the IG algorithms instantly, this signal enables the user to quickly specify a loss of interest in a particular category explicitly instead of gradual learning.

user stories

Rahul recently lost his dog Brutus and he otherwise loved watching dog videos but now they remind him of his dog so he wants to mute the topic for a while.

Rahul is trying to lost weight but all these delicious posts on IG keep tempting him so he wants to mute all food posts.

 

Hypothesis here would be that certain topics were preventing/creating friction to engagements from a certain set of users, if I allow users to weed out specific topics they should be happier and browser for longer.

The risk is that this reduces the number of posts the user can consume so we need to be tracking the impact

Metrics

Primary

(Posts consumer + stories consumed) per user 

This encapsulates impact of post reduction as well so covers risk.

other metrics to look at 

- Adoption rate: Since this is a new feature this is important to track 

- Opt out rate: Since this is a new feature this is important to track , users may try to use the feature and then opt - out for reasons that may range from quality of the product shipped to actually not wanting to use it since they didn't like how strongly it impacted their feed

- Time spent per user : This should ideally go up 

- Like rate/comment rate: This should remain unchanged, hypothesis here is that a group of users will mute topics like food that they enjoy but are looking to avoid leading to a drop in like rate BUT there will be a another cohort of users who will mute topics that they do not enjoy thus enabling them to discover /interacts with posts they actually like

 

 

Access expert answers by becoming a member
4 likes   |  
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs
badge Platinum PM

Instagram goal is to allow people to share memories through pictures/stories

 

Clarify question:

  1. Is the “mute” feature based on the category of the content or based on the hashtags? : The mute feature is based on the category defined at the explore section

  2. Does this also applies to the stories? : Nope for now 

  3. Does this feature affects only the viewer’s feed or even discovery/search feature?: Feed and discovery 

  4. Is this feature specific to “showing food” or in general allows user to mute a hashtag? : let’s say “mute showing food” feature

 

Goal: Mute feature aligns with the uber goal of instagram which is bringing people together/ building community by showing relevant/interested information to the user 

 

Two types of users

  1. Creators 

  2. Viewers: This feature focuses on increasing the experience of the viewers by allowing them to select the “type” of content they want to see on their feed.

 

Pros

  •  Better user experience 

 

Con

  • Could affect the feed content, what if the people the user is following posts “food” picutres” majorly then the viewer wouldn’t have enough content to engage.

  • Could discourage content creators of “food” channel if there a lot of people muting the topic 

 

Metrics (in order of priority)

 

  • # of people using the mute- food topic

  • Time spent by the above users in feed / day

  • Time spent by the above users in search/explore / day

  • # of posts pertaining to feed / user (if there are a lot of users using the mute feature, it might decrease)

  • # of likes against the posts regarding food topic 

Access expert answers by becoming a member
3 likes   |  
1 Feedback

You're missing a couple steps (walk through the customer journey, describe the metrics, and evaluate them). See the site's interview guide on how to answer a metrics question https://www.productmanagementexercises.com/how-to-answer-a-metrics-question

 

0
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs
badge Bronze PM

 

Firstly, I would like to understand what is meant by ‘Mute showing food” means? (Interviewer: It is a feature to mute all posts, stories, and sponsored ads related to food)

How do we decide that a particular post/story/ad is related to food? (Interviewer: We have an image recognition tool and NLP tool that go over them and if the particular item gets labeled as “food-related”, we shall mute it)

How does that feature work? (Interviewer: As and when a user opens the app, there shall be a dialog box that asks if the user wants to mute showing food)

What is the ultimate goal we are trying to achieve? From my understanding, it Is to show “food-related” posts/stories to relevant users, which shall enhance the user experience and ensure retention. Am I correct? (Interviewer: Yes, go ahead)

Possible user action would be as follows:

User opens the FB -> brows the feed/ looks at the statuses/ ads -> react to the post, comment, reshare the post -> comes across the action to mute showing food -> select yes/no -> continue browsing the fb -> occasionally come back to look back at any new replies/ comments/ react to posts etc

I have tried to compile the above user actions to cover all possible actions of an users

Based on the above actions, would want to focus on below metrics-

1)      Number of users who reply to question of muting food related posts etc.: This shall give an idea If the feature is usable and If this the right question to ask the users,

 

2)      Avg Number of food-related posts/ stories/ by users who responded “yes”

a.       Reacted upon by a user in a week

b.       Reposted by the user in a week

c.       Commented by the user in a week

Have taken week as a basis instead of day as it gives a bigger sample range to measure the success of the metric

 

3)      Avg Ratio of food-related posts shared/reacted/commented by user to total number of such items shown to the users who responded “yes”

This metric should be higher when compared to the previous ratio prior to implementing the feature

4)      Avg session time in a week comparison amongst people who responded “yes” and “no”: This metric gives an idea if because of this feature do we see some increase in engagement.

5)      Bounce rate comparison prior to and after implementing the feature: - This shall give an idea of how many people do some action in FB. This is a negative metric. Ideally, we expect the bounce rate to be reduced after the feature implementation.

 

Access expert answers by becoming a member
0 likes   |  
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs