Question

Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?​

Answer 1

Let me start with few questions:

1. How're we doing it right now? --> Users have an option to 'Like' the post

2. Why did we launch 'Likes' in the first place? --> What do you feel?

Let me think about it. First off, let me segment the different users:

1. Heavy Posters --> These are the 1% of the users who are always active on Facebook. They're posting all the time, adding new friends, commenting, etc. They're are the content creator!

2. Frequent Posters --> These are the 9% of the users, who sometimes participate on daily FB's activities. 

3. Occasional Posters/Lurkers --> These are the 90% of the user who usually lurks around

As a FB PM, I would have seen this pattern and wanted more participation/engagement from all user segments. --> Seems fair assumptions

Let me list the needs/pain-points of each segment:

1. Heavy/Frequent Posters  --> When others don't engage on their posts, they lose motivation to produce content/posts and thereby they come less on FB, and FB loses its power of having an engaging community. 

2. Lurkers --> They like the posts but don't want to comment, they want an easy way to show their appreciation.

Looking at the above problems, 'Likes' seems to be the solution. 10% of users posting and 90% 'Liking'

Now, extending the above analogy, I see we can have more emotions - “love”, “haha”, “wow”, “sad”, and “angry” catering to the distribution of the 90% of the users as not all 90% will have a need to 'Like', they have other emotions like “love”, “haha”, “wow”, “sad”, and “angry”

So, from the above hypothesis, I see # of posts having only 'Likes' will < #posts having 'reactions'

Other Success metrics will be:

1. # of posts with reactions / # posts without reactions

2. Reactions leading to more posts --> # reactions / # total posts --> Cohorts of posts before and after reactions can be measured

3. Reactions leading to more active users --> # unique users before and after reactions

4. Reactions leading to other activities --> 'Add Friends', 'Comments', and 'Chats'

 

To summarize, I feel reaction will increase overall engagement as we're targeting a larger distribution of user and their emotions, and an ability to portray these emotions will lead to an increase in engagement

 

Answer 2

C: Clarify the Question

Who are the reactions for?
What are the goals of reactions?
Why was it trimmed to the said 5?

I: Identify the Customers

The two main customers for this feature that I see are businesses who need this information, and people who have been resorting to the like button to respond to posts. Who does it add more value to? I would think this would add more value to businesses as it can directly lead to a more efficient monetization with this additional layer of understanding of a user's sentiment. 

There are two aspects of success

1. (Users) Market Adoption and Engagement due to the reactions. Will people post more, respond more, now that there is an option of an appropriate response?

2. (Businesses) How much more efficient can this information make market targeting?

• Market Adoption
  • Hypothesis: Does the availability of different response options increase the post engagement of the users?
    • Avg count of reactions per post daily pre-launch of non-like reactions
    • Avg count of reactions per post daily post-launch of non-like reactions (hope to see an increase)
    • Avg count of reactions per 7 days (assuming weekends have a spike of activity) pre-launch of non-like reactions
    • Avg count of reactions per 7 days post-launch of non-like reactions
  • Does the availability of different response options instigate people to share more news?
    • Avg number of posts per day pre-launch
    • Avg number of posts per day post-launch
  • Does biasing to positive emotions garner more engagement with the user?
    • Correlation of distribution of categorized posts according to the non-like reaction vs Avg engagements per post.
• Business Monetization
  • Does targeting using non-like reactions lead to more engagement with the user?
    • Avg count of single clicks on targeted posts
    • Avg count of two-level clicks on targeted posts
    • Avg count of responses (comment, shares, reactions) on targeted posts

Would this number differ by reaction?

I believe that the reaction type is only important for the businesses to understand how to target their products or subjects, but to Facebook it does not make much of a difference. A user is likely to engage in a post that makes them mad, sad, happy, funny etc. Being mad about a cause is not necessarily a bad thing. Sympathizing with a sad cause isn't either. I believe success would differ due to the mixture of each reaction-garnering post, not the reaction themselves. 

Comments

Excellent post my friend,  you really held true to company mission and kept goals to users engagement

Answer 3

First of all, I'd probably need some sort of clarification about what "at launch" means. Are we looking at the performance the very next day after the feature is launched or are we giving ourselves a few weeks to see how things are going to play out? I assume it's the latter and we are going to be evaluating success two weeks after launch.

The goal of reactions feature is to give users more freedom in expressing how they feel about content that gets posted on Facebook. This feature should be particularly appealing to users whose primary method of engagement on FB is via the like button, which, I assume, is going to be the vast majority of FB's userbase.

First, to set some baseline metrics I can look at the historic data to estimate things like 

1. number of likes over the span of 2 weeks
2. number of posts over the span of 2 weeks 
3. avg. number of likes per post
4. avg. number of likes per user

Second, I can look at the distribution of the sentiment of facebook comments. I am pretty sure this data will already be available because that's probably how they came up with these 5 reactions to begin with. Let's say the distribution looks like this: neutral (50%), funny (30%), sad (15%), surprised (10%), angry (5%). I'd say it's a pretty safe bet that reactions are going to follow pretty similar distribution. 

Next, I am going to make a few assumptions:

1. Given how these reactions got implemented (you need to hover before they pop up), not all users are going to immediately become aware that this option even exists
2. In the first two weeks, I'd expect that only around 25th percentile of facebook active users are going to engage with the feature and it'll take the rest of the users more that two weeks to get fully on board

I think these assumptions matter because it is pretty unrealistting to expect 100% of our active users to immediately jump on board of the reactions train. We'll be setting ourselves up for failure if we don't adjust our launch goals accordingly.

Keeping our assumptions and sentimetn distribution in mind, we can proceed to do some math for reaction vs likes volume two weeks after launch. For ease of math, I'll assume that the expected number of likes over 2 week period based on our historic data is 10million. After we roll out reactions 75% of users are going to continue relying on likes to react to post because they are not yet on board with the new feature, so that's 7.5million likes. 25% of the users are going to use reactions instead of likes whenevere appropriate. Based on our sentiment distribution data, we can expect 1.25million likes, 750k haha, 375k sad, 250k surprised, 125k angry reactions. After summing everything up we end up with the following number of likes and reactions per post:

Likes: 8.75million / # of posts
Haha: 750K / # of posts
Sad: 375k / # of posts
Surprised: 250k / # of posts
Angry: 125k / # of posts

I'd expect the numbers of reactions grow and likes decline as more and more people are becoming aware of the feature, but ultimately we are likely to end up fairly close to the distribution of general user sentiment. Unless we continue adding more types of reactions, I think "like" is going to continue being the most popular way of engaging with the post, just because it is the default and the most relevant reaction for the majority of posts.  One thing to note is that these calculations did not accout for the fact that many users were most likely not liking posts that were making them sad or angry, so I think our average number of reactions + likes per user is going to go up due to this and so will the percentage of reactions vs likes.

Answer 4

Clarification Question: What do you mean non-like features here? Do you mean that we are abandoning the likes and switch it to the expressions mentioned above?

Goal: The goal of Facebook I guess is straight forward: to enhance user engagement ( I don't think monetization here is very important)

User Journey:  Let's break it down to various users segmentations.

a) Users who hugely rely on social networks to express themselves and Facebook is already part of their lives; they care more about whether he/she gets more attention when posting. They are eager to attract more comments & reactions.

b) Users who only post when they have important occasions such as birthday, graduation...etc. They constantly update themselves to their social networks. The pain point here is also to receive as many attention as they can.

c) Users who rarely post but they are active on Facebook to keep track of friends. They will click likes to their friends but that doesn't mean they are best friends.

Metrics ( Ranked by Priority and we should conduct A/B testing to the like feature )

User Engagement:  # of posts of each user && # comments in each post && comments received by each user daily  &&  average number of messages of users daily && # of friend request and new connections

User Retention: Average Time of User to Abandon Facebook && User Churn Rate

User Acquisition:   # number of new users signed up

Drawback: My hypothesis is that this feature will distance people and discourage user engagement in a sense that people who don't have a very close relationship won't express certain deep feelings rather than likes. If the user's engagement is down, Facebook's Ads business will also be damaged.

Answer 5

Repeating the question:

Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?​

Answer:

Let’s see if we can use some of the components of STAR methodology?

S = Situation

T= Task

A = Actions

R= Results

 

Start with a clarifying question?

 

Why did Facebook added expressions (additional four expression categories in addition to the Like)? Was Facebook:

 

1- Losing users/subscribers in one of the states or countries?

2- Trying to compete or distinguish from Snapchat (Chat) and Instagram (pictures)?

3- Increase engagement that will lead to retaining what they have (and stop losing)?

4- Increase average revenue per user?

 

So, we could hypothesize that Facebook is changing to expressions to:

1: Increase engagement of existing users

 

By keeping users more engaged, they could stop losing, and perhaps attract some moree from thee millennium category that are looking for more granular expressions.

 

Next, would be the Task- let’s first categorize users in certain buckets:

 

There are four types of users:

1- Who builds relationships? 35%

2- Self Promoters - 15%

3- Bearer of important news - 15%

4- Watchers - 35%

 

I am not guessing, the ratio, but #1 and #2 should be at least 70% each.

We need to reach this 70% market and keep them more engaged.

 

So, what metric, we need to track and for what duration?

Duration of at least two weeks, to make sure the users are fully aware of this new feature launch.

 

And what tests, we need to perform? They are called A/B tests to measure engagement.

 

1- How many times do the users come back and check Facebook in a day and in a week?

2- What happened to Like, did it increase or decrease? Is the sum of Like plus five other expressions ( a total of 6) is equal to when there was Like alone or did it increase? For example, without this feature, Likes were 100 Million, and with the launch of five expressions, we now have 6 choices, so the total should exceed 100 Million.

 

We can run a significance test and compare before and after values by computing P-Value. Say, we set a null hypothesis and then set a threshold of 0.05, below this value we assume that we can reject the null hypothesis and in fact engagement has increased.

You can learn about P-Value at Khan Academy: https://www.khanacademy.org/math/ap-statistics/tests-significance-ap/idea-significance-tests/v/p-values-and-significance-tests

Answer 6

 

You launched a new feature that determines whether a Facebook message was read by a recipient. What metrics would you collect? How would you know the feature was a success?

 

So by reactions, you mean instead of just giving a like, I am able to react with different emojies right?

 

Yes you are right.

 

Also, could you remind me the reactions it is like, funny, sad and angry right?

 

Yes, but we also have wow and love.

 

Ok thanks for that. So what I will be doing is, before starting directly with metrics, I will summarize this features goal and how it helps FB mission. Then from the user perspective, I will speculate on the why users are using this reactions, so I would be able to think about product health metrics. Finally I will try to think about whether it has a negative effect on FB also any risks involved and how would I measure it if any.

 

Then I think we will end up having a lot of metrics so I will try to choose 2-3 top metrics to look for.

 

Ok sounds good, go ahead.

 

So first the mission and the goal.

 

FB's mission is bringing people together closer and I can actually see how reactions help with that, beucase with reactions you are communicating with the poster and the other viewers in an effective way. So let me write down:

M -> Reactions is easy communication

 

And from the users perspective, I can see that having reactions really enhances the ux. Because before reactions I think people were hesitated to like a sad post, because it does not really convey the meaning. So from the users side this is a good problem to be solved. Since reactions do not have a complex structure, I will not go down the customer journey here.

 

Ok is that all?

 

Before going through metrics, I would like to talk a bit about the possible negative sides of this feature actually. I think there is a good chance that reactions will decrease the number of comments that is being written, and comments might be a very critical thing here since it provokes a conversation, compared to the reaction. I see like feature in the raeactions so there is no need to think about it in terms of cannibalization.

 

So having all this assumptions in front of me I will go with these metrics:

 

For company misssion:

 

# of people reacting to story -> this should be growing because we want to understand if we are helping people to connect more

 

# of unique reactors / DAU (daily basis of couse) -> This is also a good indicator of understanding passive users vs active users, because we want people to react

 

For user pain points

 

• Avg. reactions per post -> I would expect this to increase compared to likes actually.

• Distribution of reactions -> This will also give me a good understanding overal sentiment around facebook, I do not want it to be a sad place :)

  • It is possible to deep dive to this by type of posts or type of post owner too but this will be a detailed metric

• It is also a good metic to look at people in different groups like high reactors, medium reactors and low reactors -> This will help me understand how the reactions used

 

For the risks:

• Number of comments made would be usefeul

• Also number of people (unique) commenting per week would be useful

• Avg number of comments per post could also be an indicator.

 

If I were to prioritize I would go with:

• # of people reacting to story -> To see how I help company mission

• Avg. reactions per post -> I want this to be increasing too

• avg # of comments would be useful so I can compare it