Imagine you are the PM in charge of Reactions on Facebook - the new way to interact with posts by using “love”, “haha”, “wow”, “sad”, and “angry” reactions. What would success look like in terms of number of non-like reactions per post at launch and how do you come up with this? Would this number differ by reaction? Why or why not?​

You launched a new feature that determines whether a Facebook message was read by a recipient. What metrics would you collect? How would you know the feature was a success?

So by reactions, you mean instead of just giving a like, I am able to react with different emojies right?

Yes you are right.

Also, could you remind me the reactions it is like, funny, sad and angry right?

Yes, but we also have wow and love.

Ok thanks for that. So what I will be doing is, before starting directly with metrics, I will summarize this features goal and how it helps FB mission. Then from the user perspective, I will speculate on the why users are using this reactions, so I would be able to think about product health metrics. Finally I will try to think about whether it has a negative effect on FB also any risks involved and how would I measure it if any.

Then I think we will end up having a lot of metrics so I will try to choose 2-3 top metrics to look for.

Ok sounds good, go ahead.

So first the mission and the goal.

FB's mission is bringing people together closer and I can actually see how reactions help with that, beucase with reactions you are communicating with the poster and the other viewers in an effective way. So let me write down:

M -> Reactions is easy communication

And from the users perspective, I can see that having reactions really enhances the ux. Because before reactions I think people were hesitated to like a sad post, because it does not really convey the meaning. So from the users side this is a good problem to be solved. Since reactions do not have a complex structure, I will not go down the customer journey here.

Ok is that all?

Before going through metrics, I would like to talk a bit about the possible negative sides of this feature actually. I think there is a good chance that reactions will decrease the number of comments that is being written, and comments might be a very critical thing here since it provokes a conversation, compared to the reaction. I see like feature in the raeactions so there is no need to think about it in terms of cannibalization.

So having all this assumptions in front of me I will go with these metrics:

For company misssion:

# of people reacting to story -> this should be growing because we want to understand if we are helping people to connect more

# of unique reactors / DAU (daily basis of couse) -> This is also a good indicator of understanding passive users vs active users, because we want people to react

For user pain points

• Avg. reactions per post -> I would expect this to increase compared to likes actually.

• Distribution of reactions -> This will also give me a good understanding overal sentiment around facebook, I do not want it to be a sad place :)

  • It is possible to deep dive to this by type of posts or type of post owner too but this will be a detailed metric

• It is also a good metic to look at people in different groups like high reactors, medium reactors and low reactors -> This will help me understand how the reactions used

For the risks:

• Number of comments made would be usefeul

• Also number of people (unique) commenting per week would be useful

• Avg number of comments per post could also be an indicator.

If I were to prioritize I would go with:

• # of people reacting to story -> To see how I help company mission

• Avg. reactions per post -> I want this to be increasing too

• avg # of comments would be useful so I can compare it

C: Clarify the Question

Who are the reactions for?
What are the goals of reactions?
Why was it trimmed to the said 5?

I: Identify the Customers

The two main customers for this feature that I see are businesses who need this information, and people who have been resorting to the like button to respond to posts. Who does it add more value to? I would think this would add more value to businesses as it can directly lead to a more efficient monetization with this additional layer of understanding of a user's sentiment. 

There are two aspects of success

1. (Users) Market Adoption and Engagement due to the reactions. Will people post more, respond more, now that there is an option of an appropriate response?

2. (Businesses) How much more efficient can this information make market targeting?

• Market Adoption
  • Hypothesis: Does the availability of different response options increase the post engagement of the users?
    • Avg count of reactions per post daily pre-launch of non-like reactions
    • Avg count of reactions per post daily post-launch of non-like reactions (hope to see an increase)
    • Avg count of reactions per 7 days (assuming weekends have a spike of activity) pre-launch of non-like reactions
    • Avg count of reactions per 7 days post-launch of non-like reactions
  • Does the availability of different response options instigate people to share more news?
    • Avg number of posts per day pre-launch
    • Avg number of posts per day post-launch
  • Does biasing to positive emotions garner more engagement with the user?
    • Correlation of distribution of categorized posts according to the non-like reaction vs Avg engagements per post.
• Business Monetization
  • Does targeting using non-like reactions lead to more engagement with the user?
    • Avg count of single clicks on targeted posts
    • Avg count of two-level clicks on targeted posts
    • Avg count of responses (comment, shares, reactions) on targeted posts

First of all, I'd probably need some sort of clarification about what "at launch" means. Are we looking at the performance the very next day after the feature is launched or are we giving ourselves a few weeks to see how things are going to play out? I assume it's the latter and we are going to be evaluating success two weeks after launch.

The goal of reactions feature is to give users more freedom in expressing how they feel about content that gets posted on Facebook. This feature should be particularly appealing to users whose primary method of engagement on FB is via the like button, which, I assume, is going to be the vast majority of FB's userbase.

First, to set some baseline metrics I can look at the historic data to estimate things like 

1. number of likes over the span of 2 weeks
2. number of posts over the span of 2 weeks 
3. avg. number of likes per post
4. avg. number of likes per user

Second, I can look at the distribution of the sentiment of facebook comments. I am pretty sure this data will already be available because that's probably how they came up with these 5 reactions to begin with. Let's say the distribution looks like this: neutral (50%), funny (30%), sad (15%), surprised (10%), angry (5%). I'd say it's a pretty safe bet that reactions are going to follow pretty similar distribution. 

Next, I am going to make a few assumptions:

1. Given how these reactions got implemented (you need to hover before they pop up), not all users are going to immediately become aware that this option even exists
2. In the first two weeks, I'd expect that only around 25th percentile of facebook active users are going to engage with the feature and it'll take the rest of the users more that two weeks to get fully on board

I think these assumptions matter because it is pretty unrealistting to expect 100% of our active users to immediately jump on board of the reactions train. We'll be setting ourselves up for failure if we don't adjust our launch goals accordingly.

Keeping our assumptions and sentimetn distribution in mind, we can proceed to do some math for reaction vs likes volume two weeks after launch. For ease of math, I'll assume that the expected number of likes over 2 week period based on our historic data is 10million. After we roll out reactions 75% of users are going to continue relying on likes to react to post because they are not yet on board with the new feature, so that's 7.5million likes. 25% of the users are going to use reactions instead of likes whenevere appropriate. Based on our sentiment distribution data, we can expect 1.25million likes, 750k haha, 375k sad, 250k surprised, 125k angry reactions. After summing everything up we end up with the following number of likes and reactions per post:

Likes: 8.75million / # of posts
Haha: 750K / # of posts
Sad: 375k / # of posts
Surprised: 250k / # of posts
Angry: 125k / # of posts

I'd expect the numbers of reactions grow and likes decline as more and more people are becoming aware of the feature, but ultimately we are likely to end up fairly close to the distribution of general user sentiment. Unless we continue adding more types of reactions, I think "like" is going to continue being the most popular way of engaging with the post, just because it is the default and the most relevant reaction for the majority of posts.  One thing to note is that these calculations did not accout for the fact that many users were most likely not liking posts that were making them sad or angry, so I think our average number of reactions + likes per user is going to go up due to this and so will the percentage of reactions vs likes.

Let me start to answer this Facebook metrics PM interview question with a few questions of my own:

1. How're we doing it right now? --> Users have an option to 'Like' the post

2. Why did we launch 'Likes' in the first place? --> What do you feel?

Let me think about it. First off, let me segment the different users:

1. Heavy Posters --> These are the 1% of the users who are always active on Facebook. They're posting all the time, adding new friends, commenting, etc. They're are the content creator!

2. Frequent Posters --> These are the 9% of the users, who sometimes participate on daily FB's activities. 

3. Occasional Posters/Lurkers --> These are the 90% of the user who usually lurks around

As a FB PM, I would have seen this pattern and wanted more participation/engagement from all user segments. --> Seems fair assumptions

Let me list the needs/pain-points of each segment:

1. Heavy/Frequent Posters  --> When others don't engage on their posts, they lose motivation to produce content/posts and thereby they come less on FB, and FB loses its power of having an engaging community. 

2. Lurkers --> They like the posts but don't want to comment, they want an easy way to show their appreciation.

Looking at the above problems, 'Likes' seems to be the solution. 10% of users posting and 90% 'Liking'

Now, extending the above analogy, I see we can have more emotions - “love”, “haha”, “wow”, “sad”, and “angry” catering to the distribution of the 90% of the users as not all 90% will have a need to 'Like', they have other emotions like “love”, “haha”, “wow”, “sad”, and “angry”

So, from the above hypothesis, I see # of posts having only 'Likes' will < #posts having 'reactions'

Other Success metrics will be:

1. # of posts with reactions / # posts without reactions

2. Reactions leading to more posts --> # reactions / # total posts --> Cohorts of posts before and after reactions can be measured

3. Reactions leading to more active users --> # unique users before and after reactions

4. Reactions leading to other activities --> 'Add Friends', 'Comments', and 'Chats'

To summarize, I feel reaction will increase overall engagement as we're targeting a larger distribution of user and their emotions, and an ability to portray these emotions will lead to an increase in engagement