How would you evaluate the success of Reactions in Facebook?

Question

How would you evaluate the success of Reactions in Facebook?

Asked at Meta (Facebook) over a year ago

10.1k views

Asked at

10.1k views

How to answer Metrics questions

Interview Guide

Top Meta (Facebook) interview questions

What is your favorite product? Why?89 answers | 263k views
How would you design a bicycle renting app for tourists?62 answers | 82.5k views
Build a product to buy and sell antiques.54 answers | 66.8k views
See Meta (Facebook) PM Interview Questions

Top Metrics interview questions

How would you measure the success of Facebook Likes?40 answers | 61.3k views
What metrics would you use to measure the success of the Save feature at Facebook? 39 answers | 87.2k views
How would you measure the success of Gmail?23 answers | 17k views
See Metrics PM Interview Questions

Invite members
Invite by email

Add another

PM Team · Answer 1 · 2019-01-30T06:07:02+0000

I explain why we have reactions: Because like is too limited. The human feelings, reactions is more varied than a simple like.

How does it work:

I click on reactions, I see a list of reactions. I select one reaction or I close the list.
I can click to see who has reacted how to a certain item (post, comment, etc).
Without need to click, I can see the list of reactions a post has received.

What is the BIZ goal to have reaction: To increase user engagement. FB needs users to spend more time on FB, so that the revenue which is based on CPM (impression ads) and hopefully CTR (click through rate, click ads) increase.

What does engagement mean in FB:

#of posts, comments, shares, likes, scrolls in news feed, videos watched, reactions to comments.

How do we expect reaction to change the engagement:

Hypothesis: People see more meaningful reactions to their posts, e.g. 5 people smiled, 2 people showed anger, 3 people cried. Seeing different reactions is more rewarding than seeing 10 likes ===> People will be more likely to post more.
Hypothesis: All users, especially passive users, will have a better way to express their feelings toward an image, text, video, comment. They feel that FB is a place were they can express what they think with one click.

In terms of design, our hypothesis is that reaction is a reward to both poster and viewer, and it is also very easy to interact with.

Summary:

BIZ goal: increase engagement
User goal: Increase the feeling of validation and engagement in what is happening is the social network

Since we are considering the reaction as a significant improvement to like, it is very important to do A/B testing.

Metrics:

Users can see who reacted how to an item:
1. %users who saw a post and clicked to see who had what reaction to this post (
2. The number of seconds users spend checking who reacted how
#of times like was chosen vs. reaction
#of times reactions button was clicked to see the list of reactions
%of times reactions button was clicked to see list of reactions but no reaction was chosen (could mean that the right reaction was not there)
distribution of the number of times each reaction was used. If a reaction was too few times, possibly it should be replaced with something else.
avg # of comments written for posts with reaction vs. posts without reaction (does seeing reaction of other people encourage the user to write a comment)
%increase in the engagement of passive users after introducing reactions. (see below the blue metrics for calculating engagement in this case)
%increase in user engagement after he gets a lot of reactions for his posts.

A/B tests:

We need to choose similar control and test groups. They need to have the same amount of activities before going live with reactions in test group. We can measure activities using:

#hours spent daily on FB
#posts daily
#likes daily
#comments daily
#share daily
Type of posts should have the same disribution, e.g. 50% of posts are text, 10% video, 40% images

List of metrics for A/B testing

#posts that get reaction vs. #posts that get like
total # of reactions for 1000 posts vs total # of likes for the same amount of posts (posts should be chosen with the same nature, e.g. 100 text posts, 100 videos, etc)
#of times 1000 posts were shared in A vs. B
total #of comments for 1000 posts for A vs. B
total #of times X videos were played in A vs. B (the best is if we test it for videos of the same nature, e.g. videos of cute animals)
avg #hours spent on FB in A vs. B
avg #posts daily by A vs. B
avg total #reaction in A vs. total #like in B
avg #of comments written by A vs. B
avg #of shares in A vs. B

Prioritisation:

To prioritise the metrics, I will concentrate on

BIZ goal, engagement
User Goal, being validated and that they can express their exact feeling.

(M, H) means BIZ Goal: Medium, User Goal: High.

Users can see who reacted how to an item:
1. %users who saw a post and clicked to see who had what reaction to this post (L, H) --> user wants to validate himself socially
2. The number of seconds users spend checking who reacted how (L, H) --> It matters for user who has reacted how
#of times like was chosen vs. reaction (L, H) --> user has been able to express what he thinks
#of times reactions button was clicked to see the list of reactions (L,L) --> The user is just discovering what is there
%of times reactions button was clicked to see list of reactions but no reaction was chosen (could mean that the right reaction was not there) --> (H,H) User has not been able to find the reaction he wants, BIZ Goal, user has not been able to engage with the feature ---> Requires a change from our side on the list of reactions we offer.
distribution of the number of times each reaction was used. If a reaction was too few times, possibly it should be replaced with something else.---> (H, L) Helps us to refine our list of reactions.
avg # of comments written for posts with reaction vs. posts without reaction (does seeing reaction of other people encourage the user to write a comment) ---> (L,L) does not tell us directly if we need to act, because the reason could be that posts have a different nature
%increase in the engagement of passive users after introducing reactions. (see below the blue metrics for calculating engagement in this case) ---> (H, H) Passive users now find it more rewarding to use FB, We have increased engagement for users which were passive earlier
%increase in user engagement after he gets a lot of reactions for his posts. ---> (H,H) Retention : user feels validated (rewarded) and wants to return to the cycle.

A/B tests:

We need to choose similar control and test groups. They need to have the same amount of activities before going live with reactions in test group. We can measure activities using:

#hours spent daily on FB
#posts daily
#likes/reactions daily
#comments daily
#share daily
Type of posts should have the same disribution, e.g. 50% of posts are text, 10% video, 40% images

List of metrics for A/B testing

#posts that get reaction vs. #posts that get like ---> (H, H) People feel validated and use reactions more. We have high engagement
total #of comments for 1000 posts for A vs. B (H, L)
total #of times X videos were played in A vs. B (the best is if we test it for videos of the same nature, e.g. videos of cute animals) (L,L)
avg #hours spent on FB in A vs. B ---> (H, H)
avg #posts daily by A vs. B ---> (H, L)
avg total #reaction in A vs. total #like in B ---> (H, H) People feel validated and use reactions more. We have high engagement,
avg #of comments written by A vs. B ---> (H, L)
avg #of shares in A vs. B ---> (H, L)

In case of A/B testing we need to calculate if the increase between two groups is statistically significant or not. p-value of 5% could be a good indication.

I will choose all metrics with (H, H) and (H,L)

Summary:

We defined why we introduced Reactions, and we explained what success means form the eyes of our business and from the eyes of user. BIZ wants to increase engagement and User wants to feel validated and able to express what he thinks fast and easy.

Based on this we defined two sets of metrics.

1) measures for the success of reactions among users for which reaction is launched.

2) The measures for A/B testing.

We prioritised metrics based on BIZ goal and User Goals and chose metrics that maximise both Goals and then metrics that maximise BIZ Goal. Because I believe a high engagement (BIZ Goal) means people have felt rewarded (User's Goal).

PM Team · Answer 2 · 2018-08-12T21:01:18+0000

This is a classic Facebook metrics – execution question.
For this answer, I would follow the framework below:
1. Goal and success metrics of product/feature in question (In this case, the feature is Facebook reactions)
2. Goal and success metrics/impact on eco-system because of feature in question (In this case, the eco-system is the Facebook news feed of which Facebook reactions is a sub product or feature. So I will go through the metrics that I need to monitor on the Facebook news feed to understand the overall impact of Facebook reactions)
3. Test and measurement methodology – How will you A/B test, what metrics will you track? How will you determine cohorts in each group? How will you make sure that the cohorts don’t get mixed up during the test (avoid test corruption?)
So, following this framework:
1. When I first launch, reactions, I will state my goal/purpose as:
a. Customer goal: Human emotion has a wide range and variety. The Like button limits the range of emotions that someone can express while engaging with content on their news feed – provide customers with more options in addition to “Like” to express themselves.
b. Customer goal: Increasing social validation – By making it easier to engage with the news feed through reactions, you increase the likelihood of activity across friends and increase the feeling of social validation.
c. Business goal: Increase engagement of users on the news feed- By engagement I mean – average number of shares, posts, likes, reactions, comments per visitor during a 30 day period
d. Business goal: Increase retention: Increase the rate of repeat visits (daily, weekly, monthly) per user

2. These are the metrics I will test specifically for this feature:
a. Discoverability/Acquisition: Are people able to discover Reactions? It is currently hidden under the Like button- is it easily discoverable?
b. Usage: For the visitors that discover Reactions – How many people are clicking on the reactions? What is the total number of clicks? I would measure this for at least a week.
c. Usage by reaction type: How many clicks per reaction type? This would help me understand if I even have the right set of reactions in there or do I need different reactions?
3. These are the metrics I will test for in the eco-system:
a. Usage/Engagement/Content Creation: For the user group that is exposed to reactions (Test Group)– does their average activity during a 30 day period increase due to reactions – Are they liking/reacting/commenting/sharing more as compared to people (Control group) who do not see reactions? Are they creating more content than before the test (pre-post analysis) and as compared to the control group (A/B)?
b. Retention: For the test group that is exposed to reactions – do we see higher repeat visits (to see how their friends have reacted to their posts/shares or if others have liked/reacted to posts that they have liked/reacted)
c. Engagement: Does it increase time spent and length of scroll on news feed? Test A/B and see if there is statistically significant difference?
d. Monetization – Is there higher click thru rate on ads for visitors who were exposed to reactions? (Probably least priority as FB does not like to talk about monetization much).
4. A/B test tactics
a. I already mentioned what I would test in my A/B test across test and control groups. I also talked about pre-post analysis a little bit
b. You may be asked follow up questions on how you would determine incrementality? How would you prevent control group from seeing reactions in their feed if their friends are in the test group?
c. The way to answer the questions above might be:
i. Make sure that test and control group are very apart in the Facebook social graph.
ii. If for some reason, someone on the test group and someone on the control group are part of the same social graph, have an engineering solution to convert a “reaction” from the test group in to a “like” when seen by the control group.
iii. Also, to truly measure incremental improvement in engagement, take people in test and control groups with similar levels of prior engagement. How do you determine prior engagement – You could look at average activity rate per user (average number of likes/shares/posts/visits/friends) etc and make sure that test and control groups have people with similar activity rates. This way you can test to see if “passive users” became more active? And if “active users or highly socially engaged people” also had a marked improvement in their social activity.

Simran Kohli · Answer 3 · 2024-08-12T21:43:36+0000

How would you evaluate the success of Reactions on Facebook?

What do we mean by success here?

We can define it ourselves

What do we cover in scope when we say ‘Reactions’?

All type of reactions i.e. likes, hearts, care, sad etc

Assumption
We have recently launched the reactions piece and before this we only had ‘Likes’ on to the platform

1. Describe the feature

Reactions on Instagram allows the users to expresst themselves wrt the content they consumed

These can be in the form of Likes, Heart reacts etc

2. Goal

The goal of the feature is to help people engage better on the platform and express themselves more meaningfully

3. Walk through the user journey

User A posts —> User B sees the post -> Basis how he feels, he reacts (Like or heart or care etc)

—> User A receives acknowledgement about how User B and other users like him felt about it

Other case where User C sees User B has done a ‘like’ to User A’s post and he also chooses to react or not

Overall: I have boosted Interactions/ Engagement in the system

While I have some sense of where ‘d like to focus on , I would still like to take a holistic view across various metrics and then deep-dive into what suits the most basis some evaluation

Awareness: Measure the level of awareness of the feature among the users

FAQ’s regarding the feature

Coverage of the people who have used it and impressions of the posts where reactions are live [For eg A user has the last 2 posts where Reactions have been used by some people, then what are the impressions of those 2 posts in terms of unique users]

Market Campaign Reach

Acquisition: Measure how the users are becoming aware of the feature and the number of users acquiring it

How many users have done their first reaction (apart from likes) as a % of total active users

Activation: Measure how many users are using the feature for the first time and whether they are successful in using it

First-time adopters to Reactions

Average Reacts per user [Here, my assumption is that using it twice or more is an indicator of successful usage since one-time usage can also be an error]

Engagement: Measure the frequency and duration of feature usage

Usage of Reactions overall WoW (to understand the adoption trends)

Share of Likes vs Reactions (To understand the shift)

Likes per impressions [Assuming that now I can express myself better so I would engage more with the posts]

Retention: Measure the no of users returning to the feature and how often they use it

Retention of users W/M for those who have adopted the app

Average Sessions per user per day [to understand if this has engaged the user more to the platform]

Average Session time per user

Monetization: Measure the revenue generated by the feature

This is very far fetched but still putting out some thoughts

Boost in the usage-> Any boost in the revenue per user

For instance, with more reactions more engagement, I would be able to understand the user needs more hence more relevance for ads hence better targeting- better conversions and roi

Referral: Measure the number of users who refer others to the feature

In the context of social network effects, has this boosted the DAU/MAU where inactive users would have come back to the platform in lieu of word-of-mouth

Any campaigns that we ran (eg on Playstore, Appstore etc) that had better conversions and results

[Devil’s Advocate: Looking at these campaigns in isolation would not be a good benchmark because feature release has some initial exploratory phase where every metric looks good hence it would be good to compare with other feature launch campaigns]

Shortlisting the Metric Category

Based on the overall product’s objective and the platform’s goal, I would look at a combination of Adoption and Engagement for this feature

Adoption
L0:No of the users who have done at least 2 reactions [To ensure we do not cover accidental usage]

Engagement
L0: Boost in the average reactions/likes per post per user L1: Boost the average sessions and session time on the platform

Angshuman Gupta · Answer 4 · 2022-06-06T16:09:03+0000

Assumptions/Clarifying questions

Feature of FB
Reaction feature for facebook.com itself / all countries / users
Reaction mean like/dislike/laugh/anger etc
Reactions on posts

My Approach

I want to begin with FB’s mission and then find out the goal of this feature
User Segments and their needs
North star connecting with all the user segment
Breakdown the north star into metrics / secondary metrics
Trade-off / counter metrics

FB Mission: Connecting people and power to build community

Goal of Reactions: Helps to connect with people / showcase emotion / goal -> engagement

Segments:

Content Creator - I want people to react/engage with their context / thereby validating my context and inspiring me to create new content
Interactor / Reactions - Wants to engage in an easy way
FB Overall - Increase the engagement overall

For simplicity, we have assumed Content Creator and Interactor are different but in the real world both can play the same role

--------------------------------------------------------------------------------------------------------------

North Star: # of reactions per post

If we have more reactions per post there will be more engagement and the content creator will be incentified to create more content

Breakdown the north star / secondary metrics

Content Creator →
1. # of posts per user
2. # of posts with at least X reactions
3. # of context creator
Interactors
1. # of posts viewed per user
2. # of posts reacted or reacted/viewed → CR
3. # of reactions per user
4. # of interactors

Trade-Off

Increase content creators vs interactors
1. We should have more interactors than content creators
We should consider # of posts with at least X reactions
1. X should be 10% of # of friends of the creator

Counter Metrics

Negative reactions (dislike)
Negative Content → Reactions to that

shrutc · Answer 5 · 2021-04-26T07:06:36+0000

Facebook reactions are an improvement over the likes feature where users can express their exact emotions such as love, sadness, anger, amusement, etc for a post. This helps users narrow down their emotion, instead of just like which is very generic. Its more convenient to react rather than comment the exact emotion.

Clarifying question – Are we measuring the success of FB reactions to improve usage or engagement or to attract more users or increase revenue? Based on the goal I will define the metrics.

Goal – Measure success of FB reactions to improve usage/ engagement on the platform

User Journey for FB reactions

Users will know of this feature while looking at posts on FB. When they want to show an emotion, this will trigger the use for the feature and based on the post they will click on a particular emotion. Thus, the user will engage and show a reaction if the post demands it. Users may see similar posts or some other posts and keep using the feature. Based on what posts they like or how much they scroll based on their emotions for a particular post, FB will gain as more scrolling means more money for FB.

Metrics for each step of the user journey

1. Knowing the feature – these metrics will help evaluate if the user has seen the feature and knows how to use it.

- Are users aware of this feature?

- How users came to use this feature?

- Number of new users using this feature in their initial days

2. Trigger for using the feature – have users ever felt the need to use the feature.

- percentage increase in number of clicks by users – If this is high it means more users are using this feature

- Number of reactions used per day per user

3. Do these reactions help in engagement

- percentage users that use reactions after viewing a post

- number of users that scroll further after using reactions

- number of users using reactions on a daily/monthly/quarterly basis

- Reactions most used on a daily/monthly/quarterly basis

- Number of users sharing a post after reacting on it

4. Multiple usage by the same users – are users using this feature repeatedly

- Number of reactions used per user in daily/ monthly/ quarterly

5. monetization – are these reactions generating money for FB.

- Time of day when reactions are used – FB will show more relevant posts at a particular time depending on this metric

- Type of posts that get most reactions – this will help FB to post more such posts by understanding a user’s likes.

-percentage users that scroll further after using reactions – this will help gauge if users are indeed staying on FB after having a reaction to a particular kind of post. More scrolling means more money through the ads.

The goal is to use these metrics to increase engagement on FB.

The metrics I would implement and monitor are -

1. percentage users that use reactions after viewing a post

2. number of users that scroll further after using reactions

3. number of users using reactions on a daily/monthly/quarterly basis

4. Reactions most used on a daily/monthly/quarterly basis

5. Number of users sharing a post with friends after reacting on it

Priority based on ease of implementation would be – 1, 3, 4, 5, 2

Summary

So, to measure the success of FB reactions in increasing engagement, I will first split all metrics to see which are relevant to this phase of the user journey. Based on the ease of implementation, I will implement the metrics and monitor them on a daily basis. An increase in these will mean a positive thing, whereas a dip will mean more analysis needs to be done to improve the reactions so that they engage users.

MythicCabbage · Answer 6 · 2020-11-20T06:00:21+0000

My understanding is that Reactions allow Facebook users to give quick feedback on a post, going beyond the original Like button. Reactions create engagement by allowing users who might not write a comment to connect with the poster by expressing feeling through Reactions. Reactions go beyond Like to include Love, Hug, Ha-ha, Wow, Sad and Angry and can be more appropriate than a simple like, especially on a post expressing something surprising or sad.

Facebook's goal is to connect users and give them tools to express themselves. Reactions help connect users in a deeper way.

I would think that Reactions help Facebook gather better data about a particular post and that can feed into the Newsfeed algorithm which decides which posts to display. For example, if there are a lot of Sad Reactions that could be a condolences post to display to many friends in the Newsfeed. I would track whether a post that gets an initial # of Sad Reactions for example, after distributed widely in the Newsfeed, continues to get more Sad Reactions, which would show that the assessment of a post as a "Sad" post to distribute widely was a correct move by the Facebook Newsfeed algorithm.

The metrics I would use to evaluate Reactions would be based on engagement. We can measure engagement by both time on Facebook, and also the number of posts that get any type of Reaction, comment, or share. For this particular evaluation, I would measure avg # of Reactions per post in general, and then look at the distribution of Reactions to see which sentiments are being expressed most. Also which percentage of Reactions are non-Like reactions - in other words, are users responding to the non-Like reactions and finding them useful? I would also look at whether the average # of Reactions has increased now that users have more options than just Like.

I would show Reactions first to a subset of users and measure the following: whether the engagement on posts goes up, measured by # of Reactions, and also whether week-over-week or month-over-month more posts than average are created (i.e., posters are getting more Reactions to their posts, thus that is motivating them to create more posts). Also, I would measure overall time on the site, because users could be staying on the site longer as they can easily engage with more posts and more "interesting" posts that evoke sentiment are displayed to them on the Newsfeed. All of these metrics increasing would show success of Reactions.

I would consider Reactions successful if # of Reactions on posts increases. But I would be sure to measure this over time, as there could be more use than usual of a new and exciting feature. So I would track # of Reactions month over month and make sure they increase. I would also track avg # of posts created per user over the same timeframe.

I would also make sure to track if there is any Reaction that is rarely clicked, I would consider removing it from the Reactions.

I would gather some qualitative data as well, through user focus groups or surveys to determine if any crucial Reactions are missing.