How you would prove the value of the Reactions feature? How would you decide whether to launch it or not?

Start with clarifying questions. Standard so will not dive deeper in to that portion.

Share/Align on understanding around Reaction Feature 

Allow people to express a range of emotions for a post. This means of expression is more natural way for people to express. 

 

Aligns with FB mission

Fosters a sense of more deeper connect through sense of shared emotions (vs simply likes) and aligns with Facebook a mission

 

Goal- because more natural way is aimed towards driving user engagement with content on the platform . 

 

Customer journey

Pertinent to this feature :

• Awareness
• Engagement
• Repetition
• Exit 

FB likes is a widely know feature . Awareness is not an issue. Furthermore, reactions are prominently displayed below posts, boosting awareness further. 

 

Engagement and repetition are most interesting metrics because of our goal as well. 

Keeping with our goal I want reactions to drive people react to the posts with emotions - validates my hypothesis that more people will use it, and larger percent of posts will receive reactions and overall reactions are more complete way to express than just likes

• customers are likely to react more when they can express in emotions than simply ‘likes’ 
• posts/news feed will see uptick in engagement because like was not the right way to connect with the post
• Reactions will be roughly XX% of the sim total of all likes and other reactions combined. 

1. Lift in reactions per post
2. Lift in number of reactions per user 
3. Correlate lift in reactions to number of shares and comments

• like all metrics we will need to measure these across dimensions of countries , type of posts etc.

Moreover, interesting to see the these numbers at an aggregate level as well

 

Go/ No go decision

Test hypothesis through Controled release of the feature and AB test to validate the hypothesis. Launch  if lift is statistically significant. 

This is an interesting one. Let me clarify my understaning of Reactions first. Reactions are on many products of FB but for this case we can likely focus on FB's primary product which is News feed. Reactions are enhanced "Like" in the sense that you can simple like something or you can actually like something via means of few emoticons. (correct).

Ok, now that I have confirmed my understanding of reactions and narrowed our focus to News Feed product, my sense says that we may want to measure the success of this feature against certain criterias during A/B testing / Beta testing. Does that sound like I may be alinged with you? (yes. woud like to see what how you approach this though).

Correct. Now that we are aligned, I think let's think of why we might want to implement Reactions so that it can help us drive towards how to evaluate its success or not.

I think Reactions are well associated to FB's overall goal which is to increase connections and communications amongst users across FB. FB is interested in this since this is what drives its primary business of digital advertising + FB is all about social networking thereby making the people feel better connected across the globe. Thinking of this and what Reactions are, I think it our goal should be to see if Reactions can drive greater Engagement. Engagement can result in greater euphoria amongst people leading to Retention which is important but since Engagement is the primary thing that can lead to retention and monetization business impact for FB, I think we should be more interested in Engagement. Is that fair? (sure). So I think that while we should measure Engagement we should also measure Retnetion as a secondary metric, I would like to focus on Engagemet for our case if that's ok. (sure). I also do want to mention that based on past data of Like for instance, we can see the inflection point that changes Engagement to Retention so we know when to measure for each. I also think that while Retention is important Engagement is more imp because a user can be retained as a FB user but if he/she doesn't login frequently on FB then business impact from that user is null.

Reactions may cause greater comments amongst users, greater # of posts being read, higher amount of time being spent since responding to a post via reactions is more fun and easier, etc. So I think we should measure some of the below over a week. We should conduct a Beta testing (select users) and these measurements should be done on that sample.

1. % of users using Reactions vs. Likes. (want to see people using reactions increase).

2. % comments posted on posts with greater reactions vs. likes and % of comments posted on posts with greater likes vs. reactions. (this may be too complicated and we may not have a good decision from this since all posts are not equal).

3. % of time spent by users who use reactions vs. % of time spent on News Feed by people who only have access to Likes.

There may be additional ways to measure success of Reactions, but in the interest of time, let's stop here to analyze these. I think the most important success measure here might be % of users using REacitons vs. Likes (this would have to be done on Beta testers since they would have both features). I say this since if REactions are more interesting, then I would expect mmore people to be using that vs. Likes. There is an aspect where initially you would see higher usage of Reactions initially just becasue it's that "new" thing to do or use (so you are kinda cool - think of most users of FB who are young folks so they care about coolness). So we may need to measure this for a while till that "new" fade has lost its shine. Since FB has millions of users and young people are their primary customers who tend to use FB daily, measuring and tracking a trend over a week might be sufficient, however, we may want to meausre this for upto 2-3 months to see if the trend continues or reverses. If it reverses, we may need additional time to see the data.

I believe above slow but careful measurement will provide us a definitive confidence /inconfidence in our new feature. I would decide to launch based on data I observe.

All in all, to decide to launch a feature tells me we need testing with real time users beforehand which lead me to recommend Beta testing and then measuring for success amongst Beta users.