You launched a new feature that determines whether a Facebook message was read by a recipient. What metrics would you collect? How would you know the feature was a success?

Clarifying questions:

1. What was the intent to launch this feature

Ans: To enhance user engagement. Once user knows that the sender gets time stamp of message read, user will feel to reply in timely manner Vs just reading and then forgetting it later.

 

2. Does FB has feature to turn off this feature?

Yes

 

Goal: FB's goal is to build communities and the messenger helps with this mission to help users communicate effectively. This feature creates a sense of urgency to reply and helps the communication going Vs in normal situation user may otherwiese forget to reply. So the goal will be increased Engagement on Messenger.

 

User journey: User sends a message to another friend.

The friend gets notified (if on), or reads it when opens the Messenger or FB app. 

May or May not reply right away

Metrics: Since the idea is to keep chain on messages going on in timely manner. I will focus n the folloing

Engagement:

1. # of Messages created per month and trend going forward.

2. Average Reply time should decrese with time.

3. MAU,DAU on messenger should increase over time.

4. # of unreplied messages should decrease over time..

5. Average # of messages per user per month should increase.

6. % of time people check Read status when they do not receive reply

Retention:

1. %of users who sent one message over 30 day period.

 

North star: #1 because the intent is to increase engagement by generating timely response.

 

Trade Off s to consider:

1. DAU and MAU may decrease because people may not open the message to read as they don't want to feel pressurized to answer right away 

2. Average reply time may increase as receivers may not open the message until they are ready to reply and then may forget over time to reply .

 

 

 

Structure:

1. Clarify what the feature is (Assumed it's understood for the purpose of this write up)

2. Layout FB's overall mission

3. Brainstorm how this feature can play a part in advancing that mission

• Set strategic context and determine a north star metric to anchor to
• Understand key user groups 
• Define what does success means for each group and for the feature overall (identify key buckets of metrics to measure)
• Prioritize and discuss trade offs as we go

• Stickiness (DAU/MAU)
• Session Time/Time Spent in App per user per D,W,M
• Avg Response Time (amount of idle time before responding to a message or sending a follow up message)
• Chain Length (Number of messages sent before getting a response)
• Again, Segment by Category of Messages - Friends (1:1), Businesses, Groups

• Perform cohort analysis to understand if users that have this feature are interacting with FBM more over time 

• Measure incrementality in transaction volume for businesses
• Segment by type of business
• Use learnings to see if more businesses can see FBM as a means to engage with their customers 
• Measure incrementality in discount rate revenue as a result of above

 

Clarification Questions

1. Is the feature a part of Facebook Messenger or has this feature been introduced in WhatsApp or Instagram or any of the other Facebook properties - Facebook Messenger

Structure of the Response

1. Details of the feature
2. Feature Goals
3. Metrics
4. Prioritization

Feature Discussion

A big part of the human conversation in the real world is the fact that when we talk to another person in real-time we constantly get feedback from them either from hmms, yeses, and other types of responses. This feedback that we get from the other person helps further the conversation easily. So, this feature is essential for boosting user engagement in messenger.

As we move more towards digital conversations using text, we tend to lose the human connection to the conversation. No person, no voice, no feedback. In a bid to bring back the more human elements of the conversation to text, read receipts play a big role.

Types of Conversation:

1. 1:1 conversations - Considering for this discussion
2. Conversations in Groups

Goals

Increase User Engagement in the product

Metrics for the Feature

1. Average time spent by users in messenger
2. Average conversation duration where read receipts are on
3. Average conversation length (in messages) where read receipts are on
4. Time lapsed between successive messages in a conversation where receipts are on
5. Reduction in time between successive messages when both the users are not on chat at the same time

I will prioritize the Average time spent by users in messenger as the North Star metric. Some guard rail metrics I will also look at is retention, to ensure that users are using the product for longer and just as frequently.

 

Me: first I would like to start by defining FB Mesenge, Since FB has several products like FB Messenger, instagram and Whatsapp for messaging, which one would you like me to measure?

Interviewer: Messenger

Me: So the goal of the feature is to increase user experience and engagement on the app by providing additional information about a message. Are there any other goals we are trying to meet?

Interviewer: No

Me: I would like to walk through the user journey

1. User opens FB messenger

2. User opens a 1:1 chat and sends a message

3. Recipient receives and reads the message and user is notified that it is read

 

or

1. User opens FB messenger

2.User opens a group conversation and sends a message

3. Recipients receive the message and read and user is notified of who all read the message

 

Am i missing any crucial steps in between?

Interviewer: No

Me: 

here are some metrics i would use to measure the success of read receipts on FB messenger

Awareness:

Avg. number of times user opens the "read receipts" page or popup

Acquisition:

% increase in users per session

Engagement:

Avg. number of followup conversations 

% of users message was sent vs read

% increase in time per user per session

Avg number of users removed from group conversation

Referral:

% increase in number of users in a group conversation per group 

% increase in new users conversing per user

 

RICE prioritization:

Based on Reach Impact Confidence and Effort, I would like to prioritize the above metrics. 

 

In summary, I would measure Avg. number of followup conversations, % of users message was sent vs read, % increase in time per user per session and % increase in new users conversing per user

 

So, i have to measure the success of FB read messages and determine if the product was a success.

The feature works as follows: I guess read messages will work the same way on faecbook as they do on Whatsapp.

When i send a message, there comes a single tick. When the user logs into FB account there comes a double tick( whatsapp equivalent of delivered message). When he reads it and both have read receipts opted in, the double ticks change color.

The goal of the feature is to increase conversations and provide feedback to the sender. More people turning read receipts on would mean there is a feedback info going to the sender of his messges and more replies mean more conversations.

How to measure success?

1. No. of users opting in-> Adoption of the feature

2. No. of messages sent per user before and after the feature launch-> Engagement

3. No. of new conversation/user before and after the launch-> Enagagement

4. Length of conversations before and ater the launch-> Engagement

5.  No. of people turning it off after initially turning on-> User Churn

6. No. of blocks/user from messenger before and after launch-> DisEngagement

7. No. of emojis and stickers used before and after-> Engagement

8. No. of messenger logins/user in 1 week before and after-> Retention

 

Out of these, 1,2,3,5,8 are high priority metrics for success than 4,6,7

You launched a new feature that determines whether a Facebook message was read by a recipient. What metrics would you collect? How would you know the feature was a success?

 

Feature description 

Just like the check mark in WhatApp (WA), to launch a read/unread in FB messenger. Note that this feature may have negative effects as it might be perceived as invasive and that impact needs to be measured as well. 

Questions:

1. Will there only be a state of read? Or all the statuses as in WA?

2. Will users be able to hide the indication from the sender?

   1. Will hiding it have a negative impact on the hider (like to WA) 

3. Will this only be in FB messenger? 

 

Feature Goal and overall product and company goals

1. Provide users with better viability to the level of interaction. “Why is she not responding to my message”? Oh she did not see it … 

2. Messenger  - creates instant interaction b/w users 

3. Increase user interaction 

4. Allow users to share content in small (1x1) groups 

5. Instant sharing and gratification 

6. Increase user delight through various means of self expression in visual appealing ways 

7. FB - Create and maintain relationships b/w individuals and communities 

Summary 

1. Communication 

2. Interaction (intensity, frequency, variety) 

 

Product Features 

1. Posts

2. Responds 

3. Likes

4. Shares

5. Emojis

 

Note: When analyzing the data there should be some metrics that need to be broken down by segments to insure that there is no inherent biases:

1. Platform - web/mobile/tables - IOS/Android 

2. Country - due to cultural differences

3.  

 

Audiences 

Readers and writers … in this case they would be of the same profile as this is interactive product and this feature applies to both parties 

 

User Journey 

Writer 

1. Send a message

2. See message status 

3. Message status change to read 

Reader

1. Read a message 

Metrics through the User Journey 

1. Awareness / discoverability 

   1. # of users on desktop that moused over it 

   2. Turn feature off (do not show my message status)

2. Acquisition 

3. Interaction

   1. # UU 

   2. # Conversations 

   3. # C/UU

   4. # of messages in a conversation (as defined per conversation < 10 min gap) 

   5. Delta time in responsiveness (conversation rate) 

   6. # of friends that interacted / total # of friends 

   7. # of responds to alerts (push notifications) 

   8. # of social interactions (like/ emojis) 

4. Retention  

   1. MoM returning users 

   2.  

5. Revenue 

   1. N/A

6. Referrals 

   1. Sentiment analysis 

 

Valuation of the metrics 🠊 prioritization 

 

Prioritization of metrics 

1. Sentiment analysis in this case is very important as this feature may have negative impacts 

2. # of users that turned the feature off

3. UU over time D/W/M

4. Churn 

5. # of messages/conversations

6. Rate (in time) of conversation respond rate 

 

Testing method

I would recommend that we do a sentiment analysis to ensure that overall user delight is not impacted by this new feature. And that users are able to find out how to turn this feature off. 

In addition, some metrics need to see if are no not platform and or cultural dependent 

I would recommend doing a A/B test to measure over stats on increase in the 

Can this be gamed

1. This metric is based on a few assumptions which may not hold true and thus there are risk

2. More messages exchanged May not necessarily lead to stronger social connections - this belief needs to be tested with data

3. It's a high level metric and changes in it over time may not be directly attributable to the feature

4. There is a risk that the metric may go up due to more one-sided conversations and thus we need to check message distribution also

Summarize

The read/unread feature goal is to improve the quality of the social interaction from within FB as a whole and Messenger specifically. Measuring the quality of the conversation can be tricky and will need to used via proxy metrics such as : Conversation chain length, conversation rate (time differences b/w messages) , # of conversations and retention. 

 

I urge the team to also test negative sentiment that can build regarding privacy concerns. 

Objective

For this Facebook metrics question I will assume that the feature is for Facebook Messenger. 

The objective of this product is to enable "users to have private conversations to drive strong social connections"

The read feature has an objective of "Helping Senders of message understand the conversation engagement"

Users and Use Cases

There are now different of user personas who are in the messenger eco-systems

1. Senders of messages
   1. individuals to individuals
   2. Individuals to Groups
   3. Individuals to Businesses
   4. Businesses to Individuals
2. Receiver of messages

1. Senders sends a message
2. Message gets notified once receiver logs in and sees the message

• Let senders decide on ways to forward the conversation (or not)  based on read receipts in order to drive richer more engaged conversations
• Make receivers aware that senders know that they have seen it in order to drive engagement

Metrics

We can start looking at the metrics in a number of ways

1. Usage & adoption
   1. Senders: % of senders seeing this is not very relevant as this is always visible - what is interesting is what happens after they see it
   2. Senders: % of senders logging back to see message status even without response
   3. Senders : Chain length ( Average number of messages sent by one party before other responds)
   4. Receivers : % of conversations immediately started by receivers 
   5. Receivers: Average time between visit screen and next message
   6. Businesses : Response rate of business queries ( Eg % of messages responded in 1 Hour)
2. Quality of conversations: The hypothesis is that this should improve so we should look at
   1. Average number of messages / conversation ( it can also be median ) : Conversation = Fixed time frame
   2. Average Message distribution in a 2 party conversation ( Messages by A / Total Messages)
   3. Average Time to respond between messages in an conversation OR % messages responded in X hours
3. Retention
   1. % of users reactivated due to this feature compared to base
4. Overall Impact on social value
   1. Impact of feature on DAU for Messenger ( attributed)
   2. Total messages exchanged in a 1:1 conversation ( as a proxy for strong social connections)

• A/B with control group at launch
• Blackout feature to group and test / Hold-out
• Pre-Post feature analysis

1. More messages exchanged May not necessarily lead to stronger social connections - this belief needs to be tested with data
2. Its a high level metric and changes in it over time may not be directly attributable to the feature
3. There is a risk that the metric may go up due to more one-sided conversations and thus we need to check message distribution also

The core purpose of this feature is to help improve engagement in Facebook messenger.

Read Receipts are used to foster the social obligation of responding to a message which you , the recipient have read and responding back to the message sender.

If we were to segment the messenger customer base by usage - Active(use at least once a day) and Passive users(use once a week) , this feature will help improve engagement for passive users and will help improve the engagement for that user group. I dont' see this improving the engagement metrics significantly for active users who are already using the messenger application actively.

So , the metrics I will be looking at are the following -

Feature Success

Definition of Conversation- A session where interaction happens between two users or a group of users with a maximum delay of 12 hours.

a. Number of messages per conversation (Applicable to both 1:1 as well as Group Conversations)
b. Increase in number of conversations(Applicable to both 1:1 as well as Group Conversations)

c. Time spent in the messenger per user

d. Conversation Duration (Time per conversation)

e. Number of view receipts read for a message/user

Amongst the above metrics, I would prioritize c (ultimate goal of this feature),c,b,d and e.

The way to test this hypothesis will be to launch this feature to a test group and a control group with a similar mix of Active to Passive User base and measure the above metrics and see if we see a statistical significant improvement in engagement metrics mentioned above.

One important thing is also to measure if this feature is actually leading to decrease in engagement of a segment of the user base.