Instagram is launching video calling. How would you test and measure its success?

Question

Instagram is launching video calling. How would you test and measure its success?

Asked at Meta (Facebook) over a year ago

4.6k views

Asked at

4.6k views

How to answer Metrics questions

Interview Guide

Top Meta (Facebook) interview questions

How would you design a bicycle renting app for tourists?62 answers | 82.5k views
Build a product to buy and sell antiques.54 answers | 66.8k views
A metric for a video streaming service dropped by 80%. What do you do?50 answers | 135k views
See Meta (Facebook) PM Interview Questions

Top Metrics interview questions

What metrics would you use to measure the success of the Save feature at Facebook? 39 answers | 87.2k views
How would you measure the success of Gmail?23 answers | 17k views
What metrics would you look at as a product manager for Instagram ads?19 answers | 23.8k views
See Metrics PM Interview Questions

Invite members
Invite by email

Add another

Shruti_KK · Answer 1 · 2021-06-02T12:49:41+0000

Assumption: We want to measure success at launch

I'll start by first laying out FB's overall mission/goals for context:

FB is focused on bringing people together and creating community.

There are two tangible outcomes that I can distill from there: creating connections and fostering meaningful interactions between them.

Knowing where FB is in its lifecycle, I would assume FB is currently focused on Engagement/Retention and Revenue

With that context, I'll get into the Video Call feature and the first frame the problem it solves for different users groups involved:

Today, a user that is on Instagram needs to exist the app and go to FB, WhatsApp, Facetime or any other video calling app that is available on their phone. This add friction in the user journey, that becomes even more pronounced in the context of someone wanted to call a friend after looking at a post/story from them. From business perspective, this is a drop off point. By fixing this leak, FB can retain users on their platform longer and gain continued engagement. So the overarching goal of this feature would be Engagement.

User Groups:

Person that is calling

Person getting a call

The goal will be to create value across both of these user groups for the feature to be successful, lead to network effect and higher engagement overall.

If I were to capture that in a North Star Metric, it would likely be Total # of successful video calls placed over a period of time - D,W,M. I'll prioritize per Day for launch. This will ladder up to Instagram level engagement metric around Time Spent on Instagram / User / Day or even # Active Users.

Let's now identify categories of metrics to measure in line with user journey:

User Group 1) Person that is calling

Awareness: User becomes aware that they can now make video calls through Instagram

Activation/conversion: User goes through the conversion funnel to call a friend (triggers could be any awareness placements or in response to a post they saw from a friend)

Engagement: They love the feature as they didn't lose context of what they were doing on Instagram previously, spend more time on Instagram as result, engage more with the feature

Retention: They remain engaged with Instagram long term owing to this feature

Monetization: There is no direct revenue opportunity with this feature as of now. It's indirect through improved engagement.

Since we are talking at launch, I will prioritize Awareness, Activation and Engagement categories as my top 3 priorities. I will set up an A/B Test to cross compare results against Control group that does not have the feature to understand incrementality.

Let's look at the metrics now:

Awareness: % of users that become aware of video calling feature per Day in the Test Group (measured by # of impressions for any awareness placements or Call initiation CTA - top of funnel)

# Impressions of landing page per D,W,M

# users that become aware of video calling feature

Segment by demography, geography, user tenure (new existing), Activity Level throughout the journey.

Activation/conversion:

% users that are calling a friend (base conversion rate) per D,W,M

Funnel Metrics (by entry point - during a session, from a friend's post, directly through call function)

# Users that are calling a friend per D, W, M

DAU, WAU, MAU

Engagement:

# of video calls / user / D, W, M

Avg session length per user

Time spent on Instagram per user per D, W, M

% users that continue to use Instagram after a video call

Counter Metrics:

Impact on other FB family of apps - video calling from Messenger, Whatsapp (to understand cannibalization if any)

Quality of calls (% dropped calls per day)

Time spent on Instagram per user per D on non-video call related functions

User Group 2: Receiver

Awareness: User becomes aware that they can receive calls on Instagram now

Activation: User accepts calls on Instagram

Engagement: Users log into Instagram and spend time on Instagram as a result of taking a call from there

Retention: Users use video calling feature more and more, which improves there overall engament with Instagram over time

My priorities for this segment would be Activation and Engagement (in line with impact on NS metric and overall company goals)

Metrics:

Activation:

% of video calls that get answered per day

# users that accept IG video calls in a day

Engagement:

# IG logins / user after a video call

Impact on # posts per user per D, W, M after the introduction of this feature

% of video calls that get answered / user / D, W, M

% of first time receivers that initiate calls after their first call

Counter Metrics:

Same as prior

harry YOUMBI · Answer 2 · 2020-04-01T14:34:40+0000

Metrics related PM interview questions like this should use the following structure (or something similar):

Description of the Feature: IG Video call would allow private conversations between 2 or more users (callers and receivers).

The goal of the Feature: Let's assume that the goal of the feature is to increase retention on Instagram

User Map Journey :

Acknowledgements: via a popup message that announces the availability of the feature

# users who read the popup message ;
Avg time spent on reading the popup message ;
% users who read the popup message/people who launch the feature

Activation : let assume that activation means to launch the first video call to another user

#users who launch the first call to another user
#first call that answered by the receivers
Avg time spent on the first video call
growth of activated users WoW, MoM

Engagement : Let's assume there is an Engagement when receivers answer to video call requests

#video call request well answered by receivers
Avg time spent on video calls

Retention :

# WAU (Weekly Active Users) on video call;
Avg time spent on Instagram via video call;
Growth of WAU of the Video Call feature;
% of calls sent/received per user

Revenue: No revenue tracked because the goal of the feature is to increase retention

Referrals :

#invites to video call on IG
#call sent to non activated users of the feature

Prioritizing :

To summarize based on the goal of the feature which is to increase retention on Instagram, let's focus on the following metrics :

#users who launch the first call to another user
Avg time spent on Instagram via video call;
Growth of WAU of the Video Call feature;

rahul_g · Answer 3 · 2019-12-03T06:30:39+0000

Instagram is launching video calling. How would you test and measure the success of it?

By video calling are you referring to a feature that lets you call users individually/privately unlike live which anyone can opt into

Assuming you want to identify metrics for the success here WRT launch which means a focus on adoption

Context

Instagram is a photo/video sharing app - its mission is to capture the world's moment and gives you the tools to do that

We are launching so whats the goal of the launch - adoption vs engagement , Id say former(adoption focusseD) since it takes time to build habits

Video calling

Video calling allows you to video call your contacts , as a reciever youd want this to come from someone in your network so assuming its only for 2 way follow relationships

Is it opt-in by default or do you have to opt-in manually, assumings its former

Whats the goal here

Allow people to closely connect with people in their network in a private fashion and help build a community

users needs

Caller

Share a beautiful moment with reciever

Connect with receiver regarding something they shared

Generally catch up with reciever

good quality stream

Receiver

No spam, I dont want to be called by people I don't know

Good quality stream

Assumptions

Assuming this is only enabled if the user pair is following each other
Since this is a social feature assuming that there are enough users on it already and the flag for the feature was kept off
Can users opt-out? assuming they can since this is not messaging platform at its core

Goal at launch: drive adoption and trial (Short term)

Given that IG is not the default when it comes to video/calls/messaging ( though it's moving towards that) assume that this is opportunity driven use case rather a daily use case so retention may not be a meaningful metric

I would breakdown North star metrics into the following :

Product (engagement focussed): Mins spent on live calls (with a graph for the following) = Unique users x sessions/user x avg time/session

launch focussed(adoption focussed): % of eligible users(support version + live call enabled) using calls in a meaningful fashion (actually made a 2 way call)

Health metrics - these are metrics I'll use to understand the health of the product

P0

Discovery Funnel : What % of users discover the feature , what % of these users actually tried it

NPS score: Gives me a sense of the users are feeling when they use the product

% users opting out: This is critical in terms of understanding if the defaults we've chosen make sense or do we need to make changes because you dont want to annoy users

Spam rate: What % of calls are spam - ie rejected multiple times.

Avg call length: Gives you a sense of depth of engagment this is generating

Overall timespent metrics for IG: Are the users spending more time on IG because of this feature

P1 - retention focussed

Weekly rolling retention: users using this feature in a meaningful way(call actually connected) on a weekly basis

Drop off rate: users who tried for the first time(callers only) and dropped out the next week

Anjali Mulchandani · Answer 4 · 2019-12-03T04:49:51+0000

Product Desc- Instagram Video calling is an option on top within Instagram messaging, giving the opportunity to have a video conversation with friends over audio. It acts as a quick platform tp connect rather than going on alternative apps for initiating a video call.

FB Mission- To build community and bring ppl closer together and Instagram video calling helps ppl to connect and build community through having them have video calls and take the communication with a personal touch.

Goal- To increase engagement

Customer Journey:

Open conversation with a friend- access option to initiate video call -> it connects to a friend and once the friend accepts- you have a video call.

Hypothesis: Launching VC will increase engagement ( time spent per user on Instagram)

Test: 1. Number of ppl prefer VC rather than just text

2. Time spent per user on Insta increases post-VC launch

3. Number of users not opting in for VC

Platform Health Metrics:

#video calls per user per day-->Conversion
CTR of VC feature--->Awareness
time spent on video calls / total time spent on Instagram per user per day--Engagement
%increase in time spent on Instagram post video calling feature rollout
%increase in messaging conversations per user per day-Engagement
#missed video calls per user per day
Average time spent per video call per user per day--Engagement
Drop rate/ of video calls--Retention

North Star Metric: number 3 and 4 as it aligns with our goal for user engagement.

Counter Metrics:

Average time spent per video call per user per day-- if that is too high that could be because quality may be poor and user on opposite side is unable to hear and it may take longer to keep a call going.

Amit J · Answer 5 · 2022-02-27T06:46:24+0000

Clarifying question:

Nascent stage
USA/Canada
App/Web - App
Instagram has close to 1.3B active users
Multiple user can join or just single user - multiple

Instagram - social media platform, create and share content, follow fan, like/comment/react, instagram shopping.

What are Meta core strengths?

Technology

Social graph

Competition or similar services

Zoom - business/

TEAMS - Microsoft

Whatsapp - Video calling

Messenger - Video calling

Google meet - video calling

Mission of Instagram video: To help users to stay connected and give them power to share through the use of technology.

Funnel:

User Login to instagram app

Look at recent instagram stories

Interact with friends/Fans through like and comment section

Metrics:

Adoption:

Daily, weekly and monthly new users (once a month) using instagram video feature

Engagement

Avg time used per user on Instagram video

Avg time increased and decreased on overall instagram after this feature was launched - daily, weekly and Monthly

Retention:

Daily, weekly, monthly users who uses instagram feature more than once

Users who have not used video calling on intsagram in more than a month

Monetization:

Avg. revenue increase/decrease with instagram video features being live.

Performance:

Daily, weekly, monthly video calling issues reported by users

MVP or north star:

Adoption and Engagement:

How many active users are making the call and avg time they spent on the call.

Counter-metrics: Performance

Avg star rating provided by user on video calling feature on daily, weekly, and monthly basis

Aslan · Answer 6 · 2022-02-25T04:11:16+0000

Clarify:

2 different applications: mobile and web

We can use the insta web in order to see the posts etc . I am assuming that this video calling feature will be launched on mobile app. Ok

Assuming that this video calling can be done with group of people, not just 1:1.

Since this is the “communication” part of the app, I assume that this feature will be added to the messaging/messenger part.

Instagram: this is a social media platform where users can create the content and post their pictures and videos with the closed /open group according to their settings.

User journey:

User to open the app

User to open messenger

User to click the video button

User to call the person ( video is closed first, for the privacy)

User to communicate with the other(s)

These social media products are being fed by the engagement between the users. Because if there is no product selling, you are the product. Here the main idea to launch this video calling is that they want users to engage with the product and be the part of this ecosystem.

Goal: increase the retention rate and engagement of the app and at the end of the day I want to increase the revenue.

What can I measure here? What steps are important:

Brainstorming:

- Who is going to be my base? DAU, WAU and I want to see MAU to make them more DAU, WAU

- How can I know that my friend is using the app actively to call them?

- Any technical issues?

- Can I call all my friends on my list? Like influencers? Or just ppl that we follow reciprocally?

Metrics:

- How many active users are getting the call?

- How many calls are answered? ( because I may add a feature which can show that the person is active in the app currently. So that this can increase the answers to the calls)

- If there is missed call notification on the phone, what is the opening rate

- what is the time spent on the app after the call? ( this is important bcz I want users to be engaged with the app)

- how many ppl who made the video call clicked, saved, put the comments to the adds?

- How many people after the call purchased the products through Instagram? ( conversion rate)

- How many people blocked by the others after getting the video call? ( and their reports/ complaints)

MVP of metrics:

I would start with how many active users are getting the call and answer them. After the call, how long do they stay on the app. Then the other metrics can come next since my main target is to increase the retention rate

Gaurav P · Answer 7 · 2019-12-05T11:29:43+0000

Instagram is launching video calling. How would you test and measure the success of it?

Goal

Improved User experience (Engagement time)
Increase adoption / repeat usage (retention)
Assumption - Since, instagram is a free product (goal is not monetization). Hence, not focussing on it.

Engagement

# of video calls made per day
% of sessions w/ video calls made
Average time spent on video call

Adoption

Unique daily users making 1 video calls / DAU%
Avg. calls per day = Total calls per day / DAU

Retention

Frequency clustering - # of video calls made in a week

1. 0-1
2. 1-3
3. 3-5
4. 5-10
5. 10+

7. Weekly retention cohort - users who made call on 1st day, made call on 2nd, 3rd, 4th and so on to the next day.

North Metrics: 1, 3, 6

Counter Metrics - video calls per day can increase (for a while) as user want to try the product or due to patchy network and user making multiple video calls in patch.