Instagram is launching video calling. How would you test and measure the success of it?
0 votes
236 views
in Metrics by (12 points) | 236 views

4 Answers

+1 vote

Product Desc- Instagram Video calling is an option on top within Instagram messaging, giving the opportunity to have a video conversation with friends over audio. It acts as a quick platform tp connect rather than going on alternative apps for initiating a video call.

FB Mission- To build community and bring ppl closer together and Instagram video calling helps ppl to connect and build community through having them have video calls and take the communication with a personal touch.

Goal- To increase engagement

Customer Journey:

Open conversation with a friend- access option to initiate video call -> it connects to friend and once the friend accepts- you have a video call.

Platform Health Metrics:

  1. #video calls per user per day
  2. time spent on video calls / total time spent on Instagram per user per day
  3. %increase in time spent on Instagram post video calling feature rollout
  4. %increase in messaging conversations per user per day
  5. #missed video calls per user per day
  6. Average time spent per video call per user per day
  7. Drop rate/ abandon rate of video calls
North Star Metric: number 3 and 4 as it aligns with our goal for user engagement.
 
Counter Metrics:
  1. Average time spent per video call per user per day-- if that is too high that could be because quality may be poor and user on opposite side is unable to hear and it may take longer to keep a call going.
 

 

by (17 points)
0 votes

Instagram is launching video calling. How would you test and measure the success of it?

By video calling are you referring to a feature that lets you call users individually/privately unlike live which anyone can opt into 

Assuming you want to identify metrics for the success here WRT launch which means a focus on adoption

Context

Instagram is a photo/video sharing app - its mission is to capture the world's moment and gives you the tools to do that

We are launching so whats the goal of the launch - adoption vs engagement , Id say former(adoption focusseD) since it takes time to build habits 

Video calling 

Video calling allows you to video call your contacts , as a reciever youd want this to come from someone in your network so assuming its only for 2 way follow relationships

Is it opt-in by default or do you have to opt-in manually, assumings its former

Whats the goal here

Allow people to closely connect with people in their network in a private fashion and help build a community 

users needs

Caller

Share a beautiful moment with reciever 

Connect with receiver regarding something they shared

Generally catch up with reciever

good quality stream

Receiver 

No spam, I dont want to be called by people I don't know 

Good quality stream

Assumptions

Assuming this is only enabled if the user pair is following each other 
Since this is a social feature assuming that there are enough users on it already and the flag for the feature was kept off
Can users opt-out? assuming they can since this is not messaging platform at its core

Goal at launch: drive adoption and trial (Short term)

Given that IG is not the default when it comes to video/calls/messaging ( though it's moving towards that) assume that this is opportunity driven use case rather a daily use case so retention may not be a meaningful metric 

I would breakdown North star metrics into the following : 

Product (engagement focussed): Mins spent on live calls (with a graph for the following) = Unique users x sessions/user x avg time/session

launch focussed(adoption focussed):  % of eligible users(support version + live call enabled) using calls in a meaningful fashion (actually made a 2 way call)

 

Health metrics - these are metrics I'll use to understand the health of the product

P0

Discovery Funnel : What % of users discover the feature , what % of these users actually tried it

NPS score: Gives me a sense of the users are feeling when they use the product

% users opting out:  This is critical in terms of understanding if the defaults we've chosen make sense or do we need to make changes because you dont want to annoy users

Spam rate: What % of calls are spam - ie rejected multiple times. 

Avg call length: Gives you a sense of depth of engagment this is generating

Overall timespent metrics for IG: Are the users spending more time on IG because of this feature

P1 - retention focussed

Weekly rolling retention:  users using this feature in a meaningful way(call actually connected) on a weekly basis

Drop off rate: users who tried for the first time(callers only) and dropped out the next week 

 

by (135 points)
0 votes

Instagram is launching video calling. How would you test and measure the success of it?

 

Goal 

  • Improved User experience (Engagement time)

  • Increase adoption / repeat usage (retention) 

  • Assumption - Since, instagram is a free product (goal is not monetization). Hence, not focussing on it. 


 

Engagement

  1. # of video calls made per day

  2. % of sessions w/ video calls made

  3. Average time spent on video call

 

Adoption

  1. Unique daily users making 1 video calls / DAU%

  2. Avg. calls per day = Total calls per day / DAU

 

Retention

  1. Frequency clustering - # of video calls made in a week

    1. 0-1

    2. 1-3

    3. 3-5

    4. 5-10

    5. 10+

7. Weekly retention cohort - users who made call on 1st day, made call on 2nd, 3rd, 4th and so on to the next day.

 

North Metrics: 1, 3, 6

 

Counter Metrics - video calls per day can increase (for a while) as user want to try the product or due to patchy network and user making multiple video calls in patch. 



 

ago by (57 points)
0 votes

There are few questions that I have before i dive deep into this?  

Can I describe the video calling feature as below:

Video Calling Feature dscription: Ability to make a video call to your contacts by simple tap of a button - (Sounds right)

1. Why is the video calling feature being launched? - is it to increase engagement on the platform? - (Yes)

2. Is the video calling feature being launched around the world or are we testing it in certain location? - (Across the world) 

3. Is it being launched for all the platforms? Web as well as mobile & for all OS? - (All)

Vision: Vision of instagram is to give users an ability to create professional looking photos & share it easily.

GOAL: The objective is to increase engagement on the instagram platform and hence the revenue

Let's dive into the metrics we would want to track & success criteria for each of these metrics. Initally we would target high level metrics. I would divide the metrics in the following categories:

Discoverability (Growth & acquisition)

- Feature Penetration (%)  i.e. users who have used this feature atleast once post launch  Success criteria - Target % of Instagram's base. WhatsApp's video calling data can be used as a proxy. Highest target value will be % penetration of what WhatsApp has achieved.

New users added per month i.e. first time users of video calling feature each month. Success criteria - Positive Growth compared to previous month

Engagement

- Active Users of Video call (DAU, WAU, MAU), Success criteria:  Positive Growth compared to previous period

- Total calls per Active user -  Success criteria:  Positive Growth compared to previous period

- Duration of calls per Active user - Success criteria:  Positive Growth compared to previous period

Happiness

- Quality of Service - QOS - Success Criteria - Should be excellent  (Industry defined standard for VOIP)

- Calls drop % - Success criteria - Less than X% (Industry defined standard for VOIP)

- Average Rating for the version on which the feature was launched- Success Criteria - >4

Returning users

-% Returning users for monthSuccess criteria: Positive growth compared to previous month

Revenue 

- Ads revenue per session (in which video call occurred) - Success criteria: Should be equal or more to the average Ads revenue per session

ago by
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
To avoid this verification in future, please log in or register.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
To avoid this verification in future, please log in or register.