You are the PM of Instagram stories. What goal would you set and how would you measure success?

For this question, we have some clarifying questions. 

1. Firstly, is there any particular objective we are given in terms of which we should improve Instagram, stories or in terms of which we should measure its success. For example, it can be revenue from advertisers.

 

2. Is there a particular customer segment that we should focus on? Customer segmentation on the basis of how they are consuming stories

- content consumers / Creators/Advertisers

- Less frequent user to a more frequent User

 

 If I am free to choose. Among all the things, my Approach would be the following.

 

Framework

First we will Define what Instagram stories is. What is the definition of this feature? And what what use value it is given giving to the users. Then we will look at What is the company goal? Judging from the stage of Growth At which the company is, we will choose a goal. We will also zero in on a segment, a customer segment. For the given customer segment, we will chart the user journey and we will look at the behaviors which we want to encourage. On the basis of that, we will choose a few metrics. We will choose a North Star, not star metric, which will be our primary Metric, which we will refer again. And again, then we will choose a series of supporting metrics, which we should also look at. In the end, we will share some feedback and trade-offs regarding various metrics that we have chosen. And then, in the end summarize. 

 

Defining the feature 

 Instagram stories is a feature in Instagram, where the user can make a video or a photo, which will remain on their profile for 24 hours. After that, it will disappear. Its length is not more than 30 seconds. And people who are connected or who are followers of this particular user, they can engage In the stories via. Reactions comments,shares, replies to polls etc.

Segmentation. 

We will look at content consumers. Why? Because content consumers or the people who are watching the stories, are the primary group of people who are actually engaging with the with the creators. And these are the people who are actually getting the main value in the value chain of stories. So if things are not optimized for this group, then the creators were creating the stories will also not get the required motivation to make more stories and again the whole ecosystem might get disrupted. So first priority would be to keep the content consumers happy. 

-Looking at stories once a week

- Moderate users, who are looking at stories, two to three times a week

- Power users who are looking at and engaging with stories every day consistently. 

 We will choose the consumer segment, content consumers, who are looking at stories once a week because if we work on this segment and try to increase engagement from this user set, then obviously our medium usage segment will also benefit from that and we will try to Look at some ways through which these both these segments can come into heavy users. Um, in terms of the value that they get, so we will ensure that they get good value out of stories. And in this way, we will try to maximize engagement from stories. 

 

Growth Stage of instagram

 Instagram is somewhat into a matured phase. And the goal that I can think of for Instagram stories is Ranging from engagement to revenue generation. So we will also look at The stories. Which, Are posted by advertisers and if we can maximize the value which is being provided to the content consumers. It will also ensure that the revenue which is generated for the advertisers and in turn the revenue, which is generated for Instagram is also maximized.

 

User Journey

How the users were using Instagram stories? How do they get value? So, first of all, they would get updated regarding their friends. Life updates. The options which are available to them, are to share a reaction. Or they can go one step ahead and share a message with them or if it's a post, which they can share to other users, in their PMS, in their Private messages so they can do that also. So in this way, the user is getting value and it is not just only by making them see the stories it is when they really engage or Express themselves, that the overall mission of Instagram is met and that is how they will get the value. Um, Which is the intended value of the stories.

Action that we want to encourage among the users who are using Instagram stories, very less frequently. 

Number of reactions, number of replies sent two different stories. Number of PMS sent. And number of polls Answered. 

Metrics 

 acquisition and activation 

1. number of signups on Instagram

2. number of users, who are watching stories at least one story in a week. 

So this we can look at for Activation Then the Real category of metrics, which we want to look at is engagement for Consumer.

Engagement 

1. Avg number of stories or number of accounts stories watched by user.

2. timespent by each user on stories. We can also compare it to news feed or reels.

3. DAU, WAU for story segment.

4.  number of stories that the users are responding to in terms of reactions.

-reactions

-Private messages

- shares

-polls.

 This should be a North Star metric. Because this is the one which signifies the value is being provided or whether the user is wanting to do reactions or not.

 We can also track it for advertisers where If the call to action for that story ad is a sign up or a registration. We often see that as the consumer swipes up, it is getting taken to another page maybe on Instagram, maybe outside. So if wherever we can track, we should also track how many number of purchases or number of signups? Whatever is the call to action? How many users are actually responding to that.

 Revenue

1. number of purchases, which are made from story ads

2. Revenue made from story ads.

3. Customer lifetime value.

 So our North Star metric would be number of stories that each user (We can average it out for across users) is responding to in terms of reactions, PMS, shares and polls answered. And then we will support it with the following supporting metrics 

1. timeSpent on stories per user

2.  average number of stories, watched per user

3. daily active users or weekly active users. 

4.  Increase in the number of stories that the users responded to week on week or month on month. So that we can track whether whatever steps we are taking for its success whether it is actually bringing in the result or not for a particular period of time.

5. revenue from story ads

6. customer lifetime value. 

 

Summary

 What goal we should set and how we should measure success? So, the goal that we should set is the goal that we have set is For the consumers who are not consuming stories, a lot who are looking at stories, maybe once a week or so. What all can we do so that the number of stories that these users are responding to in terms of reactions PM shares and polls answered. How they are increasing week on week or month on month so we can set a particular Target. Say we want to see a 15% increase on an average per month in the number of stories that the users are responding to So this is our North Star metric number of stories that each user is responding to supported by various metrics, which are tracking the acquisition and activation bit. Which are tracking the engagement bit and some metrics, which are tracking how much revenue Instagram is making from these stories.

My approach for this would be, I will try to understand the objective of Instagram and also assume what Instagram intends to get with the stories feature. 

Instagram's Objective: Instagram's mission is to connect people through everyday moments, and its tagline is "Capture and Share the World's Moments"

The objective for the stories feature: 
My assumption is this feature would have stemmed from the idea to improve product adoption, revisit rate, no of people posting and engaging through posts. These objectives resonate with the mission statement and strengthen my assumption. Going ahead with the assumption here are a few problems it intends to solve: 

1. Privacy for posting

- Followers will have continuous access to the posts on the wall ( calling these wall posts for convenience)
- Users do not know how the post will be received
- Users are uncomfortable with people discussing their posts on a public platform.

2. Lack of FOMO 
- The posts stay on the wall forever, I am not missing out even if I do not open the gram for a day. 

Reducing the inhibition to post and incentivizing the users to open every day can be the major objectives of the feature. So based on this narrative, these are the metrics I would be tracking.

Adoption:
- Change in posting frequency for active and inactive users to evaluate if the feature is giving intended privacy to post less important things.
- No of accounts with no posts on the wall but post stories.
- % change in time taken to revisit Instagram.
- % change in average time spent on Instagram
- % of users with first action as story viewed.
- No of users who made their first wall post.
- No of inactive users who made their first post after posting stories.
- Average time spent on stories


I would track these as well but these will be my secondary metrics: 
- No of reactions to a story
- No of new conversations started through a story.
- % change in the average number of wall posts in a specific period. ( to see if the feature is cannibalizing wall posts). 

Coming to the second part of the question I would put my focus on groups 2 and 3 with a focus to upgrading them to a more active group and specifically on group 2 at the beginning as this feature has the potential to improve the average frequency of visiting Instagram and group 2 is already warmed up enough to be converted to group 1

Ok. So let's first make sure my understanding of what Instagram stories is is correct. Instagram stories is more like FB News feed but the difference in my mind here is that FB is much more oriented around friends posting things so when you scroll through you are mostly looking at friend's feeds but I feel that Instagram stories has more advertisments resulting from explicitly following certain creators. One can like the feed, comment on it, share it, or simply scroll through variuos feeds. Instagram makes money from CTR or CPI. Correct? (yes pretty much except I wouldn't say necessarily that Instagram feeds is more about digital advert than FB. If you have ton of friends on instagram it will be the same as FB).

Ok. I think you are correct; I din't think that way since I don't really have that many friends that I have added on instagram. Anyhow, so now that we are aligned on my understanding of this product, if we had to measure success and set a goal I would like to consider our primary goal that we are trying to achieve and for whom. I think the aduiance part is important since that may help us think through what that user would more likely do on Instagram so that we can then measure success based on user journey. Ok here? (yes)

ok, so do you have any business goal in mind that we are trying to achieve success on? (nothing specifically but you tell me what you think might be a good business goal).

Sure, considering Instagram is a mature business I don't think we need to worry about customer acqusition, adoption or conversion. The main thing to keep in mind is IG makes money on digital ads and that comes from two things, user interacting with IG and coming back and back and back on IG. This way user will see more ads and if user comes back again and again it also means we are adding value to user. All of this results in more business impact for FB. So we should measure engagement but secondary north star should be retention? (ok, but why engaegment and not retention as our primary?)

Well, engagement is critical to attain retention. So if we get engagement I thikn we would have added value to user which will lead to retention. (ok. Due to interest of time let's focus on engagement).

Great! So IG has creators and consumers of the content. Creators are mostly businesses but also consumers since those could be my friends creating material and posting like their pics, etc. Let's focus on consumers here since I do believe we have more consumers so if we can get them to engage more we would make a bigger business impact through CTR and CPI revenue.

A consumer would, as noted before, heart a post, comment on it, share it, or a person like me would simple scroll through the pictures and sometimes look at a single picture fo long. (yes these actions are comprehensive).

Great! Thanks for confirmation. So I think engagement really comes from above actions my customer can take. Considering that, if our north star is engagement, those actions are the ones that we should measure to determine success or not. (fair)

1) Avg. Amount of time spent per session/user (though this could be deceiving since mostly IG is used on cell phone and people generally don't end the session. Additionally, average could aslo be quite deceiving since we are including outlier data points so maybe we eliminate those and then perform a box plot or something or measure median score).

2) % of users spending atleast 10 min scrolling per session (again session could be deceiving and not sure if there is a way we can actually get this metric measured correctly. I say 10 min here since in my experience it's so easy for IG to consume 10 min of your time easily so I am more interested in this number ticking up).

3) % of users sharing a post (this is important since if a post was shared by a friend of mine I have extremely high likelyhood of atleast openin that post and naturally IG is so attractive and addictive that I will at least look at couple posts).

4) % of posts that send out a notification to friends of the poster (this one has a downside to it since if i have a lot of friends and each even posts 1 and i am to get notification for all of them I am going to get tired of these notifications and turn them off. That's a bad taste in my mouth and the purpose of notification which is to incentivize a user to open IG is killed). So technically I wouldn't want to possibly see this number rather steady. In fact, I thikn it be important to measure a point at which (specific #) there is a reduction in user opening notifications. That's how we should set a threshold for this one.

All in all, I think #2 is the most important here since if I can get a user to spend at least 10 min I am rest assured user has seen atleast few ads and that confirms 2 things: 1) user finds value and was entertained 2)there is a massive business impact for IG.

In terms of setting a goal, I would say that if data were available, I would want to measure inflection point that results an engagement into retention of a user. Due to lack of that data, for now, let's assume that atleast 10 min of being on IG is that inflection point that gets my user hooked and has him/her come back again and again on IG. I would measure this over a week for the next month to ensure I have succeeded in achievenig engagement. IG users are mostly younger folks and they tend to use it daily + 1 month will give me enough data to measure trend which i need to know if I am successful.

Does the above make sense?

(yes).

Great! So overall, I think for a product like IG that is mature, we should measure success of engagement by measuring % of users who spend atleast 10 min on IG per session.

Assumptions:

1. I am a PM at this point of time, not at launch
2. Question is  not exactly clear  regarding  user groups,  so will assume  that group  1 is  IG daily active users, group 2 = 50% likelyhood of story post every 15  days and group size = 1/3 of group 1,  group 3 = 20% likelihood  of story post every 30 days  group size  = 1/3 group  1.

1. time to activation (post a story)  for daily/weekly cohorts
2. 7 day and 30 day retention for stories
3. % of  users who view stories daily(assume strong story viewing behavior leads  to  story creation)
4. Stories DAU, MAU and WAU
5. Avg # of stories posted per week & per cohort per week 

Question: What goal would you set for creation of instagram stories and how would you measue success?

Define the feature: user captures a moment, enhances recording with graphics/sound, posts or shares the story

Mission: provide an activity users finds enjoyable that creates content for other users to consume leading to engaged users   

Users:

- new users who have just created one story

- users who come back to create another story

Metrics:

- # of stories created

- # of likes / comments

- recurrance of behaviour indicating users finding creation of stories a worthwhile experience: growth in the average # of stories / user; increase in the number value of mode of (# of stories/user), so for example, initially it is 3 because that is the most common number of stories created by users but if we are successful then through engagment this number should increase

- # of stories shared

- increase in the # of DAU/WAU/MAU

- growth in the number of users who create a high number of stories, the high number can be determine by picking a number close to the maximum number of stories created / user  

Evaluate / prioritize/Recommend: success should be determined based on engaged users who find value in creating stories. This can be measured by looking at increase in number of users who come back to create more stories because they found creating stories rewarding.

I didn't quite understand what the meaning of showing up every day means in regards to this execution question.

Does it mean creating stories/viewing stories etc?

I will attempt this solution without the above constraints.

The framework is to understand the original goal (north star for the product). Depending on where the product is in its lifecycle select a goal. Segment the customers and choose one segment. Understand their user journey and then see the component that contributes to the selected goal. Define the associated metric.

Problem: Measure success of Instagram Stories

Instagram Stories, based on Snap Chat stories, allows you to  create interesting, fun short video consisting of your photos and/or videos . They are about 30s in length. It stays in a followers feed for 24 hours and then disappears.

Goal:  FM - Aligns well with FB mission of bringing people closer to each other using technology

    Product goal: Improves time spent on Instagram, thereby improving engagement with various features of IG.

Is there any red flag that I should know about?

No.

The goal of creating IG-Stories was to lower the barrier to entry to create user content on Instagram thereby increasing engagement and monetization opportunities.

 

Lets consider the segments - Creators and Consumers

Creators are the main engine for Instagram Stories, as they create the content that spawns activity from the product.

Creators can be segmented into - Common folk, Brands, SMBs, Corporations, Non Profits and Influencers

The common thread for each of these Creator types is the success of the stories is proportional to the number of people who view the story fully and the engage with IG because of that. Further in the case of Brands, SMGs, Influencers, success could further be defined as user behavior that causes a purchase of a service or product on IG. As IG is pretty mature the goal of the product is finally the revenue it generates for FB. 

The user journey for Creator

Create Story

Publish Story

Story is seen by followers

Receive Reactions from followers {responses and reactions}

Track conversions - commerce

Track who has seen stories.

This could lead to many metrics  - 

-  Segment into activity of creators as  HAI, MAI, LAI . Activity is number of stories created in a week. Each group will have a set of Individuals, Brands, SMBs, etc.

 - #of stories/day created by each segment 

- Power of story -  Assign different weights to Number of sales conversions(50%),  number of Replies(20%),  Number of Shares(20%), Number of Reactions(10%)- Commerce having the highest weight and reactions the lowest. Power is a weighted average of the User behavior to a story.  Measure the growth in power of stories created.

 

Of these components the Create story and Receive reactions components contribute mainly toward engagement. 

 

The primary metric would be:

 --  Daily average of the power of stories. 

 The reason I would choose this as the primary metric is that this a measure of both engagment and revenue generation for FB. 

Secondary metrics:

 -I would also put some guardrails -    Average % of response = ave number of followers responding/average number of followers

                                                        % of users who finished watching the stories

                                                        % Growth in Creators/day -  #new creators for the day/ total number of creators for the day.