A metric for a video streaming service dropped by 80%. What do you do?

Some clarifying question:

1. Did the drop in the metric occur gradually over time, or was it a sudden decline? - sudden
2. Has the decrease in streaming activity been observed on both web and mobile platforms, or is it specific to one?- Both
3. Is the decline in streaming metrics specific to any particular country? -no
4. Have there been any recent feature launches or updates that could be related to this drop? - yes

Depending on the two answers above, I’d list out the steps

1. Check the New Feature Implementation:
   • Verify if the recently launched feature is functioning correctly across all devices.
   • Ensure that there are no glitches or performance issues associated with the new feature.
2. Assess the Impact of the New Feature:
   • Analyze if the new feature has inadvertently affected other functionalities.
   • Check if it has increased the turnaround time (TAT) or caused any delays in the streaming experience.
3. Evaluate Content Availability:
   • Determine if there are any popular series or movies currently trending that are not available on your platform.
   • Consider if the absence of such content might be driving users to competitors.
4. Analyze Competitor Activity:
   • Review any recent changes in competitor pricing or new entrants in the market.
   • Evaluate if competitors have launched new features or promotions that might be attracting your users.
5. Investigate if the drop coincides with a festival or holiday period when users might be spending more time with family or engaging in outdoor activities rather than streaming content.
6. Check if there has been an increase in negative feedback or complaints related to the new feature or overall user experience.

To solve the problem , we need to 1st ask interviewer what type of video streaming service we are discussing here. Do they have any suggestion? Example like Youtube , facebook or Netflix/prime.

Suppose , they answered Youtube. Here we have to define and agree on what are the different types of metric are important for Youtube. For youtube the important metrics are , Viewership , Number of users , Time spent on videos , Number of videos uploaded , Number of videos shared, Increase in Subscription , Increase in positive feedback on videos.

If we agree on above , here I would like to pick up Viewership. As this metric can impact rest of the above metrics.

Now to understand deeper , I would like to understand from when 80% drop in viewership started happening. Is it happening on all types of contents on any specific content? Is it happening accross all locations or from any particular location? Since how long this decrease is happening?

Lets assume data analysis found that , on a particular type of content the drop is happening and its majorly happening from 3 geographical regions since last 2 days.

As next step , I would like to take below actions:

- Filter out the contents which is having drop in viewership

-Inform the content owner not to upload such content

-Put a global restriction to stop such content to render

-Enabling a content scanner to ensure no violations occured post publish

As a measurement , I would like to run A/B test and  enable the changes in only 1 geographical location and observe the uptake, if it shows a positive trend , I would like to enable it in rest of the 2 locations.

If all locations show a positive trend ,  I would like to rollout the changes and as a lesson learnt would like like to enable the restriction feature in other locations as well.

Clarifying questions

1. What kind of metric that dropped 80%? In some cases drop by 80% is good wherease in other scenarios it would be detrimental to the business. The next course of action depends on what kind of metric is it.
2. Was the drop an abrupt even or gradual (over a period of time)?
3. Does the drop pertains to any specific geographic region or market or technology (Android, iPhone, Desktop, etc)?
4. Any recent changes or software updates pushed to the production (which could have contributed to this drop)?
5. Any global security event that correlates to this drop? Any ZERO day event? DDoS attack, BOT attack, etc.
6. Any recovery noticed from the drop at all?
    

Before getting into details, I would like to establish certain points for shaping my approach in identifying the root cause of such a sudden drop.

• Which metric has met with such a huge drop? (is it acquisition, engagement or retention?)

• Since when are we facing this drop?

• Is this drop limited to a platform or observed on web, android and iOS?

• Which geographical area has seen this drop?

 

I am assuming that this acquisition metric (successful sign-ups) has seen a drastic drop of 80% from the past 1 week across all the platforms of ABC Streaming for the Indian vertical.

 

To perform root cause analysis, I would break my investigation in two areas, external factors and internal factors. I will start with external factors first and would probe into following areas:

• Is there bad PR for our product?

• Have we recently released any controversial content on the platform?

• Are there any political or regulatory changes?

• Are there any new entrants in the market?

 

If anyone or all or a few of the above external factors are said to be TRUE, then we will work with the marketing, legal & business teams to ensure that we are aligned to the external factors and will be a longer solution to the RC.

 

But if all of these factors are said to be FALSE, then I will investigate following internal factors:

• Are there any specific OS versions facing this issue?

• Has there been a recent release?

• Have we observed a sudden crash in the server?

• Is the data tool working as expected?

• Is there a change in subscription plans?

• Is there a change in permitted devices on a subscription?

• Is the User Journey working as expected:

  • Is the user able to download the application?

  • Is the user able to start the sign-up process?

  • Is the user able to make payment for their subscription?

  • Is the user able to successfully finish the sign-up process?

 

If none of these internal factors are an issue, then I will sit with the tech and data team to further deep dive into the issue. And if any of these factors are responsible for the 80% drop in acquisition metric, then I will work with all the relevant internal stakeholders to prioritise the fix for the problem.

5 Why

Problem: key  metric has dropped by 80%. 

Q: What does your product actually do?Does it only have provide streaming services or other services as well? 

A: Only Streaming

Q: What is your key metric? Is it for user retention, paid users

A: User Retention

Q: What kind of user retention are you measuring? Is it Average viewing time per user has been decreased or DAU has been decreased?

A: Average viewing time per user

Product Understanding: We are doing the RCA of the video streaming services whose average viewing time per user has been declined.

Q: Does your BI tool is giving correct information?

A: Yes

Q: Does this metric calulation has been changed lately?

A: No

Q: Since when we are seeing this change?

A: last week

Q: Does this change is due to some particular location?

A: No

Q: Does this change is due to some particular segmentation?

A: No

Q: Does this change is some device specific?

A: Yes, Android

Internal Factors

Q: Did you had any major launch lately?

A: Yes

Q: What kind of change did you have?

A: We have changed user experience for both android and Ios

Q: Is it team has missed including the changes for the android and it is break the user experience for the android user and eventually Average Viewing time is getting reduced.

A: Yes

Summarize: We looked over here why Average viewing time per user has been increased by 80% because we did some launch last week where team has missed including those changes for android user and hence decrease in average viewing time

 

Repeat the question to confirm: I'm the PM of a streaming video service and I see that one key metric has dropped by 80%. What will I do? 

State the structure of my answer to follow: I'd like to ask a series of questions that will help me narrow-in on the root cause. From there, we can figure out the best solution to implement.

Questions:

SVOD, TVOD, AVOD, or hybrid streaming service?

What is the key metric that dropped? How is it defined and logged?

When did the drop start occurring? Was it a sudden or gradual drop? (Sounds sudden since I would've noticed a decline in a KPI over time.)

Internal factors:

Did we change the way we log this key metric or its definition?

Were any new tests or features released around the time when the drop started?

Did a server go down?

Did the drop occur in all regions? All devices? All operating systems? All user types?

Looking at the user journey, were any other steps in the funnel or related KPIs affected?

External factors:

Are there any macro events happening in the world that might be related to the root cause? (e.g. holidays, war, etc.)

New competitor launch?

Partner integration breakdown?

Next steps:

If the problem is internal and the solution can be fixed immediately, I would work with my tech lead to get the right resources in place to fix the problem. If it was process-related, I would put a workflow in place to prevent this issue from reoccurring in the future.

If the problem is external, more thought and strategy definition will be required, but since 80% is a significant drop, I would rally my cross-functional partners to find and build a solution with urgency.

Summary:

80% is a significant drop for any KPI. If I were the PM, I would investigate internal and external factors to identify the root cause of the problem, and then work with my tech lead and cross-functional team to build a solution with a sense of urgency.

You are the PM for a streaming video service. You come into the office and see that one key metric has dropped by 80%. What will you do?

 

Clarification questions - 

1/ To be on the same page, a ‘streaming video service’ like YT, NFLX, Prime Video?

2/ Key metric -> Do u have a metric or I can brainstorm and pick one

 

Structure-

1/ What key metric

2/ What does a 80% drop mean

3/ What I will do to dive deep into understanding the drop

 

Details

1/ What key metric

We know the service is a video streaming service. Few metrics important to me to track are - 

Input metrics (what capabilities in a product we have)

a/ content or # of videos

b/ # of categories

c/ # of langauges

 

Output metrics (what does this drive)

a/ # of users DAU, WAU

b/ # of new users daily, weekly

c/ # of videos viewed by user

d/ # of ads displayed

e/ Revenue

 

One metric I would pick is DAU -> this is because DAU drive the usage of the product and revenue. If we see this metric drop, then it means that we MIGHT have a user stickiness problem which can hurt our product, brand, and revenue.

 

Please let me know if you have any questions on this selection?

 

Ok, now that we have agreed on what is the metric we want to focus on, lets continue on the net item.

 

2/ What does a 80% drop mean?

This might be a rhetorical question but just to make sure we are aligned, this means that if we had 100 DAU, now we see 20 DAU? Is that so? 

Ok, now that we have cleared that out. Moving on…

 

3/ Diving deep into understanding what might be the reason?

For this, I want to split my investigation into internal and external factors. Let me explain…

Internal factors are - Anything that is related to the company and the business unit

 

External factors are - Anything that is outside of the company such as customers/users, competition, market, macro factors

 

Lets start with internal factors - 

1/ 1st I will see if we had a data issues or dashboard issues -> Did all the jobs run as planned or we have any metric recording or dashboarding issues.

 

2/ 2nd I want to focus on the product itself. Specifically I want to see if we did any changes to the product -> e.g, a login change, search change, catalog change, language support change? Or changes to the devices we support. Also I want to focus on tech issues -> service outage

 

3/ 3rd I want to see if we did a price change. If we did this, this wont be sudden and I would expect a drop if the price change was in the upwards direction (aka price increase)

 

4/ 4th I want to see if our product positioning changed. What this means is that, did we say that we will only stream a particular category of videos.[this should not be suprising] 

 

5/ 5th Did our company’s position change? This can be because of a statement by our CEO or VP or the likes.

 

6/ 6th Did our “Free trial” end?

 

External factors

1/ Customers/users - do we see a drop in a particular region? Or in a age grp? Or in a specific locale speaking users?

 

2/ Did our competition make a move?

 

3/ Were there any marco changes like war or sovernishgty issues or country govt banning our product

 

Of these my priority order are

1/ Data issues or dashboarding issues, seasonal drops?

2/ Product changes/tech issues in case there are broader teams working on it

3/ Product position changes incase marketing team has released something

4/Company position

5/Users -> who, where, what

6/Macro changes -> law changes, government changes, internet outages

 

 

KEEPING IT CRISP AND STRUCTURED SO THAT THE SAME APPROACH CAN BE USED IN ALL SUCH QUESTIONS

 

> Part 1: Clarifying questions to identify the problem statement

1. What is the streaming service? (Assume: Netflix)
2. What is the key metric? (Assume: Views/day)

 

> Part 2: Metric evaluation to understand the metric

1. What: What is the period of the drop? (Daily)
2. When: Is it one time? Is it seasonal? weekend effect? Any other abnormalities? (Assume: It is one time)
3. Where: Is it in a particular region? (Assume: US)
4. Who: Is it for a particular segment? (Particular kind of videos? / Device/ Age / etc) (Assume: No) 
5. Did we change the metric/measurement method/measurement broke? (Assume: No)

 

> Part 3: Internal factors to inspect the User journey

1. Did the product change/Any new launched/Any new flows? (Assume: No)
2. Look at the user journey (Assume: all metrics are down)
   1. App download # 
   2. App sign-ins # 
   3. Traffic on the service #
   4. App Searches #
   5. Video pay
   6. Video views #

> Part 4: External factors (5Cs)

1. Company: Did the company change its strategy? Ex: Discontinue mobile app (Assume: No)
2. Competitors: Did a competitor launch anything? Ex A free service ( Assume: No)
3. Collaborators: Did our partners change anything?
   1. Video providers (Assume: No)
   2. App stores (Assume: No)
   3. iOS/Android (Assume: No) 
   4. Internet services: YES (Internet service from one of the telecom provider is down)
4. Customer: Did customer habits change  (Assume: No)
5. Climate: Did the law/regulation change? Any natural calamity etc (Assume : No)

Clarifying questions:

- What metric has dropped, and since when is the drop observed?

- What is the source of the data? Verify if the data is correct by looking at other dashboards that power from the same source (if the source is an internal dashboard)

- What is the criticality of that metric for the business

- Try to understand the macro factors that could have led to this. Is this drop happening to users from a particular geography? Is there a trend that tells us which device or network is most affected? If the metric is related to payments, there might be a play of regulation change.

- Basis on the responses to the above questions, I will start going narrower in the macro parameters. Is there some campaign/news/agenda by a competitor being run in that geography?

- Are other internet products also affected by the exact demographics of users?

- If I cannot get a clear sight of the reason, I will start checking on the micro factors, like if a recent release targeted the same segment of users. Did we communicate any particular company communication to affected users before they got affected?

- Once I arrive at the right problem using this approach, I will plan to diagnose the situation the fastest way possible.

First, I would like to understand what key metric has declined because 80% seems too high for a KPI drop. Also, I would want to understand if it's a steep or a gradual decline, assuming it to be steep since a gradual decline wouldn't have gone unnoticed over the past few days/weeks.

Then, after narrowing down on what metric has been impacted, I would try to deep dive if the decline has been due to:
1. a particular category of videos
2. in a particular geography
3. any particular user segment
4. in a certain language
5. on any particular device/browser (laptops,mobiles,etc.)
6. in any OS type (ios/android)

If there wasn't anything anomaly found above, then I would try to identify if there were any external factors impacting the KPI:
1. any PR issues
2. new competitor entry
3. current competitors offering discounts/lower pricing
4. seasonality
5. change in government policies/laws
6. any DoS attacks

I will also try to chalk down the possible internal factors that could have impacted the KPI drop:
1. Change in pricing/value proposition
2. any identifiable bugs in recent light
3. Impacted system health- latency, avg load time, app crashes, network errors, storage issues

In the end, I will try to identify the stakeholders and detail all the touch points for each stakeholder-
Stakeholders:
1. Users(Supply/Demand)- Creators/Publishers, Viewers
2. Advertising partners
Touchpoints:
UI changes, addition/deletion of any CTAs and or ads on every screen that might lead to less browsing/less view time, etc.

The answer will also depend on identifying what streaming service is this to understand the value proposition and offerings and gain more perspective on the problem.
For example, if the KPI in question was view time, then for a streaming service like Youtube & Hotstar that also offer free content - the reason could be that due to UI changes paid content is shown more on the Homepages and reccomendation than the free content and hence the average view time of a free subscriber could have gone down. This will not be true in case of the likes of Netflix.

After identifying the issue, I would try to work with the stakeholders involved- Business, Dev, DevOps, etc. to fix the issue and put checks in place to avoid such a menace in future.

80% drop in a metric is huge. I would like to understand what metric dropped so I can understand this better.

For a video streaming service the customer segments are content creators and viewers.

Some of the key metrics that are gathered for viewers are new user sign ups, activation (user has watched atleast for 30 mins weekly) engagement ( user logs in atleast once per week), retention (existing users have renewed the subscription) and referral (users refer friends and family)

We can assume any of the metrics listed above dropped to be investigated.

First step is to understand if it is a gradual decrease or sudden drop. Let's investigate both.

for a gradual decrease, there could be the following reasons:

1. External factors - competition which is taking away our customers

1. Price

2. Better content

3. Better quality

4, promotions

2. Internal factors

1. Price increase

2: decrease in quality

If this is the case where a new entrant has entered into the market or competitors are changing the economies of the market, we need to look at our marketing strategies and invest in R&D to meet customer needs so we can keep both happy.

 

If it is a sudden drop, we should look at the following factors:

1. External factors:

1. Seasonality- is there a holiday season like a festival and people spending time together

2. Is it limited to geographic area . If yea, then was there any power outage or internet service outage which impacted access to the streaming service?

 

If no, we continue to drill down.

3. Events: any major event like Super Bowl or oscars being steamed which is taking the attention away from the streaming service?

 

Internal factors:

1. There was an outage in the service

2. There were UX changes and was hard for people to login

3. Another genre of content was being streamed. Maybe even short videos.

These are some of the areas where we could look and if we find the root cause, then we work on resolving it and tracking the metric again

Note: Me – Signifies the interviewee, Int – Interviewer.

I would like to understand more about the product and metrics which is in question.

Me: 80% is a big drop. Wanted to understand more about the product.

Int – It’s the Facebook reels.

Me: Thanks. Facebook reels have two main stakeholders, creators, and consumers. Do you know which metric we are talking about?

Int – It’s the consumers.

Me-Can you provide me with some context on what this metric measure, the engagement of consumers, the time spent per consumer on the reels, etc?

Int : The metrics measure the engagement of the consumers

Me: Great. The drop in metrics is it a sudden drop or gradual and when did we start seeing this drop?

Int – It’s a sudden drop, we started seeing it in the last two days.

Me: thanks for the information. Do we notice if it’s happening for both web and mobile apps and also have we identified if it’s specific to a certain device?

Int- It’s happening for the mobile app.

Me: Are we seeing the drop globally or specific to any region?

Int- specific to a region

Me- Issue related to any specific mobile phones, iOS, Android, etc.

Int- iOS

Me: Thanks, since it’s a combination of mobile phone and geography, I would like to break it down further. If it’s any infrastructure issue, it will be device agnostic, so I will not focus on this for now. Will come back later on this. Until and unless you confirm, that some specific platforms are catering to specific types of devices

Regarding the issue seen in iOS, have we looked into if it’s specific to the iOS-specific version or seen across all of them?

Int – Same infrastructure supports all devices. And seeing issue across all of them.

Me: Good information. Is there any s/w release we have rolled out?

Int – Yes, there was a s/w release which is rolling out gradually.

Me: Assuming the release is going into different phases, have we identified if we are seeing the drop in the areas where this release has been rolled out?

Int – Yes, it seems to be the case.

Me- Since we are seeing the direct impact of the release to an extent to make a significant impact on the geographical metrics drop, can we first pause the rollout of this release to further areas until we investigate the issue?

Int- Sure we can do that.

Me: As part of the release, can you please walk me through the release changes, specifically the changes made toward the reel?

Int- Sure, we made a few changes, introduced the count for consumers on how many content creators they follow, changed the UI and positioning of the reels and introduced a way for consumers to prioritize their following.

Me: Thanks. Now that I know about the issue more, I would like to come up with the below hypothesis.

It might be that there is a bug introduced in the release which affects iOS, might be slower loading time or returning errors, or could be that with iOS people are not able to quickly figure out the new positioning.

Have the customers reported any issues on what might be the issue they are facing?

Int: No customers have not talked about it yet.

Me: Got it. Since we are not certain which specific area we might be hitting, I would like to do the below in nailing down specifics.

1)      For slower loading time- Can we run a sample smaller load test on iOS and compare the before and after release metrics for it? This will help us evaluate if there is any slowness caused by the new release, like loading time, and any specific errors the system is throwing when the consumer is attempting to watch the reels.

2)      Bug – Can we run customer journey-specific tests to figure out on which step we are seeing the issue, and what actions the customer is not able to take which is resulting in the drop?

3)      Positioning of iOS – Run a survey to reach out to iOS consumers, to figure out how this change has impacted their day-to-day watching of reels, whether is it easy to navigate, and whether are you having issues finding it.

 

Also, since it’s such a significant impact on metrics, I would suggest a rollback of the changes until we identify the next steps in solving the issue. 

Clarification : 

• Which straming video service ? Assuming it is for Netflix 
• Netfilx is in the business of long form content viewing. 
• Which  Key Metrics ? Assuming Videos watch time has reduced by 80%  .
• Understand has this 80% decrease is from previous day or previous week or previous year . Assuming this 80% decrease is from previous day.
• Which platform has this decreased assuning this happened on app.

• Internal Factors :
  • Check the data source : Data source from which we are receiving data , any delay in data updation. 
  • Technical release : Any release which happened tht has led to this decrease. any bugs that has been reported by any users.
  • User joureny funnel : Key metrics : Watch time . User opens the app , see recommendationsof videos , click on videos , video gets played .Create data funnel to analyse where is the drop. is less users watching or less user clicking on recommendations or less users on app. 
  • Check for any geography : is the app blocked or is this decrease if from particular geography. 
  • Check for any particular category where you see the drop.
  • Check for any specific device type or platform types
• External Factors : 
  • Any govt decision about any ban on app for any location or any content types.
  • Any Marketing camapign launched by competitors 
  • Any marketing campaign launched by Netfix which received backlash on social media. 
  • Any major events which is happening , any live stream of match or any other event.
• Consumer Behavior : I dont think 80% drop will happend with any consumer behavior.

Clarifying Questions:

1. Do you have a specific streaming service in mind?

   1. Lets assume Youtube

   2. Youtube has multiple products - Youtube, Youtube Premium, Youtube TV and Youtube Kids

2. Do you have a metric in mind?

   1. Do you want me to focus on any specific step in the customer funnel when thinking about the time spent drop? - Acquisition, Conversion, Engagement, Adoption Retention etc.. 

   2. Lets assume Time Spent has dropped

3. Has the time spent dropped across the geography?

   1. Yes

4. Is the time spent dropped on a specific platform?

   1. Mobile, Web, Both

5. Did we release any new features for Youtube recently that would have left for this regression?

   1. We do continuous release , so we might have

6. Has the time spent dropped suddenly or has it been gradual?

   1. Sudden drop

 

Game Plan:

1. Identify the source of the drop

2. Identify possible options to respond

3. Prioritise

4. Execute the option

 

Identify the source of the drop The way I would investigate this in two parts:

1. First, is this caused by any actions done by the product team or the XFNs

Place > Price > Promotion > Product (platform, features, bugs, terms & condition of usage) > Partnership

1. Is this a reporting error? - a bug in the tracking pipeline etc.. 

   1. If this is the case, we will see inconsistencies across metrics. In case time spent we might see the session's timestamps are not aligned with the logging frequency or not matching with the aggregation of the individual video watched during the session.

2. Is this a product related issue or defect?

   1. Have we released a recent update? Where was that released? Can we isolate the issue to that release?

   2. Can we sweep through the customer journey funnel to understand the other metrics affected? Starting with the top of the funnel , can we check how the other engagement metric - #DAU, #WAU, #number of videos watched per session? Can we isolate the problem to a specific customer segment, region, surface, customer journey step, use-case?

3. Have we made any changes to the customer pricing? Youtube TV and premium. Have we stopped any ongoing promotion?

4. Is the drop happening in locations with similar characteristics?

5. Have we made any changes on the policy for using the youtube services? The policy changes could be we have released new terms and conditions for the product?

6. How is the content creator side of the business looking? Can we also check if there is a sudden/gradual drop in the content creator engagement metric? 

 

2. Second is this caused by external factors (e.g. competition, seasonality) that is not under our control.

 

1. Social/Market Sentiment: 

   1. How is the market sentiment about youtube and Alphabet? If there is anything from the Public Relationship team that can be correlated to drop in engagement.

   2. Could this be attributed to the increasing sentiment of people towards social platforms stealing people’s time and attention?

   3. Are there any cultural trends, festivals going on that is stopping people from using Youtube services?

2. Technology: Are there any wide scope internet, power outages in some part of the worlds that led to decrease in usage?

3. Is this seasonal - For example during summer, a lot of people will be spending time outside home, will go for vacation? Can we compare this with previous years?

4. Political: Have countries or a country blocked Youtube access or overarching Google access that has created this drop?

5. Economy related: Do people need to work more because of the ongoing inflation? Does this mean that they have less time to spend on the internet?

6. Legal: Are there any new laws in a country thats affecting the overall Youtube usage or blocking specific youtube content? For example - some heavily religious countries are blocking any video content with slight adult language/scene references

7. Competition: Is the competition affecting the drop? People have limited time -they can either spend on Youtube or Other streaming and social media sites? Do we have any reports from the market insights team on this?

 

Lets assume that the drop happened because of the lack of new content on the platform. We are seeing a drop in the new & interesting content created by the creators/influencers. The reason for the drop is they find it difficult to easily create content using the youtube content editing tools.

 

Identify possible options to respond:

 

1. Retain the current content creators

2. Make the life of the content creators easier by giving them the right tools

 

Let’s talk about the content creators:

1. Sophisticated & Professional: they have got their own content creation set-ups and make high definition content using professional HW and SW and do not rely on the youtube provided tools for editing

2. Medium range sophistication and everyday youtuber: Medium and small, less sophisticated content creators, making videos from the living room, rely on 3P apps and Youtube content editing tools

3. Novice, Occasional youtuber: They are not necessarily serious about their work and posting content once a while e.g. as a hobby

 

Retain the current content creators

 

 

Make the life of the content creators easier by giving them the right tools on the platform

 

 

My recommendation - Fund expenses of the 3P tools and Build - Build these tools from scratch over the period of time.

The first thing that comes to mind is that this is a very substantial decrease and it sounds like it happened abruptly so it's not a trend that they were tracking. It says key metric but is unclear about what metric they are speaking about so clarifying questions need to be asked. Is it a service outage? Is it a drop in subscription rates? Viewership during certain times of the day? Let's say it was a service outage-related issue coming from a certain region. In my experience, many service outages are caused by network issues. A normal response could be using a phone bridge to gather together with key IT individuals and go through some troubleshooting steps. I've found that the OSI model is a great tool to try to isolate the problem,  Based on the scenario my first question might be to look at how the traffic is being routed, is a device offline, could a port be blocked, did someone change a firewall rule recently and is blocking traffic and things like that. If it's subscriber-based then you need to gather the right people and try to find the root cause by performing an analysis. This might involve customer questionnaires or even gathering a focus group. It's even possible that some payment processing services experienced a glitch and canceled everyone's service. What time of the month is it? When do payment processing and validation of customer purchases happen? Is it an internal part of the company or an external vendor doing this work?

Q- A video streaming service, a key metric has dropped by 80%, Abruptly in a day.

Clarifying Questions - 
1- What is the type of streaming services - Free trials (Netflix, Prime), Freemium, Content tiers, Free 
2-In what all platforms the product is available? Is the drop for any particular platform - Mobile (Android + IOS), Desktop, TV, etc. 
3-In which all geographies our product is available? Is there any specific geography where we see the drop? - All over India 
4- What is the metric? Can we assume it to be a total number of new subscriptions?
5- Is the drop compared to yesterday's numbers or last month's avg daily new subscription? It could be possible that today is normal but yesterday was a spike.
3- Users who complete payments but opt out of subscription on the same day or later are counted in the new subscription. Yes 

After clarification - 
There could be 3 broad categories of reasons for this drop.
1- Internal to the company- Like app updates, glitches, app slowness, advertisements, changes in the available content, etc.   
2- External - Like competitors, public sentiments, calamities, events
3- Others - Its seasonal fluctuation, industry-wide observation, calculation error, the metric was changed, the metric is captured by a third party vendor and there is a bug in their system.

Let me first clarify if any of the points from the 3rd reason is applicable, if not, then I go looking for issues in 1 or 2.
 
Define the problem area. One of the ways is to check the drop rate at major steps in the user journey and then dig deeper into it.

1- Leads: Web + App (Paid and Unpaid)
It includes web traffic associated with organic users, Ads, social media, affiliates, new app downloads, etc.

Do we see any drop in the total number of leads?

2- Platform open rate (App + website) as % of 1
Do we see any drop in platform open rate?

3- User clicks on Buying option as % of 2 
Do we see any drop in the buy option-click rate?

4- User what goes to the payment page as % of 3
Do we see any drop in the payment initiation rate?

5- Users who complete subscription purchase as % of 4
Do we see any drop in the subscription completion rate?

Reasons
There must be a drop in one or more of the above numbers. Drop at a step could be due to these reasons 

A - Changes before this and after the preceding step.
-It can be UI changes
-Dynamic variables change like content, text, images, links, dropdown options, etc.
-Glitches, app slowness

B - Changes before the preceding steps 
1- Change in composition of users basis Gender, Geography, language, age, etc at step 1
2- Changes in the composition of users basis platforms where they arrive like [Android app, IOS app, or website] at step 1
3- Changes in the user composition basis device where they arrive like a mobile, desktop, or tablet at step 1
  
C- External factors 
1- Is the change due to external forces, like content, prices, or specific shows in other sources.
2- It could be due to the availability of complementary features such as coupons, cashback by affiliates, etc. 

D - If not, draw a detailed user journey from the previous step to this step, validate with the interviewer, and then find specific issues around them.





 

1. Point out that 80% is a significant drop. 

 

2. Ask: Which metric dropped?

• Engagement - Weekly watch time

• Monetization - Monthly paid subscriber, monthly advertisement revenue.

• Growth - New users, daily active users

Interviewer: Assume the metric is weekly watch time. 

 

3. Ask: Which user segments? 

• By geography - Asia, US, Europe

• By age groups - Teenagers, working adults, mature

• By platform - mobile, TV, desktop

Interviewer: Assume the users are teenagers. 

 

4. Ask: Is this due to seasonal reasons? 

Interviewer: No, this is not. 

 

5. Ask: Is this due to any of the following reasons? 

1. New competitors that took away teenagers’ time consumption. For example, TikTok has recently garnered a lot of attention. 

2. Internal causes

• Bugs.

• New features.

• Software performance issues such as slow loading time. 

3. Shows 

   • Canceled popular shows.

   • A popular show just ended. 

Interviewer: Let’s assume that a popular show just ended. 

 

6. There are 2 potential solutions.

   1. Try to push for a new episode for that particular show. 

   2. Recommend new shows that are similar and resonate well with teenagers. 

Interviewer: Let’s go with solution b

 

7. Let’s also set the metrics funnel to find out the show that will increase the watch time for teenagers. 

 

1. Reach - number of users who see the thumbnail for at least 30 seconds. 

2. Reach - number of users who watch the trailers. 

3. Engagement - number of users who watch 50% of the show. 

4. Engagement - number of users who watch 100% of the show.

Assumption: Since the product in consideration is a streaming service, I am assuming it to have a business model similar to that of Netflix. (Subscription Model). Moreover, the 80% drop seems to be a sudden drop

First I would deep-dive and identify the metric. I would classify the metrics into - 

Acquisition metrics:

• No of new users logging-in and opting for subscription
• No of users making successful payment for 1st time

Engagement metrics:

• CTR of a button
• Average Watch minutes per user
• Conversion from Home Page to Detail Page

• No of old users coming on the landing page
• No of successful renewals

• Was the previous day a national holiday?
• Has a competitor launched a new show?
• Were there internet issues in any particular part of the world where we are a major player?

 

 

Okay, Let me first understand the value proposition & core competencies of Netflix. As well as, elaborate on the user journey of NetFlix.

Netflix is the online streaming OTT service that provides content across all languages globally(the USA & other international countries). It runs on a tiered paid subscription service that allows users to access content on devices(mobile & website).

User journey of Netflix

User opens Netflix via App/Website> Signups to the Service> Pays for the subscription(Monthly, Quarterly, or yearly)> Logs into the service> Watches the content of his/her favorite> Scans for other interesting content & continue watching> Logs out of the service when done!

Let's get into the problem step-wise!!

Is this a temporary Blip or a Permanent blip?

It would be temp as it was noticed suddenly & had it been permanent there would have been a gradual drop in the KPIs that could have been noticed.

Is this drop pertain to the US market or Globally?

Assuming it to be global since Netflix operates in multiple geographies

Is it mobile-specific or website?

Assuming it to be both

What are the KPIs that dropped?

User retention, Subscriptions, Time spent by the user,  App downloads.

Assuming there is a sudden drop in the subscriptions across markets!!!

Let's first scan through some of the factors both internal & external to understand what actually went wrong!!

Internal factors:

New Product Launch

New Feature addition

Improvisation of exiting feature that causes a sudden drop

New bugs that went unaddressed

Server down at few markets or entire globally

External factors:

A new competitor entered the fray

Negative PR

Smartphone sales numbers need to be tracked

Content is provided for free of cost by competitors?

Netflix is available for free of cost via other platforms(Like torrents)

------

Are quarterly subscriptions, monthly, or yearly subscriptions going down?

Assuming it to be Monthly!!

What are the aforementioned factors caused it?

I hope there were no new launches or any internal factors that caused the drop in subscription by comparing it to all the stages of the user journey. If at all the problem stems from there, it is immediately taken back to operations, Engineering & design teams to get it rectified & provide hotfixes/feature roll backs.

Coming to the external factors, the problem seems to stem from wide availability of free Netflix content across various spurious channels like torrents or telegrams & free services provided by the competitors.

For the first problem, Have indepth discussions with the R & D teams, engineering teams & external consultancies to deploy high-end security systems. My idea would be to encode each content on Netflix with effective algorithms that immediately report to our office whenever Netflix content gets grabbed unauthorisedly. Take legal against all such operators by removing the content from their platforms too.

For the second problem, Competitors provide service for free however, the users are initially getting attracted to that free service due to their wide Pr & Marketing. Nonetheless, this shouldn't intimidate Netflix since we have high-quality & engaging content that caters to a wider audience. Lets not leave this drop for the hope of regaining users, my idea would be to let users select their preference of content(like 5 movies max as a cap) to watch it for free on Netflix for a month only. PR & Marketing for this strategy should be spread across various channels & buzz to be created. This would even help our lost customers to experience Netflix again as well as attract new customers.

 

That is a huge drop in any one metric

Some assumptions: It was sudden. If it wasn’t sudden than I as well as others should have and would have noticed. 

Clarifications: What metric are we talking about, was it a monetization drop, a drop in thru put, drop in engagement, offloads, application downloads, what exactly dropped because all those metrics are fed by different systems and we will need to define the metric to understand which system or process may be off.

Internal v. External

Now there are internal and external factors to consider. Was it something, we introduced into market or something we failed to react to within the market. With a decline this sudden, I would be very surprised that was entirely based on competitive market conditions. It would not be a surprise otherwise but we should be thorough. 

External 

• Have their been any system outages? 

• Have our competitors had similar experiences? 

• Have our competitors had subsequent surges, meaning they are leaving our platform. And why? 

• Personally, I don’t think a drop this low would either be gradual or directly related to competitive pressure. But you should be thorough.

• Check reliable user groups, media channels, technology forums for any mention or large scale outages. If so, contact marketing and have them work on a communication plan separately.

• If nothing is available via traditional channels, then let’s check with our partners. With Video streaming platforms. One party provides the content and others provide the technology so depending upon which side of that supply chain I would check in with external partners, content delivery providers, or google, samsung, the video streaming devices and platforms.

Internal 

• Was there a release during the time of the outage?

• What time did it occur, has it continued, what has happened sinc?

• Can we define to a specific region, can it be isolated by a server, etc.

• If it’s monetization have we had any bug/ issues associated with our payment layer.

• If it was quality issues, usage issues, where our performance metrics, have been seen sudden declines in thru put or lag-time. 

• What about the content itself, are there any content-related issues that we did not expect?

Then once you do this you should have enough data to apply some Fault Tree Analysis and recommendations for the future. 

 

1. Define the system that has failed. Was this just one failure within a larger system or are there gremlins out there that may have not yet surfaced?

2. Define the starting point for the failure. So in this case, let’s say it happened at midnight in North America for all streaming devices. 

3. Identify the initial cause for the failure. What event or seuqnece of events caused the failure (can it be replicated in sandbox for further analysis so that our development team can begin triaging?

4. Identify the next level of events. Look beyond the broken vase on the floor and go deeper into other precipitating events that might have gotten us here. I.e., was it precariously placed on the table, was the table shaky in the first place, etc. 

5. Identify the root cause.

6. Provide the probability of how likely this might happen again.

7. Measure the impacts of this happening again.

8. Develop a fall back plan/future risk mitigation plans. This could be pulling back code, additional smoke testing, new vendors, etc. recommended changes in business processes,etc. 

9. Document everything including analytical assumptions, action items and notes.

The first step would be to look at the importance of the metric. In this case, it is specified in the question that it is a key metric, which means it needs to be looked into urgently.

The next step would be to look at the larger trend of the metric to understand the time frame in which the drop occurred. An 80% drop is very steep. My assumption here is that since it is a key metric, a gradual reduction in the metric would have been flagged earlier and analysed. 

It would also help to understand what the variance of this metric has been in the past. If it is a very volatile metric, a large decrease may not be a cause for alarm, but again, this is an 80% drop which is unlikely in a very short period.

The next step would be to look at the formula of the metric and the product funnel that this metric is a part of. If one of the parameters used to calculate this metric has caused this drop, we need to focus on that parameter. Similarly, if one of the previous stages in the funnel has seen a similar drop, we need to shift focus to that.

Once the exact metric has been identified, we need to look at internal and external factors.

Internal factors could be:

• Product release causing something to break
• Product release resulting in users using the app differently
• Issues on the backend (Server down, API failing)

• Competitors
• Seasonality
• Macroeconomic factors

• If it is due to a newly introduced issue in the product, an emergency fix could be released.
• If it is an issue with the backend platform, the application support and DevOps teams will need to check and rectify the issue.
• If it is due to external factors, a medium to long term plan would need to be defined on how to counter it. Though external factors would usually affect metrics over time and not cause a sudden 80% reduction unless it is a very unique event (like the COVID pandemic).

C: First i would want to know which is the Key metric we are talking about here.

I: We are talking about Avg View time/day

C: Okay. 80% seems to be a very high drop. What is the duration of the drop? Is it a sudden one or a gradual one?

I: It is relatively gradual. It has happened over the past two weeks.

C: Okay. And is the graph uniformly declining or it has peaks and troughs?

I: It is uniformly declining.

C: Okay. Is there any particular geography or demography that is contributing more to the decline or is it uniform?

I: It is more or less uniform

C: Okay. Are there any device skews? As in the App or the television?

I: No.

C: Okay. So far, as this is not a sudden decline, I can rule out any bug or something which broke at our end as they tend to give sudden declines and sharp curves.

I would like to now narrow down my focus to 3 things:

1. Anything that the competition did (External focused)

2. Any macro level or international event  (External focused)

3. Any experiment or campaign that we did ( Internal focused)

I would first like to talk about internal focused factors. Did we do any experiment or A/B test during this time?

I: We do them all the time.

C: Is the manager on any of them reporting any stark differences in the Avg view time/day in the control and variant groups?

I: No

C: Same question for any campaigns that we ran for the past fortnight

I: No, strark differences in Avg View time

C: Okay, now since, there was no difference due to any experiment/A/B test/ campaigns during this time, I am ruling out any internal focused factor. Lets talk about external focused factors now

Are we reporting any major launch by any competitor that might have led to decrease in our traffic?

I: No, we don't see any

C: Okay. Did any major interntational event happen during the past fortnight?

I: Yes, there was the soccer world cup

C: Okay, I think we are getting to the reason.

Did the time of the day when the Avg view times decreased also coincided with the times when there were live matches going on?

I: Yes, there is a correlation

C: I see, so i think that is why the Avg View time declined for us.

I: Okay. What would you have done differently to counter this?

C: I would have known that a soccer world cup is coming up. I as a streaming service could have prepared better by

1. Show some old matches, soccer documentaries or sports series

2. Partner or have rights to show soccer matches live or recorded

3. Release a new series around sports or soccer

4. Be more aggressive and focus on geographies that predominantly don't have a large soccer following like India and go local with more regional series/shows there

I: Which one would you pick out of your suggestions?

C: I would pick 1,3 and 4. I understand that getting rights to a soccer world cup is not easy and requires a lot of investment which I might not be able to recover as I have a subscription model and not advertisement based model

I: Okay, thank you

80% drop in any key metric is a huge drop. Let's try to find the problem first.

1. What metric dropped by 80%? Identify which metric has dropped by 80%. Is it related to user acquisition, user activation, user engagement, user retention, user monetization, or user referral? 

2. Which product/issue was affected by the drop? Was the drop for a product level KPI or issue level KPI? 

3. When was the drop observed? Was the drop sudden or gradual? If the drop was observed in a single day, it is highly likely that user engagement or daily active users may have dropped by 80%.

4. Where was the drop observed? Was the drop observed only in a specific country/area or was it observed across the world?

5. Why did the drop happen? When you know the answers to all the above questions, ask yourself the ultimate question: Why did the drop happen? Following are the possible reasons:

i. The drop may be due to a change in the product. The company might have launched a new feature which could result in the drop. The company might have changed the price of the product.

ii. The drop may be due to a change in the people's perception of the product. For example, since the product is a streaming video service, it is possible that a video was uploaded on the platform, which might have hurt the sentiments of the public resulting in the possible boycott of the product.

iii. The drop may be due to the company's competitors. Competitors might have come with a new feature. Or they may have reduced the price. It is also possible that a new competitor might have come.

iv. The drop may be due to a technological issue. The servers may have gone down leading to inaccessibility of the website.

 

Once the cause is identified, we need to identify its impacts. If it is due to the content published on the platform, we need to take it down as soon as possible and issue a statement in the press maybe. Because a damage in the reputation can spread like wildfire in the era of social media. If it is due to technological errors, it should get resolved by the development team. Although it may not have long term impacts because even giants like Facebook suffer from technological issues, however, it may lead to decrease in revenue. 

Clarifying question – which type of metrics are we talking about. Customer facing metrics or backend service-related metrics.

Below I am going to go with customer facing metrics.

To begin I would like to call out the most important metrics that video streaming sites would use

1.       Avg no of active users in a day

2.       Time of day that has max users

3.       No of active subscriptions

4.       No of users whose subscriptions that have renewed it

5.       Most sought after subscription plan

6.       Percent usage of personalised recommendations made by site to users

7.       Avg no of movies streaming per sec/ min/ hour/ day – also a graph displaying trends across past 7 days, 1-month etc

8.       Most watched genre of the day/ month/ year – all three

9.       Demographic that most uses my site

10.   Demographic to genre mapping – which age group watches what genre

These are few metrics that come to my mind. Also, I would explain why streaming sites would need these metrics. Such metrics basically help the site understand the trending user pulse in terms of which genre is most trending at the moment or what type of users use the site. Basically, helps the site to understand their customer segment and if its changing. This will help the site to stream relevant content and attract the right segment.

Clarifying question – Is there a specific type of metric that has gone down? Here I will go with active subscriptions metric and suggest the remedial steps that can be taken.

1. This would mean lesser users are subscribing or users whose membership has expired are not renewing. We can compare across all the metrics to narrow down the actual problem. Here we can pin point whether the site is not attracting new customers or whether older customers are not renewing membership.

2. A comparison of the trend across a longer period like a quarter or year can also help understand the avg trend of this metrics. A trend analysis will help understand if it’s a cause for alarm.

3. Compare with competitive sites how they are doing in terms of this metric. See their trend to understand if this is an industry-wide trend or specific to your site. For example, at the beginning of the year the No of active subscriptions might fall if users haven’t immediately renewed their expired memberships, but eventually this metric might pick up. This trend might be common across all subscription-based streaming sites. 

Also, once every quarter or year, go through all metrics and check their relevance with industry trends and decide which metric is actually helping improve the site and which metrics are obsolete

In this case if the problem is internal i.e competitor sites are doing well, then device a strategy to increase this metric through better marketing strategy, expanding to new customer segments, improve the content on the site and check if the content is relevant when compared to competitor sites.

If the problem is external i.e there has been an industry dip then see competitive industries that are thriving and see if this seems like a long term shift or short term one. If long term, then strategise to venture into new product categories and diversify.

Summary

Before making a conclusion and creating panic, we need pin point the root cause by comparing with other metrics of our site and of competitors. An internal and external analysis is required.

 

Here is my approach

1. Ask clarifying questions about
   1. What kind of streaming video service is it? Is it paid service like Netflix, Disney, or more like youtube, Vimeo, etc.?
   2. Ask if the interview can give more details about the metric and how it is defined?
   3. 80% is a massive drop, and what is the time range for the decline? Month over month, week over week, etc.
   4. Is the decline specific to a device type/OS/browser? Mobile or Desktop and drill more based on the response
   5. Ask if the drop is more prominent for a particular user segment, specific geographies?
2. Summarize the question based on what is learned from the clarification questions
3. State out the approach for the rest of the analysis. "My approach to such problems is to traverse down the tree branches and rule out each pathway. I generally break the factors into two categories."
   1. factors that are in our control, i.e., internal factors
   2. factors that are out of our control, i.e., external factors
4. I would start exploring the internal factors branch.
   1. Is it a data accuracy or quality issue? Has anything changed in how we capture data, run ETL, etc.?
   2. Is it a code change that we or any other product teams made recently? Or Is any team running product experiments?
   3. Is it a content quality that has gone down or a recommendation algorithm that has changed?
5. Assuming we exhaust all internal factors and cannot find a root cause, we will move to external factors.
   1. Is it a seasonal drop?
   2. Has user behavior changed about consuming such video streaming services?
   3. Has a competitor launched some new feature or product offering that is impacting us?
   4. Is it a social or political issue? A ban on the app or something else?
   5. Was there a marketing change that happened? Such as a negative article published about our service or running a marketing campaign to acquire users that stopped?
6. Summarize the analysis once we have narrowed down to the specific root cause and follow it up with few proposed solutions. 

Being a product manager of a Collaboration application and spending close to 6 years in this domain, there are certain key metrics which I believe all PM will keep a close eye. Couple of them are or could be :

Metrics 1: Number of Calls being made

Metrics 2: Duration and Type of the calls

Metrics 3: Frequency and #of Call drops

Metrics 4: Number of New Registrations

Metrics 5: Drop in Existing Enrollments

Now let me tell you where I always chose above 5 are my basic checks for everyday. Couple of years back , I was building a Audio/Video Application for small scale call centers. We build WebRTC protocol and also using a PAAS to host our application and ultimately providing a SAAS solution to end users. One day I found that the metrics 1 and 2 (above) are showing a downward curve. Definitely that was a scary moment. Now to diagonize the problem, I sat with my Scrum Team and Ops team and asked them to figure out the reasons. They started with Metrics#3 to find a root cause. We found an interesting pattern.

Users (Registered or Unregistered) when was using Video Call due to intermittent disconnection caused from sudden drop in the bandwidth of their internet connection, showed an inhibition to re-login.

From here I learnt that UX Design and Technical Design are 2 silent monsters which we have to tame all the time. We introduced a simple banner which started telling users that they are facing “Bad network and that may interrupt their call quality” and this simple improved my Metrics 1 and 2 by 100%.

Would like to hear your feedback and Suggestions, on how you would have improved this situation?

First of all, 80% is a huge drop. 

Let's see some of the metrics that matters to video streaming service

• Viewing hours or # of hours viewed per day
• # of Unique Users per day
• Session Starts per day
• Daily Active Devices (Yes, not users but devices)

Internal:

• Analytics / Reporting:  Is this a real drop or some kind of reporting issue
• Any recent code changes?  Look at the recent release notes to see all the features relaased
• Any experiements running?  If so are these in any way affecting the users - Look at the hypothesis.  Are the control and variant setup correctly.

• Competitor Product - Has the competitor launched any new product / pricing changes?
• Seasonality - Usually 80% down isn't seasonal issue but keeping a radar on this is a good idea
• If it is a subscription service - Has anything related to Pricing changes? (Our product as well as competitors)
• Is it location (Geo) specific
• 3rd Pary integrations - CloudFront or any other Hosted services issue
• Specific Device or Across the board.

Q. You are the PM for a streaming video service. You come into the office and see that one key metric has dropped by 80%. What will you do?

 

Ans.

Assuming the Video Streaming service is something like Youtube

 

One key metric has dropped by 80%, is this a sudden drop or gradual drop?

Assuming the drop was sudden, did we release some new feature/build recently ?

If not, I will analyse the drop for each of the below breakdowns-

1. Acquisiotion channels e.g. Google Organic, FB Ads etc

2. Device Type i.e. Desktop, Mobile, Tablet

3. Browsers

4. Technical factors - Page load time, Server up-time etc.

5. Geography e.g. City wise/Country wise breakup

6. Device Model e.g. iPhone XR, Xiaomi etc

7. Device OS

8. Page wise funnel/dropoff

9. External factors - Data speed of Telecom service provide has reduced/stopped, Government regulation/order etc

 

Basis the above breakdowns, I will dig deep further.

For example, if the conversion from FB Ads has gone down, I would further analyse if-

1. If the FB Ads landing page has the key metric drop for all acquisition channel

2. If yes, that means the issue is something to do with the landing page (& not for that channel)

3. If all other acquisition channels did not heve the key metric drop, then I can further analyse the FB ads

i. If there was some changes in the FB Ads creatives recently

ii. If there was any changes in the Ads that we were running recently e.g. Configuration of target audience

iii. If other companies are also facing key metric drop (can check online blogs or else with the friends/peers in other companies)