You are PM of Medium and you notice some fluctuations in % of active users. After 3 weeks of monitoring, you notice one morning that the % of active users drop by 35%. What would you do to figure out the drop and how will you restore it?

Question

You are PM of Medium and you notice some fluctuations in % of active users. After 3 weeks of monitoring, you notice one morning that the % of active users drop by 35%. What would you do to figure out the drop and how will you restore it?

Asked at Medium over a year ago

1.1k views

Problem Solving category

Asked at

1.1k views

How to answer Problem Solving questions

Interview Guide

Top Problem Solving interview questions

Invite members
Invite by email

Add another

Tushar · Answer 1 · 2020-12-11T18:47:17+0000

Clarify:

What is the timeline frequency of the active user we are talking about here? Is it D, W, M Interviewer: D
What do you mean by active users? Does it mean people who have logged in and opened a session irrespective of the time they spent on the platform Interviewer: Yes
The fluctuations seen earlier in the active users were positive or negative or both? I believe the fluctuation was in +/- < 1% and that is why we did not look at it in till now. Interviewer: Yes
Is the absolute number change is significant as well as sometimes the population is low and percentage change is thus huge Interviewer: Yes
Was a drop seen on any specific channel - Mobile(IOS/Android)/Web? Interviewer: All
Was the drop specific to a particular geography? Interviewer: USA
Seasonality: Did we see the drop last year as well during this time? IVR: No

Thank you for clarifying. The objective is to find the potential root cause of the sudden drop in the % of active users and ways to restore it. The drop has occurred across all channels in USA. Does that sound right? Interviewer: Sounds good

I would first look at the internal and external factors and go one-by-one to negate the possibilities. And then after I drill down on the cause, would look at potential solutions to restore it.

Internal factors:

Any recent change in the way we measure the metric? No
Any recent retirement on promotions campaigns? No
Do we see change in any other metrics other than #/% daily active users?
- Is it specific to a particular user group? Readers, Posters IVR: Both
- Do we see change in the number of posts as well or is the drop related to readers only and by how much ? IVR : we see a change in both but do not have specific numbers
- Are new users able to sign up and do we include the new users in the active category? Yes
- Any drop in new users sign up? No
Any chnages in the content exposed to google search crawlers? If this is the case, it is possible that people who are conducting a google search do not see relevant content from medium and thus do not go there. The organic/inorganic search numbers should also be impacted if thats the case IVR: No
Did we do any recent changes in the past three weeks related to UI that makes people unable to see sign in pages. We should be able to verify it with unique no. of ip address visiting medium should not have dropped. If thats the case, it depcits that users are visiting the platform but not able to sign in IVR: We do not make any recent UI changes
Did we see any recent problems with AuthN/AuthO - No/# of errors in signing in must be up IVR: Yes, we do see a spike in AuthN errors
- Could it be due to mediums identity solution? Any changes there IVR: No
- It could be due to SSO platforms like google, facebook not allowing users to go in medium. I would like to see the numbers for users who use SSO and granularity with google and facebook sign on individually. IVR: Bingo. Google API key was not updated and we see the numbers match with the drop.
  - So it means that all the users using google sign on in USA were not able to login using their G profile. IVR: Yes

External factors (wont be mentioning in the interview as already found the problem as part of internal factors):

Any issues with third party servers? IVR:No
Google page rank changes
Any regularoty issues with gov putting sanctions or statements against medium in the US
Bad PR - medium sued or bad advertising leading to a backlash
Major event in US and people do not have time to post/read on medium
- Elections
- Protests
- Communication system/Internet Services Down
New competitor launched or existing competitor luring users via promotions, advertising, influencers
Some major influences on the medium platform left? As a result the followers of these do not go to medium as they have lost interest

So, we have found the problem that users were not able to sign in on medium using their G profile due to G API key being not updated on the medium platform.

I would work with my engineering team to confirm the root cause and check in the test environment that user is unable to login using G profile. Post confirmation, I will work with the team to get the appropriate key updated and test the solution. The deployment should be real-time for API key update and would need no other considerations.

Once updated, I will monitor the live traffic dashboard to see that the active users have increased and the next day dashboard should depict the active user level to the previous ones (seeing a uplift)

I will work with my cross functional partners to understand how we missed the API key updates - was it not shared with our medium team, did we not update it, did we update it incorrectly, how we did not test it if it was updated incorrectly.

I found the G API Key not being updated successfully as the cause due to which users were not able to leverage SSO via their G profile. The G API Key expired and worked with the eng team to update, test and deploy it. I update the lesson learnt to understand the gap for this miss.

I monitors the D/W active user metrics and was able to restore them to previous levels. (Total time - 35 mins)

Please share your feedback. Thank you!

Rishabh Verma · Answer 2 · 2020-03-28T20:58:45+0000

Part 1: Problem Identification:

Identification and segmentation of cause for user-churn:

1. Demography (IP locations having the most increased churn by comparing with past data)

2. Ethnography (age, likes, profession)

3. User Agent (Browser info, OS , Android App, Web-App)

4. A/B split testing (whether a new feature integration has led to UX problems)

5. Decline in Content-wise user interaction (any topic which is unsuitable or not creating a good user interaction)

6. HTTP errors displayed (technical, web-page unavailability)

Metrics:

1. Bounce rate

2. Pageviews

3. Reduction in Referer vs Organic vs Social search links

4. no. of sessions

5. average sessions duration

Part 2: Problem Analysis -

After analysis of Medium Web analytics it is found that the cause of regular churn is due to:

Demography - Users from Metropolitan cities are not visiting much now.

Ethnography - People who are relatively young having affection for Product management are the churn

User-Agent - No. of users who use Mozilla Firefox and Android OS is lowering.

A/B Testing : The new feature where you can form a group of people having same likes as yours is not working as per the intention. ~ 35% churn is due to this maybe.

HTTP 504 Gateway Error (Error of 5XX represents server side error)

Metrics - Reduced no. of sessions

Part 3: Solution-

Demography - Some other blog website is becoming popular as metro people are most aware about copetititors - Competitor Analysis is required

Ethnography - Source more interactive writers from Product community

User Agent - Android team needs to roll out a new bug-fix feature update which is correlated with Error 504 and sessions reduction problems.

A/B Testing : The new feature does not create instant notifications on phones of group members instantly. Now A/B test again with instant notifications insertion.

paribest_007 · Answer 3 · 2020-03-28T00:01:08+0000

Clarify that there are 2 problems that we are solving - "some fluctuations in % of active users" & "sudden drop of 35% of active users"?

Let me start by focusing on the biggest problem today which is the sudden drop of 35% of active users. The timing of the sudden drop is very curious.

Medium is a blog where content is published and consumed. It is fair to rule out that there are big changes in the number of articles published by publishers.

I would also check if this has happened in the past to rule out seasonality. Perhaps 35% is too big of a drop because there is very less change of seasonality impacting content publishing businesses.

Was there any technology change or outage that might have causes this? Let us assume that there were no major features (or) outages.

Now, it might be best to discuss the metrics –

Analysis of the sources of traffic – direct (browse & search in site), referral (from other sites), search (from google, bing, etc.), campaign (marketing ads), social networks
Can we isolate the problem of drop in traffic to particular region?
Is there drop in registered users (or) guests (or) users able to login.

Our goal is to analyze any change that might have impacted the page ranking (indicated by drop in search traffic), drop in traffic by direct search on medium, end of any marketing campaigns, drop in users being able to login to the site to read the articles etc.

Once we are able to identify the reason for 35% sudden drop. We can investigate if the fluctuations in recent time was caused by the same issue (or) the fluctuation is in the normal range.

----
Category: Problem solving