You are the PM of Messenger, you noticed that its DAU has gone down significantly. How would you go about looking for the root cause?

Thanks for the wonderful question. I believe that the only way to go about and find the Root cause is by understanding the Question, then the Product.

How do we define our DAU of Messenger? We define  DAU as the users who open Messenger and read a message or send a message.

How long are we facing this issue? It has been for the past 4 hours and is now currently coming up slowly.

Do we see any realtions while checking for different demographics of users like age, mobile type, regions? Yeah, the users belonging to the age of 18-25 are the ones showing a huge drop. However, things are pretty same across regions, mobile type.

Is there any seasonality associated with the drop, i.e, Are we able to see the drop happened in a similar way previously? No, we are not able to map any seasonality so far in the last few years.

How are the other  metrics doing related to  Messenger and as well as the FB? Every other metrics have dropped a bit or stayed same. However, the number of stories has increased in FB significantly.

Was this a sudden change or a gradual one? Like I said, the drop came all of a sudden for 4 hours together and is increasing now.

Okay, This seems unique. A significant drop coming up and returning back to its normal stage in another 4 hours. Though, this is getting rectified already, it should be pondered and as a team we'll have to understand the cause behind this. Only, then will we be able to handle if anything similar happens next.

The fact that the drop is only significant among users of age 18-25 makes me feel that they could have jumped to some competition product or Instagram DMs or Whatsapp and would want to place my bet on External  aspects. However, since I also got to know that the drop was same across regions, mobile devices, and was a sudden one for just 4 hours make me focus on Internal Issues. This is because, External issues tend to cause us a damage slowly and gradually. By the way, Internal bucket will have factors and issues which can be controlled by FB directly and External has things that can't be regulated by Facebook like Bad PR, Government Regulations, User behaviour etc.

Since, it is 3 reasons relating to Internal Vs 1 reason associated to External, I would first want to look into Internal Bucket.

Internal Bucket:

I would first of all want to check if we are checking with the correct data as things could have gone haywire as there was a bug in the system that collects data and once the data team got it corrected things are going back. That certainly makes sense. However, that was not an issue. Infact, the data with which we are working has been checked more than twice already. So you may skip any issues related to data.

Okay, that is one thing down the list. Next, I would want to know if we did any change that is causing such an issue. For example, a recent update that we did could have a bug due to which app crashes as soon as one opens. This could be checked by checking if there are any relationship between DAU with different versions of Messenger, # of Crash reports generated. Thanks for suggesting a few metrics on how we can confirm this. On checking, we got to know that this was not a case.

Oh, nice then. Since we the issue is with DAU of messenger and at the same time Metrics related to stories have gone up, I would like to see how are the tocupoints doing, i.e, opening Messenger can be done through Directly through the app, notification from the app, from FB through message icon at the top right. Do we have any difference in this touchpoints? No, all the touchpoints are performing low.

The reason I wanted to know this was because, if there was any difference, we can check things related to that, like algorithm that works for the notification. However, since it is common, I would next go to one final probable reason I have related to internal bucket.

Was there any grand launch of new feature within FB, Whatsapp, or Instagram that has caught the attention of tech savvy people in the  recent past of 6 -12 hours, that all these people started trying that out and were sharing something related to it in their stories as well. This seems relating to what we face-drop in DAU messenger andincrease in stories. But unfortunately,we did not launch anything of that sort in the specified timeline.

Oh, if this was also not a reason, then I would like to jump  to the External Segment.

External Bucket:

 Since the change was a sudden one, I would  like to start with my discussion around PR. 

Did we face any bad  PR in the recent past and that got corrected from our legal and  PR team? For example, we could have been charged with some Cyberattack victim and lost critical info of our users to the hackers. Later we could have corrected them that nothing such thing happened. Nothing of that sort has happened.

Since it is across regions, I'm gonna exclude Govt regulations.

Next, I would want to discuss about our competition. though it is unlikely it is not something that is impossible. So, did we come to know any competition campaigning to launch something in the recent past and got  launched in the timeframe when we faced a issue. Since, the issue is worldwide, I would want to consider firms that are present at a larger scale like us and not some small firms at few locations. To be precise, firms like TickTok n Snapchat? Nothing that we are aware of. But how do you plan to check on them?

I guess, Twitter hashtags will help us in this. Trending tags related to tech can be followed to get an idea. Hmm.. okay anyways there are no such things.

So the only thing left is User behaviour. So, before analysing this, I would want to hear few more things.

So, we know the age range in which we are facing a huge issue. apart from that did we see any relation in user demographics. Like I already said, nothing much apart from ususal noises.

Okay. By the way we did see increase in stories right, Do we have any relation there? like which types of stories, links are getting posted in the Story section? That's an interesting question. So on checking we got to know that most of the stories are related to some movie. 

That's a great news. I guess we are finally going in the right direction.
Before sharing what I want to check let me share what am I trying to know.  So my hypothesis is that, Since the drop was only for 4hours, it could be that, people of age group from 18-25 had something so intriguing and intresting that they left our platform and interacted with that event or something and once things got over they came back to their usual ways of chilling through Fb and other sources.

To check this, i would want to know if there was any major  screening or movie function especially in Hollywood that happened in tha recent 4 hours? I'm specifying hollywood as that is one movie group that users from different regions can be interested at a moment. I propose to check this with help of twitter once again. Okay. On checking we got to know that, there was a Marvel  Movie function that got screened at NY and as well as telecasted across the globe through Youtube Live in the timeframe we faced issues. 

Okay this seems why we faced an issue. To be precise, what are stories conveying? were the hashtags, pics relate to Marvel as well? Yeah.

So to summarize, the issue of drop in DAU significantly is because of this activity which grabbed users attention from FB. 

Clarifying questions:

1 - Is this on mobile or desktop?  Assume it's overall

2 - By DAU, are you looking for a specific set of user groups as there can be business users on the platform, or users in general? Assume all users are impacted

3 - Can you also correct me if DAU is defined as someone who just uses the messeneger product on a daily basis by just going into messenger, but without the need to send or reply?  Yes.  We just need a user to go into the tab or the specific app on mobile.

4 - Is this an abrupt or gradual drop? gradual, it has happened for over a period of a month and is continuing

5 - What locale is this isolated to?  No - the drop is about the same for all regions in the world

6 - Lastly, do you have a quantity for signficant?  Ah, yes it's about 10%.

OK.  Thanks for clarifying my questions.  Let me give it some thoughts and I will come back with a list of possible causes both from an internal aspect, which is within Facebook, and also external aspects that are outside of Facebook.   Let's see if one or more of the causes can be confirmed and I can provide a summary.  Does that work for you?  Sure.

INTERNAL FACTORS

OK, first off I would like to look at several internal factors which may explain the drop

1 - First of all, have we confirmed that the reports reporting the drop are correct?  Yes, the analytics tools are working fine.  No issues there.

2 - Is there any seasonality user swings historically that can be attributed to?  No.  Messenger doesn;t have any type of seasonality adjustments. 

3 - Ok has there been a release about a month ago that has introduced bugs that could impact random users from using messenger?

4 - Were there any issues with the Messenger APIs?  Yes, but not all.  Our devops team has reported that the send APIs that are hosted has had some issues over the course of the last couple of weeks, but they aren't completely down.  OK - let me note this down but this alone may not alone drive the DAU by 10%

5 - Were any other Facebook products affected?  i.e. Facebook Core/Newfeeds, etc.?  None.  They are performing in line.

6 - Were there any experiments that were run that could negatively impact Messenger?  There are experiments running on newsfeed.  OK, that may not adversely impact Messenger much as those products are pretty much separate.

EXTERNAL FACTORS

OK.  let me ask if there are external factors which may be of sigificance.

1 - Were there any social events recently like national wide protests that may have driven down the usage of the product?  As far as I know, there are small events but nothing like a massive national protests.

2 - Was there any PR written recently nefatively about facebook as a company or on our messenger product?  None as far I know.  People would occasionally ask about how Messenger works but no negative PR was reported.

3 - Was there a recent competitor launch?  Yes, Google has recently pushed a new version of Hangouts and have been promoting it in Gmail.  In fact they were prompting users to start a hangout chat and giving them beta feedback in being randomly selected for their latest PIxel devices.  OK wow,  now we are onto something.  

Let me think a bit ... we have an internal issue where some of the messenger APIs and that can rresult in some of the users like businesses who integrated with FB messenger cannot reach their customers on FB, and therefore those customers won't get a message back on messenger and respond to them.  It's sporadic but signficant enough.  Now combine that with a recent launch of Hangouts beta test with a promotion, I think I can make a sound conclusion that the combination of the messenger API availability and the Google Hangout beta test +  promo likely caused the 10% DAU drop.  

Let me know what your thoughts are, or if you would like me to take action on addressing the issue.

First, I would need to clarify certain points :

• DAU = Daily active users. How is that defined ? Does the user need to take a specific action or does he just need to open the app ? For desktop, if he goes on FB, does he count as a DAU messenger ?
• Which device was the decline observed ? Let's assume it is Mobile
• How much is significantly ? Let's assume it is around 5%-10%
• Is the decline sudden or progressive ? Let's assume it is progressive

I would split my analysis into two parts :

Internal factors :

• Was there a change recently to the Messenger user experience ? This might have led to users getting dissatisfied with the new experience and coming back less. Here I would check with the product teams to see if there were any new releases and check with them the performance of their chosen metrics and how they evolved pre and post release
• Was there a release recently on another product that might have created a bug on messenger's core features ? For example, maybe users are unable to open the messenger app or unable to send messages (therefore people respond less). This might be relevant to look into since the drop was significant. We can also look into customer feedback and if we have many complaints related to messenger lately
• Was there the same drop on other FB products (Instagram, FB, Whatsapp) ? Here we need to see if the drop is structural to the company or not (this might be caused if there are some PR issues with the brand)
• Did we stop some marketing campaigns that were run previously ? This might lead to an increase in the previous period, and once stopped, we would notice a significant drop ==> Here I would analyze a larger period, and see if the current level of DAU is normal for periods with no marketing campaigns activated
• Who are the users that we are currently losing ?
  • Do we notice a drop in old users ? This might point out towards a retention issue. We'll need to look into why and how we can improve their experience to retain them more
  • Do we notice a drop in new users ? Are we acquiring less new users ? Is that an issue also observed on other FB products or just messenger ? Do we need to look into our acquisition programs ?
  • Do we notice a drop in Business related conversations ? For example, due to covid and lockdowns, a lot of businesses closed, so their customer service were no longer available on messenger

External factors :

• Is the drop seasonal ? Here I would look into historical data to see if we notice a dip in the same period in the previous years. I would assume it is not seasonal since we were not asked to investigate this issue in the previous years
• Is there any specific events happening around the world currently and might be impacting users' behavior in terms of messenging ? (holidays, strikes, protests, election...)
• Is this a market trend ? Are users using less messenging apps ? I don't think this is likely here
• Is there a nex competitor on the market that is rapidly growing and attracting our members ? Here I would look into the countries where Messenger's most known competitors are based and see if we see a decline there (Asia, ME...)