Weekly active users (WAU) for iPhone app dropped. What happened?

Structure

I’m going to start with some clarification questions, make a few assumptions, and then explore a few external and internal factors that may be the reason for the drop.
I’ll make an assumption (instead of the interviewer) and ask for additional information that could assist in understanding the problem and the root cause.

Clarification Questions

1. What is the size of the drop? Is it a problem? (we can check the top 5 WAU drops in percentage we had in the last year)

2. Is it a drop that occurred on a single day or occurs over a few days?

3. Is it a drop in existing users’ usage - Is there a decrease in installs or an increase in uninstalls?

4. Can we segment the devices\users that were missing during this week? Age, Religion, Country\Area, Devices\Operation Systems

5. Were we experiencing a drop in any other apps? (FB Lite, Instagram, WhatsApp)

6. In which device categories we experienced a drop (Desktop, Web mobile, Mobile App, Tablets) – was there an increase in other categories?

 

Assumption: The drop occurred in the regular FB Mobile app in the US.

 

Factors

Internal Factors

• Did we release a new feature? (in FB APP or other FB’s product)

• Are there any A\B tests currently running? Did we expand, limit, or turned off any A\B test?

• Did we launch the app in a new area?

• Did we block some influencers?

 

External Factors

• Is there a significant event? Holiday, memorial day, war, elections, sports events (Olympics, Mondial, Euro, NBA’s final,…)

• Were there any new features or product launches by competitors’ products?

• Were there any changes in regulations?

• Was there an update in iPhone (regulation, Operation system, privacy,..)

• Any social event? Did someone tweet, post, wrote an article about something that can affect iPhone or FaceBook?

• (How are we measuring WAU) was there a change that can affect that (e.g., If Google Analytics update the way it treats and counts sessions)

• Was there a change in FB partnerships or APIs? (Google, Taboola, Youtube, News sites, share to WhatsApp…) - Is there a drop in a specific source?

 

General Factors

• Did we experience a significant decrease in usage in one of the features? (e.g., birthdays, share, like, video plays)

• Did we experience an increase in reports\complaints? Was there something in common? (e.g., increase in the number of clicks on ads “why do I see that?)

• Did we experienced a decrease in performance (Load time | number of app crashes reports)

 

Assumption: The interviewer provides additional information that the drop was mostly in the East-Coast during 3 specific days. In addition, we suspect that:

1. During those days, we made changes in 4 A\B tests.

2. The NBA’s Eastern finals took place (but in the past, such events didn’t cause a drop in WAU)

 

Asking for additional information \ data

 

1. Investigating the A\B tests

   1. Check which A\B version the East-Coast users received? And what was the exposure? 10% 50% 90%?

   2. Was there a change in the behavior of those users? # of sessions, the average length of a session, #/% of interactions (likes, comments, shares, posts, video plays)

   3. Check that there was no intersection between experiments – users that were exposed to 2 or more A\B tests simultaneously.

2. Investigate the sports event

   1. I assume that FB has a classification of users who loves NBA\Sports. I would check if there was a drop in their usage during this week.

   2. Was there a decrease in content published about the sports event relative to previous years?

   3. Was there a decrease in engagement around content related to the sports event relatively to previous years?

   4. Were our competitors also showed a decrease in content\interactions around the sports event? Where did the users go?

   5. Is there a new player around sports events? (e.g., Amazon Prime or Netflix can start streaming the games, Ticktock can add a feature that makes stickers of players,…)

I'd use this structure to answer this Facebook exeuction question.

Weekly active users (WAU) for iPhone app dropped. What happened? 

 

Description: iphone apps are specificially made to work on the iphones. They are made so that they will work with the ios software that powers the iphone. Any iphone app that is not compatible with the ios system will not properly work on the iphone. 

 

Clarification: we know this is a progressive decline and not sudden. so we know there is a fundamental shift in way people are accessing their apps.

                    1) is this decline specific to USA or world wide (USA only)

                    2) If USA is this specific to a certain state, city, region (no)

                    3) all apps or a particular app is giving issues ( all apps)

                    4) Has this happened before, seasonal (no)

                   5) use meaning. the use of app on the phone (yes)

                   6) not related to devlopment of iphone apps( correct)

                   

 

The Hypothesis:

1. I phone users have found an alternative way to access their apps 

2. I phone users are having problems opening their apps

3. I phone users are having a bad experience when using their apps

 

I am going to first investigate external reasons that may be influencing this change and then will look at internal factors so investigate the problem further.

 

External Factors

1.  New competitors granting a more fun easier way to access the apps. people migrating to android phones. is the android market suddendly showing capturing better market share. Perhaps there is a new phone out with amazing features and iphone users are now using andriod phones to access their apps through the andriod platform. Will check for if there is a sudden spike in android phone sales. or a sudden increase in android downloads.

2.  PR Issue: Is there news about data breach on hack regarding iphone or its app that is making people uncomfortable using apps through iphone.

3.  Will check with my engineering team if thers is a possibility of a bug or cyberattack that is causing an interruption in iphone app experience

4.  I will also discover if this progressive decline in app usage is present in android app as well. This will point to the fact that there is either a significant shift in how people are accessing their apps or that a hack or bug has affected all apps and hence affecting app usage.

5.will check if there is a problem with downloading and loading apps from the ios store. That might be causing negative experience but I would imagine that to be caught by their QA team .  so i would rule that out from progressive decline.

6. Regulations: Changes in laws or regulations causing privacy concerns for users, they may feel like their information is not safe of they are being monitored, watched, regulated.  I will check to see if there is such a sentiment prevailing in the iphone users and or android users as well.

 

Assuming that the progressive decline is not caused by a new competitor or a new way to access apps. Assuming it is no android stealing mkt share. There is no known cyber attack, bug or privacy or data breaches that would affect public sentiment and no regulations that have affected the use of iphone or android app usage I will investigate the internal factors that can be causing the drop.

 

Internal Factors:

1. I will check with my engineering team to see if there is any latest update to the apple ios system that is coinciding with the drop of iphone app usage. I will investigate if there is a change in the way iphone apps functionality affected by the update.

 

2. Is it affecting all devices or a particular device? That will help me narrow the scope of the problem to see if there is a 

 

3. check if there is an issue with downloading the apps from the AppStore. An issue with that can cause a gradual negative experience 

 

4. check to see if there is a problem with pushing iphone app updates and also check if that is all over or particular in a specific device

5.  Check if there is a problem with loading time of the i phone apps. that could mean that the apps are either not being updated properly and are being stuck in a bad loop and or its not compatible with the device it is on.

6. Check if the iphone apps are just takign too much space or draining battery life on the phones and the users are now just opting to either not use the apps or turning to use it elsewhere

7. Check if there are reports of iphone apps crashing on all devices or some devices that would help narrow the focus to finding the problem. it could be not updating, or the app is not compatible with the device it is on. That also happens with alot of older phones. So scoping down the device focus would be helpful.

8.  I will check to see if there is a problem with app display in all or specific devices. apps not displaying properly can also cause a WAU to go down

 

 

 

Summary: It would help to narrow down the scope of the problem to a particular device. Look at external factors first that may be influencing any major shift in change in behavior. Following internal user flow investigation across all devices to check if the problem is focused on one device or is across all devices. 

First try to understand the situation by asking clarifications. 1. what is the rate of drop?  (assume it is statistically significant), 2. investigate possible causes: a) are there competive apps similar, b) any broader trends such as negative publicity, c) quality issues that were pending on the app, d) app is data intensive, e) app pricing model changed, f) when was this observed in past day, week, month, quarter?  (for now assume it is a health app such as breathing or pranayama and a new competitive apps is there).

Second, look at the id of users... if they are Boomers, Gen X, Y, Z; geo, psychlogical profiles.....  for now assume that these are mainly Gen X users - mainly women - office and housewives.    

Third, look at the customer needs... women want quick access to app and also want videos and guided talks to help understand and go deeper into practice.   Seems pain point is users now looking for deeper engagements.

ROI - assume drop of 10% causing loss of $100k revenue/month or $20k profit... now to make new videos and improve takes investment of $100k to $500k/... so there is   5 month to much longer breakeven period to recoup investment. however there is upside not counted. also competitor may also improve.

Solutions:  App has to be improved for deeper engagement else will fade away. prioritize with different options: 1. add video, 2. reduce time of loading and make faster, 3. improve UI, 4. add Q&A and 5. Add community forum

look at trade offs including cost and start with simple such as Q&A, then redesign for faster loading and other.

Evaluate the impact of this with reduction in loss of WAU. then iterate with further improvements.

Summary: Root cause for the WAU loss has to be first determined, then looking at potential solutions, pick fast impact one first then iterate.

Clarity

who are speaking as - apple analyst or analyst for a company like say YouTube, lets assume later since you'll never a random drop in apple device WAU owing to long lifecycle of the devices

Confirm the drop

- Over what period - is it natural or at an abnormal rate

- Was it there last year/month/week

- are we sure it's a drop and not due to a bump sometime in the recent past

- Is the data reliable, did we make changes to any metric definitions and are the data pulls updated or maybe it' didn't update correctly.

 

Start investigation 

internal factors

Any other metrics affected? this will help us narrow down the focus?

Any specific cohorts affected? OS version/Phone/Country/Location/ISP/phone service provider

Any technical issues? check error rates

What are the customers saying - check customer feedback for the same period for iphone users, talk to customers who reported issues

Any major releases that had product design changes or logging changes 

 

External

Any major updates from apple?

Any outages on apple in the recent past

Does the competition also show a simialr drop? 

Any major festivals/downtime across the world ( example california fires ) that could lead to this