What would you do if you found out our uploads were down 50%?

Following line of inquiry could serve as a good starting point to understand and scope the problem space: 

1.   [ When] Duration for which the issue has existed (transient vs gradual)

2.   [ Where] Geographical Region

3.   [Who] Customer Group Impacted > (Business accounts, Personal accounts, both)

4.   [Platforms] Web, desktop, mobile; and OS.

Initial Information

Let’s assume that after the initial questions with the interviewer, we have the following facts with us:

• Uploads dropped sharply in last 24 hours 
• North America region was impacted
• It was not limited to any particular customer segment 
• Client or platform part is unclear, and part of the investigation you are leading 

After limiting the scope of the questions, we can start by examining changes In

1.   Integrity of our Measurement Systems

2.   Customer Behavior

3.   Technical Changes made by Dropbox

4.   New Patches/OS version released by Windows

[Note: We will not pursue competition as a root cause on priority here because we established that the decline is not gradual trend. Furthermore, the sequence of investigation below is not in a particular order. Known information may help us determine the right starting point of our investigation]

1.   Integrity of Measurement System

Important to establish that drop is not error in data error. Immediate step should examine any changes to the analytic system, migrations, business rules or other code changes pushed last 24-48 hours.

Sanity Checks on the network traffic volumes, server’s health and server load, server downtime average users, average downloads, and customer complaints.

2.   Customer Behavior

Dropbox customer base is big. Consequently, a large user base has to shift behaviors to impact the average upload metrics. 

a.   Unusual social incidents/events that are infrequent and difficult to predict

(2020 presents a good example here). Events like Election Days can impact user activity and corresponding have transient impact on the metrics. The analytics systems can flag these as alerts because they are infrequent and not anticipated.

b.   UX changes switching User Behavior:  We should investigate the recent code changes or UI changes that might be impacting a switch in user behavior. (However, it should be noted that this part of investigation is a due diligence step because it is unlikely that a sudden drop in upload is UX driven)

3.   Technical Reasons

Overlay the timelines and performance metrics of

1.   dial up of code pushes to production,

2.   network traffic volumes,

3.   Servers health metrics (such as peak and average loads, downtime, load balancing efficiency etc.)

4.   New Patches/OS version released by Windows

5.   Average active users

This can help draw correlations between anomalies in one or more components of the system design and drop in uploads. This is particularly useful to know where to double click and dive deep. It can help isolate the problem within platform, client, network and server.

Essential Follow Ups

1.   If the issues are due to recent code changes, revert to the previous stable version while the code goes back through the QA cycle. Furthermore, to mitigate future issues, we should trace the bug by working backwards and further strengthen QA and testing tools and processes.

2.   If the issues were infrastructure related, then a root cause analysis should follow to identify and remedy the failure points in the design.

3.   Finally, if we identified a mistake, we should ensure that the moral of the technical team is not let down. When innovative work is done, there may be few bumps along the way. Important to squeeze every bit of learning from such incidents