How would you go about designing a dashboard of call quality for WhatsApp?

Design a whatsApp call quality dashboard

Situation
I want to first describe my understanding of the product. WhatsApp is an online instant messaging service that allows users to make audio or video calls and share media files. 

Next, I want to define the call quality measure as any aspect of the call that interrupts the fluency or continuity of the communication with the user (caller) and the receiver of the call (callee) and/ or call ends without either party intending it to.

Personas
Now, let’s discuss a scenario that a person would use the calling feature. 

Persona 1: For the sake of time, I am going to assume one persona for this use case. 

• Working millennial in her late 20s. 

• She lives away from her family and works in a different city. 

• She has a hectic schedule living on her own and working overtime to get ahead in her career.

• She misses her family and likes to touch base with her mother every other evening after returning from work and prefers to use WhatsApp to save money on her phone bill.  She likes to use the video calling feature so her grandmother (who has suffered hearing loss a few years ago) can see her every day, but the connection is not that great where her family lives so she sometimes has to make do with audio calls.

Persona 2: Again, to save time, we will focus on the persona of a WhatsApp employee (Tanvir) whose job is to monitor the call quality of people living in the region where Sally’s and her family live. I will go ahead by defining the scope quickly. I am making some very broad assumptions about his job being an operational NOC sort of role (who basically looks out for big problems), to be able to complete the case study.

• As a user Tanvir needs to monitor the call quality of the whatsapp calls in a certain region by viewing the dashboard on a web-based UI.

• Tanvir needs a UI dashboard that can provide information about the current status of calls being made through the service, should be able to get alerts for any feature or system failures affecting the region. He should be able to view the type of problems occurring before and during a call, the frequency of occurrence and consequence of the same.

• Fundamentally Tanvir will want to know if the service is continuously working or having issues on a global or regional scale or if there are many unrelated isolated incidents occurring.

• At a high level, he will want to know the real-time usage of the app and the number of users experiencing good or bad calls - to determine if there are any systemic issues he needs to address immediately. For further granularity, he will want to drill down into the type of issues callers are experiencing to alert support teams and get them resolved.

Use case Scenarios

Now, I want to list out the issues and prioritize them to begin designing my dashboard. I want to focus on Audio calls for this use case. 

The main objectives of the call at a high level are as follows:

1.  Call made successfully - User makes call and receiver is able to speak to the user in a legible manner

2. The sound clarity enabled both parties to speak to each other without interruption or disturbance for the length of time they desired.

Prioritization

Call quality can be measured in many ways, most of what I have listed below are not as technical. However, I would like to go ahead and prioritize the audio issues that Tanvir will probably need to see in his dashboard to monitor the call quality levels in real-time.

Metrics to consider:

1. # of active calls per minute - Audio

2. # of calls disconnected per minute - Audio (call was made and then disconnected)

3. # of calls successfully made per minute (where user initiates and ends intentionally)

4. # of calls with complete audio loss - Audio

5. # of calls with partial audio loss - Audio

6. # of calls failed (call not able to connect after user initiates call)

7. Overall average daily call success rate (rate of total calls made to number of calls disconnected)

8. Overall average daily call drop/disconnect rate

9. Overall average daily Customer Satisfaction Rating 

Let’s say I have to choose only 3 metrics for Tanvir’s dashboard. I would prioritize them by broad categories of service availability since Tanvir’s role is primarily to monitor the uptime of the service.

Conclusion: I would represent the following 3 in a visual chart, perhaps a barometer style to design the dashboard for Tanvir that gives him the most important information that indicates a global service disruption issue to send alerts to other teams.

1. Overall average daily call drop/disconnect rate

2. # of calls failed > by device, by internet provider, by zip code etc.

3. Overall average daily Customer Satisfaction Rating

How would you go about designing a dashboard of Call Quality for whatsapp?

Problem clarity

Let's get some clarity on why we want to do this:

Who's this for? I'm assuming PMs/People working on this core functionality

What's this for? I'm assuming this will indicative of product health - this might help with the following ensure that product health is fine, in case it's not where is the problem? when the did problem occur etc

What do you mean calls? Audio or Video? assuming both 

Customers

Who will be core user this dashboard? Within the company I can expect product/product analysts to be using this, I'm going to assume product analysts since most PMs part time do wear that hat 

Customer needs

• Quick Product health check - is the product working as expected
• Track product progress - how is my product performing over a time frame, is it worse or better or same
  • Set target for certain metrics
• Quick Analysis - be able to manipulate data
  • data comparison/insights: What fluctuation is normal vs what is not 
  • data drill down:Set up manipulated data as a dashboard view
• automated issue detection - is there something wrong? where is the issue
• Permission management: Certain data/dashboards are priviliged access/restricted by team/people 

Clarify the question

1. Can we assume that this product will be used to monitor the call quality and not the overall health of the system (like outages etc.)? Yes
2. What is the goal of the product? Increase customer satisfaction which in turn will result in increased customer acquisition and engagement.
3. Call quality can be measured for real time audio calls, recorded audio messages, group audio calls & video calls? Let us start by focusing on audio call.

We will be building dashboard which will display the quality of the calls in WhatsApp. The dashboard will be used to also identify the top issues at any given time.

We will look at the possible users of the dashboard, identify high level use-cases & features, and prioritize the feature which might have biggest value with lowest effort.

Identify the users and customers

The dashboard will be used by data analysts, business or product managers to identify the overall call quality of the WhatsApp.

What are the use cases?

There are 2-ways I can think of on how to measure the calls.

The first way would be providing rating system where the caller & recipient of the call will provide rating of 1-5 (5 being happy). We could display comments section if the users choose rating below 2. However, the system of rating would not be perfect because of the human behavior - users might not provide ratings, users tend to not use 2,3,4.

Another method might be to use technology to automatically find out metrics like

• Calls successfully completed
• Calls not answered
• Calls disconnected at callers
• Calls disconnected at recipient end
• Calls with poor audio quality

Furthermore, we can also collect data like the users, device, location, network (wired or Wi-Fi) etc.

Let us assume that we are using the metrics rather than the survey.

1. Display call quality trend graph (classify data with the 5 metrics above)
2. Display various filters - time period, location, network (wired or Wi-Fi), device, users, service provider 
3. Allow users to select multiple filters above
4. Show map of the globe with possible hotspots
5. Allow users to set alerts on the above metrics
6. Allow users to set alerts on the above metrics + filter (like location)
7. Notify users thru various channels for alerts.
8. Provide actionable suggestions to users based on past history

What are the features we will be implementing?

Wrap-up

The dashboard seems like huge project. We should thoughfully implement critical platform features and iterate on that.

The vision of the product should be to bring VOIP on par with the quality levels in traditional telephony. Furthermore, this product can also be used to provide suggestions to the users on the stregnth of the signal, market competing wireless technology which might give better audio quality.

How would you go about designing a dashboard of Call Quality for whatsapp?