How would you measure the success of Google Hangouts?

What is google hangouts

• Messaging app by google
• can be used for text chat or video calls 
• dedicated apps + integrated with gmail 
• When users use google calender it sets  up a video meeting by default on hangouts(now called meets)
• There's the B2B space and B2C space
• I think Google hangout/meets is more relevant to the B2B space given B2C space for communication with friends/family is dominated by IM apps like whatsapp/imessage 

• Is the above in line with expectation ie we shoud look at the enterprise usecase 
• Earlier users could message on hangouts but now that is a standalone app
• We are referring all platforms - desktop, app etc 

• 2 major cohorts 
  • Meeting host
  • Audience 
• User journey is as follows
  • Host wants to discuss a topic with a specific audience (can be 1-1 or many-1)
  • It may be using a calendar event or more real time where a link is shared over chat
  • in case of calendar invitees get an email/event start showing on their calendar 
  • Host may choose to insert a custom link (zoom)  for the call 
  • Users join the link, start the call 
    • Users may get disconnected due to internet connectivity 
    • Video or audio may get impacted - pause  or break
    • People may ask you to repeat yourself
  • Sometimes people 
    • present their screens
    • share docs 
    • record meeting 
  • Meeting ends 

1. Daily active overall and 
   1. Setting up meeting (host)
      1. What % of meetings had a non google link
   2. Actually joining meeting (invitee)
      1. Of the meeting no one joined what % of them had a zoom/other software link 
   3. Average meeting size (number of  participants)
2. Timespent/user 
3. Session quality
   1. Stream quality 
      1. Video
      2. Audio
   2. Buffer rate (video)
   3. Audio breakage rate
   4. Disconnect rate (did users get disconnected while they had internet connectivity) and then joined back 
4. Retention rate (WoW)
5. Growth rate - users added vs users lost (churn)

First, let me just quickly summarize what Google Hangouts is.  Google Hangouts is a web and mobile app for video and audio conferencing across both personal and professional use.

 

For the scope of this question, are we referring to a specific element of the product -- e.g., desktop or mobile, Android or iOS, personal or business use cases?  I will assume we are not focusing on a specific channel, but instead will be seeking to metrics to gauge the overall success of the product.

 

Next, let’s clarify what we mean by success because it is quite a general term.  Some different ‘success’ goals come to mind – acquiring new users, engaging existing users, retaining users, monetizing users, etc.  Is there a particular area we’d like to focus on?  For the purpose of this analysis, I will assume our goal is engagement.

 

Now, let’s walk through potential user actions:

-       User dials into a video or audio conference call

-       User shares document/screen in video / audio conference call

-       User leaves a video or audio conference call

-       User hosts a video or audio conference call

-       User sends a chat message

 

Let’s brainstorm some potential metrics for each of these:

 

User dials into a video or audio conference call	-       % Change in number of calls dialed into per month -       % Change in number of dial ins per user per meeting -     % Change in number of calls dialed into on Google Hangouts vs other platforms
User shares document/screen in video / audio conference call	-       % Change in time spent sharing screen / document per call
User leaves a video or audio conference call	-       % Change in number of minutes spent on call per month -    % Change in number of minutes spent on Google Hangouts vs other platforms -       % Change in number of times dials out per user per meeting
User hosts a video or audio conference call	-       % Change in number of calls hosted per month -   % Change in number of calls hosted per month on Google Hangouts vs other platforms
User sends a chat message	-       % Change in number of msgs sent -       % Change in number of msgs opened

 

I would segment each of these metrics by the following categories to gain further insight:

-       Age group

-       Professional / personal usage

-       Web / mobile

-       iOS / Android

-       Geography

 

Finally, I would narrow down to a 'top three metrics' to focus on.  I will prioritize metrics according to which ones measure our goal - user engagement - the best.  From the list, I would select the following:

1)    % Change in number of calls dialed into per month

2)    % Change in number of calls hosted per month

3)    % Change in number of minutes spent on call per month

Clarifying Questions:

Success can be different at different stages of the company. So, what was the business goal with Google Hangouts? Or can I make my assumptions?

Assumption -

Business Goal: Tying users into the ecosystem of GSuite

Product Goal: Google Hangouts has reached a maturity stage, so it would be important to engage and retain users so that they continue using the service. This would allow users to consume more of the GSuite ecosystem (which is the business goal).

Google Hangouts helps users to communicate with each other. The communication can be with text, VOIP, video or by sharing files. The convenience is that it is well integrated into the Google platform, making it accessible across Gmail, Calendar.

There are different types of GSuite users:
Personal Users: Individual user accounts that use Gmail and other GSuite features for personal use to communicate with friends and family
Business Users: Business accounts using GSuite for internal communication.
Professional Use: Individual users using Gsuite in a professional setting like doing interviews etc

A user can start a conversation either from within Gmail desktop site, dedicated hangouts site or through the Hangout app. A user can create an invite to share with other users who they would like to join, so the other user doesn't necessarily have to have a Gmail account.

For Hangouts to be successful, it needs to ensure that there is a steady growth in usage, engagement on Hangouts.
For each of the above user group, the metrics that would inform/drive the business goal stated above are (all measured by user segment):

1. Number of users using Google Hangouts with more than one other GSuite product - As measured by segment

2. Virality - The number of invites sent to join Google Hangouts - this would help measure the proactive intent of using hangouts. Shows that this is the preferred tool for communication.

3. Average Length of chat sessions per user segment

4. Quality of Service - Number of interrupted calls, user feedback/satisfaction on call quality

5. Churn Rate - As there are so many other communication options out there, the switching costs are low. So, it is important to keep this in check, so that users are engaged on GSuite (which is the goal).

6. Engagement - Number of engaged users /  number of GSuite users. Engagement is defined as actions taken (call / chat / share)

If the goal is user retention, the 2 important metrics to measure success of Hangouts is Engagement and Churn.

First, I would confirm if my understanding of the goal of the product is correct. Google Hangout aims to make it easy for people to have video calls with each other. I would also ask clarifying questions regarding what ‘success’ means in the eyes of the interviewer. For the purpose of this interview, I’m going to assume that we are going to define success around engagement. If the engagement is high (we will define metrics to measure engagement), Google Hangout is successful. If it’s low, it’s not successful.

Here is how I think Google Hangout works:

– User visits a chat page (group or with just another person)
– User clicks on “join call” to start the conversation
– Once call is finished, user hangs up by either leaving the room or hanging up

Note that Google Hangout has other functionalities such as ability to chat but given that the most popular and well known feature of Google Hangout is video call, I’m going to focus on measuring engagement of Google Hangout by focusing on video calls.

Here are a few metrics that I can think about for measuring Google Hangout:

– Given Hangout’s integration into Google Calendar, many users add Google Hangout to their calendar invite. One metric to look at is % of remote Google Calendar meetings which occur with Google Hangout.
– In addition, I can see what % of video conference based calendar events scheduled via Google Calendar are on Google Hangout. This number helps me determine popularity of Google Hangout.
– I think one indication of success of Google hangout is to determine length of call divided by length of Google Calendar invite. If the number is low, it’s an indication that people are ending the call early for reasons we need to understand.
– It would be important to know what % of Google Calendar meetings which include a Google Hangout actually do occur on Google Hangout. If the number is low, it could be an indication that the call participants run into problems and end up not having the call
– Another metric to look at would be what % of first timers using Google Hangout, come back and use it again within a period of time
– One more metric to look into would be number of minutes it takes for all the call participants to join the call. If the number is low, it’s an indication that it’s easy for the participants to determine how to join the Google Hangout call
– Another metric to consider would be number of times that someone starts and ends a call in one group. In some cases, I’ve seen this in Skype a lot, users struggle with starting a communication that everyone can hear video and audio. If this number is high, it means people are struggling with using the product
– To measure whether video conference is being used, I will look at % of Google hangout calls which include video enabled.