What A/B tests would you run to increase the number of messages sent and received on WhatsApp?

What A/B tests would you run to increase the number of messages sent and received on WhatsApp?

Questions:
Goal” To improve communications and engagement on the platform
Any target audience we want to focus on? – You Choose
Any constraints on the length of the A/B Tests? – No
App/Web? – App
Any other constraints such as cost/Tech? – No
Assumption: we are running this test only in the US – Ok
No distinction between iOS vs Android – Ok

Goal of WhatsApp is to connect the world and encourage interactions in a meaningful way and build relationships.

Users:
Power users – send and receive multiple messages through the day. Use, and are aware of all the functionalities of the product and communicate with a large set of people in various forms of media.
Moderate users – use the product on a need basis – receive more than send and are often not online for large parts of the day. Use it for necessary communications.
Rare users – These users have the app downloaded on their phones, but do not use it often. They receive messages much more than sending and are often not online for days on end as they use other forms for communication with friends and family.

For the purpose, I would like to choose the ‘Rare users’ group for the A/B tests.

Pain Points:
1. Don’t have any common interest from people in their circle
2. Fear of being judged for what they say
3. Discouraged from typing long messages as they often are busy with other things and would rather not spend time typing a lot
4. Often miss out on updates from groups as many people are sending messages – thereby unable to join in conversation
5. Stay in touch with their friends and family using other forms of communication such as Facebook/IG/Other social media apps.

To increase the engagement on WhatsApp – Let’s build a few tests that address these pain points.

Test1: Creation of interest-based groups (LA Lakers group, Fitness enthusiasts group) that users can join and participate
Control group: Does not have ability to search for and join groups
Test group: Has ability to join and participate in interest-based groups
Hypothesis – People who are able to find others with a common interest are able to better relate with each other and hence will be willing to share more on the platform.
Evaluation Metrics:
# total number of ‘interactions’ – Interactions – messages sent and received in groups
# Average numbers of messages sent and received by each user
# Average session time per user
Tradeoffs:
Users may join multiple interest groups and get overwhelmed
They may be silent spectators in these groups.

Test2: Ability for birthday reminders for friends and family (Integration with information on Facebook could be 1 way to get the information).
Control group: Does not get notifications on birthdays for friends and family.
Test group: Receives banner/push notifications about birthdays for friends and family.
Hypothesis: The reminders for birthdays will incentivize people to reach out and wish their loved ones thereby extending the possibility of conversations.
Evaluation Metrics:
# Reminders actioned upon
# Average interaction length based on reminders
Tradeoffs:
Users may have switched off notifications from WhatsApp
Users may not like additional notifications

Test3: AI based conversational system that can predict user’s response and allow for single click-based replies to messages.
The AI system reads the messages received from sender, understand the tone and nature of the conversation and be able to respond in an intelligent fashion.
Control group: Does not have Ai based reply feature
Test group: Is provided with an update on the app that has an AI based conversational system.
Hypothesis: AI based conversational system will increase the communications and interactions among the user base as it negates the users’ aversion of typing long messages.
Evaluation Metrics:
#conversations where AI based comm system was used
# corrections/deviations from predicted text response
#average conversation length post usage of AI based conversation engine
AI based replies as a percentage of overall replies in a conversation.
Tradeoffs:
Users may not upgrade WhatsApp to avail the feature
If there is not enough historical data, then the AI responses may not be useful

There are others as well that we could talk about, but in in the interest of time, I would stay with these tests.

Prioritization of A/B Tests:

Recommendation:
Based on the above evaluation criteria, I would suggest that we run an A/B test to create interest based groups to increase engagement and communication on the platform.

Clarifying questions:

Is the intent to increase # of messages sent for a particular user cohort or I can select any? (Any)

Assumption is that this is being done to increase Engagement? Is there any other objective? (Engagement is fine)

Objective: WhatsApp is focussed to build communities as is the mission of its parent Company 'Facebook'. So the objective of this experiment will be to increae user engagement.

User group : Users with medium engagement as they are aready engaged to some extent , so they need slight motivation to engage more.

Experiment : 1 

Push Notifications to remind users when there are some Unread messages.

Hypothesis: Reminders will initate user to send message and then also get replies and starting chain of communication

• Control Group: Will not get any user notifications
• Experiment Group: Will get notification when there are unread messages

In both cases, Users with medium engagement with same demograph wil be part of the experiment

Success Metric: 

1. # of messages in experiment group should increase

2. Average time spent by users in Experiment group should increase.

3. Average # of messages per user should increase.

Trade Off:

1. It is possible that users do not sent back the message, but instead call back. One reason that this group is not sending enough messages that they find composing a message hard and they prfer to talk Vs communicate via messages.

2. Notifications do disturb users and many people have a tendency to turn them off

Experiment : 2

UI changes to show a separate tab for Unread messages at the top. It will be in red font, bold and will show the count. On clicking this tab, all unread messages will be filtered.

Hypothesis: This new UI will stand out to the users and will prompt replying back to their friends.

• Control Group: Will not see any different UI
• Experiment Group: Will see this new UI

In both cases, Users with medium engagement with same demograph will be the part of the experiment

Success Metric: 

1. # of messages in experiment group should increase

2. # of Unread messages should decrease.

3. Average # of messages per user should increase.

4. Clicks on Unread message tabs should increase over time.

Trade Off:

1. It is possible that users do not sent back the message, but instead call back. One reason that this group is not sending enough messages may be because they find composing a message hard and they prefer to talk Vs communicate via messages.

2. # of unread messages may not truely reflect success of this experiment as it is possible that some messages would have been replied directly 

Experiment : 3

Auto-suggest replies

It is possibe that users do not reply much because they find typing hard. 

Hypothesis: System suggested messages will help users send quick reply from pre-composed message.

• Control Group: Will not see any auto-suggested replies
• Experiment Group: Will see auto-suggested replies

In both cases, Users with medium engagement with same demograph will be the part of the experiment

Success Metric: 

1. # of messages in experiment group should increase

2. # of Unread messages should decrease.

3. Average # of messages per user should increase.

4. # of automated messages/# of total messages should increase over time.

Trade Off:

1. It is possible that users still prefer to call Vs send auto messages if user did not get suggested message that user liked

To start with, I will go ahead with Experiment #1

​My structure for this AB testing question:

1. CLARIFY:
   1. Should we be focusing on a specific region: domestic or international? You choose. 
   2. Is there a specific user you want to focus on? Prospect v. current user, etc.? You choose. 
2. BACKGROUND: WhatsApp is a messaging app that is owned by Facebook. It allows users to send voice and text messages, make voice and video calls and share images / videos, etc. It is generally used on mobile devices but does have a desktop component as long as the user's mobile device remains connected to the Internet while they use the desktop app. It is popularly used to make international messages and is used in a variety of countries, including the US, India, Brazil, Indonesia, etc.
3. GOAL: The goal of the AB tests will be to increase the number of messages sent and received on WhatsApp. 
4. USER GROUPS: There are two types of user groups we can focus on. For these AB tests, I'd like to focus on existing users.  
   1. Prospects: Have never used Whats App. Would need to focus on bringing them into the platform to increase the overall user population / messages being sent and received.
   2. Existing Users: Current users of WhatsApp though usage may vary from low usage to power usage. 
5. BRAINSTORM AB TEST IDEAS: 
   1. Push Notification Reminders: Send push notification reminders to select group to get them to engage in WhatsApp. 
      1. Hypothesis: Users who receive push notification reminders to use WhatsApp will send more messages. 
      2. Experiment: 
         1. Control Group A: Does not receive any push notifications. Would have to be a similar group to Experiment Group B (ex. same region, age, etc.)
         2. Experimental Group B: Receives push notifications to get them to engage in the app. For example, "Don't forget to check in with your friends using WhatsApp." 
      3. Success Metrics:
         1. Average number of messages / user sent by Group B v. A
         2. Total messages sent by Group B v. A
         3. Time between next message sent and push notification (ex. did users in B send a message right after they received the push notification)
      4. Trade Offs: 
         1. Users may have turned off push notifications from WhatsApp
         2. Users may not enjoy additional "Marketing" like notifications 
   2. Personalized Recommendations: Send personalized notifications to users to remind them to check in with someone they may have lost touch with
      1. Hypothesis: Users who received personalized notifications will use WhatsApp to send more messages
      2. Experiment: 
         1. Control Group A: Does not receive any personalized notifications. Would have to be a similar group to Experiment Group B (ex. same region, age, etc.)
         2. Experimental Group B: Receives personalized notifications to get them to engage in the app with a person they recently lost touch with. For example, "You last messaged Person X 3 weeks ago. Do you want to check back in with them?"
      3. Success Metrics:
         1. Average number of messages / user sent by Group B v. A
         2. Total messages sent by Group B v. A
         3. Number of messages received by average recipient in Group A v. recommended recipients in Group B 
      4. Trade Offs:
         1. Users may deliberately not want to contact someone (ex. recent ex, friend they had a fall out with, etc.). Would need a strong algorithm to determine if a person should be a recommended recipient. 
         2. Users may not want WhatsApp "reading" their texts. Privacy concerns. 
   3. Connection Marketing Campaign: Create an email marketing campaign around maintaining connections via WhatsApp.   
      1. Hypothesis: Users who receive the marketing email will engage with WhatsApp more / send more messages. 
      2. Experiment: 
         1. Control Group A: Does not receive any email. Would have to be a similar group to Experiment Group B (ex. same region, age, etc.)
         2. Experimental Group B: Receives email encouraging user to engage with WhatsApp. Email could focus on how WhatsApp helps you maintain connection especially when everyone is virtual in COVID. 
      3. Success Metrics: 
         1. Average number of messages / user sent by Group B v. A
         2. Total messages sent by Group B v. A
         3. Clicks on email (assuming there is a link in email that links to WhatsApp)
      4. Trade Offs:
         1. Not every user will click email / open WhatsApp
         2. Emails could go to Spam. 
         3. Users may not like receiving marketing emails from WhatsApp. Could be seen as an annoyance / deleted without being read.
   4. Business Partnership: Partner with an ecommerce business to get them to use WhatsApp as a vehicle to contact their buyers for advertisements, notifications of delivery of items, etc.
      1. Hypothesis: Buyers from businesses who have the opportunity to sign up for WhatsApp notifications will receive more messages.
      2. Experiment: 
         1. Control Group A: Does not receive any option to sign up for WhatsApp notifications. Would have to be a similar group to Experiment Group B (ex. same region, age, etc.)
         2. Experimental Group B: Has the ability to opt-into WhatsApp notifications on business website and during check out. (For example, sign up for notifications of sales via WhatsApp or notifications of delivery of your parcel via WhatsApp.)
      3. Success Metrics:
         1. # of people / % of people in Experimental Group B that sign up for notifications
         2. Average number of messages received by user in Group B v. Group A
      4. Trade Offs:
         1. Experimental Group B is limited in size. Group must sign up for notifications on third party site. Less control over getting them to sign up.
         2. Much more complex test because of third party than quick AB test via WhatsApp app. 
         3. Higher cost to set up automated notifications of things like status of delivery. 
6. PRIORITIZE AB TESTS:

   1. Test	Impact to Goal	Cost to WhatsApp
      Push Notification Reminders	High	Low
      Personalized Recommendations	High	Medium
      Connection Marketing Campaign	Medium	Low
      Business Partnership	Low	High

7. RECOMMENDATION: Based on the prioritization, I would choose to run an AB test during which we send generic push notifications to users to get them to engage in WhatsApp. Given that WhatsApp already has push notifications built, running this test should be fairly low cost.  

• Free messaging platform with a large user base globally 

• Key features

  • 1:1 or group chats, voice or video calls

  • Save messages, take pictures

  • Basic profile - view last login of users

• Typical user journey

  • Sending from other platforms (e.g. google photos, dropbox)

    • User selects photo to share in google photos/other app

    • Export to whatapp

    • Select contacts to send to

  • In-app sending

    • Open app

    • Tap chat icon 

    • Search or scroll through contacts

      • Default display is by last contact

    • Send message

  • Receiving

    • Notification of message

    • Click on notification

    • Read message

• Objective: increase sent and receive messages

• Ideas for tests

  • H1: seeing contacts grouped by frequency of sending/receiving may prompt more messages

    • Test: UI to display contacts grouped by frequently sent or “it’s been awhile” categories

    • Control: Contacts grouped by last sent (as is)

    • Success: increase in # messages sent or received/user in 1 week

    • Tradeoff: Increased friction in sending / one additional tap needed, could slow users down in sending messages

    • Impact: L 

    • Cost: M

  • H2: Adding in a push notification to prompt send messages to groups/contacts

    • Test: Opts in to receive notification to prompting to send a message to a group/contact that hasn’t been contacted in awhile

    • Control: No notification option (as is)

    • Success: 

      • Increase in sending

      • Opt in rate for notifications

      • Click through rate from notifications

    • Tradeoff: too many push notifications cause user to abandon app or cause users to diable notifications which we may want to use for other functionality

    • Impact: L

    • Cost: L

  • H3: Prompting users to create specific types of groups -  family group, friend group

    • Test: Receive prompt during onboarding to create two types of groups - family and friend group

    • Control: As is

    • Success: increase in messages sent

      • # of groups created

    • Tradeoff: Potential privacy concern - labeling as a family/friends - could cause users to be wary of app

    • Impact: H

    • Cost: M

  • H4: Sharing data on typical uses “How you compare to others - e.g. typically people have 4 groups or “typical users send 5 messages a day” - inspire people to send more

    • Test: See a pop up upon login - typical users send x message/day

    • Control: As is

    • Success: increasing sending/receiving

    • Tradeoff: users who typically send more than average could reduce the  number of messages typically send as users tend to gravitate towards the norm

    • Impact: H

    • Cost: L

  • H5: Set Reminders for sending messages to certain contacts - e.g. “Set reminder to send message to x in x timeframe”

    • Test: New reminder feature

    • Control: as is

    • Success: increase sending / receiving

    • Tradeoff: putting contacts on the back burner / e.g. remind me in 3 mo may actually decrease frequency of contact

    • Impact: M

    • Cost: M

About WhatsApp:

Whatsapp is an application owned by Facebook which allows messages and VoIP (Voice over IP address to users.)

Objective:

Increase the number of messages sent and received on it.

Assumptions:

The AB tests would be specific to the mobile application without interference of any external forces.

Hypothesis:

1.       Shorting contact list by the last seen time i.e. A person who is online now will be on top irrespective of the person who was active 1 minute before.

2.       In recent updates of “STATUS”, shows the “PROFILE PIC CHANGES” as well.

3.       To forward any photo or URL from, list the “GROUPS” on top and give the option to select “ALL Group” as well after shorting the users based on the number of active users and their active time.

Experiment:

1.       Show the top five active users who were active recently.

2.       Disappear the profile pic changes status in 12 hours irrespective of 24 hours for other status.

3.       The users can forward only photo or URL in all groups simultaneously.

Criteria for success:

1.       Active users suggestion to conversation rate

2.       Profile pic suggestion to reply rate

3.       No. of photo/URL forwarded to the number of replies received on that from the other members.

Trade-off:

1.       Users might start avoiding sending messages to the person who was active a long time ago.

2.       Introvert kind of users may start avoiding changing the profile picture.

3.       Users may start doing viral fake news or photos to many people that can create disturbance in society.

 Prioritization: