What A/B tests will you run to increase the booking rate among Airbnb guests?

Question

What A/B tests will you run to increase the booking rate among Airbnb guests?

This was asked prior to Covid-19

Asked at Airbnb over a year ago

4.5k views

Asked at

4.5k views

How to answer AB Testing questions

Interview Guide

Top AB Testing interview questions

What A/B tests would you run to increase the number of messages sent and received on WhatsApp?11 answers | 6.1k views
What A/B tests do you suggest to make it easier to find friends on Facebook?7 answers | 3.5k views
How do you explain the P-value to a non-data person?3 answers | 2.4k views
See AB Testing PM Interview Questions

Invite members
Invite by email

Add another

Leonardo Fernandes · Answer 1 · 2022-01-10T16:10:23+0000

1) Describe the Product

Just to make sure. Airbnb is the product that enables people, travelers or guests, to rent apartments and houses of other people, owners or hosts. Guests can search on the website by using filters and an interactive map. After finding the apartment, they can book directly on the website by looking at available dates.

2) Clarification

Are we focusing on a specific segment of users? (new, existing, demographics, etc)
- You can assume.
Whey you say booking rate, you mean that guests should rent more apartments during any time range, right?
- Yes, we are talking about renting more frequently.

3) Goal

Perfect, so I understand that this objective of increasing booking rate among guests is tied to the goal of growing, thus having more revenue.

I would like to focus on existing users since they already know how the experience looks like, and we already have their trust in the platform.

A brief talk about the customer journey. People tend to travel during work holidays or long holidays, like spring and winter breaks. They tend to plan ahead of time, and then start searching for places on the internet. To decide the place, they look primarily for activities and fun places to visit. After deciding it, they start looking for hotels or Airbnb. They care about location, mobility, food, parks, cleanliness, etc. After choosing the right place, they book and pay for it.

4) Hypothesis

The prices shown on the map and listings after searching should be total instead of per night, and more highlighted.
- Hypothesis: Giving the total cost to users can help them better to decide which apartment to rent. Therefore, they would rent faster and not search anywhere else.
- Impact: Medium --> uncertain about this result
- Cost: Low --> not very difficult to change this
Show professional pictures of famous touristic spots and fun places of cities that users are searching to go to.
- Hypothesis: Showing photos of what they can experience in that city may induce them to book more.
- Impact: High --> essential information to decide where to go
- Cost: High --> hard operational work
Marketing Campaign recommending new places/experiences to users based on their past experiences.
- Hypothesis: Showing possible places that users would love to go may motivate them to book more.
- Impact: High --> Airbnb has good marketing, so it might be effective
- Cost: Low --> already have the data

Based on Impact and Cost, I would go with C), A), and B).

Now focusing only on C to design the experiment.

5) Experiment

Segmentation
- By user_id --> avoid noise and track the same user on the web and app.
- Make sure to have homogenous groups, segregated by avg number of bookings, avg $ spent, members since X period, and others.
Control Group
- People who are not receiving the Marketing Campaigns
Test Groups
- People who are receiving the Marketing Campaigns
Operation
- Campaigns will be personalized by each user based on their past data
- Campaigns will be sent by email
- Campaigns will be sent 1-3 times per week, once per day, scheduled to the time when users open the most Airbnb emails on average
Parameters of Test
- Test power: pocket rule --> 80%
- Alpha: 5% (false positive)
- Beta: 20% (false negative)
Simulation
- Simulate the test to make sure everything is working out

6) Metrics

booking rate per user_id of both groups
open and click-through rate per user_id of both groups
avg price rental per user_id of both groups
satisfaction rate per user_id of both groups
- We do not want to disturb their experience by sending emails
number of "enable emails" notifications per user_id of both groups

--> All metrics will be measured weekly/monthly, to avoid the effects of seasonality.

7) Trade-offs

Couples and groups of friends who travel together may be split into different groups, thus they might end up sharing the content.
Users might have been disabled to receive email notifications.

Pandruvada Soujanya · Answer 2 · 2021-03-31T11:55:05+0000

All feedback welcomed on my Airbnb testing question answer.

CLARIFY:
- Should we focus on a particular region? - You choose
- Is there a particular user you want me to focus on? - New User, Moderate User, Power User, etc.? - You choose
BACKGROUND: Airbnb is an online marketplace for unique homestays and experiences. The stay becomes unique because of the host. The hosts can give away their extra space for rent and build valuable connections through hosting. During the Covid times, Airbnb has expanded the platform to include virtual experiences which can either be joined individually or in a group.
OBJECTIVE: Improve the booking rate of Airbnb
USERS: There are 2 types of user groups we can focus on. I will select existing users for achieving this goal:
- New User: A user who is used to booking hotels for stays and has opened the Airbnb app for the first time
- Existing User: This user has used Airbnb before though usage may vary from low to power usage
BRAINSTORM A/B TEST IDEAS:
1. The map on the search page of Airbnb shows all the properties available for the city with price tags. The price tags can be selected to view additional details of the property. Showing up listings with Superhosts as pre-selected (in black) can increase the booking rate
  - Experiment: Experiment Group: Sees listings with super hosts on the map as pre-selected. Control Group: Sees all the listings
  - Trade-offs:
    1. The bookings with Superhosts may be more on the expensive side and this might make the users feel that the app is showing expensive listings
2. Showing the host details with a small picture and the badge on the search listings page can increase the booking rate as it increases the familiarity with the host
  - Experiment: Experiment Group: Sees thumbnail picture and badge of the host along with property attributes for each listing on the search listings page. Control Group: Sees only property attributes on the search listings page
  - Trade-offs:
    1. Unclear pictures uploaded by the hosts may make the listing unattractive for the user
3. Show nearby experiences on each of the detailed listing pages. The probability of selection of a homestay increase if the travellers find multiple nearby experiences to book
  - Experiment: Experiment Group: Sees list of nearby experiences. Control Group: Sees only property and host details on the detailed listings page
  - Trade-offs:
    1. Providing a lot of information to the user can lead to paralysis of choices.
Success metrics:
- Total bookings by the control group vs experiment group
- The average number of bookings by Control Group vs experiment group
- Time spent on the listings page
- Number of saved listings by the control group vs experiment group
- Number of clicks on listings by the control group vs experiment group
PRIORITIZE AB TESTS:

Test	Impact to Goal	Cost to Airbnb
Pre-selected listings on the map with super hosts	High	Low
Host details along with property details	Medium	Low
Nearby experiences for each listing	High	Medium

I would select showing pre-selected listings on the map to the experiment group to increase the booking rate based on the impact and cost.

DKumar · Answer 3 · 2020-05-28T22:48:03+0000

Objective - Improve the Booking Rate for AirBnB

Assumptions-

The A/B tests would be specific to the AirBnB website and app ( Not external or offsite)

The tests would be run for a substantial period to get significant statistical inference for the KPIs

Approach-

Funnel Analysis

This will help analyse at which stage of the funnel the drop-offs can be minimised.

Stages in the Funnel

Discovery : - Search- Test for relevance of search

Metrics - Differences in click distribution and search precision ( desired results/ total results) between test and control group

Consideration :- Test for different User Journeys

Refinement
Sorting
Multiple display options can be tested (Map positioning/ Attributes of listings etc.)

Metrics - Click-through rate, Average time on the page, No. of listings opened

Intent :-

Tests for Attributes of the listing ( Amenities, location, price etc.), feedback from previous guests.

Metrics - No. of reservation requests sent, No. of listings saved/shared, New sign ups/ referrals

Conversion :-

Tests for Checkout Process

Payment Methods- Split payments/ Terms & Conditions for Cancellation & Reservation

UI changes to boost recall and push for conversion

Test different ways of getting a quick response from the hosts

Loyalty scheme to encourage repeat booking

Metrics - Abandoned checkout, Conversion, Repeat booking

Vishal Shah · Answer 4 · 2021-07-19T00:16:31+0000

I assume this is for the primary and 1st product of Aibnb which is to book accomodations. (correct).

And we are trying to increase conversion rate. Here, we are saying people are looking booking acomdations but aren't booking it. (correct).

Is this from first time users or across the board? (more so from first time or less frequent users).

Is this noticed in certain regions or certain countries? (no)

Is this noticed during certain period of time (no)

Do we notice this amongst those people who spend less than x $ per night? (no. Actually this is noticed more amongst those who book considerably high rent per night compared to most others).

Ok...so I think last thing you stated leads me to believe that we aren't able to interest affording customers enough to lead to conversion. these set are very picky so they are looking for more than what we offer. We can assume here that these very affording customers typically are likely to cancel at last minute and strict cancelation policies could be refraining them from conversion. Also likely that they are specific about areas, cleanliness, safety and amenities. We can liekly address some of these.

1. We could provide a live view from each guest of the acomodation area in the last month of stay to provide more comfort of cleanliness.

2. Provide a conceirge type service by enabling these affording customers to personally talk to the host over video chat / phone call. This will enable our customers to be able to ask as many questions as possible to truly feel positive about the accomodation.

3. Easy cancelation policy by providing greater cancelation windows and reduced cancelation fees.

#3 can have huge business impact so this may need to be thought through. this also depends on how expensive we are talking about. So for now let's think of #1 since that's easy to implement and is low cost. #2 is great but AirBnb already has ample infrmation fields so I doubt talking to somenoe on the phone will provide additional value leading to conversion.

May I go ahead and strategize how we may A/B test this? (sure).

#1...I would setup a control group and experimental group both containing random set of users who generally spend prob more than $130/night on accomodation. I say $130 arbitrarily but overall hotels in good cities are generally $150/night + and if someone were to come to AirBnB it's mostly for cheaper acomodation so here $130/night is likely still pricey. This could be wrong assumption but I don't have data to formulate what the right $ amount should be. I expect that experiemntal group that can see videos of live conditions of the place by people hwo stayed in the last month, will convince them more and provide them more comfort of cleanliness and comfort of the place since reviews are subjective but what the eye can see is very real. One may say in reviews or in ratings as 3/5 but 3 is subjective. I would see if experiemntal group has had increase in % of conversions. % would be those who saw the listing how many % of those converted to bookings.

There will liekly be issues here in the sense that not everyone is as detailed in taking videos. So may not take all parts of the place in videos and some may not have the right lighting making it hard to get a good sense. Also, many may not care so that could pose significant dnager to this test. But I think it's worth to try and maybe we can incentivize people to take videos as well.

Any questions?

(No we are good. Thank you).

SS@1 · Answer 5 · 2021-01-08T03:08:12+0000

Me: AirBnB is a product where users can lend their space for fixed period of time to other users in exchange of money

I would like to walk through the user journye through booking process

Interviewer: Ok

me:

1. User logs into Airbnb

2. User searches for Destination and dates. then available spaces are displayed

3. User Filters to results based on criteria like number of bedrooms, number of beds, check-in time, free cancellation etc

4. User contacts the hosts to understand more details about the place

5. User gets preapproved by host and books the space

and the goal of the AB test is to increase the booking rate. I would like to suggest the following hypothesis and experiments

1. Hypothesis 1: Allowing hosts to upload better quality images of bigger size will allow hosts to market their place better

Experiment -

A listing with high quality HD or higher picture gets better booking rate compared to listing with pictures shot on phone

2. Hypothesis 2: A listing with creative things to do while in the space will increase marketing of the space better

Experiment: Add Things to Do around as filter criteria in search and filter results based on the criteria

Experiment: Convert "things to do" into metadata for a listing

Metrics:

% increase in bookings per user

% increase in click through rate per listing

Avg time lapsed between posting and booking per day

Tradeoffs:

I will choose the first hypothesis based on RICE score as it is easier to implement.