15% off membership for Easter! Learn more.

Back to Questions

Late deliveries lead to customer churn. What data we should look at to prove this hypothesis for a food delivery app?

Asked at Uber over a year ago

7k views

category

category

Analytical Thinking

& 2 other companies

Analytical Thinking

Asked at

& 2 other companies

7k views

How to answer Metrics questions

Interview Guide

Silver PM

Feb 23, 2021

Late deliveries lead to customer churn. What data we should look at to prove this hypothesis for a food delivery app?

Definition of the app

I understand this product as a mobile app, which connects people who want to order food to restaurants, and delivery person. Via app user chooses the restaurant and food, sets delivery location, and makes a payment. Do I understand it right?

Clarifying questions

What do you mean by “late deliveries”? Do you have any specific time range when it’s happening?
What exactly do you mean by churn (uninstallation, stop using service)? Do you have any specific definition of churn?
Is this happening in any specific location (City, country)?
Do we see customer churn across all devices (mobile, web, iOS, and Android)
Do we see this across all versions?

From the questions above I understand that:

A cohort of late deliveries (9-12 pm) leads to customer churn (they do not make orders during next month). Our goal is to find out the data that proves this hypothesis for the delivery app (mobile, across all locations, and across all versions)

Finding the cause of the problem

Overall there are internal and external factors, but first, let’s focus on internal

There are 3 types of users involved in this process of ordering food:

Those who order food
Those who cook
Those who deliver the food

Let’s think about each of them:

Those who order food

Do they have enough of a selection of the restaurants that work late?
How much time should they wait until delivery? Is there any difference between regular time and late time?
Do we see any satisfaction decrease for late-night deliveries?
Are there any price experiments for the users for late-night deliveries?
Do we see churn for all types of restaurants?
Requests to support
User cancellation
Delivery app notifications click-through rate.
Can we extract data about Sleep mode on customers’ phones?

Those who cook

Do we have any data that shows us a significant decrease /increase in orders on their side?
How the average time of handling orders change during late-night?
How the average time of cooking change during late-night?
Do we know anything about special rates for the restaurants during late-night hours?

Those who deliver the food

Do we have any data about the availability of delivery person? ( i.e # of orders vs the number of an available delivery person in the area)
Do we see any friction in time to pick up the order from the restaurant?
Do we see any friction in time from picking up until the delivery?
Do we know anything about the transportation type of the deliveryperson at late night?
Cancellation by the delivery person (unable to find or reach the customer)
Amount of tips
Do we know anything about special rates for the delivery person during late-night hours?
Are there any experiments or known bugs on the person delivery side?

External factors we need to consider:

Do we know anything about competitors’ activity during late-night hours?
Do we know anything about transportation regulations in the area, city, or country?
Are there any natural diseases in the region?
Do we know anything about public catering regulations in the country/city of investigation?
Do we know anything about criminal risks during late-night hours in the country/city of investigation?
Is there any information about cellular networks at late night time, that may affect communication with customers/delivery person?

Identify problem:

From the answers above I could understand and identify the problem of churn and focus more specifically on the data, that we should look.

Action points:

Depending on the root cause I can come up with action items that can fix the problem of customer churn.

1 like | 0 feedback

Platinum PM

May 6, 2020

Late Deliveries lead to customer churn.

To prove or disprove this hypothesis, we have to look at the below data points:

1. No. of customers who stop ordering or are churned away vs how many late deliveries they had

2. No. of customers who increase order frequency vs no. of late deliveries they had

3. No. of customers who decrease order frequency vs no. of late deliveries they had

4. No. of customers who increase order frequency vs no. of late deliveries which were explained in chat or customer support calls

5. No. of customers who decrease order frequency vs no. of late deliveries which were explained in chat or customer support calls

6. No. of orders of our App vs No. of orders of competitor App before and after X late delieveries

7. Bubble chart: On the X axis is the delta in delivery time, on the y axis is the change in no. of orders per month and the size of the bubble denotes the number of late deliveries. Basically the hyothesis is that if the delta in the delivery time is small, the user churn would be less.

Out of this, data points number 1,3,5,6,7 are relatively more important from an analytical point of view

2 likes | 0 feedback

Platinum PM

Apr 26, 2020

'Late Deliveries lead to Customer Churn', what data we should look at to prove this hypothesis for a food delivery app

Hypothesis

Late deliveries leading to customer churn

Clarity

We need to prove this via existing data, we are looking to prove this leads to churn rather than look at causes/fix them

How do you define customer churn - assuming it mean users ordered successfully in M0 and didn't order in the M1

Why do we want to look at this data? can be manifold, maybe we are deciding if we need to invest more in supply chain/is this an important enough problem to solve

What do we define as late deliveries? does it mean we took more time than expected or the time we showed was too high? assuming it means the former ie we took more time than promised to the customer

how do you measure delivery time? assuming it means time from the moment user paid to the moment the delivery person reaches the door step

How much data do we have? I'm assuming this is a mature app and has enough data

Data analysis

This is a data deep dive to understand if delayed deliveries lead to customer churn to do so we must compare data between delayed deliveries and on time deliveries

We must clean up the data to ensure we only pick relevant data set:

1) right user set To ensure we are looking at the right user set, let's look at retained customers, ie users who've placed an order last month who are not new users ( ie their tenure on the delivery app is more than 1 month)

2) right time period: Take last 6 months data

3) Assuming the delayed order frequencies will lead to higher churn , simlarly higher delay will lead to more churn

4) We can compare Month on month average retention against frequency and delay so the graph would look something like

Customers who didn't have any delayed order vs customers who had at least 1 delayed order vs customers who had 2 delayed and so on.

Similarly we can look at retention after looking at the average delay vs MoM retention.

I would at this data ideally at a city/clusters within city level to ensure we don't mask any problems at an area level with averages. It is likely that higher delays is acceptable to users in one cluster of the city owing to its infrastructure/access vs another.

Combining the 2 we should be able to see if delays impact retention, if yes, at what points both in terms of frequency/delay amount is the problem and decide actioanble in terms how to minimise.

5 likes | 0 feedback

Gold PM

Apr 19, 2020

Before diving into the finer details of this metrics question, let’s clarify what goes into churn. Churn is the # customers lost / total # customers. In the case of a food delivery service, I’d think # customers lost means customers who have been inactive for some time (have not made an order in the last 60 days, for example), as most food delivery services tend not to have subscription plans.

To determine that late deliveries lead to customer churn, we’d need proof of causation and not just correlation. Thus, it’s good practice to first consider all the reasons customer may churn.

Potential reasons for customer churn fall into two potential categories:

-       Internal factors

o   Seasonal Factor – e.g., customers tend to churn in August, when they are traveling on vacation

o   Access to Product – e.g., major outage on web or mobile app prevent customers from placing orders

o   Product Changes – e.g., modified UI that changed the size of the order button, resulting in lower # orders placed

o   Product Quality – e.g., late deliveries, inaccurate deliveries, poor customer service

-       External Factors

o   User habits – e.g., Online food delivery is down across the industry

o   Referrers – e.g., Google / Apple made a change that makes our service harder to find

o   Competition – e.g., Competitor released new delivery service that captured our users

o   Society – e.g., People are ordering less food due to societal factor like COVID-19

To isolate the effect of late deliveries on customer churn, we will want to find a data set that is sensitized for as many of these other factors as possible.

To test the hypothesis, I would separate customers into two groups: customers who churned, and customers who were retained.

-       Among the churned customers, what % of delivery orders overall met the delivery window? I would look at these metrics over daily / weekly / monthly periods and compare against customers that did not churn over the same periods.

-       Among the churned customers, what % of the last 5 / 3 / 2 / 1 orders met the delivery window? I would compare against customers that did not churn over the same periods.

-       I would also look at % of accurate orders (overall and the last 5 / 3 / 2 / 1), as well as avg order rating scores (overall and over the last 5 / 3 / 2 / 1).

Results from this analysis will give me a better idea on whether this hypothesis holds true.

6 likes | 1 feedback

1 Feedback

Platinum PM

Apr 27, 2020

0

PM

Apr 20, 2020

Description

The service is available on both mobile and web and allows users to order food from local businesses. Users can ask special requests like taking out spices, to call and not ring the doorbell, etc easily by mark the boxes

Clarification

By late you mean running late right? And not late in night? - running late.

There is no new feature / app that has been released recently without a/b test, so we can rest assured that it didn’t impact the CR differently.

External factors

The COVID situation led everyone to order food, because you can’t get out, people want to have some decent meals, and maybe to spoil a bit, and thus there are much more orders.

Competition - there other food deliveries services that partner with restaurants, and there are restaurants that have their own logistics. They all offer more or less the same service but with a different mission. There is no less competition, vice versa, there is great business opportunity and more food delivery businesses open, and more restaurants start to manage their own delivery simply because there is enough traffic in a specific area so they hire people.

Is there any renovation in the city? New roads or new sidewalks that the municipality are building? Nope, nothing new in the city, all looks normal.

User journey:

User desired to eat sushi
Download and register to the app.
Look for a good sushi place - using stars and reviews, maybe he already knows what they want.
Optional: Read reviews
Select restaurant and go into the menu
Select a few rolls add some sauces and move to checkout
In the checkout they mark whatever special request they have (baby is a sleep - don’t ring the bell, allergic to nuts)
Select preferred payment method
Capture delivery details
Complete reservation
Get sms with a delivery estimated time and link to track status.
Users get an SMS 5 min before food arrives.

Let’s run some queries:

I’ll look into some data points over a 3 months period to better understand the trend and drill down from there.

Awareness acq acti engage retention referrals revenues

# of user registrations - positive trend

# of users who created a profile (activated their account)

% out of registrations - 80% of the users activated their account, seems that the onboarding flow is working pretty well.

# the number of orders per week in the last 3 months - to see if there are more reservations over time.. Seems that there are more and more reservations in a higher rate on the beginning of march

# of users who ordered in the last 3 months - to see if the user base is growing as expected. There is a pick at the same point in time where the number of reservations goes up..

Interestingly, although the number of new users goes up, the number of reservations is constant. This is disturbing as i’d expect to have more reservations, i guess that soon we will start to see even decrease in reservations..

Avg number of orders per users in the last 3 months - to measure retention over time.. Trend started with avg of 2.2 reservations per user in a month, went up to 4.1 and started to drop and now is around 1.7 reservations in a month..

% of users who rank & wrote reviews after ordering - 10% of the users left a review - this is very low and could impact dramatically on user engagement, trust, and lead to churn or not even starting a reservation in some cases..

% of users who ordered out of registration - 54% of the user ordered within 7 days since registration.

Seems ok, just a side note I would look into it to better understand what % i should expect here, as someone who downloaded the app probably had the intent to order food, so why didn't they? Maybe they were curious.. I’d use a survey.

% of users completed the checkout out of the users who reached this screen. - so for users who selected a dish and reach to the checkout how many completed the reservation.

Is there a credit card option? I’d assume that if there isn’t it means that an interaction is required with the delivery person and considering the situation it might turn off customers.

Side note - compare menus and prices across different food ordering platforms including the restaurant.

Avg restaurant views out of the users who completed reservations

Avg restaurant views all registered users - try to see if they can’t find what they are looking for, maybe there is some content issue, could be specific kitchen they are looking for and can’t find.

Avg time from completing the order to delivering the food - it seems that trend is going up while the time to deliver that provided to the user is still the same.

Summarize:

It seems that since COVID started there are much more people that want to order food. Which increases the avg number of reservations per user, the number of delivery people is growing but not in the same rate users order.

Time to deliver food hasn’t changed or adjusted to the situation which set wrong expectations with users. Setting the right expectations with the user will set trust and the user will not be dissapointed, taking enough time and delivering earlier might even make them happy :)

When looking on the booking flow, there is dramatic friction, there is one point in the flow, where the user need to select a restaurant which seems to be a bit lower, i’d drill down into it with surveys to better understand what % of users actually intended to order food, so to get a better sense if the churn makes sense or not..

I’d also combined usability tests to see if users are struggling with the UXI or missing capability that could help them make a decision.

Post delivery, it seems that there is a huge miss with reviews and rank which could help users make a better decision when looking for something to eat. I’d review this flow in depth as well, but since the conversion rate between search to order is 53% i’d assume that related to something else then reviews as 50% of the people order without reviews. Probably they knew what they were looking for.

0 likes | 0 feedback

Top Uber interview questions

How would you design a bicycle renting app for tourists?62 answers | 82.5k views
Design a library for the future.25 answers | 23.5k views
Drivers are dropping out of a city on Lyft. How do you figure out what's going on?23 answers | 18.8k views
See Uber PM Interview Questions

Top Metrics interview questions

How would you measure the success of Facebook Likes?40 answers | 61.3k views
What metrics would you use to measure the success of the Save feature at Facebook? 39 answers | 87.2k views
How would you measure the success of Gmail?23 answers | 17k views
See Metrics PM Interview Questions

Invite members
Invite by email

Add another