15% off membership for Easter! Learn more. Close

How would you design a test to measure the success of using a gift card promotion to get Prime members to download the Prime mobile app?

Asked at Amazon
954 views
Answers (1)
crownAccess expert answers by becoming a member

You'll get access to over 3,000 product manager interview questions and answers

Campaign Background-
Prime Video subscription/year is free with Prime Membership/year (covers Prime Video, E-commerce etc)
Not all Videos on Prime Video are available for free, some need to be bought over and above the membership fee
Prime Membership - already indicates Amazon engagement (in this case, consider e-commerce engagement)

Did only Prime Members who are not users of Prime Video - via app or website - receive the promotion?
Yes

Across Android and Apple and web Amazon.com users?
Yes

Campaign Set Up-
On shopping for $X, Prime Members get a giftcard worth $Y < $X to buy a Prime movie which isn't for free with the subscription
$X=$30
$Y=$5

Did all qualifying guests receive the same offer, i.e. min basket of $X?
Yes
*Hint: All customers will not have similar basket sizes, with the offer will be more successful for some customers more than others.
If a customer has an average basket of $10 per month, and min transaction value is $30 for promo, such a customer has low probability of redeeming the offer
If a customer has an average basket of $20 per month, and min transaction value is $30 for promo, such a customer has good probability of redeeming the offer and the campaign will result in some extra sales in e-commerce (but some transactions may be pulled ahead in one transaction, reducing transaction frequency for next few weeks)
If a customer has an average basket of $40 per month, and min transaction value is $30 for promo, such a customer has high probability of redeeming the offer, the markdown of $5 is being given away for free.

Campaign Goal
To Increase Downloads of Prime Video App among Prime Members who do not use Prime Video via app or website

User Journey
Any marketing support for the promo - where is the promotion displayed?
On homepage of amazon.com and amazon app users
Product details page unders offers applicable

Add items to cart

If eligible items worth >=$30 are added, is there a message displayed in the cart or during check out that this promo is applicable, or add $5 to avail this offer?
No
*Customers may forget to add eligible items even if interested

How does customer receive the giftcard?
On making an eligible transaction, giftcard is sent to customer via email or SMS.

Is the giftcard valid only on prime video app or prime video website too?
Only on app
Is this clearly mentioned in offer heading/main T&C
Yes

*Customer has to remember to add giftcard from email or SMS
*To do this, customer has to download Prime Video App
*Scroll through Videos which need to be bought (Is the giftcard eligible for all such videos - yes)
*Add giftcard to PrimeVideo wallet during or before the transaction
*Buy a Video using giftcard to redeem the giftcard

How long does the campaign run for?
2 Weeks

How long is the giftcard valid?
6 Months

Test Design & Metrics
A/B Test:
Choose a control group which is statistically similar to test group
Criteria - prime membership, basket size, offer viewing behaviour, offer adding behaviour, usage of other amazon products etc
Test Group receives the promotion, Control Group does not

Assuming metrics will be tracked till last giftcard validity ends (last date of campaign + 6 months)
Assuming no other event in this time frame which can influence users to download prime video app

Hypothesis
Customers who receive the giftcard are more likely to install the primevideo app

Complex Campaign Set Up
Primary Metric - Users(%) with Prime Video Installs (Minus Uninstalls) at end of last giftcard validity in Test vs Control

Secondary Metrics
Primary metric segmented by different criteria -  engagement levels like basket sizes, offer redemption behavior, email/sms open and click rate, average retail session time etc
Incremental Sales  in Retail - Test vs Control, Campaign period vs Natural Baseline Baskets

Customers who made purchases >=$30 in campaign validity period but did not add eligible items - Test vs Control
If Test = Control, the proportion will be the same
But since eligibility is a factor here, test group may have higher such customers, indicating a failure in the way the campaign is set up

% of customers who received the giftcard who downloaded Prime Video app
% of those who installed Prime Video App who redeemed the giftcard
(Downloads without redemption of giftcard is best scenario)


% of those who received giftcard -
viewed email/sms
browsed primevideo website
added giftcard


% of them who uninstalled later - are these driven by customers who did not redeem the giftcard?

Prime Video Installs  (Unique no. of Users)- Redeemer Test vs Non Redeemer Test vs Control
Gift Card Redemptions/Total Group Size - Test
% of Users with Prime Video Installs - Test vs Control
Average Time Spent on Prime Video Per User- Redeemer Test vs Non Redeemer Test vs Control
Average Minutes of Video watched per User - Redeemer Test vs Non Redeemer Test vs Control

Prioritized Metrics
Activation & Engagement around Prime Video - ultimate aim, easy to measure, prime membership indicates loyalty for retail the very least

Primary Metric - Users(%) with Prime Video Installs (Minus Uninstalls) at end of last giftcard validity in Test vs Control

Prime Video Installs  (Unique no. of Users - %)- Test vs Control
Gift Card Redemptions/Total Group Size - Test
% of customers who received the giftcard who downloaded Prime Video app
% of those who installed Prime Video App who redeemed the giftcard
(Downloads without redemption of giftcard is best scenario)

Average Time Spent on Prime Video Per User- Redeemer Test vs Non Redeemer Test vs Control
Average Minutes of Video watched per User - Redeemer Test vs Non Redeemer Test vs Control

User Journey based metrics  - easy to measure, important to iterate and improve on campaign design

Primary metric segmented by different criteria -  engagement levels like basket sizes, offer redemption behavior, email/sms open and click rate, average retail session time etc

% of those who received giftcard -
viewed email/sms
browsed primevideo website
added giftcard

Customers who made purchases >=$30 in campaign validity period but did not add eligible items - Test vs Control

% of them who uninstalled later - are these driven by customers who did not redeem the giftcard?


Finally, Incremental Sales in Retail - difficult to measure, medium confidence about accuracy due to complex campaign set up, secondary goal
 

Access expert answers by becoming a member
1 like   |  
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs
Get unlimited access for $12/month
Get access to 2,346 pm interview questions and answers to give yourself a strong edge against other candidates that are interviewing for the same position
Get access to over 238 hours of video material containing an interview prep course, recorded mock interviews by expert PMs, group practice sessions, and QAs with expert PMs
Boost your confidence in PM interviews by attending peer to peer mock interview practices, group practices, and QA sessions with expert PMs