How would you design a test to measure the success of using a gift card promotion to get Prime members to download the Prime mobile app?
You'll get access to over 3,000 product manager interview questions and answers
Recommended by over 100k members
Campaign Background-
Prime Video subscription/year is free with Prime Membership/year (covers Prime Video, E-commerce etc)
Not all Videos on Prime Video are available for free, some need to be bought over and above the membership fee
Prime Membership - already indicates Amazon engagement (in this case, consider e-commerce engagement)
Did only Prime Members who are not users of Prime Video - via app or website - receive the promotion?
Yes
Across Android and Apple and web Amazon.com users?
Yes
Campaign Set Up-
On shopping for $X, Prime Members get a giftcard worth $Y < $X to buy a Prime movie which isn't for free with the subscription
$X=$30
$Y=$5
Did all qualifying guests receive the same offer, i.e. min basket of $X?
Yes
*Hint: All customers will not have similar basket sizes, with the offer will be more successful for some customers more than others.
If a customer has an average basket of $10 per month, and min transaction value is $30 for promo, such a customer has low probability of redeeming the offer
If a customer has an average basket of $20 per month, and min transaction value is $30 for promo, such a customer has good probability of redeeming the offer and the campaign will result in some extra sales in e-commerce (but some transactions may be pulled ahead in one transaction, reducing transaction frequency for next few weeks)
If a customer has an average basket of $40 per month, and min transaction value is $30 for promo, such a customer has high probability of redeeming the offer, the markdown of $5 is being given away for free.
Campaign Goal
To Increase Downloads of Prime Video App among Prime Members who do not use Prime Video via app or website
User Journey
Any marketing support for the promo - where is the promotion displayed?
On homepage of amazon.com and amazon app users
Product details page unders offers applicable
Add items to cart
If eligible items worth >=$30 are added, is there a message displayed in the cart or during check out that this promo is applicable, or add $5 to avail this offer?
No
*Customers may forget to add eligible items even if interested
How does customer receive the giftcard?
On making an eligible transaction, giftcard is sent to customer via email or SMS.
Is the giftcard valid only on prime video app or prime video website too?
Only on app
Is this clearly mentioned in offer heading/main T&C
Yes
*Customer has to remember to add giftcard from email or SMS
*To do this, customer has to download Prime Video App
*Scroll through Videos which need to be bought (Is the giftcard eligible for all such videos - yes)
*Add giftcard to PrimeVideo wallet during or before the transaction
*Buy a Video using giftcard to redeem the giftcard
How long does the campaign run for?
2 Weeks
How long is the giftcard valid?
6 Months
Test Design & Metrics
A/B Test:
Choose a control group which is statistically similar to test group
Criteria - prime membership, basket size, offer viewing behaviour, offer adding behaviour, usage of other amazon products etc
Test Group receives the promotion, Control Group does not
Assuming metrics will be tracked till last giftcard validity ends (last date of campaign + 6 months)
Assuming no other event in this time frame which can influence users to download prime video app
Hypothesis
Customers who receive the giftcard are more likely to install the primevideo app
Complex Campaign Set Up
Primary Metric - Users(%) with Prime Video Installs (Minus Uninstalls) at end of last giftcard validity in Test vs Control
Secondary Metrics
Primary metric segmented by different criteria - engagement levels like basket sizes, offer redemption behavior, email/sms open and click rate, average retail session time etc
Incremental Sales in Retail - Test vs Control, Campaign period vs Natural Baseline Baskets
Customers who made purchases >=$30 in campaign validity period but did not add eligible items - Test vs Control
If Test = Control, the proportion will be the same
But since eligibility is a factor here, test group may have higher such customers, indicating a failure in the way the campaign is set up
% of customers who received the giftcard who downloaded Prime Video app
% of those who installed Prime Video App who redeemed the giftcard
(Downloads without redemption of giftcard is best scenario)
% of those who received giftcard -
viewed email/sms
browsed primevideo website
added giftcard
% of them who uninstalled later - are these driven by customers who did not redeem the giftcard?
Prime Video Installs (Unique no. of Users)- Redeemer Test vs Non Redeemer Test vs Control
Gift Card Redemptions/Total Group Size - Test
% of Users with Prime Video Installs - Test vs Control
Average Time Spent on Prime Video Per User- Redeemer Test vs Non Redeemer Test vs Control
Average Minutes of Video watched per User - Redeemer Test vs Non Redeemer Test vs Control
Prioritized Metrics
Activation & Engagement around Prime Video - ultimate aim, easy to measure, prime membership indicates loyalty for retail the very least
Primary Metric - Users(%) with Prime Video Installs (Minus Uninstalls) at end of last giftcard validity in Test vs Control
Prime Video Installs (Unique no. of Users - %)- Test vs Control
Gift Card Redemptions/Total Group Size - Test
% of customers who received the giftcard who downloaded Prime Video app
% of those who installed Prime Video App who redeemed the giftcard
(Downloads without redemption of giftcard is best scenario)
Average Time Spent on Prime Video Per User- Redeemer Test vs Non Redeemer Test vs Control
Average Minutes of Video watched per User - Redeemer Test vs Non Redeemer Test vs Control
User Journey based metrics - easy to measure, important to iterate and improve on campaign design
Primary metric segmented by different criteria - engagement levels like basket sizes, offer redemption behavior, email/sms open and click rate, average retail session time etc
% of those who received giftcard -
viewed email/sms
browsed primevideo website
added giftcard
Customers who made purchases >=$30 in campaign validity period but did not add eligible items - Test vs Control
% of them who uninstalled later - are these driven by customers who did not redeem the giftcard?
Finally, Incremental Sales in Retail - difficult to measure, medium confidence about accuracy due to complex campaign set up, secondary goal
Top Amazon interview questions
- A metric for a video streaming service dropped by 80%. What do you do?50 answers | 135k views
- Design a library for the future.25 answers | 23.5k views
- Your new feature boosts Amazon Search by 10%, adds 2s to load time. What do you do?19 answers | 36k views
- See Amazon PM Interview Questions
Top Product Design interview questions
- How would you design a bicycle renting app for tourists?62 answers | 82.5k views
- Build a product to buy and sell antiques.54 answers | 66.8k views
- How would you design a web search engine for children below 14 years old?36 answers | 42.9k views
- See Product Design PM Interview Questions
Top Amazon interview questions
- How would you improve Amazon?14 answers | 35k views
- How would you determine if a specific block in your neighborhood is suitable for a new grocery store?14 answers | 13.4k views
- You have a grocery delivery service that delivers food within 24 hours. How many trucks would you need to operate it?12 answers | 29.8k views
- See Amazon PM Interview Questions
Top Product Design interview questions
- How would you design a consumer application for a scooter sharing business?21 answers | 18.6k views
- Build a product to solve the dog poop problem.13 answers | 9.4k views
- How would you design a "Google Refrigerator"?13 answers | 8.3k views
- See Product Design PM Interview Questions