Can you give us a few examples where you have planned and conducted A/B tests?

Example 1:

Situation- We wanted to plan for subscription groups for our streaming services platform. We wanted to create three tiers 1) fully AVOD 2) Freemium model for Cable customer 3) SVOD model. We wanted to compute the level of acceptance of customers in watching ads in the Freemium model

Task- We wanted to compute how much premium content should we package as part of the Freemium model beyond to determine the perceived value of the subscription cost

Action- We had categoried content into various Tiers Tier 1- Were new movies just released, Tier 5 was old legacy content. Our proposition was that customer would perceive paying additional subscription fee for Tier 2 content and upward. Since these were the existing subscribers of the cable customers, we could run multiple runs with various tiers of premium content being available beyond which they were asked to pay a fee. At least 65% of our customers were ready to pay for a subscription for Tier 2 content and upwards while they were ready to watch ads for content below it. 

Resolution- Giving away content Tier 3 would have resulted in a loss the company hence we tweaked the subscription fee to compensate for this loss

Data Collected- 

1) Video Completion rates for ads in every Tier

2) Attribution of user activation

3) Number of ad impressions

4) Ad completion %s

Example 2:

Situation- We wanted to compute what should the period of time for which ads should be supressed from customer if they have already seen the ad. We came up with various values based on video completion rates, # of exposures etc. and had determined that it should be atleast equal to 4-5 mins or the average length of any news video

Task- We decided to conduct a A/B Test specifically for traffic comint out ot NY since its the hub of news broadcasting. We created 3 Tiers 2 mins, 3 mins and 5 mins and ran experiments across these three tiers

Action- We found that video completion for ad significantly dropped off after 3 mins. There were other issues associated with latency due to client side ad integration

Result- We kept the ad supression at 2.5 secs till implementation of server side ad integration after which the ad suppression timing was still reduced.

Data collected

1) Quartile completion %

2) Number of ad impressions