You are the PM for a B2C product that has an advertisement-based monetization model with significant and steady daily revenues. One day, there are no ads served and the revenues plummet to zero. What would be your strategy, as a Product Manager, to deal with this crisis?
22k views
-icon.svg){kind=icon}
22k views
22k views
Access expert answers by becoming a member
You'll get access to over 3,000 product manager interview questions and answers
Recommended by over 100k members
Bronze PM
Assumption - The product is an advertising platform where a seller can publish their ads and they are directly presented to consumers. Monetization for advertising platforms is usually based on:
- Impressions - number of ads served
- Clicks - Users actually clicking on the ad
- Actions - Users performing certain actions such as signup or add to cart after clicking on the ad.
- Daily average advertisements published on platform.
- DAU for publishers
- New code deployed to the publishing platform
- Infrastructure health issues - broken servers or provider outages.
- Changes to our pricing model.
- Competitors offering a better rate.
- New regulations that impede advertiser or our activity in a particular country
- Economic factors impacting advertisers that is resulting in them scaling back such as drop in revenue.
- Major acquisitions in primary customer segment in publishers that may result in consolidating platforms and us losing business.
- Negative new about our platform that is causing advertisers to pause doing business with us.
- Look at DAU of consumers on our platform
- Look at daily average ads served
- Outage on the platform preventing users from either logging in or seeing advertisements.
- Issues with our ad targeting system
- New code/feature deployments.
- Provider/infrastructure outages bigger than our platform.
- Consumers migrating to a different platform due to a better experience.
- Consumers boycotting the platform or advertiser due to negative news.
- Government regulations restricting user access to our platform.
Silver PM
B2C company , Advertisement based monetization - Daily revenue
Revenue went to zero becasue no ads were served , what will you do as a PM?
1. Understand more about the problem
- Advertisement revenue is based on no. of impressions as well as no of engagement.
So here has there been no ads shown? or no customers to whom ad can be shown? You decide
- Operational worldwide ? Yes
- How user use the product primarily? through mobile app
2 . To examine the problem
Investigation order -
- Check if the advertisers have stopped the ad
- Any recent changes in the ad sales stategies Eg. Increased price
- Are advertisers boycotting our platform
- Check whether the ads are getting deployed
- Have we changed / released any feature related to this ?
- Have we done changes in the advertisers platform?
- Check if users are able to see the ads
- Are user able to access the application?
- Have there been any bug due to which they are not able to sign up/ login
- Has there been any external event due to which user might not be able to access
- Are user seeing the ads?
- Have the mobile company have changed theoir advertising policy
- Have user started using adblock softwares
- Are user able to access the application?
- Check if we are calculating the revenue corectly
- Has there been error in the calculation of revenue - is the analytics tool working fine
- Is the data flowing correctly?
PM
To start I would want to clarify what it is meant when they say "no Ads were served". Was it:
- Visibility issue - no ads were shown to users on the platform, hence revenue is zero
- Engagement issue - ads were shown but users did not engage with them at all, hence revenue is zero
Assumming visibility issue:
Since the problem states that one day no ads were served, it's safe to make following presumptions:
- This is likely not a specific platform issue (iOS / Android / Web). A platform specific issue would mean that we would observe a reduction of a sizeable chunk and not a complete halt
- This is likely not an app version issue since app versions don't have a clean switch, and it's always follows a gradual rollout and adoption cycle. An issue stemming from it would be observed over time, and wouldn't be sudden
- Internal factors
- Tech issues
- Operational issues
- Business issues
- External factors
- Competitors market entry
- Market reputation issues
- Regulatory issues
We can start by querying sample responses from a randomly selected user cohort
- Is BE response flowing in? If there are issues here then,
- we check the BE systems where the Ads are configured to check if the config parameters are in compliance with the built system parameters
- Check segmentation engines to clarify that it is not false negatively disqualifying all users to become ineligible for the campaign
- If any BE deployments were done around the time of deployment which may have affected the BE response
- Check for BE response timeout issues, and BE server down times
- Is FE serving an issues? If FE is unable to serve despite BE response flowing in then,
- Check for buggy code bits related to the ads serving unit specifically used in the app
Operational issues:
- Check for ad image assets - perhaps some new campaigns were configured which were not compliant to the designed system parameters, leading to a breakdown
- CTA deeplinks - if user engagement is an issue, perhaps incorrect or broken deeplinks are preventing users from being able to interact
- Incorrect configuration on app versions that are not live yet, or assets which are not yet released to production builds
- Are we able to source sufficient ads to display?
- Has there been some pricing change leading to drop in ads supply
- Competitors market entry - New competitors entry in the market who have similar serving base, but discounted pricing?
- Market reputation issues -
- Any developments with respect to reputation that could have led to public boycott, and therefore advertisers dropping out?
- Also check DAU numbers to verify this hypothesis, as DAU numbers would also reflect a similar pattern
- Regulatory issues -
- Any policy changes preventing the platform from engaging with the current supply of advertisers?
- Policy changes preventing the platform from advertising altogether?
Clarifying Questions:
Revenue streams for the company? - Subscriptions and Ads.
Which space is the company in? - Entertainment
Timeline we are looking at resolving this? - Assuming within the next 24 hours.
How long have we known this issue for - 3 hours.
What ad formats are we looking at - Video and display - across all.
How much revenue loss are we looking at? - 10k USD/day
Are ad calls itself blocked or are we not getting responses to ads? - Could be External issue
If we are getting responses we are not able to show ads → Most probably internal issues.
Platform : Issue on any specific platform.
Traffic: Health of traffic at client side.
Advertiser Ecosystem
External: Advertisers
External: Ad Platform
Internal : Technical and Sytem level changes.
Internal/System level Changes
Did we make any rollouts recently?
Any team made changes that can impact ads? Any experimentation which could have blocked ad calls itself.
Any migration we had done for ad servers or calls?
B/E health is good. Is there any flag which was turned on/off which could have done this.
Is the client sending a request to the server? -If we are getting back responses, are we not able to render – App level issue.
Are we recording metrics correct internally?
Are we able to see ads on the platform
External Changes:
Any policy change/ violation notice we have received.
Any compliance change from the ad platform side.
Any migration we had done.
Any major deprecation of service we could have missed?
Is the server responding back to calls from the App.
Is reporting an issue
If revenue dropped to 0 most probably it is not a specific demand partner issue.
Debugging:
Look at ad funnel reporting and check where the drop is : is it request, responses or impressions.
If request and responses are good, but impressions have dropped → it can be an issue at client side
If requests are good and responses have dried up → it could be a compliance issue, server/handshake issue.
If requests itself are not being sent → Client/app issue.
If all metrics are looking good (internally) and nothing is being reported on the ad platform → Reporting issue (Callbacks are not being sent) → could be a downtime issue from the ad provider (Google/meta/Amazon in this case)
Silver PM
Context setting: I understand said business has a vision to connect people with the products they need to XYZ. Customer acquisition is a cornerstone of the business model with Ym customers and growing. Ads are a critical enablement of this strategy so this crisis needs to be dealt with swiftly.
Clarifications
1. What geography? NA
2. mobile or web? both
3. At what point is this caught, was there an alert? mid day
4. Was there a taper off or a cliff? cliff
5. what portion of marketing revenue is dependent on the ad server? most
6. were not ads served or we see no clicks so assume no ads served? no clicks
One day suddenly and without a taper no ads are registered as clicked on. As a pm I need to act quickly to identify the root cause and implement a strategy to address.
The approach needs to be swift and effective. The structure is as follows
1. perform a root cause analysis
2. implement a fix based on root cause
3. if the fix is complex and will take too long (TBD) then a work around for stop gap is needed
Root cause analysis
assume we have a chain of activities like: content creation -> content loading -> web server personalization engine -> serving -> pull -> placement -> feedback through clicks
Internal
1.content creation -> loading outage
if this is batch daily then maybe content did not load
is 'old' content removed daily? if not there should still be content loaded
2.content loading -> web server engine outage
assuming content is loaded the engine is either out or unable to perform a personalization algo
there could be an error in the content format so the engine can't process
if so then the content loading ETL needs to be investigated
3.web server engine -> serving outage
assuming the engine has a personalization also running effectively but unable to serve then the serving infra may have an outage
if serving infra working maybe the format of the person also is flawed
external
1.serving -> pull outage
assuming the serving engine is working then the pull is unable to complete task so the pull may be not happening or is happening but defective
or the serve is corrupting the file so the pull is unable to recognize it
2.pull -> placement outage
if the pull is working effectively the placement also is not working
or the pull is working but corrupting to content so unrecognizable
3.placement -> feedback through clicks outage
maybe the placement is working but not one click happens so we believed ad serving was broken when bin fact it's the measurement
Next I'd assume that the problem is in fact one of the above. I would do an effort analysis and assume that any internal issues will be within our control while external are not and may or may not be straightforward to fix. There are there broad types of issues above
1. an engine or piece of infratsructure/tech has an outage
2. a file, data, content etc is corrupt
3. a process such as serving, etc, loading is failing
Note these are not mutually exclusive and any combo can be happening at the same time.
I would generally rank
1. Tech/infra outages as M-H given the length of time for added root cause analysis, uncertainty, unpredictability of hardware fails and potential reliance on third party ie. AWS/Google/MSF etc.
2. Corruption of files as L - M as even if the fix is complex and involves new pipelines, there are work arounds available such as manual loading of content, side loading etc
3. A process is also L - M for similar reasons as corruption
If the root cause reveals that indeed a tech or infra problem is at play, then I would immediately look into a work around such as working with Ad and search partners for contracted serving.
If this is not at play I would assess the level of effort needed to fix the process or corruption and if more than one business day (assumption since ad server tied to most marketing revenue) I would stand up a SWAT pod for manual serving until the fix is complete.
Bronze PM
a. What are the product goals
b. What are the User Personas : End Users, Advertisers
c. What is the User journey of end users who view these Advertisements - User Goals
c. What is the monetization model : CPC, CPV
Since Daily Revenues have dropped to 0, we can assume that Advertisements are not being shown across all users and all platforms
I will first list down probable causes across below categories
a. Product :
Advertisements are not being seen by Users
Has there been significant drop in Usage
Lets say User Goals related to Ad are :
a. Views an Advertisement
b. Skip Ad
c. Clicks on Ad
Are Ad campaigns created by Advertisers enabled and running or are they getting disabled
b. Business :
Has there been any increase in minimum CPM, CPV recently
Any new policy change to approve Ad campaigns which has increased Review time for Campaigns
Any negative publicity of the product in Social media due to which users have uninstalled the App
c. Users:
Has there been drop in Daily Active Users
Has there been increase in uninstallations of app
Are Advertisers switching to competitor platforms
d. Environment
Has Govt imposed any ban on the App usage
Lets say we come to know there has been a significant drop in Daily Active Users
We will first define User journey of an End User
Launch App - User Login -> Views video, listens to Audio -> Ad is streamed - Skips Ad/ Clicks on Ad
We need to check has there been any drop-off in these touchpoints
Is User able to launch App or it is getting failed on Launch
Is App failing when User clicks on any suggested video or any song
Is Ad streaming API working or not - is API serving ads during the videos
Are users able to skip Ad - If Skip Ad button is not working then it can lead to negative User Experience
Are users getting redirected after clicking on CTA button on Advertisement
Based on answers of above question we can then identify the root cause and check with the development team on the issue that has been identified
The fix needs to be introduced as a Hot fix if its a small bug or the team can do A/B Testing by doing a soft launch of fix on 5% of Active Users
Metrics to Track for the Fix:
Increase in Daily Revenue
Daily Active Users
Gold PM
Clarifying questions:
Is it a mobile based or web based product?
Are we catering to particular geography via this B2C product?
Is this problem with us only or a category problem? Are others also facing the same issue in this category or is it just us?
Are we experiencing decline over a period of time or it just dropped abruptly?
Assumptions:
I am assuming this product to be available across both the platform- web as well as mobile
I am considering accessibility of this product is not constrained to any particular geographic region.
Given that I am a PM, I would have noticed this thing earlier if there would have been a decline over a period of time- so considering it a abrupt decline to zero.
Also assuming it to be the problem which is concerned with out B2C product not all
Advertisement ecosystem are 3 sided
Consumers/ users - (client traffic)- Demand side
Advertisers- Demand side
platform- that is system
Firstly to handle this crisis I will look at the reason behind this situation for each and then would go for solutions to deal with this crisis.
Demand side
Fall in traffic abruptly which could be due to following reasons
Change in Gov policy which restricts using this app
A Negative PR for our product
Maybe traffic for our product depends on traffic from external partners for example fb. And there is some issue with due to algorithm change or some ther issue on fb’s part
May be recent product releases which logged out users
Another reason could be for our product - only free users get to see the advertisement. And because of some promotional scheme - significant users signed up for the subscription, leaving behind a very small number to see advertisements.
Supply side :
We are dealing with 2 or 3 large advertisers and that too from same segment and for some reason they have backed out which could be due to following reasons
They might have decided to end their campaigns as part of their advertisement strategy for eg- Particularly this segment has decided to shift completely towards traditional way of advertising
There can be some change in Government policy which prohibits this segment to advertise their products or services
As per the contract - the campaigns have come to an end, and since they all belong to the same segment. There can be a possibility that all of them have the same date campaign expiry.
There can be a possibility that they have willingly decided to back out because of
Pricing increase
Policy change
Key features inadvertently dropped
Entry of some new player in this category with better incentives for them which requires them to collaborate with it on immediate basis to avail the offer
System
Release/ software changes which caused ad system to be unavailable to advertisers
Release/ software changes which caused ads to be not visible to users
Solutions to solve above mentioned problems
Systems
Check with engineering teams and find out the root cause - and also come up with action plan with scheduled timings to solve this
Advertisers
Work with sales team to check who and why cancelled
Also work with Business development team to rectify the issue if its due to- change in policy ( see why we did make a change in policy and how can we tweak it fit in advertisement requirements), price increase ( see what is market price being charged and try aligning our prices to market price), or key feature being dropped( Do the AB testing again by bringing back the key feature and analyze the response)
If the issue is that since we have only 2-3 big advertisers who decided to back out for some reasons which are not in our control- Try onboarding new advertisers and this time ensures as our long term strategy that we don't have all from same segment
Check with the campaign status and renew the contract
Clients
-If its temporary because of technical glitch then wait for some hours and get in touch with engineering team and fix the issue on priority basis
- If it's due to negative PR policy- First thing is to get in touch with PR team and try communicating thing
Okay.so the revenues have dropped to zero as there were no ads served.
First of all, i will go about investigating the reasons for the same.
Why did it suddenly become zero? Was there a trend before this?
Did something in the tech break? Can we fix it on a war footing?
If it was not a tech issue, i will investigate the demand side and the supply side.
Are we experiencing a decline in the number of users of clicks or actions ( depending upon our revenue model)?(Demand Side)
- Are we targetting the correct keywords for the Ads?
- Are we not getting enough new or repeat users and thus advertisers are not getting the value for their advertising dollars?
Are we experiencing a decline in the advertisers? ( Supply Side)
- Are we dependent on a few large suppliers?
- What is our continuity plan?
- A Freemium Model: Free users get to see Ads and get lesser features. Paying users get ad free experience and all the features
- Subscripton Model: I will also give an option to pay up front fees or pay as a monthly/quarterly subscriber
- Tiered/ A La Carte Pricing: I will experiment with a tiered or an a la carte pricing model, depending upon the feasibility to my product specs
Liked by PM Exercises
Gold PM
My answer to this product strategy question:
Clarification
No ads served - was there an abrupt drop (sharp cut off in a few hours) or did it taper off over a period of time (say 1 month) and finally there were no ads that were served one fine day
Was there any geographic focus of the product? Did this drop happen specific to a geographic region?
Is it a web based product? Or Mobile product? Or both?
Assumptions
This was a sharp drop (as otherwise as PM i should have noticed the steady decline)
This was an overall drop
This is both web and mobile environment.
Advertising ecosystems are 2 sided
Client traffic (demand)
Advertisers (supply)
System
Root cause analysis
Client
Was there any sharp drop in client traffic on one fine day? If yes, need to explore why
Was there any significant event externally for e.g. US presidential elections which tuned users away from the app or blacklisting the product by Gov etc.
Was there a recent product release for e.g. login changes which locked out a large no of users
Did the product depend on traffic from external partners significantly say FB and there was a drop due to algo change or some other issue on their part.
Advertisers
Analyze if the campaigns inadvertently ended on the one fine day
This could be similar to Y2K problem
If bulk of the ads are from Third party Supply Side platforms, check if there are issues from their end (blacklisted , contract changes, technical changes etc)
Analyze if bulk of the revenue came from 1 or 2 specific segments of advertisers which were affected by environmental changes,
Analyze if there are external factors which lead to advertiser cancellation
Pricing increases
Policy changes
Key features inadvertently dropped
System Issues
Release which caused ad systems to be unavailable
Releases which caused key DBs to be unavailable
Release/ Software changes which caused ads to no longer called from client
Release/ software changes which caused key attributes to not send (which causes ads to be not matched)
Considering the above scenarios, based on the root cause analysis the following actions can be taken
Client
Traffic drop due to login or release issues
Hot release to fix it (considering mobile environment, cannot roll back if there have already been installs)
External issues:
If the product is left redundant due to external factors, check what other services/ content/ features which can be provided
If it is a temporary drop,then wait it out for a couple of hours to see if traffic comes back
Advertisers
Check for campaign statuses with last updates made (to see if advertisers ended the campaigns)
Work with BD teams to resolve issue with ad networks/ products
Work with Sales to reach out to the big customers to see who & why cancelled?
System
Work with engineering teams to identify any technical issues
Before I start answering the question, I’d like to list out some clarifying questions I’d ask the interviewer if there was one:
Did this happen gradually or it was within one day?
What is the overall mission of the company’s product? What is its goal?
For whom is the product for?
Are there any ongoing business objectives that I should know about? (It might influence my strategy)
I am assuming that the product is strictly dependent on advertisements. My second assumption is that the “no ads” served event did not happen on purpose (as part of some experiments like if the company was trying out different monetization strategies) given the “crisis” word. I would have know about it as well. Given that I don’t know the above clarifying questions ( mission of company, ongoing business objectives etc), I will strategize on a way forward.
The information we know is that (with some assumptions):
This is a B2C product. Our users are direct consumers.
The product is dependent only on ads for revenue
The loss of ads has happened to all of our customers segments on all platforms for our product
Urgency is key, we need to solve this crisis ASAP
I’m assuming that since this product is dependent on ads for revenue, this product lives on a site, social media, TV - anywhere where we would also be able to show ads. I am disregarding offline products like a law firm that has their ads on billboards because it’s hard to believe that in one day all the law firm ads would be taken down.
Urgency is top of mind since lack of revenue will hurt the company and since the company relies on the steady and significant revenues on a daily basis. We need to solve this problem ASAP. Before root causing the issue to get fixed, I'd do two things:
One, start a communication thread to keep stakeholders informed about the issue and our progress in resolving it, ensuring transparency and managing expectations. I'll update them twice a day via email and messaging apps.
Second, I’d contact the appropriate teams to start the initiative of conveying a message to our customers using the product. Ex. “We are experiencing technical issues and working hard to get it fixed soon.”
Third, I’d create a temporary contingency plan to mitigate revenue loss and then act on it. It would look like this:
Talk to engineering team to see if any current major bugs happening or recent deployments could have caused this crisis
Talk to ad provider
Talk to other teams who might have implemented an initiative I wasn't aware of
The first thing I’d do is a quick root cause analysis of what could have created this issue. I’d investigate what could have happened internal to my team, internal to my company and externally as well. I’d start with internal. I’d first talk to engineering team to understand if there have been any bugs in being able to show ads on the platform for our product/service. And also confirm if the issue is indeed to all our customer segments and on all platforms. I'd work with them on fixing these issues if this is the root cause. Next, I’d look to our internal ads partnerships team. Externally, if our ads were provided by an ad provider, I’d ask if there have been any issues with the provider (I’d contact our partnerships team directly). An issue with contract changes, etc. I’d remedy this issue if it’s the root cause.
I’d also question if another team in the company decided to implement new initiatives to the product. One of them could be a very easy option to disable ads on the platform. If this was the case, although the intention might have been innocent, they didn’t consider the negative consequences it could have on us. If this was the case, I’d show this team the data of what has happened and work with them to retire this feature of ad-blocking. I’d talk to the team and let them know we can work together on that ad-blocking feature with more strategy.
After the crisis, I'd work on implementing monitoring systems to detect similar issues early and explore diversifying our revenue streams to reduce dependency on a single revenue source. If the third reason was the right one, I'd maintain a better and closer relationship with this team.
This would by my approach to solve this crisis.
Silver PM
Context Setting : There are multiple products in B2C space and works purely based on Ad based monetisation model, eg : Google, Youtube etc. Google is a platform of many products and for the sake of this assignment, I will pick Google Search as the B2C product.
Problem Statement : One day, there are no ads served and the revenues plummet to zero, what is the strategy and how to deal with this crisis.
Understand the root cause : Is this happening due to an issue that the Tech team released a change that resulted in no Ads served? Was there any change in Sales channel or Marketing Campaigns or Distribution Channel? Is this Happening in a specific Geographical region? Is this sudden drop? Seasonality changes? Competitor changes or Market changes?
Assumptions : This has happened as a sudden change and no internal changes occurred nor external factors influenced. As the monetisation model is purely based on Ads served and may be PPC, there hasn’t been revenue generated for that specific day.
User Segment : B2C with a wide range of user segments based on Geographical, Demographics, Behavioural or Psychographic changes from urban and rural. For this reason, I am assuming and picking US as the geographical region.
Current State : Monetisation model is purely based on Ads served and may be PPC, there hasn’t been revenue generated for that specific day.
Brainstorm the ideas : Short term goal and Long Term
Short Term
Solution 1:Have a conversation with Marketing teams and see if Ads can be served back .
Solution 2 : SEO ranking - Business Domains which have the most impact and reach based on SEO ranking, have a conversation with the businesses to say that if they would like to stay at the top of the Search, then on a daily basis, they need to pay a X cent per click on their link through organic search . Prioritise the business based on SEO ranking.
Solution 3 : B2C users - If the search exceeds a certain limit of data, then they need to subscribe to a plan or wait till next day to continue with Search. Long term goal and to be prioritised.
Solution 4 : To see if there is a possibility to increase the plan price for adjacent products in the portfolio, Eg : Google Workspace. Medium to Long term goal and to be prioritised.
Prioritisation : Depending on the principles of the product, between information creators and consumers, consumers take a precedence and I don’t want to take a hit by asking right away for consumers to pay and use, therefore I would strategise to focus on business domains.
Solutions 1 (Deep Dive) : SEO teams to prioritise the business based on the SEO ranking and market data of the company to have conversation rolling. Conversation with the Business and get the agreement in place and the lay of the monetisation model for Google Search.
Tech changes :
If Ads are not served, the UI/UX rendition has no impact. Changes to pricing engine on capturing the payment for per click. Given the monopoly, I don’t see the change needs to be PR’d for consumers. Basically no change to experience and therefore doesn’t require external PR announcement.
Delivery : Alignment with the teams and changes to be rolled out.
Gold PM
Alright, starting with a few clarifications here.
- What is this B2C product? (Ans: Your call)
- Ok, I'll assume it's a social media or content based website, where users come to engage and I serve ads. This is a typical business model.
- It's a strange phenomena, no ads being served today, has this happened before? (No)
- Did we observe any trend with respect to engagement prior to this? (Answer: No)
- Has this happened on a particular geography or across geography? (Ans: Across)
- Is there a possible data issue? (Ans: No)
- Was there a change in instrumentation? (Ans: No)
- User logs in (Returning user are logged in)
- User checks the feeds
- User engages with the content (Read, Like, Comment)
- User closes the session
- Is there a change in the number of users who are logged in?
- Is there a change in the behavior of returning users? Are they spending less amount of time?
- If the engagement with posts have decreased
- If no. of posts created have declined
- Check: If the no. of sessions per user has declined
- Did we release some feature?
- Was there some A/B test that went live?
- Is there some updated version that's causing this?
- Is this due to server side issue?
- Is this due to some device or OS?
- Was there some change to my ad targeting engine?
- Is this happening due to some bad PR?
- Is there a regulation issue?
- Is there a competitor that's taking away the users and the advertisers?
Thank you for the question. Let me gather some thoughts.
Let me start with some clarifications -
1/ Let me understand the business model -> we sell a B2C product and we make money with ads on our product pages? Is that right? YES
2/ Are we a mobile and web platform only? YES
3/ Geo? US
4/ where did we see the revenue drop to zero? Any particular geo or every where? Everywhere
Thank you for the insights. I will make some assumptions and list them here (and validate by you)
Assumptions
1/ We have a healthy stream of advertisers that have their ads listed on our ad platform
2/ Based on the end customers interactions, we display right ads on the site
3/ probably there is a auction happening on the ad platform on whose ads are served where
4/ We make revenue based on views and clicks (views being lesser and clicks being higher)
Before i dive into the crisis dealing strategy, let me paint a picture of what happens in general
1/ We have ad platform
2/ BD team bring advertisers to this platform where they list their ads with keywords and auction amount they are willing to pay for the placement spot
3/ When our product page gets rendered for an end user, based on the keywords, an auction is run and who bid most get their ads get a highest placement and we make money
Now lets dive into understanding the problem
I want to split my investigation in the following buckets -
INTERNAL FACTORS
1/ Product pages
Did our product pages change - this is unlikely to not know until now because am the PM
Did we have a service outage on the product page - our alarms would have lit and not something we find after the fact
2/ Ad platform
Did our platform have an outage - same alarms should have lit
Did the ads licensing end - could be possible
Did the rendering algorithm or auctioning algorithm change - could be possible
3/ Other changes to our business model
Did we take a decision to not serve ads anymore - highly unlikely or even if the case, would know beforehand
Did we change the ads to be rendered before login and we are seeing all users are logged in users?
4/ Company wide changes
Did we move our ad places to different parts of the product pages or under pagination or below the fold - unlikely to find late
EXTERNAL FACTORS
1/ competitive moves
Did our competition launch a new competitive product which is taking away from visitors to product pages
2/ advertisers moves
Did advertisers move away from our ad platform
3/ legal moves
Did we get a legal hold which made advertisers move away or stop us from rendering ads?
Clarification question:
- Metrics monitored are: Number of Ads served and Revenue generated from the Ads
- Sudden decrease?
- Is this specific to a platform or a geography?
- Internal factors:
- Data sanity: Data pulled from the source is stable and not changed. Any calculated metric definition has not changed
- Tech:
- Any new release that would impact Ad serving.
- B2C product not loading and is down
- Underlying 3rd party Ad manager is down
- Users: Campaigns stopped running due to Advertisers pulled budgets
- Seasonal:
- This will never happen, unless this B2C App has a hisotric seasonal trend.
- External factors:
- DC down? - Natural calamity, abuse attack, etc
Clarifications/Assumptions
- What was the product where this crisis occurred?
Assuming Google search engine where user can search the content through any website
- When we say no ads served one day, is it due to some reasons ads were not displayed to the users or it was placed correctly but users were not seeing it due to some issues?
Assumption- Users were not using the product for some reason but the ads were displayed
- Does this has an impact on all kind of ads or only to a specific segment
- Assumption - All type of ads for every segment were down
- Is it for specific hours the issue happened or it impacted uniformly throughout the day
Assumption- For the entire day
- Only specific geographies were impacted or it was impacted across all the locations/boundaries?
Assumption- Specific geographies but not sure which one
Describe the product
Search engines typically make their revenue by carrying advertisements. Companies will pay search engines for displaying advertisements on a search results page. In this model advertisers would pay search engines for every click their advertisement received. The goal of the product is to generate revenue through ads. Users are the individuals, business who search the data and clicks the ads.
Strength:
- Strong presence across the globe with huge content lake.
- Ability to provide any kind of information in fraction of seconds in a user friendly manner at free of cost to the users
- Strong integration with the allied products like gmail, news, calendar, drives etc to engage and gain attention of the users
Weakness
- Majority if the revenue is driven from the ads. Selling the space for the extensive use of google drive yet another way of generating income but more than 60% of revenue comes from only advertisement. So it is solely dependent on ad business for revenue
List of choices
- Identify the root cause of the problem
- If it is happening due to some technical glitch
Choice- Pass it to the support team as a critical/high priority item
- Is there any change in compliance rules and regulation or governance policy that happens suddenly may be on temp basis like removal of article 370 in Kashmir leads to internet shut down
Choice- Identify which specific locations have been impacted. Wait till the situation resumes to normal and derive some discount or offer programs to the users in the nearby regions to search as much content as they can and share the information via right mediums like education institutes, walkie talkie, newspaper assuming the other means are still working for the specific period of time.
- It could be a sanction levied at international level by the trustbusters or the international apex authorities due to sudden political and economy related issues like cold wars
Choice- Assuming the impact is panoramic across various major countries. We can incentivize the usrs from other location to search and save the content and spread as much content as they can via other channels like print media, electronic media, phone, whatsup chats etc
- Focus more on the customer engagement on the allied products if it is just falls in ads and not in the internet service. Like sharing the ads over emails, calendars, meeting invites, giphys and emoticons with ads embedded in it. Displaying the ads on the you tube and other mediums like comments, google news, weather etc
Evaluation of choices
- Shutting down of internet is a biggest threat that will have potential impact on the revenues and becomes worst when coupled with unexpected events called unsystematic risk where control lies in externally.
- A war room can be created considering the criticality of the issue with tech team in case of technical glitch. We can also call out the experts from the allied areas to take help from their knowledge and expertise
- Focus more on allied product to come up with a temp solution where ads can be placed on the other allied products to continue the revenue model.
- Revenue model for such a giant organization like google should be diversified to reduce the ill impact in long run.
Recommendation
- Focus more on allied product to come up with a temp solution where ads can be placed on the other allied products to continue the revenue model and let the tech team fix the issue if it is technical one as the other choices may be having poor user experience , poor customer engagement and hard to execute that will take good amount of effort and time. Last option of strong ties with allied product to display ad is more useful considering it is completely in our control to make the changes and not been driven by the external factors.
Clarifications/Assumptions
- What was the product where this crisis occurred?
Assuming Google search engine where user can search the content through any website
- When we say no ads served one day, is it due to some reasons ads were not displayed to the users or it was placed correctly but users were not seeing it due to some issues?
Assumption- Users were not using the product for some reason but the ads were displayed
- Does this has an impact on all kind of ads or only to a specific segment
- Assumption - All type of ads for every segment were down
- Is it for specific hours the issue happened or it impacted uniformly throughout the day
Assumption- For the entire day
- Only specific geographies were impacted or it was impacted across all the locations/boundaries?
Assumption- Specific geographies but not sure which one
Describe the product
Search engines typically make their revenue by carrying advertisements. Companies will pay search engines for displaying advertisements on a search results page. In this model advertisers would pay search engines for every click their advertisement received. The goal of the product is to generate revenue through ads. Users are the individuals, business who search the data and clicks the ads.
Strength:
- Strong presence across the globe with huge content lake.
- Ability to provide any kind of information in fraction of seconds in a user friendly manner at free of cost to the users
- Strong integration with the allied products like gmail, news, calendar, drives etc to engage and gain attention of the users
Weakness
- Majority if the revenue is driven from the ads. Selling the space for the extensive use of google drive yet another way of generating income but more than 60% of revenue comes from only advertisement. So it is solely dependent on ad business for revenue
List of choices
- Identify the root cause of the problem
- If it is happening due to some technical glitch
Choice- Pass it to the support team as a critical/high priority item
- Is there any change in compliance rules and regulation or governance policy that happens suddenly may be on temp basis like removal of article 370 in Kashmir leads to internet shut down
Choice- Identify which specific locations have been impacted. Wait till the situation resumes to normal and derive some discount or offer programs to the users in the nearby regions to search as much content as they can and share the information via right mediums like education institutes, walkie talkie, newspaper assuming the other means are still working for the specific period of time.
- It could be a sanction levied at international level by the trustbusters or the international apex authorities due to sudden political and economy related issues like cold wars
Choice- Assuming the impact is panoramic across various major countries. We can incentivize the usrs from other location to search and save the content and spread as much content as they can via other channels like print media, electronic media, phone, whatsup chats etc
- Focus more on the customer engagement on the allied products if it is just falls in ads and not in the internet service. Like sharing the ads over emails, calendars, meeting invites, giphys and emoticons with ads embedded in it. Displaying the ads on the you tube and other mediums like comments, google news, weather etc
Evaluation of choices
- Shutting down of internet is a biggest threat that will have potential impact on the revenues and becomes worst when coupled with unexpected events called unsystematic risk where control lies in externally.
- A war room can be created considering the criticality of the issue with tech team in case of technical glitch. We can also call out the experts from the allied areas to take help from their knowledge and expertise
- Focus more on allied product to come up with a temp solution where ads can be placed on the other allied products to continue the revenue model.
- Revenue model for such a giant organization like google should be diversified to reduce the ill impact in long run.
Recommendation
- Focus more on allied product to come up with a temp solution where ads can be placed on the other allied products to continue the revenue model and let the tech team fix the issue if it is technical one as the other choices may be having poor user experience , poor customer engagement and hard to execute that will take good amount of effort and time. Last option of strong ties with allied product to display ad is more useful considering it is completely in our control to make the changes and not been driven by the external factors.
Bronze PM
This sounds very serious. First, I'd like to clarify a few things to make sure I understand the situation correctly. When I do that I want to see if we can narrow down the focus to better understand where we should be looking for the cause of this. Finally, I will suggest what remedial action should be taken to solve this and whether there are any lessons to be learned for the future. Agree?
Sure.
Who are we, what is our product, what kind of ads do we normally have, and where do we source these ads from?
We are a small sports website that shows written articles and sports news, with video footage also. We show banner and other display ads on every page, video ads play before video content is shown to free users, and we also have sponsored sections. We've both a mobile app and a website. Our advertising funnel has one source, we use Acme Ads Manager as our Demand Partner.
Is this affecting mobile app/web?
Both.
When you say there are no ads being served what do you mean?
Banner ads are not displaying, video ads are not playing at all. Visitors to our website and app are not seeing any ads, so we are not getting any of that ad revenue.
Is this affecting all geographic areas or some?
Well, no ads are being served so possibly it is affecting all areas, but really we don't know.
Did this happen abruptly?
Yes, ads were playing yesterday and revenue was coming in as normal.
Can you look at the daily advertising revenue figures and the daily active users for each day for the last three weeks to validate this assumption. I want to make sure there are no hidden trends that might be lost.
There have been no significant drops in daily revenue metrics and site visitors over the last month have remained consistent. This is an abrupt change.
So am I correct in summarising things as the following: We are not showing any display ads video or otherwise on our site, since some time between the time now, 9.05 am and yesterday and consequently are not receiving any of that revenue stream for that time. This is the case for our website and mobile app.
Unfortunately, yes that is accurate.
Ok. I'd like to see if we can isolate this a little. Specifically, I want to see if we can work out when exactly this first happened, and see if we can see any external or internal factors that may have triggered this. Whilst we do that I want to also check our Acme Ad Manager account to see if there are any issues with it.
Firstly, when was the last ad served?
Don't know.
Ok, well then, when was the last successful receipt of revenue for us displaying an ad?
This seems to be at 12.03 am.
Ok so sometime after 12.03 am this problem arose.
Let's look at external factors and internal factors that may have caused this.
Did a release or upgrade create a bug?
Any external issues with data centres or reports of outages anywhere?
There was no release or update last night and servers were constant. No reports of issues with data centres.
So then let me look at the source of our ads.
Firstly, does anything look amiss with the Ad account?
- Of setting up ad campaign - partners login to the portal, select type of campaign, add necessary details like duration of the campaign and then publish the campaign.
- Would check if partners were able to login to their portals and set campaigns yesterday -> as per interviewer, the metrics show that this was working fine.
- Was their any release for the campaign portal - no release. Also would check the data regarding active campaigns - this is also fine. So does not seem to be an issue in the ad campaign portal.
- Of customers clicking/viewing ads - customers login to the b2c portal and while scrolling through their posts/interacting with friends see ads in their posts.
- Would check - yesterday's data on number of logins to the portal ->lets say the login metric is fine. Number of daily active customers is trending at normal level. Thus customers were able to login but did not see any ads since impression numbers are 0.
- Next would check - if there was any release for the customer b2c portal -> yes as per engg there was a release yesterday. Would check with respective engg team regarding the items sent in the release and check if the issue at hand could be because of the release. Once it is confirmed that issue is due to the release, would discuss with the respective product owner & key stakeholders regarding whether this was expected and if not would work with the team for rollback of the feature.
Platinum PM
Interviewer: Yes
I: Also, has this happened specifically on a certain device or timeframe?
Interviewer: Time frame is essentially just yesterday's. We get data a day late in this case. As far as device, no we are looking at every device.
I: Is it likely that we looked at analytical numbers inaccurately or that the tool has had an issue or maybe db got corrupted and gave us wrong numbers? Have we looke at other stats to see if those stats make sense?
Interviewer: tool is fine, db hasn't been corrupted. There is something else bigger that's likely going on.
I: Ok, so the way I look at this is that there could be some internal factors (things we could control) and some external factors (outside of our control). Let's explore each but let's dig in further with internal factors since we already got started on this with analytical tool or data corruption issues and all.
Internal factors:
1. Any infrastructure related issues that we know of? - No
2. Any major code changes that could have been a resultant of this? - No
3. Did we hike the price excessively? - No
4. Have we consistenly seen numbers drop or has this been a just a dramatic drop to 0? I assume you mean a dramatic drop. - Yes, dramatic drop without any pattern.
5. Did we have bunch of contracts expire? I highly doubt because even if we did, our numbers have dropped to 0 and I would be shocked to see that we had enough contracts expire to result to 0. - Agreed, so no such exponential contract expiration in play here.
Ok, let's explore some external factors.
External factors:
1. Has this been seen from a certain region of US? Let's assume we are looking at US here. - Yes, we noticed Chicago has had a drop to 0.
2. So these numbers drop to 0 is only related to Chicago cutomers? - Possibly.
3. Ok we should look into that further and we found that that is true. So, do we know of any major storms or any other major outages in Chicago area? Or is it likely that our AWS server that served Chicago region didnt' fail over to the East region DC? - We just had some major snow storm in Chicago followed by severe -44 degree artic blast yesterday.
Hmm...I think what coud hvae caused his drop is major storm would have distrupted electricity major times and an Arctic blast made it too cold for electric lines to be fixed. This means, our customers didn't have any access to internet and we know that we have a drop to 0 from customers in Chicago.
Given, limited time and currently available information that we uncovered, I am going to keep a watch on these number and proceed with conclusion that the drop in revenue to 0 shouldn't be alarming. However, if Chicago issues get fixed and I don't see a spike again, I would surely be concerned. This is still a high alert watch for me.
Platinum PM
1) Was ads the only source of revenue for the product? Or was it the major source of revenue (Only source)
2) What industry/industry type did the B2C product belong to? What kind of product it is? (Mobile or web based) (Both) Or is that something that you'd like me to assume (Assume)
3) What was the reason for no ads? Was it because of any internal issue? Could you tell me a bit about this situation
4) Can you explain and tell me a bit about the product? (What it does, problem its trying to solve, competitors)
5) Was it a gradual decrease over a period of time before becoming 0 or was it a sudden drop? (Sudden drop)
6) Was the drop witnessed/seen in a particular geographical area or all across the world? (Across the world)
Step 2: State the assumptions
1) It is both a web and mobile based product
2)There was a sharp decrease in ads, if there would have been a gradual decrease over a period of time as a PM i would have witnessed and figured it out
3) The drop wasn't constricted to a geographical area. (Was seen all across the world)
State 3: Analayse the situation (Root cause analysis)
What led to this sudden drop in ads on the product all across the world?
First i would like to discuss some external factors that could have caused this situation before going to the internal factors
External factors
1) A new product or feature launched by a competitor that could have caused this issue. This product might be 10X times better than the current product and has no ads
2) Was there any external large scale event (like the presidential elections) that could have triggered this (i.e the advertisement industry in general)
3) Any change in governement policities/ban related to products showing or displaying ads
4) Negative PR about our product/platform
Internal factors (Can be from the advertisers side as well as the client side)
1) Sudden change in demand of the clients as they wanted to move towards an ad-free business model
2) Advertisers might have gotten bankrupt and out of business hence no ads from their side
3) The ad-campaign ended and there was no re-activation done
3) System related changes
1) Advertisers unable to display/post ads on-to the platform because of recent change in algorithm/privacy seetings.
2) Clients unable to see the ads that are displayed, can be possible because of some change in UI.
3) Ads are not getting updated on the DB and thus people are unable to click on those ads.
Step 4: Solving the problem (Rectify the situation)
External factors are something that we have no control on, however internal factors we can control, assess and evaluate on a periodic basis
Moving on with the internal factors
1) Address this issue with clients, as to why they want to make this decision, what made them come to this conclusion and if they are making this decision are they certain/sure about it.
However as a PM since i am working closely with the customer success teams and sales and marketing, this change in demand i would have known. As the demand however sudden doesn't happen overnight.
So i don't think this would be the problem
2)Advertisers getting bankrupt is something that doesn't happen overnight. And as a PM i would be aware of the advertisers on our platform and about their finanical health.
Search and do a market research for new advertisers who fit the bill and requirements. A/B test out the platform with different set of advertisers and types of ads and go with the one that is more favorable from a business point of view, also see the feedback you get from the end users as thats a very important metric as well.
So i don't think this is the problem either
3) If the ad campaign ended, then we will have 3 options either re-activate the ad campaign with the current advertise, go for a new ad campaign from the same advertiser or search for a new advertiser.
In this case evaluate the business impact, going to a new advertiser or reactivation ad campaign will have and choose the one that is more favorable from the business point of view for the product.
4)Now moving to system related changes
Advertisers unable to post/display ads on the platform- There was a new update with the new algorithm in place which had a lot of bugs and issues, because of which advertisers were unable to post ads and since there were no ads the consumers were not able to see anything.
Now this is soemething that can happen overnight, a new update can for sure can cause an issue. So as a PM i would try going deeper into the issue.
Firstly i would work on rolling back this feature to the previous update version, then would work with the developers to go through the bugs that were reported and fix it. Work with the QA and testing team and get this back to work so that advertisers are able to publish ads on the platform.
Clients are unable to see the ads that are posted on the platform- This might be possible because of the recent UI change because of which ads are not very promiment discovered on the main homepage screen. The feedback that i received from the users (through the customer survey and usability testing) was to not show ads on the home screen.
UI changes like these are always tested and released incrementally. Work with the engineering team to figure this out and position it in a way that satsifies the customers as well fits the new UI changes.
Work with the marketing team to broadcast these UI changes and ad-positioning and everything so that people are more aware
Ads are not getting updated on the DB work with the engineering team to fix this issue
Step 5: Summarize your answer
Top Google interview questions
- What is your favorite product? Why?89 answers | 263k views
- How would you design a bicycle renting app for tourists?62 answers | 82.5k views
- Build a product to buy and sell antiques.54 answers | 66.8k views
- See Google PM Interview Questions
Top Product Strategy interview questions
- What should Airbnb's strategy be during the COVID-19 pandemic?26 answers | 35.9k views
- How would you acquire more users for Uber?22 answers | 33.8k views
- How would you determine if a specific block in your neighborhood is suitable for a new grocery store?14 answers | 13.4k views
- See Product Strategy PM Interview Questions
Top Google interview questions
- How would you improve Google Maps?53 answers | 228k views
- A metric for a video streaming service dropped by 80%. What do you do?50 answers | 135k views
- Calculate the number of queries answered by Google per second.45 answers | 78.5k views
- See Google PM Interview Questions
Top Product Strategy interview questions
- You are the PM for Facebook Live. What are your priorities?13 answers | 19.7k views
- Evaluate the upsides and downsides of building a super app — an app having all major B2C features including entertainment, e-commerce, food ordering, hotel booking, cab booking, chat, holiday planning, gaming, med ordering, service booking, etc.11 answers | 15.7k views
- How would you sell live plants on Amazon?10 answers | 11k views
- See Product Strategy PM Interview Questions
