How could you use logs to determine if a new product is successful?

Question

How could you use logs to determine if a new product is successful?

Asked at Google over a year ago

673 views

Asked at

673 views

Top Execution interview questions

Imagine you were in charge of Facebook Watch. What metric would you want to measure?13 answers | 9.1k views
Weekly active users (WAU) for iPhone app dropped. What happened?10 answers | 6.3k views
You are the PM of Instagram stories. What goal would you set and how would you measure success?10 answers | 14k views
See Execution PM Interview Questions

Invite members
Invite by email

Add another

nehagupta · Answer 1 · 2025-06-24T12:22:46+0000

clarifications -

New product that has just gone like - MVP version and telemetry data, like funnel conversions, user clicks etc. is not available? is the right? yes
By logs do you mean - backend logs that are mainly captured to help development team debug issues and audit purposes. - yes that is right.
What does this new product do, value to users? U can assume.
1. Ok so I can explain how generically we can use logs to determine success or should i take assume a specific app and then explain? u can choose an app.
Lets assume I am the PM for the payment experience - where customers select our instrument and then pay on online store fronts using this instrument - pay later instrument. Is this fine? yes sure.
first lets define success for the product
1. Since its a new app 2 things are important
  1. Acquiring customers - growth in unique customers that perform transactions using this instrument.
  2. Seamless payment experience - i.e. high success rate and low error rates.
From a logs perspective- API calls that come to the backend, success or failure error codes are usually tracked. (debt API call)
1. figure out number of unique API calls with txn id (user id, amt etc. won;t be captured in logs as security concerns)
2. API responses for the same txn id
3. Thus we can figure out % successful transactions per day, figure out by merchant
4. day on day number of transactions
5. day on day payment success rate can be tracked.
Trade off -
1. Extracting these details from logs will be cumbersome as there are too many logs and bulk queries are difficult so every 6 hrs we might want to query so that logs are not too many and the query does not timeout.

Swati Grover · Answer 2 · 2022-05-03T16:23:05+0000

Clarification Question -

Are we talking about a specific product or any product in general?
Is there any market in particular?
When we say "new product" is it new for the company or new in terms of tech in the market (first of its kind)?
What does "successful" mean here?
What kind of logs are we looking at - Backend data of how the product is used, support data as well, licensing data of the product etc.

Framework - Product analytics can be done using the product logs as well as the backend data of a product. Now we can divide the info that we get from prod logs into 2 kinds of metrics - External and Internal metrics. External metrics refers to the inference that we can make using our logs about its performance in the market and internal refers to the inferences that we make that help us in improving the product processes.

Solution -

External Metrics -

Breath of use - How many active users we have on Daily/Monthly/yearly basis
Frequency of use - how frequently our users use the product. For instance - a product targets atleast 10 logins/month
Depth of use - Understanding which features are popular features
Efficiency - It is essential to understand the user jourbey around the product. This also corresponds to retention. For instance - User Registers -> Logs in -> Activates a feature -> Drops off. Now this feature is where user frustration needs attention

Internal Metrics -

Specific points in time, when a feature/product crashes, or disconnects from the server etc.
Time between a critical bug is raised for a feature to when it is resolved
Overall defects per new release
Support response time - SLAs