How do you prioritize a customer facing feature with X impact on engagement vs a tech driven task like upgrading the libraries to eliminate legacy bugs which have no immediately visible impact?

Approach: Clarifying Questions, Understand the goal/objective/key performance metric of the company for the quarter, Determine ROI of the tasks, recommendation

Clarifying Questions:
Q1: What is X?  is it high/medium or low impact? what is the complexity of task in terms of Man hours?
Q2: Legacy bug - Can you please define immediate and visible in this situation? By when would we start seeing a significant impact? how much in advance we should start to keep the impact under control? What is the complexity of the task in terms of man hours?
+++++

Goals/Objective/Key performance Metric for the quarter - Facebook at this stage of product life cycle would like to maintain the engagement levels and would like to ensure it can offer something new to either reduce retention (competitive offering) or improve profitability (Revenue) and would like to ensure a seamless experience for the users (security, Performance).
Hence, Metrics for the FB based on the current stage of product life cycle would be:
1. Seamless Experience (as it also has an impact on engagement and Retention) - No. of security issues, bugs, Impact on brand
2. Maintaining similar Engagement Levels - DAU, MAU, Avg. session time per user
3. Retention (competitive offering) - Reduction in customer churn (New feature impact)

Now depending on the magnitude of the impact, each of these three types of metrics can take priority over the others.

So, let's confirm with interviewer if in this quarter what is FB's major goal.
Are they focusing on releasing a feature to eliminate/minimize the competitive threat? Yes
Do they need to release a feature to ensure similar level of engagement or they can survive this quarter without it?
Is it the right time to start prioritizing the tech debt?
+++++

Elements to be considered for ROI Calculation:
1. Complexity of the tasks in terms of man hours
2. Direct impact of the tasks on the key performance metrics for the quarter
3. Blockages - Dependency on other tasks
4. Impact on other critical tasks

Based on the above Q&A and structure, we can prioritize one task over the other.

There are many ways to go about answering this Faceboook product improvement interview quetstion and it really helps to get the right data to make the decision.

With the lack of data, here is how I would approach the situation. It is exciting to move on and work on features that increase engagement while ignoring the growing trench of technical debt but at the same time managing technical debt is important because it can potentially have a snowball effect in the future.

A few different factors could be considered to evaluate whether or not such a customer-facing feature at FB should be prioritized over technical debt, even if the resources are going to be split in some form.

• Are these features going to directly influence the ad revenue that the newsfeed brings to the business? Basically, figure out whether it is going to immediately increase ad revenue as a result of deeper engagement.

• Review current engagement metric KPIs - Are they unsatisfactory, meeting expectations or exceeding expectations?
  Overall determine how the app is performing in terms of growing engagement with existing customers at the rate that is considered healthy. 

• Are market forces pulling down the app's share of voice? Is there enough data to support a future trend of flatlining or decline in engagement? Time-to-market is an important consideration too.

• Are there specific time-bound company goals that need engineering to innovate newer forms of engagement to complement growth in certain geographic zones. Timing is key in such situations and seizing the opportunity quickly can really benefit.

Assumption: The aim is to minimize the risk of technical debt becoming unmanageable for the business. We might as well end up prioritizing fixing certain types of bugs.

1. Get the complete list of bugs with their bug description. - Bug list V1

2. Separate the bugs from the feature enhancements - Bug list V2

3. Isolate the bugs that might have an impact on revenue or critical product functions. - Bug list V3

4. Now score the bugs according to factors like

   1. Does the bug affect Tier 1 revenue-generating advertisers or other sources of revenue in the next 3 months?

   2. Impact on product performance, how affects other feature(s) - Very low, low, medium, high

   3. Security-related? Yes/No

   4. % of users who are affected by the bug - Ask the defect management team for a % benchmark to use as a cut off for giving priority to bugs with a very high impact on users. It should basically affect key users or all users.

   5. Does fixing the bug enable the newer features/ future upgrades for core features to perform well? Yes/No

   6. Level of effort to fix the bug 

   7. Level of technical complexity - very low/Low/Medium/High/Very high

 

After doing the above, I would narrow down to the most important bugs to compare with the scores with the feature prioritization scores to make an information-driven decision.

If you are a startup, its always a good to write-off certain percentage of tech bandwidth for techical debts. The reason is we need to pay the techical debt as regularly as we can even if does not have direct impact on core metrics else it will have large impact on business. Just like we have goals and metrics for product, it would be good to have similar to goals for the tech metrics as well such as performance metrics, quality metrics and have a plan to improve these. You can ask any companies about the large outages that they had becuase of code or performance issues and what they lost during that period. 

For the sake of this questio, lets make an assumption that tech team is coming up with a genuine items that will have long term impact on the code (and not fancy stuff). The question that will come is when to pick these items (to be specific which sprint or which quarter). A simple rule would be pick up the customer facing items that will impact on the certain metric first in the quarter (or sprint) and deliver it so we get enough time to get the feature to be adopted and show its results and the engineering team could use the rest of the time to pay the debt. 

There will always be exception to the rule such as 

1. The new customer facing feature would require certain technial debt to be paid to begin the work. 
2. The technical debt item may take longer than a single sprint 
3. The technical debt does not necessarily impacts the tech metric that we are trying to improve (say performance) 
4. If we do not fix the item, that it will expose us to security vulnerabilities or meet any regulatory guidelines (say PCI) 

 

 

 

A few different sets of ideas clustered together to help assess the priority: 

Trade-off analysis:

Feature Group	Competitive Offering	Overall Product Health (performance, security, efficiency)	Long Term Risk	Scoring
Customer-facing	Yes (3)	No (1)	Risk of losing user engagement (5)	9
Tech debt/architectural	Doesn't impact (1)	Yes (5)	Scalability & vulnerability issues (5)	11

Categorize tech debts in terms of value to the business & measure it in terms of the effort for the work:

Category	Effort	Value
App Performance	Medium	Doesn't yield any significant change
Security Update	Low	Allows user-profiles secure
Android OS changes (compliance)	High	App/Feature Crash, App updates
Inhibits certain modal/design toolkit changes	Medium	Allow creating modern modal design & in-app experience

Based on the first analysis, we get a score that tells us that tech debt will be expensive misses and should be prioritized.

Based on the second analysis, I can choose categories of work I want to do incrementally towards tech features so I don't completely neglect the most important ones.

Lastly, I would ask questions about what the "functional feature is and see how it impacts user engagement". I will try to map how the feature of that feature may be tied to one of these categories from the second analysis and try to find a meaningful co-relation (if one exists). Having a more digestible picture of the scenario then will help us decided what efforts to prioritize.

Depends on what the quarterly goals we've set. If we have enough bandwidth after allocating Engineering resources to the products/features we've set out to release that quarter, then we can set the remaining resources to work on tech debt. Also, assessing impact of these libraries on future releases planned for Android app will tell us if we should work on it immediately or in the subsequent quarter.

What do you guys think?