First let's idenfity the goals behind this decision. There are two possible decision horizons for this decisions- first, why was the decision made to outsource the app development; second what objective we will serve by bringing it in-house.
Ask the interviewer if we have any information on why was it outsourced.
There are a few possible reasons:
1. Resource constraint: We didn't have enough bandwidth to get it done. in-house.
2. Risk: There was an opportunity for this app but there were other oppotunities as well and we wanted to execute on them all so wanted to see if we can execute on a few of them in parallel and experiment to see which one has legs. Not all of them will succeed, and we will pick the winner and scale it.
3. Skill: we may not have core skill to develop the app and with multiple parallel opportunities, it was wise to outsource the pilot so that we can test quickly (fail quickly) before deciding if we have a winner in our hands and scale from there.
Next we outline the concerns and conditions for bringing it in-house. There are some additional considerations (besides success of the pilot) that will motivate bringing the app in-house.
1. Scale core businesses in-house where you have core skills to implement the corporate requirements - integrating with core systems, implement coding standards, QA standards and automation, build and deploy automation, tracking/analytics standards, marketing automation and measurement,.
2. Security and Compliance: core apps will have to follow higher standards in data security (in-transit and on-disk), hardening againsts attacks and code vulnerability, platform securty and so on.
3. Integrate App's evolution into company;s processes for measurement, feature experimentation, approval, resourcing, scaling, business outcome attribution and even cross polination etc. These issues can leverage or depend on sensitive business processes and/or information.
Considering these cases - at the right time winners should be brought in-house. It is fine to oursource the app until its worth in market fit and experience/tech is proven (especially if it requires new skills or additional resources that we don't have) but after that pilot success it is importal to bring that app in-house. It will even justify the resource training or adding/hiring (since the fix cost can now be spread over a long term horizon)
Define the metrics to decide when the app is successful to bring it in-house. This will have to align with the organization's goal and mission.
Traffic or app awareness/interest
Activation/retention/referrals (for opportunities that require solid proof of success vs. early indicators)
Revenue or profit target (against cost to justify taking on long term fix cost)
Based on company goals and sensitivity to brand perception (from bad quality) I would suggest using an early or late metric to use for deciding when to bring that app in-house. I would choose between user acquisition (early metric) or retention (later metric). If company is sensitive to spenidng in-house resources too early (in an app's maturity cycle) I might also suggest profit/revenue metric which will prove out monitization viability as well.