Your product just caused a worldwide outage. How would you go about handling this situation?

What is the product? An erp tool

What does outage mean here? People are not able to access the tool

Is the outage related to any specific part of the tool or complete tool itself? Complete tool

Since when this issue exists? Yesterday night

Is it only specific to web or mobile (Android or iOS?)? All

Hypothesis -

Internal factors -

Did we do any bug fixes recently? No

Did we do any new releases recently? No

Did we do any changes in the UI or any internal flow recently? Yes, we did some performance optimisations within the tool for faster page load and search

External?

Are there any changes in the global data policy and compliance? No

Short term -

Acknowledge the issue, issue a public apology and revert back the changes to the latest working version of the product. 

Inform the users about the stability of the product

Long term -

Thoroughly check the issue, identify root cause, prioritise the fixes

Keep all the stakeholders updated about the latest status of the fixes and keep sharing periodic updates

Redo the performance optimizations and then test it properly and then rollout again following up with a production sanity for each of the major markets, to ensure none of the markets are impacted.

Metrics to monitor -

% user logins

% feature usage

% customer complaints

Improvement in NPS