If you were a product manager at ChatGPT and saw a rise in thumbs down on responses, how would you identify and address the root cause?

Problem statement - Rise in thumbs down on responses in ChatGPT

Clarifying questions -

Was this a sudden or gradual drop? Gradual over the last one week

Was this drop specific to any geography? No

Was this observed specific to portal or app? No

Was this specific to android or iOS? No

Was this across all users or specific to a particular set - free users? All

Is the increase in thumbs down specific to a particular domain or overall? Overall

Was there any new feature release during this period? No

Was there any bug fix which went live during this time? No

Was there any retraining of model happened during that time? No

Wa there any new AI chat launched by any competitors with better responses or more accurate responses? Yes, there was a new competitor product launch

Do we have any feedback on the responses generated by the competitor product? No significant data

Hypothesis -

Based on the above data points, it seems to me that the competitor have launched a new product during the similar timeframe, which being new is supposedly trained on a much larger data set and is likely to produced more accurate and faster responses. 

As next steps, we need to comapre the responses generated by ChatGPT and the competitor product for same questions, see the user feedback and validate that whether the quality of responses is better for the new product. Assuming, this is true, we need to retrain our model with newer datasets ( currently it is updated till end of 2021), so that the responses could become better and more accurate.

In the short term, to be really sure that users understand the menaing of thumbs down, we can add a tooltip explaining the meaning of this button, to ensure users are not using it incorrectly.

Metrics to track

• % responses with thumbs down
• % responses with thumbs up on same questions which were earlier thumbs  down
• Overall ratings from users