I ran an A/B test on Instagram. It resulted in a 10% increase in usage for Instagram stories and a 10% decrease in overall usage for Instagram. As a product manager, what metrics will you ask from me to make trade-offs on the A/B test?

Let me first clarify a couple of questions:

• Does usage of Instagram stories mean creation, viewing as well as engagement with stories?
• Overall usage of Instagram means engagement with all the different features of Instagram such as posts, stories, reels, highlights etc (creation, viewing and engagement)?
• The time frame of these changes is the time-period for which the A/B test was run?

I will structure my answer by first discussing the goal of the A/B test, then look at the user actions and ultimately discuss the metrics which can be used to make trade-offs.

We all know that the mission of Meta (the parent company of Instagram) is to give people the power to build communities and bring the world closer together. Similarly, the mission of Instagram is to provide people the means to capture and share the world’s moments. Instagram mainly generated revenue through ads.

Instagram has been in the market for quite some time and has been facing stiff competition from other platforms such as Tiktok, YouTube etc. Every person has 24 hours in a day and each of these platforms have been competing to get these users to spend maximum of their 24 hours on their platforms and engage with satisfaction. And since the question states that the A/B test resulted in a 10% increase in usage for Instagram stories, I am going to consider the high-level goal of the A/B test is to evaluate whether the new change in Instagram stories could result in increased engagement for the platform.

Now, Instagram is a 2-sided marketplace operating with creators (common users, celebrities, influencers, businesses) and the viewers (who consume content).  Creators generally create stories, posts and reels and are motivated by the views and the engagement that they receive from the viewers. Viewers generally view and engage with the stories, posts or reels of their own friends or accounts that they follow and enjoy coming to the platform only if there is good content.

Now let’s look at the metrics to be considered to make trade-offs on the A/B test:

1. No. of Instagram stories creators
2. No. of stories created/creator
3. No. of stories with engagement/creator (Instagram already has some benchmark figures or categories of engagement such as 100 likes/comments, 1000 likes/comments etc)
4. Total number of stories created
5. Total number of stories with engagement
6. Total number of Instagram stories viewers
7. No. of stories viewed/viewer
8. No. of stories viewed with engagement/viewer

Note: Engagement includes views, likes and comments

The above metrics correspond to Instagram stories. Similarly consider the metrics for other main Instagram features such as posts, reels etc and the average time spent on Instagram/user.

Also, for engagement, consider positive/negative reactions and comments.

If there is a sharp decline in the number of posts or reels created, it needs to be analyzed before the A/B test change could be implemented.

If the decline in terms of usage is only sharp decline in negative reactions and comments, the A/B test change can be implemented.

If the A/B test change is only cannibalizing engagement with other important Instagram features, the A/B test should not be implemented as the main goal of the A/B test was to check whether the overall engagement has increased. Moreover stories are short-lived (24 hours) whereas posts, reels are long lasting which contribute largely to Meta’s and Instagram's visions of building and sustaining communities.