You are PM of FB Events and you saw an overnight drop in RSVPs by 10%. How would you go about determining what happened?

A Facebook metrics question framework.

Define Product:

Facebook events lets people organize gatherings in the real world with people on Facebook.  As a creator, you can create public or private event, set date/time/location, invite people.  Invitees can see these events, view who’s coming and provide an answer (RSVP). They can share events with their connections. 

 

Define Metric: 

• Clarify what RSVP means: when an invitee responds by selecting an option that indicates if they’re going or not. 

• Clarify what’s measured: Assume: Avg # of RSVPs per invite had dropped

• Period of comparison: Over night drop of 10%

Summarize: Avg # of times an invitee responded to an event has dropped overnight by 10%. 

Narrow it down:

• One time drop or has been progressive over several weeks?: Assume one time

• Regional? Or global? : Assume global

• Seasonal pattern (i.e do we see this every year this time, perhaps on christmas or new year day?)  No

• Which platform (mobile/desktop/all): Assume Mobile iOS app

• Any specific user segments: Assume no specific segment 

• Is this for public or private events? Assume: only Public events show the decline

• Is there difference in terms of type of response (public events can be RSVP’ed as “Interested or Going”) : Assume:  both “Interested/Going” have declined, about the same rate. 

• Decline in other features in mobile? Or only event rsvps impacted? Assume: Only event RSVP

• Any A/B feature testing going at the same time? No

Summarize the problem statement: Avg # of times responses that are given on a public events dropped overnight by 10%, on mobile IOS platform, globally

 

Hypothesis:

 

External factors

 

1. A sudden global event that’s causing people to not want to attend public events. (Something like COVID-19 stay at home situation)

2. A new product / feature in a product that’s attracting people to organize/respond to public events (unlikely because it’s a sudden one time drop of rsvp’s)

 

Internal factors

• The way we track the metric is broken or something in metric calculation method changed  

• Bug introduced in mobile that is not registering people's responses on public events. 

• Public events aren't as displayed in as many places anymore as they used to be (on IOS) on the users's journey on mobile. So the user can't see the public events and hence can't respond

• Users don't see relevant and interesting public events as much as they used to

• RSVP actions aren't as visible anymore or not as prominently displayed anymore on mobile IOS

• People don't see the benefit of RSVP'ing to public events (this is unlikely because it would have been a progressive drop)

• Changing user behaviour: People have concerns about privacy and don't want others to know they're going to a public event (this is unlikely because it would have been a progressive drop, not a sudden drop)

Top Hypothesis and Testing of Hypothesis

1. The way we track the metric is broken or something in metric calculation method changed 

Test: Talk to devops and find out if feature usage is tracked properly and metric calculation is unchanged in the last 2 days

2. Bug introduced in mobile that is not registering people's responses on public events.

Test: Find out if a new release yesterday has introduced a bug. Talk to QA.

3. Public events aren't as displayed in as many places anymore as they used to be (on IOS) on the users's journey on mobile. So the user can't see the public events and hence can't respond

Test: Find out if a UI change has been made on IOS and we removed or moved the way users see public events in FB.

4. RSVP actions aren't as visible anymore or not as prominently displayed anymore on mobile IOS

Test: Find out if UI change has been made that moved the actions location, or changed the look and feel or ways of responding (for example, rather than an action word like "going" did we change to an icon like a checkbox so users can't see it? etc). (Note: as the PM for events, I would know if a change like this has been made, but nevertheless...)

5. Users don't see relevant and interesting public events as much as they used to

Test:  Find out if the algorithm that chooses what public events users are shown/recommended in their feed or Events page has changed and people are seeing events that they're less likely to respond to

Solutions

Let's assume that the algorithm decided which public events are shown/recommeded to a user in their Event page changed. Now, when you click on "Discover Events", it's sorting by a different algorithm. I'd find out what prompted that change, what were the goals we set and what metrics did we expect to impact.  I'd find out about any A/B test results.  I think Avg# of RSVPs is an important metric for the Events feature and if something is impacting it negatively, I'd consider rolling back the changes or making additional tweaks to the algorithm to overcome the impact it caused. 

Step 1: Ask Clarifying questions to narrow down the scope of the question

1) Was the drop in RSVP's on a global scale or was it constrained to a particular demographic or geographic location? (Global level)

2) Was the drop seen on the mobile/web side or across all the platforms?. (Mobile)

3) What do you mean by RSVPs? (Are we considering the numer of people) that responded to an event around them decreased by 10%

4) Is it a yearly or regulary occuring phenomenon or is this the first time that it has happened?  (First time)

5) Is there any A/B testing going on that feature or not? (No)

Step 2: Talk and explain your understanding of the product

Facebook events is used by people for creating real-life  public and private events and sharing them with people across their network.

Users can see the upcoming events in their events tab, and can RSVP to an event that they find interesting and want to attend. They can also add these events on their calendar so that they don't forget.

Event creators and event goes can see the list of people who have RSVP'ed to a particular event.

Facebook event was launched by facebook a couple of years back and is not a very old feature, a lot of people dont know about this feature thus we can say that facebook events is in the growth phase.

 

Step 3: Lets talk about the factors (hypothresis) that can lead to this decrease in RSVP, we will then test these hypothesis out.

External factors:

1) A global pandemic outbreak like Covid-19 that may have restricted people from going outside their homes and attending events. (Since there has been an overnight drop in RSVP this definitely can be one of the reasons)

2) A competitor might have launched a product, which is more convinent has a better user experience, which has resulted to people moving towards trying out that particular feature. ( Even if the competitor had launched a product, user acuqistion and user adoption would take some time so i don't think this could be the reason)

Internal factors

1) Change in the events algorithm by facebook such that people are not able to see interesting events around them for this reason they are not RSVPing and there has been a reduction in RSVP. (The algorithm would have been tested out before launching it and bring the change, thus this would also lead to a gradual decrease in RSVP and not a sudden drop)

2) Change in the UI such that the events page is not so easily accessible to the end users, thus leading to decrease in reduction in RSVP.  (As a PM i would be aware of the changes made in the UI, and this would have led to a gradual decrease in people RSVPing so this is not the reason as well)

3) Change in privacy settings which has gotten the users concerned as they don't want others in their network to know that they are going and attending any particular event for this reason there has been a decline in RSVP. (Again would lead to a gradual decrease and not such a sharp decrease, hence can't be the reason)

4) New update has introduced a bug in the feature such that people are able to view the events but unable to respond to these events (Ask the QA if this bug was reported and tested)-This can be one of the reasons for sudden decline in people RSVPing to events

 

Solution

As a PM i would check the new update and further test out the bug and if the bug wasn't reported and checked before, I would roll out to the old update version so that people can view and respond to the events while the devlopment team works on removing the bug issues and then later on we can launch the update with additional tweaks.

Clarifying questions :

is the drop globally or specific to demography ? global

Is their any seasonal holiday which is common globally ? No

is the drop due to less no. response in rsvp?yes

is there any new feature which went live very recently ? no

is there any A/B testing ongoing which can impact the rsvp feature ? no

is there any drop observed on user engagement on facebook app which in turn led to rsvp drop ? no

is there any spike in customer complaints regarding the rsvp ? no

is this the first time we observed this issue or we have some kind of trend ? yes

is there any specific platform where we have observed the drop ? yes

is it mobile or web ? mobile

is there any drop on event view / engagement  with the event but did not view ? no 

 

Facebook events ar usually created by end users or organisers as public or private events which are shared to their family and friends, people respond to these event.

user journey :

user create a public /private events

share it on the timeline for friends and family 

people recieve the event on timeline nd respond by (yes, not interested)

people also add event on their calenders to not to miss the event.

 

External Factor : 

1. There is no seasonal holiday globally as discussed above which can have the impact.

2. Hypothessis : For public events if there are too many similar events ongoing and around same time , users are tend to respond less.

         we can track it by no. of similar category publish events in same areas vs the userbase to whom those events are visible.

3. 

Internal Factor that can impact : 

1. users are unable to view the events on their timeline due to changes in settings or due to some new changes which went live.

          Metric We can use to track:  no. of view on event vs responses

2. The respond button is unresponsive :

• Metric to track is to check which mobile platform is having issue ios vs android.
• then narrow down further more by app version / ios / android version where issue is more common .