You are the PM of the Facebook timeline. How would you measure retention?
+2 votes
2k views
in Metrics by (28 points) | 2k views

3 Answers

+7 votes

Follow Up Questions First

Italics are assumed answers.

  1. Can you define the Facebook Timeline? This is the feed based section of a user's profile, which allows visibility into a chronological timeline of what has been happening with that user on Facebook. This does not include the "peak" feature when you hover over a user's profile. 
  2. Why do you want to measure retention? Is this for user growth? Is this for a specific segment's adoption of the timeline (Eg. Workplace by Facebook? Going to assume that there is no specific reason for wanting to measure retention, and we want general growth. 
  3. Am I measuring other related metrics on the Timeline? Let's assume no, we only care about retention for now.

Timeline Goals

  • First let's outline the goal of the Timeline. This feature aims to add personalization to Facebook, giving users a literal wall which they can paint with whatever they want as a way of expression. The timeline should provide both quick and in depth information into a user, with a focus on quickly accessible, but stalkable information. 
  • Many Timeline posts are expandable, allowing users to click in and dive into a post. The main use case for a user visiting a Timeline is to keep up to date with that person/page, and is typically not to browse through the entire history of the Timeline. This mostly comes from personal experience and anecdotal evidence, but I feel that typically a full scroll through someone's Timeline is not the main use case. Most of the time it's when we have not caught up with a friend before, and just want to check in with them from the last time we interacted. 

Metrics

  1. timeline visits per user per day
  2. time spent on timeline per user per day
  3. time spent on timeline / total time spent on Facebook per user per day
  4. timeline interactions per user per day

For measuring retention, I am aiming to both grow my slice of the pie and the entire pie. These metrics give a good simple overview of how users utilize the Timeline and how it relates to the Facebook product as a whole. Metrics are ranked in order of importance.

  1. Metric 1) measures simply total time spent on the timeline per user. This is simple aggregate level metric, grouped by user, gives us a glimpse into broadly how much the timeline is viewed. I purposefully did not include time spent as the top metric, as quick viewability is more important as stated above.
  2. Metric 2) is more focused around the stalk-ability of Facebook. Facebook is often used as a platform to retrieve or lightly online stalk others information. This aspect of the Timeline, while not the main goal, is still a very important metric to understand how people are using it. 
  3. Metric 3) measures cannibalization versus the rest of Facebook. If we want to grow Timeline retention, we also want to grow this as a proportion of time versus other pages on Facebook. This metric does not measure page visits, as there is no singular page visit metric which is measurable and standard across Facebook. This is mainly due to the idea of infinite scrolling on Facebook, where visiting the News Feed with one visit could take hours of time. 
  4. Metric 4) measures user interactivity. As is the case with any product, how users not only visit but interact with it is a strong signal of how they feel about the product. 

User Surveys

Outside of the metrics mentioned above, which are passively collected, I would also utilize active collection of an basic score to determine how people feel about their timelines. This would be a simple survey, sampled across all Timeline users for how they feel about the Timeline. "How to do you feel about the Timeline from 1-10?" This score is a simple way to collect user feedback to understand how users feel about the Timeline. This question would then be followed up with an open ended input to allow users to elaborate on their answer. 

by (41 points)
0
Great answer! loved it.
+1 vote

What is the purpose of Facebook Timeline? 

Facebook Timeline helps users in the following ways - 

a. View a chronological list of timeline posts which has been posted by them or addressed to them.

b. Create Posts, Milestone etc. 

b. Update Profile Information

c. Launch and Organize Photos, Friends

If Facebook newsfeed is the place where users see what is happening in their friends life, Facebook timeline is the place where users can organize and manage their personal life and curate what they want to tell the world.

Coming to the question, the next step is to identify if the question is meant for one particular segment of customers or for all the customers. Let us assume it is for the entire customer base of Facebook. Retention metrics consists of measuring behaviours of users in timeline and the success of that can be measured as follows - 

a. Number of timeline page visits/user/day 

b. Median time spent in the timeline/user/day (Measure active time spent in the application. Active Time is the time spent where the user has scroll, clicked or performed any other activity in the page for greater than 1 minute) - Look at 20 percentile, 50 percentile, 90 percentile and 95 percentile metrics for this would give a sense of how long users spend in the timeline page and track retention for the Low, Medium and High Users of Timeline 

c. % Time Spent in Timeline vs Rest of the application

d.  Posts Created via Timeline/ Posts Created via Newsfeed (Measure this metric by each kind of post - video, life event etc.) 

e. Comments, Likes , Shares from Timeline per day

I would prioritize b, e, d, c, and a metrics since those metrics are the key indicators which denote positive behaviours of Timeline users.

by (44 points)
0
Good structure. Some feedback:
- Why does c matter? If your concern is cannibalization, I suggest you mention it.
- I also recommend describing your evaluation criteria in more detail.
Good luck!
0 votes

Thanks for the question. I have a couple of follow-up questions: 

1. WHY? Why do we want to measure retention - are we concerned that people might be leaving the platform? Or is this part of a bigger strategy for something else?

Answer: We just want to understand our user behavior and make them spend more time on FB.

2. Thanks. Please give me a couple of minutes to think thru this question.

Think:

What is the question trying to test? 

  • Knowledge of Metrics and Measurement
  • North Star Metric and Guardian Metrics
  • Critical User Journeys
  • Product Intiution
  • Come up with multiple top line metrics and decompose general ones into specifics
  • More than one way to solve the problem

Answer Flow:

Define CUJ

Define what we are trying to measure?

Connect CUJ with measure

Multiple ways to measure

Isolate to one way - decompose it into sub metrics

Provide guardian metrics

Start Answer

We'll start off by thinking about the feature itself - i.e. FB timeline, then we'll define retention and what it means to retain users. After that we'll connect the user journey and try to come up with ways to measure retention.

The feature - FB Timeline. The FB timeline is a rolling wall of updates where each user can post something (image, video, text etc.) and other users can see it. There are two primary modes for the timeline - your personal timeline and the social timeline (what everyone else sees). I think we should first focus on the social timeline for measuring retention. Is that okay? (Answer: Yes).

Okay now let's try to think about what retention is. I'd define retention as the total time that a user spends on a feature. i.e. the point in time in between them first interacting with the feature and last interacting with the feature. 

Before moving forward let's take a step back and think whether there are any other ways to define retention. Another way to define retention would be the total number of users who are active at any point in time. i.e. the total users that are being retained due to the timeline feature.

Or at a higher level, it could be the time that a user spends browsing (not actively interacting) with the timeline.

I'm going to stick to the first definition for now i.e. time between user's first interacting with the feature and last interacting with the feature.

Let's apply this definition to the FB timeline. So we need to measure two things - First interaction and last interaction.

First interaction can be - first time user sees another user's post. first time user likes or does an action on the post. first time the user posts something. To be broad, I'll take the first time the user performs any action on the timeline - like, click, comment etc. 

Last interaction - this is tricky. Last interaction can be the user unsubscribing from FB. But that's too harsh. Let's define last interaction as a period of inactivity from a user for 1 week. 

We can use this metric Retention as the North Star metric - i.e. a metric that guides us. But we should also try to have some guardian metrics to make sure we are not optimizing too much on one behavior. 

What are the things that are important for a robust timeline? People should post, people should share others' posts, people should post things that increase the retention time of others, people should click on ads shown in the timeline. All of these could be our guardian metrics. i.e. when we optimize for retention, we should also make sure the guardian metrics don't suffer dispropotionately. 

by (13 points)
+1
You don't need to spend much time to describe what retention is. Both you and interviewer know what it means. Spend your time discussing the metrics that help you measure retention and prioritizing them.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
To avoid this verification in future, please log in or register.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
To avoid this verification in future, please log in or register.