You are the PM of the Facebook timeline. How would you measure retention?

Product

Facebook timeline is a feature that displays the profile information of a person along with the historical information in chronological order on all the activities that took place in his wall

Users and Value Proposition

Profile owner - The owner gets to see all the posts that he posted or recived in his wall as a memory catchup and gets to recall all the vents that happened in a specific time period

Profile Visitor - The timeline feature allows for the users to go in to the profile of a newly added friend or a connection to be up to date with all the posts and information

Goal: 

As the product has been launched way back, it is important to understand how frequently the feature has been used in order to arrive at a conclusion if the users are gaining the real value proposition. A part of the strategy would be to make reuse the same idea for workplace

Metrics:

1. Average number of times a profile owner visits his timeline (By month)

2. Average time spent by a Profile owner on the timeline

3. Average number of time a user visits time line of other users

4. Average time he spends on the timeline of others

5. Average number of interactions made on the past posts by the user

6. Total number of friend requests sent after viewing a timeline

Priority 

1, 3, 4

Counter Metrics

1. Average Number of links that look expired while visiting the timeline

Structure: 

1) Frame the problem that Timeline solves for different user groups involved 

2) Frame the strategic context and why measuring retention is important right now 

3) Layout data analysis approach / metrics 

Starting with #1

Problem: 

Timeline is a user's social identity on FB and for many, a system of record. It's a place for people to tell the world who they are and set stage for how they want to be perceived. So it's a pretty important tool for User's own self expression. 

Timeline also helps users stay interested and up to date on the community that they are a part of. It's an important way for people to catch up on what their friends/acquaintances are up to or just to get to know someone a bit more (like a celebrity). 

To summarize, that's 2 main journeys: 

• Page Owner keeping their own timeline up to date
• User visiting someone else's timeline 

• Given Timeline is so central to what a user does on FB, I would imagine is it important to make sure users continue to gain value from it, in order to be interested in FB overall. 
• In the context of overall FB mission/goals, I would assume persisting user's engagement with their own and Friends' Timelines is very important in bringing people together and creating that sense of community. I would retention with Timeline ladders up squarely into overall Engagement and Retention goals for FB at app level. So more than Growth, I would assume measure Retention for Timeline is very important to ensure users remain engaged with FB overall. 
•  

Let's now dive into the journeys we identified and determine a methodology for measuring retention across both journeys. 

For either journey, I would look to identify and track key user actions wrt to Timeline that best correlate with long term engagement with the feature. I will look at historical data to identify these patterns.

Starting with the first journey: 

Journey 1) User using the Timeline as a means of self expression

Here are the actions this user might take:

• Updating their profile information: picture, About, other info in Bio: Metrics - # of Bio Updates /User /Month, # Profile Pic Updates / User / Month, Time Spent on Profile / User / D, W (segment by user type like Public Figure, General etc, channel)
• Spend time scrolling through their own Timeline to see how their identify/views appear to others: Metrics - Time Spent on their own Timeline / User / D, W, M, # Visits to their own Timeline / User/ D,W,M, # Visits to their own Timeline / User/ Session
• Check Timeline after posting something to see how it appears their Timeline: Metrics - #Visits to their own Timeline after performing an action like Posting, Commenting, Sharing etc. 

Of these, I would assume # Visits to their own Timeline / User/ D,W,M correlates strongly with user's overall engagement with FB. As in the user that checks in on their own Timeline often might post more, interact with Friends more. On the contrary, someone that is disengaged does not feel the need to check their own Timeline often. So I would prioritize this metric and work towards giving users Tools/Reasons that would make them interested in visiting their own Timeline often. 

I would also measure some Counter Metrics to this, which could be:

% of User's Time Spent on their own Timeline (as a ratio of overall time spent of FB). Because it is important to balance the time users spend on themselves vs the community to sustain their interest in community over time. 

Journey 2) Users visiting others' Timelines 

Key Actions: 

• Visit a Friends Timeline: Aggregate Metrics: Time Spent on others' Timelines / User/ D, W, M, # Visits to Others Timelines / User / D, W, M - I would segment by type of user profile visited like Public Figure, Friend, Acquaintance, Stranger, Channel
• Explore a Friends Timeline: # Visits to someone's Profile/Bio/User/D,W,M, # Clicks into Details like posts, photos, comments on posts / Visit, Similar Metrics By Session

Amongst these, I would prioritize # Visits to Others Timelines / User / D, W, M as it might correlate best with user's long term engagement on FB. It implies user is interested in learning about others in the community, which will keep them coming back to App. 

The key Counter Metric I would measure to this journey would be the same as Journey 1. % of User's Time Spent on Other's Timelines as a portion of their overall time spent on FB over a period of time to ensure they are staying interested in other actions as well. 

Facebook timeline is essentially the profile page of a user where the user can update her personal information such as About section, Education, Job, Interests and demographics. The user can also update her profile picture, cover picture, add photos, videos and a status. When user A visits user B’s profile, user A can see all the basic information and past posts including status, pictures, videos, third-user posts shared by B based on B’s privacy settings for the timeline.

The Timeline feature is important for a user to inform the network about her latest details including her relation ship status/job status/how she/he looks currently and view same kind of information for her/his friends.

Since there are 2 users involved with a timeline, the creator (or profile owner) and the visitor, I will be talking about how to  measure retention for both the type of users starting with first with the Profile Owner and then moving onto the Visitor if time permits.

Profile Owner’s journey

For the profile owner’s journey let me consider an existing profile of a user who updates her timeline page.

1. The user locates timeline button on her home page which is the feed page
2. The user clicks the timeline button (which is the name of the user located on the bar at the top of the feed page)
3. User selects the information to be updated, for example, profile picture
4. User uploads a new picture or selects a picture from previous uploads
5. User beautifies the picture by adding effects and frames or chooses not to beautify the picture
6. User selects the privacy status of the item although current profile pictures have only 1 privacy status available i.e. “Public”
7. User clicks save/update button
8. User starts receiving notifications related to the recent profile change

To measure retention for Profile Owner’s journey related to timeline the metrics that I will use are:

1. # timeline button clicks per month per user: to know whether a user is still using the timeline feature
2. # profile changes per month per user: to understand whether a user is still making any changes in the profile or is satisfied with existing information
3. # profile change initiations per month per user: to understand whether the user is thinking of changing any profile information or not. If yes and is not completing the profile change then FB has an engagement problem and needs to work on the features to provide a better experience in changing profile information

Profile Visitor’s journey

For the profile visitor’s journey let me talk about the journey first.

1. Visitor views a profile suggestion in the feed/ User wants to find a specific profile using the search box
2. Visitor opens the profile of the suggested/searched acquaintance
3. Visitor navigates through the timeline page of the acquaintance and looks at each information in detail or looks for a specific kind of information for example, education/relationship status/pictures/status/etc.
4. Visitor sends a friend request/closes the profile by navigating to another profile or her own feed page or logs out

To measure retention for visits, I will use the following metrics:

1. # external timeline visits per month per user
2. Time spent per external timeline visit - to test whether the user is actively going through each information or just views the profile picture, views the name and performs action including sending a friend's request or closing the profile immediately. Although, I think this might be more of an engagement metric than a retention metric

I would like to prioritize the importance of tracking each metric based on its impact.

So I would first implement measurement of the following metrics as these metrics will tell me whether the feature is being used at all or not.

• # timeline button clicks per month per user and 
• # external timeline visits per month per user

Then, I will measure “# profile change initiations per month per user” to understand whether the user has become less/more interested in updating her profile since the time she joined FB. Another, similar metric that can be measured is “Time of last profile update since the user joined FB”.

Finally, I believe if the user is at least visiting the timeline of her own or others once per month, the use of timeline still exists so “# profile changes per month per user” and “Time spent per external timeline visit” metrics become lesser important as retention metrics.

What is the purpose of Facebook Timeline? 

Facebook Timeline helps users in the following ways - 

a. View a chronological list of timeline posts which has been posted by them or addressed to them.

b. Create Posts, Milestone etc. 

b. Update Profile Information

c. Launch and Organize Photos, Friends

If Facebook newsfeed is the place where users see what is happening in their friends life, Facebook timeline is the place where users can organize and manage their personal life and curate what they want to tell the world.

Coming to the question, the next step is to identify if the question is meant for one particular segment of customers or for all the customers. Let us assume it is for the entire customer base of Facebook. Retention metrics consists of measuring behaviours of users in timeline and the success of that can be measured as follows - 

a. Number of timeline page visits/user/day 

b. Median time spent in the timeline/user/day (Measure active time spent in the application. Active Time is the time spent where the user has scroll, clicked or performed any other activity in the page for greater than 1 minute) - Look at 20 percentile, 50 percentile, 90 percentile and 95 percentile metrics for this would give a sense of how long users spend in the timeline page and track retention for the Low, Medium and High Users of Timeline 

c. % Time Spent in Timeline vs Rest of the application

d.  Posts Created via Timeline/ Posts Created via Newsfeed (Measure this metric by each kind of post - video, life event etc.) 

e. Comments, Likes , Shares from Timeline per day

I would prioritize b, e, d, c, and a metrics since those metrics are the key indicators which denote positive behaviours of Timeline users.

Answering this Facebook metrics question, at least in my mind, should start with the following. 

Follow Up Questions First

Italics are assumed answers.

1. Can you define the Facebook Timeline? This is the feed based section of a user's profile, which allows visibility into a chronological timeline of what has been happening with that user on Facebook. This does not include the "peak" feature when you hover over a user's profile. 
2. Why do you want to measure retention? Is this for user growth? Is this for a specific segment's adoption of the timeline (Eg. Workplace by Facebook? Going to assume that there is no specific reason for wanting to measure retention, and we want general growth. 
3. Am I measuring other related metrics on the Timeline? Let's assume no, we only care about retention for now.

Timeline Goals

• First let's outline the goal of the Timeline. This feature aims to add personalization to Facebook, giving users a literal wall which they can paint with whatever they want as a way of expression. The timeline should provide both quick and in depth information into a user, with a focus on quickly accessible, but stalkable information. 
• Many Timeline posts are expandable, allowing users to click in and dive into a post. The main use case for a user visiting a Timeline is to keep up to date with that person/page, and is typically not to browse through the entire history of the Timeline. This mostly comes from personal experience and anecdotal evidence, but I feel that typically a full scroll through someone's Timeline is not the main use case. Most of the time it's when we have not caught up with a friend before, and just want to check in with them from the last time we interacted. 

Metrics

1. timeline visits per user per day
2. time spent on timeline per user per day
3. time spent on timeline / total time spent on Facebook per user per day
4. timeline interactions per user per day

For measuring retention, I am aiming to both grow my slice of the pie and the entire pie. These metrics give a good simple overview of how users utilize the Timeline and how it relates to the Facebook product as a whole. Metrics are ranked in order of importance.

1. Metric 1) measures simply total time spent on the timeline per user. This is simple aggregate level metric, grouped by user, gives us a glimpse into broadly how much the timeline is viewed. I purposefully did not include time spent as the top metric, as quick viewability is more important as stated above.
2. Metric 2) is more focused around the stalk-ability of Facebook. Facebook is often used as a platform to retrieve or lightly online stalk others information. This aspect of the Timeline, while not the main goal, is still a very important metric to understand how people are using it. 
3. Metric 3) measures cannibalization versus the rest of Facebook. If we want to grow Timeline retention, we also want to grow this as a proportion of time versus other pages on Facebook. This metric does not measure page visits, as there is no singular page visit metric which is measurable and standard across Facebook. This is mainly due to the idea of infinite scrolling on Facebook, where visiting the News Feed with one visit could take hours of time. 
4. Metric 4) measures user interactivity. As is the case with any product, how users not only visit but interact with it is a strong signal of how they feel about the product. 

User Surveys

Outside of the metrics mentioned above, which are passively collected, I would also utilize active collection of an basic score to determine how people feel about their timelines. This would be a simple survey, sampled across all Timeline users for how they feel about the Timeline. "How to do you feel about the Timeline from 1-10?" This score is a simple way to collect user feedback to understand how users feel about the Timeline. This question would then be followed up with an open ended input to allow users to elaborate on their answer. 

Thanks for the question. I have a couple of follow-up questions: 

1. WHY? Why do we want to measure retention - are we concerned that people might be leaving the platform? Or is this part of a bigger strategy for something else?

Answer: We just want to understand our user behavior and make them spend more time on FB.

2. Thanks. Please give me a couple of minutes to think thru this question.

Think:

What is the question trying to test? 

• Knowledge of Metrics and Measurement
• North Star Metric and Guardian Metrics
• Critical User Journeys
• Product Intiution
• Come up with multiple top line metrics and decompose general ones into specifics
• More than one way to solve the problem

Answer Flow:

Define CUJ

Define what we are trying to measure?

Connect CUJ with measure

Multiple ways to measure

Isolate to one way - decompose it into sub metrics

Provide guardian metrics

Start Answer

We'll start off by thinking about the feature itself - i.e. FB timeline, then we'll define retention and what it means to retain users. After that we'll connect the user journey and try to come up with ways to measure retention.

The feature - FB Timeline. The FB timeline is a rolling wall of updates where each user can post something (image, video, text etc.) and other users can see it. There are two primary modes for the timeline - your personal timeline and the social timeline (what everyone else sees). I think we should first focus on the social timeline for measuring retention. Is that okay? (Answer: Yes).

Okay now let's try to think about what retention is. I'd define retention as the total time that a user spends on a feature. i.e. the point in time in between them first interacting with the feature and last interacting with the feature. 

Before moving forward let's take a step back and think whether there are any other ways to define retention. Another way to define retention would be the total number of users who are active at any point in time. i.e. the total users that are being retained due to the timeline feature.

Or at a higher level, it could be the time that a user spends browsing (not actively interacting) with the timeline.

I'm going to stick to the first definition for now i.e. time between user's first interacting with the feature and last interacting with the feature.

Let's apply this definition to the FB timeline. So we need to measure two things - First interaction and last interaction.

First interaction can be - first time user sees another user's post. first time user likes or does an action on the post. first time the user posts something. To be broad, I'll take the first time the user performs any action on the timeline - like, click, comment etc. 

Last interaction - this is tricky. Last interaction can be the user unsubscribing from FB. But that's too harsh. Let's define last interaction as a period of inactivity from a user for 1 week. 

We can use this metric Retention as the North Star metric - i.e. a metric that guides us. But we should also try to have some guardian metrics to make sure we are not optimizing too much on one behavior. 

What are the things that are important for a robust timeline? People should post, people should share others' posts, people should post things that increase the retention time of others, people should click on ads shown in the timeline. All of these could be our guardian metrics. i.e. when we optimize for retention, we should also make sure the guardian metrics don't suffer dispropotionately.