How should you decide whether to autoplay videos on the newsfeed? What are the metrics for success?

How should you decide whether to autoplay videos on the Newsfeed? What are the metrics for success?

 

Clarification questions

1. Should we focus on videos on the news feed or also videos in the FB watch section? (only Newsfeed)

2. Autoplay w/o a sound? (Yes)

3. Does autoplay w/o mouse hover once the video is visible? (Yes)

 

Structure

Mission -> Product -> Current Metrics -> Concerns -> Summary\Recommendation

 

Mission and product

FB’s mission is to (1) make the world closer together (2) build tools for communities.

The Newsfeed is the main component at FB and performs as the homepage; It presents content created by friends and others and additional features as friend recommendations. 

There are three types of content: text, images, and videos. 

There are three categories of videos 

1. Videos posted by friends

2. Videos posted by not friends

   1. Videos with friends’ reactions 

   2. Recommended videos

3. Ads

The Newsfeed is a mature product, and the relevant goals at this stage are engagement, monetization, and retention.

 

Why should we auto-play videos?

• Auto-playing videos can intrigue users with the content.

• Remove decision points.

• Increase consumption of videos.

• Increase engagement with videos.

• Increase time spent on site.

 

Current Metrics

Direct Metrics (Video Player)

• # of videos seen and by how many users.

• The ratio between videos completed \ videos watched \ videos exposed to

• #/% types of engagement with videos

• # of videos uploaded 

• # of users uploading videos \ Average uploads per user

• Performance (devices \ internet usage \ speed)

• Time spent on the Newsfeed.

 

Higher Level Metrics (Newsfeed)

• Feed’s composition: Posts \ Images \ Videos

• Time spent on site \ depth of the engagement (how much content seen)

• Original content created and # of content creators.

• # of reactions as share, tag, comment, like, etc.

 

High-Level Metrics (Facebook)

• Average # of content seen during a session.

• Feed’s composition: content created by friends \ content not created by friends

• Revenues

 

Concerns

• The recommendations algorithm will need to be adjusted since we will need to define better what it means that a user watched 5 sec \ 15 sec \ 50% \ entire video.

• Auto-playing videos will probably attract users’ attention and can affect on:

  • Feed’s composition: Posts \ Images \ Videos.

  • Time spent on the Newsfeed.

• Content Creators - Since videos will receive greater attention, content creators can create more videos and fewer text and image posts.

• Newsfeed’s Composition - Today, most of the videos are not made by friends; having more videos on the feed can affect the Newsfeed’s composition and present less content created by friends and, by that, going against FB’s mission of making the world closer together.

• Engagement with videos - I assume that there is less engagement with videos that friends didn’t create, which can cause a decline in engagement.

• Time Spent on Feed

  • We need to track that users don’t spend “too much time” watching videos,

  • Spending more time in the feed can reduce engagement with other products such as Marketplace, FB’s Stories, Watch, groups, etc.

  • We need to track if we are gaining more revenues rather than in other products.

• Auto-play videos might be considered harmful.

 

Summary and Recommendation

Consuming more videos can affect users’ behavior; it can increase usage metrics and revenues, weakening relationships with friends.

 

I recommend exploring it with A\B test over time and understanding the overall impact on users and Facebook’s goals.

I'd first like to clarify what Autoplaying videos on newsfeed means - Assuming it is the ability to automatically start playing videos as the user is scrolling through their Newsfeed. This is currently with audio off. 

Before getting into success measures for this feature, let me lay out FB's overall Mission and goals for context: 

• # videos watched / user / session, D, W, M (CTR for videos) - segment by Activity Level Low, Med, High (similar to above) 
• Time spent watching videos / user / session, D, W, M - though this is the north star, it is more of a lagging indicator. The purpose of this feature is to get users to "view" a video and as a result, they might spend more time watching videos
• Total Video Watch Time on Newsfeed / D, W, M
• Avg length of video watched / user
• Overall engagement level with videos per user (reactions, comments, shares)
• Stickiness: DAU/WAU 
• Segment by Type of Content, Tenure of user (new, existing), demography and geography
• Avg session length / user

• Interaction with consumers that react/comment on a post might improve overall # of shares, reach of a posts, resulting in higher satisfaction for the 'Creators' encouraging them to post more video - so I might prioritize getting Content Creators to interact more with their audience  
• Getting users to post at least 2 videos a month

• % of businesses that are paying for Ads 
• % of businesses that posting video Ads 
• Revenue realized per Ad 

My idea of how to best answer this Facebook metrics question.

Feature definition: Autoplay plays a video without touch/click when it is in the focus area of the screen( mid of the screen).

Goal of the feature: Primarily the feature serves two goals:

1. Reducing human effort to view the video

2. Increases user engagement with the video

Customer Journey:

A user is going through and scrolling his news feed. When he reaches a video, the video may start playing itself with/without sound. 

If the customer touches on the video it starts to play with sound.

To skip the video the customer should scroll his news feed.

Metrics: 

To evaluate whether to autoplay videos or not, we should look at two factors:

 

1. What is the user scroll behavior when he sees autoplay videos?

2. How frequently a user clicks on autoplay videos?

Coming to the first question, if a user stops scrolling news feed in certain percentage of autoplay videos( this can come out of product data), we can assume he is watching them. So, he is an engaged and interested user and we should show him auto play videos. if this is not the case, we should consider second level question.

Coming to the second question, how often a user is clicking on autoplay videos can be a second level metric.

If the user is clicking and then skipping, this is an indication of lack of content or relevance to the user. To such a user, we should show relevant but different videos.

if a user is clicking and not skipping for a certain time( coming from data), he is highly engaged user and can be showm more relevant autoplay videos even sponsored content.

If a user is not clicking and scrolling through, he should not be shown autoplay videos if he shows this behavior for 7 days/ 7 sessions

Coming over to the metrics part, we will evaluate the following

 

 

User Stage	Metric	Priority
Acquisition	No. of New Users of the product	Medium
	No. of New Videos uploaded	High
Enagagement	No. of Videos watched	High
	No. of Videos clicked upon	High
	Avg Time per video watched	High
	Total time of videos watched	High
	No of videos liked, commented, shared etc.	Very High
	No. of Videos uploaded	Medium
Retention	Users who watch or click on videos at least once every session: retention rate	High
	users who stop interaction:churn rate	High
Monetisation	No. of ad videos watched/clicked	High
	ARPPU	High

 

 

Thus to sum up, We will focus on user intercation with auto play videos to determine if to show him the autoplay videos or not.

We shall focus on metrics which have been marked as High, very High in the above on priority.

 

How should you decide whether to autoplay videos on the newsfeed? What success metrics?

 

Define the objective 

What would be the impact of auto playing videos in the feed. Videos in the feed are a way in which users express themselves and share information in their community. 

Question:

1.  Does autoplay include sound ? 

2. Can users set (easly) a way to revoke the autoplay?

3. Do creators choose to autoplay ?

Product Objectives  

FB overall mission is to bring the world closer together by creating interaction between community members. Videos are a great way to share information and generate interactions. Autoplay of a video is a great way to bring users attention to the content being shared and generate interactions b/w users. However this might have a negative impact as be viewed as invasive and violates users privacy as it is played on the user’s phone without his or her consent. 

Product features  

1. Video player screen 

2. Mute 

3. Do not show again (like this) 

4. Setting - do not autoplay 

5. Posts 

6. Likes 

7. Shares

8. Comments 

Target Audiences  

1. Uploaders 

   1. Casual 

   2. Professional 

2. Viewers 

User Journey / interaction with the feature  

Content Creator

1. In the post module - add video 

2. Setting autoplay 

3. Description, tags,  etc…  

Content Viewer

1. Goto the feed 

2. View movie

3. Listen/ Mute 

4. Stop /Start 

5. Remove video

6. Report video 

7. Change setting of autoplay 

8. Social aspects 

   1. Share

   2. Comment

   3. Like 

9. Discussions in groups about this feature 

Metrics that measure each step in the journey  

Creator (we can look as the creator segments in the analysis) 

1. Awareness / Acquisition  

   1. Hard to measure from the user interaction, clicks on the video upload button 

2. Activation 

   1. # UU that upload videos 

      1. First time uploaders 

   2. # Videos 

3. Engagement

   1. # Circular effect -  those that upload are they more engaged with others’ videos 

   2. Sum time spent 

   3. # videos/UU 

4. Retention 

   1. # of returning users in given time (D/W/M) 

   2. Churn (cohort analysis) 

5. Monetization 

   1. # users that share (for pay) videos 

   2. # Rev per video 

   3. Sum of paid videos 

6. Referrals 

   1. Positive sentiment 

   2. Share outside on FB 

   3. Share in groups ??

Viewer 

1. Awareness / Acquisition

   1. # UU in the feed (look at delta)

2. Activation 

   1. # video views 

3. Engagements 

   1. Listen

   2. Mute 

      1. UU

      2. Actions 

      3. Actions/ User

   3. Stop

   4. Play 

   5. Remove video

   6. Report video 

   7. Change setting of autoplay 

   8. Social aspects 

      1. Share

      2. Comment

      3. Like 

         1. Dislikes 

   9. Discussions in groups about this feature 

      1. # words

      2. # sentiment 

         1. Positive 

         2. Negative 

4. Retention 

   1. # UU returning 

5. Monetization 

   1. # UU viewing paid content

   2. Sum of Revenue 

Creators 

 

Viewers

 

Prioritized metrics 

1. # videos 

2. UU users

3. # interactions (likes, share, comment)

4. Sentiment analysis 

5. Churn 

6. Revenue metrics 

7. Full video played 

8. # report videos (do not show again)

9. # disable feature/ viewer 

How to collect the data ?

A/B test this feature. A/B test can be on multiple levels.  

1. On / Off (with an option to turn on) 

2. On with high discoverability of off setting 

3. With sound 

4. Not all (might be a measure of the impact) 

 

Summarize 

This is a measure change to feed that can have a massive impact on user engagement, retention and revenue. To some users, this might be a delight and to others it will have a major negative impact. Given that this feature is superimposed on user’s feed it’s impact can be measured relatively quickly and the stats (metrics) above 

 

 

 

 

 

1. As I understand the newsfeed feature, users see a newsfeed, usually there is infinite scroll, and as a newsfeed item comes up it can have a video and the video can auto-play or not with sound on/off.

2. Assuming I am an FB PM, FB mission is to bring people together and the goal of newsfeed is to inform user of information relevant to them

3. A typical user journey may be that a user lands an a page w. newsfeed, they scroll until they encounter a post with a video, that video can play automatically or not and with sound on or off, the video can also be in an ad post. When the user encounters a post with a video they can choose to engage with it, press play or just watch it, add a reaction or comment, clink on a link, go to posters page, re-share and continue on in the news feed.

4. Some metrics to keep an eye on could be:

- time spent on newsfeed

- # of posts per session

- # of reactions/comments for video posts

- re-shares

- engagement with video posts (click play, pause, turn on/off sound)

5. Keeping in mind the broader facebook mission and goals of newsfeed, the most impactful metrics to keep track of would be:

- time spent on newsfeed, # of posts viewed per session

- reactions, comments on video posts

- actions taken on initial video state

- ARPU related to ad video posts

I would also segment these metrics based on video posts or not, % video in feed content, ad vs. non-ad posts.

6. I would prioritize measure how users interact with initial state of video since that is pretty low effort and would be pretty effective at communicating user preferences with a high level of confidence. I would A/B test playing the video with sound off (to avoid disrupting users) against not playing the video, and look at % of instances where users alter video initial state by clicking play or pause, and also secondarily make sure there were no large changes in # of posts viewed, avg. time per video post, # of likes and comments, to measure engagement.

7. To summarize, as a FB PM, based on the FB mission and newsfeed goals and with the user journey of viewing and engaging with video content there are several areas to measure during the user journey but I would prioritize how users react to an initial state of a video while measuring resulting engagement both with video posts, ad/not, as well as newsfeed at large. Depending on how clear the results and how confident I am in the data I might also look into lettering users set their own preferences as well as persisting their last chosen state, such as keeping playing with sound if the last video was played with sound.

Assuming, I have to evaluate this as a PM for FB newsfeed.

Newsfeed as a product has a goal of helping users find the most relevant content via the newsfeed.

Success for news feed will lead to greater discovery of useful content and thus stronger user engagement. We can measure useful content discovery by looking at the bounce rate of users. 

In order to decide if Newsfeed should have autoplay, I need to understand its impact on newsfeed success and fb mission. That is, does it lead to stronger amount of connection between users, entities on FB and improve their retention on the platform.

Primary Key Metrics That Represent the above are:

1. DAU, MAU for FB App as a whole [Main success metric]

2. CTR and bounce rate from the newsfeed clicks

3. Engagement on newsfeed (comments, reactions, shares)

4. ARPU (Impact on advertising revenue)

In order to evaluate the impact of this, I will create an AB test:

1. Treatment - will have a autoplay of videos on the newsfeed

2. Control - Will have the non autoplay for videos

I will work with data scientist to setup the sample sizes and run the experiment until I get statistical significance for impact on above mentioned key metrics and make a decision.