You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that % of people that add addtl. profile information increased by 8%. However, 7 day retention decreased by 2%. What do you do?
+3 votes
2.8k views
in Problem Solving by (15 points) | 2.8k views

5 Answers

+3 votes

You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that % of people that add addtl. profile information increased by 8%. However, 7 day retention decreased by 2%. What do you do?

Let's start from the WHY behind the change- likely that you were implementing this change to improve retention ; let's proceed with the assumption that this is a social app where profiles play a critical role.

An increase of 8% for sign up flows is a significant increase in number of people completing profile information - I would ask how was this implemented because from experience I know that every time we add a step it introduces a 3-4% drop off. 

My assumption here is that this is an added pop up screen during sign up which is causing a drop off in total number of users completing sign up successfully since they drop off on the profile info screen.

*Interviewers nods yes*

Also I would recommend changing the way we measure success of the A/B test , let me tell you why - consider the following scenarios

In case A for every 100 people signing up 50 people ended up signing up of which 20 completed profile information

In case B ( winning variant ) for 100 people signing up 48 people ended up signing up of which 22 complete their profile 

So the blocking screen is causing an overall drop off which reduces D7 while increase number of people completing their profile

I would re-configure the experiment hypothesis to " users with better profile information have higher retention than users who don't - how can i increase in the number of users filling their profile information on D0 ( primary metric) while increasing/not affecting the successful sign up rate ( secondary metric that doubles up as a kill metric) "

if my goal is to increase profile information of new sign up I would focus on passive methods  ( push notif, in app pop ups, incentives)  POST users successfully signing up so as to negate this drop off

if my larger business goal is to increase retention I would reduce the steps of sign to increase successful signup and focus on passive methods ( push notif, in app pop ups, incentives) POST users successfully signing up so as to increase total user successfully signing up and help improve D7

by (135 points)
+2 votes

Clarifying questions: 

  1. What was the new sign up the flow and how is it different than the old sign up flow. My assumption is the new sign up flow asks for more user information, thus the increase in the % of profile information. 
  2. What was our goal? Was it to increase profile information? What was the acceptable counter metric decrease? Is the result within range? 
List pros + cons of both metric
  1. Increase in profile information - the more data we have, the better our recommendation and personalization system and the network becomes more valuable to the users. Cons, depending on the types of information we ask and require of the user, the user may have a different level of comfort and privacy concerns
  2. Decrease in 7 day retention - if a user does not come back in 7 days, this is not great for the platform, however, I want to also consider whether 14 day or monthly retention has decreased, perhaps the new user no longer needs to come back on 7th day and add a profile pic or other information. 
What would I do to validate my hypothesis
1. One hypothesis is 7th-day retention may decrease due to the new user no longer come back to fill out additional information, I would then compare 14 day and 30 day retention to see if it decreased. 
by (32 points)
+3
I agree with the clarifying questions here, it is a must before making assumptions. I also like the structure and approach of the answer.

Merging both answers above would be ideal in a way tho.
0
1. So far this the most logical answer here. I would clarify that if there was a CRM campaign in place to "bring back" users to add more profile info earlier, is so, then the current drop in retention is well justified. E.g. of profile pic above is a good one...so it would have materialized like me getting an *email* saying..."hey buddy good job getting started on the profile and btw we noticed you didn't upload a pic so when you get a chance, do stop by and let us see your smiling face"

Seeing such an email within first 7 days, is definitely going to drive up 7 day retention

2. Another possibility could be having asked additional information in the sign up flow which
"turns off" users from coming back. So for e.g. if I get asked a question which goes like "hey you do know that we can bring in all your contacts if you want from facebook with the email address you provided, should we? Y/N" Likely the result of this question would be getting "additional information" N as an answer which would turn off folks who'd likely think this social network is creepy as heck too.
+2 votes

Hi, 

First of all, I make to make certain assumptions or inferences here:

1. The % users who filled addl information increased by 8% which means that providing additional information was optional.

2. The overall decrease in retention includes people with both the variations (control as well as variation version)

Now, I want to analyze further which user segment retention rate is affected and would suggest strategy to address the problem - 

1. People who did not provide additional information and bounced off Landing Page or in subsequent usage -  

I would verify these users retention rate between control version and variation version. Ideally it should be same. If it is not, then we should look at user profiles at each variation to seek the differences.

2. People who provided additional information and then bounced off Landing Page or in subsequent usage -

If retention number is lower here, then we need to focus on providing relevant customized offerings basis additional information and check retention rate in subsequent 7-day period.

3. People who signed up but bounced off additional information page and never made it to the landing page - 

If % of users is higher enough to reduce the overall retention rate, then we should look at alternative placement of additional information or content of additional information (this can be gathered by analyzing which page and field had the maximum bounce-off rate). May be we can ask user to provide information on 2nd or 3rd usage etc., as we must give a chance to user to use our application before exiting it. 

by
0
I love how you broke it down to three user groups. Nice work.
+1 vote

Assuming no other change was implemented that could have affected these metrics, lets start by breaking down the question. First piece of information is that additional profile information increased by 8%, which is desirable. We do not have information about what exactly we are asking the user to provide, so lets assume that the additional profile information consists of user's interests and experiences. And second info we have is that 7 day retention decreased by 2% because of this change.

The objective of getting more profile information from the user is to find better content and connections related to the user's interests and experiences. If we are not able to use the additional user info to find more relevant and engaging content for the user, then this effort is a waste. So, our primary focus should be to make the best use of this info to improve the experience for the user. This would have an effect on the churn rate too and generate positive word-of-mouth marketing for the product.

Now lets try to understand the possible reasons for the decrease in user retention. A thought that would come to our mind is that probably we have made the process so elaborate and complicated that user did not even complete the process and chose to leave the app. But since retention period metric tracks the user after they have completed the signup process, we can rule out this reason. We ought to be more specific about what kind of information we are asking from the user. For privacy reasons, many users will not be comfortable in sharing more personal info. Make sure that users are satisfied with your product's security and privacy policies and convinced that they have full control over how your product uses their data.

In summary, we need to do the following to improve retention -

  1. Create a more engaging experience for the user by finding and displaying relevant content based on user's interests.
  2. Communicate your product's compliance and privacy standards and make sure they trust you with their data.
by (32 points)
0 votes

<!--[if gte mso 9]> <![endif]--><!--[if gte mso 9]> Normal 0 false false false EN-US ZH-CN X-NONE <![endif]--><!--[if gte mso 9]> <![endif]-->

<!--[if gte mso 10]> <![endif]--><!--StartFragment-->

C: Clarifying questions

How are we “encouraging new users to add more profile information”, is it another step in the profile process? Is it an optional step or a mandatory step? Is it a pop up that is removing the user from the main workflow?

What is the objective of the business? To increase retention? To increase user engagement? To increase the amount of targeting data to sell to businesses?

 

An increase of 8% from sign up with additional information is a significant increase assuming that the total number of signups have not decreased..However, for the 2% retention drop, I would first look at the activation %. Has the total Activation % dropped from when the “encouragement” was introduced? Is the retention counting from when the user “signed up” or when the user “completed their profile?” If there was a 2% drop when the retention was counted from sign-up, one thing we can hypothesize is that the # of sign ups increased but the # of people who fully completed their profile to activate their accounts stayed stagnate or dropped. Or, it could be that the # of sign ups stayed the same, but the # of activated devices dropped.

 

Assumption. We are seeing a higher drop off flow of users who are not completing their profile because additional steps are demotivating the users to finish sign up. Retention is measured from Activation not sign-up.

 

A/B testing should measure three aspects

1. Total number of users signing up

2.  Number of users who finish completing their profile

3. The number of users who finished completing their profile still using retained. 

 

What to do: 

(Preliminary) Change the way the data is measured

% of users who completed their profile with additional information from signing up

% of users of retention counted from activation among the people with the additional profile information

 

 

VS

 

% of users who completed their profile without additional information from signing up

% of user retention from activation among people without additional profile information (from before the feature is introduced)

 

If the additional information is optional, and activation is completed, then ideally retention should be the same, and we should focus on moving consumers from without additional information from signing up without additional information to additional information.

 

If the retention drop with the measure adjustment is happening in one of the two categories, we need to look into the user profile and understand if there is a reason among engagement and retention efforts that changed.

<!--EndFragment-->

 

by (41 points)
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
To avoid this verification in future, please log in or register.
Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
To avoid this verification in future, please log in or register.