You are PM of FB Login used by 3rd parties and you notice successful logins metrics is down on a Monday morning. What will you do?

Question

You are PM of FB Login used by 3rd parties and you notice successful logins metrics is down on a Monday morning. What will you do?

Asked at Meta (Facebook) over a year ago

2.8k views

Asked at

2.8k views

How to answer Metrics questions

Interview Guide

Top Meta (Facebook) interview questions

What is your favorite product? Why?89 answers | 263k views
How would you design a bicycle renting app for tourists?62 answers | 82.5k views
Build a product to buy and sell antiques.54 answers | 66.8k views
See Meta (Facebook) PM Interview Questions

Top Metrics interview questions

How would you measure the success of Facebook Likes?40 answers | 61.3k views
What metrics would you use to measure the success of the Save feature at Facebook? 39 answers | 87.2k views
How would you measure the success of Gmail?23 answers | 17k views
See Metrics PM Interview Questions

Invite members
Invite by email

Add another

Chetan Channa · Answer 1 · 2021-06-18T04:59:32+0000

The question says that the sucessful login metrics is down on a Monday Morning.

Let me first mention the understanding of the product

Facebook login is a product through which users of facebook can use their facebook credentials to login to other 3rd party websites which have integrated this feature. Once the user uses this product, facebook authenticates the user and shares a unique authentication token.

[Interviewer]: You are right.

Let me first clarify the problem.

This is a trend on every Monday or just this Monday, I want to know if this is a trend or one time thing? [Interviewer]: It happened only on this Monday and it is a one time thing. Haven't observed any regular dips on Monday.
To clarify the metric, the number we looked at was the absolute count of sucesful logins and not like a % ? [Interviewer]:Yes, we are talking about absolute count of logins.
Also, we have verified that this is not a reporting/data issue. [Interviewer]: You are right, this is not a data/reporting issue.

Let me know mention the approach I will use

Since # of logins is actually dependent on number of users attempting to login, we first need to figure out if attempts went down and/or user attempted but couldn't login. User flow is

User visits a 3rd party website(A)----a%--->Clicks on Facebook Login(B)----b%----> Successfully Login(C)

Now the count of succesful login can dip because :-

Less # of users visit the 3rd party website
Less number of users click on Fb login (a% went down)
Less number of users were able to login even after attempting (b% went down)

Let me first look at b% which is

% of succesful logins = # of successful logins(B)/Number of logins attempted(C)

-----------From this point on depending on inetrviewer, the case can take 2 direcrions. For sake of this answer, I am covering both---------

Case 1: If '% of successful logins' hasn't changed much that would mean that the dip in logins is because of decrease in number of logins attempted

Case 2: If '% of successful logins' has dipped, it means that the attempts are similar but users are not able to sucesfully login mostly because of a tech failure.

Other use cases are very unlikely

---------------------------------------------------------------------

--------Case 1---------

Has the % of successful logins changed ? [Interviewer]: No

Since the % of succesful logins hasn't changed, it means that the attempts of login via Facebook(B) has decreased. This would mostly be because of 'external' factors.

Let's formulate some hypothesis around it.

Is the dip specific to a region ? For this, I would look at the login attempts broken down at country level. If a particular country or a region shows a dip then the dip could be because of infra issues and or holiday etc in the region/country. [Interviewer]: No, dip was across most of the countries.
Was there any major negative news with respect to Facebook/privacy published or gone viral. [Interviewer]: No, nothing that sort of happened. There was no negative mention of facebook apart from usual stuff over the weekend.
Was there an issue with a major product using Fb login, For eg say bbc.com is the biggest user of Fb login and it had issues on Monday morning ? I would like to check if requests for login decreased from a particular product [Interviewer]: Indeed, there was an issue with site x which gives us 15% of the logins and it went down.

----------------------------Case 1 & Interview Ends--------------

---------Case 2-------

Has the % of successful logins changed ? [Interviewer]: Yes, it has decreased.

So it looks like that the users are attempting but not able to login. [Interviewer]: Yes

This would mostly be due to technical reasons which could be both internal or external.

Let me first look at internal reasons

Did we roll out any new release or increased roll-out of any experiment ? [Interviewer]: No, we didn't
Was there any infrastructure issue within Facebook, which prevented the users ?[Interviewer]: Not that I am aware of
Was dip specific to new or old users ? I am asking this because, for the first time users have to give consent on privacy, if those have changed or that flow is broken.[Interviewer]: Didn't notice any specific difference, there was a dip in both.

Let me now look at external reasons

Was there an issue with a major product using Fb login, For eg say bbc.com is the biggest user of Fb login and it had issues on Monday morning ? I would like to check if requests for login decreased from a particular product [Interviewer]: Not that I am aware of
Was dip specific to a platform ie. Web, Android, iOS or any other ?[Interviewer]: Yes, the dip was majorly seen on iOS.
In that case, I am aware that Apple roled out a new iOS release few days ago, may be people downloaded it over the weekend. Was this dip specific to iOS XX? [Interviewer]: Yes, indeed this dip was specific to iOS XX

In that case, may be there was some incompatibility between the releases. So I would request my QA team to do a thorough testing and also analyze the issue from internal logs

-----Interview ends--------

Nimasha Jain · Answer 2 · 2021-06-17T18:39:48+0000

Facebook’s 3rd party logins is place where other website use FB authentication layer to provide their Login support. The FB login credentials would be used as tokens for login into their respective websites. Metric of successful logins for 3rd part are down. First of all, I would like to understand the following queries

C: Which metricseactly ; Is metrics of login into the 3rd party or login into the Facebook?

I: Success metric on Facebook login is working fine. There is no notable change into it.

C: Is it only for specific geography?

I: No, Its global observation.

C: Is it for any particular handheld device or on operating system?

I: As such, we haven’t seen into the data for same but let’s assume it’s same. There is no segregation.

C: Is there any decrease in metric of users trying to login or they are same?

I: Number of users trying to login into the

C: Is there any one particular 3rd party facing the issue?

I: There could be. Let’s fetch the info. Meanwhile go ahead with troubleshooing.

C: I would like to transverse user journey. User opens website. User click button Sign in using facebook. Now there would be 2 scenarios

Scenario A: If user is already logged using Facebook. He will automatically login into the website. In case of any Authentication APIs of FB are intact, this scenarios is less likely to get some errors. I would like to check the following metrics

# Of signins on Saturday and Sunday on Facebook

# Of signins on 3rd party using Facebook.

# Of errors on Facebook Signin API.

Scenario B: If user is not logged in FB previously, they will be asked to enter FB credentials. In case the screen appears but users are not able to click submit button or API collaboration of 3rd party to Facebook Login Api is failing. Here we need to check the error of API and 3rd party.

Lastly we also need to check, there is no leak in privacy concern of users, no cyberattack or hot fix rolled out busted in production.

Summarizing

I would check error, success metrics. I would also be browsing the external factors along with checking the internal factors of metrics and errors.

spk · Answer 3 · 2021-06-16T14:58:38+0000

Product Understanding:

Facebook login is a system through which users can use their Facebook accounts to log into 3rd party apps and websites instead of creating a new account on that service. Users will receive a notification that they used Facebook login to log into a 3rd party app or website. Is that correct?

Clarify:

If it is a first time user or a returning user who logged out of his previous session, they will be shown the login page to login using Facebook login

So how do you measure a successful login metric? Since a “successful login” is when a user logs into a 3rd party apps using FB login successfully, I am assuming that you are tracking # of logins on a daily, weekly and monthly basis? Yes

Okay, so the number of logins into 3rd party apps using FB login is down. I have few more questions to understand more about the context so that I could formulate my hypothesis and go deeper into the problem

Some more clarifying questions to form a theory:

Since this is observed on a Monday morning, should I assume that it is a sudden decline and not a gradual one? YES
Was the decline specific to a region or all over the world? It’s not specific to a region
Was the decline specific to a particular platform like ios or browser or android? If it is specific to a platform, I want to calculate the number of logins done using each platform and compare it with the total number of logins. If there is no change, then it is possible that some users are just logging through a different platform. Let’s say it is total logins and not platform-specific

Okay so we observed a sudden decline in the number of logins into 3rd party apps using FB login and it is not platform-specific and it is not region-specific

I want to make some hypothesis regarding this and I will mention how I will test each hypothesis to get to the bottom of the problem

Hypothesis

Users are not preferring to login using Facebook because of privacy concerns. Test: The user behavior would be gradual and not sudden.
Users are trying to login but they are not able to login. They are getting authentication errors. Test: There might be some infrastructure changes made yesterday. I would check this with the infrastructure team
There was news of a cyberattack that compromised several users data Test: I would check with the security team if there were any reports of cyberattacks news
Were there any feature releases with respect to Facebook logins yesterday that made users wary of 3rd party logins using Facebook? Test: I would check back with the product team and if there are any feature changes that lead to this decline, I will look back to the goals of the a/b test that lead to the launch of the feature. There can be a possibility that in this case even though the number of logins through 3rd party apps is reducing, the number of Facebook deactivations that happened due to the feature launch didn't significantly change or might have even significantly reduced. Since users are not completely churning from Facebook, the value that is being created in a long-term perspective with respect to this feature launch is more than the value lost in launching this feature. If the Facebook deactivations are also significantly increasing, then I would discuss with the stakeholders to roll back the feature

Summary:

I came up with a metric that measures successful logins and formulated my theory to test. I would check for possible infrastructure changes and any reports of cyberattacks. I would also look back to the goals of the feature release to understand why the feature release happened even though there is a negative impact on the success metric.

shrutc · Answer 4 · 2021-07-08T10:15:04+0000

Facebook (FB) logins for third parties is a feature which allows other applications to allow users to login via FB to avoid having to filling up tedious sign up forms. This also allows the thrid party read-only access to certain user data such as name, profile pis, list of contacts from FB which they can make available on their own platforms to users.

The possible metrics for FB Login used by 3rd parties:

Usage Metrics:
# of logins through third parties per day/ month/ year
# of logins by category of third party
# of logins per third party
% growth in third party logins per quarter/ year

Awareness Metrics
# of new third party registrations per day/ month/ year
% increase in new third party users

Engagement/ Retention Metrics:
# of repeated logins per third party

Revenue Metrics:
% Revenue generated per third party for logins
% Revenue generated through third party logins

Clarifying questions:

Did an internal activity such as new feature additions or hardware/software change affect the metric. - No

Is there a new competitor in the market who is providing third party apps with better access to user profiles - No

Is this occurring globally or is it specific to a location? - let’s assume it’s a generic problem happening in all locations

Has the traffic to Facebook reduced in general? - no

Has there been a drop of logins from a particular third party? - yes

Is the third party not doing well and seeing a decrease in usage? - no

Has the third party stopped using Facebook as a login option? - yes

Is it known why they have stopped using FB logins - no

Problem: A particular third party has stopped providing FB as one of the options for logging in to its users.

Possible pain points:
We need to assess possible pain point/ improvement areas for FB Login used by 3rd parties.

1. Security concerns of users about personal data being transferred between two platforms and hence they prefer to register directly with thr third party
2. Limited access to user data and hence third parties may as well ask users to register with them and acquire a new customer.

Next I would find solutions for each of the problem areas and try to prioritise them based on the ease of implementation and how much it would actually increase adoption by third parties.

Summary:
To summarise, when I first see a metric drop I will first narrow down to the possible root-cause based on internal and external factors. Once I am aware of the root cause, I will brainstorm possible solution options along with design and engineering teams and then prepare a roadmap to implement these based on priority.

Lokesh Kumar R · Answer 5 · 2021-06-27T13:56:49+0000

Product Understanding

Facebook's Login used by 3rd parties, mean that these are the options that one finds while signing-in/ Logging-in at thirdparty websites. They use FB's API and get few information from their profile and setup the profile in their site.

Before jumping onto the question, I would like to clarify a few things.

Is this issue faced by a particular 3rd party site or common across different third party sites? Common across different 3rd party sites

Is the drop consistent throughout everywhere or is it present only at certain places? The drop is present worldwide

Do we see any seasonality in this issue like this happening weekly once or during a particular month every year? This is the first time we are facing this

Is the drop consistent across different types of devices? It is consistent across devices

Is the change a more gradual one or a sudden issue? It happened all of a sudden.

Thanks for clarifying the above questions. This has certainly helped me get some initial ideas on where the issue could be.

So, The way I would like to solve is by dividing the possible places where the issue could be into External and Internal Buckets where External = The factors that are out of FB's control, like Government Regulations, Competition, PR etc..

Internal = The factors that are in FB's Control like Technology Change, Metric Definition or Database related changes etc..

After breaking like stated above, I would like to dive deep into one bucket and understand various reasons related to that bucket and the things that could influence the above issue.

Hope the above structure is okay? (Interviewer Nods a yes).

So from the clarification questions, we got to know that the change has been a sudden one. So, My initial bet would be on the Internal bucket as External buckets influence will always be a more gradual one. For example, if a competition is doing something that is more attractive to the user, You will tend to see a gradual change in your user's behaviour to the new place and not a overnight change. However, I would also want to mention here that Bad PR is one thing that could change things drastically all of a sudden,

So, due to the above reasons, I would jump into Internal Bucket

Internal Bucket:

Was there any change made to our metric definition? If yes, it could be possible that the drop is not actually a drop instead a different number that shouldn't be compared with our prev data. No, We haven't made any changes to the definition

Was there any change made to our Database or reporting tools that we use? If yes it could be probable that this change has mage some issue and the data that we have collected could be totally wrrong. No, We are sure that the data that we are checking with is Accurate.

Oh, Okay. Then, I woudl like to move to the other bucket within Internal Bucket, which is, Tech Changes.

So did we make any change to our API, which could have caused a issue that prevents people to login/signup from FB SSO and inturn reduces Logins through FB option. It could have also increased the LoadTime and thus users exiting from the attempt midway. No, we haven't made any API changes in terms of Tech aspect.

Okay, the other possibility is that we could have made some UI change in the API, meaning, when one clicks FB SSO option, often a new pop comes asking people to sign in to their FB. Had we made any change which could have made the users feel something wrong about the option or could have made it difficult for them to find the button to proceed? No, we haven't made any UI changes in the recent past.

Since, it has been a NO everywhere, I would like to go into External Bucket.

Out of all the possible factors under External Bucket, I would directly reject Competition Factor, Governmental Regualtion issues. This is because, it is highly unlikely to have a competition affect us worldwide that too as a sudden issue. Similarly, Government Regulation could be probable factor had the issue had happened at a particular location.

However, I would like to look into Bad PR as it is one external factor that has the power to turn tables upside down within a day and could have a effect worldwide.

So, My question will be did we face any bad PR news in the recent passt espescially in the recent Saturday or Sunday? Yeah, a news came stating that FB has faced a cyberatack and as a result, more than 300,000 users information got released online. However this is a false claim.

Though this is a false claim, It is is a huge issue, that could make any user think twice before using FB SSO as this means that FB will have access to certain things that they do in these 3rd party sites. This doubt would increase multi-folds when the 3rd party site is a Dating App as people consider this as their most private information.

Yeah, you've cracked the issue. This was the reason that has caused us the problem.

So as a PM, the very next action that I would take will be to publish a clarification that the whole news was a false claim and nothing as such has happened if the same has not been done so far.

So to summarize, the issue was because of a FalseNews which claimed that the FB underwent a Cyberattack and critical information of users got leaked.

prisingh · Answer 6 · 2020-05-21T00:48:54+0000

Clarifying questions:

1.Is this happening for new user login via 3rd party or even for returning users? - New users

2.Is this a sudden drop or gradual? Also when you say down, down by what % to understand the magnitude of drop. - sudden 10%

3. Are we seeing this on a specific browser? Specific platform ? - NO

4. Are we seeing this on a specific type of client and HW combination? such as some specific Samgun phones etc? _ NO

5. Is this happening across all 3rd party apps or any specific 3rd party app? - ALL

6. Did we release a new SDK for login recently that is being used by these 3rd party app? If so, did we change anything specifically in the login process?

7. Did we release a new login notification that lets people know what data FB provided as part of login that is causing people to not allow login via FB

If the above is all NO, then I will look closely to see what happens when a 3rd party app sends the Oauth dialog and the exchange of exchange token happens. where in the process is that getting broken ?

I will also look at customer support tickets, feedback on forums to isolate the problem.