Around 40% of reviews in Amazon are fake. As a PM in Amazon, how will you tackle the problem? Break your answer into 3 parts — 1. How will you identify a review is fake? 2. What action will you take? 3. If the number of fake reviews are decreased, what will be the impact?

First of all - CQ -

1. How do we know 40% reviews are fake? - by Data team
2. Reviews as in - reviews written by users under products be a particular seller… and these reviews help in buying decision - yes.
3. Has it started happening recently or it is happening since beginning? - It is an all time trend. Leadership thinks it is high time to now find a solution to this.

OK…

First of all I will try to understand what are the consequences of this and why is it a big issue now?

What reviews are -

When a customer buys a product, then he is supposed to write a review regarding his experience of the product. They write their reviews and give a rating to the product, seller and delivery service.

How are they useful to Amazon -

1. While customer is exploring the product, then he considers going through the customers reviews. And these reviews contribute a lot to the buying decision of the customers.

   If majority of reviews are fake, these will have a adverse impact and eventually customer experience will hamper.

2. These reviews keep a check on sellers. As market is very competitive (even for online sellers), sellers always strive to have good reviews for more orders in future. If these reviews are not authentic, then whole reason of having review system fails. And sellers have no aspiration for providing good product and services.

3. Ratings are like testimonial of customer experience and communicate a lot about what’s good and what is bad on platform. fake reviews serve no purpose and just mislead platform’ algorithm to rank sellers and customers.

Now that we have understood impact of fake reviews on platform, Qs is how to identify fake reviews. And then how to get rid of them. Because if we continue to have them, they will keep on impacting the system. And finally, how to stop fake reviews in future.

Step 1 - Identify fake reviews -

1. Only a customer who has bought a product from that seller is supposed to write review about product/seller. First and foremost step is to check order history for each of the review writer and see if he has bought the product or any product from that seller. If not, mark that review as fake.
2. there can be bots writing reviews. Such reviews will be having same wordings. These can be identified and marked as fake.
3. Some reviews are written long after product is purchased. Those reviews also dont make sense as they dont reflect the actual product… rather how the product was used. So along with checking for the order history, we will have to check order date also. if difference between ordering and writing review is too large, mark that review as fake.

Step 2 - getting rid of them

1. We can either remove them completely or add a marker saying these are fake reviews. I will suggest to remove them and have only verified reviews going forward.
2. To get rid of them, we will have a marker against each review during identification phase. Every time we will find a fake review, we will toggle that FakeMarker to 1. Once all reviews are checked, we will run a program to remove all the reviews, where FakeMarker is on.

Step 3 - Stop the same in future -

Now, to have only verified reviews on system, we will have to allow only those users to write the reviews for product, who have actually bought the product. Moreover, review has to be written with in stipulated time period post that duration, turn the review section inaccessible.

Impact of decreasing fake reviews -

1. Reviews will give genuine representation of product, seller and service thus helping customers big time in their decision process.
2. Sellers will strive for good reviews and will maintain their product and service quality to have good sale in future. It will be a fair marketplace in that case with healthy competition.
3. Platform will earn trust from customers as reviews are serving the right purpose now.

Clarifying questions:

• Are we talking about the Amazon B2C app? Yes
• Any specific geography where we are wishing to tackle this? (World wide / globally)
• Any specific category / sub-category of products that we are hoping to solve for? (all categories)
• What kind of fake reviews are we referring to? (Purchased fake reviews / Bot reviews / Users reviewing products and haven’t bought the products/ Users intentionally giving wrong reviews?) All type of reviews
• Also do we have any technical constraints as to how we will manage this problem? No

Types of fake reviews:

• Fake reviews that have been manipulated by sellers / agencies
• Users intentionally giving wrong reviews
• Bots giving reviews

How will we identify whether a review is fake?

Fake reviews that have been manipulated by sellers / agencies

Key markers:

• new user accounts or accounts with no purchase history
• Similar user leaving similar comments across unrelated categories.
• Generic wordings used for comments indicating the user has not used the product
• short length of comments / no comment
• leaving only ratings and no reviews.

Users intentionally giving wrong reviews

Key markers:

• Similar behaviour observed in the past as well for multiple products
• History of purchasing and initiating returns / not paying for orders.
• has raised several tickets in the past to customer support and was unhappy with the service.

Bots giving reviews

Key markers:

• Unnatural / non-human language
• Heat mapping indicating the cursor has a robotic pattern of movement
• Leaving reviews at unnatural timings
• Historical order experience of that particular profile

2. What action will you take?

Tackling this problem involves building a comprehensive engine that would analyse reviews across Amazon and create flags for reviews that fall under any of the three buckets. These buckets would then be analysed on a second level by a human agent and upon confirmation that the reviews are indeed fake we can remove the reviews and put these profiles on standby for action in the future.

What would these engine analyse:

• Segment them on the basis of categories/ sub- categories/ geographies
• Timing when the review was given (flag any review that was posted at late night / or very early in the morning as per that geography)
• Flag reviews that were left for a product from a geography where the product is not available
• Use NLP to detect unnatural language / robotic language
• Use NLP to detect whether a review given has any sync with the product in question
• Flag reviews from accounts which don’t have a good order history with Amazon (very frequent returns/ not accepting orders)
• Create events on our heatmapping tool to check whether any account has an unnatural cursor movement while leaving reviews.
• Flag profiles who have left monosyllabic or short length reviews under orders various times across categories.

Once we have placed reviews under these buckets, human agents can be employed to do a double check and remove any profiles/ reviews that are confirmed to be fake.

3. If the number of fake reviews are decreased, what will be the impact?

Pros:

✅ authentic reviews on the platform, we will boost our NPS and general trust of the consumer

✅ likely to experience less returns/ refund requests as the users would be reading authentic reviews and then placing orders

Cons:

❌ lesser reviews for some products will impact visibility in the catalog page

❌ fewer orders from products with no reviews

Thank you for the question. I want to start off with some clarification questions - 

 

1/ When we say reviews on Amazon - are we talking about Amazon.com retail product? If yes, is there any idea on what products such as built and sold by Amazon, built by 3rd party and sold by Amazon, built and solde by 3rd party on Amazon?

 

2/ Are these reviews always +ve , or -ve, or combinations?

 

3/ What are the after effects of these reviews, such as do other users find it useful, or has someone flagged these?

 

In general there are 2 types of users - 

 

1/ Genuine purchasers of a product -> buy a product, use it, leave review - either +ve or -ve and sometimes return the product

 

2/ Product reviewers -> are given a product to review and keep it (can be biased)

 

How to identify review is fake

 

Category 1 - Obvious fake reviewers

1/ Seller is leaving a review - this is ver obvious and i assume less frequent

2/ Repeat of IP address with different user names - part of a blacklist IP addresses

 

Category 2 - Potential fake reviewers

1/ Review done before product is delivered - red flag

2/ Bad review posted without a return - potential bad user which we can start to track

3/ poorly formed phrases or repeat of a pattern from different posts

 

Category 3 - Flagged reviewers

1/ Track the reviews and their past and current reviews to identify patterns

 

What action will you take?

Category 1

1/ Block them immediately and remove their reviews

 

Category 2 and Category 3

1/ Create a list of such reviewers with IP addresses, names, and other patterns

2/ when we have a list of reviewers who need to be closely tracked then make sure their posts are not made live immediately and are reivewed by a human reviewer to be added to the reviewers to be blocked

 

Make sure the feedback is created for this process for tracking false negatives and false positives

 

Impact is # of fake reviews are reduced?

Category 1 -  # of reviews

Overall number of reviews can drop - could be significant when it comes to sheer numbers

 

Category 2 - Ratings

Ratings could change from a leaning left or leaning right to more centered

 

Category 3 - Brand

Recognized as a market place where there are true reviews which can be trusted.

 

So even though initial impact would be drastic with loss of reviews and drop in ratings, with a brand promise of true reviews, we will gain customer trust which will enable more products and purchases in long term.

First I would clarify that what does it mean by fake reviews.

Let people who give reviews within some specific time like within 5 minutes after sign up process or people who have not ordered the item but giving reviews.

I will take help from data that we have & will remove the fake reviews also I would like to ask those people the issues they are facing maybe they are not comfortable with some of our policies or features or customer support. Once we get the real problem we can go in that way.

Now as we have removed the fake reviews then the number of fake reviews is going to decreased. There will be several impacts: -

1.It will help us to know which product customer is liking and if not liking then we can get a clearer sense what was the problem in the delivered item.

2.It will provide our customers a better experience on the platform.

3.it will also improve the trust of customers on the platform.

Describe the product- Online buying platform where users can buy the product virtually

Clarification

1.      Which specific product segment is throwing fake reviews at a large scale?

Assuming – It is unequally divided

2.      Which specific geographic regions is showing higher number of fake reviews?

Assuming – It is specific to the bottom 5 countries where rate of corruption and malpractice is high

3.      What is the impact in the present situation? Is impacting the sales, revenue or the customers footfall? Assuming all have been falling that is also goal of app to increase the same

4.      Are the virtual sellers allowed to sell the products through other competitors website? Assuming yes for certain category of products

5.      Assuming the fake reviews are the positive reviews and not the negative or neutral ones

Who are the users?

A.     Business users who want to sell the product online but have the presence in the physical market as well

B.     Business users who are selling the products only at Amazon website

C.      Buyers: who usually buys through online medium

D.     Buyers who prefer to visit physical stores but occasionally buys through online site/Amazon

E.      Buyers who mostly buys online through multiple websites and occasionally visits physical stores

Business value Proposition

1.      As the users buys the product virtually so they comment and reviews as a parameter to understand the quality and Genuity of the products

2.      Reviews helps in increasing the sale, in turn the revenues and increase the footfalls in the website

3.      Reviews tells us about the customer’s likes and dislikes to make the product better

4.      Positive Reviews also helps in building the goodwill which in turn motivates the seller to work harder to meet the customer satisfaction

5.      Reviews could be positive or negative or neutral

How will you identify the fake reviews?

1.      Sales return and the refund request tied with a reason for the specific line of products will tell if the customers views is actually matching the with the stated reviews like

Difference in colour

Quality of the product

Size for apparels

Satisfaction level on the exchanged items etc

 

2.      Do the customers ask for refund or replacement with reason that shows the amount of dis-satisfaction for a specific reason which can be re-visited for the category of the products

3.      Customer complaints through the customer care

4.      Written feedback through email or chats, social media where the aggrieved buyer tags Amazon

5.      Verifying the above stated points given by the buyers with respect to the reviews posted for a category of the product

6.      Once we found the category of the product then we can do a sample investigation of different products with same segment where too many reviews are excellent with the physical stores

7.      Checking the authenticity of the consumer complaints on the basis of their experience. Example those who frequently visits the physical stores, see the product and buys it will have better understanding on the product. Again we can tie it back to the reviews to check if they are genuine

8.      A survey on the specific segment of the products where anomalies have been found at a large scale can be shared to the customers

9.      FMCG product customers can provide quick reviews other durable goods like appliances etc takes time for the user to comment on reviews. This can be checked on the number of times the maintainnence services have been taken during the warranty period

10.   Tally the review owners’ details with the actual invoice holders. Meaning the person who bought the product and used it should give a review. This could fail because at times people who buy the products are the same who uses it.

Metrics

1.      Average Number of transactions made by a user

2.      % Increase or decrease in the sales periodically for the products where reviews might be investigated as fake

3.      Average rate of return and refund

4.      % increase/decrease on the return and refund

5.      Average number of people initiating the exchange request

6.      Average number requests are received for return, refund for the same segment and product on given period

7.      % Increase number requests are received for return, refund for the same segment and product on comparing last quarter

8.      % Change in Similar type of of reviews compiling on each segment/product type before and after fix

9.      % Increase or decrease in revenue

10.   % increase/decrease in customers from physical to virtual shopping

Impact if the number of fake reviews will decrease

 

1.      Sales might go down for some time but then rise again eventually. This problem can be controlled by putting a tag on reviews as verified or genuine to gain the trust of the customers

2.      People might move to other online website for the time being or towards the physical stores

3.      Sellers might need to release lightening deals for some time and heavy discounts to increase the customer base as gaining trust will take some amount of time

4.      Operational cost might increase as we need to allow the user to post only genuine reviews. This might need additional efforts to build a system that can trace the accuracy of reviews with the actual sales made to the customer.

After identification fake reviews should be penalised and restricted, eliminated the old fake reviews. A mechanism needs to be build to restrict the fake review like a tagging a spam automatically if the reviews are not matching the aforesaid crietera and only those reviews are allowed to be posted that passes the match crieteria