How do you define success for Yelp reviews?

Question

How do you define success for Yelp reviews?

Asked at Google over a year ago

4.3k views

& 1 other company

Asked at

& 1 other company

4.3k views

How to answer Metrics questions

Interview Guide

Top Metrics interview questions

How would you measure the success of Facebook Likes?40 answers | 61.3k views
What metrics would you use to measure the success of the Save feature at Facebook? 39 answers | 87.2k views
How would you measure the success of Gmail?23 answers | 17k views
See Metrics PM Interview Questions

Invite members
Invite by email

Add another

Sandy M · Answer 1 · 2020-07-15T15:51:25+0000

C: Just to clarify, Yelp is "Reviews" app where users can get see how others have rated a restaurant or any service. I: Yes
C: Since it is a product that has been around for over 10+ years and is well known, It is definitely a mature product.. not in early adoption or growth phase. Does that sound good? I: Yes
Ok, for the structure of my answer, I would like the think about the users, their journey and then come up with a few metrics... and then prioritize them. Does that sound good? I: Yes
C: For the goal, is it ok to focus on engagement and retention since it is a mature product? I: Yes

Product: Yelp Reviews
Goal: Increase engagement and retention
Users: There are 4 main user groups. For now, let's just prioritize end users and review writers.
- End users: Those who use the reviews
- Review writers: Content creators who post reviews
- Businesses & Services: Like restaurant owners
- Yelp the business
User Journeys
- Search: End user opens the app, searches for a business and reads reviews
- Discover: End user does has some criteria in mind like "open now" or price or location and tries to discover new restaurans
- Review posting: Writers open a business's page and post a review/ photos etc.
- Businesses: Businesses sometimes respond to both good and bad reviews on Yelp

Metrics

#MetricEnd UsersReview Writers

1.	Engagement	- # of business pages seen - Time on site - # of reviews interacted with like likes, comments - # of calls made, directions to locations to show actual action taken after reading reviews. - Businesses visited as known from user locations	- # of Reviews - # of reviews per user per week
2.	Retention	- # D/W/M active users - % of repeat end users every week, month	- # D/W/M active writers - % of repeat writers every week, month
3.	Monetization	- Ad Revenue $, Coupon Revenue $ - Avg Revenue per user	- Ad Revenue $, Coupon Revenue $ - Avg Revenue per writer
4.	Satisfaction	-Avg Review/NPS in survey -Reviews in Apple/play store	-Avg Review in survey -Reviews in Apple/play store

6. Prioritization: Knowing Yelp's business model, I would like to use the criteria of impact to user to pick the metrics. When the user is engaged and spends more time on the site, it leads to better monetization for Yelp.

7. Tradeoffs & Risks: From the writer side, While we do want a good amount of quanity of reviews, we also need quality (in depth reviews) with tags and other details to train our ML recommendation models. We need to ensure that we don't accidentally lead to just the "star" reviews without the details/comments/tips. This can be monitored by

8. Chosen metrics are:

North Star: Avg. Time on site for end users and # of Reviews for Writers.
Guard rails: Avg Review in Survey for end users and % of Quality/In-depth reviews for Writers
Secondary metrics: Weekly/Monthly active users and writers

productmanager123 · Answer 2 · 2021-06-27T19:50:01+0000

Yelp:

Its an app (mobile, or web) with listing of various businesses. People search for information on local businesses (restaurants, handyman, salons etc) usign yelp.

People read ratings and reviews to inform their decision about the service.

User segments.

a/ Customers who use a service and rate and review that service - small fraction

b/ CUstomers who read the reviews to inform their decision.

Yelp also uses reviews as a signal for ranking search results. Listings with good ratings and reviews will be shown higher in the search algo.

1/ Writer

They use the service → are motivated to leave a review → search of the service → sign in → leave rating and write a review.

2/ Readers

Search results - select listing -> reviews. Can sort by best reviews, recent reviews, relevant reviews and so on. They can also engage with the review -- expand the review (see more), “like” the review, give feedback “found it useful”

Key goal behind reviews -- reviews help in informing the decision for the customer. If there are a a lot of highq quality reviews, then user can use Yelp to inform their decision instead of going to other websites such as google search.

Key metrics:

Coverage of reviews.

a/ % of listings that have reviews. Category - restaurants. Most popular categories - user search and views. FOr those catgeoreis -- you want higher coverage of reviews.

b/ # of reviews per listing

Quality of reviews

# of reviews that have a rating but no review.

% of reviews with user engagement - they expanded (see more), liked or said “found useful”

# of fake reviews

# of disputed reviews

# reviews relevant --

Creation of reviews

# of users who created a review in last month

# of users who created a review last 3 months but not in last month

# Listings / user

# of users who left a review/ # of users who had a conversion event w.r.t the service (called, directions, actually went there)

PM Team · Answer 3 · 2022-05-02T00:45:40+0000

Product Value & Users: Yelp's mission = connect people with local businesses. Yelp reviews help consumers learn about product/services and be better informed before making a purchase decision or request a quote from a business. There are user groups when it come to reviews, review creators and review viewers

Review viewers get value from being more educated about a business and its product/services
Review creators get intangile value from informing others. Some reviewers create so many reviews that they are selected to join Yelp Elite Squad and get invited to join exclusive Yelp events.

2. Metrics

NS Metric: Total # of engagement (Call, Request to Quote, Message etc) comes from reviews per month per year. This metrics is a good proxy that tells us whether reviews are useful and lead to contacts with local business

Review Creator	Review Viewer	Yelp Metrics
# of review creator per month, per year	DAU & MAU of reviews	Geo/regions with highest review coverage Geo/regions with lowest review coverage
Avg # of reviews per creator per month, per year	Avg # of reviews viewed per session	Biz categories with highest review coverage, cut data by Geo Biz categories with lowest review coverage, cut data by Geo
	Avg time spent on reviews per session per month/year	% of Biz with >= X # of reviews, cut by categories and by Geo % of Biz with
	Total number of reviews engagement (useful, funny, cool, compliment)	Total # of engagement (Call, Request to Quote, Message etc) comes from reviews per month, per year

Downstream Impact: # of review sharing per month per year and cut by Geo/categories. The more review sharing happens, the more likely new users will join Yelp as a result thus creating the network effect.

3. Tradeoff: # of reported reviews per month per year, # of fake reviews detected by ML model

KSM · Answer 4 · 2023-08-04T03:51:54+0000

First, I want to start with some clarifying questions. These will help me think through a structure that I want to present and follow. I might make/take assumptions which I will capture in this doc.

Clarification questions -

Product clarity - When we say Yelp reviews - are we talking about the product where users/customers post their reviews/ratings on a restaurant? - YES

Geo scope - are we talking about Yelp Reviews in US or other countries? US

Platform - I dont think this would matter (all platforms)? Mobile, Web, Smart devices?

Users and their journey

a/ End user giving reviews about restaurants -

J1: Access the app, searches for the restaurant, logs in (i think), enters the reviews, reviews their review, posts the review

J2: Search for a restaurant on a partner site or on the restaurant site, enter reviews their (where yelp widget is available)

b/ End users consuming reviews

J1: Access the app, searches for the restaurant, logs in (i think), enters the reviews, reviews their review, posts the review

J2: Search for a restaurant on a partner site or on the restaurant site, read the review via a yelp widget

c/ Restaurant owners

J1: List their restaurants on the yelp site

J2: Bring reviews on their site via a widget / APIs (less so)

d/ Partners who use reviews for their content

J1: Use yelp widget to get yelp review on their site

J2: Use yelp APis to get the yelp review content on their site

Few goals are - Awareness, Acquisition, Activation, Engagement, Monetization. I will pick Engagement of users entering reviews and reading/accessing reviews. Reason - Product is mature, well known in the US regions. No point focusing on Awareness, Acquisition, Activation. We can talk about Engagement and Monetization. I believe one feeds the other. We need to focus on engagement which then leads to monetization. Of the user segments, I want to focus on users leaving reviews and reading reviews - because they are more in number, are the main people that contribute content and hence are the main user type.

Main Goal	End User Writing Reviews	End User Accessing Reviews
Engagement	# of unique users per week/month writing reviews	# of unique users per daily accessing reviews
	# of reviews per user /week and /month	# of reviews accessed per day
	# of reviews written per day	Avg time spent by a reader
	Avg time spent by a writer	# of unique requests call the restaurant
	# of Reviews getting high rating in a period of time	# of unique requests to book a table (reservation)
	# Yelp tagging poor reviews or flagging as fake in a period of time	# of unique requests call the enter get directions
		# of unique requests deep link to uber

Top metric

1/ Engagement of writer - # of top quality reviews written per week

2/ Engagement of reader - # of reviews reviewed per day

Secondary metric -

1/ Engagement of writer - # of unique writers per week or month

2/ Engagement of reader - # of unique readers per day

SPD_2017 · Answer 5 · 2022-06-27T01:50:52+0000

Goals:

Yelp offers an app and website where users can review businesses, read reviews by other users, rate these reviews and make restuarant reservations. There are targetted ads based on user search, which allows Yelp to monetize this platform.

User Journey:

For this assessment, I will focus on the business review section of Yelp. There are following steps that user will go through while using Yelp.

User signs up on Yelp
User logs in to Yelp App or website
User searches for a business
User submits a review for a business
User reads reviews submitted by others
User deletes a review that they submitted
User edits a review that they submitted
User rates a review by another user

Metrics:

Activation/adoption	How many users have signed up?	new users signin up over a period Market share compared to other review websites such as Google reviews.
Retention	How many users log in to Yelp over time? How often do they log in? How much time do they spend here?	Number of users loggin in growth in the user logins over time Avg time spent using Yelp
Engagement	How do these users engage on Yelp? daily/weekly/monthly/YoY	how many reviews are submitted reviews over time How many reviews are deleted/edited How many reviews are rated by the user
Monetization	Ad revenue per customer, reservations	revenue per user

Prioritization:

For a product like Yelp, it is crucial that the user spends time on the website, which means stronger crowdsourced feedback on the businesses, making Yelp a reliable source of information and better monitization through ads. Hence retention and engagement are the key metrics to evaluate the product's success hence I would like to prioritize:

% growth in number of submitted reviews over time

% growth of reviews over users for a given period

Tradeoffs:

The number of reviews may not always provide useful content for the users, there is risk of business opting for paid reviews to get a better rating. The users and reviews need to be vetted by an algorithm to confirm the are authentic. Yelp needs to keep check to continue to succeed.

amitbp · Answer 6 · 2020-09-01T23:29:01+0000

Clarifying Question - If I understand correctly, Yelp Ratings or Reviews are for Businesses listed on the website that helps consumers to make better decisions. These Reviews can also be sponsored ? Yes

Feature Description -

Users give reviews based on their experience with the Business
Users can react to a particular review.
Businesses have the opportunity to reply to reviews posted by users.

Goal

Yelp is a product that is built on UGC(Reviews, Ratings, Photos etc). This differentiates Yelp from other listing websites. I would say as a company Yelp wants more users to post reviews and engage in the platform.

More users to post reviews
More users to read reviews
More users to engage on reviews

So, Primary objective for Yelp Review is Engagement and Retention.

User Journey

Pre-Journey-

User opens the app or website and searches for business by name or category. If by category then gets a list of business in that category, user have the option to filter and sort the listing. User clicks on a particular business and read details about the business , reads the reviews and makes a decision

Journey -

User decides to place an order or call the business for an appointment, visits the business.

Post Journey -

User decides to share her experience on Yelp, logs in to yelp and shares her experience in the form of review.

Metrics

Based on our Primary Business Objective below are the important Metrics

More users to post reviews

Avg # of reviews posted per business
Avg # of reviews posted per user
Avg # of reviews posted per business type

More users to read reviews and engage

Avg # of views posted per business
Avg # of reactions per review.
Avg time spent per user reading reviews

Also,

Avg # of calls(to business) or orders after reading a review.
Avg # of time user shuts down the website or App after reading a review or without reading a review

Prioritization

Based on the reach and impact keeping in mind our business objective, my primary metrics would be -

Avg. # of reviews posted per business
Avg. # of views per review
Avg time spent per user reading reviews

vineetagarwal · Answer 7 · 2020-07-16T15:21:55+0000

What is Yelp's business model?
1. Ad revenues from businesses
What is Yelp's value proposition for businesses?
1. Discovery, Information, and traffic
What would Yelp like to achieve?
1. Higher traffic and engagement from customers
What role do 'reviews' play in that goal
1. Information about the quality of service
2. Voice of the customer
3. Popularity measurement
What is the success of 'reviews' in simple terms?
1. An increasing number of customers review posting
2. An increasing number of customers review browsing/searching.
3. An increasing number of customer visits to businesses driven by reviews.
Which metrics to we use to track the success of reviews?
1. Primary metrics to measure the engagement factor
  1. Number of reviews posted
  2. Amount of time spent by customers in reading reviews
  3. Number of review-responses logged (Customer finding other reviews helpful)
  4. Number of Business responses to customer reviews
  5. Number of searches ran on reviews
2. Secondary metrics to measure business growth factor
  1. How many businesses tagged as 'trending/hot' based on an increasing number of reviews? Did the number of customer check-in increase after this event?
  2. How many businesses tagged as 'new/upcoming' based on the reviews pattern? Did the number of customer check-in increase after this event?