Your new feature boosts Amazon Search by 10%, adds 2s to load time. What do you do?

First, I’d like to clarify what this data reflects.  Is this overall data for the platform or is it specific to a certain channel / geography / browser / device / etc.?

 

Assuming it is generally reflective of the overall platform, next I’d like to clarify what was the goal in releasing the new items search feature.  If we only cared about increasing the number of searches, then the analysis could stop here.  But if I were the PM for this offering, the metrics I care more about are average order size and search conversion rate.  Increasing the number of searches are a means to achieve these revenue-related goals.

 

Next, I’d move onto a hypothesis regarding the slower load time.  My initial reaction is one of concern, as I believe this degraded performance could cause users to grow frustrated and leave the site or close out of the search.  Let’s think about this in the context of the user journey:

 

A) User lands on homepage

B) User performs search

C) User lands on search results

D) User navigates to product page

E) User adds to cart

F) User purchases item

 

From my prompt we know we’ve increased B, but we also potentially decreased C due to degraded performance.  Let’s check some metrics to see if the change has impacted C, D, E, and F by looking at: 

% search results exited before vs. after the new feature.

% search product pages viewed before vs after the new feature.

% search results resulting in add to cart before vs. after the new feature.

% search results resulting in purchase before vs. after the new feature.

 

 

If we notice declines in these metrics, then we know we’ve got a problem with our new feature, as it is negatively impacting users.  We can begin to diagnose the cause of the degraded performance with our engineers.

Approach

1. Understand the feature, its goals, the changes to the feature  and their impact
2. Understand tradeoffs i.e. is the change net positive to the goals - Determine whether increased searches > Search Load time increase by 2 seconds+ by evaluating on different parameters such as customer satisfaction, Order value increase, order frequency changes, avg check out times among others. 
3. Based on the data and our business goals determine the right course of action

 

Clarification

• Is searches increasing a good thing or bad thing I.e. are customers searching more because they are searching for new products or they need multiple queries for the same product
• What product did this impact (web or mobile) —> cross all platforms

 

Product Description and Goals

Amazon search is the first point of interaction for most customers in the Amazon app when they are searching for things (known or browsing). It is key is helping users find the right product in the shortest amount of time based on the search query. To that end, the product goals would be

User Goals

1. Provide accurate and relevant search results across the Amazon catalog
2. Minimize the effort customers need to make to find the right product (number of search queries, time spent searching, modification to search query)

Business goals

1. Make amazon the de facto leader in Product search i.e. customer originate product searches on Amazon and not Google 
2. Grow marketplace transactions

 

Tradeoffs

I would start by into the impact of the feature on the following metrics pre/post the feature launch so that I can get a sense of the impact. 

For that I would collect the following metrics:

 

1. Number of users joining Amazon —> Did searches increases because of new users or because of changes to the product. (Increase is bad as we are seeing slower load times without any material impact on search queries from existing users. The increase could be due to new users.)
2. Number of searches per user for each product —> Has the number of searches gone per user gone up (More searches are good if accompanied by more products)
3. Search to cart conversion rate —> What is the change in add to carts. (More items added to cart are good)
4. Search to checkout conversion rate —> What is the change in single click checkouts  (More checkouts are good, less checkouts are bad)
5. Avg number of orders per customer —> Has the frequency of orders being placed gone up / down (increase in orders is good)
6. # of sessions terminating in search —> Are users not taking any actions post search i.e. abandoning their journey (this is bad as users are unable to find what they are looking for)
7. Customer reviews of the Amazon app --> See how the Amazon app reviews have changed post feature. (Drop on ratings might mean feature has negative impact)

 

Based which direction the above metrics are trending pre and post the search enhancement we can decide whether to persist with the feature i.e. do nothing or whether to address the page load times. 

 

Summary

If the increases in Search queries is leading to more marketplace transactions and more frequent product searches and check-outs the feature is a net positive and we should not do anything. If the greater search queries isn’t impacting the metrics that point to more transactions and better customer experience we should work on address the slower load times. 

Clarifying questions -

Increase in searches may not imply that users are finding it relevant and useful - it could be the opposite. What was the goal of the feature that was implemented? Was it to reduce search time or increase the total number of orders? - Total number of orders
Has the CTR for search results been affected? No
Have there been any marketing campaigns recently that could add to increased search? No
Was the feature rolled out across both Web/App platforms? Yes
Was the featured rolled out globally or across specific geographies? Globally
Has the increase in searches considerably increased in specific countries? No.

Does the 2-sec increase in page load negatively impact business goals? Yes, user drop rate has increased and total orders have decreased.
Is the drop rate consistent across both the platforms? Yes
Is there any specific demographic for which the drop rates have increased? No

Let's consider the user journey - 

A% - User Lands on the Search Page
B% - Performs the Search
C% - Clicks the search result
D% - Adds the item to cart and places the order

Business Goal is to maximize A% x B% C% x D%, increasing revenue for the company. For the users who wait for the search to complete, CTR for search results has not changed - 10% increase in search (B%) seems to be a positive indicator. 

Hypothesis - Resurrecting the dropped users by minimizing search time will lead to more conversions.

Search time is the amount of time that the user waits after performing search action before the results are displayed on the screen. 

Typical request journey => 1. Request from client to server => 2. Server to process the request => 3. Response from server to client => 4. Response is parsed on the FE.

To identify the specific issue - I'd want to work with the engineering team to measure how performance across each of these steps has changed before and after the feature was released.
1. Request from client to server 
I would validate if the request object size is too high.
Is there a new layer that is introduced between frontend and backend? Eg. A middleware or third-part API service
Is the issue specific to a browser or platform? Are the coding standards-compliant and optimal?

2. Server to process the request
Is the algorithm complexity acceptable?
More third party integrations? Can those be optimised?

3. Response from server to client (Same as 1)

4. Response is parsed on the FE
Are the loop iterations optimal?
Is FE waiting for a large response? Can it be lazy-loaded?
Is there room to optimise how information is delivered to the user?

 

 

 

* First of all this should have been found in testing this functionality
* Second is my customer getting impacting with this delayed response. I will do this by checking the data for the customers.
* I mean 2 seconds is not too much time ( you just blink your eye and 2 seconds are over) so this can be ignored.
* If my new feature is not affecting any customer then  no need to do anything.
* There might also be be some  connection issue with DB, as  there might be multiple hits on this feature on the server at the same time so the page will take time to response.
* I will optimise my server load here and will also have a load balance in place to handle the request.

 

Clarifying Questions

1. When we are talking about the search functionality, this is the main search of the Amazon website, right?
2. Before we deep dive, I want to clarify what we mean by search - user enters a query, hits search and gets the result. A search is supposed to be completed when the results are retrieved, is my understanding correct? - Yes
3. There could be multiple page load times in the entire amazon website. Are we talking about an increase in the general page load time when someone opens the the website amazon.com or are we talking about the page load time when a user queries something? - this is the page load time after the user queries something
4. To narrow this further, I want to understand a few more specifics about the search feature.
   1. Over what time has this change been noticed - sudden, week on week, month on month - month on month
   2. In which geography has this change been noticed, since the teams at amazon operate pretty independently, it could be possible that these changes were rolled out only in a specific geography - India
   3. Were these changes noticed on any specific platform or device - Web, mobile web, ios, android app, alexa etc. - the changes have been noticed across all the platforms

Quickly summarizing the findings - Due to recently introduced change, Week on week searches in India increased by 10% and page load times post search increased by 2 seconds.

Evaluation of the Situation:

The search experience is a crucial part of any user's journey when on Amazon. It is fair to assume that most of the high intent purchases start with the search. A poor search experience can deteriorate the entire business and even affect the rest of the metric adversely. For example: poor searches could mean not being able to find products, fewer products added to cart, few checkouts etc... So it is crucial to identify and fix this problem.

For an ideal experience for the user, the user should be able to find the products as quickly as possible and go through the results also quickly. A few quick questions:

1. Increase in Searches: When we say the searches increased by 10%, is it in proportion to the number of new users added, rather are doing more searches than before a making a purchase/adding a product to the cart. - Yes, this is normal. Then we can probably make an assumption that the search itself is functioning properly and that there is no cause for concern about the searches itself.
2. Page Load Times: The increase in the page load time is definitely a cause for concern and interferes with the ideal shopping experience for a user on the product. There could be multiple reasons why the page is taking longer to load. I will break down the search into various parts and try to diagnose possible issues. Is that okay?
   1. Loaded page before the search - Sometimes some common search results are pre-cached to ensure the results are done
      1. Are our server caches working properly to be able to serve the results correctly - Yes
   2. Running the Query
      1. Is the query taking longer to run than now than earlier?
   3. Loading the page - Page loads take longer generally when there is more data requested from the server. This could happen due to various reasons.
      1. Have we changed the layout of the products and categories in the main search results page - of the products
      2. Are we loading more images for a product than before
      3. Are we loading more products on the page in comparison to earlier

Solutions:

1. Improve caches in multiple places to pre-load the results to the most common searches
2. Improve the indexing on the search to reduce the query load time
3. Reduce the results size
   1. by scaling images
   2. by progressively loading the images based on the desktop - top few results to load immediately
   3. loading fewer products at first
   4. scaling the images down to ensure overall results download is small

Implementation:

Once we have an understanding of which of the above issues is causing the page load times to increase, I will work with the engineering teams to improve the situation. Further, I will also track other metrics - such as searches, cart additions, checkouts etc. to see how it has improved the situation.

 

Clarifying questions -

1. Boosts amazon search? What is the meaning of this? - Increases searched by 10% i.e. if 100 users used to perform search in a day, after the feature release - 110 users perform search.
2. Basically, that means search has become more comprehensive and granular.
3. 2s load time to - search page? - Yes, to page that results after user adds search query.
4. It is for all search - not any particular category search. - Yes

User journey -

User opens the app

Searches for the item from search textbox

Search page loads

User browses the product and selects the product by going to product description page

User adds to cart

User goes to cart

User completes the payment and buys the product.

Let us understand the cause and effect first -

I will try to understand if the increase in 2s PLT is attributable to the change in search. A few questions related to that -

1. Have we checked health of our servers? - Health seems to be ok, no fluctuations observed off late
2. Are we sure all the servers are responsive? - Yes.
3. This release was done across all the demography and geography. - Yes
4. And the observation is not for any region or demography - Yes
5. Deviation is not platform specific - Yes
6. There is no other change that has gone live which may have impacted the PLT? - No

Ok. It looks like the causation is correct.

Now, first of all, I want to see the impact of this on not main goal - i.e. number of transactions.

Is there any decrease in user percent who are transacting finally? No -

Is there any deviation from goal? Were we expecting more conversions after search enhancement? - Yes, our success criteria for search enhancement is % increase in conversion rate.

That means page load delay may be causing user churn after search. Let us see -

1. Understand the side effect

   How big is increase in page load time - What was the average page load time earlier? and how big is the impact? - Average page load time earlier was 1.7 seconds. Now its 3.7 seconds.

   This is considerable increase in page load time, and in today’s time of instant gratification, users are not ok with this increase in PLT, hence leave the platform if they have to search operation again and again to find the desired item.

   What was the category wise variation in PLT earlier? Was there a big variation earlier? and kindly help me with variation now also. - Earlier, the least PLT was .7 second for category A and the highest was 4.5 seconds for category B. Now variation has decreased. for almost all the categories, PLT is about 3.7 - variation (1-4 seconds)

   Ok, we are seeking improvement in a few categories where plt was high earlier. But because of the implementation, we may have compromised on less PLTs on some other categories.

   Cool,

   Now I have a hypothesis -

   Transaction for the categories where PLT has increased must have decreased and the categories where PLT has improved, must have increased - balancing the conversion rates. Could we have that insight please? - yes, data is in accordance with this hypothesis.

   There is last area that I would like to explore -

   What was the purpose of search enhancement- to make search more granular and decrease PLT (as less and relevant products will be loaded).

   Isnt it resulting in people doing more searches now and increasing burden on servers? - Servers health has been consistent. As already tole, we have increased keywords in search. That is encouraging users to search the exact products with specific keywords in some categories.

   Ok, now that we have understood the problem let us see what can be done.

   1. What is the contribution of Category B searches overall? ~15%
   2. That is considerable. What is the change in search? - It has become more granular, and we have added more keywords in search options.
   3. Those changes have been added to overall search engine - Yes

   Solution -

   We will roll-back the last changes first and limit the search optimization changes to selected categories where PLT is improving.

   For rest of the categories, we will continue with the earlier search algorithm, and we will assess scope of enhancement for these searches again.

 

To detmine next steps, I would start with what the new feature attempted to do. 

Assumption:  

1. Data we have is from a clean A/B test and is not tainted by other factors external to the feature. 
2. Features goal was to make it easier for the users to search the products they wish to buy. Thus the success metrics could be :

• Primary metric:  "Average $ value of checkout per user session"
  • User session is defined as a period of time with continuous user activity i.e. no user break greater than ~15 mintues. 
• Secondary metric:  Search performance:  Ratio of -  number of searches / clicks on the SRP.
  • As response time goes up, this metric would get worse because of bounce / abandonment of search. 

• Primary metric (Avg $ per user session)	Secondary Metric (Bounce rate)	Action
  Pass	Pass (Unlikely, based on info we are given)	Roll out. No further action needed
  Pass	Fail (Most likely, based on info we are given)	1. Roll out.  2. Root cause source delay and try to fix it.  3. If we have an improvement, make an additional release with the fix and measure again.
  Fail	Pass  (Unlikely)	1. Roll back the feature 2. Restrategise how to improve search
  Fail	Fail	1. Roll back the feature 2. Root cause source of delay and try to fix it. 3. Run the experiment again if we have a fix

some clarification questions to answer this issue:

 

1.        What do we mean by boosts amazon search by 10% - does it mean the search results are increase by 10%

2.        By adding 2s – what is the impact in terms of %?- don’t know. And which metric is it affecting? Is it causing users to abandon the search? – yes.

3.        So that means effectively it is resulting into less purchases bringing the average order value down and also a bad experience for the user.

4.        Is this feature done on both web and mobile? - yes

 

So typically a user journey is that a% of users are using amazon search > B% are then checking the search results > C% are clicking on the results and further down some of them are making the purchase.

Ideally we would like to keep that B% and C% as same.

 

So, I would like to rephrase my problem statement as –

I want to decrease the search result time , so that I do not abandon the search midway and get the results?

 

We need to optimize the query to get the results. Now, I am sure amazon has very good search mechanisms but it can be seen if by applying different search mechanisms like elastic search or may be by indexing in the database the result time could be improved.

 

Now, let’s say that in any case the return time of the query cannot be improved. In that case, we need to analyse the benefit of increasing the search results. Because the ultimate goal here to make a sale.

So, if without the new feature, the overall sales – A*B*C*D remains high then I think it makes sese to rollback the feature.