Find out how a data-driven approach to Internal search could help to improve your product optimisation processes.
Internal search analysis
Over the last 10 years or so I've been involved in many product optimisation processes. As a consultant I find the trick is to strike a balance between prioritising the (sometimes) endless development backlog with giving stakeholders tangible improvements to their products that add value for their customers.
One area that often proves difficult to improve is internal search. There are many reasons for this but in my opinion the main reason is user expectation. The area of search has been redefined by Google. I imagine everyone using an internal search on a website has performed exponentially more searches on Google, so the bar is already very high.
As product people we need to design products that meet user expectations. This means understanding why customers choose to search and how we can make the journey as seamless as possible. In this blog I'll share what’s worked for me and some analysis techniques I've found successful, hopefully, you to will be able to add value to your customers via internal search.
Data-driven meets user-centric design
I’m a big believer in using data to drive prioritisation and product roadmaps. I’ve been privileged to work with some fabulous Product Designers who advocate user-centric design. I don’t see these approaches as mutually exclusive, but as complimentary. At Red Badger we’ve created a process we call DUXD (Data Driven UX and Design) that takes the best of these two processes and combines them.
Ultimately there are always things the data can't tell you - things like intention, emotion or motivations - but there are others it definitely can: conversion rates; churn rates; engagement; customer lifetime value etc. For me, playing to the strengths of these two disciplines and accepting we need both to build fantastic products that people want to use, is key to building fantastic products.
When it comes to internal search there are many things we can do to improve the customer experience. Making search easier to find, making sure the most relevant results are at the top of page 1, making inputs easier etc will all improve the user experience, but which one should you do first? It’s possible to use your data to analyse user behaviour and quantify the likely impact changes will have. We can also speak to our users and ask them for help.
The basics
Before making changes to any area of a product it’s important to understand the current lay of the land. For digital products this should always include data analysis as well as user interviews and UX reviews - this is integral to the Red Badger DUXD process.
In order to start with data analysis you’ll need some good data. Data capture doesn’t work retrospectively so you need to get this right. This is really important. As soon as tracking requirements are deprioritised you’re saying goodbye to understanding how users interact with what your team releases.
The data you capture impacts what analysis you can do. Try to think of the types of analysis you want to do before implementing any data capture. It may help to imagine the types of user journeys you want to analyse or the issues your users may encounter. This is a task Analysts and UXers can collaborate on.
Here are some data points that I’ve captured for internal search before;
- Search performed
- Search box clicked in to
- Some search results returned
- No search results returned
- Search keyword
- Intuitive search used (if applicable)
- Search algorithm confidence score
- Page the search was performed from
- Site section the search was performed from
- Number of search results returned
- Search result clicked
- Position of search result clicked (works for both lists and grids)
- Search results page of search result clicked (if using pagination)
- Title of search result clicked (or a way to identify the result, product or article)
- Bounced searches - a way of indicating if a person found what they are looking for (good for article searches)
- Modified search (was a search changed and what was it changed to?)
- Was the search term spelt correctly (usually assessed by the algorithm)
- Type of search performed (if your algorithm can perform multiple types of search)
- Were search filters used (if so how many and what were they?)
- Were search resulted sorted and by what?
It’s not necessary to capture all of the above. It’s dependent on how sophisticated your internal search is and the analysis you want to perform.
Getting started with basic analysis
Most web analytics tools will have some capacity to capture internal search data. Most of the above functionality won’t be available out-of-the-box but it can be set up using custom features. I’ll keep the analysis in this blog tool-agnostic but i’ve done this type of analysis in GA, Adobe, Amplitude etc as well as customer data warehouse setups.
Establish baseline usage
For any analysis you need a goal, something you want a user to do. Generally the number of people performing this task is represented as a percentage. Conversion rates and click-through-rates are two common examples.
For internal search the goal is usually ‘search rate’ - the number of sessions/visits that contain a search. This means we now have two segments of users, those who searched and those who didn’t. Some web analytics tools can give you this in a report. However, they include people who bounced in the ‘users who did not search’ bucket. Personally I don’t think this is comparing apples with apples as users who bounced didn’t engage with the site but users who take time to search were clearly engaged. I always remove people who bounced from my ‘did not search’ segment.
Establish success
Different sites will have different measures of success. It could be purchases, downloads, articles read etc. Usually Product Owners will want to tie activity back to a goal such as this. This means when we look at success for internal search there are 2 or 3 basic metrics to look at;
- Successful searches (searches that yield 1 or more result)
- Search result clicked (successful search that results in a click to a search result)
- Goal completion after search (linking the main goal of the site to internal search)
The above can all be viewed as a percentage to give a rate. Rates are often better to trend over time as you can see the impact of changes that have been made. Looking at the raw goal numbers can give the impression metrics are going up or down when they are often influenced by site traffic volumes.
The above measures should give a good baseline of performance. Different metrics will become important at different points of the optimisation process. If a customer pain is that your search algorithm doesn’t account for spelling mistakes or you’re trying to gather evidence to put forward a business case for developing intuitive search, you’re most likely to need the successful search metric. However, if your focus is on the performance of the algorithm in terms of bringing back the most relevant search results then search results clicked or goal completions are likely to be more important.
Customer journey analysis in search
In order to prioritise work to be done, getting an idea of the impact a change is likely to have is critical. You’ll also need to consider development effort. Once you have an idea of impact and effort prioritisation becomes a lot easier.
Here are some examples of how I've done analysis to quantify the impact of internal search optimisation.
The case of improving visibility / usage of internal search
So why do we want more people to search? The easiest way to demonstrate this is to show what extra value searching brings to the site. Here’s an example using an eCommerce site. Split the traffic (sessions/visits) into those that include a search and those that don’t (removing bounces). Calculate the conversion rate for the two segments (you can also include other metrics such as average order value (AOV), return frequency, average basket size etc). If internal search is having a positive impact there will be a difference in these metrics. Let’s assume the average order value for both segments is the same at £100 per order, but conversion rates are 10% Vs 15% in favour of sessions with a search.
Next calculate the percentage of sessions that include a search - the numbers have already been used in the above calculation. For this example let’s imagine that 20% of site visits include a search.
To quantify the impact of getting more people to search we can model the impact of getting more visits to include a search. Pick a sensible traffic number for the site. For the example here we’ll use 100,000. So in a given time period the site gets 100k visits - we’re making the assumption here that none of these visits bounce. This doesn’t impact the calculation as we removed them from the segments anyway.
From the above, 100k visits where 20% searched means 80,000 sessions without a search and 20,000 with a search. At £100 per order and conversion rates of 10% and 15% here is the revenue for the two groups;
Without a search - 80,000 x £100 x 10% = £800,000
With a search - 20,000 x £100 x 15% = £300,000
Total revenue = £1,100,000
If we can improve search uptake up 5% how much would that increase site revenue? Applying this to the equations above;
Without a search - 75,000 x £100 x 10% = £750,000
With a search - 25,000 x £100 x 15% = £375,000
Total revenue = £1,125,000
That’s an increase of 2.3% or £25k per 100k visits.
It’s easy to take this further and model different levels of uptake.
In reality things are often more complicated, but we can still apply the same logic. In my experience internal search usually increases average order value. Amusing average order value for those who search is £125 the calculations become;
Without a search - 80,000 x £100 x 10% = £800,000
With a search - 20,000 x £125 x 15% = £375,000
Total revenue = £1,175,000
Increasing uptake of search to 25% becomes;
Without a search - 75,000 x £100 x 10% = £800,000
With a search - 25,000 x £125 x 15% = £468,750
Total revenue = £1,218,750
That’s an increase of 3.7% or £44k per 100k visits.
The above logic can be used to build a case for improving the accessibility of search on a site or app. Working closely with the UX and Design Team can help craft ideas around how to do this and how we measure improvements.
The case for improving the search functionality
If the above example, the search functionality itself wasn’t improved. If we have a number of backlog items around improving search visibility and improving search capability, which should be tackled first?
There are many ways to improve search functionality, let’s look at a high level view. Here’s a basic calculation to demonstrate the impact of improving search performance.
If we assume we can improve search performance that would directly impact conversion rates and increase this from 15% to 20% (this is an estimation, more detailed analysis comes later on).
Using the numbers from the above example we get;
Without a search - 80,000 x £100 x 10% = £800,000
With a search - 20,000 x £125 x 15% = £375,000
Total revenue = £1,175,000
Improving search conversion rates to 20% (notice search usage hasn’t changed);
Without a search - 80,000 x £100 x 10% = £800,000
With a search - 20,000 x £125 x 20% = £500,000
Total revenue = £1,300,000
That’s an increase of 10.6% or £125k per 100k visits.
Given the above it seems to be more beneficial for the team to work on search performance over accessibility.
Search features
Improving search performance can take many forms. Here are some to think about;
- Flexible search - allowing spelling mistakes in keyworks
- Intuitive search - suggesting keywords after a user has typed 3 letters of the keyword
- Search relevance - ensuring that the most relevant results are at the top of the search results page
- Introducing filters - allowing users to filter search results
- Intelligent search - does the page/section of the site when a customer searches give relevance to the search term?
The methods for calculating the impact of the above don’t differ much from the earlier examples. The important thing is to be able to measure the impact of an action (or sometimes the lack of action). Once the impact is quantified the above methods can be used to quantify the impact of improving a feature or removing a customer pain point.
Below I'll run through how to identify the impact but I won't take the calculation all the way through to quantifying the impact on revenue as it’s the same process as above. Most of these analyses rely on creating models that are not 100% accurate but they allow you to get a feel for the impact changes and improvements may have.
Flexible search
Here I’m making the assumption that if keywords are spelt incorrectly they do not return any search results.
This analysis can be a little tricky if you don’t capture if a keyword is spelt correctly or not. If that data is available segment those who search into the two buckets; ‘correct spelling’ and incorrect spelling’. The same logic for those who searched Vs. those who didn’t search can then be applied to these two new segments to get a quantifiable impact.
When the spelling data isn’t available we can estimate the impact by analysis to top keywords. Take the top ten search terms for a given site and analyse them independently. For example, if the top keyword for a given site was ‘alcohol’. Download all of the search keyword data and the number of times the keywords have been searched for. Next, filter the list for terms beginning with ‘alc’. I’m assuming here that the user has spelt the first 3 letters of the search term correctly. Then manually check the list and assign ‘correct spelling’, ‘incorrect spelling’ or ‘N/A’ labels to each of the search terms in the list. Use N/A if you feel the user was not trying to spell ‘alcohol’.
By comparing the search volumes for the ‘correct spelling’ and ‘incorrect spelling’ labels you can get a feel for the percentage of time a keyword is spelt incorrectly. Repeating this for a number of keywords can be used to give an average for the percentage of time keywords are spelt correctly. The more keywords analysed the better the average and the better the model.
To quantify the impact of introducing a flexible search algorithm that would account for spelling mistakes is more straightforward now and the above logic can be applied. For modelling purposes I would assume a flexible search algorithm could eradicate 80% of error but not 100%.
Intuitive search
Intuitive search is when the search tries to guess the keyword a user is inputting based on the first few letters the user types. It can greatly decrease the number of abandoned searches, especially if your site attracts keywords that are difficult to spell or prone to predictive text errors. For best results intuitive search should be constantly refreshed with the keywords users type into your site, this optimises the suggestions towards what others have searched for.
For intuitive search. I'd want to know how many users began a search but abandoned it before submitting a search query. This tracking is not available out-of-the-box with most analytics platforms but can be added as a customer data point. The important piece of data to capture here is clicks into a search box. Creating a segment where the visit includes a click in to the search box but does not include a search creates the abandoned search segment.
Most of the time abandoned searches will result in zero conversion and so the quantification for the value of intuitive search would be to assume these abandoned searches performed as well an non-abandoned search. Again, I would assume a 80% success rate for intuitive search rather than 100%.
Introducing filters
When introducing a brand new feature the impact is often unknown and so a model is needed to quantify the outcome. I’ve seen good filters increase search conversion by over 100% so the assumed impact of the filter is something to be careful of. As a guide, the more possible search results you have or if there is similarity between potential search results, the bigger impact filters will have.
I’d recommend introducing simple search functionality on a single factor. This will give you the data you need to estimate the impact of further filters. The improvement would be to the conversation rate from search metrics. Creating segments for those who did a successful search and used a filter vs. those who did a successful search but did not use a filter should be used in this analysis.
Search relevance
Google is the benchmark for this and I doubt anyone will surpass them. This year I read that 99% of all clicks to a Google search results pages occur on the first page and almost a third of clicks are to the first position.
To estimate the impact of getting more ‘position 1 clicks’ split your data by conversion rate from clicks to position 1, 2, 3 etc. Usually you see a drop off below a certain position. Create 2 segments, clicks from top positions (above this drop off) and clicks from lower positions (below the drop off). Moving a percentage of clicks from the lower clicks bucket to the top clicks bucket gives you change in segment volumes that you can model as above.
Intelligent search
This is a more advanced feature. The area / page of a site that a user is on when performing a search can give vital information about their intent. Searching for ‘tv’ on a help site probably means a user is looking for different information than if they searched for ‘tv’ from a product page.
Capturing the ‘assumed intent’ and the ‘likelihood score’ of the assumption can help optimise search performance. To quantify the impact of this we can segment search keyword performance by the page / site section the search occurred on. In the above example, are users more likely to click on a search result for the keyword ‘tv’ if they were on a product page vs a help page? Should the search algorithm treat these searches differently?
This analysis can be repeated for different keywords and different site areas to quantify the impact.
Conclusion
Internal search is one of the hardest things for a product team to get right. It’s complex and the benchmark set by Google is one that few, if any, will ever match. However, the benefits of getting this right are huge. Searching functionality is expected by users and most expect Google-like levels of performance.
In a wider context, I hope I've demonstrated that data can be used to prioritise the features a product team can work on and how outcomes can be quantified. This methodology can be applied to all aspects of product development. Using data to drive product development ensures you’re building the right thing that will have a positive impact on your customers.
If you’d like to discuss how a data-driven approach could improve your product delivery methods or are keen to know more about the Red Badger DUXD process please get in touch.
