How to analyze and find high value test ideas
Recently I have ran several tests for a specific team where we’ve seen double digit improvements in a goal we were measuring and testing against. The method I used to find the “What to test?” question was not that difficult. Over time, I believe it may be more difficult to find more wins, but here are some ideas which I think you can apply in your work to start with.
A lot of times, when you go to conferences people will tell you things like “use analytics”, but I know you folks want to know a little more than that. Here we go.
1. Find the top potential areas of issues or opportunity for high impact results
Best way to start the conversation with your business partner is to understand their business and what is important to them. They are the experts in their field. As an analytics expert you have to be a great listener.
In my case and I’m sure this applies to many of you, a lot of times the analytics may not be set up perfectly to track the key outcomes for the business users. So first typical thing to check is, rather the analytics tools are capturing the key end action data. Example, PDF downloads, contact sales form submit, order checkout complete, etc. That said, you have to understand what your business partner’s key end action they want the web site to accomplish. Track and analyze all your micro conversions (end actions)!!
Once your analytics tools are set up to track those end actions, work from the bottom and understand what drives the end action that really matter to the business. Goal is to understand and find key pages that are leading to that end action. So if you’re using Google Analytics, use ‘segments’ to create segments around people who converted or not converted, and analyze the pages people land on, bounce, or leave if they don’t convert.
You’ll soon find out particular key pages that aren’t home page, and start to give you questions that makes you want to know what is happening to those pages that showed up into your radar.
2. Understand the users, and capture qualitative data if you can
Never underestimate the voice of the customers. Tools like Qualaroo make it so easy for marketers and analysts to gather user’s voice from the site, and you have to start leveraging the data to better understand customer’s intent, pain points, opportunities, ideas, etc. User feedbacks in context of this topic could focus on the pain points or to understand if users were able to find the key elements on the page. Like asking “Did you come to the site for XYZ reason?”, and if they answer ‘yes’, then ask them “Were you able to accomplish XYZ” and focus generating questions to gain further insights on why customers could not accomplish.
All these qualitative insights can help you gain better understanding of the size of the problem, or potentially gain ideas on what to test. Remember that the quantitative data from tools like Google Analytics will only give you what is happening to the site, but doesn’t give you the “Why” things are happening in such way.
Heatmap and session recordings could be a great tool to use to further dive into the details of usability issues as well. Since heatmap and recordings are qualitative data those are great tools to have in your analysis exercise.
3. Prioritize based on what is important
By now, you have a list of pages where you think you can improve and test some ideas to make it better. Now the question to ask is which pages are more important. You could based that importance on several things like (not limited to): traffic volume, close to the end action, high value touch point, etc.
For example, look for high traffic page with high bounce rate. That could be the first indicator for something to look at and dig deeper. You can also look into surveys to see if there is anything site visitors are complaining about.
Another idea is to look at your valuable segment and the key landing pages that differs to that of the low value visitors and it’s landing pages. Give higher priority to the ones where visitors who convert are touching.
These priorities will help you set where the engineering or design resources go.
4. Focus on what you can control
Why work on something that you can not influence. You find an opportunity on a site that requires an entire layout change that requires some kind of engineering effort. That is probably not a good test to start with if there is an alternatives, like optimizing a bad content on a landing page. You want to be able to go after the wins you can really win and build up.
In one of my experience, I came across a micro site where the site was owned by another group. However, VOC survey indicated some issues on that micro site. Since my team couldn’t immediately impact and take control of that micro site, we then had to focus on what we could immediately take control of. Such as messaging from the site we have authority to change, etc.
5. Look for easy set up, easy changes
Take into consideration of resource, budget, and time factoring it into your assessment. In order to operate fast and test fast to try many things you’ll have to focus ease of test set up, and ease of change on site.
One important factor to think about is the quality of the test. I may come to sound like doing a lot of test fast is the way to go, but if you’ve done the steps 1 to 4 really well, then it doesn’t have to be about numbers of tests at a cost of quality of test. That is not the point. The main point is doing these steps or homework well so that you identify the valuable test ideas and really contribute to the business.
Hopefully these steps were abel to give you some good ideas on where to start and leverage data to build high value tests. Enjoy analyzing and testing!!