Website Experimentation Made Fast and Easy
With the recent surge in popularity of the Lean Startup philosophy developed by Eric Ries, many people have become familiar with terms such as MVP (minimum viable product, the least-effort deliverable that can be used to validate assumptions with real customers).
Split or A/B testing is another Lean Startup concept and one that we’ve been using at Go Daddy for as long as I’ve been around (9 years this month!). A split test is an experiment on a website in which some visitors are shown a modified version of a page or pages while others continue to see the “old” version. Some goal, such as increased clicks or conversions, is measured and compared between the test and control groups to determine if the new version achieves the desired goal.
Split tests are a great way to validate changes, but they’re also a challenge. Ideally, we want to run a large number of tests and quickly learn from each one. But, changing something on a website – especially a highly dynamic eCommerce site like GoDaddy.com - requires effort and care. This work can be minimized by using yet another Lean Startup concept called continuous deployment (I’ll save that topic for another day). Ultimately, to perform the volume of tests we want, we need to go from continuous deployment to no deployment.
A project I’ve been working on recently led to an “Aha!” moment: there are products out there that do exactly this for websites. These products fall under the generic category of “multivariate testing software.” There’s even a website that does comparisons and reviews of these tools at www.whichmvt.com (I’m not endorsing the reliability of the ratings on this site – do your homework).
With all of these products, the modifications being tested are made via JavaScript in the browser. This provides a number of benefits, the most important of which is that no server-side code needs to be changed to implement a new split test. These systems also make it easy to keep track of all your tests in one place and turn them on and off at will. And, if for some reason the testing service fails, you may lose the ability to run tests, but your default website will not be impacted.
To gain insight from a split test, you have to measure the impact it has on your goal. The testing systems we reviewed all contained their own tracking mechanisms and reports. Most can also be integrated with an analytics service like Google Analytics. The result is a complete solution that puts the power of split and multivariate testing into the hands of the marketing and product teams, enabling a lean startup practice in your organization.
Got something to say? Go for it!