I was emailing a prospect recently who mentioned that a competing firm had proposed doing A/B multivariate testing. If you’re familiar with the jargon of testing different ads/landing pages, you would know that A/B testing is different from multivariate testing. I can’t blame the prospect or my competition however, because ours is an industry enamoured with jargon and it sometimes gets me confused too! In any case, let’s see what A/B testing and multivariate testing really mean, and the implications for your test results.

(Bucket testing image by mil8.)

A/B testing is defined as testing different values for the same variable. For example, suppose you have three landing pages, called A, B, and C. You can carry out A/B testing by changing the titles on each landing page. The variable there would be the title, and each the A’s title, B’s title and C’s title would each be a value for that variable.

It isn’t about having only two versions and comparing one to the other. You can do A/B/C testing or even go super deep and do A/B/[...]/Z testing! A/B testing is also sometimes referred to as split testing or A/B split testing because you split your traffic (or your “sample,” if you’re testing something offline, like in-store displays)

Multivariate testing is defined as testing multiple values for multiple variables, simultaneously (hence the name “multivariate”). Let’s say we’re still working with pages A, B and C. Instead of just testing different titles, we’d also try different colour ‘buy now’ buttons, emphasizing different benefits in the copy and displaying different testimonials.

This is all done at the same time. If you tested each of these variables separately, you’d really just be carrying out A/B testing!

To give a human analogy, A/B testing is like having two brothers growing up side-by-side. They’re raised exactly the same way and are identical in every way, except that they play different sports. You’d be testing for the result of being enrolled in each of those sports and see who got a better job, or earned more money, for example.

Multivariate testing would take those brothers’ children, and have them being raised very differently. The variables there would be the brothers’ different wives, the ways they were being raised, their environments etc. You could still test to see who got better jobs and/or earned more money.

As this example shows, there’s a big difference in what you can learn from the results of these two types of testing.

With the two brothers, we know that since everything else was the same, any difference in job quality or salary is directly attributable to the sports they played. If the soccer-playing one got a better job, then you know that raising a kid to play soccer is better for the child than raising him on say, baseball. (Statistical significance is another issue for another day.)

With the various cousins, we can make educated guesses about what caused the differences in job quality and salary. Perhaps brother A’s wife worked in the recruitment industry and helped her kids more than her sister-in-law could help her kids. Perhaps the neighbourhood where brother B and his wife raised their kids didn’t have as good schools. You can make good guesses about why you got a particular result, but you can’t be certain.

Note: To my more advanced readers, I’m vaguely aware that depending on your sample size and how you play with the numbers, you can get a very high degree of certainty (if not 100%) with multivariate testing. My point above is made at a basic level, and is the main distinction between A/B testing and multivariate testing.

Update 1: Sphinn button is broken, but you can [should? ] sphinn the story here. Also, I forgot to point it out, but if analytics are your bag, check out these 3 new metrics for measuring social media and this post on messing with competitors’ analytics.

Update 2: Tim from Convert Offline gets our first dofollow of the day . He’s a local hound apparently. Me too! Miriam’s going to interview him too, so keep your eyes peeled! (No, that doesn’t mean that you should put a sharp metal implement near your cornea. )

Update 3: More link love abounds! The very succesful Brian Chappell gets his and Hannah Smith gets some for the Gravy Train and her post on rogue Adwords due to personalization.

Please let me know in the comments whether this helped clear up to you what A/B testing is vs multivariate testing. Also, please do comment about how you’re using A/B testing and/or multivariate testing and any case studies/experience you might have in the matter! Testing is a fascinating topic and while I know a little bit about it, I’m sure many of you could teach me more! Good comments get dofollow links, as usual.
Liked the post? Get my rss feed.