Choosing Between A/B or Multivariate Test Design: Optimization Testing 101

If you’ve been trekking along with us in our Site Testing 101 series, you’re already familiar with the difference between A/B and multivariate testing (opens in new window if you haven’t read the article and would like to get up to speed). Today’s installment looks at choosing the appropriate testing method for your experiment.

A/B Split Testing

Use A/B split testing when:

You are just getting your feet wet with testing. A/B testing is simpler, so you can complete tests faster. Get a few “wins” under your belt before moving on to more complex experiments.

You need to test only one variable. Your hypothesis may be "showing testimonials on the landing page will increase conversion." You need only to test the presence of the testimonials against the absence. Or, you may hypothesize that a security seal will improve conversion. The security seal is one variable, but you test it in the top right hand corner, the bottom left, the top left, adjacent to the cart button and no security seal at all for an A/B/C/D/E split test.

You are testing a radical redesign. Radical redesign may refer to a completely new web site look-and-feel with different content template, logo, search box design, navigation menus, categorization, etc. Or, it could refer to a radically redesigned landing page. It’s a good idea to test radical concepts before multivariate testing to avoid micro-tweaking a sub-optimal design template.

You are restricted by your tool. Though less common now that Google Website Optimizer is available to all, you may be restricted to A/B split testing by the tool you use (perhaps it’s a part of your ecommerce platform).

The page or process you are testing has relatively few conversions per day. It's not just about traffic. To reach statistical significance, you need to measure completed goals. If conversions are too low (despite high traffic), you simply can’t test more than a few versions at once. It will take too long.

The longer your test runs, the more bias the test is exposed to which may skew results (users whose cookies expire or are deleted may see multiple versions of your test, your product catalog may turn over, economic changes may change consumer behavior, your promotion strategy may change, seasonal traffic spikes may occur, etc). It also takes longer before you can benefit from your learnings.

Multivariate (MVT) Testing

Use MVT when:

You’re tweaking a winner. If you’ve performed radical redesign tests with A/B testing and have reached diminishing returns testing challenger designs, you’re ready to fine-tune your page to squeeze even more improvement.

When you need to test more than 2 variations for each variable. Remember, A/B tests are either univariate or “radical redesigns.” Though you could test 2 variables with 2 variations each with a split test tool (it’s only 4 combinations), your experiment is by definition a multivariate test. When you implement your test as multivariate, you will be able to analyze a “Page Section Report," which allows you to see which variations perform best for each variable in aggregate. An A/B design will only show you the winning recipe. What Google Website Optimizer calls the "Page Section Report" may be called something else in other tools.

You need to test more than 10 “recipes” at once. When your test design requires testing of variables that impact each other (headline, image, price and testimonial combination, for example) you may quickly exceed the maximum number of versions a single A/B test will bear. Rather than perform successive rounds of A/B testing (which exposes you to testing bias and is inefficient), it’s better to test all versions with the same traffic during the same time period.

Your traffic and conversions are sufficient. This is a pre-requisite for proper multivariate testing. There’s no “magic threshold” of traffic/conversions that you need to qualify, as it all depends on the number of recipes you want to test, your existing conversion rate, your desired confidence level and your expected lift. There are test duration estimators out there, but they are not perfect. The trick is estimating the conversion improvement. If you over-estimate your improvement, it will take longer to gather enough data to “prove it” statistically.

You cannot test a radical redesign. You may have technological restraints or design/branding guidelines you must stick to, which restricts you to only testing variables within your template. In that case, using MVT will be more efficient than testing rounds and rounds of univariate experiments.

Choosing the appropriate testing method is part of your overall testing process. Join us next post for Part 4 in our series: Developing an Optimization Process.

Previous posts in Site Optimization 101

Optimization Testing 101: Part 1 – Introduction to A/B and MVT
Optimization Testing 101: Part 2 – 4 Reasons for Site Testing