Loading Form...
Thank you
Aug 15, 2011 | 3 minute read
written by Linda Bustos
You may have heard the advice to avoid A/B and multivariate testing during a traffic spike. This is sound advice for the most part, but I suggest there are times when testing during a traffic spike is exactly what you should do.
Testing at the beginning of a predictable spike can actually help you optimize for sales during the remainder. More traffic means faster test completion, and changes can be applied for the rest of the wave. For example, if you can run test beginning Black Friday that reaches statistical significance in 5 days, you have a few weeks to benefit from the learnings of the test.
The reason why testing during a traffic spike (whether a seasonal spike or due to a new product launch, sales event or promotional campaign) is not advised is customer behavior may change during the busy period. This may introduce bias into your test which can skew your results. But we need to think this through, and ask ourselves which tests could be affected by different user behavior. Will back-to-school shoppers behave differently than regular folks during a checkout test where a button is made more prominent? Or can a test be reasonably applied to all visitors under any circumstance?
A common test variable is the size/color/placement/label of a checkout button. We wouldn't expect holiday shoppers to be more influenced by a red or green button than any other shopper at any other time. The test is measuring the conversion impact for a change in design, not merchandising or promotions - the variables that can truly move the needle during high season and other heavy traffic periods. These are variables you can't reliably test during slower periods - behavior is different.
Here are some examples of seasonal test ideas (hint: they are all related to merchandising, pricing and promotions):
Tips for testing during high traffic periods
1. Know when to start and stop your test. Don't start before the spike or let it continue beyond2. Craft your test to match expected behavior during the spike (relevance!)3. Focus on variables that have the highest potential impact on sales4. Understand traffic and conversion improvement estimates to ensure your test will finish quickly during the spike (use an estimate calculator)5. Use a limited number of test recipes so the test finishes quickly6. Complete other 'general' tests before the seasonal/spike test to reduce bias from other tests (e.g. customer sees free shipping over $50 on home page but encounters a one-page checkout test when ready to buy7. Communicate with your email and PPC departments to sync up on dates, or exclude email and PPC traffic from seasonal tests)
What about unexpected spikes?
Was your business featured on Oprah Rachel Ray during a test? Did your PPC squad turn up the juice on your paid search campaign? You may want to re-test after traffic returns to normal and compare to the spike to validate the results.
Wanna learn more about site optimization testing? We're teaming up with Marketing Sherpa for our August webinar Putting Your Best Site Forward: Making the Case for Consistent Testing on Enterprise Ecommerce Sites on Wednesday, August 24, 2011 from9:00 to 9:45am PDT.