Are You Ever Done Testing A Landing Page?

If you missed the case study on WhichTestWon.com conducted by Wider Funnel for WineExpress.com, here is your opportunity to test your gut-feel. Can you guess which version resulted in 41% higher revenue per visitor and 5% lift in conversion rate?

Version A: Note the video content is higher above the fold, and the pricing box has a smaller cart button, and less emphasis on sale messaging:

Version B: Larger, more prominent calls to action and greater emphasis on value and low shipping price:

Time's up.

If you picked Version A, you are correct! Ding, ding, ding, tell 'em what they've won, Bob...

Contrary to ecommerce "best practice" of a big, juicy, in–your–face cart button and sale, sale, sale messaging, the winner was the classier, and more subtle design. This confirms the WineExpress customer cares more about the emotional experience of a fine wine than the price. The design that de-emphasizes the discount and shows more of the expert wine tasting video above the fold better reflects the value proposition of learning about and enjoying great wine, not belonging to a bargain club.

With this kind of result, it may be tempting to break out in high fives, pop a cork and celebrate - and move on to another testing page. But why stop at one victory, when there may be some more juice in this page's grapes?

Testing further

For a page or process that is this important to your bottom line, further rounds of testing can bring you more revenue lift than testing a less important page from scratch. The follow-up test for WineExpress took the learnings (yes, I'm using this as a word) about consumer behavior from the first win and applied it to another round of A/B testing.

Again, let's test our own gut feel, but this time with some context. The WineExpress customer is affluent, older, sophisticated, knowledgeable about wines and values the experience of buying wine. Wine Express' real competition is not discount wine site, but the local specialty wine shop. The biggest barrier for this customer profile is shipping costs, when there is a local shop available. This test is to determine whether showing a countdown clock at the top of the page that emphasizes urgency and reinforces the $.99 shipping value proposition will outperform a landing page without it. So, which test won round two?

Version A: Countdown clock to create urgency

Version B: Countdown clock removed

The result was an additional 7% lift for Version B, the one without the clock (and remember, this is lift upon lift, an additional lift out of the previous winning page). WineExpress customers were once again psychologically influenced by a cleaner, more sophisticated design.

A/B testing tells us more than even customer surveys can. If you surveyed customers and ask them whether they like to receive discounts, or if they are price sensitive, or that if would behave a certain way under certain circumstances, they are likely to respond from the conscious mind. The unobserved customer will behave according to the subconscious mind, and we can learn what truly motivates.

But I digress. Back to the purpose of this blog post.

When is it time to stop testing a page?

There is no magic number of times that you should test one page or process. While it's wise to move on when you reach the point of diminishing returns, unimpressive test results can also be the result of testing small ideas instead of big ideas.

Some landing pages are so critical to your business, that ongoing rounds of testing will continue to improve your performance. But keep in mind, when you have a control that's hard to beat, there is also a risk in sending half of your traffic to an underperforming test version.

It is as much art as it is science, and you must exercise judgment. The key is to not sell yourself short by testing once or twice when there is untapped potential on your page, while avoiding the trap of testing micro-variables that provide little if any incremental lift, or which are unlikely to have long-term impact [half-life].

Looking for help with ecommerce? Contact the Elastic Path consulting team at consulting@elasticpath.com to learn how our ecommerce strategy and conversion optimization services can improve your business results.