Conversion Optimization: Dont Rely On Case Studies

Making a decision on adding a new usability feature or modifying your design for higher conversion? You may be tempted to use case study data to support your decision. But could case studies lead you astray?

The danger in relying on case studies

With A/B testing, there are many variables that affect test results. Every analysis must consider test context. What works for a pureplay may not work for a bricks-and-clicks retailer or manufacturer. Customers in the UK may behave differently than US shoppers, both on domestic and international sites. Product category, price points, purchase role, and even time of year can be factors. A strong brand with die-hard fans will find it more difficult to move the needle in a checkout redesign than sites with many comparable competitors. Customers will put up with all kinds of usability issues for must-have, exclusive items.

Often, test variables are interdependent with the website's design. A red cart button may show a greater lift on a site with a blue "theme" than one with a yellow or red color scheme. A free shipping banner may be more effective on a site with few "merchandising zones" than one that is jam-packed with banners. Not to mention, you may measure a conversion differently than the website in question. We must the tendency to conclude "this worked for xyz.com, this must be how all web users behave."

Recall the case we covered on Get Elastic recently where WineExpress.com found more emphasis on sale messaging lowered conversion. This would not be the case on every site.

Does that mean they case studies can never be helpful?

I'm not case study bashing, we've covered several on this blog. But I believe case studies should be taken as food-for-though, not proof-text.

• Case studies can help you build your case to your upper management/C level executives who may be on the fence about testing or skeptical about testing. (You know, the win-over-your-HiPPO-thing.)

• They can also spur you on to challenge the “sacred cows” on your site, and give you ideas for testing. For example, you may be married to your one-page checkout process -- everyone believes it's "best practice," but it's never been tested against alternatives. A case study that found multi-step outperformed one-page could open your mind.

• Case studies from your industry or a competitor are typically more relevant and may help you understand your customer/industry more. They can help you shape a hypothesis for testing.

Case study data sources

Where can you find case study data? Here are my favorites:

WhichTestWon.com
MarketingSherpa
Marketing Experiments Research Journal
ABtests.com
e-commerce and conversion optimization blogs (including those of testing consultants and vendors, like the Eisenberg Brothers, Unbounce and WiderFunnel)
Google search
InternetRetailer and Shop.org
Slideshare

Wanna learn more about site optimization testing? We're teaming up with Marketing Sherpa for our August webinar Putting Your Best Site Forward: Making the Case for Consistent Testing on Enterprise Ecommerce Sites on Wednesday, August 24, 2011 from
9:00 to 9:45am PDT.