One of the best reasons to do website optimization is for the wins, when you’ve found a change that delivers real revenue to the bottom line.
But before you celebrate, check out this infographic for Marshall Downy’s sobering presentation at Which Test Won’s The Live Event. The event was held in Austin, Texas, The Conversion Capital of the World.
Marshall is with Build.com and gave several examples of post test analysis that changed the decisions he would make based on the pure test data.
What is post-test analysis? It’s what comes after you’ve completed a split test or multivariate test and have a winning change.
The problem is that we often can’t test the right metric to determine if a winner is actually helping the business. A typical example of this is products with long sales cycles. You can’t really test much if you’re waiting six months to see which leads close. You can increase the lead conversion rate, but you’ll always wonder if the lead quality was the same.
Another example is subscription services. Your test may show you how to get more subscribers, but what if the cancellation rate goes up?
Marshall lists the following types of post-test analysis to help us evaluate the true impact of our test results.
1. Customer Satisfaction Scores – If the customers aren’t as satisfied, it may not matter if you’re selling more.
2. Return Rate – If significantly more people are returning the product, increased sales may not have been good for profits.
3. Profitability – I can increase your conversion rates by slashing your prices, but will that really help the business?
4. Customer Lifetime Value – An important metric for subscription and repeat-purchase businesses.
5. Brand- and Category-specific sales – What if we increase sales of one product line at the cost of another. Also see “Cannibalism.”
6. Signup to Purchase Rate – You may get more triers, but are they turning into buyers?
Marshall didn’t share his slides, but here is my Instagraphic infographic from his presentation.