split testing

Multivariate testing offers high-traffic websites the ability to find the right combination of features and creative ideas to maximize conversion rates.

However, it is not sufficient to simply throw a bunch of ideas into a pot and start testing. This article answers the question, What is a multivariate test?, explains the advantages and pitfalls of multivariate testing, and offers some new ideas for the future.

If you run a relatively high-traffic site, consider this question: Will I profit from running multivariate tests?

Before we dive into the question, let’s be sure to define the terms. I’ll talk about the dangers of doing multivariate tests (MVT) and when you should consider using them.

What Is Multivariate Testing?

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified.

Multivariate testing is distinct from A/B testing in that it involves the simultaneous observation and analysis of more than one outcome variable. Instead of measuring A against B, you are measuring A, B, C, D & E all at once.

Whereas A/B testing is typically used to measure the effect of more substantial changes, multivariate testing is often used to measure the incremental effect of numerous changes at once.

This process can be further subdivided in a number of ways, which we’ll discuss in the next section.

Multivariate, Multi-variant or Multi-variable

For this article, we are focusing on a specific way of testing in which elements are changed on a webpage. Before we dive into our discussion of multivariate testing, we should identify what we are talking about and what we are not talking about.

One of the frequent items tested is landing page headlines. Getting the headline on the page can significantly increase conversion rates for your landing pages. When testing a headline, we often come up with several variants of the words, testing them individually to see which generates the best results.

A multi-variant tests multiple variants of one variable or element. Find out more about multivariate testing.

A multi-variant tests multiple variants of one variable or element.

This is a multi-variant test. It changes one thing–one variable–but provides a number of different variants of that element.

Now suppose we thought we could improve one of our landing pages by changing the “hero image” as well as the headline. We would test our original version against a new page that changed both the image and the headline.

An example of a multi-variable test. Here we are testing the control against a variation with two changes, or two variables.

An example of a multi-variable test. Here we are testing the control against a variation with two changes, or two variables.

This is a multi-variable test. The image is one variable and the headline is a second variable. Technically, this is an AB test with two variables changing. If Variation (B) generated more leads, we wouldn’t know if the image or the headline were the biggest contributors to the increase in conversions.

To thoroughly test all combinations, we would want to produce multiple variations, each with a different variant of the variable.

Two variables with two variants each yield four page variations in this multivariate testing example.

Two variables with two variants each yield four page variations in this multivariate testing example.

In the image above, we have four variations of the page, based on two variables (image and headline) each having two variants. Two variables times two variants each equals four variations.

Confused yet?

A multivariate test, then, is a test that tests multiple variants of variables found on a page or website.

To expand on our example, we might want to find the right hero image and headline on our landing page. Here’s the control:

The Control in our multivariate test is the current page design. A multivariate test, then, is a test that tests multiple variants of variables found on a page or website.

The Control in our multivariate test is the current page design.

We will propose two additional variants of the hero image–for a total of three variants including the control–and additional two variants of the headline, three including the control.

Here are the three images:

We want to vary the hero image on our page. This variable in our test has three variants.

We want to vary the hero image on our page. This variable in our test has three variants.

Here are three headlines, including the existing one.

  1. Debt Relief that Works
  2. Free Yourself from the Burden of Debt
  3. Get Relief from Debt

A true multivariate test will test all combinations. Given two variables with three variants each, we would expect nine possible combinations: three images x three headlines.

Here’s another example that will help you understand how variables, variants and variations relate. An ecommerce company believes that visitors are not completing their checkout process for any of three reasons:

  1. The return policy is not visible
  2. They are required to register for an account
  3. They don’t have security trust symbols on the pages

While these all seem like reasonable things to place in a shopping cart, sometimes they can work against you. Providing this kind of information may make a page cluttered and increase abandonment of the checkout process.

The only way to know is to test.

How many variables do we have here? We have three: privacy policy, registration and security symbols.

How many variants do we have? We have two of each variable, one variant in which the item is shown and one variant in which it is not shown.

This is 2 x 2 x 2, or eight combinations. If we had three different security trust symbols to choose from, we would have four variants, three choices and none. That is 2 x 2 x 4, or sixteen combinations.

We’ll continue to use this example as we explore multivariate testing.

Why Multivariate Testing Isn’t Valuable In Most Scenarios

A multivariate test seeks to test every possible combination of variants for a website given one or more variables.

If we ran an MVT for our ecommerce checkout example above, it would look something like this:

Variations multiply with multivariate tests requiring more traffic and conversions.

Variations multiply with multivariate tests requiring more traffic and conversions.

There are many reasons that multivariate testing is often the wrong choice for a given business, but today, I’m going to focus on five. These are the five reasons multivariate tests (MVTs) are not worth doing compared to A/B/n tests:

  1. A lack of time or traffic
  2. Crazy (and crappy) combinations
  3. Burning up precious resources
  4. Missing out on the learning process
  5. Failing to use MVT as a part of a system

Let’s take a closer look at each reason.

1. Multivariate Tests Take a Long Time or a Whole Lot of Traffic

Traffic to each variation is a small percentage of the overall traffic. This means that it takes longer to run an MVT. Lower traffic means it takes longer to reach statistical significance, and we can’t believe the data until we reach this magical place.

Statistical Significance is the point at which we are confident that the results reported in a test will be seen in the future, that winning variations will deliver more conversions and losing variations would deliver fewer conversions over time. Read 2 Questions That Will Make You A Statistically Significant Marketer or hear the audio.

Furthermore, statistical significance is really measured by the number of successful transactions you process.

For example, MXToolbox offers free tools for IT people who are managing email servers, DNS servers and more. They also offer paid plans with more advanced features. MXToolbox gets millions of visitors every month, and many of them purchase the paid plans. Even with millions of visits, they don’t have enough transactions to justify multivariate testing.

It’s not just about traffic.

This is why MVTs can be done only on sites with a great deal of traffic and transactions. If not the tests take a long time to run.

2. Variations Multiply Like Rabbits

As we saw, just three variables with two variants resulted in eight variations, and adding two more security trust symbols to the mix brought this to sixteen combinations. Traffic to each variation would be reduced to just 6.25%.

Multivariate testing tools, like VWO and Optimizely offer options to test a sample of combinations — called Partial, or Fractional Factorial testing — instead of testing them all, which is called Full Factorial testing. We won’t dive into the mathematics of Full Factorial and Partial Factorial tests. It gets a little messy. It’s sufficient to know that partial factorial (fractional factorial) testing may introduce inaccuracies that foil your tests.

What’s important is that more variations mean larger errors… because statistics.

Every time you add another variation to an AB test, you increase the margin of error for the test slightly. As a rule, Conversion Sciences allows no more than six variations for any AB test because the margin of error becomes a problem.

In an AB test with two variations, we may be able to reach statistical significance in two weeks, and bank a 10% increase in conversions. However, in a test with six variations, we may have to run for four weeks before we can believe that the 10% lift is real. The margin of error is larger with six variations requiring more time to reach statistical significance.

Now think about a multivariate test with dozens of variations. Larger and larger margins of error mean the need for even more traffic and some special calculations to ensure we can believe our results aren’t just random.

Ultimately, most of these variations aren’t worth testing.

All eight variations in our example make sense together. As you add variations, however you can end up with some crazy combinations.

Picture this:

It’s pouring down rain. You are camping with your son.

While huddled in your tent, you fire up your phone’s browser to find a place to stay. While flipping through your search results on Google, your son proclaims over your shoulder, “That one has a buffet! Let’s go there, Dad!”


The last time he ate at an all-you-can-eat buffet, he was stuck in the restroom for an hour. Not a pretty picture.

Then again, neither is staying out in the wretched weather. So you click to check out the site.

Something is off.

The website’s headline says, “All you can eat buffet.” But nothing else seems to match. The main picture is two smiling people at the front desk, ready to check you in.

As you scroll to the bottom, the button reads “Book Your Massage Today”.

Is this some kind of joke?

As strange as this scenario sounds, one problem with MVTs is that you will get combinations like this example that simply don’t make sense.

This leaves you with two possibilities:

  1. Start losing your customers to variations you should not even test (not recommended).
  2. Spend some of your time making sure each variation makes sense together.

The second option will take more time and restrict your creativity. But even worse, now you need more traffic in order for your test.

With an A/B/n test, you pick and choose which tests you like and which to exclude.

Some may argue it can be time-consuming to create each A/B/n variation while a multivariate test is an easy way to test all variations at once.

Think of a multivariate test as a system that automatically creates all possible combinations to help you find the best outcome. So on the surface, it sounds appealing.

But as you dig into what’s really going on, you may think twice before using an MVT.

3. Losing Variations are Expensive

Optimization testing can be fun. The chance of a breakthrough discovery that could make you thousands of dollars is quite appealing. Unfortunately, those variations that underperform the Control reduce the number of completed transactions and fewer transactions means less revenue.

Every test — AB or Multivariate — has a built in cost.

Ideally, we would let losing variations run their course. Statistically, there is a chance they will turn around and be a big winner when we reach statistical significance. At Conversion Sciences, we monitor tests to see if any variations turn south. If a losing variation is costing us too many conversions, we’ll stop it before it reaches statistical significance. This is how we control the cost of testing.

This has two advantages.

  1. We can control the “cost” of an AB test.
  2. We can direct more traffic to the other variations, meaning the test will take less time to reach significance.

When tests run faster, we can test more frequently.

On the other hand, multivariate tests run through all variations, or a large sample of variations. Losers run to statistical significance and this can be very expensive.

Lars Lofgren, former Director of Growth at KISSmetrics, mentioned that if a test drops below a 10% lift, you should kill it. Here’s why:

What would you rather have?

  • A confirmed 5% winner that took 6 months to reach
  • A 20% winner after cycling through 6-12 tests in that same 6 month period

Forget that 5% win, give me the 20%!

So the longer we let a test run, the higher that our opportunity costs start to stack up. If we wait too long, we’re foregoing serious wins that we could of found by launching other tests.

If a test drops below a 10% lift, it’s now too small to matter. Kill it. Shut it down and move on to your next test.

Keeping track of all the MVT variations isn’t easy to do (and also is time consuming). But time spent on sub-par tests are not the only resource you lose either.

4. It’s Harder to Learn from Multivariate Tests

Optimization works best when you learn why your customers behave the way that they do. Perhaps with an MVT you may find the best performing combination, but what have you learned?

When you run your tests all at one time, you miss out on understanding your audience.

Let’s take the example from the beginning of this article. Suppose our multivariate test reported that this was the winning combination:

If this combination wins, can we know why?

If this combination wins, can we know why?

What can we deduce from this? Which element was most important to our visitors? The return policy? Removing the registration? Adding trust symbols?

And why does it matter?

For starters, it makes it easier to come up with good test hypotheses later on. If we knew that adding trust symbols was the biggest influence, we might decide to add even more trust symbols to the page. Unfortunately, we don’t know.

When you learn something from an experiment, you can apply that concept to other elements of your website. If we knew that the return policy was a major factor, we might try adding the return policy on all pages. We might even test adding the return to our promotional emails.

Testing is not just about finding more revenue. It is about understanding your visitors. This is a problem for multivariate tests.

5. Seeing What Sticks Is Not An Effective Testing System

Multivariate tests are seductive. They can tempt you into testing lots of things, just because you can. This isn’t really testing. It’s fishing. Throwing a bunch of ideas into a multivariate test means you’re testing a lot of unnecessary hypotheses.

Testing follows the Scientific Method:

  1. Research the problem.
  2. Develop hypotheses.
  3. Select the most likely hypotheses.
  4. Design experiments to test your hypotheses.
  5. Run the experiment in a controlled environment.
  6. Evaluate your results.
  7. Develop new hypotheses based on your learnings.

The danger of a multivariate test is that you skip steps 3, 4 and 7, that you:

  1. Research the problem
  2. Develop hypotheses.
  3. Throw them into the MVT blender
  4. See what happens.

Andrew Anderson said it well

The question is never what can you do, but what SHOULD you do.

Just because I can test a massive amount of permutations does not mean that I am being efficient or getting the return on my efforts that I should. We can’t just ignore the context of the output to make you feel better about your results.
You will get a result no matter what you do, the trick is constantly getting better results for fewer resources.

When used with the scientific method, an A/B/n test can give you the direction you need to continually optimize your website.

Machine Learning and Multivariate Testing

Multivariate testing is now getting a hand from artificial intelligence. For decades, a kind of program called a neural network has allowed computers to learn as they collect data, making decisions that are more accurate than humans using less data. These neural networks have only been practical in solving very specific kinds of problems.

Now, software company Sentient Ascend has brought a kind of neural network into the world of multivariate testing. It’s called an evolutionary neural network or a genetic neural network. This approach uses machine learning to sort through possible variations, selecting what to test so that we don’t have to test all combinations.

These evolutionary algorithms follow branches of patterns through the fabric of possible variations, learning which are most likely to lead to the highest converting combination. Poor performing branches are pruned in favor of more likely winners. Over time, the highest performer emerges and can be captured as the new control.

These algorithms also introduce mutations. Variants that were pruned away earlier are reintroduced into the combinations to see if they might be successful in better-performing combinations.

This organic approach promises results faster and with less traffic.

Evolutionary neural networks allow testing tools to learn what combinations will work without testing all multivariate combinations.

Evolutionary neural networks allow testing tools to learn what combinations will work without testing all multivariate combinations.

With machine learning, websites that had too little traffic for pure multivariate testing can seriously consider it as an option. At Conversion Sciences, we leverage the use of machine learning tools for AI Optimization of high-traffic websites with outstanding success.

If you would like to explore how our CRO with machine learning services can give you a more accurate targeting of your audience and a revenue lift, schedule your call.

Final Thoughts: Is There Ever a Case For doing MVTs?

There are instances when introducing many variables is sometimes difficult to avoid or better to focus on.

Chris Goward of WiderFunnel gives four advantages to doing MVTs over A/B/n tests:

  1. Easily isolate many small page elements and measure their individual effects on conversion rate
  2. Measure interaction effects between independent elements to find compound effects
  3. Follow a more conservative path of incremental conversion rate improvement
  4. Facilitate interesting statistical analysis of interaction effects

He later admits, “At WiderFunnel, we run one Multivariate Test for every 8-10 A/B/n Test Rounds.”

Both methods are valuable learning tools.

What is Your Experience?

It is a bit of heated subject between optimization experts. I’d be curious to hear from you about your ideas and experience on what matters the most. Please leave a comment.

The fight for online leads and sales has traditionally been fought at the search engine. That is changing.

Web analytics, bid management, competitive intelligence, ad testing and ad management tools are all common staples of any serious paid search effort. Return on ad spend (ROAS) is being tracked all the way through the sign up or purchase process and ad strategies are being adjusted accordingly.

Quietly, the battle for online leads is moving to a new front. This new front is measured by revenue per visit, and it’s kissing cousin, conversion rate. Like the tide that floats all boats, website optimization is being seen as the way to reduce all marketing costs by dropping the acquisition cost of new prospects and customers.

Why do we say this is happening quietly? That is the conclusion we came to when examining an unusual data set from SpyFu.com. We were able to determine which businesses had conversion optimization tools installed on their website. This, we reasoned, gave us a pretty good idea of which businesses would dominate in the world of online marketing — assuming they were actually using the tools.


In this month’s podcast, based on the Marketing Land column Data Exposes Scandalously Low Adoption Of Conversion Optimization Tools, Brian the Conversion Scientist explores the usage of conversion optimization tools for two industry segments: Higher Education and B2B Software.

In one report, 73% of businesses are spending between $500 and $5000 per month on paid search ads. Almost a quarter are spending between $5000 and $50,000 per month. Yet, only 14% of businesses have at least one website optimization tool installed.

Who are going to be the winners in this new front? Where does your business fit in this statistic?

To get the most out of his column, download one of the free reports that share all of the data he uses.

In these reports you will learn:

  • Why your team needs time to review analytics.
  • Why businesses with smaller ad budgets should focus more on acquisition costs.
  • How to decrease your Search Ad costs.
  • Why you shouldn’t invest in social media sharing.

You’ve read the blog posts and you’ve heard from the vendors. A/B testing is a lot more difficult than you can imagine, and you can unintentionally wreak havoc on your online business if you aren’t careful.

Fortunately, you can learn how to avoid these awful A/B testing mistakes from 10 CRO experts who tell all in this Content Verve article. Here’s a quick look at some of their greatest pitfalls:

Joel Harvey, Conversion Sciences Worst A/B Testing Mistake

“Because of a QA breakdown we didn’t notice that the last 4-digits of one of the variation phone numbers displayed to visitors was 3576 when it should have been 3567. In the short time that the offending variation was live, we lost at least 100 phone calls.”

Peep Laja, ConversionXL Worst A/B Testing Mistake

“Ending tests too early is the #1 mistake I see. You can’t “spot a trend”, that’s total bullshit.” Tweet

Craig Sullivan, Optimise or Die Worst A/B Testing Mistake

“When it comes to split testing, the most dangerous mistakes are the ones you don’t realise you’re making.” Tweet

Alhan Keser, Widerfunnel.com Worst A/B Testing Mistakes

“I had been allocated a designer and developer to get the job done, with the expectation of delivering at least a 20% increase in leads. Alas, the test went terribly and I was left with few insights.”

Andre Morys, WebArts.de Worst A/B Testing Mistake

“I recommend everybody to do a cohort analysis after you test things in ecommerce with high contrast – there could be some differences…”

Ton Wesseling, Online Dialogue Worst A/B Testing Mistake

“People tend to say: I’ve tested that idea – and it had no effect. YOU CAN NOT SAY THAT! You can only say – we were not able to tell if the variation was better. BUT in reality it can still be better!”

John Ekman, Conversionista Worst A/B Testing Mistake

“AB-testing is not a game for nervous business people, (maybe that’s why so few people do it?!). You will come up with bad hypotheses that reduce conversions!! And you will mess up the testing software and tracking.”

Paul Rouke, PRWD Worst A/B Testing Mistake

“One of the biggest lessons I have learnt is making sure we fully engage, and build relationships with the people responsible for the technical delivery of a website, right from the start of any project.”

Matt Gershoff, Conductrics Worst A/B Testing Mistake

“One of the traps of testing is that if you aren’t careful, you can get hung up on just seeing what you DID in the past, but not finding out anything useful about what you can DO in the future.”

Michael Aagaard, ContentVerve.com Worst A/B Testing Mistakes

“After years of trial and error, it finally dawned on me that that the most successful tests were the ones based on data, insight and solid hypotheses – not impulse, personal preference or pure guesswork.”

Don’t start your next search marketing campaign without the guidance of our free report. Click here to download How 20 Search Experts Beat Rising Costs.

21 Quick and Easy CRO Copywriting Hacks to Skyrocket Conversions

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

  • 43 Pages with Examples
  • Assumptive Phrasing
  • "We" vs. "You"
  • Pattern Interrupts
  • The Power of Three
  • This field is for validation purposes and should be left unchanged.

My partner Joel Harvey is fond of saying, “My favorite part of a design is the money.” He’s been part of many a web design project. His perspective comes in response to the number of times he’s heard things like:

“I want the design to pop!”

“I want my site’s design to be groundbreaking like nothing else out there!”

“Let’s turn it up a notch on the design.”

“I want the site’s design to reflect the high value of our product.”

In and of themselves, none of the above statements are unworthy pursuits. But if your goal is to increase online sales conversion and fill your coffers to the brim, you will fall woefully short if you believe that web design alone can do the heavy lifting of convincing your visitors to take action. If increasing sales is your goal, the most important person on your split testing team is the accountant.

Designers Don’t Design for the Accountant

A while back, a client sent us a couple of different mocks of some new designs they were entertaining. They ask which one I liked. The first thing I said is I like the one that makes you the most money. Up until that time their team was arguing over color palettes, white space,and rounded edges.
When I reminded them about the bigger goal, their conversation evolved. In a clock tick, we were all discussing the quality of content on the pages rather than the design elements. When their offer and call to action were right, everyone seemed to forget about the trivia of the actual design.

Designing For Your Ego

Another client brought to us a new landing page campaign they had just launched and were baffled and disappointed by the early results. They went on to explain that they thought this was the best designed landing page they had ever done. They had just hired a new graphic designer that ‘got it’, and even the CEO was impressed with his work. One problem, their paying customers didn’t seem to agree. No doubt, the design was gorgeous. Rich colors, curvy rectangles, sexy images, even the header and body fonts were crisp and clean.
So why wasn’t this campaign working? We had them show us their most recent successful campaign. The design was a tad dated, and compared to the new landing page it looked like a high school hobbyist in the company basement eating Cheetos and suckling energy drinks.
Still, by comparing we immediately saw the problem with the new landing page. The copy on the old page was much better. The headers screamed the product’s value proposition and benefits. The body copy answered relevant questions, and helped the reader imagine themselves buying the product. The call to action button was big, bold, and in your face. The new page looked stunningly attractive but said very little.
To add insult, the hot shot designer was a minimalist and had an aversion to big gawky buttons, so his primary call to action was tiny button that blended in with the hero image, and , by design, was easy to ignore. We instructed them to use the old page copy on the new design (they had to make a few adjustments to make it all fit), and we asked the designer to create a bigger and bolder call to action button. They obliged us and that new design finally beat the old landing page.

How Much Time Are You Spending With Your Designer vs. Your Banker?

So my lesson is this. Beautiful, eye-popping design and effective, profitable web design are two different things. And it always seems easier to mistake those eye-popping designs for profitable ones. Split testing will always lead you in the right direction.
Some companies spend more on design than they do on organic SEO, and almost all companies spend more on design than on Conversion Rate Optimization. Search engine spiders don’t evaluate site design, only content and links. And I have yet to see a company design their way into a better conversion rate and better RO.
Some companies spend way more time going back and forth about a design element than they do actually testing it. Makes you wonder how far ahead of your competitors you could get if you spent more time and resources on conversion optimization and testing.
So when considering a redesign of your entire site, of a successful landing page, or even a banner ad, do the following:

  • List the things about the page experience (not just he design) work. Keep those in the new design.
  • What about the experience doesn’t work?
  • Why do we want to change this (especially if it is working)?
  • Before you launch a radically new design, test what you believe is NOT working about the current design.

Above all, use web designers that deeply understand the web and principles of conversion. Otherwise they are just an artist, and the value of an artists works usually increases only after their demise. Can you wait that long?

How helpful would it be to know what prices and features your competition was thinking about using?
One of my readers just sent me a very revealing screenshot. It is one of the pricing pages that Optimizely is testing. It was found by “spying” on their test data.

We hid the pricing on this test treatment from Optimizely

We hid the pricing on this test treatment from Optimizely

We are able to see this because of an “exploit” that allows anyone to see what a site is testing if they are using the Optimizely testing software. Oh, the irony.
Venture Beat recently “revealed” this in an article. Those of us who use these tools have known about it for some time. It’s quite easy to decipher this test data.
Try dragging the following link to your browser bookmark bar.
Optimizely Spy
Now visit Optimizely and click on the bookmark to see what they are testing.

How is this possible?

Whenever we run a split test with Optimizely, the software uploads scripts and data into all of our visitors’ browsers to change the experience and track the results. Along with this is included not just the test our visitor is being entered into, but all of our tests for that account.
So it’s relatively easy to decipher this information and see what we’re testing.
Note that the snooper can’t see any actual results, just what kinds of things you’re testing.
We like this approach because it speeds up the delivery of tests. When we use one file with everything, it changes less frequently, and the file it can be cached on a content delivery network (CDN) specifically designed to deliver files faster.
Faster tests mean more reliable tests.
Convert.com also uses this technique, though they take steps to obsure the test information.

Why Aren’t We More Concerned?

In a worst case scenario, a competitor can see what hypotheses you are testing. They can then test those same ideas and perhaps win more customers.
However, only a small percentage of sites are even testing, let alone stealing your tests. I did a quick survey of sites selling plastic surgery and cosmetic surgery who are spending at least $500 per month on search advertising.
Of 2,958 domains, only 33 had some form of split testing software installed, such as Optimizely. That’s just 1.1% of these domains. Furthermore, we know that some portion of these testing are not actually using the software they have installed.

Plastic and Costmetic Surgery Websites with Testing Software

Plastic and Cosmetic Surgery websites are missing a significant opportunity to get more patients. Source: SpyFu.com

Here’s another surprise. There are ninety-seven (97) domains in this space spending over $50,000 per month on search ads. Only five of them have A/B Testing software installed, only 5%.
If you’re in the plastic surgery space and are testing, you have a major advantage over your competitors. So, the odds of someone stealing your ideas are far outweighed by the gains you will see from testing.

Our Recommendation

We recommend that you continue to test using Optimizely unless your page contains sensitive information, such as price.
If you feel uncomfortable with your test information being publicly available, move to Convert Experiments for some protection. Another popular tool, Visual Website Optimizer, does not use this technique meaning past and future tests are safe from prying eyes. There are also a variety of other highly recommended AB testing tools available.
Whatever you do, don’t let this issue take the steam out of your testing program. As you can see, testers have a significant advantage, snoopers or not.
PS: If you are in the plastic and cosmetic surgery industry, you should contact us.

We find out which half will work.

How can it be that a team with our experience, intelligence and good looks could be wrong so often? It’s a mystery to us.
The truth is, that the audience for any website is unique. Given two sites selling the exact same product, you cannot assume that what works on one will work on the other.
People are complex. Groups of people are only slightly less complex.
This is why our optimization process is so powerful. We let your visitors decide what works best, and they vote with their money or their contact information.
We call it the Conversion Catalyst.

Why We Win

We live by a few mantras at Conversion Sciences to guide our decisions.
Our opinion doesn't matter
No matter how smart we think we are, the only thing that matters is what gives your visitors a better experience on your site. Don’t get caught up in the words “better experience.”
The best judge of our success is the accountant.
Visitors vote with their dollars. A good experience is one in which more visitors find what they need, want, or think they want. The person we are trying to impress really is the person that counts the income.
It’s not that we’re humble, but testing has a way of humbling you.
You don't have to be humble to optimize, but you can't optimize without being humbled.

We’d like to humbly offer to optimize your website.

Let us make your accountant smile.
You can get a free strategy session with a Conversion Scientist. We’ll help you see the possibilities for your visitors.

Dennis van der Heijden is in an enviable position. He is able to see the results of hundreds of split tests through his awesome split testing service, Convert Insights at Convert.com.
He’s noticed a few things about how successful businesses are at finding winning tests.
These numbers plus his ideas on why some have tests that frequently yield conversion rate lifts while others don’t is the subject of my Instagraph. This was recorded live at Conversion Conference East 2012 in Ft. Lauderdale, Florida on October 10.
Here is a time-lapse video of the creation of the Instagraph.

Here is the final result.