tools

Heatmaps are just the first step to obtaining useful insights on your website visitors. Today we’ll find out how heatmaps helped increase prospective student inquiries by 20% for a University and have a chat with Andrew Michael of Hotjar. Find out what he has to say.

Andrew Michael | Understanding Your Users: Leveraging Tools to Grow Your Website

Subscribe to the Podcast

iTunes | Spotify | Stitcher | Google Podcasts | RSS

Resources and links discussed

How Heatmaps Helped Increase Prospective Student Inquiries by 20%

We were looking at the heatmap report for the website of Northcentral University, a non-profit online university headquartered in Arizona.

Reading a heatmap report is like looking at a weather radar, but instead of blobs of green, red and yellow showing us where rain is falling around us, a heatmap report shows us where visitors are clicking on a web page.

And it was raining clicks in an unexpected spot on the NCU website.

Specifically, visitors were clicking on one of the fields in the middle of a form, and only on that field. Not the name field, not the email field. The majority of them weren’t completing the form.

So, why were visitors so interested in this one field?

It was an important question, as this form was the primary invitation to get more information on the University. It was on almost every page, ready to start a more in-depth conversation with any visitor.

The field visitors were clicking on was “program of interest”, a dropdown field that listed the degrees offered by NCU. It was meant as a way for prospective students to tell NCU which degree program they were interested in.

These prospective students were using it as an information source.

While the copy on the page was regaling visitors on the value of NCUs one-on-one learning, it’s 100% doctoral professors and it’s diversity, visitors were telling us that they had one question first.

Do you offer a degree program I’m interested in?

At least, this was the hypothesis. So we designed a test.

At the top of every page, we placed a dropdown menu that listed the university’s programs, just like that on the form. When a degree program was selected, we took them to the part of the site that described that degree program.

Half of NCUs visitors would see this dropdown. The other half would not. They’d have to use the dropdown in the form.

When we measured the results, the visitors who saw the dropdown in the page were 20% more likely to fill out the form completely, requesting information.

This indicated that the change would increase prospective student inquiries by 20%, a very significant improvement in the key metric for the site.

The current site offers a complete section designed to help visitors find a degree program they’re interested in.

This is something that we would not have been able to find any other way than through a heatmap report. It doesn’t show up in analytics. No one would have complained.

This is the power of a class of report called user intelligence reports.

Anyone who knows how to read rain chances from a weather radar can use this kind of report. More and more of us are doing this.

These reports are surprisingly easy to generate and the tools are inexpensive.

You can bring people to websites all day long but if it’s not optimized and it’s not user friendly and you’re going to lose all day and you just can end up throwing money down the drain.

Leading the way is a company called Hotjar. On today’s show we’re breaking down HotJar with Andrew Michael. A tool focused on helping you understand your users. Andrew got into marketing because he’s intrigued by psychology – understanding what drives people’s decisions.

An Insightful Chat with Andrew Michael from Hotjar

Time is precious for overburdened marketers. On this show, we seek to understand which tools are truly valuable, and which are just giving us “interesting” insights.

We install something like Hotjar on every one of our client sites when optimizing.

Tools like Hotjar are a part of what I call ‘the golden age of marketing’. These tools are continually evolving, getting easier to use and less expensive.

These are the tools that buy you more time to be creative, ground breaking and successful.

We start off the podcast talking about all of the things Hotjar brings to the table under a single subscription. Then we talk about the outcome of leveraging tools like this – how do they actually empower marketers serve their online prospects better?

Listen to the Podcast. It’s well worth it.

When You Get Back To The Office

I’m not a shill for Andrew. I just know these tools are a great value and easy to learn.

When you get back to the office, i recommend that you do a trial of Hotjar. Add it to your homepage, or one of your “money” pages where you ask visitors to take action. Setup a heatmap report on it.

Let it run for a few days, and then look at the scroll report. This report tells you how far visitors are scrolling on your page. This is one of the first things we look at when we start analyzing our clients’ sites.

Where is the report turning blue? This is the place on the page that visitors stop reading. Look in the blue area. What key content are they missing?

If more than half of your page is blue, you have a scroll problem. Visitors aren’t being engaged enough to get through your content.

Reasons for this include: false bottoms, where visitors think the page ends when it doesn’t. It can mean that your content isn’t engaging them enough high on the page. It can mean that you’re not handling a key objection.

Your strategies include moving key content to the top of the page, putting arrows, chevrons and “v”s on the page to tell visitors to keep going, or re-thinking the story you tell on this page.

Don’t be discouraged. This is progress! Next, share this report with your design team and see what they think.

This is how pages get better and businesses grow.

You can get all these links discussed on this week’s episode in our shownotes. One thing to remind you all of is that Hotjar is a freemium model so it’s one you can definitely

Alright scientists, that’s it for this week.

Andrew Michael | Understanding Your Users: Leveraging Tools to Grow Your Website

Subscribe to the Podcast

iTunes | Spotify | Stitcher | Google Podcasts | RSS

Don’t miss the first episode of the first Podcast season, where we chat with Mouseflow, a user-behavior analytics tool and cover recordings, heatmaps, funnels. Plus, how to manage helicopter executives.

Collecting Qualitative Data on Your Visitors

Subscribe to the Podcast

iTunes | Spotify | Stitcher | Google Podcasts | RSS

Resources and Links Discussed

Intended Consequences Podcast Season 1 Episode 1: Key Takeaways

  1. Exit Intent: Not sure what this means? You’ll learn about that and why it matters.
  2. What Happens When a Site Bug Goes Unchecked: You’ll hear stories on the impact a site bug can have on your website – and we’re talking a $1.2 million impact.
  3. Tips on Conversation with Executives: Gain knowledge and tips from Evan on how to have conversations with your marketing executives.

Excerpts from our Conversation with Mouseflow

Avoid the Bias

This stuff really fascinates me just because it’s a psychological. It’s diving into the minds of of your visitors. And one thing that I always encourage people to do when they’re using Mouseflow on their website is PLEASE FOR THE LOVE OF GOD DO NOT go in with any bias. You have to be willing to test and identify ways that you can improve your form or specific parts of your Web site.

The Add-to-Cart Bug

This is going to have to remain anonymous, so I can’t share the company. But there is e-commerce store in America that uses Mouseflow and they were recording 100 percent of all visitors. So lots of data coming in. We’re talking millions and millions of sessions per month. And there was a pretty serious bug / error that was deployed live onto the Web site after they had finished a redesign. And Mouseflow picked it up. They had notifications set to send to one of their product marketers e-mails whenever a JavaScript error occurred.

All of the sudden at 2:00a.m. on Monday night their e-mail just starts getting absolutely blown up. It turned out that there was an Add to Cart button that was not working on about 40% of their product pages.

It was a huge huge error.

Mouseflow estimated it ended up being like $400K revenue loss. So, it ended up being a serious deal . And if that had gone further unnoticed, obviously this would have stretched into the millions of dollars.

Get ready, Marketers

“So I would I think that’s one of the most exciting things for a marketer who finally grabs this tool installs it, because they’re about to get the data they need to have a really really interesting meetings.”

Conversion Sciences Podcast with Mouseflow, a user-behavior analytics tool.

Conversion Sciences Podcast with Mouseflow, a user-behavior analytics tool.

Helicopter Executives

That executive who doesn’t feel comfortable with the work that a marketer has done, because that marketer doesn’t have any data, will come in and change things based on their experience with a customer their experience or their own preference.

In other words executives are coming in with all their biases and making changes to a campaign, and that’s really frustrating to a marketer marketing team who’s worked hard on a redesign, to have a sample size of one person come nin and upend those assumptions.

Celebrating Design

“But there’s something to be said for installing this tool before you launch a redesign, and then going in and celebrating, with heatmap and session recordings, where the redesign has really improved things. That’s going to get the design team and the UX team more interested in working with you.”

It’s going to make your boss look good because he or she shepherded this fantastic redesign. And then you can go and say here’s the next things we can be improving on.”

Multivariate testing offers high-traffic websites the ability to find the right combination of features and creative ideas to maximize conversion rates. However, it is not sufficient to simply throw a bunch of ideas into a pot and start testing. This article answers the question, What is a multivariate test?, explains the advantages and pitfalls of multivariate testing, and offers some new ideas for the future.
If you run a relatively high-traffic site, consider this question: Will I profit from running multivariate tests?
Before we dive into the question, let’s be sure to define the terms. I’ll talk about the dangers of doing an multivariate test (MVT) and when you should consider using them.

What Is Multivariate Testing?

Multivariate testing is a technique for testing a hypothesis in which multiple variables are modified.
Multivariate testing is distinct from A/B testing in that it involves the simultaneous observation and analysis of more than one outcome variable. Instead of measuring A against B, you are measuring A, B, C, D & E all at once.
Whereas A/B testing is typically used to measure the effect of more substantial changes, multivariate testing is often used to measure the incremental effect of numerous changes at once.
This process can be further subdivided in a number of ways, which we’ll discuss in the next section.

Multivariate, Multi-variant or Multi-variable

For this article, we are focusing on a specific way of testing in which elements are changed on a webpage. Before we dive into our discussion of multivariate testing, we should identify what we are talking about and what we are not talking about.
One of the frequent items tested is landing page headlines. Getting the headline on the page can significantly increase conversion rates for your landing pages. When testing a headline, we often come up with several variants of the words, testing them individually to see which generates the best results.

A multi-variant tests multiple variants of one variable or element.

A multi-variant test with multiple variants of one variable or element.


This is a multi-variant test. It changes one thing–one variable–but provides a number of different variants of that element.
Now suppose we thought we could improve one of our landing pages by changing the “hero image” as well as the headline. We would test our original version against a new page that changed both the image and the headline.
An example of a multi-variable test. Here we are testing the control against a variation with two changes, or two variables.

An example of a multi-variable test. Here we are testing the control against a variation with two changes, or two variables.


This is a multi-variable test. The image is one variable and the headline is a second variable. Technically, this is an AB test with two variables changing. If Variation (B) generated more leads, we wouldn’t know if the image or the headline were the biggest contributors to the increase in conversions.
To thoroughly test all combinations, we would want to produce multiple variations, each with a different variant of the variable.
Two variable with two variants each yield four page variations in this multivariate testing example.

Two variables with two variants each yield four page variations in this multivariate testing example.


In the image above, we have four variations of the page, based on two variables (image and headline) each having two variants. Two variables times two variants each equals four variations.
Confused yet?
A multivariate test, then, is a test that tests multiple variants of variables found on a page or website.
To expand on our example, we might want to find the right hero image and headline on our landing page. Here’s the control:
The Control in our multivariate test is the current page design.

The Control in our multivariate test is the current page design.


We will propose two additional variants of the hero image–for a total of three variants including the control–and additional two variants of the headline, three including the control.
Here are the three images:
We want to vary the hero image. This variable in our test has three variants.

We want to vary the hero image on our page. This variable in our test has three variants.


Here are three headlines, including the existing one.

  1. Debt Relief that Works
  2. Free Yourself from the Burden of Debt
  3. Get Relief from Debt

A true multivariate test will test all combinations. Given two variables with three variants each, we would expect nine possible combinations: three images x three headlines.
Here’s another example that will help you understand how variables, variants and variations relate. An ecommerce company believes that visitors are not completing their checkout process for any of three reasons:

  1. The return policy is not visible
  2. They are required to register for an account
  3. They don’t have security trust symbols on the pages

While these all seem like reasonable things to place in a shopping cart, sometimes they can work against you. Providing this kind of information may make a page cluttered and increase abandonment of the checkout process.
The only way to know is to test.
How many variables do we have here? We have three: privacy policy, registration and security symbols.
How many variants do we have? We have two of each variable, one variant in which the item is shown and one variant in which it is not shown.
This is 2 x 2 x 2, or eight combinations. If we had three different security trust symbols to choose from, we would have four variants, three choices and none. That is 2 x 2 x 4, or sixteen combinations.
We’ll continue to use this example as we explore multivariate testing.

Why Multivariate Testing Isn’t Valuable In Most Scenarios

A multivariate test seeks to test every possible combination of variants for a website given one or more variables. 
If we ran an MVT for our ecommerce checkout example above, it would look something like this:

Variations multiply with multivariate tests requiring more traffic and conversions.

Variations multiply with multivariate tests requiring more traffic and conversions.


There are many reasons that multivariate testing is often the wrong choice for a given business, but today, I’m going to focus on five. These are the five reasons multivariate tests (MVTs) are not worth doing compared to A/B/n tests:

  1. A lack of time or traffic
  2. Crazy (and crappy) combinations
  3. Burning up precious resources
  4. Missing out on the learning process
  5. Failing to use MVT as a part of a system

Let’s take a closer look at each reason.

1. Multivariate Tests Take a Long Time or a Whole Lot of Traffic

Traffic to each variation is a small percentage of the overall traffic. This means that it takes longer to run an MVT. Lower traffic means it takes longer to reach statistical significance, and we can’t believe the data until we reach this magical place.
Statistical Significance is the point at which we are confident that the results reported in a test will be seen in the future, that winning variations will deliver more conversions and losing variations would deliver fewer conversions over time. Read 2 Questions That Will Make You A Statistically Significant Marketer or hear the audio.
Furthermore, statistical significance is really measured by the number of successful transactions you process.
For example, MXToolbox offers free tools for IT people who are managing email servers, DNS servers and more. They also offer paid plans with more advanced features. MXToolbox gets millions of visitors every month, and many of them purchase the paid plans. Even with millions of visits, they don’t have enough transactions to justify multivariate testing.
It’s not just about traffic.
This is why MVTs can be done only on sites with a great deal of traffic and transactions. If not the tests take a long time run.

2. Variations Multiply Like Rabbits

As we saw, just three variables with two variants resulted in eight variations, and adding two more security trust symbols to the mix brought this to sixteen combinations. Traffic to each variation would be reduced to just 6.25%.
Multivariate testing tools, like VWO and Optimizely offer options to test a sample of combinations — called Partial, or Fractional Factorial testing — instead of testing them all, which is called Full Factorial testing. We won’t dive into the mathematics of Full Factorial and Partial Factorial tests. It gets a little messy. It’s sufficient to know that partial factorial (fractional factorial) testing may introduce inaccuracies that foil your tests.
What’s important is that more variations mean larger errors… because statistics.
Every time you add another variation to an AB test, you increase the margin of error for the test slightly. As a rule, Conversion Sciences allows no more than six variations for any AB test because the margin of error becomes a problem.
In an AB test with two variations, we may be able to reach statistical significance in two weeks, and bank a 10% increase in conversions. However, in a test with six variations, we may have to run for four weeks before we can believe that the 10% lift is real. The margin of error is larger with six variations requiring more time to reach statistical significance.
Now think about a multivariate test with dozens of variations. Larger and larger margins of error mean the need for even more traffic and some special calculations to ensure we can believe our results aren’t just random.
Ultimately, most of these variations aren’t worth testing.
All eight variations in our example make sense together. As you add variations, however you can end up with some crazy combinations.
Picture this:
It’s pouring down rain. You are camping with your son.
While huddled in your tent, you fire up your phone’s browser to find a place to stay. While flipping through your search results on Google, your son proclaims over your shoulder, “That one has a buffet! Let’s go there, Dad!”
Ugh.
The last time he ate at an all-you-can-eat buffet, he was stuck in the restroom for an hour. Not a pretty picture.
Then again, neither is staying out in the wretched weather. So you click to check out the site.
Something is off.
The website’s headline says, “All you can eat buffet.” But nothing else seems to match. The main picture is two smiling people at the front desk, ready to check you in.

As you scroll to the bottom, the button reads “Book Your Massage Today”.
Is this some kind of joke?
As strange as this scenario sounds, one problem with MVTs is that you will get combinations like this example that simply don’t make sense.
This leaves you with two possibilities:

  1. Start losing your customers to variations you should not even test (not recommended).
  2. Spend some of your time making sure each variation makes sense together.

The second option will take more time and restrict your creativity. But even worse, now you need more traffic in order for your test.
With an A/B/n test, you pick and choose which tests you like and which to exclude.
Some may argue it can be time-consuming to create each A/B/n variation while a multivariate test is an easy way to test all variations at once.
Think of a multivariate test as a system that automatically creates all possible combinations to help you find the best outcome. So on the surface, it sounds appealing.
But as you dig into what’s really going on, you may think twice before using an MVT.

3. Losing Variations are Expensive

Optimization testing can be fun. The chance of a breakthrough discovery that could make you thousands of dollars is quite appealing. Unfortunately, those variations that underperform the Control reduce the number of completed transactions and fewer transactions means less revenue.
Every test — AB or Multivariate — has a built in cost.
Ideally, we would let losing variations run their course. Statistically, there is a chance they will turn around and be a big winner when we reach statistical significance. At Conversion Sciences, we monitor tests to see if any variations turn south. If a losing variation is costing us too many conversions, we’ll stop it before it reaches statistical significance. This is how we control the cost of testing.
This has two advantages.

  1. We can control the “cost” of an AB test.
  2. We can direct more traffic to the other variations, meaning the test will take less time to reach significance.

When tests run faster, we can test more frequently.
On the other hand, multivariate tests run through all variations, or a large sample of variations. Losers run to statistical significance and this can be very expensive.
Lars Lofgren, former Director of Growth at KISSmetrics, mentioned that if a test drops below a 10% lift, you should kill it. Here’s why:
What would you rather have?

  • A confirmed 5% winner that took 6 months to reach
  • A 20% winner after cycling through 6-12 tests in that same 6 month period

Forget that 5% win, give me the 20%!
So the longer we let a test run, the higher that our opportunity costs start to stack up. If we wait too long, we’re forgoing serious wins that we could of found by launching other tests.
If a test drops below a 10% lift, it’s now too small to matter. Kill it. Shut it down and move on to your next test.
Keeping track of all the MVT variations isn’t easy to do (and also is time consuming). But time spent on sub-par tests are not the only resource you lose either.

4. It’s Harder to Learn from Multivariate Tests

Optimization works best when you learn why your customers behave the way that they do. Perhaps with an MVT you may find the best performing combination, but what have you learned?
When you run your tests all at one time, you miss out on understanding your audience.
Let’s take the example from the beginning of this article. Suppose our multivariate test reported that this was the winning combination:

If this combination wins, can we tell why?

If this combination wins, can we know why?


What can we deduce from this? Which element was most important to our visitors? The return policy? Removing the registration? Adding trust symbols?
And why does it matter?
For starters, it makes it easier to come up with good test hypotheses later on. If we knew that adding trust symbols was the biggest influence, we might decide to add even more trust symbols to the page. Unfortunately, we don’t know.
When you learn something from an experiment, you can apply that concept to other elements of your website. If we knew that the return policy was a major factor, we might try adding the return policy on all pages. We might even test adding the return to our promotional emails.
Testing is not just about finding more revenue. It is about understanding your visitors. This is a problem for multivariate tests.

5. Seeing What Sticks Is Not An Effective Testing System

Multivariate tests are seductive. They can tempt you into testing lots of things, just because you can. This isn’t really testing. It’s fishing. Throwing a bunch of ideas into a multivariate test means you’re testing a lot of unnecessary hypotheses.
Testing follows the Scientific Method:

  1. Research the problem.
  2. Develop hypotheses.
  3. Select the most likely hypotheses.
  4. Design experiments to test your hypotheses.
  5. Run the experiment in a controlled environment.
  6. Evaluate your results.
  7. Develop new hypotheses based on your learnings.

The danger of a multivariate test is that you skip steps 3, 4 and 7, that you:

  1. Research the problem
  2. Develop hypotheses.
  3. Throw them into the MVT blender
  4. See what happens.

Andrew Anderson said it well,

The question is never what can you do, but what SHOULD you do.
Just because I can test a massive amount of permutations does not mean that I am being efficient or getting the return on my efforts that I should. We can’t just ignore the context of the output to make you feel better about your results.

You will get a result no matter what you do, the trick is constantly getting better results for fewer resources.

When used with the scientific method, an A/B/n test can give you the direction you need to continually optimize your website.

Machine Learning and Multivariate Testing

Multivariate testing is now getting a hand from artificial intelligence. For decades, a kind of program called a neural network has allowed computers to learn as they collect data, making decisions that are more accurate than humans using less data. These neural networks have only been practical in solving very specific kinds of problems.
Now, software company Sentient Ascend has brought a kind of neural network into the world of multivariate testing. It’s called an evolutionary neural network or a genetic neural network. This approach uses machine learning to sort through possible variations, selecting what to test so that we don’t have to test all combinations.
These evolutionary algorithms follow branches of patterns through the fabric of possible variations, learning which are most likely to lead to the highest converting combination. Poor performing branches are pruned in favor of more likely winners. Over time, the highest performer emerges and can be captured as the new control.
These algorithms also introduce mutations. Variants that were pruned away earlier are reintroduced into the combinations to see if they might be successful in better-performing combinations.
This organic approach promises results faster and with less traffic.

Neural networks allow testing tools to learn what combinations will work without testing all multivariate combinations.

Evolutionary neural networks allow testing tools to learn what combinations will work without testing all multivariate combinations.


With machine learning, websites that had too little traffic for pure multivariate testing can seriously consider it as an option.

Final Thoughts: Is There Ever a Case For doing MVTs?

There are instances when introducing many variables is sometimes difficult to avoid or better to focus on.
Chris Goward of WiderFunnel gives four advantages to doing MVTs over A/B/n tests:

  1. Easily isolate many small page elements and measure their individual effects on conversion rate
  2. Measure interaction effects between independent elements to find compound effects
  3. Follow a more conservative path of incremental conversion rate improvement
  4. Facilitate interesting statistical analysis of interaction effects

He later admits, “At WiderFunnel, we run one Multivariate Test for every 8-10 A/B/n Test Rounds.”
Both methods are valuable learning tools.

What is Your Experience?

It is a bit of heated subject between optimization experts. I’d be curious to hear from you about your ideas and experience on what matters the most.
Please leave a comment.

Nothing gives you confidence and swagger like AB testing. And nothing will end your swagger faster than bad data. In order to do testing right, there are some things you need to know about AB testing statistics. Otherwise, you’ll spend a lot of time trying to get answers, but instead of getting answers, you’ll end up either confusing yourself more or thinking you have an answer, when really you have nothing. An A/A test ensures that the data you’re getting can be used to make decisions with confidence.

What’s worse than working with no data? Working with bad data.

We’re going to introduce you to a test that, if successful will teach you nothing about your visitors. Instead, it will give you something that is more valuable than raw data. It will give you confidence.

What is an A/A Test

The first thing you should test before your headlines, your subheads, your colors, your call to actions, your video scripts, your designs, etc. is your testing software itself. This is done very easily by testing one page against itself. One would think this is pointless because surely, the same page against the same page is going to have the same results, right?

Not necessarily.

After three days of testing, this A/A test showed that the variation identical to the Original was delivering 35.7% less revenue. This is a swagger killer.

This A/A Test didn't instill confidence after three days.

This A/A Test didn’t instill confidence after three days.

This can be cause by any of these issues:

  1. The AB testing tool you’re using is broken.
  2. The data being reported by your website is wrong or duplicated.
  3. The AA test needs to run longer.

Our first clue to the puzzle is the small size of the sample. While there were over 345 or more visits to each page, there were only 22 and 34 transactions. This is too small by a large factor. In AB testing statistics, transactions are more important than traffic in building statistical confidence. Having fewer than 200 transactions per treatment often delivers meaningless results.

Clearly, this test needs to run longer.

Your first instinct may be to hurry through the A/A testing so you can get to the fun stuff – the AB testing. But that’s going to be a mistake, and the above shows why.

An A/A test serves to calibrate your tools

An A/A test serves to calibrate your tools

Had the difference between these two identical pages continued over time, we would call off any plans for AB testing altogether until we figured out if the tool implementation or website were the source of the problem. We would also have to retest anything done prior to discovering this AA test anomaly.

In this case, running the A/A test for a longer stretch of time increased our sample size and the results evened out, as they should in an A/A test. A difference of 3.5% is acceptable for an AA test. We also learned that a minimum sample size approaching 200 transactions per treatment was necessary before we could start evaluating results.

This is a great lesson in how statistical significance and sample size can build or devastate our confidence.

An A/A Test Tells You Your Minimum Sample Size

The reason the A/A test panned out evenly in the end was it took that much time for a good amount of traffic to finally come through the website and see both “variations” in the test. And it’s not just about a lot of traffic, but a good sample size.

  • Your shoppers on a Monday morning are statistically completely different people from your shoppers on a Saturday night.
  • Your shoppers during a holiday seasons are statistically different from your shoppers on during a non-holiday season.
  • Your desktop shoppers are statistically different from your mobile shoppers.
  • Your shoppers at work are different from your shoppers at home.
  • Your shoppers from paid ads are different from your shoppers from word of mouth referrals.

It’s amazing the differences you may find if you dig into your results, down to specifics like devices and browsers. Of course, if you only have a small sample size, you may not be able to trust the results.

This is because a small overall sample size means that you may have segments of your data allocated unevenly. Here is an sample of data from the same A/A test. At this point, less than 300 sessions per variation have been tested. You can see that, for visitors using the Safari browser–Mac visitors–there is an uneven allocation, 85 visitors for the variation and 65 control. Remember that both are identical. Furthermore, there is an even bigger divide between Internet Explorer visitors, 27 to 16.

This unevenness is just the law of averages. It is not unreasonable to imagine this kind of unevenness. But, we expect it to go away with larger sample sizes.

You might have different conversion rates with different browsers.

You might have different conversion rates with different browsers.

Statistically, an uneven allocation leads to different results, even when all variations are equal. If the allocation of visits is so off, imagine that the allocation of visitors that are ready to convert is also allocated unevenly. This would lead to a variation in conversion rate.

And we see that in the figure above. For visitors coming with the Internet Explorer browser, none of sixteen visitors converted. Yet two converting visitors were sent to the calibration variation for a conversion rate of 7.41%.

In the case of Safari, the same number of converting visitors were allocated to the Control and the calibration variation, but only 65 visits overall were sent to the Control. Compared this to the 85 visitors sent to the Calibration Variation. It appears that the Control has a much higher conversion rate.

But it can’t because both pages are identical.

Over time, we expect most of these inconsistencies to even out. Until then they often add up to uneven results.

These forces are at work when you’re testing different pages in a AB test. Do you see why your testing tool can tell you to keep the wrong version if your sample size is too small?

Calculating Test Duration

You have to test until you’ve received a large enough sample size from different segments of your audience to determine if one variation of your web page performs better on the audience type you want to learn about. The A/A test can demonstrate the time it takes to reach statistical significance.

The duration of an AB test is a function of two factors.

  1. The time it takes to reach an acceptable sample size.
  2. The difference between the performance of the variations.

If a variation is beating the control by 50%, the test doesn’t have to run as long. The large margin of “victory”, also called “chance to beat” or “confidence”, is larger than the margin of error, even at small er sample sizes.

So, an A/A test should demonstrate a worst case scenario, in which a variation has little chance to beat the control because it is identical. In fact, the A/A test may never reach statistical significance.

In our example above, the test has not reached statistical significance, and there is very little chance that it ever will. However, we see the Calibration Variation and Control draw together after fifteen days.

These identical pages took fifteen days to come together in this A/A Test.

These identical pages took fifteen days to come together in this A/A Test.

This tells us that we should run our tests a minimum of 15 days to ensure we have a good sample set. Regardless of the chance to beat margin, a test should never run for less than a week, and two weeks is preferable.

Setting up an A/A Test

The good thing about an A/A test is that there is no creative or development work to be done. When setting up an AB test, you program the AB testing software to change, hide or remove some part of the page. This is not necessary for an A/A test, by definition.

For an A/A test, the challenge is to choose the right page on which to run the test. Your A/A test page should have two characteristics:

  1. Relatively high traffic. The more traffic you get to a page, the faster you’ll see alignment between the variations.
  2. Visitors can buy or signup from the page. We want to calibrate our AB testing tool all the way through to the end goal.

For these reasons, we often setup A/A tests on the home page of a website.

You will also want to integrate your AB testing tool with your analytics package. It is possible for your AB testing tool to be setup wrong, yet both variations behave similarly. By pumping A/A test data into your analytics package, you can compare conversions and revenue reported by the testing tool to that reported by analytics. They should correlate.

Can I Run an A/A Test at the Same Time as an AB Test?

Statistically, you can run an A/A test on a site which is running an AB test. If the tool is working well, than your visitors wouldn’t be significantly affected by the A/A test. You will be introducing additional error to your AB test, and should expect it to take longer to reach statistical significance.

And if the A/A test does not “even out” over time, you’ll have to throw out your AB test results.

You may also have to run your AB test past statistical significance while you wait for the A/A test to run its course. You don’t want to change anything at all during the A/A test.

The Cost of Running an A/A Test

There is a cost of running an A/A test: Opportunity cost. The time and traffic you put toward an A/A test could be used to for an AB test variation. You could be learning something valuable about your visitors.

The only times you should consider running an A/A test is:

  1. You’ve just installed a new testing tool or changed the setup of your testing tool.
  2. You find a difference between the data reported by your testing tool and that reported by analytics.

Running an A/A test should be a relatively rare occurrence.

There are two kinds of A/A test:

  1. A “Pure” two variation test
  2. An AB test with a “Calibration Variation”

Here are some of the advantages and disadvantages of these kinds of A/A tests.

The Pure Two-Variation A/A Test

With this approach, you select a high-traffic page and setup a test in your AB testing tool. It will have the Control variation and a second variation with no changes.

Advantages: This test will complete in the shortest timeframe because all traffic is dedicated to the test

Disadvantages: Nothing is learned about your visitors–well, almost. See below.

The Calibration Variation A/A Test

This approach involves adding what we call a “Calibration Variation” to the design of a AB test. This test will have a Control variation, one or more “B” variations that are being tested, and another variation with no changes from the Control. When the test is complete you will have learned something from the “B” variations and will also have “calibrated” the tool with an A/A test variation.

Advantages: You can do an A/A test without stopping your AB testing program.

Disadvantages: This approach is statistically tricky. The more variations you add to a test, the larger the margin of error you would expect. It will also drain traffic from the AB test variations, requiring the test to run longer to statistical significance.

AA Calibration Variation in an AB Test

AA Test Calibration Variation in an AB Test (Optimizely)

Unfortunately, in the test above, our AB test variation, “Under ‘Package’ CTAs”, isn’t outperforming the A/A test Calibration Variation.

You Can Learn Something More From an A/A Test

One of the more powerful capabilities of AB testing tools is the ability to track a variety of visitor actions across the website. The major AB testing tools can track a number of actions that can tell you something about your visitors.

  1. Which steps of your registration or purchase process caused them to abandon your site
  2. How many visitors started to fill out a form
  3. Which images visitors clicked on
  4. Which navigation items were most frequently clicked

Go ahead and setup some of these minor actions–usually called ‘custom goals’– and then examine the behavior when the test has run its course.

In Conclusion

Hopefully, if nothing else, you were amused a little throughout this article while learning a bit more about how to ensure a successful AB test. Yes, it requires patience, which I will be the first to admit I don’t have very much of. But it doesn’t mean you have to wait a year before you switch over to your winning variation.

You can always take your winner a month or two in and use it for PPC and continue testing and tweaking on your organic traffic. That way you get the both worlds – the assurance that you’re using your best possible option on your paid traffic and taking the time to do more tests on your free traffic.

And that, my friends, is AB testing success in a nutshell. Now go find some stuff to test and tools to test with!

About the Author


21 Quick and Easy CRO Copywriting Hacks to Skyrocket Conversions

FREE: Click to Download

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

  • 43 Pages with Examples
  • Assumptive Phrasing
  • "We" vs. "You"
  • Pattern Interrupts
  • The power of Three

We are often guilty of writing about the AB testing process as if it was something you can jump into and start doing. We believe an AB testing program can keep you from making expensive design mistakes and find hidden revenue that your competitors are currently getting. It’s not an overnight switch, however. It takes some planning and resources.
It’s a journey not unlike that taken by many heroes throughout history and mythology. We invite you to join the ranks of heroic journeymen.
The journey looks something like this: You befriend a helpful stranger who gives you something magical. Soon, you are called to adventure by events beyond your control. When you act, you enter a strange new world and must understand it. You are set on a journey to right some wrong. Allies and helpers work with you to move past trials and tests. Your magical talisman helps you understand what to do. With patience you gather your reward and return home, the master of two worlds.
Consider this blog post a harbinger of the adventure that awaits you. Here are the things you’ll encounter on your journey.

Executive Champion: Magical Helper

Every hero story involved some kind of “supernatural help”. In the story of the Minotaur, Ariadne gave Theseus the golden string to find his way out of the Minotaur labyrinth. In Star Wars, Obi-Wan Kenobi gave Luke Skywalker a light saber showing him how to dodge blaster shots and face Darth Vader.
Each barrier in your path to an amazing AB testing program will have obstacles, and each obstacle will require a little magical help. This is the role of your executive champion. Your executive can impart to you special gifts, such as the Magical Elixir of Budget, the Red Tape Cleaver: Blessed Blade of Freedom, as well as the One Ring of Power to rule them all…but let’s not get carried away.
In my experience – and I’d like to hear yours – AB testing is not something that can be done “under the radar” until you have some successes. Use this post to guide you, prepare a presentation, and convince someone with pull to support your efforts.

Traffic: The Call to Adventure

It is when we finally have a steady, reliable stream of visitors that we are called to our hero’s journey. Traffic is like the taxes imposed by the Sherriff of Nottingham. The hero just can’t stand by and watch injustice.
Likewise, you must feel uncomfortable about having thousands of visitors coming to your site – most of them paid for – and then seeing 99% of them turn and leave. This is injustice at it’s most heartbreaking and a clear conversion optimization problem. Many companies will just up the Adwords budget to grow revenue. Heroes fight for the common visitor.
AB testing is a statistical approach to gathering data and making decisions. There is a minimum number of transactions you will want each month in order for your AB tests to reach statistical significance. In general, you can test an idea a month with 300 monthly transactions.
To see if you have the traffic and conversions, use our simple Conversion Upside Calculator. It will tell you how quickly you would expect a positive ROI on your AB testing program.

Analytics: Understanding the Unknown

Upon accepting the call to adventure, the hero will find herself in a strange new world. Here the rules she is so familiar with will no longer apply. She will see things differently. In the tale of The Frog Prince, a spoiled princess agrees to befriend a talking frog. In exchange the frog will retrieve her favorite golden ball from a deep pool. Upon making the deal, her world changes.
You, too, have lost your golden ball.
Most websites have Google Analytics, Adobe Analytics, Clicky, Mixpanel, or some other analytics package in place. I recommend that you not look at this as a scary forest full of strange tables, crooked graphs and unfathomable options. Instead, look at this as a constantly running focus group. It’s a collection of answers to your most pressing questions.
You should get to know your web analytics tools, but don’t get wrapped around the axel thinking you need to have fancy dashboards and weekly updates. That’s the work of admins.
Instead, sit down with a specific question you want to answer, and figure out how to drill deep into it.
“Where are visitors entering our website?”
“How long are they staying?”
“Which pages seem to cause the most trouble?”
“Are buyers of Product A acting different from buyers of Product B?”
“How important is site search to our visitors?”
This gives you amazing cosmic powers to make decisions that otherwise would have been coin tosses.

Hypothesis List: The Yellow Brick Road

One of our favorite hero stories is that of Dorothy and her journey through the Kingdom of Oz. This is a favorite because it has all of the elements of the hero’s journey. Our hero’s journey needs a path to follow. Just as Dorothy was told to follow the yellow brick road to Oz, our hypotheses are the yellow bricks in our path to AB testing success.
As you become more familiar with analytics, you will have many ideas sliding out of your head. [pullquote]Ideas are like the slippery fish in an overflowing barrel.[/pullquote] You probably already have a lot of questions about how things are working on your site. You’ve probably collected dozens of ideas from well-meaning coworkers.
It can be overwhelming.
The magical helper for unbounded ideas is called the Hypothesis List. It is like Don Quixote’s Rocinante. It is your powerful steed on which you can rely to carry you through your journey to testing. By building out this Hypothesis List, you will eliminate ideas that aren’t testable, refine ideas that are, and rank them based on expected ROI.
[pullquote]If AB testing tells you which of your ideas are good ones, the Hypothesis List tells you which are most likely to be good ones.[/pullquote]

Ideas are not Hypotheses

A Hypothesis is an “educated guess”. To be a hypothesis, an idea must be somewhat educated: informed by data, supported by experience, or born from observation. Any idea that begins with “Wouldn’t it be cool if…” is probably not a hypothesis.
When you take an idea, and try to write it into the format of a hypothesis, you quickly realize the difference. Here’s the format of a hypothesis:

If we [make a change], we expect [a desirable result] as measured by [desired outcome].

The change is a modification to copy, layout, navigation, etc. that tests a hypothesis. it is insufficient to say “Get more clicks on Add to Cart”. You must state a specific change, such as, “Increase the size of the Add to Cart button”.
The result is a desired outcome. For most tests, the desired out come is a bottom-line benefit.

  • “increase transactions”
  • “decrease abandonment”
  • “increase phone calls”
  • “increase visits to form page”

[pullquote]Soft results such as “increase engagement” are popular, but rarely correlate to more leads, sales or subscribers.[/pullquote]
The outcome is usually the metric by which you will gauge the success of the test.

  • Revenue per Visitor
  • Lead Conversion Rate
  • Form Abandonment Rate

The ROI Prioritized Hyptohesis List Spreadsheet

Download your copy of the spreadsheet we use to prioritize winning test ideas.

  • List and categorize your ideas
  • Rate and rank to find “Low Hanging Fruit”
  • Place into buckets to identify key issues.

Many of your ideas will spawn several detailed hypotheses. Many ideas will simply die from lack of specificity.

Too Many Ideas

It is not unusual to have more ideas than you can manage. Nonetheless, it makes sense to capture them all. A simple Excel spreadsheet does the trick for collecting, sorting and ranking.

Too Few Ideas

It may be hard to believe, but you will run out of good hypotheses faster than you know. Plus, there are many hypotheses that will never be obvious to you and your company because as the old saying goes, “You can’t read the label from inside the bottle.”
This is where focus groups, online user testing sites, surveys, and feedback forms play an important role. Too many marketers use input from the qualitative sources as gospel truth. This is a mistake. You’re working toward an AB testing process that will let you test this input.

Ranking by Expected ROI

We recommend ranking your hypotheses so that the “low hanging fruit” bubbles up to the top. Our ROI Prioritized Hypothesis List ranks them based on four criteria, all ranked on a scale of one to five:

  1. Level of Effort: How difficult is this to test and implement?
  2. Traffic Affected: How much traffic will this hypothesis affect, and how important is that traffic?
  3. Proof: How much evidence did we see in analytics and other tools that this hypothesis really is a problem?
  4. Impact: Based on my experience and knowledge, how big of an impact do I really think this hypothesis can drive?

Once you’ve plugged a value in for these criteria for each hypothesis, add 2, 3 and 4 and subtract 1. That’s the weight of each hypothesis. The higher the weight, the lower the fruit hangs.

The Scarecrow: JavaScript Developer

Every hero has helpers, allies and maybe even a sidekick. You are no exception. Dorothy had the Scarecrow, the first ally she met on the banana-colored road to Oz. The Scarecrow had an amazing brain, but didn’t really know it.
At the end of your journey, you are going to have complete control of your website. You won’t need to put requests into IT. You will have the power to change your website for every visitor because the change happens in the visitor’s browser. Each visitor gets the same website from the server. It’s the JavaScript that instantly transforms what they see into a test treatment, something different and measureable.
Your JavaScript developer will be your Scarecrow. This person must be comfortable with JavaScript, HTML, CSS, the browser DOM and cross-browser issues. This person will enable you to make changes that put your hypotheses to the test.

The Tin Man: Designer

Dorothy was also befriended by the man who didn’t know he had a heart on the way to Oz.
You’ll want  a designer that isn’t interested in redesigning your pages. All you need is a designer who can change portions of a page. It may be a design change as simple as a new image, or as complex as a new page layout.
Avoid designers who like to add to their egos to your design.

The Lion: Copywriter

I made the copywriter the lion in this journey because writing and defending bold headlines and copy takes courage. Like Dorothy’s friend, the cowardly lion, most copywriters have been taught to write business speak to online visitors. They have PTSD from having their copy bled on by executives. This won’t work in testing.
Your copywriter needs to be able to writer for personas. He must be brave enough to create “corner copy”, or headlines that test the extremes of emotion, logic, spontaneity and deliberateness.
One of our biggest winning headlines was “Are you ready to stop lying? We can help.” It took bravery to write and defend this headline that delivered a 43% increase in leads for an addiction treatment center.

Tests and Trials: QA Methodology

Every hero is tested. Villains and the universe put obstacles in place to test the hero’s resolve. You, too, will be tested when you realize that you don’t have just one website. You have ten or twenty. Or thirty.
Your website renders differently in each browser. Safari looks different from Chrome. Internet Explorer seems to be dancing to it’s own HTML tune.
Your website renders differently on smaller screens. Smartphones, phablets, tablets and 4K monitors squeeze and stretch elements until your HTML and CSS snap.
Your website renders differently based on your connection. Fast Wi-Fi is often not available to your mobile visitors. Your development team is probably testing on fast Wi-Fi.
The permutations of these issues means that you can’t design for one site, even if you have responsive design.
Add to this JavaScript that moves things around on each of these different websites, and you have the potential to bring some of your visitors to an unusable website.
Quality Assurance, or QA is your defense against your new-found responsibility.
At Conversion Sciences, we go to the extreme of purchasing devices and computers that let us test a the most common browsers, screen sizes and operating systems.

Conversion Sciences' QA station

Conversion Sciences’ QA station


There are a number of sites that will provide simulators of a variety of devices, browsers and OSes. These have names like BrowserStack, Sauce Labs, and Litmus.
How do you know which of these you should be QAing on? Your analytics database, of course. Look it up.

Magical Help: AB Testing Tools

As we said above, your executive champion can bestow on you supernatural aids to help you in your journey. This is not the only source of magical helpers. Aladdin found the magic lamp in a Cave of Wonders.
Your magic lamp and “genie” are your AB testing tool. These marvelous tools make our agency possible. AB Testing tools have magical powers.

  • They inject JavaScript into our visitors’ browsers, allowing us to change our website without changing the backend.
  • They split traffic for us, letting us isolate individual segments of visitors to test.
  • They track revenue, leads and subscribers for us, so we know if our changes really generate more business for us.
  • They provide the statistical analysis that tells us when we can declare a winner in our tests.
  • They provide lovely graphs and reports.

The market leaders currently are Optimizely, Adobe Target, and Visual Website Optimizer (VWO). We have also used Convert.com, Maxymiser, Monetate and Marketizator to test websites.
We call these tools the “supreme court” of analytics. They control many of the variables that pollute data, and give us confidence that our changes will deliver more revenue, leads and subscribers to our clients.

The Belly of the Whale: Patience

The story of “Jonah and the Whale” appears in the Bible and Quran. In short, God asks Jonah to go to the city of Ninevah. Jonah hems and haws. So God arranges for Jonah to be swallowed by a big fish. After three days of apologizing, the whale spits Jonah out, and he begins his God-ordained journey.
It turns out that the belly of the whale is a theme in many hero myths. Like them, you will find yourself waiting and wondering as your tests slowly gather data. Some will go more slowly than others. Pressures from executives will mount. You must persevere.
[pullquote]Do not rush to make decisions, even if it looks like your test is going to deliver you a winner. Let the process run its course.[/pullquote]

The Reward: Revenue, Subscribers and Leads

In the end, you will have winning hypotheses and losing hypotheses. Because you won’t launch the losers and will push live the winners, you’ll begin to collect your boon, your reward, your Benjamins.
Be sure to show off a bit. Toot your own horn. Heroes come home to fanfare and celebration. Let your organization know what your AB testing program is doing for them and revel in the glow of success.

Master of Two Worlds: AB Testing Program

Your journey has taken you from magical helpers to reward. Along the way you entered a new world, learned its rules, overcame tests and trials, and used magic to win the day.
You are now Master of Two Worlds: Master of the old world of pray marketing, and Master of the new world of data-driven marketing. This will be a career builder and a business boon.
This is everything you need to build your AB testing program. Conversion Sciences offers a ready-made turnkey conversion optimization program. Ask for a free strategy consultation an carve months off of your hero’s journey.
[signature]

The AB Testing JavaScript that powers tests is powerful, but can lead to many unintended consequences.
Conversion Sciences offers a pretty amazing ability to our clients: A completely “turnkey” testing service. By “turnkey” we mean that our clients don’t need to do anything to their website in order for us to analyze the site, design creative, develop code, QA, test, review and route traffic to winners.
Why is this? Don’t we need new pages designed? Interactions with IT? Release schedules? Sprints?
The reason we have this “phenomenal cosmic power” is that our AB testing tools take control of each visitor’s browser. The changes are made on the visitors’ devices, so the website doesn’t have to be modified.
Well, not until we’ve verified that a change brings in big money.
While this makes us feel all high and mighty, it comes with a big drawback.

The Magic and Mania of JavaScript

All of the major browsers have a scripting engine built into them. It allows programs to be run inside the browser. The programming language used is called JavaScript. It’s JavaScript that makes websites interactive. Website developers use JavaScript to make text accordion when you click a heading. It is used to rotate images in a carousel.
Unfortunately, developers use JavaScript to do silly things, like parallax animations.

This unnecessary "parallax" motion may be reducing the conversion rates for this site.

This unnecessary “parallax” motion may be reducing the conversion rates for this site.


And then there’s this.
Don't use JavaScript animations just because you can.

Don’t use JavaScript animations just because you can.


Yes, JavaScript gives us the power to make our websites harder to use.

JavaScript is Used in AB Testing Software

Our developers use JavaScript to modify a website when you visit it. The AB testing software “injects” our JavaScript into the browser when the page is loaded.
First the web page is loaded as is from the webserver. Some visitors will see only this default version of the website.
For some visitors, our JavaScript is then injected and executed. The AB Testing software determines who sees the default web page and who will see a variation of the page.

Phenomenal Cosmic AB Testing Power

We can change almost anything about a page using AB Testing JavaScript.
We change the headline to something different.
We change the order of a rotating carousel, or slider.
We hide elements…

AB Testing flicker can be cause by simply removing elements.

AB Testing flicker can be cause by simply removing elements.


and insert them as well.
We insert video.
We completely change the look of a website.
We completely change the look and feel of this site. The test showed the white site performed the same as the brown site.

We completely change the look and feel of this site. The test showed the white site performed the same as the brown site.

AB Testing JavaScript is a Step Toward Personalization

If we wanted, we could deliver a different website to every visitor who arrives. If you’re thinking “personalization” at this point, then you have an idea of where our industry is heading. [pullquote]AB testing produces the data that makes personalization truly work.[/pullquote] Here are my instagraphic notes from a presentation by Chris Gibbins at Conversion Conference 2016 on using AB testing and personalization.

Instagraphic shows why and how to use AB Testing for personalization.

Instagraphic shows why and how to use AB Testing for personalization.

AB Testing Flicker, Flash, Blink and Ghosting

Unfortunately, JavaScript can introduce an error into our AB tests. Because we always load the website “as it is” on the server first, there is a possibility that the visitor will see the original for a fraction of a second before our changes get executed.
This has been called a “flash”, “flicker”, and a “blink”. It can have a significant effect on test results.

With AB Testing JavaScript, Flash is not cool.

With AB Testing JavaScript, Flash is not cool.


The problem with this AB testing JavaScript flicker is that it won’t be there if a change is made permanent on the site. Our tests may say the new change generated more revenue, and we’ll change the website to make that change manifest. But there will be no flicker. This means there is another variable in our test.
Was it the new headline we tested or was it the flicker that made more people buy?
[sitepromo]

The Human Eye is Drawn to Motion

Our eyes and brains have evolved to make motion important. When something moves, our eyes and brains work hard to figure out if the thing moving is edible, if it will eat us, or if we can mate with it. Technology has moved faster than evolution. Even though we’re looking at a website, where there is no immediate source of food, fear or fornication, our lizard brains continue to favor motion.
Imagine this scenario. We are testing the text on a button. That means that for some of our visitors, the button will change. Others will see the default button text. Then we’ll see who buys the most.

The changing element on this page draws your eye, and can make the page impossible to focus on.

The changing element on this page draws your eye, and can make the page impossible to focus on.


If there is a flicker, flash or blink when the text changes, the button will be immediately more visible to those visitors who see it. More of them will see the call to action. This may make more of them consider buying than those who simply scrolled past. If this new treatment generates more revenue, we are left with the question, “Was it the text or was it the motion that created the lift in sales?”
We won’t know until we push the new text onto the server and do a post-rollout analysis to see the real lift. At this point, we may find that the text didn’t increase purchases. It’s a real bitch.
How many buttons have been change because of flicker?

AB Testing Software Tries to Eliminate Blinking

The AB testing software that we use works to eliminate this blinking and flashing. The issue is so important that Convert Insights has patented their method for eliminating “blink”.
AB testing software like Optimizely, Visual Website Optimizer, Adobe Test, and Marketizator, load asynchronously, meaning that they load our changes as the page loads. This makes it easier for changes to be made before the visitor sees the page.

How Convert Insights Eliminates “Blink”

“The first thing this snippet does is hide the body element for max. 1.2 seconds (this can be set), this prevents blinking of elements already visual on the site that load under 1.2 seconds (we have yet to get a client that loads faster than this). During the 1.2 seconds, the SmartInsert(R) technology search and replaces DOM-elements every couple of milliseconds and loops through the entire available DOM-elements in the browser of the client. After all elements are replaced the body hidden attribute is set to display the site again either at 1.2 seconds or when the signal is given that all elements have been replaced (or DOM-ready).
Everybody can see how this technology works by loading our Chrome Extension.”
— Dennis van der Heijden, Convert.com

Eliminating Flash Flicker and Blink in AB Tests

In addition to this, our developers can do things that reduce and eliminate flicker and blink. Every test you do has different issues, and so a variety of tactics can be used to address them.

Avoid Google Tag Manager

Don’t use a Tag Manager like Google Tag Manager to serve your AB testing JavaScript software tags. Add them to the page manually. Tag managers can counteract the asynchronous loading of the tool and delay when changes can be made.

Make Changes with CSS

If the change can be made with the cascading style sheets (CSS), we favor making changes with CSS over using JavaScript. CSS can make changes to an element – like an image or font – that can be applied before an element displays.

Modal Dialogs and Overlays

Modal dialogs usually don’t display until the visitor takes an action. They can be edited before they are shown without flashing.

Use a Timer for DOM Changes

All of the images, headings, copy, widgets, and forms are stored in the browser in a big database called the DOM (Document Object Model). When JavaScript makes changes to the DOM, the content on the page changes. The DOM is slow-loading, as you can imagine.
Our developers will set a timer in JavaScript to check for when a changing element is loaded. By watching for our element to be loaded, we can execute a change it before the DOM – and the page – is loaded.

For the AB Testing Software to Load Our Changes Immediately

The AB testing software provides a synchronous loading mode. Optimizely and VWO use different approaches for this.

Rethink the Test

Sometimes, we have to go back to the drawing board and design a test that doesn’t cause flash-inducing changes. We will refactor a test to eliminate items that cause a flash or flicker.

Delay Display of the Content

We can delay the display of the content until the DOM is loaded and all elements have been changed. This causes a different kind of issue, however. The visitor sees a blank page or blank section for longer if they get the page with the change.
In one case we added a spinning wheel to indicate that the page is loading.
loading_small

Insert Placeholders Using Fast CSS

When inserting elements, we’ll use CSS to insert a placeholder, then use JavaScript to modify the element after the page has loaded. This reduces redrawing of the page when elements are inserted.

We created a blank box in CSS to minimize AB Testing Flash on this mobile website.

We created a blank box in CSS to minimize AB Testing Flash on this mobile website.

Optimizing for Mobile

Mobile pages load more slowly. This is because visitors don’t always have a high-quality connection when visiting from the festival, or from inside the walls of a bank while standing in line. For mobile devices, flash can be an even bigger issues.
Fortunately, the tactics we use on the desktop also work on mobile websites. But don’t forget to QA your test treatments on 3G and 4G connections. You may find flicker and blink on mobile devices that didn’t appear on your desktop.

Great JavaScript Developers are a Must

We spend a considerable amount of our time making sure our JavaScript and CSS changes are like the “native” implementation would look.  It’s one of the things that makes testing hard. Our team has the experience to ensure your tests aren’t falling victim to flicker, flash, blink or ghosting.
If you’d like our team to take over the effort of developing your AB tests, contact us for a free consultation and an overview of our process.
[signature]

Here are several questions about applying conversion science to ecommerce sites. These questions came from the sponsors of the GP Ecommerce Summit in Bucharest, Romania.

  1. Can we consider Conversion Rate Optimization a real science?

What defines a science? The Scientific Method.

  1. Assume we know nothing about a problem
  2. Research it
  3. Develop hypotheses
  4. Select the most likely hypothesis for testing
  5. Design a test that isolates that hypothesis
  6. Run the test using sound statistical methods
  7. Evaluate with post-test analysis
  8. Draw a conclusion
  9. Use the new information to formulate new hypotheses
  10. Repeat

I’ve just described our six month Conversion Catalyst process to you. We “science the sh*t” out of websites. Without the science, we make bad decisions, emotional decisions, decisions based on superstition and myth.
There is also a component of sport in conversion optimization. We are in this to win. While we must be objective, we like to find revenue and hate when our tests are inconclusive.

  1. What are the first steps you have to take if you wish to increase your conversion rate on your e-commerce website?

My recommendation is that ecommerce sites focus on the value proposition their offering. This is a combination of your categories (what you sell), your shipping policy, your return policy and your brand.
Zappos built an amazing online brand by putting its value proposition front and center, “Free shipping both ways. 365 day return policy. Empowered customer support people.”
What is your value proposition? Fast delivery? Local manufacturing? Free installation? Donations to charity with every purchase? Emphasize it on your site, in your cart and throughout checkout.

  1. How do you create a good landing page and what are the best ways to test it?

The best landing pages keep the promise of the ad, link or post that brought the visitor there. They make an offer that matches the promise as exactly as possible. They show the product, even if it is a service or a PDF or a video series. Good landing pages provide proof points that are specific and supported by fact. Good landing pages build trust by borrowing from customers and customers. Good landing pages make the call to action the most prominent thing on the page. And good landing pages don’t add any distractions, such as social media icons, links to other pages or corporate site navigation.
This is the chemical equation for landing pages: Offer + Form + Image + Proof + Trust = Landing Page

The chemistry of the landing page

The chemistry of the landing page

  1. Can persuasive writing help you sell more online or do you need more than that? For example, how do you test a good headline?

Most of our biggest wins come from copy changes, like headlines. We are even testing different kinds of testimonials on one site to see which build the most trust. The words are very important. This is related to the value proposition I discuss above. When you learn the emotional language that brings visitors into your site, you learn something about your audience. This insight can be used anywhere.

  1. What is an important point you want to drive home?

There is a wave of ecommerce sites rushing to rebuild their sites using responsive web design (RWD). This is in part due to Google and Mobilegeddon, but few can ignore the growing influence of mobile devices on our revenue. This rush to RWD is a mistake for many businesses who will find themselves with a poorly performing mobile site and a lower conversion rate on their redesigned desktop site. Tragic.
You should embrace your mobile visitors, and there are alternatives to RWD. I’ve seen some redesign horror stories and some pretty amazing success stories. Mobile design is still too new for there to be best practices, but our testing tells us what successful mobile designs should begin to look like.

  1. How do you remember the ecommerce market in the USA from 10 years ago?

Ten years ago, we didn’t have the data tools we have today. We relied much more on qualitative research. Most of my work was building out personas, making content recommendations and working with “best practices”. Google Analytics was young. We had been using server logs to get unreliable data on visitors. Only a few years before I had written my own web analytics package to get an idea of what was working on my sites.
Today, we have amazing qualitative and quantitative tools to uncover problems with our websites. We enjoy powerful testing tools to help us determine exactly what effect our changes will have on our businesses. We are creating revenue in the laboratory using science and creativity. We have moved from the tool-building phase into the human creativity phase. It’s a very exciting time to be an online business.
[sitepromo]
[signature]

What has the Conversion Scientist been reading lately?

AdExchanger: Why Do Mobile Users Not Buy On Mobile?

We believe that mobile traffic is every bit as important as desktop traffic. Many businesses walk away from their mobile traffic because it doesn’t convert well. This is a mistake.
Two points found in this article drive the point home:

  • App and Mobile Functionality (sucks)
  • Mobile Represents a Different Type of user

Spend some time on your mobile site. Don’t just create a responsive version of your desktop website.
Read more.

Marketizator: 25+ Tools That Conversion Rate Optimization Pros Can’t Ignore

I often say we’re living in a golden age of marketing, in which we can find data to answer almost any question we have. And these tools aren’t expensive. Every marketer can benefit from these tools with a little curiosity and patience.
[sitepromo]

Nielsen Norman Group: Long-Term Exposure to Flat Design: How the Trend Slowly Decreases User Efficiency

I reviewed 47 wordpress templates for a competition earlier this year. 98% of them used a “flat” design approach. Of course, we’re seeing this style of design pervade websites.
Is this a good thing? Nielsen Norman Group says we can use flat designs if we follow some smart guidelines.
Read more.
Got suggestions for what we should be reading? Share them with us!
[signature]

Search engine algorithms are evolving at higher paces than ever before. The frequent updates to these algorithms – especially Google’s search algorithm updates – have made it harder to “game” the system using Search Engine Optimization (SEO). This has forced companies to bring at least one SEO specialist on board in order to gain and keep high rankings for their websites in the search engine results pages (SERPs).
At the same time, advances in data-collection tools has made conversion rate optimization (CRO) one of the highest returns on the marketing investment (ROI). Ironically, CRO is one of the most underused activities in the marketing department.
This paradox becomes apparent once you consider that obtaining the click that brings someone to your website is only the first step toward converting the visitor into a paying customer. From this perspective, CRO carries the burden of managing the entire user interaction, as opposed to SEO, which arguably only brings the visitor to the “front door.”

SEO and CRO Are Meant to Work Hand-in-Hand

With SEO, the basic point of focus is the webpage. In conversion optimization, the central concept is a PPC ad and a matched landing page. Nevertheless, the principles of search engine and conversion rate optimization are undeniably compatible. In fact, here are a few fundamentals that apply to both SEO and CRO:

  • A conversion optimized page will prove user friendly and more likely to receive inbound links and referrals, thus improving SEO.
  • Having clear and relevant headlines, as opposed to excessively creative ones, will improve both SEO and CRO.
  • Using clear content hierarchy with proper heading tags will help with SEO and keep focus on the progression of the message, which will help with conversion.
  • A conversion optimized page should be using plenty of relevant keywords that match what visitors are searching for.
  • Replacing complex presentations with digestible pieces of content will improve your SEO and conversion rate.
  • Search engines will favor pages that are updated frequently. Keeping layouts and content fresh will prove beneficial for both SEO and CRO.
  • Pages that focus on a single topic or product achieve better search engine rankings and improve conversion rate.

SEO Factors Inform CRO Efforts

The SEO field has been revolving around the standards imposed by search engines, especially Google’s ranking factors. Some of these are documented by Google, some are relatively obvious, others are not confirmed, and some sit at the brink of speculation or wishful thinking.
Since SEO revolves around ranking factors, which basically dictate the actions and tools needed in this field, it’s only natural that the SEO insights most relevant to CRO are rooted in these ranking factors.

1. Focus on User Behavior

Conversion optimization is data-driven, much like SEO. Web analytics are your greatest asset, but you will need to do additional research into user behavior. Segmentation analysis becomes quite important. Ask yourself this: “How do different segments interact with your website, and how can you optimize their particular experiences?”
The user interaction factors most likely to be useful in CRO and impact on conversion optimization are:

  • Dwell time and click backs focus on how long people spend on your page before returning to the original SERP. Session duration is also important. It measures the amount of time people spend on your site and may be used as a quality signal by Google.
    Average session duration in Google Analytics

    Average session duration in Google Analytics


    If you’re having trouble differentiating dwell time, session duration, and bounce rate, read this article published by Neil Patel on Search Engine Journal. It will clarify the topic.
  • Bounce rate is used to calculate the percentage of users who navigate away from your site after viewing a single page. Bounce rate probably cannot be a ranking factor by itself. Metrics that can’t be applied broadly, with the objective of identifying relevant and quality content, usually are not Google algorithm factors. However, bounce rate will surely influence the way you strategize for conversion, especially in creating the A/B tests fundamental to CRO.
  • Direct and repeat traffic are powerful indicators of quality for Google. They use data collected through Chrome to determine how often users visit any particular site. Pages with a lot of direct traffic are favored in SERPs, because they are much more likely to contain quality and engaging content.

2. It’s Not Just the Landing Page, It’s Also the Website

Conversion optimization extends beyond single pages, creating what we call conversion paths throughout the website. SEO dictates that breaking up content into multiple steps is usually a bad idea. CRO specialists tell us that multiple-step landing pages can convert better, by engaging respondents in a mutually productive dialogue and facilitating proper segmentation. For this reason, some form of consensus needs to be achieved in order to allow both SEO and CRO specialists to reach successful results.
Some of the site-level SEO factors most likely to influence CRO are:

  • Site Architecture and Sitemap improve your site’s relationship with Google, since they allow the engine to index your pages and more thoroughly organize your content. Make sure your website can accommodate conversion paths without messing up its logic.
  • Domain TrustRank is a very important ranking factor. TrustRank is a link analysis technique described in the famous paper Combating Web Spam with TrustRank by researchers Zoltan Gyongyi, Hector Garcia-Molina of Stanford University, and Jan Pedersen of Yahoo!. SEO by the Sea tells us more about TrustRank.
  • Google indexes SSL certificates and uses HTTPS as a ranking signal. People are reluctant when offering credit card details and other personal data over the Internet. Obtaining an SSL certificate is crucial to offering assurance to customers and letting Google know that you are running a legitimate business.
  • Mobile friendly sites rank better with Google. Even before the April 2015 “Mobile Friendly” Google algorithm update, it was not unthinkable to assume that mobile friendly sites had an advantage in searches from mobile devices. Google actually displays “Mobile friendly” tags next to mobile search results.
    Google's mobile friendly tags

    Google’s mobile friendly tags


    Also, keep in mind that Google has precise standards for evaluating what constitutes mobile friendly design. Google WebMaster Central offers details about mobile friendly requirements. To assess your website’s current mobile performance, check out this Mobile Friendly Test.

[sitepromo]

3. If Content Is King, the Webpage Is Its Kingdom

In both SEO and CRO, content is king. In SEO, this wins you links. In conversion optimization, it wins you customers. [pullquote]You should never allow technical aspects to eclipse what is truly important: compelling value propositions and meaningful brand experiences.[/pullquote]

Page-level SEO factors that will prove crucial for conversion

Using keywords correctly throughout webpages is critical when trying to improve your search engine ranking and your conversion rates as part of your online marketing strategy. Keywords must be used in:

  • URLs.
  • Title tags. Place top-performing keywords in descending order and make sure that the title tag reflects the most important keywords used on that particular page. Here are 9 best practices for optimized < title > tags (Search Engine Land).
  • Description tags. This MOZ article states, “While not important to search engine rankings, [Meta Description Tags] are extremely important in gaining user click-through from SERPs.”
  • Heading tags. The heading tag is useful in outlining whole sections of content. It impacts both the SEO and usability of websites. For information on how to use these tags, consult this article from Woorank.com.
  • The body text. Fairly distributing the keywords throughout the content is crucial. You may want your keywords to be the most frequently used elements on the page. However, do not overstuff content with keywords. Use them intelligently and always favor usability. A link or review from an established source – thanks to the quality of your content – will weigh much more than keyword density. On the other hand, keyword prominence might be an important relevancy signal. Make sure to include your keywords in snippets and in the first 100 words of your content.

A great page layout influences rankings and conversion, if not directly as a quality signal, at least by scoring in the “user friendly category.” This keeps readers coming back for more. The page layout on highest quality pages makes the main content immediately visible.
Content length. While life on- and off-line speeds up and our attention span keeps narrowing, you would expect content to get shorter in order to efficiently catch the attention of users. On the contrary, long articles rank and convert better than short ones. Review the results of an A/B testing experiment conducted by Neil Patel, demonstrating the superior efficiency of long copy.

4. Build Links, Build Trust, Build Rapport

One of the driving goals of SEO is link building. Conversion optimization deals with links mostly in terms of conversion paths. Landing pages usually do not contain links themselves other than for the call to action (CTA). However, many SEO factors concerning link building can apply to CRO in crucial ways. Here are some examples:

  • The quality and word-count of the linking content make a big difference in link value. For example, receiving a link from a 2,000+ word well-written article weighs in much more than a link from a short comment or a poorly written blog post.
  • “Poisonous” anchor text pointed toward your site may be a sign of spam or a hacked site. Either way, it can hurt your ranking and your conversion rates, particularly when the anchor texts in question are stuffed with pharmaceutical keywords.
  • If there are low-quality links pointing to your landing pages, or you receive unnatural links warnings from Webmaster Tools, you can always use the Disavow Tool. It will not remove the harmful links themselves, but at least it will eliminate them from Google’s assessment of your site.
    You have the option to disavow links

    You have the option to disavow links

  • Contextual links – links placed within the content of pages – are more valuable than links found in sidebars, footers, or anywhere else on the page. So on top of the PPC ads, try getting your landing pages mentioned in relevant content on relevant websites.

5. Your Brand Needs a Social Identity to Attract and Convert

In terms of the decision to purchase, user behavior has been shifting toward a multi-source, multiple stage process over the last few years. Regardless of how persuasive your landing pages are and how well they bring customers to the realization that you have the answer to their specific needs, your brand needs to back up its claims with a healthy social media presence and an SEO effort that encompasses social factors. Here are a few of the factors that can inform CRO specialists on what needs to be done:

  • Google officially favors real brands and real businesses, with real offices and real people, so it only makes sense they would verify businesses and brands by their website and social media location data. MOZ goes even further and suggests that Google looks at whether a website is associated with a tax-paying business.
  • Brands have Facebook pages with many likes and Twitter profiles with many followers. Moreover, serious businesses have proper company Linkedin pages. Interestingly, Rand Fishkin, co-founder of MOZ, states that having many Linkedin profiles that list working for your company will improve your rankings and might actually constitute a brand signal.
  • Social media account authority weighs considerably in SERPs, especially since social media has become a major influencer of consumer behavior. This infographic published by Social Media Today shows how social media influences consumers, the types of content that deliver the most impact, and more.
A link shared on multiple accounts will be more valuable than the same link shared multiple times on one account.

A link shared on multiple accounts will be more valuable than the same link shared multiple times on one account.

Wrapping It Up

Looking ahead, experts predict a major detachment from traditional ranking factors to a much deeper analysis of perceived site value, authority, structured data, and social signals. Automation is transforming digital marketing, turning SEO and CRO into much more precise and effective fields in the process. Ideally, within this decade Google’s services and search algorithm will evolve to a level that will allow us to fully customize our proposals according to our customers’ buying cycles.

Alexander Kesler headshotAbout the Author

Alexander Kesler is the President of inSegment, a Boston-based digital marketing and advertising agency. He is a graduate of Babson College, where he earned a B.S. in Entrepreneurship.
Feature image licensed by Bgubitz through Creative Commons and adapted for this post.

If you ever went to the government and asked them what your fair share of taxes should be, they would first ask you how much you made last year.
And that would likely be the answer.
Likewise, a conversion optimizer would probably be the last person to ask how much to budget for conversion optimization. “How much budget do you have?”
Nonetheless, I’m going to give you the tools to add conversion optimization to your budget next year. Then, when you call us next year, you’ll be ready.

Where to Get Your CRO Budget

One key question you need to ask is, where will I get my CRO budget? I have some suggestions.

1. From IT

The basis of any conversion optimization effort is a sound analytics and measurement foundation. This consists of tools that slide under your website and are bolted in place. This is IT stuff.
Our research has shown that most businesses’ websites have some level of implementation of analytics. You don’t want to be left behind. This is a crucial behavioral database that will be invaluable as you begin to vet ideas for testing.

2. From the Things You Should be Testing Anyway

It is a golden age of marketing. We have more tools, data sources and shiny objects to drive our online businesses than any marketers have ever had. We can mobile gamify our ratings and review process using direct visitor feedback to drive personalization throughout our content funnels.
In other words, we’re overwhelmed, and the first sign of a marketing department that is overwhelmed is the decision to redesign. [pullquote]Your website probably doesn’t need a redesign. It probably needs to be optimized.[/pullquote] Put the redesign money into an optimization program and see immediate results.
There is a good way to get your head around all of the things you could be doing to your site. You could test the ideas. Instead of blindly pouring money into exit-intent popovers, live chat, or personalized recommendations, you should test them. We have seen these work and we have seen them fail.
Your conversion optimization team will know how to use data to make good decisions on where to spend your money. Budget for optimization first.

3. From Your Ad Spend

Paid search is a great way to generate qualified traffic. However, our success in search causes our fundamentals to “regress”. It becomes harder to increase traffic, and the new traffic often is less qualified, less profitable.

When you spend more, get less traffic and make less money, it's time to try optimization.

When you spend more, get less traffic and make less money, it’s time to try optimization.


[pullquote]When your traffic is flat, ad spend is rising and profit is dropping, you know you should be putting some of that into optimization.[/pullquote] This may seem like a no-brainer, but there is a period of sweat and anxious hand-wringing.
You see, conversion optimization takes time. There is a very real dip in performance. When you reduce spending on ads you reduce your traffic and your revenue. For a period of time, your revenue drops until your optimization efforts get traction.
It might look something like the graph below. This assumes a modest 5% increase in revenue per visit (RPV) each month for one year, and that 8.9% of ad spend, or $8900, is invested in optimization each month. In this example, we began with a conversion rate of 1.7%.
If you can make it through a short valley of death, borrowing from your ad spend can be very profitable.

If you can make it through a short valley of death, borrowing from your ad spend can be very profitable.


Monthly revenue dips due to the reduction in PPC traffic. Revenue returns to baseline levels in month four. Revenue is positive in month six compared to investing in PPC only.
The Return on CRO (green line) turns sharply north, even though we are still investing 8.9% of ad spend each month. This is what powers conversion optimization. You have a compounding effect working in your favor, but you have to invest on the front end.
Send me an email if you want to see all my assumptions.
It’s this four-to-six month dip that marketers and managers fear. How do you sell a drop in revenue to your boss?

4. Pony Up

The other option is to reach into your own profits and slap down some cash on your conversion optimization team.
I’m not going to sugar coat this. There are three costs you must deal with when investing in optimization.

The Components of a Conversion Optimization Budget

The Software

The first cost is the least bothersome. Conversion optimization requires a certain amount of data to succeed. [pullquote]Testing is not that hard. Deciding what to test is quite difficult.[/pullquote]
The competition in the marketplace is pretty brutal. Each year, we get more functionality from cheaper and cheaper tools. At a minimum, you’ll want a good click-tracking tool, a good session recording tool, a strong analytics database and a split-testing tool.
Depending on your traffic, these can be had for a few hundred dollars each month up to several thousand dollars each month.

The Team

None of these tools matter if you don’t have someone to pull the levers, turn the knobs and read the graphs. The main functions found on a conversion optimization team are:
A researcher to collect qualitative data.
A statistically-responsible person to collect and evaluate quantitative data.
A developer to create the changes in each test.
A designer to implement design changes.
A patient QA person to be sure nothing is broken by a test.
A project manager to keep the momentum going.
It is possible to have one super-amazing person who can do all of this. It is the death-knell of your conversion optimization program to ask someone to do all of this in addition to another job. Your PPC person is not going to be able to do all of this and their job too.
These are fairly expensive employees. Consider hiring an outside agency, like us, to get started. As of this writing, Conversion Sciences can provide these functions for less than ten-thousand dollars a month.

The Opportunity Costs

There is a cost to testing that is not seen in reports. It’s the cost of losing treatments. In any list of “good” ideas for increasing your conversion rate and revenue per visit, fully half will actually do more harm than good. We don’t know which of our ideas are “losers” until we test them. When we test, some percentage of your visitors will see these losers, be turned off, and won’t buy.
This is lost revenue. With proper management, this downside can be minimized, but it is the cost of doing business. It’s the price of admission, the overhead, the burn, that funny smell in the kitchen.
It’s hard to budget for this particular line item, but it should be part of your discussion.

Be Clear About Your Upside

If I haven’t scared you off, there is good news. We call it the upside, the green bling, statistical bignificance, and sometimes we just dance.
You should understand what your statistical bignificance is. You must know the answer to the question, “What happens if my conversion rate goes up a little?” We call this a Basic Unit of Upside.

Conversion Upside Calculator
Click for a Conversion Optimization Upside Report that does the math for you.

We offer our Conversion Optimization Upside Report to help you understand your upside. It calculates what your yearly increase in revenue would be if you only added 0.1 to your conversion rate or revenue per visit. Plug in a few numbers and you’ll see what small changes mean for your bottom line.

A Little More Motivation

For most businesses, conversion optimization is a ten-thousand-dollar a month investment or more. Many businesses are spending a whole lot more than that.
If conversion optimization is on your “maybe next year” list, consider what might happen if you give your competitors a year’s head start on you.
The business with the highest conversion rate has the lowest acquisition cost and can profitably boost bids on their paid advertising. Plus, Google favors high-converting landing pages when assigning ad placement.
With a realistic understanding of the costs of conversion optimization and a real appreciation for the potential upside, you should be able to make the case for adding it to your shopping list in 2016.
[signature]
Feature image by frankieleon via Compfight cc and adpated for this post.