AB Testing Research: Do Your Conversion Homework

Infogrphic: AB testing research delivers conversion evidence.

IF we had to pick one thing that has made Conversion Sciences a successful AB testing agency, it would be this: We are very good at picking what to test.

This isn’t because the team is made up of geniuses (except you Brian, we all know you’re a genius). It’s because we have a consistent methodology for conducting AB testing research. In other words, we do our homework.

Like we talked about in our our rundown of the best AB testing tools, “your AB tests are only as good as the hypotheses you are testing.”

With the proper research, we can consistently make better hypotheses, leading to more profitable testing results and a better experience for our visitors.

100 Million Neurons in Our Guts

Would you believe there are 100 million neurons in the human gut? This concentration is second only to our brains, even prompting scientists to refer to it as our “second brain”. While we don’t use our gut to make conscious decisions, it can greatly influence our mental state and is likely the reason we have “gut reactions” or “gut feelings”.

There are times when “going with your gut” makes sense. That time is when you don’t have any other options. If there is no information available to you, your gut may be a good second opinion for your brain.

On the web, there is rarely a need to go with your gut. Let’s redefine these terms.

Whenever someone says “My gut reaction is…” you should hear, “I don’t really know. Let’s do some more research.”

Whenever someone says, “I have a gut feeling that…” you should hear, “I don’t have enough information. How can we better inform ourselves before making this decision?”

We are living in a golden age of digital marketing information. With such easy access to research methods, there is no good reason to ever go from the gut on web design, copywriting, value proposition, etc. You don’t need your intestines to design your website.

After all, the primary output of a healthy gut is… well… crap.

What research skills can keep you from resorting to your colon for inspiration? To answer that question, we worked with KlientBoost to capture many of the key AB testing research methods and enjoy the satisfying feeling of winning AB tests.


Conversion Research Evidence Infographic


Research + Framework = Growth

AB testing research feeds an AB testing framework for test results that are consistently positive and repeatable. Feed it well, and it will poop out revenue growth month after month. This is the only resemblance to your gut I could think of.

It doesn’t make sense to test an idea without some evidence that it will make a difference. Good research is full of nutrients, vitamins and fiber. And that is the last time I’ll refer to the digestive system in this article.

The Heart of AB Testing Research

Before we get into the details, it’s important to understand the core of testing research, and ultimately, the core of conversion optimization itself:

“The definition of optimization boils down to understanding your visitors.” – Brian Massey

Optimization is just a fancy word for bettering our understanding of our customers and giving them more of what they want.

Behavioral data is the best, most reliable research you can get. With it, we can eliminate tripping points and optimize the experience. We find this in our analytics databases. But much of our AB testing research will not be behavioral, and that is fine.

In general, there are two types of research:

  1. Quantitative Research
  2. Qualitative Research

Understanding Quantitative Research

Quantitative data is generated from large sample sizes. Quantitative data tells us how large numbers of visitors and potential visitors behave. It’s generated from analytics databases (like Google Analytics), trials, and AB tests.

The primary goal of evaluating quantitative data is to find where the weak points are in our funnel. The data gives us objective specifics to research further.

There are a few different types of quantitative data we’ll want to collect and review:

  • Backend analytics
  • Transactional data
  • User intelligence

Understanding Qualitative Research

Qualitative data is generated from individuals or small groups. It is collected through heuristic analysis, surveys, focus groups, phone or chat transcripts, and customer reviews.

Qualitative data can uncover the feelings your users experience as they view a landing page and the motivations behind how they interact with your website.

Qualitative data is often self-reported data, and is thus suspect. Humans are good at making up rationalizations for how they behave in a situation. However, it is a great source of test hypotheses that can’t be discerned from quantitative behavioral data.

While quantitative data tells us what is happening in our funnel, qualitative data can tell us why visitors are behaving a certain way, giving us a better understanding of what we should test.

There are a number of tools we can use to obtain this information:

  • Surveys and other direct feedback
  • Customer service transcripts
  • Interviews with sales and customer service reps
  • Session Recording

Usability and User Experience

Two of our key objectives in going through all this data is to evaluate our website’s Usability and User Experience.

Usability deals with how easy it is for someone to learn and use the functions of our site. If we can make any part of that customer journey easier or more intuitive, we are increasing Usability.

User Experience deals with the emotions and attitudes users experience as they use our site. If we can make the customer journey more enjoyable, we are improving User Experience.

While these two concepts often go hand in hand, they are not the same, and both need to be kept in mind when collecting data.

The Importance of Segmentation

It’s not enough to simply know that “visitors” are doing ____ when they visit a given webpage or flow through a given funnel.

Which visitors?

  • Are they on mobile or desktop?
  • Are they here via paid ads or organic search?
  • Are they using Chrome or Firefox?
  • Did they click-through via a Facebook post or a Tweet?
  • Are they a new or returning visitor?

In order to properly understand and evaluate our visitors and customers, divide them into strategic segments to understand the differences across each segment.

It’s especially important to know what these key segments are before we run our AB tests, because otherwise, our tests won’t tell us anything about them.

Follow the Proven System A/B Testing Agencies Use

AB testing research is a fundamental part of any proven CRO framework, and it’s an important part of what separates an ROI-generating A/B testing agency from a waste of money.

As you finish up the year and move into 2017, it’s time to take things up a notch. In 2016, hundreds of businesses drastically improved their bottom lines via a proven, systematic CRO process.

Why not join the party? Take our free gift and click here to schedule a call with one of our CRO professionals.


Think Twice: How the Gut’s “Second Brain” Influences Mood and Well-Being

Categories: AB Testing