Page 18 – Conversion Sciences

How an A/A Test Gives You Confidence

Nothing gives you confidence and swagger like AB testing. And nothing will end your swagger faster than bad data. In order to do testing right, there are some things you need to know about AB testing statistics. Otherwise, you’ll spend a lot of time trying to get answers, but instead of getting answers, you’ll end up either confusing yourself more or thinking you have an answer, when really you have nothing. An A/A test ensures that the data you’re getting can be used to make decisions with confidence.

What’s worse than working with no data? Working with bad data.

We’re going to introduce you to a test that, if successful will teach you nothing about your visitors. Instead, it will give you something that is more valuable than raw data. It will give you confidence.

What is an A/A Test

The first thing you should test before your headlines, your subheads, your colors, your call to actions, your video scripts, your designs, etc. is your testing software itself. This is done very easily by testing one page against itself. One would think this is pointless because surely, the same page against the same page is going to have the same results, right?

Not necessarily.

After three days of testing, this A/A test showed that the variation identical to the Original was delivering 35.7% less revenue. This is a swagger killer.

This A/A Test didn’t instill confidence after three days.

This can be cause by any of these issues:

The AB testing tool you’re using is broken.
The data being reported by your website is wrong or duplicated.
The AA test needs to run longer.

Our first clue to the puzzle is the small size of the sample. While there were over 345 or more visits to each page, there were only 22 and 34 transactions. This is too small by a large factor. In AB testing statistics, transactions are more important than traffic in building statistical confidence. Having fewer than 200 transactions per treatment often delivers meaningless results.

Clearly, this test needs to run longer.

Your first instinct may be to hurry through the A/A testing so you can get to the fun stuff – the AB testing. But that’s going to be a mistake, and the above shows why.

An A/A test serves to calibrate your tools

Had the difference between these two identical pages continued over time, we would call off any plans for AB testing altogether until we figured out if the tool implementation or website were the source of the problem. We would also have to retest anything done prior to discovering this AA test anomaly.

In this case, running the A/A test for a longer stretch of time increased our sample size and the results evened out, as they should in an A/A test. A difference of 3.5% is acceptable for an AA test. We also learned that a minimum sample size approaching 200 transactions per treatment was necessary before we could start evaluating results.

This is a great lesson in how statistical significance and sample size can build or devastate our confidence.

An A/A Test Tells You Your Minimum Sample Size

The reason the A/A test panned out evenly in the end was it took that much time for a good amount of traffic to finally come through the website and see both “variations” in the test. And it’s not just about a lot of traffic, but a good sample size.

Your shoppers on a Monday morning are statistically completely different people from your shoppers on a Saturday night.
Your shoppers during a holiday seasons are statistically different from your shoppers on during a non-holiday season.
Your desktop shoppers are statistically different from your mobile shoppers.
Your shoppers at work are different from your shoppers at home.
Your shoppers from paid ads are different from your shoppers from word of mouth referrals.

It’s amazing the differences you may find if you dig into your results, down to specifics like devices and browsers. Of course, if you only have a small sample size, you may not be able to trust the results.

This is because a small overall sample size means that you may have segments of your data allocated unevenly. Here is an sample of data from the same A/A test. At this point, less than 300 sessions per variation have been tested. You can see that, for visitors using the Safari browser–Mac visitors–there is an uneven allocation, 85 visitors for the variation and 65 control. Remember that both are identical. Furthermore, there is an even bigger divide between Internet Explorer visitors, 27 to 16.

This unevenness is just the law of averages. It is not unreasonable to imagine this kind of unevenness. But, we expect it to go away with larger sample sizes.

You might have different conversion rates with different browsers.

Statistically, an uneven allocation leads to different results, even when all variations are equal. If the allocation of visits is so off, imagine that the allocation of visitors that are ready to convert is also allocated unevenly. This would lead to a variation in conversion rate.

And we see that in the figure above. For visitors coming with the Internet Explorer browser, none of sixteen visitors converted. Yet two converting visitors were sent to the calibration variation for a conversion rate of 7.41%.

In the case of Safari, the same number of converting visitors were allocated to the Control and the calibration variation, but only 65 visits overall were sent to the Control. Compared this to the 85 visitors sent to the Calibration Variation. It appears that the Control has a much higher conversion rate.

But it can’t because both pages are identical.

Over time, we expect most of these inconsistencies to even out. Until then they often add up to uneven results.

These forces are at work when you’re testing different pages in a AB test. Do you see why your testing tool can tell you to keep the wrong version if your sample size is too small?

Calculating Test Duration

You have to test until you’ve received a large enough sample size from different segments of your audience to determine if one variation of your web page performs better on the audience type you want to learn about. The A/A test can demonstrate the time it takes to reach statistical significance.

The duration of an AB test is a function of two factors.

The time it takes to reach an acceptable sample size.
The difference between the performance of the variations.

If a variation is beating the control by 50%, the test doesn’t have to run as long. The large margin of “victory”, also called “chance to beat” or “confidence”, is larger than the margin of error, even at small er sample sizes.

So, an A/A test should demonstrate a worst case scenario, in which a variation has little chance to beat the control because it is identical. In fact, the A/A test may never reach statistical significance.

In our example above, the test has not reached statistical significance, and there is very little chance that it ever will. However, we see the Calibration Variation and Control draw together after fifteen days.

These identical pages took fifteen days to come together in this A/A Test.

This tells us that we should run our tests a minimum of 15 days to ensure we have a good sample set. Regardless of the chance to beat margin, a test should never run for less than a week, and two weeks is preferable.

Setting up an A/A Test

The good thing about an A/A test is that there is no creative or development work to be done. When setting up an AB test, you program the AB testing software to change, hide or remove some part of the page. This is not necessary for an A/A test, by definition.

For an A/A test, the challenge is to choose the right page on which to run the test. Your A/A test page should have two characteristics:

Relatively high traffic. The more traffic you get to a page, the faster you’ll see alignment between the variations.
Visitors can buy or signup from the page. We want to calibrate our AB testing tool all the way through to the end goal.

For these reasons, we often setup A/A tests on the home page of a website.

You will also want to integrate your AB testing tool with your analytics package. It is possible for your AB testing tool to be setup wrong, yet both variations behave similarly. By pumping A/A test data into your analytics package, you can compare conversions and revenue reported by the testing tool to that reported by analytics. They should correlate.

Can I Run an A/A Test at the Same Time as an AB Test?

Statistically, you can run an A/A test on a site which is running an AB test. If the tool is working well, than your visitors wouldn’t be significantly affected by the A/A test. You will be introducing additional error to your AB test, and should expect it to take longer to reach statistical significance.

And if the A/A test does not “even out” over time, you’ll have to throw out your AB test results.

You may also have to run your AB test past statistical significance while you wait for the A/A test to run its course. You don’t want to change anything at all during the A/A test.

The Cost of Running an A/A Test

There is a cost of running an A/A test: Opportunity cost. The time and traffic you put toward an A/A test could be used to for an AB test variation. You could be learning something valuable about your visitors.

The only times you should consider running an A/A test is:

You’ve just installed a new testing tool or changed the setup of your testing tool.
You find a difference between the data reported by your testing tool and that reported by analytics.

Running an A/A test should be a relatively rare occurrence.

There are two kinds of A/A test:

A “Pure” two variation test
An AB test with a “Calibration Variation”

Here are some of the advantages and disadvantages of these kinds of A/A tests.

The Pure Two-Variation A/A Test

With this approach, you select a high-traffic page and setup a test in your AB testing tool. It will have the Control variation and a second variation with no changes.

Advantages: This test will complete in the shortest timeframe because all traffic is dedicated to the test

Disadvantages: Nothing is learned about your visitors–well, almost. See below.

The Calibration Variation A/A Test

This approach involves adding what we call a “Calibration Variation” to the design of a AB test. This test will have a Control variation, one or more “B” variations that are being tested, and another variation with no changes from the Control. When the test is complete you will have learned something from the “B” variations and will also have “calibrated” the tool with an A/A test variation.

Advantages: You can do an A/A test without stopping your AB testing program.

Disadvantages: This approach is statistically tricky. The more variations you add to a test, the larger the margin of error you would expect. It will also drain traffic from the AB test variations, requiring the test to run longer to statistical significance.

AA Test Calibration Variation in an AB Test (Optimizely)

Unfortunately, in the test above, our AB test variation, “Under ‘Package’ CTAs”, isn’t outperforming the A/A test Calibration Variation.

You Can Learn Something More From an A/A Test

One of the more powerful capabilities of AB testing tools is the ability to track a variety of visitor actions across the website. The major AB testing tools can track a number of actions that can tell you something about your visitors.

Which steps of your registration or purchase process caused them to abandon your site
How many visitors started to fill out a form
Which images visitors clicked on
Which navigation items were most frequently clicked

Go ahead and setup some of these minor actions–usually called ‘custom goals’– and then examine the behavior when the test has run its course.

In Conclusion

Hopefully, if nothing else, you were amused a little throughout this article while learning a bit more about how to ensure a successful AB test. Yes, it requires patience, which I will be the first to admit I don’t have very much of. But it doesn’t mean you have to wait a year before you switch over to your winning variation.

You can always take your winner a month or two in and use it for PPC and continue testing and tweaking on your organic traffic. That way you get the both worlds – the assurance that you’re using your best possible option on your paid traffic and taking the time to do more tests on your free traffic.

And that, my friends, is AB testing success in a nutshell. Now go find some stuff to test and tools to test with!

About the Author

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

43 Pages with Examples
Assumptive Phrasing
"We" vs. "You"
Pattern Interrupts
The Power of Three

"*" indicates required fields

June 17, 2016/by Kristi Hines

The Hero's Journey to an Amazing AB Testing Program

CRO Tests | Multivariate | AB Testing

We are often guilty of writing about the AB testing process as if it was something you can jump into and start doing. We believe an AB testing program can keep you from making expensive design mistakes and find hidden revenue that your competitors are currently getting. It’s not an overnight switch, however. It takes some planning and resources.
It’s a journey not unlike that taken by many heroes throughout history and mythology. We invite you to join the ranks of heroic journeymen.
The journey looks something like this: You befriend a helpful stranger who gives you something magical. Soon, you are called to adventure by events beyond your control. When you act, you enter a strange new world and must understand it. You are set on a journey to right some wrong. Allies and helpers work with you to move past trials and tests. Your magical talisman helps you understand what to do. With patience you gather your reward and return home, the master of two worlds.
Consider this blog post a harbinger of the adventure that awaits you. Here are the things you’ll encounter on your journey.

Executive Champion: Magical Helper

Every hero story involved some kind of “supernatural help”. In the story of the Minotaur, Ariadne gave Theseus the golden string to find his way out of the Minotaur labyrinth. In Star Wars, Obi-Wan Kenobi gave Luke Skywalker a light saber showing him how to dodge blaster shots and face Darth Vader.
Each barrier in your path to an amazing AB testing program will have obstacles, and each obstacle will require a little magical help. This is the role of your executive champion. Your executive can impart to you special gifts, such as the Magical Elixir of Budget, the Red Tape Cleaver: Blessed Blade of Freedom, as well as the One Ring of Power to rule them all…but let’s not get carried away.
In my experience – and I’d like to hear yours – AB testing is not something that can be done “under the radar” until you have some successes. Use this post to guide you, prepare a presentation, and convince someone with pull to support your efforts.

Traffic: The Call to Adventure

It is when we finally have a steady, reliable stream of visitors that we are called to our hero’s journey. Traffic is like the taxes imposed by the Sherriff of Nottingham. The hero just can’t stand by and watch injustice.
Likewise, you must feel uncomfortable about having thousands of visitors coming to your site – most of them paid for – and then seeing 99% of them turn and leave. This is injustice at it’s most heartbreaking and a clear conversion optimization problem. Many companies will just up the Adwords budget to grow revenue. Heroes fight for the common visitor.
AB testing is a statistical approach to gathering data and making decisions. There is a minimum number of transactions you will want each month in order for your AB tests to reach statistical significance. In general, you can test an idea a month with 300 monthly transactions.
To see if you have the traffic and conversions, use our simple Conversion Upside Calculator. It will tell you how quickly you would expect a positive ROI on your AB testing program.

Analytics: Understanding the Unknown

Upon accepting the call to adventure, the hero will find herself in a strange new world. Here the rules she is so familiar with will no longer apply. She will see things differently. In the tale of The Frog Prince, a spoiled princess agrees to befriend a talking frog. In exchange the frog will retrieve her favorite golden ball from a deep pool. Upon making the deal, her world changes.
You, too, have lost your golden ball.
Most websites have Google Analytics, Adobe Analytics, Clicky, Mixpanel, or some other analytics package in place. I recommend that you not look at this as a scary forest full of strange tables, crooked graphs and unfathomable options. Instead, look at this as a constantly running focus group. It’s a collection of answers to your most pressing questions.
You should get to know your web analytics tools, but don’t get wrapped around the axel thinking you need to have fancy dashboards and weekly updates. That’s the work of admins.
Instead, sit down with a specific question you want to answer, and figure out how to drill deep into it.
“Where are visitors entering our website?”
“How long are they staying?”
“Which pages seem to cause the most trouble?”
“Are buyers of Product A acting different from buyers of Product B?”
“How important is site search to our visitors?”
This gives you amazing cosmic powers to make decisions that otherwise would have been coin tosses.

Hypothesis List: The Yellow Brick Road

One of our favorite hero stories is that of Dorothy and her journey through the Kingdom of Oz. This is a favorite because it has all of the elements of the hero’s journey. Our hero’s journey needs a path to follow. Just as Dorothy was told to follow the yellow brick road to Oz, our hypotheses are the yellow bricks in our path to AB testing success.
As you become more familiar with analytics, you will have many ideas sliding out of your head.

Ideas are like the slippery fish in an overflowing barrel.

Ideas are like the slippery fish in an overflowing barrel. You probably already have a lot of questions about how things are working on your site. You’ve probably collected dozens of ideas from well-meaning coworkers.
It can be overwhelming.
The magical helper for unbounded ideas is called the Hypothesis List. It is like Don Quixote’s Rocinante. It is your powerful steed on which you can rely to carry you through your journey to testing. By building out this Hypothesis List, you will eliminate ideas that aren’t testable, refine ideas that are, and rank them based on expected ROI.

If AB testing tells you which of your ideas are good ones, the Hypothesis List tells you which are most likely to be good ones.

If AB testing tells you which of your ideas are good ones, the Hypothesis List tells you which are most likely to be good ones.

Ideas are not Hypotheses

A Hypothesis is an “educated guess”. To be a hypothesis, an idea must be somewhat educated: informed by data, supported by experience, or born from observation. Any idea that begins with “Wouldn’t it be cool if…” is probably not a hypothesis.
When you take an idea, and try to write it into the format of a hypothesis, you quickly realize the difference. Here’s the format of a hypothesis:

If we [make a change], we expect [a desirable result] as measured by [desired outcome].

The change is a modification to copy, layout, navigation, etc. that tests a hypothesis. it is insufficient to say “Get more clicks on Add to Cart”. You must state a specific change, such as, “Increase the size of the Add to Cart button”.
The result is a desired outcome. For most tests, the desired out come is a bottom-line benefit.

“increase transactions”
“decrease abandonment”
“increase phone calls”
“increase visits to form page”

Soft results such as “increase engagement” are popular, but rarely correlate to more leads, sales or subscribers.

Soft results such as “increase engagement” are popular, but rarely correlate to more leads, sales or subscribers.
The outcome is usually the metric by which you will gauge the success of the test.

Revenue per Visitor
Lead Conversion Rate
Form Abandonment Rate

The ROI Prioritized Hyptohesis List Spreadsheet

Download your copy of the spreadsheet we use to prioritize winning test ideas.

List and categorize your ideas
Rate and rank to find “Low Hanging Fruit”
Place into buckets to identify key issues.

Many of your ideas will spawn several detailed hypotheses. Many ideas will simply die from lack of specificity.

Too Many Ideas

It is not unusual to have more ideas than you can manage. Nonetheless, it makes sense to capture them all. A simple Excel spreadsheet does the trick for collecting, sorting and ranking.

Too Few Ideas

It may be hard to believe, but you will run out of good hypotheses faster than you know. Plus, there are many hypotheses that will never be obvious to you and your company because as the old saying goes, “You can’t read the label from inside the bottle.”
This is where focus groups, online user testing sites, surveys, and feedback forms play an important role. Too many marketers use input from the qualitative sources as gospel truth. This is a mistake. You’re working toward an AB testing process that will let you test this input.

Ranking by Expected ROI

We recommend ranking your hypotheses so that the “low hanging fruit” bubbles up to the top. Our ROI Prioritized Hypothesis List ranks them based on four criteria, all ranked on a scale of one to five:

Level of Effort: How difficult is this to test and implement?
Traffic Affected: How much traffic will this hypothesis affect, and how important is that traffic?
Proof: How much evidence did we see in analytics and other tools that this hypothesis really is a problem?
Impact: Based on my experience and knowledge, how big of an impact do I really think this hypothesis can drive?

Once you’ve plugged a value in for these criteria for each hypothesis, add 2, 3 and 4 and subtract 1. That’s the weight of each hypothesis. The higher the weight, the lower the fruit hangs.

The Scarecrow: JavaScript Developer

Every hero has helpers, allies and maybe even a sidekick. You are no exception. Dorothy had the Scarecrow, the first ally she met on the banana-colored road to Oz. The Scarecrow had an amazing brain, but didn’t really know it.
At the end of your journey, you are going to have complete control of your website. You won’t need to put requests into IT. You will have the power to change your website for every visitor because the change happens in the visitor’s browser. Each visitor gets the same website from the server. It’s the JavaScript that instantly transforms what they see into a test treatment, something different and measureable.
Your JavaScript developer will be your Scarecrow. This person must be comfortable with JavaScript, HTML, CSS, the browser DOM and cross-browser issues. This person will enable you to make changes that put your hypotheses to the test.

The Tin Man: Designer

Dorothy was also befriended by the man who didn’t know he had a heart on the way to Oz.
You’ll want a designer that isn’t interested in redesigning your pages. All you need is a designer who can change portions of a page. It may be a design change as simple as a new image, or as complex as a new page layout.
Avoid designers who like to add to their egos to your design.

The Lion: Copywriter

I made the copywriter the lion in this journey because writing and defending bold headlines and copy takes courage. Like Dorothy’s friend, the cowardly lion, most copywriters have been taught to write business speak to online visitors. They have PTSD from having their copy bled on by executives. This won’t work in testing.
Your copywriter needs to be able to writer for personas. He must be brave enough to create “corner copy”, or headlines that test the extremes of emotion, logic, spontaneity and deliberateness.
One of our biggest winning headlines was “Are you ready to stop lying? We can help.” It took bravery to write and defend this headline that delivered a 43% increase in leads for an addiction treatment center.

Tests and Trials: QA Methodology

Every hero is tested. Villains and the universe put obstacles in place to test the hero’s resolve. You, too, will be tested when you realize that you don’t have just one website. You have ten or twenty. Or thirty.
Your website renders differently in each browser. Safari looks different from Chrome. Internet Explorer seems to be dancing to it’s own HTML tune.
Your website renders differently on smaller screens. Smartphones, phablets, tablets and 4K monitors squeeze and stretch elements until your HTML and CSS snap.
Your website renders differently based on your connection. Fast Wi-Fi is often not available to your mobile visitors. Your development team is probably testing on fast Wi-Fi.
The permutations of these issues means that you can’t design for one site, even if you have responsive design.
Add to this JavaScript that moves things around on each of these different websites, and you have the potential to bring some of your visitors to an unusable website.
Quality Assurance, or QA is your defense against your new-found responsibility.
At Conversion Sciences, we go to the extreme of purchasing devices and computers that let us test a the most common browsers, screen sizes and operating systems.

Conversion Sciences’ QA station

There are a number of sites that will provide simulators of a variety of devices, browsers and OSes. These have names like BrowserStack, Sauce Labs, and Litmus.
How do you know which of these you should be QAing on? Your analytics database, of course. Look it up.

Magical Help: AB Testing Tools

As we said above, your executive champion can bestow on you supernatural aids to help you in your journey. This is not the only source of magical helpers. Aladdin found the magic lamp in a Cave of Wonders.
Your magic lamp and “genie” are your AB testing tool. These marvelous tools make our agency possible. AB Testing tools have magical powers.

They inject JavaScript into our visitors’ browsers, allowing us to change our website without changing the backend.
They split traffic for us, letting us isolate individual segments of visitors to test.
They track revenue, leads and subscribers for us, so we know if our changes really generate more business for us.
They provide the statistical analysis that tells us when we can declare a winner in our tests.
They provide lovely graphs and reports.

The market leaders currently are Optimizely, Adobe Target, and Visual Website Optimizer (VWO). We have also used Convert.com, Maxymiser, Monetate and Marketizator to test websites.
We call these tools the “supreme court” of analytics. They control many of the variables that pollute data, and give us confidence that our changes will deliver more revenue, leads and subscribers to our clients.

The Belly of the Whale: Patience

The story of “Jonah and the Whale” appears in the Bible and Quran. In short, God asks Jonah to go to the city of Ninevah. Jonah hems and haws. So God arranges for Jonah to be swallowed by a big fish. After three days of apologizing, the whale spits Jonah out, and he begins his God-ordained journey.
It turns out that the belly of the whale is a theme in many hero myths. Like them, you will find yourself waiting and wondering as your tests slowly gather data. Some will go more slowly than others. Pressures from executives will mount. You must persevere.

Do not rush to make decisions, even if it looks like your test is going to deliver you a winner. Let the process run its course.

Do not rush to make decisions, even if it looks like your test is going to deliver you a winner. Let the process run its course.

The Reward: Revenue, Subscribers and Leads

In the end, you will have winning hypotheses and losing hypotheses. Because you won’t launch the losers and will push live the winners, you’ll begin to collect your boon, your reward, your Benjamins.
Be sure to show off a bit. Toot your own horn. Heroes come home to fanfare and celebration. Let your organization know what your AB testing program is doing for them and revel in the glow of success.

Master of Two Worlds: AB Testing Program

Your journey has taken you from magical helpers to reward. Along the way you entered a new world, learned its rules, overcame tests and trials, and used magic to win the day.
You are now Master of Two Worlds: Master of the old world of pray marketing, and Master of the new world of data-driven marketing. This will be a career builder and a business boon.
This is everything you need to build your AB testing program. Conversion Sciences offers a ready-made turnkey conversion optimization program. Ask for a free strategy consultation an carve months off of your hero’s journey.

June 7, 2016/by Brian Massey

AB Testing JavaScript: Great Power, Great Problems

CRO Tests | Multivariate | AB Testing

The AB Testing JavaScript that powers tests is powerful, but can lead to many unintended consequences.

Conversion Sciences offers a pretty amazing ability to our clients: A completely “turnkey” conversion optimization testing service. By “turnkey” we mean that our clients don’t need to do anything to their website in order for us to analyze the site, design creative, develop code, QA, test, review and route traffic to winners.
Why is this? Don’t we need new pages designed? Interactions with IT? Release schedules? Sprints?
The reason we have this “phenomenal cosmic power” is that our AB testing tools take control of each visitor’s browser. The changes are made on the visitors’ devices, so the website doesn’t have to be modified.
Well, not until we’ve verified that a change brings in big money.
While this makes us feel all high and mighty, it comes with a big drawback.

The Magic and Mania of JavaScript

All of the major browsers have a scripting engine built into them. It allows programs to be run inside the browser. The programming language used is called JavaScript. It’s JavaScript that makes websites interactive. Website developers use JavaScript to make text accordion when you click a heading. It is used to rotate images in a carousel.
Unfortunately, developers use JavaScript to do silly things, like parallax animations.

This unnecessary “parallax” motion may be reducing the conversion rates for this site.

And then there’s this.

Don’t use JavaScript animations just because you can.

Yes, JavaScript gives us the power to make our websites harder to use.

JavaScript is Used in AB Testing Software

Our developers use JavaScript to modify a website when you visit it. The AB testing software “injects” our JavaScript into the browser when the page is loaded.

First the web page is loaded as is from the webserver. Some visitors will see only this default version of the website.

For some visitors, our JavaScript is then injected and executed. The AB Testing software determines who sees the default web page and who will see a variation of the page.

Phenomenal Cosmic AB Testing Power

We can change almost anything about a page using AB Testing JavaScript.

We change the headline to something different.

We change the order of a rotating carousel, or slider.

We hide elements…

AB Testing flicker can be cause by simply removing elements.

and insert them as well.

We insert video.

We completely change the look of a website.

We completely change the look and feel of this site. The test showed the white site performed the same as the brown site.

AB Testing JavaScript is a Step Toward Personalization

If we wanted, we could deliver a different website to every visitor who arrives. If you’re thinking “personalization” at this point, then you have an idea of where our industry is heading.

AB testing produces the data that makes personalization truly work.

Here are my instagraphic notes from a presentation by Chris Gibbins at Conversion Conference 2016 on using AB testing and personalization.

Instagraphic shows why and how to use AB Testing for personalization.

AB Testing Flicker, Flash, Blink and Ghosting

Unfortunately, JavaScript can introduce an error into our AB tests. Because we always load the website “as it is” on the server first, there is a possibility that the visitor will see the original for a fraction of a second before our changes get executed.
This has been called a “flash”, “flicker”, and a “blink”. It can have a significant effect on test results.

With AB Testing JavaScript, Flash is not cool.

The problem with this AB testing JavaScript flicker is that it won’t be there if a change is made permanent on the site. Our tests may say the new change generated more revenue, and we’ll change the website to make that change manifest. But there will be no flicker. This means there is another variable in our test.
Was it the new headline we tested or was it the flicker that made more people buy?

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

43 Pages with Examples
Assumptive Phrasing
"We" vs. "You"
Pattern Interrupts
The Power of Three

"*" indicates required fields

The Human Eye is Drawn to Motion

Our eyes and brains have evolved to make motion important. When something moves, our eyes and brains work hard to figure out if the thing moving is edible, if it will eat us, or if we can mate with it. Technology has moved faster than evolution. Even though we’re looking at a website, where there is no immediate source of food, fear or fornication, our lizard brains continue to favor motion.

Imagine this scenario. We are testing the text on a button. That means that for some of our visitors, the button will change. Others will see the default button text. Then we’ll see who buys the most.

The changing element on this page draws your eye, and can make the page impossible to focus on.

If there is a flicker, flash or blink when the text changes, the button will be immediately more visible to those visitors who see it. More of them will see the call to action. This may make more of them consider buying than those who simply scrolled past. If this new treatment generates more revenue, we are left with the question, “Was it the text or was it the motion that created the lift in sales?”

We won’t know until we push the new text onto the server and do a post-rollout analysis to see the real lift. At this point, we may find that the text didn’t increase purchases. It’s a real bitch.

How many buttons have been change because of flicker?

AB Testing Software Tries to Eliminate Blinking

The AB testing software that we use works to eliminate this blinking and flashing. The issue is so important that Convert Insights has patented their method for eliminating “blink”.

AB testing software like Optimizely, Visual Website Optimizer, Adobe Test, and Marketizator, load asynchronously, meaning that they load our changes as the page loads. This makes it easier for changes to be made before the visitor sees the page.

How Convert Insights Eliminates “Blink”

“The first thing this snippet does is hide the body element for max. 1.2 seconds (this can be set), this prevents blinking of elements already visual on the site that load under 1.2 seconds (we have yet to get a client that loads faster than this). During the 1.2 seconds, the SmartInsert(R) technology search and replaces DOM-elements every couple of milliseconds and loops through the entire available DOM-elements in the browser of the client. After all elements are replaced the body hidden attribute is set to display the site again either at 1.2 seconds or when the signal is given that all elements have been replaced (or DOM-ready).
Everybody can see how this technology works by loading our Chrome Extension.”
— Dennis van der Heijden, Convert.com

Eliminating Flash Flicker and Blink in AB Tests

In addition to this, our developers can do things that reduce and eliminate flicker and blink. Every test you do has different issues, and so a variety of tactics can be used to address them.

Avoid Google Tag Manager

Don’t use a Tag Manager like Google Tag Manager to serve your AB testing JavaScript software tags. Add them to the page manually. Tag managers can counteract the asynchronous loading of the tool and delay when changes can be made.

Make Changes with CSS

If the change can be made with the cascading style sheets (CSS), we favor making changes with CSS over using JavaScript. CSS can make changes to an element – like an image or font – that can be applied before an element displays.

Modal Dialogs and Overlays

Modal dialogs usually don’t display until the visitor takes an action. They can be edited before they are shown without flashing.

Use a Timer for DOM Changes

All of the images, headings, copy, widgets, and forms are stored in the browser in a big database called the DOM (Document Object Model). When JavaScript makes changes to the DOM, the content on the page changes. The DOM is slow-loading, as you can imagine.

Our developers will set a timer in JavaScript to check for when a changing element is loaded. By watching for our element to be loaded, we can execute a change it before the DOM – and the page – is loaded.

For the AB Testing Software to Load Our Changes Immediately

The AB testing software provides a synchronous loading mode. Optimizely and VWO use different approaches for this.

Rethink the Test

Sometimes, we have to go back to the drawing board and design a test that doesn’t cause flash-inducing changes. We will refactor a test to eliminate items that cause a flash or flicker.

Delay Display of the Content

We can delay the display of the content until the DOM is loaded and all elements have been changed. This causes a different kind of issue, however. The visitor sees a blank page or blank section for longer if they get the page with the change.

In one case we added a spinning wheel to indicate that the page is loading.

Insert Placeholders Using Fast CSS

When inserting elements, we’ll use CSS to insert a placeholder, then use JavaScript to modify the element after the page has loaded. This reduces redrawing of the page when elements are inserted.

We created a blank box in CSS to minimize AB Testing Flash on this mobile website.

Optimizing for Mobile

Mobile pages load more slowly. This is because visitors don’t always have a high-quality connection when visiting from the festival, or from inside the walls of a bank while standing in line. For mobile devices, flash can be an even bigger issues.

Fortunately, the tactics we use on the desktop also work on mobile websites. But don’t forget to QA your test treatments on 3G and 4G connections. You may find flicker and blink on mobile devices that didn’t appear on your desktop.

Great JavaScript Developers are a Must

We spend a considerable amount of our time making sure our JavaScript and CSS changes are like the “native” implementation would look. It’s one of the things that makes testing hard. Our team has the experience to ensure your tests aren’t falling victim to flicker, flash, blink or ghosting. Best in class Developers are one of the things that help to make use the most effective Conversion Optimization Agency on the planet!

If you’d like our team to take over the effort of developing your AB tests, contact us for a free consultation and an overview of our process.

May 26, 2016/by Brian Massey

4 Types of Useful AB Testing Tools You May Not Realize You Have

CRO Tests | Multivariate | AB Testing

Talking about useful AB testing tools to the readers of this blog may be like preaching to the choir. But if you are new to this blog, or new to conversion optimization in general, you may be wondering which AB testing tools you can start using without making a huge investment. Fortunately, there are some AB testing tools out there that are either free or won’t cost you any additional money because you already have them – you just don’t know it yet.
In this post, we’re going to look at AB testing tools that you may have had all along and how to use them to optimize different aspects of your marketing strategy for conversions.

Useful Website AB Testing Tools

Since most people will want to do AB testing on their website, we’ll start with the tools you can use here. Did you know that if you have fewer than 50,000 unique visitors per month, you can use tools like Optimizely for simple AB testing for free? It’s a really simple tool to use. You just sign up for your free account and start up a website project.

Create a project with Optimizely for free when you have fewer than 50,000 visitors a month

Once you’ve entered your URL, you will be taken to a screen where you can immediately start creating a variation to test on your website.

Create a variation for your AB test

Once you’ve created your variation, you click the Start Experiment button and get the code you need to add to your site.

Start your Optimizely experiment

You will set up a goal so you know which variation leads to the most conversions.

Create a goal for your AB testing experiment

And then sit back and wait for visitors to come to your website to determine which variation gets the most conversions!
If you’re stuck for ideas on what to test on your landing page, you can try the common elements – headlines, subheads, images, calls to action, etc. – as well as some creative options listed in our landing page testing ideas post.
If you have more than 50,000 visitors each month, or would prefer to not add another tool to your toolkit, you can also look into Google Analytics Content Experiments. This allows you to conduct testing with your Google Analytics.
To start, you go to Behavior > Experiments for your website and click the Create Experiment button. Then you define the experiment you want to perform, starting with the goal of your test. You will use your Google Analytics goals to ultimately determine which variation of your AB test is the winner.

Create an experiment in Google Analytics

The key difference between Google Content Experiments and your average AB testing tool is that you have to create an additional page on your website that has the variation, whereas most AB testing tools (like Optimizely) will let you “edit” your page in their editor. So depending on what you want to change, it may be an easy or difficult process to create that second page.

Setting up variants in Google Analytics may require more steps than using a traditional AB testing tool like Optimizely.

Next, you will receive the code you need to insert on your website to start your experiment.

You code for your AB test in Google Analytics

Finally, you will confirm that the code has been installed and you will start your experiment. Once your experiment is completed, Google Analytics will declare a definitive winner.

Alternatives to AB Testing Your Own Website

An alternative to doing AB testing on your own website is to monitor the tests of others. There’s a free way to do it and a paid way. First, you can try to find your competitor’s website history in the Internet Archives. The downside to the free is that you’ll have a lot of clicking to do.

The Internet Archive brings you the tool Wayback Machine which lets you see how a website appeared on a particular date.

The other option I mentioned in my landing page testing ideas post, Rival IQ, allows you to see your competitor’s website history in a much easier to digest format.

RivalIQ is a paid tool for viewing website histories.

There’s a good chance that if you look through your competitors designs over the last couple of years, you’ll see subtle changes to headlines, images, colors, etc. that will relate to some AB testing. So instead of testing on your own, you can learn from their tests and pave your own unique way from there.

Email Marketing AB Testing

If you are running email marketing campaigns, chances are you are using a popular email marketing software platform that likely has an AB testing component built in. MailChimp, for example, allows you to select an AB testing campaign and then allows you to test four aspects of your email campaign: the subject line (highly recommended), the from name, the content, and the send time.

AB testing options in MailChimp

You can choose a certain percentage of your recipients to test with and you can choose click rate, open rate, revenue, or other goals to judge the results of your testing. For example, if you chose to test subject lines, you would simply enter two subject lines for your recipients instead of one.

AB testing email subject lines in MailChimp

Or, if you were going to test two different types of newsletter content (such as a text-only email versus an HTML newsletter), you would get two email templates to send to your recipients.

AB test your email content

Most email marketing software offers AB testing. At the bare minimum, you can at least test your subject lines. Some go further with the from name testing, email content testing, send time testing, and other forms of AB testing.
But considering that your subject line is the make or break point of whether someone opens your emails, it’s safe to say that so long as you have the option of testing that, you are good!

Alternatives to AB Testing Your Own Email

There is a simple and free way to monitor your competition’s email and potentially see what headlines are working for them – just sign up for their emails. And be sure to at least open them. If you just ignore them, some will automate you out of their main line of emails. And that might mean you’ll miss out on some good subject lines!
Bonus tip: if your competition is using email marketing software like Infusionsoft, ActiveCampaign, or others that allow automations, you should open the emails and click on the links on occasion. You may get to see one of their automation funnels in action too!

Blog Content AB Testing

Similar to email AB testing, blog content performance can rely heavily on one specific element: the title. If you choose a great blog title, people will click through and read your post from your blog’s homepage, search engines, social networks, and other sources. If you choose a bad blog title, then you may not get an clicks or readers at all.
That’s why AB testing your blog post titles can be a crucial key to the success of your content marketing strategy. If you have WordPress, Nelio A/B Testing is a tool you can use to do just that.
While it’s not free, it starts at $29 a month for websites with 5,000 views per month. And it will allow you to test crucial elements of your blog, beyond just the headlines of your blog posts.

You can use Nelio A/B Testing to test WordPress blog content

For serious publishers, WordPress website owners, and WooCommerce website site owners, this can be a powerful AB testing tool that can help you test a variety of things that other testing tools simply can’t.

Alternatives to AB Testing Your Own Blog Headlines

Going back specifically to blog headlines, if you don’t want to test your own, there are ways of finding out the best headlines for a specific topic. The free way would be to use BuzzSumo – even without an account, you can usually get the top five to ten headlines about a specific topic based on social sharing.

Find the top headlines for a topic using BuzzSumo

If you don’t mind paying, a similarly priced tool that offers even more information that you can try or compare to BuzzSumo is Impactana. Both start at $99 per month, but Impactana goes a step further by allowing you to see headlines that are not popular based on social shares alone, but also based on views, backlinks, comments, and other metrics (based on the type of content).

Impactana uses more metrics than BuzzSumo to show you the top headlines for topics

This can give you a strong idea of what headlines and content generate the most social buzz, search authority, traffic, and audience engagement.

Social Media Ad Campaign AB Testing

While social media advertising is not free, AB testing for some social media ad platforms is because it’s built right in. Take Facebook, for example. You can create an Ad Campaign, an Ad Set that is targeted to a specific audience through specific placements, and multiple Ads under that set that help you test variations so you can determine which one drives the most conversions.
Here’s how the process looks in the standard Facebook Ads Manager. First, you will start by choosing your ad objective.

Choosing your ad objective in Facebook

Then you will name your Ad Campaign.

Naming your Facebook ad campaign

Next, you will define your Ad Set by choosing your target audience, ad placements (the desktop news feed, the mobile news feed, Instagram, etc.), and setting your budget.

Defining your Facebook Ad Set

Before you continue, you can save the name of your Ad Set.

Your budget and schedule for your Facebook ad

Finally, you will configure your first Ad.

Configuring your Facebook ad

Once you’re finished with your first ad creative, you will place your order. Once you do, that ad will go into review and you will get the option to create a similar ad.

After finishing your order for your Facebook ad, you’re ready to get set-up for your variation

This will allow you to create another Ad under the same Ad Campaign and Ad Set. You will get the option to modify the Ad Set if needed.

Modifying your Facebook ad set

Otherwise, you can click Continue to create your next Ad variation. This will bring up the same Ad you created before so you can create your variation by changing one specific element, such as the image, originating page, the headline, the text, the call to action, the news feed description, or the display link.

Change the element you want to test in your Facebook ad AB test

Once you have made your variation and placed your order, you will again get the option to create a similar ad or to continue to your Ads Manager. You can also decide to add more variations from the Ads Manager by clicking on the Create Similar Ad button.

You can add more variations by clicking Create Similar Ad

The downside, as you can see above, is that you can’t name the individual ad variations. Therefore, unless you’ve changed the images between them, they all look the same in the Ad Manager view. Hence, to know which variation in terms of originating page, the headline, the text, the call to action, the news feed description, or the display link is working, you will have to click through to the winning variation and view the post to learn from it.

It’s easy to toggle off an ad if it’s not working out

The upside, however, is you can easily toggle off the losing variation of your ad based on its performance.
But overall, this is a great way to use AB testing in your Facebook Ad Campaigns. And it’s the simplest way as it doesn’t require you to use Power Editor, although if you are more comfortable in Power Editor, it can be done there as well.
If you use LinkedIn Ads, they also offer an option for creating variations with their text and image ads. Fortunately, their variations editor is even simpler. Start by going to LinkedIn Ads Manager, select your LinkedIn advertising account, create a new Campaign, and then select Text Ads.

How to begin a LinkedIn ad campaign

Start by giving your campaign a name.

Name your LinkedIn ad campaign

Next, you will create your first ad.

Creating your first ad

When you save your first ad, you will get the option to continue or create up to 100 variations of your ad.

After you create your ad, LinkedIn makes it easy to create your variations

When you click to create another ad, you will be able to create an entirely new ad from scratch to test different URLs, headlines, descriptions, and images.

Building your LinkedIn ad variations

Once you are finished creating your variations, you will continue through the campaign creation process to select your target ad audience and set your ad campaign budget.

Select your targeted audience and your budget

Once you have finished creating your campaign, you will get a clear view in your ads dashboard of how each of your ad variations are performing. This will allow you to learn what works and what doesn’t quickly, as well as allow you to toggle the losing variations off.

Your dashboard will tell you how your ad variations are performing

Alternatives to Doing Your Own Social Media Ad AB Testing
There are two free alternatives when researching paid advertisements. The first is Moat. Moat allows you to look at other companies display banner ads. While this isn’t specific to social media ads, it can help you learn about the images and ad copy that big brands use to drive paid traffic to their websites.

Use Moat to discover what your competitors are doing with their ad testing

If you notice particular imagery, copy, calls to action, button colors, or other elements have been used over and over again, you can assume that said elements have been doing well, considering you can almost guarantee big brands are testing the elements that they are paying for.
The second resource you can tap into that is specific to Facebook is the AdSpresso Facebook Ads Gallery. Simply sign up for their newsletter and start searching for brands that advertise on Facebook. Not only will you get to see the variations of ads they have used over time, but you can click on their ads to get some pretty good details about the ads.

AdSpresso shares a lot of valuable information about Facebook ads

Between these two resources, you should learn a lot about how to create a successful ad campaign on social media and beyond. And they’re both better options than sitting around and refreshing your Facebook or LinkedIn news feed, hoping to see some ads from your competitors.

In Conclusion

As you can see, between free and premium tools, there are various ways to A/B test many aspects of your online marketing beyond just your landing pages. Be sure to look at the different aspects of your online marketing strategy and think about the ways you should be testing it to improve your results today!

About the Author

Kristi Hines is a freelance writer, blogger, and social media enthusiast. You can follow her latest tweets about business and marketing @kikolani or posts on Facebook @kristihinespage to stay informed.

May 20, 2016/by Kristi Hines

The AB Testing Process that Empowers Marketers

CRO Tests | Multivariate | AB Testing

At Conversion Sciences, conversion optimization is so important that we think every site should benefit from it. We take every chance to teach businesses about it. The AB testing process is an important part of conversion optimization and is within reach of almost any business that prizes data-driven marketing.

Conversion rate optimization (CRO) is a systematic process for increasing the rate at which website visitors “convert” into leads and customers.

When visitors arrive at your site, you want them to take a certain action. You might want them to subscribe to your mailing list, purchase a product, call your company, make a donation, fill out your contact form, or any number of things. CRO seeks to maximize the percentage of visitors who perform this desired action.

And as traffic becomes more and more expensive to acquire, CRO continues to become a bigger and bigger deal for online businesses.

When to Use Conversion Optimization

Conversion optimization has become an important part of digital marketing because available tools are becoming easy to use and inexpensive. Businesses generate more sales from the same traffic.

Businesses will turn to conversion optimization when:

Search ads, like Adwords get too expensive.
Their organic search traffic isn’t growing fast enough.
They aren’t getting enough revenue from their email list.
They want to compete with bigger companies on the Web.

Conversion optimization gives the business more control over its own destiny, increasing revenue and delighting more customers. AB testing is a powerful tool in the conversion optimization game.

Understanding Conversion Optimization

At a high level, a website’s basic revenue model looks like this:

Traffic x Conversion Rate = Revenue

Let’s say you are getting 100,000 visitors each month and converting 3% of them into customers. In order to double your revenue, you can either (A) double traffic by getting 100,000 extra visitors each month, or (B) increase your conversion rate from 3% to 6%.

As you can imagine, it is usually much cheaper to fix a few things on your site and increase the conversion rate than to increase traffic by 100,000 people. And this is based on a simple three variable formula.

In reality, many websites and online enterprises consist of numerous steps in a complex chain of conversion funnels.

Conversion funnels are complex, and it's easy to lose someone along the way without CRO and AB testing.

Conversion funnels can be complex, and it’s easy to lose someone along the way without CRO and AB testing. Image credit: Digital Marketer

Low conversion rates at any point in this lengthy funnel can gut revenue totals, and consequently, optimizing the conversion rate even slightly throughout the multiple stages of this funnel can result in a massive increase in overall revenue.

The AB Testing Process

The best data-driven marketers take a systematic approach to optimize a website’s overall conversion rate. And while that approach is fairly complex, the core process includes the following:

Data gathering & analysis
Hypothesizing & Prioritizing
Design & Run AB Tests
Interpretation & implementation

To summarize, you begin by gathering intelligence on the your target audience. Next, you predict a series of website changes that will improve the overall conversion rate and then test those changes with a live audience. You run tests to confirm or refute your predicions. Finally, you implement changes that improve the conversion rate and discard those that don’t.

1. Data gathering & Analysis

The CRO process begins with gathering and analyzing both quantitative and qualitative data in order to achieve a well-rounded understanding of the website’s target market and how they are engaging with the site.

Data software, surveys, and usability tests are often used to collect and analyze this data.

2. Hypothesizing & Prioritizing

Once data has been collected, it’s time to hypothesize a series of site changes that will potentially increase conversions. Each idea for increasing conversion rate is called a hypothesis, or “educated guess”. These predictions are usually based off the data collected, “best practices”, and the personal experience of the data-driven marketer. The hypotheses you focus on will be based on your core testing strategy.

Changes are then made and compared against the original site in front of a live audience using a AB testing.
Each hypothesis should have the form:

If we describe change , we expect more visitors to describe desired outcome
as measured by metric .

For example:

If we add “Free shipping on all orders” to our product pages, we expect more visitors to purchase as measured by revenue per visit.
If we include the phone number in our headline, we expect more visitors to call as measured by web-based phone call rate.

Taking the time to write out each your hypotheses ensures:

That you are testing something very specific.
That you are testing something that results in a desired outcome.
That you can measure the results.

Conversion Sciences enters our hypotheses into a spreadsheet and rates each on a scale of 1 to 5 for four categories:

Based on my experience, how big of an impact do I expect this hypothesis to have? (1-5 with 5 being a big impact)
How much traffic sees the page on which this hypothesis applies? (1-5 with 5 being a lot)
How much evidence do I have that this hypothesis is really a problem? (1-5 with 5 being most)
How hard is the test to implement? (1-5 with 1 being best)

Add 1, 2 and 3, then subtract 4 to get your hyopthesis weight. Do this for each test to get a ranking and sort the spreadsheet by weight. Those hypotheses with the highest weighting will jump to the top. These are your “low-hanging fruit”, the first things you should test.

3. Designing & Running AB Tests

This is where the new tools come into play. AB testing tools offer ways to change your website for some visitors, while leaving it the same for others. The tools allow you to do this without changing your website because the changes are made in the visitors’ browsers.

One visitor will get a page (A) as it is, then the next will get a version of the page (B) with your change. The tools manage this so that about the same number of visitors see each page. These AB testing tools then report on which version generated the most revenue and tell you how much more revenue you would expect to get.

If the original generates more revenue — we call it the Control — you can be assured your change would have hurt your site. If the modified version generates more revenue — we call it a Treatment — then you’ve found an improvement and can code it into the site.

AB Testing tools have a learning-curve. Most offer “WYSIWYG” interfaces for changing elements. Some tests will require that you have a resource familiar with Javascript, HTML and CSS.

4. Interpretation & Implementation

After running a series of AB tests, results are analyzed and interpreted and additional tests may be run. The goal is to identify a slate of changes that yield statistically significant improvements in the site’s overall conversion rate.

Verified improvements are implemented as permanent changes to the website, and then new hypotheses may be made and tested until the target conversion rate is achieved.

What You Can Do With an AB Test

When evaluating page elements to test and improve, CRO specialists typically start with “best practices”. Best practices are techniques that tend to work for many websites. Testing page elements based on best practices will often improve the site’s conversion rate immediately, and they provide a good baseline from which the data-driven marketer can plan and implement more tailored tests.

It’s important to note here that “best practices” do not work for every site. Especially on your mobile site, there really aren’t any best practices. This is why AB testing is so important.

That said, it’s good to have a basic understanding of best practices when attempting to optimize conversions on a website.

1. Develop an Effective Value Proposition

Your website must convey a value proposition that gives the visitor a reason to stay and explore. The value proposition is constructed out of copy and images.

The value proposition doesn’t have to be unique, but it must describe the reason you occupy space on the Web. It should include who your offering is targeted at and why they should care about it.
Your value proposition may also include pricing, delivery, return policy, and what make you unique.

Each of your visitors will come in one of four modes: Competitive, Methodical, Humanist, or Spontaneous.

COMPETITIVE visitors are looking for information that will make them better, smarter or more cutting-edge. Use benefit statements and payoffs in your headings to draw them into your content.
METHODICALS like data and details. Include specifics and proof in your writing to connect with them.
HUMANISTS want information that supports their relationships. They will relate to your writing if you share the human element in your topic.
SPONTANEOUS visitors are the least patient. They need to know what’s in it for them and may not read your entire story. Provide short headings for them to scan so that they can get to the points that are important to them.

Your goal is to write copy directed at whichever of these groups visit your site.

In addition to understanding what motivates your audience to buy, it’s also important to understand what stands in the way of them making that decision.

Most buyers can be broken down into two categories based on two different fears:

Transactional buyers
Relational buyers

Transactional buyers are competitive bargain hunters whose greatest fear is paying a dollar more than they need to. They aren’t looking for “cheap”. They are looking for the greatest possible value they can find for the lowest possible price.

In order to appeal to transactional buyers, your copy should be focused features, price, and savings.

Relational buyers, on the other hand, are focused entirely on quality. Their greatest fear is buying the wrong thing, and they are more than happy to seek out expert help and pay a premium in order to assure they receive a quality product.

In order to appeal to relational buyers, your copy should be benefits focused with educational content, copious social proof, ratings, and reviews to demonstrate that selecting your product is a guaranteed win.

Things to test:

The language on links and buttons.
Headlines and subheadings.
Wording of discounts and specials.
Description of return policy and shipping policy.
Adding bullets and highlights to copy.
Change copy that talks about your company into visitor-focused copy.

Remember that nobody cares about your business or products until they’ve found what they are looking for. People only care about how your business can solve THEIR problems. Remember to keep the copy and messaging consistently focused on the customer on every platform and at every point of interaction. Personal stories and intros have their place and can be quite effective, but again, only when the context is customer benefit.

For further reading, check out these great value proposition examples and the case studies on their implementation.

2. Design to Help Your Visitors Choose What’s Next

Design is important to your conversion rate but not for the reasons most people think. Your site’s design should be focused two things:

Highlighting your persuasive copywriting
Helping the visitor choose to act or to take the next step in their journey

Visitors should have an idea of what your landing page is all about with five seconds of arriving. They should then be taken through a streamlined journey rather than needing to browse around and find their own way.

A simpler, more intuitive, and more straightforward site design a great place to start.
Things to test:

Making links and buttons with calls-to-action stand out.
Move important information, such as free shipping offers “above the fold.”
Swapping columns.
Increasing the size of images.
Placing security badges near the “Add to Cart” button.
Increasing the font-size of important information like price and stock.

These are some places to start.

3. Focus on Entry Pages

A good conversion funnel isn’t just a webpage. It’s a combination of ads, search results, blog posts, email marketing, social media, webinars, and much more. Each of your campaigns will bring visitors to your site on different pages. Start on these pages and find look for ways to help visitors choose the next step.

For ecommerce sites, product pages are often the entry page for search traffic.

For many sites, the most common entry pages will be:

The Home Page
Category and Product Pages for ecommerce sites
Post pages for blogs
Signup pages for webinars, reports and other offers

The Behavior > Site Content > Landing Pages report in Google Analytics will tell you which pages are your most frequently visited entry pages.

Google Analytics offers a list of entry pages, which they call "Landing Pages"

Google Analytics offers a list of your entry pages, which they call “Landing Pages”

If you can get more visitors into your site from these entry pages, reducing your bounce rate, you will have more opportunities to win prospects and customers.

Conclusion

AB testing is a tool that is within reach of more and more marketers. It is powerful because it

Ensures you aren’t making changes that hurt your online business.
Helps you understand what your visitors are really looking for.
Disciplines you to make smaller stepwise changes to your site.

To continue your journey into the world of CRO, check out Conversion Sciences Free CRO Training.

If you have any questions or if you noticed I left out some key info, don’t hesitate to let me know in the comments. And of course, don’t forget to share this post with any of your colleagues who could benefit from an introduction to AB testing.

May 12, 2016/by Jacob McMillen

4 Mobile AB Testing Ideas that Worked for Our Clients

CRO Tests | Multivariate | AB Testing

Joel Harvey, Conversion Scientist™

This post is excerpted from Designing from the Mobile Web 2.0 by Joel Harvey. Joel discusses what he’s been learning through the tests that Conversion Sciences has done for the mobile web. You can begin to take these as new hypotheses in your business so that your mobile devices and your mobile traffic is converting at higher and higher levels.
I’m sure mobile’s on the top of your mind.
We have tested hundreds of mobile hypotheses over the last couple of years and we’ve learned a lot. There’s still a lot that we don’t know.
We’re going to share some of the key things that we’ve learned along the way with you and show you what you need to be focusing on to start driving wins in your mobile traffic.
[mobile-web-20]

What Mobile AB Tests Can You Try Right Now?

What are some of the things you can AB test right now to start driving higher mobile conversions?
Here are the hypotheses we’ve tested and things that we’ve seen give us increases across dozens of mobile websites. Some of them are very site specific like the offer link and the copy. This is one thing that yields big results in any business and there’s no one rule of thumb that we can share.
However, there are some things you can do with layout and with execution that, with the right words and the right elements, work across almost many mobile sites.

Sticky Headers and Footers
Persistent Calls to Action
Footer Content
Optimizing for the Right Goals

1. Sticky Headers and Footers

For anybody that doesn’t know, what we mean by a “sticky” header or footer, we mean a “bar” of content that persists at the top of the screen. It sticks even when the visitor scrolls.
By locking this on the screen as people scroll we always keep in their view. Just making the existing header sticky, we saw a 9% increase in phone calls for this site. We’ve seen this on ecommerce sites, increasing form fill completion and purchases.
For a follow-up test, we simply added a call to action. The text delivered a relatively large increase in phone calls on top of the increase from adding the sticky header.
There are a number of things you can put into a header.

Calls to action
Search boxes
Add-to-cart button
A menu icon
The company logo
Shipping policy

Over time this has continued to evolve and conversions have increased.
In our experience, sticky footers, or bars that persist across the bottom of the screen, may work better for your audience. We recommend that you try them both.

2. Persistent Calls to Action

Persistent calls to action or parachutes are offers that remain on the screen as the visitor scrolls. These are usually found as the footer.
It’s very similar to a sticky header.

The site at left enjoyed a 20% increase in registrations from a sticky header. The site at right saw a 45% from top and bottom stickies.

Since we found that these parachutes work well as top “stickies” and bottom “stickies, we wondered if we could do both. Our original thought was that it took up too much screen real estate, that it was too much to keep in front of the visitor. However, we found that in most cases you can have both and they’re very complimentary to one another.
We recommend testing to find the right call to action first, and then testing a persistent, or “sticky” call to action.
So why do we call it a parachute?
We call it a parachute because we know that mobile visitors will scroll much farther down a page and much faster than desktop visitors. This is an interesting fact: Mobile visitors are more likely to see all of the content on your page than a desktop user, especially if you have a lot of content and it’s a long page. You can see this in your own scroll maps and in session recordings.
The problem is that they do it fast and they sometimes get lost. Having a parachute someplace for the them to parachute out of this purgatory they’ve gotten themselves into is proving to be very helpful.

Check Your Tel-Links

The Safari mobile browser doesn’t necessarily do a fantastic job of identifying phone numbers and turning them into click-to-call, or tel-links.
Anywhere where you have a phone number on your site, make sure you’ve explicitly written the tel-link around that phone number.
In the example above, we found this site didn’t have click-to-call. We tested adding the tel-link functionality explicitly, and we saw a 20% increase just in clicks to call. It makes sense. You’re on your mobile device. You don’t have to put it down, write the number down, and then type it in.
There are phones built into these mobile devices, aren’t there?

3. Footer Content

This issue dovetails in with what we were just saying about how far a mobile visitor will scroll.
Mobile visitors are much more likely to see the footer of your site. They bounce on the bottom after fast scrolling. On the desktop the footer of the site is this graveyard of crap that we throw into the bottom. It’s the detritus of the site. It usually includes the privacy policy and some legalese, things that are not really going to compel anybody to take action.
Yet, the visitors that scroll to the bottom of your page may be very interested. What are you telling them? “Copyright 2015?” This is not really a deal closer.
We saw on our scroll map reports that about 50% of mobile visitors reach the bottom of these pages. Page footer are rarely inspiring.

“You’re here. Desktop version. Back to top. Copyright.”

Why not do something a little bit more compelling? We changed this to reiterate the value proposition. Why should someone call, and how we can this company help?
We saw about an 8% increase in calls with this very simple change.

The Footer on Mobile is not a Graveyard

We’ll take 8% lifts all day and night.
Mobile will make up between 40% and 60% of almost anybody’s traffic at this point in time. We see some outliers, but more or less that’s the range. So let’s just say it’s 50%.
An 8% increase on mobile is a 4% increase on your entire business. There’s not many things that you can do to magically get a 4% increase for your entire business. This is an example of one of those things you can do.
It’s the beauty of conversion optimization. It’s a huge lever to grow your business. Something as simple as making an offer in the footer to reiterate your value proposition can have a meaningful impact on your business.
Get people to take action once they get to the bottom of that page.
[mobile-web-20]

4. Optimize for the Right Goals

What mobile goals are you optimizing for? Are they the right ones?
Conversion for your mobile visitors should look different than conversion for your desktop visitors. One of the ways to really leverage the growth in mobile is to understand and accept that.
One of our clients, eSigns.com, allows visitors to design vinyl banners and other signs for their event, yard, etc. It’s a phenomenal tool and a great site. Unfortunately, it is Flash-based tool, and it doesn’t work on iOS or Android devices. Even if it did work, visitors are less likely to design a banner or a sign on their small-screen mobile device.
Essentially we had this mobile dead end. Mobile traffic simply could not convert. eSigns had become resigned to ignoring their mobile traffic.
But we said, “What could we do with this traffic?” We chose to focus on getting email addresses instead.
We shifted our objective.

We used an entrance overlay to entice visitors to give us their email address.

This mobile entrance overlay enticed mobile visitors to give us their email address.

Instead of saying, “We need to get more conversions and more immediate revenue out of mobile visitors,” we said, “Let’s just focus on getting your email address”. We could tell from upstream indicators that these people are really just kicking tires.
Remember, we’re addressing a consideration set. We decided to do something to remain in their consideration set. Ask them to give us their name and email address. In return, we offered to email them a link to this page so they can come back to it later, on their desktop, when they are ready to do the design.
The results are that 5.3% of the visitors gave us their name and email address. Not bad, especially for an unoptimized experience. Each person who completed this form received an email with a link to that page, and the link was specially coded so we could track it and track those visitors.
Of those who got our email, we found that 26% of the recipients of this email clicked back to the site. Now this didn’t really result in an immediate boost in revenue from those clicks, but it lit a fire under their email list growth rate. They gathered almost 1,400 new email addresses in the first month of this test.
This is a business like many of your businesses that ultimately live and die by their email list. Building your email list gives you control over your own destiny. The marginal cost of delivery is almost zero. Not just for every single recipient of your email, but for every email. It’s so cheap to create new customers and satisfy repeat customers.
To get an idea of just how valuable these new email addresses were, we calculated the value of every recipient of eSign’s emails. We’re fond of a metric whenever we’re evaluating an email called revenue per recipient (RPR).

Collecting emails from the mobile deadend delivered an estimated $200,000 in additional annual revenue.

For eSigns, we calculated that the average annual revenue per recipient to be $11. With an average of 1,400 new email addresses per month, we’re generating almost $200,000 in additional annual income. Instead of ignoring mobile traffic, we took a longer-term approach by getting their email address, and using email to convert them.
Take a hard look at what your mobile visitors will be willing to do to begin a conversation with you.

Do This Now

These are our main premises. We hope you agree.

That mobile is a one of the fastest growing channels for your online business.
That mobile doesn’t have to convert at levels lower than desktop.
That mobile visitors want and need a unique mobile experience.
That there are no best practices and that you should test.

If you buy into our beliefs, these are the things you should try right now on your mobile traffic.

Optimize your header and determine if a sticky or non-sticky header works better.
Start testing those persistent calls to action – those parachutes. Test placement, test color, test language, and test the page you’re sending them to.
Take a hard look at the footer content on your mobile device right now and ask yourself, “If this is the last thing that someone sees on this page, am I compelling them to take action?” If the answer is no, the next question is, “What could we do here to compel them to take action?”
Ask yourself, “Are we optimizing for the right goals on mobile? Do we really believe that doing X and Y is what we want people to do on our desktop site. Is the best use of our optimization efforts on mobile?” Then test some alternative goals.

Once you’ve nailed your mobile experience, you can start having fun with your mobile site, like this add-to-cart animation.

You can have fun when you’ve delivered the mobile experience your visitors want.

May 5, 2016/by Brian Massey

How to Avoid Mobile-Friendly Redesign Disasters

Conversion-Centered Design

Joel Harvey, Conversion Scientist™

I’m sure mobile’s on the top of your mind.

We have tested hundreds of mobile hypotheses over the last couple of years and we’ve learned a lot. There’s still a lot that we don’t know.

We’re going to share some of the key things that we’ve learned along the way with you and show you what you need to be focusing on to start driving wins in your mobile traffic.

Mobile-Friendly Redesign and Best Practices

We are not here to talk about mobile optimization best practices. No, mobile best practices don’t really exist yet. Mobile best practices are a monster that lurks in our marketing. We call it the Bestpracticlops.

The Bestpracticlops reminds us how dangerous the phrase “best practice” is, and how dangerous best practices can be. We’re going to talk a little bit about that before we really dive into the meat of mobile-friendly redesigns.

Don’t let the Bestpracticlopse devour your mobile website.

A Best Practice is a commercial or a professional procedure that is accepted or prescribed as being the most effective.

What does that really mean?

It’s something that works everywhere all the time. That’s what a true best practice would be.
Now let’s compare that with the definition of conversion optimization

Conversion Optimization is “The process of generating and using data to improve a site’s unique experience for key segments.”

In other words, conversion optimization is finding the unique things on your unique site that drive your unique visitors to do more of what you want them to do. This is fundamentally different than the definition of best practice – something that works everywhere all the time.

They don’t live together nicely.

There is an almost infinite number of things that you can consider for testing on a website. There are an infinite number of things you could test on a website. Clearly, many of them aren’t really worth testing. Just in the mobile world alone, you could ask, “What kind of device type are we talking about?”

What’s the operating system?

Is it Android?
Is it iOS?
Is it Windows?
What browser are they using?
What’s the screen size?
What do we think the load time is of the page?
Is this a new visitor or a return visitor?
Is this a regular desktop site being shown on the mobile device?
Is this a responsive site adjusting itself to the mobile device?
Is it a mobile dedicated site?
What promotion brought them here?
What’s the site speed?
What’s the speed of the network connection?
Is it in landscape or horizontal mode?

Those are just a few of the things that you have to consider, not to mention the most important things like:

What is my offer?
What is the content?
What is the copy?
What is the design and layout?

We don’t want to make you feel overwhelmed. We want you to avoid this misconception of best practices: Whenever you’re faced with an infinity monster of choices, what do you do? You might want to reach for something easy. You might look for a best practices unicorn riding a rainbow that you can get on and ride to higher conversion rates.

Best practices are like Unicorns. They sound great, but don’t really exist.

You might be tempted to say, “Best practices. That’s going to make this easy. We don’t have to worry about the infinity monster anymore. We’ve got something in the bag.” The truth is that 50% of your ideas, at best, are going to be wrong and I can say this from years of experience.

Our process at Conversion Sciences is to painstakingly go through the data and the site and come up with a list of hypotheses. We have a fairly sophisticated methodology for going through and ranking those hypotheses based on what we think have the best chance of winning.

Yet, on our best day, half of those hypotheses don’t result in a lift in conversion.

Yet, on our best day, half of those hypotheses don’t result in a lift in conversion. That’s not because they’re ill-conceived or they’re just filler and fluff that we throw into the list. It’s because your visitors are very different from us and we really don’t understand them.

If you take that and apply it to the concept of best practices, particularly mobile best practices, you’re faced with a 50% success rate at best if you just start rolling things onto your site.

I’m going to show you some things that have worked for us. If you want to, you can just start rolling stuff out onto your site. However, I would highly encourage you to test it because what works on one site in the same industry doesn’t work on another site in that same industry.

And this is where we get the Bestpracticlops. You have a few failed tests and say, “We tried some things that those experts said worked and it didn’t for us, and therefore, optimization doesn’t work. We should focus on something else.”

Don’t fall into that trap.

Mobile Visitors Want Something, Well, Mobile

Whenever we think about mobile websites — and this is really true of all websites – we find that visitors don’t want what we’ve built for them.

Your mobile visitors will cut their own path to a solution. Will that path include your mobile website?

Your mobile visitors will cut their own path to a solution. Mobile-friendly redesign must take this into account.

You may be familiar with the concept of desire lines. Universities, state capitols, and other places, landscape engineers go to great pains to lay out what they think are the best routes for sidewalks, and what you inevitably find is that the visitors, the users of those sidewalks, pick their own path.

We know for a fact that this is absolutely true in mobile. The problem is it’s harder for a visitor to cut through to a desire line on mobile. You design for what you think your mobile visitors want, and this is different from what they actually want. I guarantee you these are two distinctly different things.

Furthermore, your optimized mobile experience is going to be different from everybody else’s, even more so than on the desktop. The best way to understand the desired path is through analysis and testing. The data will tell you.

Mobile visitors want different features, faster load times and mobile calls to action.

Mobile Traffic is One of Your Fastest Growing Segments

If this sounds like a lot of work, it is most likely worth it, no matter the industry you are in. The red-dotted line shows in the graph below shows that in just 14 months the percentage of people visiting this ecommerce site on mobile has grown from 20 up to 35%.

Mobile growth as a percentage of all visits for an ecommerce community.

This has real life implications on your site-wide conversion rate. For this particular audience, most people don’t buy on a mobile. So you have an ever increasing percentage of your traffic coming out on a device that, so far, doesn’t produce the same level of revenue.

The Mobile Screen is Screen One

More and more of your visitors’ first interactions with you will be on a mobile device. It’s important to understand this. In this section, we’re going to talk about dedicated mobile websites or heavily QA’d responsive websites.

Mobile is the first impression.

When someone’s decided they’re going to buy a new TV, what do they do? They pick up their mobile device whenever they’re stopped at a traffic light, whenever they’re waiting in line, whenever they’re sitting in a restaurant waiting for a friend. They start doing some simple searches.

“Best LCD TV over 52 inches”

These visitors are trying to get a sense of what their consideration set is. The consideration set is comprised of all the different specs and brands that you want to consider for the television.

The next question is, “I’ve discovered what the one or two things that I want to buy are. Now I have to figure out who I want to buy them from.”

The research that creates those two consideration sets doesn’t happen in mutually exclusive websites either. In doing these exploratory searches and they’re more likely to land on real merchant sites than on sites like Consumer Reports. They’re going to land on Best Buy. They’re going to land on Fry’s. They’re going to land on competing sites.

They’re not coming to buy right then. They’re landing to understand what they want and who might be able to provide it for them.
And this happens across any industry, whether it’s ecommerce or lead generation.

In the beginning, when people are in tire-kicking mode, at the very top of the funnel, they’re using their mobile device to figure out what they want and who might be able to provide it for them. The key is making sure that you are positioned to make a great first impression, and putting yourself in a position to be the website that they come back to whenever they’re actually ready to make a decision about who to buy from or who to submit their information to.

It’s a pretty simple concept. Give them a win.

If you forget that, you’re going to start making your mobile site work like your desktop site, and you’re going to lose when you do that.

Responsive Web Design. What Could Go Wrong?

One of the hottest topics in web design today is responsive web design (RWD). Remember the Bestpracticlops? The Bestpracticlops thrives on mobile responsive design.

Many sites have gone responsive in the last year. Many of you moved quickly, quicker than you would have liked to, in making your sites responsive thanks to Google’s Mobilegeddon update. It turned out not to be such a big deal.

I certainly know we were taking phone call after phone call saying, “We’ve got to go to responsive because of Google and can you help us do that in like six weeks?”

The truth about responsive templates is that they make decisions for you and, unfortunately, those decisions are not always good. Think about this: You have very unsophisticated technology trying to say in real time, “Based on the screen size of this device, I need to move this element, eliminate this one and to change the order of these others.” It’s not artificial intelligence here. It’s really dumb technology making decisions about how to fit stuff on a screen that’s 3 or 5 inches wide from a site designed for a 17 inch screen.

What could go wrong?

When Mobile-Friendly Redesign Becomes User-Unfriendly Redesign

The following image shows a pay-per-click landing page for Banfield Pet Hospital. They are spending pay-per-click money to get people to this page. The mobile landing page is inset over the desktop landing page.

Banfield’s responsive web design decided the company logo wasn’t important on small-screens.

Do you see what’s missing?

The most important trust-builder on the page has been removed – the logo. This initial credibility and trust builder has been amputated by the responsive design. This responsive template made a decision to simply remove the company’s logo. The headline is quite ironic in this case. We have not been properly introduced because there’s no branding.

Layout issues, “scroll jails” and “click jails” are symptoms of responsive web design.

Responsive designs can make a number of poor decisions, so many in fact that we wonder, “Why not just design for mobile specifically?”

Responsive designs can make a number of poor decisions, so many in fact that we wonder, “Why not just design for mobile specifically?”

What’s missing from the mobile image below?

The mobile screen on the left shows that an important headline was removed by the responsive template. Adding it back increased calls by 26%.

The responsive template decided to remove the headline on small-screen devices. Since mobile visitors are different than desktop visitors, we tested what would happen if we fixed this responsive decision. We told the responsive design, “Don’t make that decision. You’re bad. You made a stupid decision. Now sit in the corner.”

We found that returning this powerful headline to mobile devices delivered 26% more mobile phone calls.
This is real. This happens. We see this happen even today.

Responsive Web Design Delivers Slower Mobile Experiences

Responsive web designs don’t eliminate the fat when switching to mobile screens.

Internet Retailer did a study and found that, for large responsive ecommerce sites, the average load time just for the homepage was 18.24 seconds. We’re not talking about a category page that could have dozens of images in this case.

That is about 16 seconds too long probably, especially for mobile visitors.

Mobile visitors are really impatient. They are in the moment. We’re getting more and more impatient, and mobile visitors make desktop visitors look like the Dalai Lama when it comes to patience.

A mobile site with an 18 second load time is like shooting yourself in the foot. It’s like shooting yourself in both feet, and both knees. If you’re lucky it stops there.

Redesign Your Desktop Site to be Mobile Friendly? Really?

You don’t just get to make the mobile site responsive. The desktop site won’t remain the same.

The desktop site is probably your bread and butter for most of the people listening to this. You have to redesign your desktop site to have a responsive mobile site. Here’s what we know about redesigns: they almost always result in a decrease in conversion rate.
Sorry, we’ve seen it too many times.

In short, we recommend that you keep your high-converting desktop site as it is and optimize for that traffic. Build out your mobile-friendly site using a separate, adaptive web design. Then optimize that mobile traffic for the unique experience they want.
We’ll talk about what you should try on your mobile site.

NOTE: we find that a responsive web design works well for desktop and tablet-sized screens.

Mobile-Friendly Redesign Based on Data

Since every mobile audience is different, you’ll be well-served if you use your behavioral data to guide your mobile-friendly redesign. This includes:

User Testing
Surveys
Web Analytics
AB Testing Results

At the very least, you have the web analytics data and the recommendations here to guide you to mobile AB testing for results.
Conversion Sciences make mobile a part of every conversion optimization project. Contact us for a free consultation.

May 4, 2016/by Brian Massey

3 Very Different Meanings of the Word “Redesign”

Conversion Optimization

Have you ever found yourself in the middle of a conversation or argument and it suddenly hits you that the two of you aren’t talking about the same thing? Then you have that brilliant “aha” moment where you can actually start making some progress.

One workplace conversation that can be particularly tricky is whether your company should redesign its website. It’s important to make sure everyone is talking about the same thing when you talk about redesign because it’s costly, risky, and emotionally charged.

There are a few common reasons companies choose to redesign their websites:

The site performs poorly
The desire to be “mobile-friendly”
The site is “dated”
The desire to be “unique”

These reasons have a common denominator: you’re not happy with a very particular aspect of your site. There are many ways you can approach finding a solution to the problem, and we – the universal We – attach the word “redesign” to those solutions even though it means one of many methods are used to get the result we want.

Synonyms for redesign from dictionary.com

According to Dictionary.com, the above are some of the more common synonyms for “redesign”. The way Conversion Sciences uses this term is very industry-specific, so it has a certain jargon-y quality. Someone working in marketing at a tech or ecommerce company probably understands our jargon more than their colleagues in other departments.

If you’re that marketing person and you’re trying to convince your boss and other departments that you need conversion optimization, it’s really important that you’re all speaking the same language. You might be experiencing some miscommunication and not even realize it.

What are the different ways each of you might be using the word “redesign”?

Before you dismiss it as juvenile to keep returning to basic, dictionary definitions of “redesign”, make a mental tally of important people who don’t work in marketing, conversion optimization, or graphic design.

Your CEO and CFO, maybe your boss
Your customer service representatives answering chat, phone calls, and emails
Your customers

All of us feel great satisfaction in knowing the real definition, but ultimately being right isn’t helpful if no one understands each other.

A Full Redesign: Starting Over From Scratch

When we say “redesign” in its purest sense, we mean a brand spanking new website. You hired a designer, you have a new color palette and CSS, you completely threw out the old. Every page is new, the entire structure is different.

“Redesign” can be used to mean a brand spanking new experience

When Conversion Sciences cautions against redesigns, this is the definition we’re using. We say there are only two good reasons to undertake a website redesign:

You are re-branding or
Your CMS (content management system) is too limiting

When I worked at Westbank Library our website used a proprietary CMS built by the company that built our ILS (integrated library system). An ILS is used to search for books or connect to an online resource or check to see when books are due back. In other words, an ILS isn’t meant to be the platform for a very specific kind of online application.

Westbank’s homepage in 2008, built with a CMS that was only intended to be used for online library catalogs (screenshot via the Way Back Machine)

The ILS wouldn’t support some very important non-book-related features:

We couldn’t optimize the site for the search engines
We couldn’t embed a calendar
We couldn’t choose which photos appeared where on the page
We couldn’t create customized landing pages for special events
We couldn’t make the site ADA compliant
We couldn’t add widgets other libraries were using

We needed a new site built on a new CMS, one that met our present-day needs. The only way to do that was to dump the old one. The new website was built using Drupal, and it meant everything was new. The change was necessary and long overdue.

Westbank’s new homepage after the from-scratch redesign (screenshot via the Way Back Machine, which is why the images aren’t loading)

We were excited that on smartphones, the phone number was tel-linked and that the site was now searchable without going back to Google. Best of all, we had an actual, legitimate calendar. Before the redesign, the best we could do was make a list of what was going on.

Calendar of events on old site

After the redesign, people could see an actual calendar with clickable events where they could go find more information.

Calendar of events on new site

Without a doubt, the new site was an immense improvement. The lack of functionality on the old site was crippling us.
In this case, a full redesign was justified, but the results weren’t what we had hoped.

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

43 Pages with Examples
Assumptive Phrasing
"We" vs. "You"
Pattern Interrupts
The Power of Three

"*" indicates required fields

Conversion Optimization as Redesign: Making Incremental Changes

When the new site launched, our traffic went through the roof – hundreds of times more people were visiting our website. But since the change was long overdue, people who used the old site for a decade were totally lost.

Dozens of people called us saying “I can’t find anything on this new website, you need to redesign it!” and dozens more sent us angry emails saying the same. With the amount of time we spent working on the new website, it was disheartening to hear. Small public libraries don’t have the resources to do projects like this often – and in some cases, they can’t do projects like it at all. We knew we’d been fortunate, and we were suddenly terrified we had blown our only chance to fix our site. There were very serious discussions of applying for grants, then hiring a new design team to start over.

But after spending time talking to our patrons, we’d find out what they actually meant by “redesign”.

In one case, a gentlemen received an email reminding him to renew his books and included a link for him to do it online. Before our redesign, that link took him to his library account where he was automatically logged in on his home computer. All he had to do was click “renew”. After the redesign, this link took him to our homepage, so he had no idea where to go. When we say that your landing pages need to fulfill the promises you’ve made in your ad, this is a great example of what we’re talking about. Instead of changing the design of anything, we needed to fix that link.

Another way we knew people were lost is by analyzing how they used the site.

One problem in particular was how people used our site’s search box. All of the searches were for titles of popular books and movies, but the search box wasn’t connected to the online catalog. Our old site had one search box, and its only use was to look for books and movies. Everyone assumed the new search box had the same function, but it didn’t.

Our search bar at the launch of our new site.

Search options on the new website

We used the data from our search box the same way you can use heat maps. You can accommodate how your visitors are already using your site with the data you gather. Instead of forcing them to use our search box the way we wanted, we changed it to do what they wanted.

But that change meant our visitors, once again, didn’t have a site search option.

We changed the site search bar to be a catalog search, but it still wasn’t perfect

From this point, we found a widget that gave us a more dynamic search bar. Then we replaced images at the bottom of the page that linked to adult, teen, and children’s programs with widgets featuring new books and the library’s Instagram account. And we featured upcoming events more prominently, moved the contact information into the footer, added navigational links along the top of the page, and worked to make the site ADA compliant. The current homepage design is very different compared to what it was when we first rolled out the new website.

The homepage as it is now

These changes were slow-going, careful, and made one at a time. The redesign 1.0 and current iteration look similar because of branding and tabbed browsing, but for library patrons, these are two very disparate experiences. It is safe to say the new homepage underwent another redesign, but you might hesitate to use that word because the changes didn’t happen all at once.

Interested in Conversion-Centered Website Redesign Services?

Looking back at synonyms of “redesign”…

“Redesign” can be used to describe incremental changes

The website wasn’t perfect, but there was a lot to work with. We couldn’t start over every time we realized the site could be doing better.

Big Swings as Redesign: Changing Several Variables at Once

We use the term “big swing” to talk about sweeping changes we make on a page. Often these changes are on a page that’s particularly important or special, like a homepage or landing page.

It means we’ve changed several features all at once instead of testing one thing at a time. The downside of this strategy is that no matter how the page performs after the test goes live, we don’t really know why. If the page continues to perform with the exact same conversion rate, we don’t really know why: our changes may have offset each other.

Big, sweeping changes are exciting when they are successful, and people love to share these kinds of successes. They make great headlines and engaging stories. They give us hope that our big change will work out the way we want, or perhaps even better than we imagine. The problem is that there are usually third variables at play in these stories.

Think about the diet book industry. Every book boasts of its followers’ drastic life improvements due entirely to the diet. But when someone starts to pay attention to what she eats, she may also make other changes like exercising more, quitting smoking, and getting more frequent checkups with her doctor. Was her success really due to the diet book? Or was it purely chance since she made so many other changes? There’s no way to know.

Michael Scott’s Big Swing

Humans have the potential to be rational, logical creatures, but we often fall prey to our emotions when we make decisions, dole out praise, or attach blame. In an episode of The Office, Office Manager Michael Scott has the brilliant, big idea to send out paper shipments with five golden tickets tucked into the boxes at random. Each ticket awarded a 10% discount to its recipient.

The promotion quickly goes south when Dunder Mifflin’s largest client receives all five tickets, and there are no disclaimers or expiration dates. Michael arranges for a fall guy who will be fired for the idea, but then finds out this client has decided to send Dunder Mifflin even more business because of the discount. Naturally Michael wants the credit but doesn’t want to be reprimanded for almost bankrupting the company.

Michael Scott dressed as Willy Wonka, presenting his Golden Ticket idea

The Golden Ticket promotion was a big swing because Dunder Mifflin didn’t isolate the variable Michael was hoping to test: will current clients be more loyal to Dunder Mifflin because of a special, one-time-only, 10% discount?

The consequences of the Golden Ticket run the gamut of possible results of big swings:

Positive Result: When it seemed like the promotion would put Dunder Mifflin out of business, the responsible party was fired
Negative Result: When it became apparent the promotion would solidify a relationship with an important client, the responsible party was publicly commended
Neutral Result: Dunder Mifflin lost a huge amount of revenue due to the promotion, then gained more revenue, also due to the promotion

Big Swings at Conversion Sciences

In a staff meeting last week, Conversion Scientist Megan Hoover told us, “We completely redesigned this landing page for our client, and it was a big improvement”. In a different staff meeting, fellow Conversion Scientist Chris Nolan told us, “Our first test was to redesign our client’s homepage, and it was a huge success”.

Conversion Sciences doesn’t do website redesigns, we do conversion optimization. So what did Megan and Chris mean?

We switched from two columns to three
We wrote a new headline
We changed the copy
We changed the wording on the call to action

These changes mean they were speaking accurately when they described their big swings as “redesigns”.

Redesign describes what we do when we do big swings

“Redesign” describes what we do when we make big swings

We didn’t change the functionality of the page, the page’s purpose, or the CMS. We definitely made some big changes, but we certainly didn’t start from scratch, and all of the changes were very localized to a landing page and a homepage.

It’s worth noting that even though it’s tough to measure results when you make a big swing type of redesign, we still take the risk sometimes because Conversion Sciences has run so many successful tests. We are very good at making educated hypotheses about what kinds of changes will work well together, but we don’t attempt these big changes often. There is a lot of room for error in the big swing.

What is Your Desired End Result?

We covered three approaches to redesign in this post:

Throw-out the old, start from scratch
Incremental changes
Big swings

Let’s return to the most common reasons a company chooses to redesign:

The site performs poorly
The desire to be “mobile-friendly”
The site is “dated”
The desire to be “unique”

When you have the conversation at work about redesigning your site, try starting with the end goal.

If you work backwards, the conversation has a good chance of staying on track because it’s likely that everyone wants the same thing, even if it comes out of their mouths sounding very different. I’m willing to bet that everyone wants a home page with lower bounce rates. Everyone wants to reduce cart abandonment rates. Everyone wants more downloads of your industry reports. Everyone wants to sell more merchandise.

Redesigns are seductive. They come with big budgets and a chance to make a visible impact. The question at the heart of my arguments is this: do you need a website redesign, or do you need a website optimization program?

An optimization program can begin delivering results within weeks. Full redesigns take months and months to develop. An optimization program tells you which of your assumptions are good ones. Full redesigns are big gambles.

With a short Conversion Strategy Session, you will be able to make the case for a full redesign or optimization program for your growing online business. Request your free session.

April 28, 2016/by Conversion Sciences Team

AB Testing Results are Half-Filled with Losers, and That’s a Good Thing

CRO Tests | Multivariate | AB Testing

The law of unintended consequences states that every human endeavor will generate some result that was not, nor could have been foreseen. The law applies to hypothesis testing as well.

In fact, Brian Cugelman introduced me to an entire spectrum of outcomes that is helpful when evaluating AB testing results. Brian was talking about unleashing chemicals in the brain, and I’m applying his model to AB testing results. See my complete notes on his Conversion XL Live presentation below.

Understanding the AB Testing Results Map

In any test we conduct, we are trying everything we can to drive to a desired outcome. Unfortunately, we don’t always achieve the outcomes we want or intend. For any test, our results lie on one of two spectrums defining four general quadrants.

Map of possible outcomes from hypotheses.

On one axis we ask, “Was the outcome as we intended, or was there unintended result?” On the other axis we ask, “Was it a negative or positive outcome?”

While most of our testing seeks to achieve the quadrant defined by positive, intended outcomes, each of these quadrants gives us an opportunity to move our conversion optimization program a step forward.

I. Pop the Champaign, We’ve Got a New Control

With every test, we seek to “beat” the existing control, the page or experience that is currently performing the best of all treatments we’ve tried. When our intended outcome is a positive outcome, everyone is all smiles and dancing. It’s the most fun we have in this job.

In general, we want our test outcomes to fall into this quadrant (quadrant I), but not exclusively. There is much to be learned from the other three quadrants.

II. Testing to Lose

Under what circumstances would we actually run an AB test intending to see a negative outcome? That is the question of Quadrant II. A great example of this is adding “Captcha” to a form.

CAPTCHA is an acronym for “Completely Automated Public Turing test to tell Computers and Humans Apart”. We believe it should be called, “Get Our Prospects to Do Our Spam Management For Us”, or “GOPDOSMFU”. Businesses don’t like to get spam. It clogs their prospect inboxes, wastes the time of sales people and clouds their analytics.

However, we don’t believe that the answer is to make our potential customers take an IQ test before submitting a form.

These tools inevitably reduce form completion rates, and not just for spam bots.

CAPTCHAs reduce spam, but at what cost?

So, if a business wants to add Captcha to a form, we recommend understanding the hidden costs of doing so. We’ll design a test with and without the Captcha, fully expecting a negative outcome. The goal is to understand how big the negative impact is. Usually, it’s too big.

In other situations, a design feature that is brand oriented may be proposed. Often a design decision that enhances the company brand will have a negative impact on conversion and revenue. Nonetheless, we will test to see how big the negative impact is. If it’s reasonable, then the loss of revenue is seen as a marketing expense. In other words, we expect the loss of short-term revenue to offset long term revenue from a stronger brand message.

These tests are like insurance policies. We do them to understand the cost of decisions that fall outside of our narrow focus on website results. The question is not, “Is the outcome negative?” The question is, “How negative is the outcome?”

III. Losers Rule Statistically

Linus Pauling once said, “You can’t have good ideas without having lots of ideas.” What is implied in this statement is that most ideas are crap. Just because we call them test hypotheses doesn’t mean that they are any more valuable than rolls of the dice.

When we start a conversion optimization process, we generate a lot of ideas. Since we’re brilliant, experienced, and wear lab coats, we brag that only half of these ideas will be losers for any client. Fully half won’t increase the performance of the site, and many will make things worse.

Most of these fall into the quadrant of unintended negative outcomes. The control has won. Record what we learned and move on.

There is a lot to be learned from failed tests. Note that we call them “inconclusive” tests as this sounds better than “failed”.

If the losing treatment reduced conversion and revenue, then you learn something about what your visitors don’t like.

Just like a successful test, you must ask the question, “Why?”.

Why didn’t they like our new background video? Was it offensive? Did it load too slowly? Did it distract them from our message?

Take a moment and ask, “Why,” even when your control wins.

IV. That Wasn’t Expected, But We’ll Take the Credit

Automatic.com was seeking a very specific outcome when we designed a new home page for them: more sales of the adapter and app that connects a smartphone to the car’s electronic brain. The redesign did achieve that goal. However, there was another unintended result. There was an increase in the number of people buying multiple adapters rather than just one.

We weren’t testing to increase average order value in this case. It happened nonetheless. We might have missed it if we didn’t instinctively calculate average order value when looking at the data. Other unintended consequences may be harder to find.

This outcome usually spawns new hypotheses. What was it about our new home page design that made more buyers decide to get an adapter for all of their cars? Did we discover a new segment, the segment of visitors that have more than one car?

These questions all beg for more research and quite possibly more testing.

When Outcomes are Mixed

There is rarely one answer for any test we perform. Because we have to create statistically valid sample sizes, we throw together some very different groups of visitors. For example, we regularly see a difference in conversion rates between visitors using the Safari browser and those using Firefox. On mobile, we see different results when we look only at visitors coming on an Android than when we look at those using Apple’s iOS.

3.9% more Android users converted with this design, while 21% fewer iPhone users converted.

Android users liked this test but iPhone users really did not.

In short, you need to spend some time looking at your test results to ensure that you don’t have offsetting outcomes.

The Motivational Chemistry and the Science of Persuasion

Here are my notes from Brian Cugelman’s presentation that inspired this approach to AB testing results. He deals a lot with the science of persuasion.

My favorite conclusions are:

“You will get more mileage from ANTICIPATION than from actual rewards.”

“Flattery will get you everywhere.”

I hope this infographic generates some dopamine for you, and your new found intelligence will produce seratonin during your next social engagement.

Motivational Chemistry infodoodle by Brian Cugelman at ConversionXL Live

April 21, 2016/by Brian Massey

9 Imaginative Approaches to AB Testing Landing Pages

CRO Tests | Multivariate | AB Testing, Landing Page Optimization

After reading our Ultimate A/B Testing Guide, we thought you might want to take advantage of a few creative approaches to landing page AB testing. They go beyond basics to improve your digital marketing ROI.

There are a surprising number of AB tests smart marketers can run on their landing pages to ensure they’re getting the highest conversion rates possible. You already know you can experiment with different landing page elements for conversion rate optimization. Headlines, subheads, calls to action, and design elements can all be considered when building your testing hypotheses’ list.

Now, let’s kick it up a notch with these nine approaches for landing page AB Testing that may inspire you to create your own unique AB tests.

1. Create Targeted Landing Pages for A B Testing

Let’s say you are launching an advertising campaign. And you want to know what page will convert best: a generic one or a highly targeted to the audience and message. For this split test you could have ad variations that would use your generic landing page and one or more targeted landing page versions.

For example, if Shopify did an AB test between their generic vs. targeted landing pages in an advertising campaign, the control would apply to anyone looking for an eCommerce solution.

Shopify’s homepage with a generic target audience would be the control in this AB test.

For a variation or version of a landing page, they could use their page that targets booksellers.

This landing page addresses the ecommerce needs of a specific audience.

To maximize your conversions with this test, analyze your customer database to determine who your best customers are and create targeted landing pages focused on those specific groups of customers. If you do a quick Google search for Shopify’s landing pages you can see they have created targeted landing pages for their top customer segments.

Different landing pages convert different demographics and interests.

You can use these as examples of how to create targeted landing pages for your top customer groups and demographics.

2. Landing Page AB Testing: Experiment with Animated Headlines

Headlines can make or break your landing page, as they are the first words that capture your visitor’s attention. It’s a simple AB test that can make a dramatic impact on conversions, and it can be done using most AB testing tools.

The key to the headline AB test is to change nothing but the headline. For example, you can see how ActiveCampaign changed from a simple headline about their features…

ActiveCampaign’s headline about features.

…to a better headline about the benefits their customers can expect when using their service.

Headline about benefits.

This change proved to be a winner, as they have kept it since 2015 with a constantly changing message about their benefits. Therefore. test your headline.

This animated headline with benefits proved to be a winner

3. Landing Page Split Test Idea: Vary Featured Homepage Products

In most cases, your homepage will be the most popular landing page on your website. Hence, it will be a page where you should do extensive AB testing.

One test you can do to see if you can increase conversions is simply changing your featured product.

Nest does this by swapping out its popular monitoring camera (formerly Dropcam)…

Nest featuring its Nest Cam on the homepage.

…with its popular thermostat controller.

Nest featuring its thermostat on its homepage

This test is currently running (as noted by the ?alt=1 at the end of their homepage URL), so we’ll have to see which one wins in the end.

In some cases, you may want to change your featured product based on the one that is currently getting featured coverage in the media. Or whether that coverage is positive or negative.

4. Creative Landing Page AB Testing Idea: Explore Different Stories

Do stories resonate with your customers and if so, which stories translate into the most sales? Find out through AB testing. Apple did this in this past by running multiple campaigns on their website, social media, and television ads featuring stories about musicians…

Apple’s landing page story appealing to musicians

…explorers…

Apple’s story appealing to explorers

…environmentalists…

Apple’s story appealing to environmentalists

…parents…

Apple’s story appealing to parents

…and many other page visitors. The goal was to show how their products could help tell everybody’s story. No matter what they did or how much of an impact they made in the world.

Apple ultimately went back to a homepage focused on its latest products, but without AB testing, Apple couldn’t just assume that approach would convert the highest.

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

43 Pages with Examples
Assumptive Phrasing
"We" vs. "You"
Pattern Interrupts
The Power of Three

"*" indicates required fields

5. AB Testing your Landing Page: Shift the Focus to One Product at a Time

Landing pages have better conversion rates when there is one, clear call-to-action; sometimes that CTA is buying a product. But when your top landing page is your homepage, you can’t focus on just one product, right?

Maybe you can. In the past, Logitech had a pretty standard homepage that offered up all of its products all at once to homepage visitors.

Logitech’s original, more traditional homepage

But now, Logitech gives visitors a tour through their top products, one at a time.

Landing page focusing on a single product.

Logitech’s new homepage focusing on a single product

In a few moments, you are completely immersed in a particular product and its main benefit, thanks to this new landing page. It’s definitely a way of landing page AB testing you will want to try if you have a few products you can highlight in this fashion.

6. Transform Your Hero Image to Video

You know that great image of your product that you use to compel people to sign up or buy? Why not convert it into a video that dives even deeper into exactly what happens with that feature of your product. MailChimp made that change with their landing page, going from a screenshot of their newsletter designer…

Mailchimp’s landing page with an image of newsletter editor

…to a video of how their newsletter designer worked.

Video of their newsletter editor increased Mailchimp’s landing page conversion rate.

Instead of hoping that an image would convince visitors that their newsletter designer was easy to use, as the landing page claimed, the video was right there to prove it.

While their current landing page features a different, shorter animation, it still features one that demonstrates the ease of use of their newsletter designer. Thus proving that since 2012, video and animation on the landing page beats a screenshot for conversions.

7. Try a Change of Scenery: AB Testing Landing Page Copy vs Image

Sometimes it’s not text or functionality that will make your site convert better, it’s simply imagery that matches the story of your value proposition. Zillow experimented with this idea by changing the background of its search.

One variation was a neighborhood overview with home sale prices, which actually contradicts the line below the image about looking for rentals.

AB testing landing pages message: Zillow’s background showing homes with their prices isn’t relevant to people looking for rental property. Especially with a message at the bottom of the page specifically written for renters.

Zillow’s background showing homes with their prices isn’t relevant to people looking for rental property. Especially with a message at the bottom of the page specifically written for renters.

Another variation used an image of a specific home, which could appeal to both for sale and for rent searchers.

Zillow’s new image says “home” whether you’re buying or renting

It seems that they have stuck with the individual home view as it works with what most searchers are looking for.

8. Landing Page A B Testing Basics: Rearrange the Elements

It may not be your product, your service, your copy, your CTA button, your colors, or other elements on your page that are lowering your conversions. It may simply be the order in which they are presented.

Just like when you rearrange all the furniture in your house because it just doesn’t quite feel right, you might want to do the same with your landing page.

Take AgoraPulse, for example. They went from this…

AgoraPulse home page before ab testing

…to this.

AgoraPulse home page after split test

It’s easy to see why the latter layout works. It flows right into starting your free trial after a simple and convincing headline and subhead. And for visitors still not convinced they should convert, there’s simple video and bullet points to convince them to click that call to action button.

9. Copy Your Competitors

The most creative AB tests might be ones you don’t run on your own website. In addition to AB testing tools, there are tools that will alert you to when your competitors make changes to their websites. Potentially based off of their own AB Testing landing page experiments.

Rival IQ monitors your competitor’s website to see if changes have been made to it recently. The entry level plan allows you to track up to 15 companies. You’re able to track each company’s website design history along with their social media accounts, organic search rankings, and paid search traffic.

In the website history dashboard, you can view a variety of web pages from your competitor’s websites.

Depending on how long the company has been in Rival IQ’s database, you can get a couple of year’s worth of design history from each company.

ActiveCampaign’s design history over a few years

When you click on a particular month, you see the breakdown of when particular changes occurred. As well as the ability to click upon a particular design to see the full landing page.

Zeroing into the competitor’s design changes.

This will give you an idea of what AB tests a competitor has run in the past. Based on the length of time a competitor has stuck with a particular design, you will know which test was the presumed winner. That is, it was statistically significant in terms of increasing their conversions. Be it for their homepage, pricing page, features page, or other significant landing pages.

ActiveCampaign’s pricing page history

In addition, you can sign up for email alerts when your competitors make major changes to their website. This will let you know when your competitors have run new tests on their website and made changes based on their results. Or you may even see the tests themselves in action as the pages change from their original to alternative versions.

If you have a lot of competitors, and you’re not sure which to monitor, use the BuiltWith Google Chrome extension. It will help you find out if a particular competitor’s website is using AB testing software. Chances are, the ones that are will be the ones that will be making frequent changes.

BuiltWith browser extension for analytics and tracking

What are your unique approaches to Start A B testing your landing page?

If you’ve already done the standard AB tests on your landing pages, then we hope that these additional split test ideas will further help you increase website conversions. Just like it has for the brands and online businesses mentioned here.

Have you been running or reading about some interesting AB tests? We’d love to hear about them in the comments.

April 14, 2016/by Kristi Hines