Don't stop tests early.

What if this test had been stopped on Thursday?

We encourage every online marketer to begin experimenting with split testing as soon as possible. It is a skillset that empowers marketing to really understand their visitors and generate more revenue from existing traffic.

One of the first hurdles you will have to get through is the “damn lying test” phenomenon. This is when a test has been run and has found a way to significantly increase results. The confidence level reported by the testing tool is 95%. You are going to be a hero.

Then you roll out the change to the site and nothing happens. Conversions don’t go up. Revenue per visit doesn’t move. What’s going on?

It’s the Damn Lying Test. One of the reasons tests lie is that they didn’t take into account the ebb and flow of traffic on your site. Tests must take into account peak and off-peak times. Form most sites, a test must run for a full week and end on seven-day boundaries.

How can you discover the rise and fall of tides in your website?

In Ride The Tides Of Your Website To Make More Money With A/B Split Testing, I discuss the various cycles and how to detect them on your site using analytics.

Listen to me read the column

The Conversion Scientist Podcast


Subscribe to the Podcast
If you enjoyed this, please leave a review on iTunes!

I provide graphs and examples to illustrate a variety of website tides:

  • Intra-day cycles
  • Weekly tides
  • Monthly cycles
  • Yearly seasons
  • Device-dependent cycles
  • Sales cycles
  • Artificial cycles

Understanding the rise and fall of the tides in your website will help you design better tests that deliver results you can take to the bank. Use your knowledge of website tides and some discipline to steadily increase the profitability of your site.

Special thanks to Craig Sullivan for inspiring this column with his Digital Elite Camp presentation 20 Simple Ways to Mess Up Your AB Testing.

A “split” is not usually considered a good thing in the world of prophylactics. However, a condom shop that is “split testing” is going to have happy customers and prosperous owners.
I had the great pleasure of visiting Estonia and the Netherlands on a speaking tour. While in the Netherlands I stayed in a real 19th century windmill, hung out in an 18th century farm house, and walked by a condom shop housed in a 16th century building.
There are a lot of old places in the Netherlands.
I mention the condom shop because I discovered that this shop, called Condomerie® was split testing their store front.
It begs an important question: if a condom shop in the red light district of Amsterdam is split testing, why aren’t you? We can get you started.
While I was impressed that the Condomerie was split testing their products, I was just as excited about the number of condom jokes I could make in a blog post.
Start counting.

The Windows of Condomerie

The store front of the Condomerie was tasteful, consisting of two large windows separated by a door. In each window a selection of condoms were well-hung, having been blown up like a balloon to show off the size and shape.
I found the collection of colorful condom most interesting. As you can see from the photo below, these condoms were animals, vegetables, soccer balls, skeleton heads… there was even a Eiffel Tower condom. You needn’t worry about size when you’re sporting the Eiffel Tower.
And what says “I love you” better than a condom that says “I love you?” See them all in the picture below.

The front window of the Condomerie

Click to “Enlarge” (snicker)


A condom with a QR code on it.

The Qondom


Being a geek, my attention was drawn to an artificial condom made from laminated paper that contained a secret QR code. I call it a QR condom, or Qondom.
Scott Stratten says that every time a QR code is used a kitten dies. However, you have to get creative when designing tests. Use whatever tools are at your disposal.
I scanned it with my phone to see where it would take me. The URL contained a word my Dutch friend understood: “Right”.
We looked to the left window. Sure enough, there was a qondom there too. When I scanned it, the URL that was revealed contained the Dutch word for “Left.”
The Condomerie was split testing its windows! They had placed a Trojan Horse in the window designed to trap sightseers with smartphones.

The Hypothesis

If they were doing a good job of testing, they would be working from a specific hypothesis. At a glance, the hypothesis became pretty clear:
“If we display our funny condoms in the window, we will draw more attention and sharing as measured by QR Code scans and visits to our website.”
Their test placed the colorful cartoon condoms in the right window. I’m going to call this the right-brained window that’s creative, emotional and completely unconcerned that these condoms are far more intricate than necessary.
On the other side were condoms arranged primarily by size. This is the left-brained, “just the facts” side of the store. You can imagine that those interested in this window would be looking for a calibration scale to ensure they got the best fit.

The left window of the Condomerie is about size and fit.

The left window of the Condomerie is about size and fit.

The right Condomerie window features colorful and whimsical condoms.

The right Condomerie window features colorful and whimsical condoms.

Which window would you linger at? The left-brain, functional window? The right-brain imaginative window? Or, the window of the cheese shop next door?
At this point, only the proprietors of the Condomerie know.

An Inflated Sample

A problem with this test is that only tech-savvy condom users with smart phones are going to be part of this sample. Those with feature phones and those who’ve never bothered to load a barcode scanner app will not be able to participate.
This is a real problem, as it probably skews the results, inflating the sample toward early adopters. The sample taken wouldn’t be expected to represent the typical condom-buying public.
There are some fundamental issues with the landing pages that may be suppressing conversions, but we can expect all marketing channels to improve as an organization learns from split testing.
This shop is learning something about their tech-savvy audience, an audience who can often afford the more expensive condoms and profitable condoms. Revenues can be expected to rise when tested changes are rolled onto the shop floor.
What are you learning about your website visitors? How are you testing your assumptions about what people want, like and are persuaded by? Testing is the most effective way to consistently increase your revenue. The Condomerie knows this and so should you.
Let Conversion Sciences get you started on a website split testing program that will get more revenue from the traffic you already have.
Get a free copy of my book if you are the first person to provide a correct count of the double entendre in this post. Put your number in the comments.
[sitepromo]
[signature]

How helpful would it be to know what prices and features your competition was thinking about using?
One of my readers just sent me a very revealing screenshot. It is one of the pricing pages that Optimizely is testing. It was found by “spying” on their test data.
 

We hid the pricing on this test treatment from Optimizely

We hid the pricing on this test treatment from Optimizely


We are able to see this because of an “exploit” that allows anyone to see what a site is testing if they are using the Optimizely testing software. Oh, the irony.
Venture Beat recently “revealed” this in an article. Those of us who use these tools have known about it for some time. It’s quite easy to decipher this test data.
Try dragging the following link to your browser bookmark bar.
Optimizely Spy
Now visit Optimizely and click on the bookmark to see what they are testing.

How is this possible?

Whenever we run a split test with Optimizely, the software uploads scripts and data into all of our visitors’ browsers to change the experience and track the results. Along with this is included not just the test our visitor is being entered into, but all of our tests for that account.
So it’s relatively easy to decipher this information and see what we’re testing.
Note that the snooper can’t see any actual results, just what kinds of things you’re testing.
We like this approach because it speeds up the delivery of tests. When we use one file with everything, it changes less frequently, and the file it can be cached on a content delivery network (CDN) specifically designed to deliver files faster.
Faster tests mean more reliable tests.
Convert.com also uses this technique, though they take steps to obsure the test information.

Why Aren’t We More Concerned?

In a worst case scenario, a competitor can see what hypotheses you are testing. They can then test those same ideas and perhaps win more customers.
However, only a small percentage of sites are even testing, let alone stealing your tests. I did a quick survey of sites selling plastic surgery and cosmetic surgery who are spending at least $500 per month on search advertising.
Of 2,958 domains, only 33 had some form of split testing software installed, such as Optimizely. That’s just 1.1% of these domains. Furthermore, we know that some portion of these testing are not actually using the software they have installed.

Plastic and Costmetic Surgery Websites with Testing Software

Plastic and Cosmetic Surgery websites are missing a significant opportunity to get more patients. Source: SpyFu.com


Here’s another surprise. There are ninety-seven (97) domains in this space spending over $50,000 per month on search ads. Only five of them have A/B Testing software installed, only 5%.
If you’re in the plastic surgery space and are testing, you have a major advantage over your competitors. So, the odds of someone stealing your ideas are far outweighed by the gains you will see from testing.

Our Recommendation

We recommend that you continue to test using Optimizely unless your page contains sensitive information, such as price.
If you feel uncomfortable with your test information being publicly available, move to Convert Experiments for some protection. Another popular tool, Visual Website Optimizer, does not use this technique meaning past and future tests are safe from prying eyes. There are also a variety of other highly recommended AB testing tools available.
Whatever you do, don’t let this issue take the steam out of your testing program. As you can see, testers have a significant advantage, snoopers or not.
PS: If you are in the plastic and cosmetic surgery industry, you should contact us.

testing, valley of death
There is a point in almost every testing program in which your confidence will be shaken. Even seasoned professionals experience it.
We start off with high confidence having identified opportunities to really improve a site. We’re good at getting that low-hanging fruit early on, but we will inevitably find that many of our great ideas just don’t work.
It’s humbling, the right kind of humbling.
With patience, our picture of the visitors coming to a site become clearer. On-going success is due not to our own brilliance, but to what we learn in the Valley of Death that kills so many testing programs before they’ve matured.
We can take you through the Valley of Death.
Have a conversation with a Conversion Scientist this week to see what our testing team can do for your bottom line.
[signature]
[sitepromo]
Inspired by the excellent blog Indexed.

Those who are successful have a certain way of looking at things. They are committed to the end goal, but focused on the next ten percent.

The Next Ten PercentOur goals are reached one step at a time.

The next ten percent is the next face to scale on the climb, the next set of downs in the game, or the next year of raising a healthy kid.
The next ten percent is more than another step or a rung on the ladder. It’s a complete process that, when repeated results in success. Why ten percent? Because ten percent makes a difference. It’s a reasonable goal, but not necessarily an easy one.
You can get your first ten percent once by luck or fortune. Success comes from getting that ten percent time after time.
It’s invigorating.
For us, the next ten percent is the next cycle of website tests. Ten percent is a reasonable goal. Repeat the process five times and you will see a 50% increase in website performance. If revenue is your goal, that’s 50% more revenue. If leads make your business go, then expect 50% more leads. Grow your revenue per visit by just 7% a month and you’ll double your revenue in one year.
We’d like to introduce you to our ten-percent at a time process. We call it the Conversion Catalyst.
It’s a proven system to get you that next step month after month.
Reply to this email or give us a call at 888-961-6604.
[signature]
Brian Massey

In his blog post announcing a new version of Visual Website Optimizer, Paras Chopra makes it clear that they want to bring the scientific method to online marketers.

Internet Microscope DoodleYou can get the tools. Who is sitting in your chair?

We’ve been doing this since 2006.
Paras says, “[pullquote]Scientists and engineers have been using the scientific method of research-based experimentation for hundreds of years to make the world a better place. With VWO, we want to bring that philosophy to the world of marketing.[/pullquote]”
He also says something that I think is somewhat profound:

Think like a marketer, execute like a scientist.

This is analogous to our motto of “Rigorous creativity.” It means that the human side of marketing doesn’t get pushed aside by numbers and spreadsheets. If we use data to understand the human animal better, we will become powerful, compassionate online marketers and business owners.
VWO is just a tool, one we like and use. Let us show you how to use a little science to become the online business you’re capable of becoming.
Have a conversation with a Conversion Scientist.
[signature]
[sitepromo]

Joel Harvey presents at Conversion Conference San Francisco 2014. Here’s the pictures and comments from his unusual presentation.

[signature]
[sitepromo]

We find out which half will work.

How can it be that a team with our experience, intelligence and good looks could be wrong so often? It’s a mystery to us.
The truth is, that the audience for any website is unique. Given two sites selling the exact same product, you cannot assume that what works on one will work on the other.
People are complex. Groups of people are only slightly less complex.
This is why our optimization process is so powerful. We let your visitors decide what works best, and they vote with their money or their contact information.
We call it the Conversion Catalyst.

Why We Win

We live by a few mantras at Conversion Sciences to guide our decisions.
Our opinion doesn't matter
No matter how smart we think we are, the only thing that matters is what gives your visitors a better experience on your site. Don’t get caught up in the words “better experience.”
The best judge of our success is the accountant.
Visitors vote with their dollars. A good experience is one in which more visitors find what they need, want, or think they want. The person we are trying to impress really is the person that counts the income.
It’s not that we’re humble, but testing has a way of humbling you.
You don't have to be humble to optimize, but you can't optimize without being humbled.

We’d like to humbly offer to optimize your website.

Let us make your accountant smile.
You can get a free strategy session with a Conversion Scientist. We’ll help you see the possibilities for your visitors.
[signature]

No, we aren’t talking about The Blob, or The Mist or even The Thing. We’re talking about “The Thing” that keeps visitors from taking action on your site and converting. Brian has a great post on Search Engine Land talking about The Thing and how do deal with it.

theblob3

To be fair, The Blob is pretty fearsome as well.

That is this elusive Thing? According to Brian, the thing usually falls in one of five areas; risk reversal, value proposition/messaging, social proof, user interface/user experience, and credibility/authority.

Conversion-Scientist-Podcast-Logo-1400x1400


Subscribe to Podcast


Once you identify The Thing, what can you do to combat that concern and give your visitors the confidence to take action.

Brian walks you through each area and what changes you can make to reduce visitor concern. Think of it as your tools to fight The Thing.

Read Brian’s entire post on Search Engine Land or listen to the audio below, and rid your site of The Thing today.

Transcript

There is an insidious voice speaking to your visitors from the moment they click through to your site. It camps in the back of their minds, setting up a tent and proceeding to talk your visitors out of taking action. While there are many angles this voice can work to fill visitors with doubt, there is likely one that is common to a large number of your visitors.

It’s “the thing.”

If you can discover and address the thing — the major concern shared by a significant number of your visitors — then you can make some major improvements in your conversion rates and revenue per click.

When testing, we have found that this thing will fall into one of five areas: risk reversal, value proposition/messaging, social proof, user interface/user experience, and credibility/authority. All of these nagging questions may be present to some degree, but one of them is more pressing than the others — and addressing it will give you wins early and often.

I recommend that you open a spreadsheet so you can capture the hypotheses that come to mind as you read these gems. Your hypotheses should read like this:

If I [change something] then more visitors will [do something good or stop doing something bad] as measured by [some metric like revenue per visit or conversion rate].”

Let’s get started.

Risk Reversal

The thought camping at the back of your visitors’ heads may be, “What if I regret this purchase?”

  • What if I don’t like it?
  • What if it doesn’t fit?
  • What if I didn’t consider something before buying?
  • What if I feel tricked?
  • Will you sell my contact info to a spammer?
  • What is the likelihood that you waste my time?
  • Will you protect my data?

Risk reversal tests start with the return policy. The most famous return policy of late is the Zappos “return within one year and we’ll pay shipping both ways” policy. It is clearly visible throughout the site, summarized in the header.

If you’re generating leads, the most important way to communicate risk reversal is your privacy policy and privacy statement. You could test link anchor text such as “We respect your privacy,” or “Your privacy is important to us,” or “We will never share or sell your contact information.”

Did you know that free shipping falls into the category of risk reversal? It means that I won’t be surprised by high shipping rates when I get into the shopping cart. Knowing what to expect is often more important than dollars saved — after all, we know shipping is factored into the pricesomehow. Test free or flat-rate shipping.

Value Proposition & Messaging

While munching on a blackened hot dog, the voice camped at the back of your visitors’ mind is whispering, “What’s in it for me?”

  • What’s my payoff?
  • Will you make me look better, smarter, cooler, more interesting?
  • Are you low price, high quality or good service?
  • Are you making an offer I can’t refuse?
  • What is the one thing I need to know about your offering?
  • Why would I put my career on the line by considering your solution?
  • What is your story?
  • Does that girl with the headset really work for you?

Communicating your value proposition and messaging is the job of the page content — this includes text, images, video, audio and almost any other media.

Headlines and calls-to-action are always important, and testing often starts there. The inevitable hero image should be tested, especially if it is a rotating banner style so prevalent today (and so often a bad idea).

Long-form versus short copy is another way to find out what your visitors prefer.

Test more detailed pictures of your products. Test getting rid of any stock photography you have on the site.

Never underestimate the power of the words on your site. Some of the most transformational tests we’ve seen involve honing in on the right words.

Social Proof

The voice camping at the back of searchers’ mind may start a fire, roast some marshmallows and whisper, “What would others think?”

  • Am I being reckless?
  • Does the rest of the herd approve of you?
  • Has anyone had a really bad experience with your brand?
  • Has anyone had a really good experience?
  • Do others confirm what you say about yourself?
  • What are other businesses in my industry doing?

We are social animals, and the herd mentality never really leaves us.

Ratings and reviews are a powerful addition and should be tested if you can get your customers to chime in with their opinion.

Test testimonials near your calls-to-action and in your shopping cart.

Come up with some big numbers to describe your success. Rather than counting customers served, consider measuring your success in dollars saved, bites eaten, or seconds spent so that readers can relate to what you’ve done.

Test social media in moderation. It can be a distraction. Will your social customers post pictures of your products? Write reviews? Provide testimonials?

User Interface & User Experience

While carving a snake out of a stick with a pocket knife, the voice in the back of your visitors’ head may be saying, “Nice job. You’re lost.”

  • Can I explore your offering the way I like to explore?
  • I’m new here. Where do I start?
  • I’m back again. Where do I go?
  • What’s the next step for me?
  • How many more steps do I have?
  • How do I take action?
  • Can I scan your site or do I have to (gasp) read?
  • Where’s the discount you promised?
  • What if I’m not ready to act?
  • Where can I find your risk reversal, your value proposition, your social proof and evidence of your credibility?

How you present information on a page can have surprising effects on your bottom line. In general, your designer should be skilled at the use of white space, position, font, color, and proximity to guide the visitor through a page.

To start with, test making important things stand out, such as calls-to-action. Test the contrast and size of text to see if readability is an issue.

Test completely different layouts for pages to find the right ballpark to do more detailed tests. Simplify or complexify.

Never underestimate the power of ugly to add more dollars to your bottom line. Don’t get attached to your creative. Your opinion doesn’t matter.

Credibility & Authority

The voice at the back of your visitors’ head may be whispering, “Will I get duped?”

  • Will I look stupid?
  • Will the product be high-quality?
  • Have I had a positive experience with you in the past?
  • Would a reasonable person buy from you?
  • Would a genius buy from you?
  • Will you keep your promises to me?
  • Are you good people?
  • Do you care?
  • Will you get me fired if I recommend you?

The first way to communicate credibility and authority is with your company logo. In general putting it in the upper left on your site does the trick. However, it may actually hurt you on targeted landing pages.

Borrowing authority is a favored strategy. Test the addition of client logos to key pages (landing pages, home page, etc.). If you take credit cards online, be sure to include Visa, MasterCard, Amex, and others, even if you take everything.

Test logos for associations you belong to. Test adding shields for certifications you’ve earned. Test the placement of site security logos, such as McAfee secure and VeriSign. Should they be at the bottom? In the header? Near the call to action button?

Test moving blog post titles to the home page to show your thought leadership (but don’t let them get stuck on your blog).

Picking A Direction For Your Testing

Hopefully, you’ve been jotting down hypotheses about your site as you’ve read this article. Now, you need to prioritize them.

Chris Goward offers PIE as prioritization criteria in his new book. Bryan Eisenberg uses a 5x5x5 model.

At Conversion Sciences, we use a Proof/Impact/Effort/Traffic model that doesn’t seem to spell anything clever. Please offer suggestions in the comments.

Those hypotheses that are supported by analytics, are expected to have a high impact, and require the least effort will bubble to the top of the list.

Now, pick one hypothesis from the top of the list that falls into each of our buckets: risk reversal, messaging, social proof, user experience and credibility. Test these first.

If and when one shows a significant win, you’ve got a good idea of what the voice in the back of your visitors’ mind is whispering to them. Try more hypotheses from this bucket.

Following The Rabbit All The Way Down The Hole

If a headline performs well (messaging), then test a hypothesis about copy length next. If a new layout provides a bottom-line boost (user experience), you might then test a hypothesis that says choices should be reordered.

What you’re doing is finding out what the biggest issue is for your visitors, and then diving in to see how far the rabbit hole goes.

When do you stop and look back at the others? When the wins become scarce and small. Switching to a new category can reinvigorate a testing schedule that needs some big news.

Test hypotheses from each of the five “buckets” to find the major concern of your visitors. This gives you the direction to take for early increases in conversion rate and revenue per click.


21 Quick and Easy CRO Copywriting Hacks to Skyrocket Conversions

FREE: Click to Download

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

  • 43 Pages with Examples
  • Assumptive Phrasing
  • "We" vs. "You"
  • Pattern Interrupts
  • The power of Three

Big Data is Good—But Big Testing is Better | Chief Marketer

@chiefmartec “out of 12,000 experiments that Google ran in 2009, only about 10% of them resulted in adopted changes.” You are not Google. But you must realize that in every industry, an online leader is going to emerge, and it will be the company that adopts a testing culture.
Intuition is no longer your friend. Your intuition cannot comprehend the variety of ways our visitors are coming at us. Data and testing are your friends, or your online audience will continue to abandon you.
Get excited about testing and taking that leadership role in your marketplace. 
Want to get Brian’s For Further Study posts delivered right to your inbox? Click HERE to sign up.