AB Testing JavaScript: Great Power, Great Problems

The AB Testing JavaScript that powers tests is powerful, but can lead to many unintended consequences.

Conversion Sciences offers a pretty amazing ability to our clients: A completely “turnkey” testing service. By “turnkey” we mean that our clients don’t need to do anything to their website in order for us to analyze the site, design creative, develop code, QA, test, review and route traffic to winners.
Why is this? Don’t we need new pages designed? Interactions with IT? Release schedules? Sprints?
The reason we have this “phenomenal cosmic power” is that our AB testing tools take control of each visitor’s browser. The changes are made on the visitors’ devices, so the website doesn’t have to be modified.
Well, not until we’ve verified that a change brings in big money.
While this makes us feel all high and mighty, it comes with a big drawback.

The Magic and Mania of JavaScript

All of the major browsers have a scripting engine built into them. It allows programs to be run inside the browser. The programming language used is called JavaScript. It’s JavaScript that makes websites interactive. Website developers use JavaScript to make text accordion when you click a heading. It is used to rotate images in a carousel.
Unfortunately, developers use JavaScript to do silly things, like parallax animations.

This unnecessary "parallax" motion may be reducing the conversion rates for this site.

This unnecessary “parallax” motion may be reducing the conversion rates for this site.

And then there’s this.

Don't use JavaScript animations just because you can.

Don’t use JavaScript animations just because you can.

Yes, JavaScript gives us the power to make our websites harder to use.

JavaScript is Used in AB Testing Software

Our developers use JavaScript to modify a website when you visit it. The AB testing software “injects” our JavaScript into the browser when the page is loaded.

First the web page is loaded as is from the webserver. Some visitors will see only this default version of the website.

For some visitors, our JavaScript is then injected and executed. The AB Testing software determines who sees the default web page and who will see a variation of the page.

Phenomenal Cosmic AB Testing Power

We can change almost anything about a page using AB Testing JavaScript.

We change the headline to something different.

We change the order of a rotating carousel, or slider.

We hide elements…

AB Testing flicker can be cause by simply removing elements.

AB Testing flicker can be cause by simply removing elements.

and insert them as well.

We insert video.

We completely change the look of a website.

We completely change the look and feel of this site. The test showed the white site performed the same as the brown site.

We completely change the look and feel of this site. The test showed the white site performed the same as the brown site.

AB Testing JavaScript is a Step Toward Personalization

If we wanted, we could deliver a different website to every visitor who arrives. If you’re thinking “personalization” at this point, then you have an idea of where our industry is heading.

AB testing produces the data that makes personalization truly work.

Here are my instagraphic notes from a presentation by Chris Gibbins at Conversion Conference 2016 on using AB testing and personalization.

Instagraphic shows why and how to use AB Testing for personalization.

Instagraphic shows why and how to use AB Testing for personalization.

AB Testing Flicker, Flash, Blink and Ghosting

Unfortunately, JavaScript can introduce an error into our AB tests. Because we always load the website “as it is” on the server first, there is a possibility that the visitor will see the original for a fraction of a second before our changes get executed.
This has been called a “flash”, “flicker”, and a “blink”. It can have a significant effect on test results.

With AB Testing JavaScript, Flash is not cool.

With AB Testing JavaScript, Flash is not cool.

The problem with this AB testing JavaScript flicker is that it won’t be there if a change is made permanent on the site. Our tests may say the new change generated more revenue, and we’ll change the website to make that change manifest. But there will be no flicker. This means there is another variable in our test.
Was it the new headline we tested or was it the flicker that made more people buy?

21 Quick and Easy CRO Copywriting Hacks to Skyrocket Conversions

21 Quick and Easy CRO Copywriting Hacks

Keep these proven copywriting hacks in mind to make your copy convert.

  • 43 Pages with Examples
  • Assumptive Phrasing
  • "We" vs. "You"
  • Pattern Interrupts
  • The Power of Three
  • This field is for validation purposes and should be left unchanged.

The Human Eye is Drawn to Motion

Our eyes and brains have evolved to make motion important. When something moves, our eyes and brains work hard to figure out if the thing moving is edible, if it will eat us, or if we can mate with it. Technology has moved faster than evolution. Even though we’re looking at a website, where there is no immediate source of food, fear or fornication, our lizard brains continue to favor motion.

Imagine this scenario. We are testing the text on a button. That means that for some of our visitors, the button will change. Others will see the default button text. Then we’ll see who buys the most.

The changing element on this page draws your eye, and can make the page impossible to focus on.

The changing element on this page draws your eye, and can make the page impossible to focus on.

If there is a flicker, flash or blink when the text changes, the button will be immediately more visible to those visitors who see it. More of them will see the call to action. This may make more of them consider buying than those who simply scrolled past. If this new treatment generates more revenue, we are left with the question, “Was it the text or was it the motion that created the lift in sales?”

We won’t know until we push the new text onto the server and do a post-rollout analysis to see the real lift. At this point, we may find that the text didn’t increase purchases. It’s a real bitch.

How many buttons have been change because of flicker?

AB Testing Software Tries to Eliminate Blinking

The AB testing software that we use works to eliminate this blinking and flashing. The issue is so important that Convert Insights has patented their method for eliminating “blink”.

AB testing software like Optimizely, Visual Website Optimizer, Adobe Test, and Marketizator, load asynchronously, meaning that they load our changes as the page loads. This makes it easier for changes to be made before the visitor sees the page.

How Convert Insights Eliminates “Blink”

“The first thing this snippet does is hide the body element for max. 1.2 seconds (this can be set), this prevents blinking of elements already visual on the site that load under 1.2 seconds (we have yet to get a client that loads faster than this). During the 1.2 seconds, the SmartInsert(R) technology search and replaces DOM-elements every couple of milliseconds and loops through the entire available DOM-elements in the browser of the client. After all elements are replaced the body hidden attribute is set to display the site again either at 1.2 seconds or when the signal is given that all elements have been replaced (or DOM-ready).
Everybody can see how this technology works by loading our Chrome Extension.”
— Dennis van der Heijden, Convert.com

Eliminating Flash Flicker and Blink in AB Tests

In addition to this, our developers can do things that reduce and eliminate flicker and blink. Every test you do has different issues, and so a variety of tactics can be used to address them.

Avoid Google Tag Manager

Don’t use a Tag Manager like Google Tag Manager to serve your AB testing JavaScript software tags. Add them to the page manually. Tag managers can counteract the asynchronous loading of the tool and delay when changes can be made.

Make Changes with CSS

If the change can be made with the cascading style sheets (CSS), we favor making changes with CSS over using JavaScript. CSS can make changes to an element – like an image or font – that can be applied before an element displays.

Modal Dialogs and Overlays

Modal dialogs usually don’t display until the visitor takes an action. They can be edited before they are shown without flashing.

Use a Timer for DOM Changes

All of the images, headings, copy, widgets, and forms are stored in the browser in a big database called the DOM (Document Object Model). When JavaScript makes changes to the DOM, the content on the page changes. The DOM is slow-loading, as you can imagine.

Our developers will set a timer in JavaScript to check for when a changing element is loaded. By watching for our element to be loaded, we can execute a change it before the DOM – and the page – is loaded.

For the AB Testing Software to Load Our Changes Immediately

The AB testing software provides a synchronous loading mode. Optimizely and VWO use different approaches for this.

Rethink the Test

Sometimes, we have to go back to the drawing board and design a test that doesn’t cause flash-inducing changes. We will refactor a test to eliminate items that cause a flash or flicker.

Delay Display of the Content

We can delay the display of the content until the DOM is loaded and all elements have been changed. This causes a different kind of issue, however. The visitor sees a blank page or blank section for longer if they get the page with the change.

In one case we added a spinning wheel to indicate that the page is loading.

Insert Placeholders Using Fast CSS

When inserting elements, we’ll use CSS to insert a placeholder, then use JavaScript to modify the element after the page has loaded. This reduces redrawing of the page when elements are inserted.

We created a blank box in CSS to minimize AB Testing Flash on this mobile website.

We created a blank box in CSS to minimize AB Testing Flash on this mobile website.

Optimizing for Mobile

Mobile pages load more slowly. This is because visitors don’t always have a high-quality connection when visiting from the festival, or from inside the walls of a bank while standing in line. For mobile devices, flash can be an even bigger issues.

Fortunately, the tactics we use on the desktop also work on mobile websites. But don’t forget to QA your test treatments on 3G and 4G connections. You may find flicker and blink on mobile devices that didn’t appear on your desktop.

Great JavaScript Developers are a Must

We spend a considerable amount of our time making sure our JavaScript and CSS changes are like the “native” implementation would look.  It’s one of the things that makes testing hard. Our team has the experience to ensure your tests aren’t falling victim to flicker, flash, blink or ghosting.

If you’d like our team to take over the effort of developing your AB tests, contact us for a free consultation and an overview of our process.

Brian Massey
9 replies
  1. Chris Desadoy says:

    There are also full stack ways of doing this with angular and node.js you can use angular itself to pre-load and route pages based on headers presented dynamically.
    You can load balance and do a 50/50 split that way too, mostly onpage with a jquery plugin, w00t
    do you guys use angular for the preload?

    • Brian Massey says:

      We rely on the AB Testing tools for a raft of additional features. We can set interim goals, segment when visitors get entered into tests, generate graphs in a variety of formats, and do statisitcal analysis, both Frequentist and Bayesian.
      We’ll look at Angular for preload assuming it has compatibility across devices and browsers.
      Thanks for the suggestion.

      • Chris Desadoy says:

        I mean, obviously you guys really know your stuff, I was just curious to the tech, it seems like angular is a great solution for pre-load. Happy to maybe show you a new way./

        • Brian Massey says:

          If we’re understanding your approach, we’d need to restructure a client site to use the Angular loading approach effectively. Always open to new ideas. Have we missed something?


Trackbacks & Pingbacks

  1. […] change happens in the visitor’s browser. Each visitor gets the same website from the server. It’s the JavaScript that instantly transforms what they see into a test treatment, something different and […]

  2. […] a single page, you can make small tweaks that generate great results. Usually, A/B testing involves a thorough research of your entire […]

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *