Proving that Website Testing Works [Case Study]

What would happen if you stopped optimizing today?

The Light Switch Test

We had been working with a client in the addiction treatment center marketplace for three years. Our website testing has generated a steady increase in phone calls from their website. In this space a phone call is valued over an online form fill by about 7 to 1.

At one point, we saw seen a sudden, deep drop in leads overnight. This wasn’t a change in traffic makeup. The conversion rate dropped simultaneously with the drop in leads. There was no Google algorithm change to blame here.

As it turns out, this was an accidental “Light Switch” test. To do this type of test, you have to be able to revert all of your changes back to one of your controls. You then turn your changes off to see what happens.

When you’re testing, the change in performance — the lead conversion rate or revenue per session — tells you how your changes have affected the bottom line. Then you turn them all back on, expecting a return to the optimized performance.

It’s like flipping a light switch off and on to see what light(s) it controls.

This is a test we would never recommend. It can be very expensive, and its only use is to validate the optimization efforts. You don’t learn anything new from it. In this case, it occurred inadvertently.

Here’s how it happened.

How to Undo Your Website Testing

We had found several winning changes in the month prior to this. We were using the split testing tool to “harvest” leads at the higher conversion rate until the client made these changes permanent. We were showing 100% of this client’s visitors the winning treatments using our testing tool and Javascript to morph their site for every visitor.

During routine maintenance, the client inadvertently removed the script for the testing tool from the site. Immediately, all of our changes had been removed. The light switch had been turned to “off,” and the site was almost exactly like it had been months before.

The conversion rate dropped to almost the exact level it had been before we started testing, and the drop matched the lift that we had claimed from our analysis.

It was several hours before the light was turned back on. As expected, the conversion rate returned to normal.

This kind of validation is rare. But it demonstrates that our process was working as expected for them.

Comments are closed.