How Big is the Optimizely "Test Snooping" Problem
How helpful would it be to know what prices and features your competition was thinking about using?
One of my readers just sent me a very revealing screenshot. It is one of the pricing pages that Optimizely is testing. It was found by “spying” on their test data.
We hid the pricing on this test treatment from Optimizely
We are able to see this because of an “exploit” that allows anyone to see what a site is testing if they are using the Optimizely testing software. Oh, the irony.
Venture Beat recently “
Try dragging the following link to your browser bookmark bar.
Optimizely Spy
Now visit Optimizely and click on the bookmark to see what they are testing.
How is this possible?
Whenever we run a split test with Optimizely, the software uploads scripts and data into all of our visitors’ browsers to change the experience and track the results. Along with this is included not just the test our visitor is being entered into, but all of our tests for that account.
So it’s relatively easy to decipher this information and see what we’re testing.
Note that the snooper can’t see any actual results, just what kinds of things you’re testing.
We like this approach because it speeds up the delivery of tests. When we use one file with everything, it changes less frequently, and the file it can be cached on a content delivery network (CDN) specifically designed to deliver files faster.
Faster tests mean more reliable tests.
Convert.com also uses this technique, though they take steps to obsure the test information.
Why Aren’t We More Concerned?
In a worst case scenario, a competitor can see what hypotheses you are testing. They can then test those same ideas and perhaps win more customers.
However, only a small percentage of sites are even testing, let alone stealing your tests. I did a quick survey of sites selling plastic surgery and cosmetic surgery who are spending at least $500 per month on search advertising.
Of 2,958 domains, only 33 had some form of split testing software installed, such as Optimizely. That’s just 1.1% of these domains. Furthermore, we know that some portion of these testing are not actually using the software they have installed.
Plastic and Cosmetic Surgery websites are missing a significant opportunity to get more patients. Source: SpyFu.com
Here’s another surprise. There are ninety-seven (97) domains in this space spending over $50,000 per month on search ads. Only five of them have A/B Testing software installed, only 5%.
If you’re in the plastic surgery space and are testing, you have a major advantage over your competitors. So, the odds of someone stealing your ideas are far outweighed by the gains you will see from testing.
Our Recommendation
We recommend that you continue to test using Optimizely unless your page contains sensitive information, such as price.
If you feel uncomfortable with your test information being publicly available, move to Convert Experiments for some protection. Another popular tool, Visual Website Optimizer, does not use this technique meaning past and future tests are safe from prying eyes. There are also a variety of other highly recommended AB testing tools available.
Whatever you do, don’t let this issue take the steam out of your testing program. As you can see, testers have a significant advantage, snoopers or not.
PS: If you are in the plastic and cosmetic surgery industry, you should contact us.
Expertise: conversion rate optimization, AB testing, marketing strategy, digital marketing, analytics, landing page design, computer programming, entrepreneurship
Speaking: IBM, Inbound, LeadsCon, Content Marketing World, Affiliate Summit, and more
Books and Training: Your Conversion Creation Equation, Video CRO, Marketing Videos that Convert, CRO Best Practices
As Seen On: ClickZ, Search Engine Land, Marketing Land, Conversion Sciences Blog, Intended Consequences podcast
Education: Texas A&M University, BS Computer Science and Management
- Confirmation Bias: What It Is and How It’s Hurting Your Website Conversions - August 20, 2024
- The Conversion Optimization Process for High Converting Websites - August 20, 2024
- Two Guys on Your Website: The Surprising Link Between CRO and SEO - June 27, 2024
So Brian, what I would like to know is how you were able to identify the 2958 domains spending $500 per month in AdWords (or the 97 spending $50k per month). I’m not familiar with a tool to do that. Did you use a publicly available tool for that?
I used data from http://www.SpyFu.com. Check it out.
These are really salient points and a good balance of the pros and cons. We also use Optimizely quite a bit and aren’t concerned by this. Also I’ve picked up quite a few great tips over the many years you’ve been blogging about conversion rate testing so thanks for that too!
Thanks. It’s always great to hear from readers.
Optimizely has taken steps to partially mitigate this issue http://blog.optimizely.com/2014/06/05/3-new-optimizely-settings-to-give-you-more-control/
Thanks, Leho.