How To Create Testing Hypotheses That Drive Real Profits: 8 Simple Steps

Here’s what he wrote back:

I think it needs a more home-like homepage. It looks like it’s jumping in to the “pick a game” attitude a bit early. A homepage that shows the most played and featured games that also includes tabs at the top listing the different genera. This is how I would like to look at the first page of a website.

(This is verbatim. He uses the word “genera,” which is the plural of the word “genus” and is used to classify populations. I had to look it up. Doesn’t this make you think that conversion science is a genetic thing?)

Many who looked at the site would come to a similar conclusion. Conversion expert Tim Ash says, “The job of the home page is to get people off of the home page.” This means the homepage must “help” me.

I found the following suggestions in my son’s feedback:

1. Make the homepage look more “home-like.”
2. Don’t ask visitors to pick a game right away.
3. List “Most Played” and “Featured” games.
4. Categorize games into genera.
5. Use tabs across the top to select genera.
6. Design the homepage for my son.

No doubt, you are getting suggestions from your team stated in very similar ways.

Let’s craft some hypotheses from our Jr. Conversion Scientist’s statements. Then you can turn your team’s input into something that you can actually test.

1. Follow The Hypothesis Form

A hypothesis takes the form:

If _________________________, then _______________________ as measured by _________________.

Plugging in our first statement, we get:

If we make the homepage more home-like, then more visitors will play a game as measured by homepage bounce rate.

The testing team won’t know what to do to make the homepage more “home-like.” Here’s how we whittle it down into a testable hypothesis.

2. Only Change One Thing

There are a lot of ways to make a homepage more “home-like.”

We can only change one thing in a hypothesis. We may design a test that presents multiple hypotheses, but each hypothesis must be specific and unambiguous. My son has some more specific suggestions: identify the most played, identify featured games, and list games by category. A more “home-like” homepage may also have side navigation, hero images, and more.

How do we choose?

3. Find Proof In Analytics

Through the site’s analytics data, we should be able to see which games are more popular than others. “Popularity” may be measured by visits to those games’ pages or by the time spent on individual game pages.

However, we know that visitors tend to click on those items near to top-left corner of a page. Is there a correlation between the order that the games are listed and the popularity of the games?

By adding tools like Crazy Egg, Clicktale, and SessionCam, we can see if visitors favor games at the top of the page. These tools offer “scroll maps” which tell you how far down visitors are scrolling.

Finally, we may want to look only at new visitors, since their behavior is often very different from returning visitors looking for a specific game.

Together, this information gives us a more specific hypothesis:

If we list the most popular games at the top under the title “most popular,” then more visitors will play a game as measured by homepage bounce rate.

4. Split Hypotheses

Smart readers will realize that the hypothesis above violates our first law: Only change one thing. As written, the hypothesis changes both the order of the games and adds the label “Most popular.” We need to test one or the other. In this situation, we can simply split the hypothesis into multiple hypotheses:

Hypothesis 1: If we highlight the most popular games, then …

Hypothesis 2: If we place the most popular games at the top of the page, then …

This is more challenging to implement, but is very important.

5. Avoid Predictive Metrics

The site we’re discussing makes money by selling advertising. This means that it generates revenue when gamers click on ads. While a high bounce rate on the home page may indeed lead to fewer ad clicks and lower revenue, it is a predictive metric. This means that it predicts the success of a website, but doesn’t actually measure it.

We like the same metrics that the accountants like: revenue, leads, subscribers and ad clicks.

We will need to track ad clicks on a per-visitor basis, meaning the reports we get from our ad network will not be specific enough. Our approach will be to track ad clicks across the site and use this as our metric of choice. We can set this up in most split testing solutions, such as Optimizely, Convert.com, Visual Website Optimizer, or Test and Target.

If, for some reason, we weren’t able to track clicks, we might select a more specific predictive metric. Measuring visits to game pages or tracking ad impressions for each visitor might estimate reality sufficiently. But, don’t bet on it.

6. Impact As Many Visitors As Possible

Our hypothesis needs to be “big” enough to make a difference. If my brilliant son had said, “They need a better ‘About us’ page,” we would run into trouble as “About us” pages are typically visited by a small portion of visitors.

Pages with little traffic can lie to us. A game page with 20 visits may have generated 10 ad clicks. This indicates a 50% ad click rate, which is huge. But, the samples size of 10 conversions is too small to bank on.

If the sample size is small, only very big improvements can be trusted.

Hypotheses on shopping carts, registration pages, and category pages are more likely to deliver the sample sizes you need to discover winning changes.

7. Pick the Right Outcome

Similar to picking the right metric, you’ll want to pick an outcome that is important to the business.

This business lives by advertising revenue. While getting visitors to play games is an outcome that predicts success, the issue is getting them to play more games during each visit or to play longer when they come.

For this site, ad clicks and ad impressions will be the defining goals. Both of these can be monetized with advertisers that want to reach gamers.

8. Write It Out

If I were to write out the hypotheses from my son’s recommendations, they might look something like this:

If we added links to game categories to the homepage, then ad revenue would increase as measured by ad-clicks per visit.

If we labeled most popular games on the homepage, then the number of first time visitors clicking ads would increase as measured by ad-clicks per new visitor.

If we put tabs on the homepage with game categories, then the ad impressions visitors see would increase as measured by ad impressions per visit.

If we listed featured games on the homepage, then more new visitors would see an ad as measured by ad impressions per visitor, new visitor segment.

Doing This Will Change Your Thinking

It is common for someone to come to you with ideas for improving the website. When they do, you will have a hypothesis checklist to think through.

What impact will it have on the business? If the answer is “can’t tell,” then it’s not a good idea.

How many things would change if we tested the idea? If the answer is, “a lot,” then you have a lot of hypotheses to sort through.

Is there enough traffic to test? You can test low-traffic pages, but the approach is different.

How would we measure the results? If the answer is, “I don’t know,” do some more research.

Is there evidence to support the idea? If not, find something that does.

Knowing how to create a valid split test hypothesis will increase your critical thinking power, help you prioritize your marketing tasks, and give you a way to talk others out of bad marketing ideas.