Hypotheses: Deciding what to Test
Today’s question is at the heart of AB testing. “How do you decide what elements of a site to test?” We call the test “hypotheses.”
But, a better question is, “How do you determine what NOT to test.”
It’s relatively easy to come up with ideas that might increase your conversion rate. We typically come up with fifty, seventy-five, one-hundred or more ideas for each of our client sites. Filtering through this list is the hard part.
The Five Steps
In this week’s podcast, I take you through the five steps we use to determine what to test on a website.
- Step One: Look for Evidence
- Step Two: Rate the Traffic
- Step Three: How Hard is it to Test?
- Step Four: What does experience tell you?
- Step Five: Bucket the Winners
We’re pretty good at picking low-hanging fruit. Last year 97% of our clients continued working with us after our initial six-month Conversion Catalyst program that uses this approach.
Each of our hypotheses gets an ROI score using the following formula:
ROI = Evidence + Traffic Value + History – Level of Effort
Once we’ve ranked all of our hypotheses, we classify them into buckets.
The top ten hypotheses reveal an interesting pattern when you bucket them.
Bucketing Your Hypotheses
I also talk about how we classify hypotheses into buckets.
- User Experience: For hypotheses that would alter the layout, design, or other user interface and user experience issues.
- Credibility and Authority: For hypotheses that address trust and credibility issues of the business and the site.
- Social Proof: For hypotheses that build trust by showing others’ experiences.
- Value Proposition: For hypotheses that address the overall messaging and value proposition. Quality, availability, pricing, shipping, business experience, etc.
- Risk Reversal: For hypotheses that involving warranties, guarantees and other assurances of safety.
This helps us understand what the primary areas of concern are for visitors to a site. Are there a lot of high-ranked hypotheses for Credibility and Authority? We need to focus on building trust with visitors.
There’s much more detail in the podcast and my Marketing Land column 5 Steps to Finding the Hidden Optimization Gems.
Expertise: conversion rate optimization, AB testing, marketing strategy, digital marketing, analytics, landing page design, computer programming, entrepreneurship
Speaking: IBM, Inbound, LeadsCon, Content Marketing World, Affiliate Summit, and more
Books and Training: Your Conversion Creation Equation, Video CRO, Marketing Videos that Convert, CRO Best Practices
As Seen On: ClickZ, Search Engine Land, Marketing Land, Conversion Sciences Blog, Intended Consequences podcast
Education: Texas A&M University, BS Computer Science and Management
- Your Traffic Fell 30 Percent. Your Targets Did Not. - September 3, 2026
- How to Filter Bot Traffic in Google Analytics With Ask Advisor - August 20, 2026
- Using AI to Chat with your Online Business - July 10, 2026
