Apply AB Testing Tactics to Before and After Testing
AB testing tactics can tell you whether your website changes are having a meaningful effect on your visitors. Before and after testing, sometimes referred to as BA testing, is similar, and it’s one of the easiest ways to learn how your design changes are impacting your visitors.
In this guide, you’ll learn what BA testing is, how it works, and how to get reliable results from before and after analysis.
What Is Before and After Testing?
Every change you make to your website is a test. Yet, changes are often made without analyzing the impact. Even small changes can have a material impact on your conversion rates.
Before and after testing (BA testing) is a type of conversion testing that evaluates the impact of changes to your website. As the name implies, it means comparing the performance of a new website or webpage to the previous version, so you know whether conversions increased or decreased after the change was pushed to production.
This can be as simple as looking at the number of conversions before the new webpage was launched and comparing it to the conversions after.
However, there are some problems with this approach that can lead you to the wrong conclusions. You might decide that the new design is an improvement, when in fact it materially lowered sales or leads. Alternatively, you may not notice a winning design because it looked like conversions went down when you launched it.
We are going to show you how to minimize the chances of making a bad call by using AB testing tactics for analyzing your changes.
Read the Complete AB Testing Guide here.
The Problem with Before and After Testing
The problem with before and after testing is that the results can be influenced by things that have nothing to do with the design change itself.
Here are some of the reasons leads or sales might drop after publishing a change to your website.
Your new design is not as good as the old one.
Thanks to AB testing, we have learned that even small changes to the layout of a page, copy, or images of a page can have a surprising effect on your visitors.
Your new design may not seem as credible to your visitors, making them hesitant to buy or submit a form.
You may tighten up your copy, eliminating information you felt was insignificant. Yet that information may have been important to a large segment of your visitors, leaving an important question unanswered for them.
Seasonality caused a drop in leads.
Many businesses have seasonal increases and decreases in their conversion rates. For example, most ecommerce websites enjoy more eager traffic during the holiday shopping season.
Businesses selling to other businesses (B2B) may experience an increase in leads at the end of the month, end of the quarter, or end of the year due to budgetary influences.
If you release your new design as these seasonal changes are happening, you may decide the new design is performing more poorly, when in fact the change was not the reason for the drop.
Your business changed its traffic mix.
We’d love to believe that the teams bringing traffic to the site are in close communication with the web development team. We know this is not always true.
For example, your paid search agency may change the bidding strategy, add keywords, or change ad copy at the same time you release your new design. This can have a material effect on the quality of the traffic, reducing your conversion rates.
Your email team may have changed the email schedule, reducing this highly qualified traffic.
Again, the change in the performance was not related to your design.
Your business ended a promotion.
It is not uncommon for new designs to be released at the beginning and end of promotional periods. This muddies the water when trying to ascertain if a new layout is truly worse, or if a discount was the reason for a change in sales.
A competitor increased their ad spend or started a promotion.
Your competitors will impact your traffic mix, and you may never know. If a competitor is siphoning off your prized ad traffic, you would expect your conversion rates to drop.
If this happened at the same time as your new design launched, you might conclude the new design is inferior to the original.
There was a technical error on the new design.
Sometimes a great new design is crippled by a bug, a glitch, or a longer load time. Your visitors may have preferred the new design and copy, but wouldn’t tolerate the slow or broken page.
AB Testing vs. BA Testing
The solution to the problems listed above is AB testing. In an AB test, the new design is shown to half of the traffic and the other half of the traffic sees the original. This ensures that any seasonal effects, changes in traffic, promotions, and competitor shenanigans impact both versions equally.
AB tests are designed to ensure that there are no differences between the original and new design. We can use the same AB testing tactics to give us more confidence in our before and after analysis.
With a little discipline, we can evaluate a new page against its predecessor with greater confidence in the conclusions.
Why Not Just Do an AB Test?
There are many reasons we will do a before and after test instead of an AB test.
- We do not have the tools and team to do AB testing.
- There are too many other AB tests running.
- We don’t think a change is impactful enough for an AB test.
- The change is temporary, but we still want to learn from it.
- The site doesn’t get enough traffic to run effective AB tests.
The Benefits of Before and After Analysis
There are some serious advantages to doing before and after analysis.
Before and after tests don’t require sophisticated testing tools.
The benefit of a BA test is that you don’t need to use any fancy AB testing tools. Your Analytics tool (such as Google Analytics) has the data you need.
Before and after tests don’t require sophisticated planning and setup.
While we are going to apply AB testing tactics to our before and after analysis, before and after testing doesn’t require all of the planning and setup required by an AB test.
You can go back and evaluate past changes.
As long has you know the dates of changes made to your website, and the specifics of the changes, you can determine how those changes are impacting your conversion rates. This is not true of AB testing.
Before and after analysis uses analytics reports, and you probably have data going back a year or more.
AB Testing Tactics to Use for Before and After Analysis
To get the best analysis of the change, we want to follow the rules that an AB test follows.
- The sample sizes should be similar.
- The traffic should be similar.
- We want to control for seasonality.
- We want to compare conversion rates, not conversions.
Create a segment for your change.
Not every change will be a sitewide change. Often your change will affect only a single page or a page template.
To increase the accuracy of the analysis, only consider traffic that has seen the pages on which the change was made.
For example, if you change the layout of the product page template for your ecommerce website, you will want to create a segment of visitors that includes only those sessions that saw at least one product page. If you changed something in the design of your site’s header, you will use all of the traffic in your analysis.
Select the proper parameters for your analysis.
You are going to compare the performance of a page or website for a period before and a period after the launch of a change. If you’ve just launched a design change to production, you will have to wait to generate data for the new design.
How long?
Ideally, you would wait a period of time long enough to generate at least 100 conversions. Since before and after testing is prone to errors, you may wish to double this number.
For example, if you had 80 conversions before your BA test and 120 conversions afterwards, it appears that the changes have had a positive impact on the conversion rate. But with such a small sample size, it’s hard to be sure. By waiting until you reach 200 conversions after your changes, you may see a different result. At the very least, it minimizes the margin of error in before and after testing.
Select appropriate before and after periods.
To gain reliable insights from BA testing, you must run the test for a sufficient duration. Select a period of time that’s long enough to capture a significant amount of data and account for any short-term fluctuations.
Consider selecting durations of at least a month or longer to ensure an adequate sample size of conversions.
1. Compare similar timespans.
The length of the time that you choose for the original “before” version of the page should be the same as for the new page.
Assuming that traffic is pretty consistent, this should give you a similar sample size for the before and after analysis.
BA Testing Tips
Evaluate similar traffic.
The kind of traffic you are receiving and the quality of the traffic coming to your changed page will impact your before and after analysis.
In the following example, the company began sending email at the same time that changes were launched. Email traffic typically converts at rates several times the site average. (Hint: Are you doing enough with email?)
Because of this, it will appear that the change made to the page caused a jump in conversions.
The solution here is to use only those traffic channels that remained consistent. In Google Analytics, use the Channels report to see if any traffic source changed.
Watch out for intra-week and intra-month seasonality.
Google Analytics 4 (GA4) allows you to compare two periods quite easily. But, you should be aware that there is “seasonality” within each week and, often, within a month.
In the following example, the before period has five weekend days. The after period only has four. Since conversions typical drop on weekends, this may skew your analysis in favor of the original design.
In this example, the before period covers the end of the month while the after period covers the beginning of the month. This can cause skew in certain kinds of businesses.
Beware of year-over-year analysis.
Year-over-year analysis is one way to control for seasonality when doing a before and after test. With this analysis, you choose the period from last year that maps onto the period from this year that includes your site changes.
Be sure to take into account big changes in site traffic and buyer behavior, such as those we saw in the pandemic. Visitor behavior changed when we were encouraged to stay at home. And as the virus surged and waned, that behavior changed.
These types of behavioral changes lead us to put less faith in year-over-year analysis.
Compare conversion rates, not conversions.
While we hope to minimize changes in traffic when selecting time periods for our before and after test, there will be changes. A drop in traffic may reduce conversions, even as changes to our pages makes the conversion rate go up.
We don’t want to make this mistake.
Be sure to choose the conversion rate to evaluate your change, not just the number of conversions.
For ecommerce businesses, you should evaluate Revenue per Visit. This metric combines changes in conversion rate plus changes in average order value (AOV) that your change may have impacted.
Look for BIG changes in performance.
Even with all of this careful selection of time periods, we are victims of changes over time that we cannot control. As I’ve pointed out, we cannot know if changes in competitor behavior or the sentiment of visitors have changed.
As a result, we cannot put faith in small changes from our before and after analysis.
For a well-designed A/B test, we set our maximum P-value at 0.05. This ensures we have a 95% or better confidence that any change we’re seeing between the two periods is not due to random chance. For a before-and-after analysis, I recommend setting the P-value to 0.01. This means that we want to be 99% sure that a change in conversion rate (or other relevant metric) is not just random chance.
How can we calculate this P-value? We’ll use an A/B test calculator.
6. Use an A/B Test Calculator for before and after analysis.
To get a statistical calculation of how our change affected our conversion rate, we can use one of the free A/B Test Calculators available on the web. We like the CXL Test Calculator.
Since we are analyzing existing data as if we had run an A/B test, we’ll choose “Test Analysis.”
We can then go to Google Analytics and create a report with the date ranges we’ve selected for the before and after periods. It might look something like this:
In this case, our report shows Conversions (purchases), Users and the conversion rate for purchases.
We will plug these numbers into our A/B test calculator.
We can see in the report that:
- We have a 15.8% lift from the original page (control) to the new page (variation). Is this a believable lift?
- Our P-value is less than 0.001, which is a confidence interval higher than 99%.
In this case we would be proud of our change and would keep the new variation on the site.
Here’s an example of a test that went the other way:
In this case, the change we made had a negative implact on the purchase conversion rate. We can switch to a “two-tailed” test to see if the variation is statistically worse. In this case the P-value is less than 0.001, or against the control with 99% confidence.
We would recommend that you go back to the original page.
What if the P-value isn’t < 0.01? For instance, what if it was 0.04? What if the Control was better with a P-value of 0.04?
We call this “inconclusive.” There is not enough evidence that the variation is worse than the control to eliminate it. In such a situation you can choose to keep the variation or roll back to the original. Even if the variation had a lower conversion rate, if it wasn’t statistically significant, either could be the long-term winner.
However, most of the time, people feel safer keeping the one with the higher conversion rate.
When to Use Before and After Testing
Before and after testing, or pre-post analysis is like having design insurance. Designers and IT people are all to eager to make changes to the website. However, many of those changes will inadvertently decrease marketing performance. This is a statistically valid way to ensure others aren’t working against the business.
You can learn something about your visitors from these tests. When you understand what increases (or decreases) the business performance of the website, you can infer the preferences of your visitors and customers.
Finally, before and after testing lets you “sign the flowers,” a way of saying that you can take credit for conversion-improving changes. If you’ve earned it, take it.
Get Help from the Conversion Scientists®
BA testing, AB testing, and designing experiments can be complicated and confusing. With Conversion Sciences, you have a skilled team of Conversion Scientists who can work with you or provide a done-for-you conversion optimization program.
Looking for AI optimization services from people who understand the critical importance of data? When you partner with Conversion Sciences, you get better conversion rates as well as a well-optimized process. Let’s chat.
Expertise: conversion rate optimization, AB testing, marketing strategy, digital marketing, analytics, landing page design, computer programming, entrepreneurship
Speaking: IBM, Inbound, LeadsCon, Content Marketing World, Affiliate Summit, and more
Books and Training: Your Conversion Creation Equation, Video CRO, Marketing Videos that Convert, CRO Best Practices
As Seen On: ClickZ, Search Engine Land, Marketing Land, Conversion Sciences Blog, Intended Consequences podcast
Education: Texas A&M University, BS Computer Science and Management
- Confirmation Bias: What It Is and How It’s Hurting Your Website Conversions - August 20, 2024
- The Conversion Optimization Process for High Converting Websites - August 20, 2024
- Two Guys on Your Website: The Surprising Link Between CRO and SEO - June 27, 2024
Leave a Reply
Want to join the discussion?Feel free to contribute!