How to Interpret Your A/B Test Results

A/B test results are influential in making good decisions in site redesign because they help you see what elements are important for your audience and customers in a controlled environment. Once you have seen the test results, you can take the new knowledge to your site-wide design. Best of all, A/B tests can be lightning fast, as long as you know how to read your results and make the right choices.

This article will help you out with:

• Understanding random selection variants
• Conducting an A/B split test
• Reading confidence percentages
• Understanding the A/B testing process in Unbounce

Random Selection of Variants

When you run a split test between multiple variants (we'll say two in this case; A and B), you will need to assign a weight to each. This weight is the percentage of your overall traffic that you'd like to see either A or B.

Now, every time a new visitor comes to your page, we'll randomly serve up a variant based on the weight you have specified. If you've assigned both A and B a 50% weight, then this will be no different than a coin toss for each page request.

Something to note is that some imbalances may occur while the traffic to a page is still low. So, as in flipping a coin, you may get heads 5 times in a row.

We should also note that when a visitor first accesses your landing page, we deposit a cookie on that visitor's machine to track the view and ensure that they see the same variant each time they return.  If you visit your own page's URL, you might notice that you continually see the same variant no matter how many times you return or refresh your browser--this is because of the tracking cookie.  To work around this, clear your browser's cache and try again but keep in mind that you will then be counted as another 'new' unique visitor.

Confidence Percentage

The confidence rating displayed in the A/B Test Center indicates whether or not your challenger variant has achieved statistically significant results.  For those of you who are interested, this is a Chi Square test for significance.  A confidence rating of 95% or better is enough information to decide either 1) promote the variant to champion if it has the highest conversion rate, or 2) discard it if its conversion rate is less than your current champion.

Another way to think of the confidence rating is that it indicates how often, if you repeated the same experiment, you could expect to get differing results.  If you achieve a 90% confidence rating, there's a 1 in 10 chance that you might get different results if you ran the same test again.  With 95% confidence, your chances are 1 in 20, and with 99% confidence, they're 1 in 100.

To make sense of all this, think of your testing activities as managing an investment portfolio.  Let's say you ran ten tests, and the average conversion rate lift of those ten tests was 50%.  So, that might be taking a 10% conversion rate to 15%.  If all of those 10 tests achieved about 90% significance, you could reasonably expect that in 1 of those 10 tests, you didn't actually find the best performing page.  Now, if you're getting 50% returns on your testing investment, you have a hefty margin to absorb the possibility of being wrong that 1 time out of 10.  However, let's say you were only getting a 5% average lift.  In that case, being wrong 1 out of 10 times would almost wipe out your overall portfolio returns.

Do note: You'll want to wait until your A/B test is live for some time for the confidence rate to appear in the Test Center. The rating depends on the traffic volume, number of variants, etc. As a suggestion, wait for roughly two weeks from the time of your A/B test launch, to analyze those results confidently. 

Glossary

Unique Visitors: number of new visitors that arrive to your landing page gets counted once, regardless of how many times they go back to your landing page.

Views: number of times all visitors come to your landing page.

Conversion: the ultimate goal of your campaign (e.g., submitting a form, signing up, downloading a piece of content, etc.). Counts an individual's submission-only once regardless of how many times they submit.

Conversion rate: the percentage of people who complete your campaign goal.