How to Interpret Your A/B Test Results

A/B test results are influential in making good decisions in site redesign because they help you see what elements are important for your audience and customers in a controlled environment. Once you have seen the results of a test you can take the new knowledge to your site wide design. Best of all, A/B tests can be lightning fast, as long as you know how to read your results and make the right choices.

This article will help you out with:

  • Understanding random selection variants
  • Conducting an A/B split test
  • Reading confidence percentages
  • Understanding the A/B testing process in Unbounce

Random Selection of Variants

When you are running a split test between multiple variants (we'll just say two in this case; A and B) you will need to assign a weight to each. This weight is the percentage of your overall traffic that you'd like to see either A or B.

Now, every time a new visitor comes to your page, we'll randomly serve up a variant based on the weighting you have specified. If you've assigned both A and B a 50% weight, then this will be no different than a coin toss for each page request.

Something to note is that some imbalances may occur while the traffic to a page is still low. So, as in flipping a coin, you may get heads 5 times in a row.

We should also note that when a visitor first accesses your landing page, we deposit a cookie on that visitor's machine to track the view and to ensure that they see the same variant each time they return.  If you visit your own page's URL, you might notice that you continually see the same variant no matter how many times you return or refresh your browser--this is because of the tracking cookie.  To workaround this, just clear your browser's cache and try again but keep in mind that you will then be counted as another 'new' unique visitor.


Confidence Percentage

The confidence rating displayed in the A/B Test Center indicates whether or not your challenger variant has achieved statistically significant results.  For those of you who are interested, this is a Chi Square test for significance.  Most times, a confidence rating of 95% or better is sufficient to make a decision to either promote the variant to champion if it has the highest conversion rate, or to discard it if it's conversion rate is less than your current champion.

Another way to think of the confidence rating is that it indicates how often, if you repeated the same experiment, you could expect to get differing results.  If you achieve a 90% confidence rating, there's a 1 in 10 chance that if you ran the same test again, you might get different results.  With 95% confidence your chances are 1 in 20, and with 99% confidence they're 1 in 100.

To make sense of all this, think of your testing activities as managing an investment portfolio.  Let's say you ran ten tests, and the average conversion rate lift of those ten tests was 50%.  So, that might be taking a 10% conversion rate to 15%.  If all of those 10 tests achieved about 90% significance, you could reasonably expect that in 1 of those 10 tests, you didn't actually find the best performing page.  Now, if you're getting 50% returns on your testing investment, you have a hefty margin to absorb the possibility of being wrong that 1 time out of 10.  However, let's say you were only getting a 5% average lift.  In that case, being wrong 1 out of 10 times would almost wipe out your overall portfolio returns.


Glossary

Unique Visitors: number of new visitors that comes to your landing page gets counted once, regardless of how many times they go back to your landing page.

Views: number of times all visitors come to your landing page.

Conversion: the ultimate goal of your campaign (e.g., submitting a form, signing up, downloading a piece of content, etc.). Counts an individual's submission only once regardless of how many times they submit.

Conversion rate: the percentage of people who complete your campaign goal.

 

Back to: Testing Your Landing Pages >>

Was this article helpful?
0 out of 0 found this helpful
Powered by Zendesk