What is A/B testing? How to use it to your advantage

What is A/B testing?

In short, A/B testing, A.K.A. “split testing”, is a testing method where the performance of two similar things are compared to see which performs better.

For example, if you were trying to optimize an email campaign using A/B testing, you would split your audience in half and send one half a standard email.

The other half would be sent an email very similar to the first but with some changes, whether they be large or small.

From there, you would monitor the audience’s interaction with both emails and determine which of the two is better based on which brings in more (positive) interactions.

“Interactions” can mean different things in different industries; in B2B it could be leads, in e-commerce it could be sales, for websites it could be page view time, so on and so forth.

Keep in mind that email marketing is just one example of how A/B testing can be used, it can also be used on webpages, for SEO, and a number of other things.

In more technical terms, A/B testing is data-driven decision making; you compare the performance of two elements and determine which of the two guarantees you a higher ROI, whether that return be time or money. This testing follows the scientific method, detailed below using the email campaign example:

Observation: You notice that an email campaign either isn’t performing as well as it has been previously or as well as you think it should.
Research topic area: You define your goals, in this case increasing the conversion metrics of your audience. This could mean viewing an email, clicking a button, visiting your website, etc.
Hypothesis: You start generating ideas for your A/B tests and why you think those tests will perform better than your current version. You could redesign your emails, change a heading, adjust colors, or a number of other things.
Test with experiment: You segment your audience and send half of them the original email, then send the other half the adjusted email.
Analyze data: You keep an eye on conversion metrics like open rate, viewing time, or if they followed any links in your email. Wait until your sample size is large enough for your interests, then see how the two emails performed in relation to one another.
Report conclusions: You say either that the original email performed better, or that the adjusted email performed better. From there, you adjust your campaigns to include the qualities that changed when creating the adjusted email.

A/B testing, graph — The benefits of A/B testing, via Optimizely.

Why should I use A/B testing?

A/B testing helps to identify leaks in the system, then patch them up in order to maximize conversions of whatever kind you’re looking for. If at any point in a sales funnel, there is a problem where potential converting visitors drop out, then conducting A/B testing on individual parts can help make sure you aren’t missing out on any conversions in your system.

Testing changes one at a time makes it very easy to figure out which factors influence visitor behavior most, and which ones had little or no influence at all.

This testing method also makes sure your audience has the best user experience possible, on top of making sure that your situation results in the most possible conversions and is as effective as possible for your investment.

A/B testing can also be used by designers or developers to determine the impact of a new feature or a change to user experience in scenarios like product onboarding or in-product experiences.

What mistakes should I avoid when A/B testing?

If done properly and following the methods mentioned above, A/B testing can be an invaluable asset to a business.

However, if done improperly, you’ve essentially doubled your effort for potentially no return.

A/B testing requires proper planning, precision, and patience in order to avoid wasting time and money. Here are some of the pitfalls to avoid when A/B testing, following the steps of the scientific method:

Incorrect/invalid hypothesis:

To combine the first three steps, an invalid hypothesis means that your entire experiment is based on something untrue or irrelevant, where you could then spend time and money making changes that ultimately hold no bearing over your conversion metrics.

If your initial observation concerning your conversion metrics is faulty or the research following that observation isn’t thorough enough, it’s very possible that your hypothesis could lead to a flawed experiment.

Testing multiple elements at once:

The best tests change one element at a time, thus making it very easy to determine what caused a change, whether it be positive or negative.

If you test while changing multiple elements at once, it’ll be much harder to attribute a result to a specific change, and you may have to test each change made individually after the fact to figure out which change was responsible for the outcome.

Testing with too little time/traffic:

When it comes to statistical significance, you have to give your experiments time to breathe in order to reach an actionable conclusion.

If your traffic is unbalanced between your A and B elements or if you don’t give the experiment enough time to get an adequate amount of traffic, then your results won’t be statistically significant and thus shouldn’t be acted upon.

Conclusion

This is just a short guide to A/B testing; this shouldn’t be taken as final or as the in-depth guide to A/B testing that other people have made. If you’re looking for something more in-depth on either applications or use cases of A/B testing, I recommend you read these articles by VWO and Optimizely.

A/B Testing: A Short Guide