What is A/B testing?
In short, A/B testing, A.K.A. “split testing”, is a testing method where the performance of two similar things are compared to see which performs better.
For example, if you were trying to optimize an email campaign using A/B testing, you would split your audience in half and send one half a standard email.
The other half would be sent an email very similar to the first but with some changes, whether they be large or small.
From there, you would monitor the audience’s interaction with both emails and determine which of the two is better based on which brings in more (positive) interactions.
“Interactions” can mean different things in different industries; in B2B it could be leads, in e-commerce it could be sales, for websites it could be page view time, so on and so forth.
Keep in mind that email marketing is just one example of how A/B testing can be used, it can also be used on webpages, for SEO, and a number of other things.
In more technical terms, A/B testing is data-driven decision making; you compare the performance of two elements and determine which of the two guarantees you a higher ROI, whether that return be time or money. This testing follows the scientific method, detailed below using the email campaign example:
Why should I use A/B testing?
A/B testing helps to identify leaks in the system, then patch them up in order to maximize conversions of whatever kind you’re looking for. If at any point in a sales funnel, there is a problem where potential converting visitors drop out, then conducting A/B testing on individual parts can help make sure you aren’t missing out on any conversions in your system.
Testing changes one at a time makes it very easy to figure out which factors influence visitor behavior most, and which ones had little or no influence at all.
This testing method also makes sure your audience has the best user experience possible, on top of making sure that your situation results in the most possible conversions and is as effective as possible for your investment.
A/B testing can also be used by designers or developers to determine the impact of a new feature or a change to user experience in scenarios like product onboarding or in-product experiences.
What mistakes should I avoid when A/B testing?
If done properly and following the methods mentioned above, A/B testing can be an invaluable asset to a business.
However, if done improperly, you’ve essentially doubled your effort for potentially no return.
A/B testing requires proper planning, precision, and patience in order to avoid wasting time and money. Here are some of the pitfalls to avoid when A/B testing, following the steps of the scientific method:
To combine the first three steps, an invalid hypothesis means that your entire experiment is based on something untrue or irrelevant, where you could then spend time and money making changes that ultimately hold no bearing over your conversion metrics.
If your initial observation concerning your conversion metrics is faulty or the research following that observation isn’t thorough enough, it’s very possible that your hypothesis could lead to a flawed experiment.
Testing multiple elements at once:
The best tests change one element at a time, thus making it very easy to determine what caused a change, whether it be positive or negative.
If you test while changing multiple elements at once, it’ll be much harder to attribute a result to a specific change, and you may have to test each change made individually after the fact to figure out which change was responsible for the outcome.
Testing with too little time/traffic:
When it comes to statistical significance, you have to give your experiments time to breathe in order to reach an actionable conclusion.
If your traffic is unbalanced between your A and B elements or if you don’t give the experiment enough time to get an adequate amount of traffic, then your results won’t be statistically significant and thus shouldn’t be acted upon.
This is just a short guide to A/B testing; this shouldn’t be taken as final or as the in-depth guide to A/B testing that other people have made. If you’re looking for something more in-depth on either applications or use cases of A/B testing, I recommend you read these articles by VWO and Optimizely.