What Is A/B Testing in Marketing?
A/B testing compares two versions of a page, ad, or email to see which performs better. The scientific method applied to marketing.
Key Takeaways
- A/B testing shows version A to half your audience and version B to the other half
- Statistical significance ensures the result is real, not random
- Test one variable at a time to know what caused the difference
- Run tests for long enough to capture a full weekly cycle of traffic
What A/B testing is
A/B testing (split testing) is a controlled experiment where two versions of a marketing asset — a webpage, email subject line, ad creative, or call to action — are shown to randomly selected halves of an audience simultaneously. By measuring which version achieves the better outcome, you gain objective evidence about what works rather than relying on subjective opinion or gut feel.
What can be tested
Almost any variable in a marketing asset can be tested. Common test variables: email subject lines (the single highest-impact email test), landing page headlines, call-to-action button copy and colour, product page images, pricing display format, checkout flow design, ad copy and visuals, social proof placement, and product recommendation algorithms.
Statistical significance
Statistical significance is the standard by which you determine whether the difference between your variants is real or due to chance. Typically, marketers require 95% confidence — meaning less than 5% probability the result occurred by chance. Most A/B testing tools calculate this automatically. A small improvement requires a very large sample to achieve significance; a large improvement requires a smaller sample.
Test one variable at a time
The fundamental rule of A/B testing is to change only one thing between the two variants. If you change the headline and the image and the CTA simultaneously and one version wins, you do not know which change caused the improvement. This is the most common A/B testing mistake. Test one thing. If it works, ship it. Then test the next thing.
Test duration
Running a test for too short a time is the second most common mistake. Traffic volume and conversion rates vary by day of week — consumer behaviour on Monday is different from Saturday. Best practice: run tests for a minimum of one full week (ideally two), regardless of whether statistical significance is reached earlier.