How is incrementality testing different from A/B testing?

A/B testing compares two variations of a creative, landing page, or experience. Incrementality testing compares the presence of marketing against its absence to measure causal impact. An A/B test asks which version works better; an incrementality test asks whether the activity works at all.

How long should an incrementality test run?

Typically two to four weeks, but the right duration depends on your conversion volume. You need enough conversions in both groups to reach statistical significance, usually at least a few hundred per group. Low-volume businesses may need longer test periods or larger audience splits.

Which channels should you test for incrementality first?

Start with your highest-spend channels, especially those where attribution credit is ambiguous, like brand search or retargeting. These are the channels most likely to be over-credited by last-touch attribution. Proving or disproving their incremental value directly impacts your largest budget line items.

Marketing IntelligenceAdvanced5 min read

What Is Incrementality Testing?

Incrementality testing measures the true causal impact of a marketing activity by comparing outcomes with and without it. Learn how to run these tests.

Key Takeaways

Incrementality testing isolates the causal effect of marketing by comparing a test group to a control group.
It answers whether a campaign created new conversions or simply captured demand that would have happened anyway.
Well-designed tests require statistical rigour in sample sizing, randomisation, and duration.

What incrementality testing measures

Incrementality testing determines whether a marketing activity actually causes additional conversions or merely takes credit for conversions that would have occurred regardless. It works like a scientific experiment: you expose one group to the marketing activity (test) and withhold it from a similar group (control). The difference in outcomes between the two groups is the incremental lift. This is the gold standard for understanding true marketing effectiveness.

How to run an incrementality test

Select the campaign or channel you want to evaluate. Split your audience or geography into test and control groups using randomisation to eliminate bias. Run the test for long enough to achieve statistical significance, typically two to four weeks depending on conversion volume. Measure the conversion rate difference between groups. If the test group converts at 5% and the control at 3%, your incremental lift is 2 percentage points, or roughly 40% of test group conversions.

Types of incrementality tests

Ghost ads or intent-to-treat tests show a public service ad to the control group instead of your ad, measuring the difference in behaviour. Geo-based tests compare regions where a campaign runs against similar regions where it does not, which is useful for offline or broad-reach media. Holdout tests suppress a percentage of a retargeting audience to measure true retargeting lift. Each method suits different channels and objectives.

Common pitfalls

Running tests that are too short or on audiences that are too small produces unreliable results. Contamination between test and control groups, where control users are accidentally exposed to the campaign, undermines validity. Testing during unusual periods like major holidays or product launches introduces confounding variables. Start with high-spend channels where even a small percentage improvement in efficiency yields meaningful budget savings.

What Is Attribution Modelling?4 min · Intermediate What Is Marketing Mix Modelling?5 min · Advanced What Is Content Marketing ROI?4 min · Intermediate