How Incrementality Testing Works
Over the past several months, a number of Liftoff customers expressed interest in learning more about incrementality testing, often with a misconception of what it is and how it works. To clear up these misconceptions, we thought it we thought it would be helpful to dig into the topic.
What is an incrementality test?
Running an incrementality test provides knowledge around the incremental value added by a marketing campaign alongside your other marketing efforts. For example, if you want to understand the incremental value provided by adding in-app advertising alongside your web campaigns, an incrementality test will provide the answer.
Why run incrementality tests?
There are a number of reasons you might want to run an incrementality test. These reasons might include: (1) to identify the lift provided by marketing campaigns against organic traffic, (2) to identify the lift of an advertising campaign compared to other ad campaigns, (3) demonstrating the lift that an ad campaign delivers against a given audience, and other less common reasons.
The most common reason Liftoff hears for running an incrementality test is to demonstrate the lift that paid user acquisition has on incremental growth on top of organic installs. And most incrementality tests that we run at Liftoff are designed to answer this question. We also receive requests to run incrementality tests to demonstrate the incremental lift from serving Liftoff ads to a given audience. Do the ads work? Does machine learning have a meaningful impact? Running an incrementality test helps answer these questions.
How does Liftoff run incrementality tests?
Running incrementality tests is complicated. There’s no way around it. To keep things simple we start by splitting the target audience into two distinct groups – a control group and a test group. The control group will be shown public service announcement ads (PSAs) and gives us a baseline with which to compare the test group. The test group will be shown Liftoff Dynamic Ads.
A key to incrementality tests is making sure that each group of users is drawn from the same user pool, and that they are randomly selected for either of our two groups. This allows our incrementality tests to have statistical significance and compare Liftoff driven users to organic users in a meaningful way.
Why are these specific groups of users important? If we do an incrementality test on purely randomized users, without using Liftoff’s machine learning platform, this simply shows the incremental value of showing ads. If we do an incrementality test on users that Liftoff believes will convert, this once again shows the incremental value of using ads, but against a group of users more likely to convert which, by the way, is what we do for our creative A/B tests.
Now, if we want to show the incremental value that a Liftoff driven campaign will have, against all users, we need to make sure that our control group is made up of randomized users, some of which may convert (marked as organic conversions). We also make sure that our test group contains users that Liftoff thinks are going to convert. This will determine if Liftoff is actually doing a good job at identifying users that are going to convert and actually driving conversions.
The reason that the control group is shown a PSA is so that we can track their actions, and show attribution for the organic users, i.e. if those users download the app and perform any desired actions. Once we have a good understanding of organic conversions vs Liftoff driven users, or once the incrementality test has reached statistical significance, we can calculate the incremental value provided by Liftoff driven campaigns. The incremental value is calculated by finding the ratio between the conversion rates of the control group and the test group.
One important detail here is that by showing ads and PSAs that can be tracked using your MMP, you can track the progress of the incrementality tests using your MMP. Here at Liftoff we want to be sure that we are completely transparent in our test findings. By running the tests in this manner you can be sure that our findings are accurate.
If you have questions about running incrementality tests, please reach out to us and we will be more than happy to answer your questions.