Estimating the Incrementality of Your Paid Media Investments
Guilherme is Senior Director of User Acquisition at Mercado Libre, an e-commerce and fintech leader in Latin America. Before Mercado Libre, Guilherme worked at Wildlife Studios as Director of UA, at Bain & Company as a consultant, and at various Brazilian corporates as a general manager.
Learn more about Mobile Hero Guilherme. Read Guilherme’s blog in Portuguese here.
Why Audience Segmenting Matters
Performance marketers maximize business volume metrics by matching efficiency and budget constraints. A business volume metric—revenue, gross margin, or active users—usually comes from an attribution model that uses a rule to split the total volume across different marketing channels. Usual attribution rules include:
- Last click: give all volume to the marketing touchpoint that is the closest to the sale
- First click: the opposite of last click
- U-shape: places value on the first and last clicks and undervalues the clicks in between
Unfortunately, these attribution rules provide only vague answers to the central question: “what part of our sales only happens because of our ads?” The incrementality of paid advertising can help answer this question. Incrementality is the volume that would vanish from your company’s total numbers if paid is turned off.
Having an estimate for incrementality can be crucial to your campaigns, especially now. The target CPA or ROAS is usually set according to the unit acquisition cost a business can afford. Suppose attributed sales are above incremental ones. In that case, the actual CPA will be higher than what the company sees in its reports. Estimating incrementality in this scenario is essential for maintaining sustainable and profitable growth.
Recently, increased privacy for users has become a central concern—think Apple’s ATT, Google’s announcements of deprecating 3rd party cookies, and browsers that restrict access to user-level identifiers. These measures obscure the data needed by existing attribution models. Estimations of incrementality are, in this context, an important alternative for adjusting attribution.
Let’s now walk through an experiment to demonstrate how to estimate incrementality.
How to Estimate Incrementality
You can estimate incrementality through an A/B test. I have used geography as a bucket to separate comparable populations. One population receives ad impressions, and the other does not. Comparing total sales trends between regions will help estimate incrementality. Usually, during the test phase, you do not want large shifts in the biggest part of the market. For this reason, you should pick a test region substantially smaller than the control region. Your marketing team can use tools like the Causal Impact or Geolift packages to address this. Such tools help you use the trend from the control region (with marketing investment) to estimate what would have been the sales of the treatment region (no marketing investment). Results are similar to the ones observed below:
In this case, the treatment of turning off performance marketing investments is visible in the chart as the difference between the green and purple lines after the experiment started (marked by the red line). This difference represents the sales attributable to paid performance marketing.
With the last click rule, our attribution proxy attributed almost double the sales to performance marketing. This led to a combined effort between marketing and finance to revisit the cost of acquisition decisions. These decisions later prompted attribution model adjustments and contributed to profitable growth for the company.
Before we move on, I want to stress the importance of two enablers for the success of the test:
- Involve your data science team early on. The wrong experiment design wastes time, energy, and money.
- Suppose you want to detect total sales movements caused by paid investment changes. In that case, such changes must be representative (usually investment deltas higher than ~30%).
Going beyond the Incrementality Experiment
As mentioned above, adjusting bidding decisions and calibrating attribution models are the most immediate actions. This captures the first-order effects, but it is not enough.
The second step in this journey is transforming this ad-hoc experiment into a day-to-day practice. Incrementality changes over time. So it’s critical to experiment continuously to capture new fluctuations and turn them into bidding decisions. Beyond that, it is possible to de-average conclusions by doing specific channels and countries, always ensuring that the changes in paid will generate detectable moves in total sales.
In a nutshell, it is important to start even if the experiment looks complex. Always remember that transformation begins with small steps in the right direction!