The Core Pillars Of Creative Testing & How To Apply Them

Lidia Pérez is User Acquisition Manager at Socialpoint. Lidia began her career in user acquisition at From The Bench, a gaming startup. She worked there from 2014 to 2019, and her role eventually evolved to leading the Ad Monetization department and UA strategy. After a brief stint at a social gaming company, Lidia joined the word games team at Socialpoint. She currently drives growth for Word Life and oversees their creative testing framework & growth strategy.

Learn more about Mobile Hero Lidia.

In an increasingly complex and competitive market, it can be difficult to hit ROI and volume targets. To succeed, casual games with hybrid business models are looking for highly efficient CPIs. UA managers rely on creatives to drive CPI—that’s why at Socialpoint, we keep creative at the core of our growth strategy.

To empower our creative team, we built a comprehensive creative testing framework that systematized creative production in pursuit of maximum efficiency. I want to share some of the takeaways from our experience, so I put together this guide on how to build a robust system with well-defined processes that ensures all the stakeholders are involved.

1. Find the Best Testing Method

Choose The Right Partner

We all know that choosing the right testing partner is crucial. But how do you know if a partner is a good fit? When selecting a platform, ask yourself the following questions:

• Are there audience similarities with your main partner? Depending on the type of partner (i.e. social, network, programmatic), you may want to select different partners for creative testing.
• Do they have flexible targeting capabilities? For example, gender, age, app genre, app category.
• What testing tools are available? For example, Facebook counts with a well-known split testing feature. An advantage of using the split testing solution is that it ensures that the audience is equally distributed among the different content versions—this in turn ensures statistically comparable results. On the other hand, the disadvantage is that you are not testing in a real environment and cannot expect the algorithm to function optimally. You can never be 100% sure that success in split testing will mean success in the live environment.
• Can you choose different optimization events, ad-hoc budgets, fast-learning phases, or a broad enough audience to achieve competitive CPIs? Optimizing for a top-of-the-funnel event like installs will provide a lot of data points. This can feed the algorithm in a cheap way and give more margin for extra experiments. However, if your testing campaign and live campaigns are optimizing towards different events, you may experience misalignment in the results.
• Do they provide comprehensive creative reports? The reporting should have the main KPIs per individual asset.

Choose Your Country Wisely 

The ideal approach would be to test directly in your main market. This way, you can expect consistency when comparing test and live results. But it may not always be possible to test in your core market due to budget constraints.

Even when there are no budget restrictions, for the sake of cost efficiency, it is a good idea to explore your options. Run sanity tests to find out which countries are a good proxy for your core market. Being cost-efficient with creative testing means you can test a higher number of creatives simultaneously and acquire more data points. This way, you can expect higher significance in a shorter period of time.

Perform Quality Checks  

Depending on your optimization goals and the countries you choose for testing, you may wish to perform an extra quality assurance test before rolling out the creative winners across the board. Competitive acquisition costs are key, but they’re not the only important factor to take into account. Creatives with amazing IPMs are useless if users don’t engage with the game and have a high churn rate.

Ultimately, you’re aiming to balance CPI with quality. Perform extra QAs to avoid unnecessary risks in the live environment. That said, you will produce a lot of creatives, which means checking cost-quality goals could be expensive. Consider only checking the quality of creatives with higher potential.

2. Focus on the Numbers

Select the Right KPIs

We know that IPM is an important metric for evaluating creatives. However, sometimes, CPI could be a more holistic metric and a better proxy to creative success. This is the case when:

• You test different creative formats such as playables and videos. It is very common for videos to get a lower IPM than playables. But despite a slightly lower IPM, they may also be more viable when you look at CPI.
• You want to broaden your campaign testing audience to increase your reach. Looking at CPI could mean no restrictions when it comes to different ad placements. For example, in the case of Facebook Ads, you’ll see different IPMs depending on the investment allocation within their different placements (FAN, feeds…).

Analyze Test Results

Once you identify the creatives that have passed the testing phases, you can start rolling out the potential winners in the live environment. Here are some tips:

• Avoid a massive creative rollout across the board. Try a progressive approach. Start the submission process with your less “creative-sensitive” partner. Add potential winners to campaigns that are struggling the most, showing ad fatigue, or have the lowest budget. For main campaigns that are meeting ROI goals, leave the new creatives in the backlog. If something is working, let it run!
• One sign that your new creative is a winner is if a partner’s algorithm feeds investment to it. But keep in mind that some creatives that pass all the test filters will not perform in live campaigns. When this happens, leave the creatives in the backlog and give them a second chance in the mid-term. The market—and your app—are constantly changing. Creative testing results are not written in stone!

3. Include the Stakeholders: Performance Artists

Artists are a key piece of a growth team, but the skillsets they need are a little different from those of UI/UX specialists. It’s important to onboard artists in the performance marketing field so they can be fully aware of what UA needs in order to succeed. Train your artists to:

• Know the KPIs and how to deal with reporting.
• Understand the different types of partners you work with and their creative nuances (social, programmatic, networks).
• Know how to prioritize creative production (balance new concepts and variations) based on performance data. A Performance Artist needs to transform KPI learnings into appealing creative winners.

To assist a Performance Artist, create accessible reports tailored to their needs and knowledge levels in order to streamline analysis. And when it comes to KPIs, don’t overwhelm the team with unnecessary information, but ensure that they’re familiar with CTR, CVR, IPM, CPI, Retention Rate and ROI.

Track Learnings

To apply a data-driven approach properly, set up an effective management system. This could be as simple as creating an excel log to track each significant learning or hypothesis, next steps, and the results. Get everyone involved in the tracking, and make a habit of reviewing the log in weekly meetings.

Set Guidelines for Creative Ideation

Finally, to reduce “gut feeling” creative decisions, define guidelines around ideation and create a prioritization system. Start by identifying best-practice guidelines. Here are a few to help you get started:

• Choose main competitors and networks to identify market trends.
• Dive into your historical creative data and transform them into learnings.
• Work directly with account managers for expertise on each partner’s creative nuances.

The budget for “out of the box” creative production should be significantly lower than the budget for data-driven creative production. If we were suffering from ad fatigue—or we had tight budget constraints—I might even reduce the “out of the box” budget to $0.

Lidia features as a Mobile Hero duo with Danika Wilkinson from Socialpoint. Danika’s article covers how to align product, marketing and art teams for a successful user acquisition strategy. To learn more, read her article here.