🧪Real Experiment Case🧪

I think you guys are gonna love this 🧪𝐑𝐄𝐀𝐋 𝐄𝐗𝐏𝐄𝐑𝐈𝐌𝐄𝐍𝐓 𝐂𝐀𝐒𝐄🧪 simply because I can’t be more transparent than this 😂

Yesterday I saw one of the Growth Gems from Sylvain Gauchet on my TL and there was one slide that shocked me because I recently tested the hypothesis of that slide with one of my clients!

I am posting the slide with the gem that Flippos Drogitis wrote so you can see the hypothesis in question.

If you have been following me, you know that I love experimenting if I believe that experiments can help me to achieve the goals of my clients and as I mentioned, this specific hypothesis was something I tested in June with one of my most successful clients.

With this client (a mobile game), we have a payback period of 8-9 months on average but we have campaigns that are outperforming that benchmark and bringing positive results in a shorter period. One of these campaigns is the tROAS/VO campaign that we had in Google Ads using the default 30d attribution window.

𝗧𝗵𝗲 𝗵𝘆𝗽𝗼𝘁𝗵𝗲𝘀𝗶𝘀 𝗼𝗳 𝘁𝗵𝗲 𝗲𝘅𝗽𝗲𝗿𝗶𝗺𝗲𝗻𝘁 𝘄𝗮𝘀 𝗾𝘂𝗶𝘁𝗲 𝘀𝗶𝗺𝗽𝗹𝗲: I wanted to see if by using a shorter attribution window of 7 days for the purchase event as many games use, I could push the algorithm to bring better quality in less time so I created a new ad account and edited the attribution settings for the purchase event accordingly.

I ran the campaign for 2 weeks with a lower budget because it was going to help me control the efficiency of the delivery and get a better CPA (the old ad account started in the same way and got incredible results starting at a low pace). I also used the same creatives because I didn’t want to affect the hypothesis with more variables (I know this could somehow create some cannibalization but our budgets are humble so I was not worried about that).

The results are simply attached in the screenshots:

🔴CPA in the campaign using a 7D att window was triple that of the campaign using the default attribution window of 30D.
🔴ROAS was extremely worse in the 7d attribution window as we will recover the money of the main campaign in much less than the average payback period
🟢CPI was slightly better (29%) in the 7d campaign but we have to keep in mind that the 30d campaign was spending much more
🟢Retention was also better in the 7d campaign but that didn’t help to optimize better in terms of ROAS
🟡I got more discrepancies in Adjust with the 7d campaign (39%) than with the 30d campaign (4% only)

I thought this was a cool coincidence and I hope you find this experiment helpful!

I always recommend testing different attribution windows if your game needs more time to recover the investment so don’t be afraid and test!