While experiments are often used to validate existing strategies, they are just as powerful when used to challenge assumptions and uncover new opportunities. At Objective Platform, we help marketers design experiments that move them from gut feeling to grounded insight and from reactive optimisation to proactive strategy.
In this example, a telecom provider faced an unexpected disruption in their media mix and turned it into an opportunity for learning.
Social media drives meaningful incremental value in the media mix. That was one client's long-standing internal assumption. Like many brands, they invested consistently in social media advertising but could not clearly quantify its impact.
When unforeseen circumstances in early 2022 resulted in a temporary pause of social media campaigns, they saw a rare opportunity to run a natural experiment. Using Objective Platform, the team compared performance during the pause with historical benchmarks and modelled expectations.
Social campaigns were paused across all products for a defined period. Using Objective Platform, the team compared performance during the pause with historical benchmarks and modelled expectations. They focused on three key products and monitored shifts across all media channels including Display, Programmatic, SEA and Affiliate.
The findings varied by product, highlighting how media mix effectiveness is context-dependent.
Product A: SEA saw increased conversions, absorbing some of the traffic lost from social media. Display and Affiliate, however, declined — suggesting partial compensation but not full recovery.
Product B: All channels experienced a decline in conversions, indicating that none could effectively compensate for the absence of social. This was a strong signal that social media had been undervalued.
Product C: Display showed a slight increase but overall performance declined, again pointing to the important role of social media in the mix.
These results challenged the assumption that social media spend could be cut without consequences. Instead, they revealed that social media was more integral to performance than previously thought, especially for Products B and C.
The experiment proved that assumptions, even long-held ones, are worth testing. Rather than cutting or increasing spend without evidence, the provider was able to ground future media planning in observed outcomes. The nuanced results across products also illustrated that channel value is not uniform, what holds for one product in the mix may not hold for another.
Objective Platform enables teams to do more than model performance, they can test it in the real world. By building learnings from natural experiments into your MMM, you can uncover channel dependencies that would otherwise remain invisible in aggregate attribution data. You can quantify the true cost of disruption before it becomes a budget decision rather than after. And you can rebalance your media mix with greater confidence, because the inputs to your model are grounded in causal evidence rather than correlation alone.
This kind of insight helps marketers align spend with impact and adapt quickly when market conditions change. For a parallel example using geographic rather than audience-based experiment design, see Validating Marketing Assumptions with Geo-Lift: Branded Search in Focus.
Planned experiments are valuable, but natural experiments — where an unintended disruption creates a real-world control condition — can reveal things that planned tests cannot easily replicate. A social media pause affecting all products simultaneously, across a defined period, creates a clean before-and-after comparison that would be difficult to design artificially without significant budget risk.
The telecom case study above is a good example of this. The disruption was not welcome, but it produced evidence that changed how the business valued a channel it had been investing in for years without being able to fully justify. That evidence is now part of the MMM, improving the accuracy of future scenario planning and budget decisions.
When disruptions happen — channel outages, budget freezes, seasonal gaps in activity — treating them as measurement opportunities rather than just operational problems is one of the most practical ways to build a stronger evidence base over time.
For a guide to how brand health measurement connects to the channel dependency findings in this case study, see 5 Reasons Why You Need to Measure Your Brand Health.
What is an A/B lift test in marketing?
An A/B lift test measures the incremental impact of a marketing channel or tactic by comparing the performance of a group exposed to the activity against a group that is not. The difference in outcomes between the two groups represents the incremental lift attributable to that activity. Unlike platform-reported attribution, lift tests provide causal evidence of a channel's contribution rather than a correlation-based estimate.
What is a natural experiment in marketing measurement?
A natural experiment occurs when real-world circumstances — rather than a deliberate research design — create a situation that approximates a controlled test. In the case study above, an unplanned social media pause created a natural control condition: a period without social activity that could be compared against historical benchmarks and modelled expectations. Natural experiments are valuable because they produce causal evidence without requiring artificial budget manipulation.
How do A/B lift tests improve Marketing Mix Modelling?
A/B lift test results can be fed directly into an MMM as calibration inputs, anchoring the model to causal evidence rather than relying entirely on observational data. This improves the accuracy of ROI estimates for the tested channel and makes scenario planning more reliable. At Objective Platform, experiment results are integrated into the Bayesian modelling framework as prior information, improving model accuracy over time as more experiments are conducted.
What is the difference between an A/B lift test and a geo-lift test?
An A/B lift test typically splits audiences at the individual or product level, comparing performance between exposed and unexposed groups within the same geography. A geo-lift test splits at the geographic level, pausing or modifying activity in entire regions while keeping it constant in others. Both approaches measure incrementality but suit different channels and situations. Geo-lift tests are particularly useful for channels where individual-level targeting is not possible, such as TV, radio and out-of-home.
What does channel dependency mean in media measurement?
Channel dependency refers to the relationship between channels where the performance of one channel is partially dependent on the activity of another. In the case study above, SEA partially absorbed traffic lost from social media in Product A, suggesting that branded search and social were operating in a dependent relationship. Identifying these dependencies is important for budget allocation because cutting one channel may reduce the efficiency of others in ways that are not visible in channel-level attribution data.
