Analytics

Incrementality Testing: How to Prove Campaign ROI With First-Party Data

Krishna Poda
CEO & Co-founder
Last updated:
November 20, 2026
15
min read
Platform attribution overstates campaign ROI. Run a holdout test with your own customer data to measure true lift, with a worked supplements example.
TL;DR
  • Platform attribution and third-party cookies give conflicting numbers, so nobody can say which one is true.
  • Incrementality testing compares an exposed group with a matched control group using your own customer data.
  • Run it in 4 steps: audience matching, measurement window, incrementality calculation, decision making.
  • Rank campaigns by measured ROAS, then scale, maintain, or kill each one.
  • Start by measuring your biggest active campaign manually once.

Incrementality testing measures how many purchases a campaign actually caused. You compare customers who were exposed to the campaign with a similar control group who were not, using only your own first-party data. The difference in purchase behavior between the two groups is the campaign's true lift.

If you're a CMO at a health and wellness brand and you've given up on proving campaign ROI because attribution tools are a black box, this approach will change how you measure. It works without platform attribution or third-party cookies.

Below is a real example of a supplements brand that moved from "we can't prove campaign ROI" to "we can prove campaign ROI within 1 week using first-party data." You'll also get a framework to test campaigns, measure incrementality, and make decisions with confidence. Read this if your team spends more time debating attribution methodology than optimizing campaigns.

‍

FOR $20M+ BRANDS

Is your data actually Decision-Grade?

9 questions. 3 minutes. Score your Profitability Visibility and Readiness for AI-driven growth.

Start Free Diagnostic
Data Decision Grade Report

‍

Why Platform Attribution Overstates Campaign ROI

Your paid media platform says: "This campaign drove $500K revenue."

Your analytics tool says: "No, it was only $200K."

Your CFO asks: "Which one is true?"

Nobody knows. Welcome to the attribution apocalypse.

For years, CMOs relied on platform attribution (Facebook/Google/TikTok telling you "we drove this revenue") or third-party cookies. Both are broken. For a full view of the attribution models themselves, see our guide to ecommerce attribution.

One health and wellness CMO put it this way: "Platform attribution was clearly gaming the numbers. Facebook was taking credit for sales that happened for other reasons. I had no idea which campaigns actually moved the needle. So I stopped trying to measure. Just ran budgets on gut feeling."

What Is Incrementality Testing?

You can measure campaign performance without relying on platforms or cookies. You use only first-party data: your own customer list, purchase history, and behavior.

The method is cohort comparison. You run a campaign and track who was exposed. You create a control group of similar customers who were not exposed, then compare purchase behavior between the two groups.

That difference is true campaign incrementality. For the ad platforms' own description of this holdout method, see Google's guide to Conversion Lift.

Example: a supplements brand's collagen campaign

Here's a real example from a supplements brand. The setup looked like this:

  • The campaign: Instagram ads for new collagen product
  • Exposed group: 50,000 customers who saw the ad
  • Control group: 50,000 similar customers who didn't see the ad (matched by past purchase history)

The measurement covered 30 days post-campaign.

Metric Exposed Control Lift
% who purchased collagen 12% 4% 8 percentage points
Average collagen order value $45 n/a n/a
Revenue per exposed customer $5.40 $0 $5.40
Total campaign revenue (50K × $5.40) $270K n/a n/a
Campaign spend n/a n/a $80K
ROAS n/a n/a 3.4x

You don't debate platform numbers. You measure actual customer behavior.

You see: 8% of exposed customers purchased the product; only 4% of control purchased (via organic). The 4% difference is campaign-driven.

That's proof.

How to Run a Holdout Test in 4 Steps

A health and wellness brand with 500K customers can measure campaign ROI using 4 steps.

Step 1: Audience Matching

Start by building the two groups you will compare. If you already run customer segmentation analysis, those segments make the matching easier.

  • Identify all customers exposed to a campaign (email, retargeting, etc.)
  • Build control group: similar customers with similar purchase history, NOT exposed
  • Match on: past LTV, purchase frequency, category interest (so comparison is apples-to-apples)

Step 2: Measurement Window

Next, fix the period you will measure:

  • Define measurement period (30 days post-campaign is typical)
  • Track purchase behavior during that window

Step 3: Incrementality Calculation

Then calculate the lift and the return:

  • % of exposed who purchased in category = X%
  • % of control who purchased in category = Y%
  • Incremental lift = X% - Y%
  • Campaign ROAS = (Incremental revenue) / (Campaign spend)

Step 4: Decision Making

Finally, act on the measured ROAS:

  • ROAS > 3x: Scale campaign
  • ROAS 2 to 3x: Maintain or test optimization
  • ROAS < 2x: Kill or redesign

ROAS is still a revenue measure. For the profit side of the same decision, see ROAS vs contribution margin.

Example: 5 Campaigns, Platform ROAS vs Measured Lift

One skincare brand ran 5 campaigns simultaneously. Platform attribution said all 5 drove positive ROAS. But the CMO was skeptical.

They measured each campaign using first-party data (cohort comparison).

Campaign Platform Says First-Party Measurement Truth
Campaign A 4x ROAS 2.1x ROAS Overstated by 90%
Campaign B 3.8x ROAS 3.2x ROAS Slightly overstated
Campaign C 2.1x ROAS 2.8x ROAS Understated
Campaign D 1.5x ROAS 0.8x ROAS Negative actual ROI
Campaign E 2.9x ROAS 2.7x ROAS Accurate

Using platform numbers, they would've scaled campaigns A and B. Using first-party data, they made five different calls:

  • Maintained Campaign B (confirmed good)
  • Killed Campaign A (actually underperforming)
  • Scaled Campaign C (discovered winner)
  • Killed Campaign D (actually negative)
  • Scaled Campaign E (confirmed good)

Budget reallocation based on first-party truth: 35% improvement in overall campaign ROAS.

How Saras iQ Measures Lift by Customer Cohort

Saras iQ automates first-party cohort comparison for every campaign, as part of Saras iQ sales and marketing analytics. For each campaign, Saras iQ:

  1. Identifies customers exposed to each campaign (from your ad platform data + email data)
  2. Builds matched control groups (similar customers, not exposed)
  3. Measures incrementality (% lift in purchase, revenue, retention)
  4. Calculates true campaign ROAS (without platform bias)
  5. Ranks campaigns by ROAS (highest to lowest)
  6. Recommends: Scale, maintain, or kill each campaign

You don't argue about platform attribution. You measure real customer behavior.

Next Step: Measure Your Biggest Campaign With First-Party Data

This week, pick your biggest active campaign. Gather three things:

  • List of customers exposed to the campaign
  • List of similar non-exposed customers (control)
  • 30-day purchase behavior for both groups

Calculate incrementality manually once. See how different first-party measurement is from platform attribution.

That insight alone will change how you optimize going forward.

Saras iQ automates first-party cohort measurement for all your campaigns. See your own numbers before you commit.

Frequently Asked Questions (FAQs)

What is incrementality in marketing?
+

Incrementality is the share of purchases that happened because of a campaign and would not have happened without it. You measure it by comparing customers who were exposed to the campaign with a similar control group who were not. The difference in purchase behavior between the two groups is the incremental lift.

How big should a holdout group be?
+

There is no fixed number. The control group needs to be large enough, and similar enough to the exposed group, for the comparison to be apples to apples. In the supplements example above, 50,000 exposed customers were matched with 50,000 similar customers based on past purchase history.

Can you measure incrementality without cookies?
+

Yes. Cohort comparison uses only first-party data: your own customer list, purchase history, and behavior. You track who was exposed to a campaign, build a matched control group that was not, and compare what each group bought. No third-party cookies or platform attribution numbers are needed.

What is the difference between attribution and incrementality?
+

Attribution assigns credit for a sale to the campaigns a customer touched, and platforms can take credit for sales that happened for other reasons. Incrementality asks a different question: how many sales would not have happened without the campaign. It answers that by comparing exposed customers against a control group.

How long should an incrementality test run?
+

A 30-day window after the campaign is typical. Define the measurement period before you start, then track purchase behavior for both the exposed group and the control group across that same window. Use the same window for every campaign so the results are comparable.

+

What to do next?

See Saras in Action
If you're ready to stop pulling reports manually and centralize your eCommerce data, see exactly how Saras does it in a 25-minute demo. No prep required.
Book a Demo
Test your Data Readiness
Take the Quiz
Take a quick 5-min quiz and find out how future-proof your stack really is.
Check out Saras Analytics × 9 Operators Podcast
Listen to how top eCommerce operators think about data, growth, and analytics
Listen Now
Table of Contents
Heading one of the blog
Heading one of the blog
Heading one of the blog
Heading one of the blog
Heading one of the blog
Heading one of the blog
Get instant, trusted answers across your business functions
Meet Saras iQ: The AI analyst for your e-commerce business

Must read resources

Get instant, trusted answers across your business functions

Meet Saras iQ: your e-commerce AI analyst, powered by your business data, certified by us