I once sat in a meeting where a team from one of the top consulting firms was presenting a targeting engine they had built for a large company.
The presentation was polished. Slide after slide showed how “good” the model was: lift curves, AUC, out-of-sample performance, clean-looking gains.
The room was excited. It felt like a win.
Then someone asked a simple question:
“What percentage of the people you target would have bought anyway?”
The mood changed instantly.
The answers became vague. There was a lot of hand-waving. No one had a clean, defensible number.
Later, when I looked at the model drivers, the story became obvious. The strongest predictor was a variable like:
- how many times a household had purchased the product in the past
That feature doesn’t measure persuasion. It measures prior loyalty.
In other words, the model’s main skill wasn’t persuasion. It was recognition—identifying people who already liked the brand.
And the question in that meeting was the right one. It’s the question many targeting engines quietly avoid.
The difference targeting engines often blur
A model can be excellent at predicting who will buy.
That does not mean it is good at predicting who will buy because you targeted them.
Targeting engines are commonly built and rewarded on metrics like:
- click probability
- conversion probability
- “likelihood to purchase”
But “likelihood to purchase” is not the same thing as “likelihood to be influenced.”
A conversion answers: Did they buy?
Incrementality answers: Did we cause it?
A simple way to hold the distinction:
Incrementality = P(buy | targeted) − P(buy | not targeted)
If your engine is optimized on propensity, it can look brilliant while barely moving that difference.
This shows up beyond targeting engines, too
This isn’t limited to vendor-built engines. It shows up anywhere a system is rewarded for conversions.
Search is a clean example.
Search systems can be excellent at finding people who are likely to click and likely to purchase. But that does not mean search created the intent. Often, it is capturing intent that already existed.
That’s one reason search can look like a “hero channel” in measurement. It sits close to the moment of purchase, so it receives credit for conversions even when the desire to purchase was built earlier through other channels.
A second example: “high-quality leads” that weren’t incremental
I saw a version of the same issue in a manufacturing business—this time from an internal team.
They built an engine to identify “high-quality leads” and route them to sales.
The results looked strong: the leads the model selected were ~10% more likely to convert. The team could show clean performance differences and tell a revenue story.
But the same question applied:
Would many of those people have purchased anyway?
If the model is mostly identifying the people already most likely to convert, then the routing engine may be prioritizing inevitabilities—improving reported efficiency without increasing total outcomes.
That’s the uncomfortable gap: “better conversion rate” is not the same thing as “more conversions.”
The question that matters
If you’re evaluating any targeting engine—whether it comes from a vendor or from your own team—this should be one of the first questions you ask:
How many of the people you target would have bought anyway?
A model can look incredible while doing something trivial:
- concentrating spend on loyal customers
- rediscovering repeat buyers
- harvesting late-funnel intent
- claiming credit for demand created elsewhere
That’s why fit metrics are not enough.
You can show me R², AUC, lift curves, ROC plots, precision-recall—out of sample, beautifully packaged—and still not answer the question you actually care about:
What did the model change?
What a truly useful engine is trying to predict
The valuable targeting problem is not:
Who is likely to buy?
It is:
Who is likely to buy if targeted, but unlikely to buy if not targeted?
That is a much harder problem.
It requires rigor and disciplined measurement because you are estimating a counterfactual: what would have happened without the intervention.
But that is the engine you actually want—because otherwise you spend limited marketing dollars on people who were already going to purchase.
That spend isn’t merely inefficient. It can become actively destructive.
The hidden costs of “anyway targeting”
Even if targeting people who would have bought anyway doesn’t annoy them, it has two costs.
Opportunity cost: your budget is finite. Money spent harvesting inevitable conversions is money you didn’t spend on persuading someone who was undecided or leaning toward a competitor.
Margin destruction: it gets worse when targeting is paired with incentives. If you discount people who would have purchased at full price anyway, you’re not just wasting spend—you’re reducing margin.
Why this gets worse in mature companies
This shows up most in mature businesses because loyal, habitual buyers generate strong signals and convert reliably.
A model can look impressive while mostly concentrating spend on familiarity.
The harder task—finding persuadable customers who might have chosen a competitor—is exactly what many engines avoid.
How to measure whether this is happening
If you’re handed a targeting engine—whether from a vendor or internally—there’s a clean standard of proof:
Run a lift test with a real holdout.
Not a lift curve. Not an AUC. Not an “out-of-sample” chart.
A holdout.
Measure the same quantity explicitly:
Lift = P(buy | targeted) − P(buy | holdout)
If you can’t run holdouts, you shouldn’t treat the engine as incremental.
The point
Targeting engines are rewarded for conversions, so they optimize for conversions.
And the most reliable way to get conversions is to find people who were already on their way.
If you don’t measure incrementality directly, you’ll end up optimizing for propensity—and mistaking that for impact.