Does Advertising Actually Work? (Part 1: TV) | Freakonomics Radio | Episode 440
Executive Summary
Stephen Dubner examines advertising measurement with Steve Levitt, Keith Weed and Anna Tuchman. The central question is which purchases an ad causes, rather than which purchases happen alongside it. Research and channel claims come from a 2020 conversation, not a current benchmark. The linked YouTube video was uploaded in 2023. Selected measurement passages only; health-product and policy examples omitted. Topics are untimed. Source: https://freakonomics.com/podcast/does-advertising-actually-work-part-1-tv-ep-440/
Key Takeaways
Ask what would happen without the ad
Levitt describes the challenge of isolating sales that would not have happened without advertising. Attractive charts can show two things moving together without establishing cause. Define the outcome you need and a credible comparison before treating observed sales as an ad's contribution.
Account for the reason ads run when they do
His retailer example ran television pushes around major selling periods. Sales could rise because of the season as well as the campaign. Compare like with like and account for timing, rather than crediting the ad for all demand during a predictable busy period.
Test assumptions instead of defending routines
Levitt recalls a market accidentally missing newspaper inserts and the company finding no sales impact in that period. It was an opportunity for further testing, not final proof that all newspaper ads were useless. Investigate surprises with a designed comparison instead of dismissing them because the routine is familiar.
Match reach with the audience
Weed distinguishes broad mass-market products from a niche example of premium surfboard wax. A large audience is wasteful when most people are irrelevant to the offer. Choose a channel based on who needs to see the message and what action you are trying to cause.
Understand the limits of a natural comparison
Tuchman explains comparing similar people near borders between television markets with different ad exposure. That can provide useful variation when a randomized experiment is unavailable. The method still depends on its assumptions and data; a nearby comparison is not automatically a valid control.
Keep a study's scope with its result
Her broader analysis uses consumer-packaged-goods data from2010to2014 and she warns against calling it the whole advertising picture. Established brands and other industries can differ. Preserve the category, period and selection limits rather than converting an estimate into a universal rule.
Check incentives and missing results
Her broader analysis uses consumer-packaged-goods data from 2010 to 2014 and she warns against calling it the whole advertising picture. Established brands and other industries can differ. Preserve the category, period and selection limits rather than converting an estimate into a universal rule.
The Conversation Breakdown
Incremental sales
⏱ Untimed topicLevitt distinguishes ad-caused purchases from sales that would have happened anyway.
Campaign timing
⏱ Untimed topicSeasonal selling periods complicate a simple correlation between advertising and purchases.
An accidental missing campaign
⏱ Untimed topicA market without newspaper inserts becomes a lead for further testing.
Broad reach and niche offers
⏱ Untimed topicWeed explains why an audience and channel need to fit the product.
Advertising-market borders
⏱ Untimed topicTuchman discusses a natural-comparison method for different ad exposure.
Research scope and data period
⏱ Untimed topicThe consumer-packaged-goods analysis has limits that should stay with its conclusions.
Incentives and null findings
⏱ Untimed topicThe episode considers why unflattering results can be harder to produce or publish.
Core Frameworks & Ideas
Incrementality - discussed concept
Estimate outcomes beyond what would have happened without the advertising.
Randomized comparison - discussed method
Design treatment and comparison groups before attributing a sales change to a campaign.
File-drawer problem - discussed concept
Unpublished null findings can distort the evidence available to later readers.
Actionable Next Steps
- ✓Write what purchases you think would occur without a campaign.
- ✓Separate seasonal demand from the campaign period in your analysis.
- ✓Define a comparison before increasing one channel's budget.
- ✓Keep the date, category and limits beside any advertising benchmark you use.
Notable Quotes
“So, don’t get me wrong. I’m not implying that advertising doesn’t work.”
“I’m implying that we don’t have a very good idea about how well it works.”
“It’s because the company knows when the big selling days are, and they target the ads around it.”
The One Thing to Remember
Measure the sales an ad causes, not all the sales that happen while it runs.