The User Research Reckoning

Judd Antin

Research Consultant & Former Head of Research, Airbnb

RESEARCH STRATEGY

The Problem

User-Centered Performance vs. Real Impact

"Every time a PM comes to a researcher at the end of a product process and says, 'Can you just run a quick user study just to validate our assumptions,' that's user-centered performance. It's too late to matter. What they want is to check the box."

Most research is symbolic, checking a box rather than changing decisions
Research rarely drives real product impact at scale
Researchers want to be right, not to be wrong
The system is broken, not individual researchers

Research Framework

Three Levels of Research (And Why Middle Range Fails)

research tiers

Macro: Big picture, competitive landscape, concept car projects, aligned to annual planning
Middle: "How do users feel about X?", interesting but unfocused, not tied to business OKRs
Micro: Specific, measurable, 48-hour turnaround, highest ROI per hour spent

The multimillion dollar buttonResearch revealed the CTA text scared users away from completing purchases. Changing seven characters added 1% conversion. That's micro research done overnight.

The Integration Problem

Research as Reactive Service vs. Strategic Partner

The vicious cycle: Researchers hired without being integrated → do reactive, low-impact work → executives conclude they're not valuable → layoffs → cycle repeats
The fix: Researchers in the room from day one, shaping questions, participating in decisions, driving the framing
The result: A great researcher embedded consistently drives product metrics, growth, and business impact
Most research teams are service functions, not strategic partners

Research's true role

Researchers are the repository of insights needed for growth. But they're rarely in the room where decisions are made.

The PM-researcher relationship

When PMs and researchers are deeply integrated, the researcher shapes the right question before work even starts.

The Five Tools

What Great Researchers Actually Have

1. Formative research, ethnographic, generative, open-ended, look ahead
2. Evaluative research, usability testing, usability research
3. Survey design, rigorous, scaled, gets you answers fast
4. Applied statistics, you can't AB test without basic stats
5. Technical skills, SQL, or now prompt engineering to work with your data

The evolution of researchQualitative research alone is no longer enough. Great researchers are multi-method, combining qual + stats + technical chops.

Tropes (Myths PMs Believe)

Research Misconceptions That Kill Impact

✗Research just slows us downINSTEAD →✓ Great research goes fast (48 hours to 1 week). Broken research at the end of a process is what slows you down.

✗I can do my own researchINSTEAD →✓ Talking to one user isn't research. It's idiosyncratic. Researchers turn garbage into signal and avoid bias.

✗AB test everything, numbers don't lieINSTEAD →✓ AB tests show what changed, not why. Research + data scientists = causal claims + deep understanding.

✗"We knew that already, that was obvious"INSTEAD →✓ Hindsight bias. Duncan Watts: "Everything is Obvious If You Already Know the Answer." We selectively remember and self-gaslight.

Based on Judd Antin's episode on Lenny's Podcast. All ideas on this page are from the episode.Watch on YouTube

GO DEEPER IN THE EPISODE

04:16 Critiques and responses to Judd’s post “The UX Research Reckoning Is Here”07:33 The state of user research 08:53 Macro, middle-range, and micro research 14:05 What teams get wrong when it comes to research 15:46 The importance of integrating research from the beginning 17:30 Traits of great researchers 19:53 Advice for evaluating user researchers 21:10 Balancing business and product focus 23:55 User-centered performance 26:42 The role of intuition in product development 30:15 Checking your gut instincts 32:54 Common tropes about PMs, from researchers 41:02 A/B testing vs. user research 43:15 Hindsight bias and narrative fallacy 44:55 Making recommendations based on research 47:26 Advice for teams on how to leverage researchers 51:18 How product managers can be better partners to user researchers 56:53 The ideal ratio of researchers in a company 59:43 Empowering user researchers to drive impact 01:03:39 The limitations of NPS as a metric 01:06:48 The risks of dogfooding 01:08:51 Lightning round

Every sketch here began as a transcript.

Send us yours and we’ll draw it. $1 a transcript.

Sketch mine →

0:00