Based on Lenny's Podcast data
The SurveyThe Gaps Between
AI Hype and Practice
"If you make practitioners sit together and ask them what's real — the answers are very different from the conference keynotes."
- Most AI practitioners are more optimistic than public discourse
- The biggest gap: evals — everyone knows they need them, few do them well
- Deployment != adoption: shipping AI features ≠ users actually changing behavior
- Safety concerns are real but nuanced — practitioners vs. researchers diverge
FrameworkPractitioner Reality Check
- The eval gap: critical skill, widely acknowledged, poorly executed
- Deployment gap: 60% have shipped AI; <40% have users relying on it daily
- Trust gap: practitioners trust AI output less than leadership does
- Measurement gap: most AI ROI is qualitative, not quantitative
The honest findingAI is genuinely useful. The hype is about capabilities; the gap is in safe, reliable deployment at scale.
What Practitioners KnowThe Real State of AI in 2026
- Works well: Generation, summarization, classification at scale
- Works with care: Reasoning chains, multi-step workflows, coding assistance
- Hard: Consistent factual accuracy, complex reasoning, novel domains
- Underappreciated: Prompt brittleness — small changes cause large output swings
The evals blindspot
Most teams ship AI features with vibes-based QA. That's fine for v0.1; it's not fine at scale.
The adoption reality
AI features ship. AI behavior change doesn't. User re-education is the hardest part of AI PM.
PlaybookClose the Practitioner Gap
- Build an eval framework before your 2nd AI feature, not your 5th
- Define "good enough" explicitly — vague quality targets produce vague quality
- Measure user behavior change, not just feature usage
- Create a practitioner community inside your org — share what works and what doesn't
The community findingThe best AI practitioners share notes obsessively. Isolation is the enemy of good AI practice.
ContrarianAI Research vs Practice Myths
✗Research papers = production realityINSTEAD →✓ Research papers are existence proofs. Production reality is 10× harder.
✗Bigger model = better productINSTEAD →✓ Better evals = better product. Model size is one lever; evaluation is all the levers.
✗AI is deterministicINSTEAD →✓ AI is probabilistic. Design your product around this, not despite it.
✗Safety is a research problemINSTEAD →✓ Safety is a deployment problem. Practitioners own it as much as researchers.