The PodKnowledge|← All Episodes
Based on Lenny's Podcast data
Lenny's Knowledge Sketch

What AI Practitioners
Actually Believe

Aishwarya Naresh Reganti & Kiriti Badam
AI researchers & practitioners
JAN 11 2026
The Survey

The Gaps Between
AI Hype and Practice

BELIEVE ITUSING ITSKEPTICALIGNORING IT
"The bad stuff is the execution is still all over the place."
  • Most AI practitioners are more optimistic than public discourse
  • The biggest gap: evals, everyone knows they need them, few do them well
  • Deployment != adoption: shipping AI features ≠ users actually changing behavior
  • Safety concerns are real but nuanced, practitioners vs. researchers diverge
Framework

Practitioner Reality Check

Enterprises: reliability = #1 problem75%AI PMs: time on workflows, not models80%
  • The eval gap: critical skill, widely acknowledged, poorly executed
  • Deployment gap: shipping AI features does not equal users actually relying on them daily
  • Non-deterministic API: LLMs are sensitive to prompt phrasings and stay black boxes
  • Agency-control trade-off: every autonomous decision the team hands over is control it gives up
What Practitioners Know

The Real State of AI in 2026

  • Enterprise taxonomies are messy and undocumented; agents can't infer the rules real humans learned on the job
  • One-click agents are pure marketing
  • Even with strong data and infrastructure, replacing a critical workflow takes four to six months
  • Prompt injection stays unsolved as agents go mainstream
"Evals" means different things

A labeling company saying "our experts write evals" means expert error notes. A client saying "we do evals" may mean checking LM Arena. Same word, different work.

Calibration is never done

Users don't talk to your system the same way they did six months ago. When a model like GPT-4o is deprecated, calibration resets. Recalibrate on every shift.

Playbook

Close the Practitioner Gap

  • Start with high control and low agency; move up only as the agent earns trust
  • Be problem-first, not tool-first — the slippery slope is chasing solution complexity and forgetting the problem
  • Log every human intervention so you build a flywheel of what "good" looks like
  • Recalibrate when the model changes or user behavior evolves
Pain is the new moatSuccessful teams go through the pain of iteration until they know what works and what doesn't. That lived knowledge is the moat.
Contrarian

AI Research vs Practice Myths

✗One-click agents will just workINSTEAD →✓ If someone's selling one-click agents, it's pure marketing. The flywheel takes four to six months.
✗AI is deterministicINSTEAD →✓ AI is probabilistic. Design your product around this, not despite it.
✗Just build fastINSTEAD →✓ Building is cheap today. Design and problem-selection are what set teams apart.
Based on Aishwarya Naresh Reganti & Kiriti Badam's episode on Lenny's Podcast. All ideas on this page are from the episode.Watch on YouTubeFollow @aish_reganti on X
GO DEEPER IN THE EPISODE
05:03 Challenges in AI product development07:36 Key differences between AI and traditional software13:19 Building AI products: start small and scale15:23 The importance of human control in AI systems22:38 Avoiding prompt injection and jailbreaking25:18 Patterns for successful AI product development33:20 The debate on evals and production monitoring41:27 Codex team’s approach to evals and customer feedback45:41 Continuous calibration, continuous development (CC/CD) framework58:07 Emerging patterns and calibration01:01:24 Overhyped and under-hyped AI concepts01:05:17 The future of AI01:08:41 Skills and best practices for building AI products01:14:04 Lightning round and final thoughts
Every sketch here began as a transcript.

Send us yours and we’ll draw it. $1 a transcript.

Sketch mine →
𝕏︎ X / Twitterin LinkedIn📸 Instagram🔗 Copy link
0:00