The PodKnowledge|← All Episodes
Based on Lenny's Podcast data
Lenny's Knowledge Sketch

AI Evals Are the Hottest
New Skill for Product Builders

Hamel Husain & Shreya Shankar
Creators of the #1 AI evals course on Maven
SEP 25 2025
The Thesis

Evals Are Data Analysis
for AI Applications

Vibe checksOK at startEvalsat scaleStraight to testsskip the data1–5 ratingmeans nothing
"The goal is not to do evals perfectly, it's to actionably improve your product." — Shreya
  • Evals: a way to systematically measure and improve an AI application (Hamel)
  • At its core, it's data analytics on your LLM app — not a scary new thing
  • Vibe checks are fine early, but they get unmanageable fast as the app grows
  • A domain expert — often the PM — should own the labeling, not a committee
Framework

Building Your Eval System

OPEN CODEAXIAL CODESLLM JUDGE
100
traces to sample when starting error analysis
2 kinds
of evals: code-based & LLM-as-judge
4–7
LLM judges is typical (Shreya)
  • Step 1: Sample real traces from your app's logs (~100)
  • Step 2: Open code — write a quick note on the first thing that's wrong
  • Step 3: Ask an LLM to group notes into axial codes; count them
  • Step 4: Build an LLM-as-judge for the biggest failure, align it to human labels
Hamel on ROI"Look at your data more than anything else — it's the highest ROI activity you can engage in."
Eval Types

Two Kinds of Evals

  • Code-based: a Python check for things you can test in code — output is JSON, has the right formatting, is short enough
  • LLM-as-judge: use an LLM to evaluate one specific, subjective failure mode with a binary pass/fail
  • Where to run them: in unit tests / CI before shipping, and as online monitoring on real production traces
  • Always align first: measure your judge against your own labels before you trust it
Why judges work

The judge does one thing — evaluate one failure mode with a binary output. That's a tightly scoped problem LLMs handle reliably. (Shreya)

Watch out for % agreement

If an error is rare, a judge that always says "pass" can look 90% aligned. Look at the full confusion matrix, not a single agreement number. (Hamel)

Playbook

Build Your Eval Framework

  • Keep judge outputs binary — pass or fail. Skip 1-to-5 scales, no one knows what 3.2 vs 3.7 means
  • Fix the obvious stuff in the prompt first — not every failure needs its own eval
  • Only write LLM judges for the pesky failures your prompt fixes can't reach
  • Wire the judges into unit tests and into online monitoring on real production traces
Time investmentShreya: 3–4 days upfront for the initial rounds of error analysis, then about 30 minutes a week to maintain.
Contrarian

Common Misconceptions

✗"Can't the AI just eval it?"INSTEAD →✓ It doesn't work. An LLM without your product context can't tell that a "virtual tour" answer is a hallucination. (Hamel's #1)
✗Skip the traces, just write testsINSTEAD →✓ People go straight to tests and that's where things go off the rails. Ground your evals in actual errors you see in the data. (Hamel's #2)
✗There's one right way to do evalsINSTEAD →✓ Many wrong ways, many right ways. Match your resources and where the product is at — it always starts with error analysis. (Shreya)
✗Evals vs A/B testsINSTEAD →✓ A/B tests are a form of evals. Both use systematic metrics to compare app quality. (Shreya)
Based on Hamel Husain & Shreya Shankar's episode on Lenny's Podcast. All ideas on this page are from the episode.Watch on YouTubeFollow @HamelHusain on X
GO DEEPER IN THE EPISODE
04:57 What are evals?09:56 Demo: Examining real traces from a property management AI assistant16:51 Writing notes on errors23:54 Why LLMs can’t replace humans in the initial error analysis25:16 The concept of a “benevolent dictator” in the eval process28:07 Theoretical saturation: when to stop31:39 Using axial codes to help categorize and synthesize error notes44:39 The results46:06 Building an LLM-as-judge to evaluate specific failure modes48:31 The difference between code-based evals and LLM-as-judge52:10 Example: LLM-as-judge54:45 Testing your LLM judge against human judgment01:00:51 Why evals are the new PRDs for AI products01:05:09 How many evals you actually need01:07:41 What comes after evals01:09:57 The great evals debate1:15:15 Why dogfooding isn’t enough for most AI products01:18:23 OpenAI’s Statsig acquisition1:23:02 The Claude Code controversy and the importance of context01:24:13 Common misconceptions around evals1:22:28 Tips and tricks for implementing evals effectively1:30:37 The time investment1:33:38 Overview of their comprehensive evals course1:37:57 Lightning round and final thoughts
Every sketch here began as a transcript.

Send us yours and we’ll draw it. $1 a transcript.

Sketch mine →
𝕏︎ X / Twitterin LinkedIn📸 Instagram🔗 Copy link
0:00