The PodKnowledge|← All Episodes
Based on Lenny's Podcast data
Lenny's Knowledge Sketch

AI Engineering 101:
The Definitive Primer

Chip Huyen
Author "AI Engineering"; Stanford, Nvidia, Netflix alum
OCT 23 2025
The Discipline

AI Engineering Is a
New Discipline, Not Just ML

RESEARCHENGINEERINGML OPSAI ENG
"If you talk to the users who understand what they want or they don't want, look into the feedback, then you can actually improve the application way, way, way more."
  • ML engineers build models. AI engineers use existing models to build products.
  • The stack: prompting → RAG → fine-tuning → training. Most users never touch pre-training or training.
  • The base model performance is strong enough that entry barriers are low and the demand for AI applications is a lot bigger.
Framework

The AI Engineering Stack

PROMPTRAGFINE-TUNE
  • Layer 1: Prompt engineering, most product work lives here
  • Layer 2: RAG (Retrieval-Augmented Generation), inject relevant context
  • Layer 3: Fine-tuning, adjust weights on your data for a specific use case
  • Layer 4: Training and pre-training, the vast majority of users don't touch this at all
Chip on RAG"The biggest performance in their RAG solutions coming from better data preparations, not agonizing over what databases to use."
Evals: The Critical Discipline

Measure What You Build

  • Evals = tests for AI systems: Unit tests for model behavior
  • What to evaluate: Correctness, safety, latency, cost, user satisfaction
  • How many evals? "I don't think of just a fixed number on the evals" — the number really depends on your app
  • The eval loop: Run evals before each model upgrade or prompt change
The eval trap

Most teams skip evals until things go wrong in production. By then, you have no baseline to fix from.

The human eval

Automated evals scale; human evals set the ground truth. You need both. Human evals define what good looks like.

Playbook

Ship Better AI Products

  • Invest heavily in evals when you operate at scale or where failures can have catastrophic consequences
  • Use evals to uncover where the product is doing poorly on a specific segment, then focus improvements there
  • For RAG, focus on data preparation: chunk design, contextual summaries, hypothetical questions, rewriting content into question-answer format
  • Sampling strategy is underrated. It can boost performance in a huge way.
The bookChip's "AI Engineering" has been the most-read book on the O'Reilly platform since its launch.
Contrarian

AI Engineering Myths

✗Constantly evaluating what model is smarterINSTEAD →✓ Talk to users, build more reliable platforms, prepare better data, optimize end-to-end workflows, write better prompts (from Chip's viral LinkedIn table).
✗Adopting the newest agentic frameworkINSTEAD →✓ "How much of the improvement could you get from optimal solutions versus non-optimal solutions?" Often, not much.
✗Fine-tune on your dataINSTEAD →✓ Prepare better data. Chip: "The biggest performance in their RAG solutions coming from better data preparations."
✗Every AI feature needs full evalsINSTEAD →✓ "It's all about the question of return investment." Sometimes good enough is good enough and you can vibe check it.
Based on Chip Huyen's episode on Lenny's Podcast. All ideas on this page are from the episode.Watch on YouTubeFollow @chipro on X
GO DEEPER IN THE EPISODE
04:28 Chip’s viral LinkedIn post07:05 Understanding AI training: pre-training vs. post-training08:50 Language modeling explained13:55 The importance of post-training15:20 Reinforcement learning and human feedback22:23 The importance of evals in AI development31:55 Retrieval augmented generation (RAG) explained38:50 Challenges in AI tool adoption43:19 Challenges in measuring productivity45:20 The three-bucket test49:10 The future of engineering roles55:31 ML Engineers vs. AI engineers57:12 Looking forward: the impact of AI01:05:48 Model capabilities vs. perceived performance01:08:23 Lightning round and final thoughts
Every sketch here began as a transcript.

Send us yours and we’ll draw it. $1 a transcript.

Sketch mine →
𝕏︎ X / Twitterin LinkedIn📸 Instagram🔗 Copy link
0:00