The PodKnowledge|← All Episodes
Based on Lenny's Podcast data
Lenny's Knowledge Sketch

How to Measure AI
Developer Productivity

Nicole Forsgren
Partner at Microsoft Research; co-creator of DORA & SPACE; author of Accelerate
OCT 19 2025
The Problem

The AI Productivity
Measurement Gap

HOW ENGINEERS SPEND TIME WITH AI TOOLSReviewing code~50%Writing code~50%MSR CUPS-model paper cited in the episode
"That's not what productivity's about when you're using an AI tool, frankly."
  • People fundamentally shift how they work with AI-enabled tools like Copilot: more time reviewing than writing
  • Faster on a single task (a paper showed engineers built an HTTP server 50% faster) is not the same as more productive
  • "Never pick number of lines of code." Pick balanced metrics, in tension with each other
  • Pick balanced metrics, metrics in tension, so you don't throw something out of whack
Framework

DORA & SPACE

SPACE(pick metrics)DORA(4 keys)ELITEPERFORMANCE
SPACE
Satisfaction · Performance · Activity · Communication · Efficiency
  • DORA four keys: deployment frequency, lead time, change fail rate, MTTR (2 speed + 2 stability)
  • Elite performance: deploy on-demand, lead time < 1 day, MTTR < 1 hour, change fail rate 0-15%
  • SPACE is how you pick metrics for complex creative work; use at least three dimensions so they stay in balance
  • "DORA is actually implementation of SPACE" — DORA covers the outer loop
Where AI changes the frameNicole expects the SPACE dimensions to hold with AI, but sees the need to add a dimension like trust or reliability: can I rely on it?
What Nicole Sees In The Data

AI Is Shifting The Work

  • Time split: "About 50% of your time now is spent reviewing versus writing" (MSR CUPS-model paper)
  • Task speed: A colleagues' paper showed engineers can build an HTTP server 50% faster with AI
  • What that unlocks: AI frees up cognitive space so engineers can do harder things, not so leaders can lay off half the workforce
  • 80% signal: "80% of the folks that I work with, this is their biggest problem" — teams don't agree on what "developer experience" even means
A team sport

"Software is a team sport." That is why Nicole's work includes a community effect and studies productivity, community, and wellbeing together.

The measurement trap

Activity metrics (pull requests, check-ins, AI tool usage counts) are easy to instrument, but "that's not what productivity's about when you're using an AI tool, frankly."

Playbook

Measure Productivity Right

  • Start with words, not data. Agree what you mean (friction? culture? inner vs outer loop?) before you pick a metric
  • Use SPACE to pick at least three dimensions so metrics stay in balance or in tension
  • Instrument the automatable metrics continuously; ask the satisfaction ones periodically, once every few months
  • Push both top-down and bottom-up, with good communication throughout
New bookNicole's new book Frictionless is a practical guide for helping teams move faster in the AI era.
Contrarian

Engineering Productivity Myths

✗Number of lines of codeINSTEAD →✓ "Never pick number of lines of code." Pick balanced metrics, metrics in tension.
✗Faster on a task = productivityINSTEAD →✓ Building an HTTP server 50% faster is not what AI productivity is about. The gain is cognitive space to do harder things.
✗Productivity is a solo pursuitINSTEAD →✓ "Software is a team sport." Measure productivity holistically, with a community effect.
✗Two-week change approvals give you stabilityINSTEAD →✓ "That's not right. It was just kind of an old wives' tale." Speed and stability move together.
Based on Nicole Forsgren's episode on Lenny's Podcast. All ideas on this page are from the episode.Watch on YouTubeFollow @nicolefv on X
GO DEEPER IN THE EPISODE
05:09 The concept of developer experience (DevEx)08:33 Flow state and cognitive load in the age of AI12:02 Challenges in measuring productivity with AI21:19 The importance of developer experience for business value22:20 Common issues and solutions in developer experience26:49 Signs your eng team is moving too slow29:52 How AI is improving productivity33:32 Real examples of productivity improvements36:35 Introducing her new book, Frictionless43:40 How to get started building a DevEx team45:15 The impact of forming developer experience teams46:15 How to measure the impact of DevEx teams48:53 Measuring the impact of AI tools on productivity55:16 Survey design for developer experience57:59 Popular AI tools for developers59:08 Bringing a product mindset to DevEx improvements01:00:40 AI corner01:02:33 Lightning round and final thoughts
Every sketch here began as a transcript.

Send us yours and we’ll draw it. $1 a transcript.

Sketch mine →
𝕏︎ X / Twitterin LinkedIn📸 Instagram🔗 Copy link
0:00