The PodKnowledge|← All Episodes
Based on Lenny's Podcast data
Lenny's Knowledge Sketch · Growth

A/B Testing, Onboarding & the
Psyche of Your User

Laura Schaffer
VP of Growth, Amplitude
Former Head of Growth, Twilio
MAR 9 2023
Core Insight

Good Friction
Is Good

BAD FRICTIONadds steps, no value GOOD FRICTIONanswers hidden fears Drops conversionFrustrates user +5% conversion liftUser feels confident VS
"Bad friction is bad and good friction is good. There's no such thing as all friction being bad."
  • Adding 4 signup questions at Twilio improved conversion by 5%
  • New users ask: Am I in the right place? Can I do this?
  • Questions that mirror those fears feel comforting, not annoying
  • User psyche matters as much as product capability
Framework

Laura's Growth Operating System

1. HYPOTHESIS: define before you start2. PAINTED DOOR / MOCKcheap, fast, kill bad ideas3. FULL A/B TESTonly for validated ideas
  • 80%+ of experiments fail, Netflix, Microsoft, and Twilio all confirm this
  • A/B testing is the most expensive validation method, reserve it for survivors
  • Painted doors and user mocks can kill bad ideas cheaply in days
  • Failure is a compass, not a wall, let it point you to the right answer
80%
of experiments fail
+5%
signup lift, 4 questions
Career Framework

Don't just work hard in your role. Become the person who knows the customer. That brand travels beyond your manager's line of sight and creates opportunity across the org.

The Confidence Interval Hack

You don't always need 95% confidence. Lower threshold + qualitative backup + more experiments = more net wins per year. Set the rules before you run, never after.

Deep Dive

User Psyche: Onboarding & Selling to Developers

The Pill-in-the-Hotdog Principle

  • Twilio's guided onboarding flopped, Step 1 was "get a phone number," terrifying to developers
  • Fix: redirect users out of the console to a docs page where telecom was buried inside familiar code samples
  • Result: conversion improved even though users left the core flow
  • Lesson: never lead with the "bogeyman", embed unfamiliar steps inside something comfortable
"You bury the pill inside the hotdog. The scary thing is still there but now it's manageable."

What Makes Developers Unique

  • Skip your marketing site, they go straight to signup with zero context
  • Will not take a sales rep's word for it, they must self-prove value through a POC
  • A FANG company ran a full Twilio integration to production with zero sales contact
  • Another enterprise team used personal emails to avoid sales outreach
  • Reason: if the integration fails, it is their reputation, their job, not the vendor's
PLG Transition Trap Don't chop up your enterprise product and call it PLG. Start fresh, the self-serve customer's problems, fears, and buying journey are completely different from an enterprise buyer's.
Tactics

Build Your Brand from the Ground Up

  • Start a Voice of Customer digest, even a weekly Slack share, and watch who asks to join
  • Host quarterly VoC sessions for all of product; at Twilio, the CEO started attending within months
  • Propose things by doing them, not pitching them, Laura's growth team started with a guerrilla Friday-night experiment
  • Surface insights tied to North Star metrics, those are the ones executives act on immediately
  • "If it's not embarrassing you've gone too far", ship the ugly MVP, learn fast, invest after validation
Growth Team Health Short-term pressure breeds vanity metrics and data-massaging. Give teams a year-long horizon with low/medium/high impact buckets, not a weekly scorecard. Educate executives on the 80% fail rate.
Contrarian

Growth Myths Laura Busts

✗ Friction always hurts conversion INSTEAD → ✓ Friction that answers user fears increases conversion. Four extra signup questions at Twilio lifted sign-ups by 5% with zero personalization behind them.
✗ Always require 95% confidence before shipping INSTEAD → ✓ Lower thresholds + qualitative backup + more experiments = more net wins per year. Define the bar before the test, never after.
✗ Career growth comes from working harder in your current role INSTEAD → ✓ Career acceleration comes from being the person who knows the customer best. That brand travels beyond your manager and opens doors across the entire org.
✗ PLG = chop up the enterprise product and add a free tier INSTEAD → ✓ PLG requires restarting with the self-serve customer's unique problems. Their psyche, fears, and journey bear no resemblance to an enterprise buyer's.
Based on Laura Schaffer's episode on Lenny's Podcast. All ideas on this page are from the episode.Watch on YouTube
GO DEEPER IN THE EPISODE
04:15 How to carve your own career path, and an example from Bandwidth05:50 Laura’s career growth framework10:18 The value of customer insights12:25 The “voice of the customer” report16:14 Leaning into your strengths18:16 The experiment that shifted the way Laura thinks about friction20:20 Questions that improved Twilio’s onboarding and conversion rate28:53 Thinking about the psyche of your users31:26 The hot dog analogy for burying “scary stuff”33:58 Why it’s better to be iterative and why experiments fail36:21 Saving money by validating fast41:58 Where the best ideas come from49:51 Experimentation lessons52:54 The amount of time a growth team needs to be successful54:43 The big change at Twilio that led to tens of millions of dollars58:41 The need for both PLG and enterprise, and how Amplitude plans to tap into PLG1:05:42 What it’s like to serve developers1:11:16 Lightning round
Every sketch here began as a transcript.

Send us yours and we’ll draw it. $1 a transcript.

Sketch mine →
𝕏︎ X / Twitterin LinkedIn📸 Instagram🔗 Copy link
0:00