← All Episodes
Based on Lenny's Podcast data
Lenny's Knowledge Sketch · Growth

A/B Testing, Onboarding & the
Psyche of Your User

Laura Schaffer
VP of Growth, Amplitude
Former Head of Growth, Twilio
MAR 9 2023
Core Insight

Good Friction
Is Good

BAD FRICTION adds steps, no user value GOOD FRICTION answers user's hidden fears Drops conversion Frustrates user +5% conversion lift User feels confident VS
"Bad friction is bad and good friction is good. There's no such thing as all friction being bad."
  • Adding 4 signup questions at Twilio improved conversion by 5%
  • New users ask: Am I in the right place? Can I do this?
  • Questions that mirror those fears feel comforting, not annoying
  • User psyche matters as much as product capability
Framework

Laura's Growth Operating System

1. HYPOTHESIS — define before you start 2. PAINTED DOOR / MOCK — cheap, fast, kill bad ideas 3. FULL A/B TEST — only for validated ideas
  • 80%+ of experiments fail — Netflix, Microsoft, and Twilio all confirm this
  • A/B testing is the most expensive validation method — reserve it for survivors
  • Painted doors and user mocks can kill bad ideas cheaply in days
  • Failure is a compass, not a wall — let it point you to the right answer
80%
of experiments fail
+5%
signup lift, 4 questions
Career Framework

Don't just work hard in your role. Become the person who knows the customer. That brand travels beyond your manager's line of sight and creates opportunity across the org.

The Confidence Interval Hack

You don't always need 95% confidence. Lower threshold + qualitative backup + more experiments = more net wins per year. Set the rules before you run — never after.

Deep Dive

User Psyche: Onboarding & Selling to Developers

The Pill-in-the-Hotdog Principle

  • Twilio's guided onboarding flopped — Step 1 was "get a phone number," terrifying to developers
  • Fix: redirect users out of the console to a docs page where telecom was buried inside familiar code samples
  • Result: conversion improved even though users left the core flow
  • Lesson: never lead with the "bogeyman" — embed unfamiliar steps inside something comfortable
"You bury the pill inside the hotdog. The scary thing is still there but now it's manageable."

What Makes Developers Unique

  • Skip your marketing site — they go straight to signup with zero context
  • Will not take a sales rep's word for it — they must self-prove value through a POC
  • A FANG company ran a full Twilio integration to production with zero sales contact
  • Another enterprise team used personal emails to avoid sales outreach
  • Reason: if the integration fails, it is their reputation, their job — not the vendor's
PLG Transition Trap Don't chop up your enterprise product and call it PLG. Start fresh — the self-serve customer's problems, fears, and buying journey are completely different from an enterprise buyer's.
Tactics

Build Your Brand from the Ground Up

  • Start a Voice of Customer digest — even a weekly Slack share — and watch who asks to join
  • Host quarterly VoC sessions for all of product; at Twilio, the CEO started attending within months
  • Propose things by doing them, not pitching them — Laura's growth team started with a guerrilla Friday-night experiment
  • Surface insights tied to North Star metrics — those are the ones executives act on immediately
  • "If it's not embarrassing you've gone too far" — ship the ugly MVP, learn fast, invest after validation
Growth Team Health Short-term pressure breeds vanity metrics and data-massaging. Give teams a year-long horizon with low/medium/high impact buckets — not a weekly scorecard. Educate executives on the 80% fail rate.
Contrarian

Growth Myths Laura Busts

Friction always hurts conversion INSTEAD → Friction that answers user fears increases conversion. Four extra signup questions at Twilio lifted sign-ups by 5% with zero personalization behind them.
Always require 95% confidence before shipping INSTEAD → Lower thresholds + qualitative backup + more experiments = more net wins per year. Define the bar before the test, never after.
Career growth comes from working harder in your current role INSTEAD → Career acceleration comes from being the person who knows the customer best. That brand travels beyond your manager and opens doors across the entire org.
PLG = chop up the enterprise product and add a free tier INSTEAD → PLG requires restarting with the self-serve customer's unique problems. Their psyche, fears, and journey bear no resemblance to an enterprise buyer's.
𝕏︎ X / Twitterin LinkedIn📸 Instagram🔗 Copy link
0:00