How to find hidden growth opportunities in your product | Albert Cheng (Duolingo, Grammarly, Chess.com)

Transcript from https://podmenti.com/t/e12c953580ef4397

Growth as the job is to connect
Users to the value of your product. Growth sometimes gets this reputation that it's just pure metrics hacking. You worked at
Three of the most successful consumer subscription products in the world. Where do you think is the biggest missing piece that people don't get about building a successful consumer subscription product? User retention is gold for consumer subscription companies. If you don't retain your users, then a lot of the onus is on getting them to pay on day one. Noam Levinsky. He said that I need to ask you about the biggest.
Monetization win that you found at Grammarly. The lived product experience for most of the free users was that Grammarly was just a product to fix your spelling and grammar because those were the free suggestions. What if we actually sampled a number of different paid suggestions and interspersed them to free users across their writing? All of a sudden, people were seeing Gramly as a much more powerful tool.
Than they were before. What's the most counterintuitive lesson you've learned about building teams? I saw some of the highest performers just being people that had very high agency, had that clock speed, had that energy, but they didn't necessarily need to have deep
Experience on that matter. Sometimes experience could be a crutch, especially in this world where the grounds are shifting so fast with AI. A lot of your learned habits actually need to be intentionally discarded.
Today, my guest is Albert Chang. Albert is known as one of the top consumer growth minds in the world.
He led growth and monetization at three of the most successful and beloved consumer products in the world Duolingo, Grammarly, and OCS.com.
Earlier in his career at YouTube, he worked on streaming and gaming features used by over twenty million people.
His unique approach to growth blends marketing, data, strategy, and product management.
And in our conversation, we cover a lot of ground.
Including his explore and exploit framework to find growth opportunities, his biggest and most interesting growth wins at Duolingo Grammarly and Chest dot com, how he uses AI to accelerate his growth work,
What he's come to realize about the power of brand.
and community in your growth work, his top experimentation best practices, why his goal at every company is to run 1,000 experiments a year, and so much more, a huge thank you to Eric Albest, Noam Lebinsky, and Jorge Mazal for suggesting topics for this conversation. If you enjoy this podcast,
Don't forget to subscribe.
And follow it in your favorite podcasting app or YouTube.
It helps tremendously. Also, if you become an annual subscriber of my newsletter, you get 15 incredible products.
For free for an entire year.
Including lovable
Replit, bolt, NADN, linear superhuman descriptorflow, gamma, perplexity, warp, granola, magic patterns, raycast, jabierd, and mobbin. Head on over to Lenny's newsletter.com.
and click product pass.
With that.
I bring you.
Albert Chang.
My podcast guests and I love talking about
Craft and taste and agency and product market fit.
You know what we don't love talking about? Sock too.
That's where Vanta comes in.
Vanta helps companies of all sizes get compliant fast and stay that way with industry leading AI, automation, and continuous monitoring.
Whether you're a startup tackling your first SOC two or ISO twenty seven oh one.
Or an enterprise managing vendor risk, Vanta's trust management platform makes it quicker, easier, and more scalable.
Vanta also helps you complete security questionnaires up to five times faster so that you could win bigger deals sooner.
The result, according to a recent IDC study.
Vanta customers slashed over$500,000 a year and are three times more productive.
Establishing trust isn't optional.
Fanta makes it automatic.
Get$1,000 off.
at banta dot com slash lenny
This episode is brought to you by Jira Product Discovery.
The hardest part of building products isn't actually building products, it's everything else.
It's proving that the work matters, managing stakeholders.
Trying to plan ahead.
Most teams spend more time reacting than learning.
chasing updates, justifying roadmaps, and constantly unblocking work to keep things moving.
Jira product discovery puts you back in control.
With Jira Product Discovery, you can capture insights and prioritize high impact ideas.
It's flexible, so it adapts to the way your team works, and helps you build a roadmap that drives alignment, not questions.
And because it's built on Jiro, you can track ideas from strategy to delivery.
All in one place.
Less chasing, more time to think, learn, and build the right thing.
Get Gira Product Discovery for free at atlassian dot com slash Lenny.
That's at Lassian dot com slash Lenny.

Albert, thank you so much for being here and welcome to the podcast.
Thanks for having me, Lenny. Excited to be here.
I'm even more excited to have you here.
So as I do for every podcast conversation, I reached out to a bunch of people that you've worked with that know you well to find out what to ask you about and what topic to spend time on.
Uh Jorge Mazal, who
Uh is
famous in my world for writing what was for the longest time the most popular newsletter post
It's actually
People have usurped it now, but it was like
Stuck there for a long time.
So here's what he wrote.
It is a mystery to me how Albert is able to do what he does. I am actually eager to listen to this episode and learn from him.
That is super nice. Thank you, Jore. I've learned so much from him.
I'm the type of weird person that likes to wake up before their kids and like pull up
A bunch of browser tabs and look at experiments, so
It was perfect that Jorge brought me into the growth world at Duolingo. Learned a ton of uh best practices and he's just a great guy.
Thanks, More.
We're already getting into these tactics. I love it. Uh
Let me just give a little framing on what I want to do with this conversation.
What I wanna try to do is to help people learn
tools and mental models.
for finding growth opportunities for their own products.
And essentially.
learn the growth mentality that you bring into the companies and products that you work on.
What I wanna start with is to give us a little insight into
How you became what you became.
There's an interesting pattern I found across a bunch of
recent guests which is
Many people were very good at piano.
When they were younger.
And we're very serious.
For example, head of chat GPT Nick Trelli was like almost gonna become professional.
Jazz pianist.
You were very serious in as a piano player earlier in career.
How did you go from pianist to one of the top growth minds in the world?
Briefly.
Well that's very flattering, um, but I appreciate it.
Yeah, I I grew up playing a lot of piano. Um
My parents were were immigrants from Taiwan and I was the oldest kid.
That they had and so
I definitely felt that.
strong uh encouragement, if you will.
To
learn a bunch of things, take them seriously, study hard.
And so I did, right? And like
My parents, even though they weren't musically proficient.
They had a like deep love for classical music.
So I was the stereotypical like baby that would listen to Mozart, I guess, when I was sleeping, type of thing.
And I still vividly remember like
We had this upright Yamaha piano and at the very top of the piano we had this countdown clock from ninety minutes.
Literally every single day of my childhood.
Um just practice really, really consistently.
At first like
I really was irritated by that thing, but as I grew older I started to appreciate like music quite a bit more.
But anyway, like
I think what really accelerated my
my interest and abilities in
uh piano was like I I feel like I hit the lottery. I had perfect pitch and so I was able to
You know, quickly
Understand.
whether I was like playing the right stuff or the wrong stuff and just pick up music pretty pretty rapidly. What does p perfect pitch even mean? Does that mean you know which note is exactly playing? Okay. Exactly. Wow. So I can listen to a song and then like
Just a very, very clear.
understanding of which note I'm supposed to start with and if I'm playing something wrong. So it's it's it's very helpful.
It's unfair, definitely.
So anyway, yeah, I got I quite got quite good like as a teenager in high school and even considered like studying at a music conservatory.
My intrinsic motivation for music wasn't necessarily
as strong at that point. And so I decided to go to engineering school uh instead. But that would have been an incredibly different career.
And to your original point around the relationship between like music and growth.
I didn't really reflect on this until recently. Yeah, yeah, I have a four year old and I'm like starting to teach him how to
Bang on the keys a little bit, but a couple of things stand out.
I mean one is that
I think music and growth, they both rely on this just consistent
repetition like you're constantly making mistakes.
you have the super tight feedback loop. You have to get really resilient.
to just making mistakes all the time and
You know that the way of learning is through those mistakes. Right. So that's kind of a thing that I learned very early.
And the second thing that
uh occurred to me is that
They both have this like structural underpinning to them.
With growth you have a growth model.
You have metrics, you have experiments, you have channels, things like that.
But you also need on a day to day basis to have creativity. You gotta come up with like interesting solutions and hypotheses to test.
And the same is true on the music side, right? You have music theory, you have scales and stuff, but
To create beautiful music.
You need that passion, that emotion, that flow.
Um, so I think that's the beautiful combination between the two.
Fun fact, my wife bought me
piano slash singing lessons for Father's Day recently, and I've gotten really into the stuff.
So I'm learning how to play very basic piano now and and learning to
Uh identify notes and hit notes with my voice.
Nice.
It could be I could go the reverse. I could become a professional piano player.
Oh man. No, it's so fun. So hard though. I'm just like my fingers really. How do you how do you do four freaking keys at once? Yeah. Just like what is going on here?
Okay. So let's get let's get into the meat of it.
I want to talk about growth.
There's a very specific framework that as we were chatting that I think would be really helpful for people to hear and learn from you.
You call it explore and exploit.
I think there's a different bunch of different ways to think about this. Talk about this framework and how that informs the way you think about growth. Yeah.
I initially came.
Up or heard with heard about it.
Explore and exploit through uh my engineering partner at uh Grammarly Normal.
And I think he actually had taken some Reforge classes. So maybe the original inventor of it might be Brian Balfour, who I know has been on your pod.
Um but anyway, it's a great concept.
The gist of it is that when you're in exploratory mode.
Think of it as like finding the right mountain to climb.
And then when you're in exploitation mode, it's like focusing your resources.
on climbing that mountain effectively.
And
certain companies I think the the warning is to
basically spend too much of your time on one and in the spectrum, right? If you do too much exploration
You can have your team feel a little bit too scatter shot.
Just trying a hundred different random ideas.
What's the through line? What's the strategy? How do you pattern match, you know, successes across them?
And if you do too much in exploitation, which is often the MO of growth teams
It can lead to this like
saturation and stagnation where you're just locally maximizing a thing.
And even though this principle of e explore and exploit, like
It's typically thought of as a as a macro thing.
I like to work with my teams more on the the micro on the insight level.
So I'll give you a concrete example.
So I work at chess dot com.
And um
one of our priorities is to encourage chess players to improve to learn and improve.
So one of the PMs that we have, Dylan.
He works on all the learning features.
The most used learning feature in our product is called game review. So you play a game of chess.
after the game's over, we have this virtual coach that teaches you about your worst moves, best moves, et cetera.
And his job is to like
improve user engagement and retention.
And so he's in this exploratory phase trying to figure out like how do I drive more of that type of activity.
And what he observes is that
eighty percent of people that review their games actually do so after a win.
And that's really counterintuitive to
When we initially built the feature, we thought that
People would want to use it after losses or to see their mistakes such they could like work on their mistakes.
It turned out not to be the
truth when it came to the human psychology and the actual data of it of it.
And so we made some changes in the product experience when you lose a game.
Now as opposed to surfacing your blunders and your like horrible stuff that you did.
We flip it on its head.
And so we show you
Your brilliant moves, your best moves.
And we have coach say something encouraging, you know, losing just part of learning, like keep it up, that type of thing.
That change alone was pretty dramatic for us.
It grew game reviews by twenty five percent.
subscription by twenty percent.
User retention by a lot as well.
Um, so that was fantastic.
But the point is that it doesn't just stop there, right? You have to take that insight.
share it broadly across the company.
Right.
Now adjacent product managers like
The PM working on puzzles.
Can now think about, okay, how do I audit these cold patterns in my product and think about making them more positive, right? I can change the success rating, I could tweak some copy.
Change the colour of some buttons.
And so
You now can take this like
Experiment when
and expand it out ten X across your organization. And that's the kind of exploitation phase of it.
So
When done right, right, you can oscillate between the two until you saturate out of exploitation mode, and then you encourage the teams to to brainstorm and get more creative again.
Amazing. Okay, so there's a lot here to follow up on
One is
Is the core piece of advice when you find something that works really well.
find ways to build on that learning. One is
Here's an insight.
It can apply to other parts of the product. Hey teams, here's something we learned unexpected. Maybe this can help you.
Also it's just keep
Find more like you run more experiments in the same zone, I imagine is a part of that. Yeah, exactly right. I mean, in my experience
The typical
win rate, and I hate to use that term for experiments, is is often something like thirty to fifty percent. Like
Usually you're not actually g like
You're trying a bunch of things, a lot of hypotheses turn out not to be true. You know, consumer products are very unpredictable like that.
But
when you do find a thing that breaks through the noise, and it could actually be a hugely losing experiment, too. Those are also super valuable, right?
Surfacing those across the company.
Like
The
original PM running that experiment doesn't necessarily need to be the person that
figures out what you should do for all the other parts of your product experience.
But the onus is on them too.
clearly articulate what their hypothesis is, what they found.
such that then as like a growth leader I can encourage people to
Kind of.
swarm around that and try a bunch of different ideas such that the success rate is up and the impact is up.
So it's just kind of oscillating back and forth between the two, that is uh
The magic bullet.
I think another
takeaway here slash something that I think about when I hear what you're saying is
There's often a lot more wins in an area than people expect.
that you can continue to find wins and
growth in something for a long time.
Exactly right.
Yes.
Okay.
At the end of the day, like
Users.
I think within a company sometimes you can have this siloed approach where
you break apart the product experience in, you know, fifty different ways and distribute them across different teams, and you assume that
Users interact with each of the different features with a different mentality.
But oftentimes that's actually not necessarily the case. And so sometimes you can surface an insight that's more
You know, human psychology based.
that can resonate across the entire product experience. And so I think
When you can find that.
Um you can double down.
People hearing this might feel like
Okay. Yes. Uh find big wins and then find more.
Is there something
you find that helps you figure out when to explore?
versus when to exploit and when you've exploited too far, just like any heuristics or I don't know, ways of
helping people guide them along this process of exploring and exploiting.
One one thing that I I try to focus on at
A company of our scale of like a chess dot com, right? We're running roughly
two hundred and fifty experiments a year. So we're not
Like
The highest in the industry, but we run a we run a decent volume, right? And so when that happens
I invest in these like experiment explorer tools, and we can talk about AI as well as another way to kind of uncover and pick out these nuggets of wisdom.
But basically these explorer tools
can allow me to look across the spectrum of experiments that are going on.
Try to figure out if there are patterns between the the hypotheses and the learnings that are happening.
And if I'm starting to see like
more and more experiments that are
Not statistically significant.
That may be a signal to me to say okay, we might have expected

Too far.
Like there might not be as much juice to squeeze.
Hey guys, let's like
You know, get back to the table and
Brainstorm and
be a little bit more divergent with our thinking.
Well, let me follow the thread on AI and how you you're using AI to
Help you figure this out. That is very cool. Talk about that.
I think one of the the latest things that we've been tinkering around with is this text uh uh sequel.
Capability
It's actually pretty powerful. Um
We have this data request Slack channel.
Where
For the longest time, and this is still true today, like people will toss in.
All sorts of just one off questions.
Yeah.
How many
subscribers do we have in South Africa, or like, you know, how long did somebody play puzzles like last
Last month or something.
And these ad hoc questions, they often take a lot of like human time to just go in and
you know, a data analyst needs to prioritize it and find time to go run the query.
And yes, you can invest in self serve tooling to improve at this.
But also I found that AI is quite
good at doing that first pass answer.
As well.
Um
And so we're working on like kind of training some of these Slack bots to essentially be the

uh provider of a lot of these answers, which makes
the company as a whole a lot more
Did uh
Informed, I guess.
And I think what's also kind of interesting is that
Just human nature is that if you have a question that you feel like, you know, you might be a bit embarrassed to ask or you don't want to bother someone, you just don't ask the question, right?
And so by the nature of having these tools, you get actually a
pretty large explosion of
questions being asked and I think you see this in chat GPT too, right? It's like just having a thing, right, that you can converse with that you feel
Uh comfortable and makes a huge difference.
Okay, this is extremely cool. So is this something you build basically it's a Slack bot that
gives you the SQL query or does it actually do the analysis. No, it does the analysis. Yeah. Whoa. So cool. Okay. Is this gonna something you guys are gonna release? Or is this just like somebody you guys should just build this at every company. We should. It's a good idea. Okay. Well there's an episode where everyone in the comments is like open source this. Yeah so we'll see if that happens again.
Uh
That is very cool. Are there other examples of that kind of stuff that you've done or seen? I mean a an adjacent example is a lot of the product managers like we're
Tinkering around with all sorts of different prototyping tools right now, right? It's just like go from an idea to a representative solution.
Today, right, there's a lot of humans involved in
taking an idea, writing up a spec, doing a review, doing design, et cetera.
Uh I'm sure you've
interviewed plenty of people that have talked about this specific problem, right?
And so
For us, like we've invested a bit in at least
Carving out the main screens of our product experience, things like our onboarding flow, our home screen.
Our chess board as an example.
And building like
Essentially AI prototypes of those using tools like a a V0 or like a lovable, right? And when you have those foundational pieces, you can then share them with
the rest of the company and they can use that as a starting point.
And then they can try to, you know, put their ideas on top of that and then
they become a lot more discussable and hopefully testable relatively soon.
What's in your AI stack along those lines?
The PMs are mostly using V Zero. The designers love Figma, so they're using Figma make.
Uh the engineers are using a
A combination of tools right now.
Um so cursor, cloud code, GitHub Copilot.
marketing teams use all sorts of tools for translation, subtitles, you know, content adaptations, et cetera.
Customer support uses intercom, Finn?
So there's quite a lot of tools that are kind of used across the company.
I would say though that something that is
Kind of annoying to me is that we haven't yet figured out
the bridging from the tinkering to the workflow quite as seamlessly as I would like, right?
And so each sub function, even though the common
I guess wisdom now is that
AI is going to strip away this these like functional titles.
It is kind of true that based on your
experience like you may gravitate to using a type of tool more.
And if that tool isn't as interoperable with some of the other tools that you need to pass down the chain to actually ship it into production, at least at our scale, right? I think for smaller startups, sure, PM should just go ship it.
But for us like
We are still doing some handoffs between functions. I expect that to change over time and we are investing in some of like
You know.
design system components and MCPs and stuff to make it a little bit easier.
But yeah, it's a it's an investment and it takes time to to smooth things out.
I wanna come back to this topic of how things have changed and how you work.
as a product person, as a growth person across the companies you've been at, but
First of all, when I
I wanna talk about another example of
Binding growth wins and monetization wins.
Uh Noam La Lavinsky, who is Chief Product Officer at Grammarly, you worked with him.
While you were grandmother.
He said that I need to ask you about the biggest monetization win that you found at Grammarly.
And how you discovered the opportunity.
I had the pleasure of working with Gnome and his product team at at Grammarly.
Um
Some context first for those that don't use Grammarly. So Grammarly is a AI powered writing assistant.
And so typically people will use it as a Chrome extension or a downloadable desktop client.
And basically what it does is it overlays your writing with a bunch of different correction. So you're a big thing. And it saves my life. Fantastic. Glad to hear that.
The Gramly is a freemium business model, which means that over ninety percent of our users are on the free
Service.
And the rest of it pay for subscriptions, essentially, right?
And so one of the teams they work on subscriber conversion, PM there is Kyla. Um, that team's great.
And their job is to figure out the free to paid
uh subscription path, right?
And so one of the realizations
One is that we weren't actually
Tracking.
Um
The events that well for the types of
Essentially suggestions that people were getting and how often were users seeing paywall and stuff like that. That's kind of step number one. We have to put that instrumentation in.
Step number two is that hey we noticed
Actually
Um
First let me explain some of the logic. So as a free user
you basically get these underlines across your writing.
And if you accept all of them.
Um, then you see the paywall and that encourages you to like subscribe for more nuanced features.
As a free user, the main things you get are spelling, grammar. They're basically correctness things.
And as a paid user, you get the like
How do you improve your tone to be more empathetic? How do you improve your writing to be more clear? How can you rewrite entire sentences? That type of thing.
And so the observed behavior from all that tracking and data was that actually a very small percentage of our free users.
Was
Deciding.
to accept all of their suggestions. They were more kind of picking and choosing as they go. And I wonder if
Your uh
Experience is kind of similar to Definitely. Cause yeah, I'm always like Wait, we stop rewriting everything. Just like this part is wrong. I will fix it. Yeah. I'm very much uh
Pick and choose that's right. And then the second thing, which is I think equally, if not more
Interesting is that
You know, I was at this company during this generative AI transformation, which is obviously still going on, right?
And
Quite frankly, both the company brand as well as the like lived product experience for most of the free users.
was that Gramly was just a product to fix your spelling and grammar because those were the free suggestions we were showing people, right?
And so we decided to flip that on its head entirely.
And we said, Okay.
What if we actually sampled
a number of different paid suggestions and interspersed them.
to free users, right, across their writing.
Such that they were intermingled and we would provide a limited taste.
of what the paid offering had to
Provide.
And on the surface, like
Even though it's rational, the the the concern is that if we give too much of this away, then will people want to subscribe.
We found completely like
That was not the case, right?
All of a sudden.
People were
seeing Gramley as a much more powerful tool than they were before.
And our upgrade rate's like nearly doubled.
Just through this change.
So I think this is interesting just modernization learning that
Especially if you work on a freemium product.
Try to have your free
product be a reflection.
Of like everything.
That your product can offer you.
Obviously to an extent there's some costs involved with some of the paid features and things like that, but
It generally will pay its for itself if you're able to put your best foot.
Forward.
Um and go do that. So that really worked well for us there.
I think this is what converted me to being a paid grammarly subscriber.
Wow. What an what a a genius move. So essentially it's here's uh here's a bunch of improvements.
But you were you get like three, I think.
Max and then it's like okay, now you get upgrade.
It's basically like a reverse free trial, but in real time, like while you're writing, as opposed to a time based one.
So we kind of adopted some
You know, patterns that are in the industry, but molded it to Grammarly's specific use case.
Right, I was gonna ask. So it's not like a full trial. It's like a
A capp trial where you get a certain number of
things and then you run out. And then you they get refreshed, I think, once a day or something like that is what I found. Yeah, you got it.
Yeah. Grammarly is
uh the best slash most devious at their upsells. I was I'm always just like
God damn it. You're about you have I'm so close to seeing it I'm
Improvement I just ca I just have to upgrade.
Just like right there. It's right there where my mouse is.
Yeah, well I'm not proud of being devious, but devious is uh really getting me to buy the thing.
Good job. Who's it, Kyla? Okay, nice job, Kyla. It's it's very effective.
I love that.
And so uh okay, so in terms of the free trial
I don't know, is there anything there of just
There's always this question of
Premium, give things away.
And then there's Pro County get it.
There's like trial versus time.
Some features or limit. I don't know. Do you have like a for consumer subscription products, like here's the way to go?
Yeah, I think first of all, why do freedom subscription in the first place is a common question that, you know.
Like I've joined all these companies that are for you in subscription. Like w what do I like about it, I guess.
Uh um
Well, one, I think it ties really nicely to like mission orientation of a lot of these companies. It's often like
You wanna spread the product as wide as possible because that's why the founders built the thing, right? You're you're trying to like improve education with like
Duolingo or you know.
Grammarly or chest.com. Like these are meant to be widespread products with a really wide value proposition that fits globally, right?
And so
obviously the lowest friction to that is going to be a free product. So that alone is part of it.
Another part of it is that a lot of these products primarily grow through word of mouth.
And especially if you can build
you know, network effects in the product, like Duolingo has a bunch of social features.
Or with Grammarly, like they have a bit of a B to C to B play as well.
So you see Gramley being used by teams and by companies and whatnot, right?
And even if users are on the free plan, they still provide quite a lot of value.
And
Making sure that Gramley can be
purchased by a coworker or by a team member or whatever, right?
So
I think these things are are usually why I lean toward
Make sure that the core value proposition that you're providing users is free and is sort of permanently free.
And then you layer on kind of a sampling or a taste of some of the premium features that are on top of it. That's usually the sweet spot that I've seen.
As to the trials
Reverse trials type of thing.
I think it largely depends. I think if you have
Especially a B to B feature where you may have some lock in.
Reverse trials can be super powerful.
You just wanna get people in there. You don't need to ask for their credit card because
they're using your CRM or they're investing quite a lot of time in like building out
You know.
material and content.
And so by the time that window drops, you actually like feel, Oh man.
I probably should.
Keep this and and start paying.
I think for a lot of consumer products it's a little bit harder for that to work. And so I've typically seen more
Just
normal free trials be be the norm.
Let me follow this thread of just consumer subscription products.
I feel like this is the category that
Every indie developer dreams of building a product in
Because
It's easy to build. Cool. Build an app.
Oh, I'd have pay wall.
And then they realize this is a lot harder than I thought.
from a perspective of distribution and CAX and growth like that.
I g is that the biggest missing
Pon about the okay.
Building a successful consumer subscription product.
Yeah, I mean user retention is gold for consumer.
subscription companies, if you don't retain your users, then a lot of the onus is on getting them to pay on like day one, that's super hard, right? Then you're dealing with
Totally different business models where you're paying for users, you're trying to like aggressively upsell them before they, you know.
hit any sort of habitual usage patterns with your product.
A lot of apps naturally do that'cause that's how they break the mold and get their first
You know.
Users to to do it. But
I don't know. I've been fortunate to join companies sort of after that initial phase, but especially like take Duolingo and Chess.com. These are organic, word of mouth kind of driven businesses.
And in in kind of both ways, like they grew the market.
Right, from a much smaller market.
And
As opposed to
it being a a very competitive space where you're kind of competing and taking market share from others and bidding for higher terms and stuff like that.
So I don't know, that there's something to that.
So what I'm hearing here is
You need to find a way to grow through word of mouth for this to have any chance of success.
And also retention needs to be very high.
Do you have a heuristic of what retention needs to be for you to have a chance building a successful consumer subscription business?
I think consumer companies tend to track like
Essentially two main types of like user retention. There's more of like the new user one, kind of D one, D seven, et cetera.
Um, I think when you have your D one retention
Somewhere around like the thirty or forty percent mark, like that's quite
solid, I think, for for a consumer app.
If it's much lower than that.
Then
Sometimes I might question like the intent of the user or the ability for that.
Uh
you to I guess acquire
Just mathematically acquire enough users such that you can grow a
A big enough daily active user base.
That's surprisingly low.
Yeah.
So it feels achievable. It's achievable.
It's achievable in theory, but there are so many options out there in the market and people are feeling a lot of like
app and product boat bloat. And so just to be clear, you're saying twenty to thirty percent of people come back the next day. You're thirty to forty thirty to forty to forty. Forty percent I think are in an okay place.
I think even more importantly, and you know, you mentioned Jorge to kick this off, but like
You know, he wrote that very, very popular article, uh
about the growth model, right? And how like current user retention rate was the biggest thing for them. I think especially if you have a product that has daily frequency.
Like that's actually the retention that matters the most is that
Like
existing user base that has developed a habitual pattern. How sticky is your product?
And it's that retention rate that really compounds and build that builds that daily habit.
So over time
Especially when companies mature a little bit. You actually focus most of your energy on the existing user retention mechanics.
Um, you find that that's a much, much bigger lever.
One exception is that
Squarely was a different type of product and that you install it.
And you don't proactively open it every day?
So that was kind of interesting to me because I assume that you should always just focus on existing user retention. But for a product like Grammarly, it's actually the activation, installation, aha moment that's really, really critical and
Well carry the user.
For a very, very long time.
That makes sense. Like
Yeah, the stats would show someone's a daily active user because they're typing things and that's not an accurate stat for Grammarly.
The other interesting trend I've noticed across successful consumer subscription products is they always start very scrappy.
and very cost efficient and spend efficient because
I think it's because it takes them a long time to find something that's working.
And
they're surviving on that margin of retention to
Growth cost essentially.
Yeah, that's right.
Yeah, the and the retention piece, that's such a good point. It's
Like my newsletter is very much along these lines. It's just like
How many people are joining every day, how many people are leaving? And it's
It's a difficult
treadmill to be on because people, you know, they want to save money, they want to spend it on Netflix and things like that.
So as amazing as you are that people are always gonna leave.
So the trick is how do you find more people coming than going?
Yeah, and I think
Um
Just to take chess.com an example, like I think probably eighty ish percent of our
Daily or weekly active users. I mean I'll check the numbers, but something like that would be like a current
a current user or an existing user.
And then a new and a like
reactivated or resurrected user, those are actually about similar size for a company of our our stale.
So even though there's a lot of attention on that new user experience, it's actually like pretty interesting that
the components of your active user base.
Are actually not heavily weighed.
In the new user set.
After you mature to a certain degree.
Can you explain that a little bit more? Yes. So
After some period of time you kind of stack up a lot of inactive users in your product.
Right.
And
You also stack up sporadic users, right? People that may not have a daily habit, but they will use it, you know, once or twice a week or once or twice a month type of thing.
And so
Eventually.
That
Nath sort of adds up where you have
Let's say hundreds of millions of kind of dormant
Users that are coming back.
And it's actually worth spending some time making sure that that
Kind of
resurrected, for lack of better word, experience inside the product is really excellent.
And that you find
novel ways to try to bring them back.
Duolingo as an example, they did a good job of using social notifications.
And so if people would use like
Contact sync or something.
you might get a push notification that one of your like best friends just started using Duolingo and that might encourage you to
Come back and resurrect into the product.
And whether you resurrected in the product.
It might be the case that you're
proficiency of the language you were learning.
Like you were learning French three years ago.
But now you like forgotten most of it, right?
And so
When you open the app again, it encourages you to essentially replace yourself, like do another placement test and put you in the right spot. And so some of these types of mechanics.
For a more more mature company.
Can
uh lead to pretty good ROI, I guess is what I'm trying to say.
So many people have already tried in the past.
That
To grow, you need to resurrect people that have been there.
And so thinking through
It's almost like a
User experience for resurrected users. Exactly.
Okay.
Let's zoom zoom out a little bit.
You've worked at three of the most successful consumer subscription products in the world.
What is
What is the difference between how these three operate? I think there's many ways to be successful. It feels like these companies are very different.
What's kind of the gist of
what each of these kind how they operate.
Well, first of all, like there's obviously a lot of similarities, but I'll just focus my answer on the the differences. So
I think Duolingo, what struck me most working there is they're very
particular. They have a pro an approach of product development that is infused across like everyone in the company.
And they tend to they actually wrote a playbook about this. It's called The Green Machine. They tweeted about it. Yeah. That was one of my most successful tweets ever. They just tweeted something about Duolingo just released their playbook and I
Screenshot like the the owl's butt and screen like page and it was like five thousand likes.
That's hilarious. Yeah. So yeah.
But yeah, I mean the ethos of the company, I mean they they hire a lot of
intelligent, energetic people out of
College, basically.
And they give them
A lot of amazing experimentation tooling.
And they care a lot about like the clock speed of the company, right? So
It's a lot of creativity, lot of ideation.
the product experience of Dual Ego actually like changes multiple times per day for each user.
Which is pretty shocking. And so I'd never worked in a place like that before.
Um
But it's
really struck me about how consistently the company operated.
And they had specs and processes for doing like
each of those steps in their product development cycle and they were really, really tight about it.
Okay, so that's Duolingo. Yeah, that's Duolingo.
Grammarly.
You know, this is an interesting company because they started as a
paid product oriented at students.
Then they expand it into more of a freemium model tailored to everyone.
Gradually focusing more on the professional
Base.
And then as a
accumulate a lot more professionals, they realize hey, there's patterns, right?
We're seeing that a bunch of marketing teams or a bunch of sales teams or a bunch of customer support teams or whatever, right?
particular functions within particular companies were really adopt grammarly
at scale.
And so
they were able to then layer on much more of a managed kind of enterprisey motion.
And while I was there, I was focused on the consumer self serve motion.
But they weren't siloed.
Right.
They were intermixed with each other.
And so a big part of my job was not just to grow
Like
to self serve revenue and self serve active users, but it was also how do you uncover
Kind of the right.
uh teams, the right functions, the right companies.
for like demand gen and sales to go reach out to.
Um so that was a very interesting it's kinda product led sales work, right? And it's really
fascinating thing for me to learn.
And then on top of that, with all the transformation going on with generative AI.
And even recently with them acquiring
uh coda and superhuman and becoming more of a productivity suite.
Like the company is just evolving pretty rapidly. It's a really exciting uh thing for me to be a part of and to see from the sidelines.
But that just made it at its core kind of a different growth job than
um than Duolingo for sure.
Essentially a B2B business versus a very consumer business. Yeah, and a lot more meaningful strategic decisions as well.
And then the core product team also
You know, I'm used to in growth.
like laying out the entire user journey.
that a user go through, you know, acquisition, activation, engagement, so on and so forth, right?
And typically
Growth teams, if they're well resourced, they can do enough to move each one of these various levers, right? And it's just a matter of like,
The sequencing of them and what you want to prioritize first.
But Gramley was kind of unique in that the core
product experience itself.
was what drove
repeated activity, right?
It's that.
I I previously mentioned that current user retention thing.
What most drives that is the frequency and the quality of the suggestions that you get every day, right?
And so it was an interesting learning in that I staffed up a growth team, tried to work on this metric, and then I realized actually, like I'm kind of just getting in the way. Like this is
really a thing that the core product team
Um
Most influences.
Let me have a conversation with the core product leader and then shift that over to them.
Um so yeah, just a super interesting experience.
And then just outcom.
The thing that's most unique about chess dot com is that they are super fanatical.
Like about Jeff.
Uh.
Crazy crazy. I mean, you shouldn't be surprised, obviously the name of the company is like this. Um
But they've always hired people from around the world. The company's always been globally remote.
They just hire people that love
Chess they play all day, they watch the streams.
Our Slack is always blowing up with people's chess moves and games and whatnot.
You know, I think
I wanna say this a little bit delicately, but
Like
Duolingo, even though the product they're providing is around language learning.
I think the original ethos of how to start the company was really around motivation.
Right. The hardest thing to
It's habits, right? How do you build that daily habit?
And I actually in many ways see language learning as like their first vehicle.
And what they have a superpower in is that.
Again.
The motivation, the habits, et cetera.
So that's kind of Duolingo.
And Grammarly actually kind of similarly, right? Like people know them for the spelling and grammar corrections, but what's really unique about them is they're they're integrated across
Tons and tons and tons of applications. There's not many many products that work like that. That's really unique.
And so now if you hear like
to share their new CEO talk about like the AI superhighway and all that type of stuff, right?
They can now
use that technology to provide a lot more than just grammar writing.
And so my point is just that like
Chess is about chess, a hundred percent. It's in the ethos, people are crazy passionate.
That just means we're always dog fooding the product. There's a and then
It's just an amazing energy in the company.
to just use the product all the time, come up with ideas.
And I love that environment. I think that's fun for me.
That is so cool.
And what I love about what you're saying is
There's no right or wrong answer. All of these companies are killing it.
I think Duolingo is worth like
ten billion dollars, something like that, and keeps growing. Like I'll look it up in a second. Uh and Grammarly is worth a ton and then
Chest.com is doing super well.
So I think that's a really interesting takeaway here is
Is you can succeed in a lot of different ways.
Yeah.
What's really cool about Duolinga, I was just thinking as you were talking is
Yeah, it's just interesting that
This very structured methodical way of building is working so well. Cause you could listen to that and be like, Oh, that's I don't want to work like this is rigid way.
But the fact that it
is killing it tells us this actually works really well. If you find something that works.
Lean into it.
Yeah, that's right.
Yeah, and the the structure is rigid, but the ideas are the farthest away from rigid as possible, right? Like you have seen their
I don't know, Super Bowl commercials, their memes, their gamification tactics. Like it's a super fun creative environment.
So like rigid is uh the the farthest possible word to use. But what I just mean is they're they're consistent. They have templates for everything and like their product reviews are like 10 or 15 minutes. It's just people go in and out. So it's just it's kind of a surreal environment.
About how rapidly and consistently they work.
Awesome. They're worth a twelve billion dollars.
And they were much higher actually not too long ago. They're coming down a little bit.
Uh so speaking of Diolingo, when people think Diolingo, they think of
the brand and the owl and the success they had on TikTok and things like that.
I'm curious to get your take on as a very growth oriented person.
Watching that.
work and your take on
Growth experimentation data versus marketing viral TikTok videos.
Mascots, things like that.
Yeah, I mean, I used to think it was versus, but now I realize that they combine really well. It could be rocket fuel for for your growth.
Yeah, being a product person, you know, I joined a lot of these companies because they're literally on the home screen of my phone and I like I like using them and I consider myself someone that's not easily swayed by
no ads or T V commercials telling me what to buy.
So I always like had an element of scepticism.
on the marketing side for much of my career.
But then yeah, you you join a place like
Like Duolingo, and you see how duo the owl
has developed a personality through the push notifications and the product experience.
And then seeing the marketing team.
leverage that personality in their TikTok and in their YouTube and all throughout social media.
And just feed into those like
Memes.
And then we would track
back in the product experience, like how did you hear about us and put all those channels in there.
And some days it would be like holy shit, like it's bringing in twenty, thirty percent of like our new users and any given day.
Um, so those two things really go hand in hand.
And that
feeling has only been reinforced by chess dot com, you know, over the last five years.
Like the first fifteen ish years of this company
was really under the radar. Like
eight hundred million people play chess around the world, but most of that is
Over the board.
Until recently there wasn't actually that much online.
But
Five years ago everything changed. You had the pandemic, you had Queen's Gambit.
You had a lot of
like YouTube and Twitch streamers, you had a bunch of kids playing it in school, et cetera.
And so it's really the combination of those two things that that make it take off.
And it's like the growth experimentation is more the the
the slow and steady or fast and steady, I should say, uh approach where you're just continually iterating, you're making the product experience better.
But then every so often, right, there's a big wave that comes in.
You can quadruple your, you know, registrations overnights and
Yeah, you'd be a fool not to take advantage of that.
I was actually speaking at chess dot com and blank chess. I was at a coffee shop this weekend.
There's a little like a family of dad and mom and a
Daughter.
Ordering and the dad's sitting at the table and he's just like on his phone, just like opened up chess dot com secretly and just playing while he's waiting.
No man.
Uh I will not uh admit or deny that I've done that before.
But it's like
That is if I can think of anything
uh more wholesome. I I can't. Like that's an amazing thing to be doing while you're just
You know, my four year old can actually set up the pieces, which is pretty great. So he he enjoys the game quite a bit. Oh man, this four year old already a pianist playing chess. That's right. What a stud.
This episode is brought to you by Miro. Every day new headlines are scaring us about all the ways that AI is coming for our jobs, creating a lot of anxiety and fear. But a recent survey for Miro tells a different story.
Seventy six percent of people believe that AI can benefit their role.
But over fifty percent of people struggle to know when to use it.
Enter Miro's Innovation Workspace, an intelligent platform that brings people and AI together in a shared space to get great work done.
Miro has been empowering teams to transform bold ideas into the next big thing for over a decade.
Today, they're at the forefront of bringing products to market even faster by unleashing the combined power of AI and human potential.
Guests of this podcast often share Miro templates. I use it all the time to brainstorm ideas with my team. Teams especially can work with Miro AI to turn unstructured data, like sticky notes or screenshots into usable diagrams, product reefs, data tables, and prototypes in minutes.
You don't have to be an AI master or to toggle yet another tool. The work you're already doing in Miro's Canvas is the prompt.
Help your teams get great work done with Miro.
Check it out at Miro.com slash Lenny.
That's M I R O slash Lenny.
Okay.
You talked about AI a little bit here and there. I wanna follow that, Thread.
That's a growth person.
I imagine AI informs chess.com in a lot of day ways, so there's kinda like two buckets here.
How is AI changing the product, say HS and other places you work?
And then how is AI impacting
your work as a growth person.
So pick one or
Both buckets and Yeah, I'll I'll tackle them in sequence. I'll start with the the chess one, um, just'cause I have maybe a slightly unique take on that one. So
Chess and AI, they've been intertwined for almost a century. Like some of the early computing pioneers, like they just figured, yeah, chess is an interesting game. We can test machine intelligence and write some algorithms, we're not.
And then fast forward to like nineteen ninety seven.
And you had IBM, they had their deep blue.
Application.
Who actually
um beat the the world champion back then, which was Gary Kasparov.
That was like a huge moment.
of like
Of shock and reckoning of like, Oh man.
Is AI gonna take over humans? Are we gonna have jobs and like all this stuff? And this is, you know, thirty years ago.
And thankfully we're all still here.
And more people are playing chess.
Than ever, right? And so the game of chess.
And chest.com specifically.
have learned how to augment, I guess, the human playing experience.
with the power of chess engines, which you know are definitely a powerful form of AI. It's not LMs, to be clear.
But
There's engines like stockfish these days.
That are just dramatically better than the top grandmasters in the world.
Like uh where we're at? It's just like I remember when it
beat humans and now it's just dramatically better. It's dramatically better. Wow. Yeah. I think
There's a rating system that compares like relative skill level.
And an average chess player somewhere like
A thousand.
maybe a fifteen hundred on the high end.
A top grandmaster like Magnus Carlson's like a twenty eight hundred.
And then
Stockfish and similar engines are like thirty six hundred.
Wow. And so to put that in comparison.
Yeah. Like at least it's not ten thousand or million. I don't even know if that's it's not ten thousand, but it's similar to like if the chess engine was playing without a major piece, like a rook or something, they would still be competitive against the best players. And this is the ELO score? Yeah, the ELO score, ELO rating.
Magnus is is what you said about twenty eight hundred. Yeah.
And then the the stock fishes would you say thirty six hundred, thirty.
deep evaluation on specific chest lines, they can calculate tens of millions per second.
So it's not realistic for for a human to compete against that. But yet, like
Watching some of these chess engines played has opened up a lot of creativity, new strategies, new lines, new appreciation for the game.
And our chest dot com approach is that we can bring this technology
For every user, even people that have never moved a piece before.
I talked earlier about that game review product. That's exactly what this does.
So behind the scenes we're running chess engines.
T
To basically spit out
evaluations for every move that you make.
And then we translate that and make that approachable to the user.
using, you know, their native language in plain, approachable
uh style, right? And even with audio and things like that as well.
And that part of it.
Like the personality, the speech back to the user, that part is LMs.
And so I guess my point is that
Again, chess and AI have been intertwined forever.
But for us what's most important is that we keep the customer at the north.
star of it. We're not just applying LMs just because it's the new hot thing. You gotta apply the right technology for the right
you know, feature to provide value to the user.
And so we never we try not to ever lose sight of that and get
Let's get
Uh two carried away.
It just sh
Really surprising.
I think people would not have expected
AI and can outbeat every human alive.
Never.
And we s and chess is at an all time high. People
want to keep playing and are playing more and more than ever played.
Yeah. You know, interestingly L LMs themselves
are quite bad at playing chess. Like they hallucinate moves.
They look at patterns, right? They're they're very good at pattern recognition, but not so good.
at going super, super, super deep on a specific chest thing.
And if you've even tried to like
create or look at chessboard images on chat GPT. A lot of them have the wrong number of squares. They're not set up properly. And so
You know, I don't want to be too dismissive. I'm sure it's gonna get much stronger at at reasoning.
And actually Google recently sponsored a tournament where all the top LMs
played a tournament against each other.
Um, so that was pretty fun to watch.
They're improving, but
Yeah.
This chess is specifically a game that having a a trained
you know, deep, deep computing engine is just gonna be much, much, much more powerful than LMs.
And not to go down this track too far, but uh Alp Zero, famous for beating the top go player.
I imagine that all is that was that trained specifically for Go? Obviously not an LM, but that was a Go specific model. Yeah, my understanding is.
That
The one that the documentary is incredible, by the way. I don't know if you've watched Alpha Go. It's it's it's uh
It's amazing how they took something so technically deep and made it like
Yeah, so it's so emotional and human. But I think that's the that's the cruci of how we feel, or I guess, about
about AIs and the products that we build, actually.
Um, but to your point, I my understanding is that
The way alpha zero is
primarily trained is that it just plays a bunch of games against itself.
Right.
And so through the neural network it just gets smarter every time and because it can
You know, have that repetition times
you know, a billion or a trillion. I don't know exactly what number, but
It's gonna get pretty damn good.
Okay, let's go back on track to w where we were going. So this was a
How AI is impacting
Chess dot com.
How is
A I changing the just the work of a growth person.
I
Like to describe growth as the the job is to connect users to the value of your product.
And in order to do that
Um
What I like to do is
Think about that user journey.
Again.
And essentially staff teams that are oriented around each
Element of that user journey. And those teams have specific metric goals, they have roadmaps, et cetera, and then they go run against them. So that's like how it's structured.
Um AI
I think can be applied to speed up some elements of that.
Excellent essentially experiment cycle that you get through.
So one example was in product discovery.
As opposed to
core product which tends to have longer time frames than you might do like
you know, thorough user research or market research. You know, it's more
More foundational, more for first principles, et cetera.
Growth is a little bit less like that. It's like you're running a lot of experiments and you're the output of any given experiment.
is the input like to your next idea.
Right.
And so
historically, I don't even I don't even mean historically, but just a few months ago, right? Like we were operating in a
It's got kind of history, I I suppose.
But you know,
There would be a lot of manual writing of these like analyses docs. You'd have to read them. You'd have to understand what insight you want to kind of grab from them and then write another spec to

That's still happening to some degree, but I think that's a spot where
Even tools like ChatGPT are super helpful, right? You can just plug in like a
an analysis that another person wrote and just have it summarized for you and give you advice on, you know, ideas to go try.
And so that ideation, that research
Cycle was much, much faster.
I talked a little bit about prototyping also just becoming much, much faster than before.
We have not yet gotten to the point where
Like.
product managers themselves are actually shipping the code into production.
But it's dramatically shortened the amount of time it takes to
Conceive.
of especially like a bolder idea that you might have.
And so
when I talked earlier about explore and exploit, right?
a lot of the explore was harder to do.
But now it's a little bit easier to do. You can take a
a broader concept and visualize it. And when you can visualize it, center it around the team, get people to click around it, that makes a world of difference.
Um
So those are just a couple of examples that come to mind.
Awesome. I'm gonna go back to this phrase right at the beginning of this answer that you shared that I think is really helpful.
that you see growth as
Simply your job is to connect users to
The value of your product.
Yeah.
Can you speak more to that?'Cause I think that's such a nice way of
Clarifying what is Growth's role.
Yeah.
Uh
It resonates deeply with me because
I
I feel like growth sometimes gets this
Um
reputation, I guess, that it's just pure like metrics hacking.
Like we're called
cold people that just are trying to move a particular metric up and we're gonna do whatever it can to like
You know.
throw walls and pay walls and add friction and those all these spots.
Um
And even though that could theoretically work in at like a micro level on a specific feature or specific metric.
I think what's most healthy for a company and you know, I want to work at durable companies, right, is to think about
The user, holistically, right?
And
When you take that framing of
connecting um users to the value of your product.
That
value can change for a user over time. And that also lines up really nicely to the journey, right? Like what a
Uh, someone that's not even a user yet needs to understand about the value proposition is super different than what a
habitual user of three plus years uh might need, right? And so the teams working on them should
Think from that perspective.
And then
From there, right, they ladder into like specific problems to solve hypotheses, et cetera.
Following that thread a little bit more.
People listening to this are
Imagining
How do I get better at experimentation? How do I run more experiments? How do we do this better?
What are two or three tips
And best practices that you think people
Need to hear maybe our
not totally aware of when they think about
Getting better at experimentation on our teams.
I think the first thing is just start.
Somewhere.
You know, I I just read this Alassian
uh state of product report and it's like forty percent

Basically don't run experimentation at all.
And there may be some good reasons for it. I mean, it could be philosophical or maybe you're more, you know, B to B oriented or whatever. So I I get it.
But
I think for a lot of especially if you work on a consumer product that has some degree of scale, some degree of frequency with your product.
You can collect enough data.
And also
I have found, you know, I can pattern match all day long. I've worked a lot of companies, right? But I'm wrong all the time. And I think consumer behavior can be very
Fickle.
And especially when you work at a company, you become a power user naturally.
So sometimes you you may forget like what the actual user experience is for a brand new user.
And so you leave a lot of opportunities on the table if you don't even try to experiment. So I just encourage taking that first step.
Just run an A B test.
Find a third party tool or something that you can integrate quickly and or or even just work with your engineers to spin something up. Just get in the practice of, you know.
crawl then walk then run type of thing.
Do you have a favorite tool, by the way, just to throw out? Is there like a go to tool for you?
We used uh Statsega at Grammarly.
And I saw that they recently got acquired, so that was exciting news.
Duolingo and chess.com both have an in house experimentation approach.
Um
Pros and cons to either, obviously.
Duolingo is an experimentation machine.
And so it's been
uh a huge accelerant to have our own thing specifically tailored.
to uh to be excellent at that.
But uh no I I

encourage companies to build
Experimentation in house from day one.
You know, at a certain scale it can make sense. And some of these companies, right, they were started fifteen years ago when these tools weren't out. So it was just something they had to do. Something that you mentioned to me uh at
Chess dot com, your goal is to run a thousand experiments a year. You said you're at two hundred fifty.
Talk about just that as a as a North Star.
Yeah, so part of having team members that are fanatical about chess is that
The company can get
pretty damn far just like building for themselves, building for the community, and not actually being very experimentation and data oriented.
The problem with that
is that you can have relatively lumpy growth.
Right. And so
Part of the the kind of
excitement of me joining the company was to help
smooth that out and bring in that experimentation mindset.
So prior to twenty twenty three, the company practically didn't experiment at all.
Last year they did about fifty. This year they're
On pace for about
Two hundred and fifty.
And then next year we have that ambitious target of a thousand.
Did I make it up? Yes, absolutely I made it up.
But but it's still a target and uh a thing for the teams to to think about.
And a thousand
Experiments by itself.
Like if you just did that, but you didn't learn, you didn't make an impact, then that's kind of a waste of time, right?
The whole point of setting a goal is that you can have conversations about what
would need to be true to actually hit that goal.
And so that leads to
Um insights. Like
Actually we need
not just product management or or engineering to be running these experiments.
Um
We can experiment with life cycle marketing, changing copy of push notifications and emails.
We can experiment with app store screenshots and you know keywords and stuff like that.
That we have all sorts of content marketing teams, et cetera, right?
We could have engineering enable no code for specific screens. Think about our home screen or our pricing screen, where we might want to do a lot of just tests that are configurable without engineering support.
Um
We might want to just like
track our progress and and look at it from time to time and make sure that we have the right, you know, observability around this.
So anyway, that's the stuff that really matters as opposed to the
You know, hitting that goal.
Um itself. So don't tell the team, but I don't actually care that much if we actually hit a thousand.
But uh
I think if we get pretty close and we accomplish some of these things, we'll be in really good shape.
Yeah, we'll make sure none of them watch this.
Um
I think Chess.com is in this ex this is just a such a cool example of a culture shifting dramatically from zero experiments to
Sounds like two years later.
A thousand.
Which is
Like three a day. Like you know, that's you know, it's there's many teams running experiments in parallel, but that's a lot.
What has helped you most shift that culture? Is it just the CO being like this is the way we're gonna go?
Well, what what have you learned about helping shift a culture from no, we're not doing experiments to a thousand experiments a year.
Yeah, I mean definitely a lot of credit to the the CEO and co founders like Eric and Danny. They're amazing. It's not their intuitive way of thinking about growing companies, but
They're
mental flexibility and encouragement, right, to to evolve and add this as a
as a tool for the company has been awesome. And they've been on the front lines preaching product led growth and experimentation just as much as as I have. So I'm glad that you brought that up because I think that is critically important.
For me joining a company to not be at odds with
you know, the the co founders and the existing approach of the company. I think that's absolutely, absolutely critical.
I think the
You know, I started this podcast with the example of the game review and the positivity and how that was shared. I mean, I think those types of things are really what
motivate people, right? They need to see this working in practice. Wins. You can yeah, you need wins. You gotta celebrate them. People feel good about the learning. It's applied across the board. Like
Who's not gonna be energized by that, I think, right? So
Um, you can't just set goals in a vacuum and
you know, create it from from top, right? People have to see it working and
And when it works, like the metrics move and you learn faster and you ship faster, and that that's a that's a great environment to be part of.
What was the first experiment you guys ran, do you remember?
I don't know. Before my time, actually. Okay, okay, got it. So they're already going down this track before they brought you in. They had run they had run a they had run some.
Yeah.
Sweet.
Are there any other
He
Lessons that you think people need to know.
To
be successful running experiments at scale.
The system matters just as much as any given experiment, probably even more.
Right. I think.
Starting with a
growth model so you have an understanding of
how your company grows in the first place and which channels you're going to leverage is critical.
You need to make sure that you are instrumenting your product.
In and out. Otherwise you're gonna run experiments and have wonky results.
Um
I won't name which company, but I I was I was part of a company that had an in house experimentation tool.
About three months into the company, we're running like some experiments and
We realized that user retention was actually configured backwards.
So all positive results were negative results. Oh. So that was kind of embarrassing.
And uh, undo all those experiments and just like that drive up retention. It's kinda weird. Like we're seeing people use the the features a lot more. Why is user retention going?
going negative. Um so I have plenty of horror stories around that type of stuff, but Oh my God.
On the flip side of horror stories, you've shared a bunch of cool examples of experiment wins.
Is there another that comes to mind if one you're really proud of or that was
really trajectory changing either at
Du lingo or Grammarly or or chess.
The so I already shared one at chess.com and one of Grammarly. I mean I could talk a bit about about Duolingo as well. Yeah. Um
Duolingo. Uh and you had Jackson, right? While you talked about this streak. Yes, uh streak. I also don't want to steal his thunder because I was gonna think about that, but
The amount of learning through
um commitment and putting streaks on a calendar and just getting people started.
Right.
As opposed to achieving some large milestone.
Um that was huge.
I I think we
We did something interesting. We spun up a virality team.
And
Virality is this like
Really amorphous thing to me. I think it's really hard to generate.

But Duolingo is a product that is shared quite a bit.
And so we invested actually in some time to
Um, essentially add screenshot tracking for like a brief period of time in the app.
Just so we could find out the hotspots of where users were doing screenshots. And you see this in other apps too. It's not necessarily like
You know.
S some horrible thing, but
We we did this for some period of time and we're able to
basically articulate and say, Okay, um
You know streak milestones was the obvious one.
Really funny.
challenges that you get in the Duolingo experience is also super highly shared.
um advancing in the top three of a leaderboard's not a thing. Anyway, so you can find these different moments where that's the case.
And then we staffed those moments with illustrators and animators and created these really delightful experiences around them.
And I that worked.
Amazingly well. So as opposed to going
Against
I guess human intuition and trying to get them to share stuff that they otherwise wouldn't
on the margins want to share.
Like lean into it more. Actually, like grab the moments where users are already organically screenshotting and make those
Much, much, much.
Better?
And you can kind of five X or 10 X and and drive a lot of growth that way too. So that's not so much an experiment. It's more a core product thing, but yeah, it's just
Resonated with me that that was interesting.
Well connects to your explore and exploit methodology, just find where
Explore where things are happening and then
try to exploit in a nice positive way. You got it.
Speaking of that, you mentioned this with Duolingo is just very good at habit.
Formation and
Motivation, behavior.
Feels like chess is good at this too. Mm-hmm.
You worked at
both these companies. What have you learned about
How to motivate people, how to create habits.
Again, like Duolingo would not have started without this
uh insight from day one, right? The
They aim to to focus on motivation and build a lot of these like tactics.
Um
Jorge actually had this model of like gamification uh patterns having
Essentially three pillars to it. You have the core loop.
You have the uh metagame and then you have the profile.
And so we actually thought about it that way too, where you know, your core loop is is your lesson.
that you go through, you do a lesson, you get some rewards, you extend your streak, and then the next day you get a push notification. It's kind of the core loop of the product and making that really tight is is super important because people need a habit to stick to.
Then you need a meta game.
Which for Duolingo is kind of like the path.
but it's also the leaderboard achievements, kinda long term things that you're gonna strive to.
such that you have like long term
I guess motivation uh to continue doing the thing.
And then the profile is also critical because you build up a profile over time. It's a reflection of your
investment inside the product experience.
And so when you nail those
three things you can end up with a long term learning journey.
That can be quite successful.
And then to flip over to the chess dot com side, like what we see is that
Over seventy five percent of our new users, they classify themselves as like I'm completely new to chess or I'm a beginner.
And unfortunately
If you're new to chess and you're a beginner, you're not gonna have that fun of a time playing live games.
We see this in the data. It's like
Less than a third of those users actually win their first game.
And when you lose a game, user retention is ten percent worse than when you win a game. That's not so bad, but my at scale that's bad. Yeah, and and it could be worse. That's true.
But
And so typically what like a lot of mobile games will do is they'll just create like a super simplified version of the game. It's harder for us to do.
HS.
And so without changing the rules of that, right.
I think that that's uh
I don't know. It's just very
eye opening to me that
When you're trying to learn something, whether that be language learning or or chess or whatever, um
Usually those first steps are fraught with, you know.
A lot of
self doubt and reinforcement that you're not good at the thing. And so
It's it pays to be very intentional to craft experiences that
You know, guide the user around that.
Well I can't help but ask, is there anything that helped that along?
Yeah, so like something we're experimenting right now is just like clearly if you say that you're new to chess.
we're gonna craft a more delightful learn how to play experience as opposed to dropping into a live game. That's an example. Another is like hiding your ratings for the first five times, such that you're not seeing your rating kind of plummet.
Um so there's a lot of tips and tricks you can do.
I'm just imagining a little guide that's like here's how you win. Sh play against a coach, play against a friend, play against a bot. There's there's a bunch of different avenues you can take. What I'd love is play against someone real and then here's like tr here's where you should move. Just like here.
Here's like a hint in real time. Yeah, yeah. Yeah, yeah. Well, I don't I don't wanna be playing you then.

Okay, let me ask you a couple more questions.
One is just zooming out a little bit.
What's
What's the most counterintuitive lesson you've learned about?
Building products or building teams across the many companies you've worked at?
Yeah, I've talked a lot about products, so maybe I'll flip to the team side for a bit.
Um
I think the standard way to hire and build a team
is you know, you fill out a J D, it's got a whole bunch of different characteristics that you're looking for.
You typically will find
You know, a short list of companies that are kind of similar to yours, and then you try to hire for that, right? I think that's kind of the the typical default path that that a lot of companies take.
And I was really struck by, you know, my experience working at you know some smaller startups or you know take Duolingo as an example where
Um
over and over and over like I saw some of the highest performers just being people that
had very high agency, had that clock speed, had that energy.
Um yes, they they cared about the mission.
But they didn't necessarily need to have deep
Experience on that matter?
And in fact, sometimes that experience could be a crutch in in certain ways.
Especially in this world where
The grounds are shifting so fast with AI.
a lot of your like learned habits actually need to be intentionally discarded. You know, you need to have a beginner's mind on this type of stuff.
So I think this is more true than ever. Um
Like looking for people that
respond and move quickly and and think
Um
You know, just
just faster and move faster, right? I think speed
The fastest speed of learning.
Uh those types of companies are the ones that I want to bet on because I think those
Those will end up uh surviving and thriving.
So just to double click on this, this idea of high agency is very
Uh
trending these days of just like higher high agency people.
To unpack that a little bit, you mentioned a few of these
Traits.
So let's just help people see what you see. So one is clock speed just
They think fast, they move fast.
Uh
They learn fast.
What else what else do you look for that helps you see that they're high agency people?
Yeah, I mean a lot of it actually happens outside of the interview process, interestingly. So a lot of it is uh, you know, the types of questions they asked. Have they actually tried your product and gone deep into it? A lot of it is the, you know.
It's the references. It's the like communication that they have to even set up your interview. Like it's the energy they bring into the conversation.
You can actually pick up a lot of soft signals.
On some of these traits.
Um
Yeah, over time you kinda pick up on some of these patterns. I don't know that I'm
perfect at it, but I've I've learned to
balance those things quite a bit more than I did in the past when I would just purely read from my questions in my rubric and not care about anything else.
Yeah, there's like a vibes component to it.
This is also kind of support for the work trial way of interviewing versus
Just a talk interview where you have them actually work with you for a week or whatever.
That's a great point.
Okay.
Uh one other question I want to ask you. You worked at a bunch of different sizes of companies.
From startup to
Grammarly. I don't know if it's called a big company, bigger company.
Duolingo's Duolingos. I don't know. How big is Duolingo's like they're about a thousand people. Okay, cool. Yeah. But I wor I worked at Google too to start my career. Oh right. Okay.
What have you learned about just the size of company that
makes you happy. What have you learned about just helping
Other people that you talk to decide what size of company is good for them.
I I definitely believe that everyone has a company stage that they shine best at.
I've personally gone through this journey of
Big tech.
To like tiny, tiny, tiny startup.
then landed in the middle, which I consider like my own gold lock zone.
I talked earlier about like
What actually gives me
personally a lot of energy is seeing
across a company's
Efforts.
But also
The company being small enough that I can get into the details. I can work with the specific teams. I can read experiment.
You know.
Results. I can look at the pixels.
And so I find that the balance of those two things tends to fit best with medium sized companies.
But that's me, right? I I think at big companies
uh like a Google
You're dealing with immense scale.
Which is interesting by itself.
You learn a lot of best practices from your peers. They have all the kind of
tools and functions that you would possibly want to go learn from.
But they can tend to move slower.
And
it's harder to kinda ship things and get them out the door, which, you know, eventually drove me nuts a little bit.
On the flip end of the spectrum, these tiny startups
They move incredibly fast.
But I grew like all my gray hair from those tiny startups because no one knows about your company. And so you're recruiting people one by one. You're you know, trying to get users one by one.
So yeah, you can you can learn fast and ship a lot of things, but if you're trying to make a big impact on the world
it can be actually pretty grueling to do so at at really, really, really small startups.
Now some of them do hyper scale and make it out and um
Obviously.
I I'm not one to to trash that'cause that's the path that I tried for for quite a while.
But for me, like I really like the zone where I can
contribute at scale, but also execute.
At a pace that's more on like the daily and weekly scale, right, as opposed to monthly and quarterly.
And when you say medium, what size of company is that roughly?
Yeah, so
Um, these companies that we've talked about in the podcast are about five hundred to a thousand people.
Typically these companies will have been around let's say ten to twenty years. Like they're durable, ideally profitable.
Um, they have a good leadership team.
But there's still a lot of dimensions to go figure out.

Key inflection points.
Um so they're
Certainly not.
Stagnant, right? You you need to find a place that's dynamic too.
Interesting. Ten to twenty years old.
Uh I don't know.
Not many people would feel like that's where I want to be.
Yeah. Uh I love that you found a number of companies like that that you enjoyed working at.
Last question.
And this is gonna be taking us to a recurring segment on the podcast that I call Failed Corner.
People hear all these stories of all these experiments on all these companies it worked at, they're all killing it up and to the right.
Uh in reality, you've touched on this
A lot of things don't work out great.
So do you share a story when
Something.
went wrong when you failed.
And what that taught you.
First of all, in the growth world, you're failing all the time. So I'm not going to pick a specific growth story because those don't actually in my ego too much, but
Earlier in my career I did a lot of core product work.
Um, I worked for this startup called Chariot.
I don't know if you ever lived in San Francisco, but the bus super. The blue commuter shuttles, like fifteen person shuttles. Uh they would essentially drive from
various neighborhoods into downtown San Francisco. It's kind of a commuting use case a cross between
Like the the public bus system and and an Uber and Lyft.
Um so I was there for some time. I I led product there.
And
You know, the the core service was was really loved by its users. Like it was, you know, reliable and fast and you know, affordable enough.
But
We got pretty interested in this idea that
You know, maybe we can improve utilization
Uh, maybe we can make the service a little bit more innovative if we offer dynamic routes more similar to Uber and Lyft.
Like how could we
Like the drivers are driving these fixed routes, but you know, if they have spare time, they can go out of their way, go pick up somebody like at their house or something and like keep going.
Um
So we tried this, we called the chair direct.
Really interesting.
attempt, but I learned a lot of lessons there,'cause ultimately it didn't work out.
One lesson is like this was kind of a solution searching for a problem. Like you never just purely want to chase a like, you know, wouldn't it be nice if we did this as opposed to
Yeah, this is our our user and this is the problem that we're solving. This is why it's gonna delight them, et cetera.
That's kind of one.
Second is um
You gotta consider, especially in
these more like marketplace type businesses, there's more than just one end user.
And uh we focus so much of our attention on the writer app without realizing
Oh yeah, the drivers are carrying a lot of the brunt.
of this experience.
And our operations team is as well, right?
And so when the di the drivers are confused or disgruntled, that can lead to a a challenging
uh overall experience for for the product. Right. So that's definitely another one.
And the third one is like we did a lot of actually
P R
uh prior to the service going out, just to get the word out.
Um
And you know, PR has has its time and place, uh, but I think
Doing it before you have validation.
That customers.
definitely want the thing is quite risky.
It can lead to a lot of sun costs.
Uh once you get it out because you're s you're just
You know, you need to see it through, you want to see it succeed.
So yeah, this is a decade ago. Honestly I had a great time at that company, but I still remember that.
Devidly,'cause it contained
You know.
three or more kind of key lessons that carried forward.
As I have built
Many products since then.
Yeah, feels like you went to the complete other end. Run experiments of everything before you tell anyone about it. That's right.
Yeah. I remember the chariot bus showing up at the Air B office and people getting in like what the hell's what the hell is this? That's right.
Yeah. Very cool. I didn't know you work there.
Uh Albert.
We've covered so much ground. Everything I was hoping we'd cover. Is there anything else that
You wanted to cover anything else?
you want to leave listeners with before we get to our very exciting lightning round.
Um no, this is great. Like hope it was useful for your listeners. I will say like over the last

I w I was honestly a little bit anxious about like, you know, do I have enough deep
you know, independent frameworks that I need to come up with.
But just being authentic to my actual experience at these companies, you know, a lot of my lessons learned.
have been off the backs of other people that have tried.
you know, similar things and have succeeded or failed. And I think what's important is that you're
You have that like
You're a mental sponge, right? You can try a bunch of different things, you can absorb them and then put them in practice right away.
discard the things that don't work, right? And evolve them.
For yourself and for the company's needs.
Um
And so I I don't know, I think that was just a realization that I had as I was was thinking through.
this podcast. And I think that's partly why I haven't done too much public speaking.
I know exactly what you mean when I left Airbnb. I was just like
And and that was the first time I ever took a break in my career of like
Like thirty years of just working straight in school.
I was just like, what have I actually learned?
I've never just sat down and thought about here's a thing I've learned.
And that
led me to writing this medium post that did really well what I learned at Airbnb and then that
Basically the led to what I do now. So there's a lot of power and like I love that this is the excuse
To make you
think through what have I learned concretely.
That I can share.
That's right.
Thank you for that.
Yeah, and I actually so at the beginning of this podcast, before we started recording, I always like to ask guests
What is your goal? What do you want to get out of this conversation?
And you know, usually it's like we're hiring, we want to make sure people know about our company, or we wanna get the users.
Uh and your answer is just I just want to give back things I've learned.
Which I love.
That's it.
And you've done that.
With that, we reached our very exciting lightning round. I've got five questions for you. Are you ready? I'm ready.
What are two or three books that you
I and yourself recommending most to other people.
Yeah, the truth of it is like I have a not just the four year old, but I also have a one year old. So most of the books that I'm reading these days are are kids' books.
Um trying to make them laugh and all kids' books. Well you said that you started singing. There's a book called Snuggle Puppy.
That that has uh a song in it that just makes my daughter crack up. So that is heartwarming for me.
But um
No, I like a book that I recommended recently at work is uh Ogulvie on advertising.
You know this book.
I don't know the book I've seen is like tenants of Martin. Yeah, it's interesting. So it's forty years old.
But it's just packed with a bunch of different practical examples about copy and creative that that work in
You know, these are old school ads, right? But
You know, he took a very experimentation oriented approach.
to just try a lot of things. I think in the book it makes a good reminder that
What ultimately matters is to compel
your users to some action.
you know, for him is like buying a product, right? It's not about just creating clever ads or sexy, you know, creatives.
It's to do things that
You know, compel that action.
I think that's very true for many of our product and, you know, life cycle teams. And so I shared that around as a as an interesting recommendation.
Is there a movie or TV show? Sorry, were you gonna share another book?
Uh yeah, actually. So yes, please. My our our co founder at chess.com. Uh his name's Danny Rensch, and he is quite well known in the the chess circles.
Um, he's releasing a memoir called Dark Squares, and it is super fascinating. Um, he grew up in an abusive cult.
And was a chess prodigy.
And so it is just this like
Uh
unbelievable story and I'm about halfway through it.
It's a reminder that sometimes the people that you work with you don't realize like how deep their past go.
Um, but this is something else and uh I think it should be out by the time this this podcast uh
Releases.
And it's called Dark Squares. Dark Squares. Which is a reference to the chessboard and also imagine the
Difficult past. Exactly.
Wow. How cool.
Okay.
Is there uh
Are there movie or TV shows you really enjoyed the you uh
You've recently watched. I mean, these days it's football season.
So I'm consumed by all the hot takes.
of my favorite teams that I love and
The teams I love to hate as well, so
Uh
Who's your team?
Uh the forty niners.
I have season tickets and um I go all the time. We had a rough season last year, so hope hoping to turn around.
Okay, very cool. Okay. Uh

Yeah, so last twenty years of my life, roughly, uh I've moved around a lot.
But I've always been within walking distance of a coffee shop. It's just like a ritual that I go and
Get coffee and it starts my day right.
Two years ago I bought a house.
And for the first time ever in my life, I'm like not by a coffee shop. And I was so depressed about this for a little while.
So my favorite product is uh is the Breble uh Barista.
And it just starts my day off right.
Um, I like making horrible latte art with it.
And uh I think it's just a reminder, I don't know.
Like
The products that most
um impact me, I guess are the ones that uh
I use all the time.
It's a daily.
And then the most caffeine. You got it.
Amazing.
Diva
Favorite life motto?
That you find yourself using
Uh in work or in life.
I was I was thinking about my piano stories, I also remember that my mom uh used to have a quote that's just she just said, like, nothing is more important than your reputation.
And she used to say this and
I think the charitable
understanding of this is that a lot of the
small decisions that you make each day. How do you treat people? How do you show up?
What's your character, et cetera.
They can compound.
And they open doors for you and
Many surprising and amazing ways, right? Like a lot of these companies that have actually joined
Have come through relatively light connections.
And even just being on this podcast, right? I think I've I've seen
A number of folks that I've worked with before, uh, beyond the show and
Um so I think
You know, doing the right thing.
building a good reputation that can carry a long way. And the flip side of that is, you know
reputations are fragile too, right? So if you do the wrong thing, take a long time to repair that.
Um
So I don't know, it just stuck with me my entire life. I I thought that was a
Interesting life motto.
Last question. York at chess dot com. How's your chess?
Terrible compared to serious, serious players, but but quite good compared to the the casual ones. Yeah. Okay. My my yellow rating is about uh eighteen hundred.
For Rapid Beleza, and about fifteen hundred for Blitz.
Yeah.
And I play many times every day.
Blitz is like fast chess. Blitz is like faster chess, kinda three minute
games Rapid is more like a ten minute game, which is still pretty fast.
And you say you play multiple times a day.
Okay, like a Patagonia, uh there's a famous book the founder wrote called Let My People Go Surfing and the rule at Patagonia is you can go surfing if the waves are great. Is that how it works at chess dot com? Absolutely.
Okay. Chess is always fun, so we play all the time. And they even have chess coaches like on staff.
Yeah, so staff just like you can book you. You can book. So I get bi weekly lessons and it's helping me improve.
Wow. Okay. This is gonna drive a lot of hiring for you guys, uh
Saved it for the end. Albert, this was awesome. Thank you so much for doing this. Thanks so much for giving back and sharing all these stories.
Two final questions. Where can folks find you if they wanna follow up on some of this stuff.
And how can listeners be useful to you?
Yeah, thanks for having me. This is great. Um you can find me on LinkedIn or Twitter.
Not a super active poster, but I read it all the time.
Uh, if there's something that I said today that resonates with you and you know, you just want to get in touch, trade notes, feel free to reach out.
And can they play with you on like can they find you on just dot com to play? They can.
Okay. Uh do you want to share your username or you don't want that?
Uh I'm happy to. I just mentioned that I'm a forty niners fan, so my username is Go Niners.
Uh wow.
I'm sure I'll get a lot of game requests. Eighteen hundred. Okay, Albert, thank you so much for for being here.
Yeah, thank you so much.
By everyone.
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.
Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast.
You can find all past episodes or learn more about the show.
at Lenny's podcast dot com.
See you in the next episode.