Inside ChatGPT: The fastest-growing product in history | Nick Turley (Head of ChatGPT at OpenAI)

0:00 You were a product leader at Dropbox, then Instacart, now you're the PM of the most consequential product in history. I didn't know what I would do here because it was a research lab. The first task was like fix the blinds or something like that. When someone offers you a rocket ship, don't ask which seat. We set out to build a super assistant. It was supposed to be a hackathon code base. What was it called before? It was gonna be chat with GBT three point five. Because I really didn't think it was gonna be. And then Sam Alman's just like hey let me tweet about it. This is a pattern with a you won't know what to polish until after you ship. My dream is that we ship daily. By the time people hear this, they're gonna have their hands on GPT five. About 10% of the world population uses it every week. With scale comes responsibility, it just feels a little bit more alive, a bit more human. This model has taste. Kevin We all, your CPO, said to ask you about this principle of is it maximally accelerated? I just really want to jump to the punchline. Why can't we do this now? I always felt like part of my role here is just set the pace and the resting heartbeat. Everyone's always wondering is chat the future of all of the stuff? Chat was the simplest way to ship at the time. I'm baffled by how much it took off. I'm even more baffled by how many people have copied. Chat GPT is now driving more traffic to my newsletter than Twitter. That is a type of capability that has been incredibly retention. I've been really excited about what we've been doing in search. Can you give us a peek into where this goes long term? ChatGPT feels a little bit like MS DOS. We haven't built windows yet, and it will be obvious once we do. Today my guest is Nick Turley.

1:17 Nick is head of Chat GPT at OpenAI. He joined the company three years ago when it was still primarily a research lab. He helped come up with the idea of ChatGPT and took it from zero to over 700 million weekly active users, billions in revenue. And arguably the most successful and impactful consumer software product in human history. Nick is incredible.

1:38 He's been very much under the radar. This is the first major podcast interview that he has ever done, and you are in for a treat. We talk about all the things, including the just launched GPT five. A huge thank you to Kevin We, Claire Vogue, George O'Brien, Joanne Jing, and Peter Ding for suggesting topics for this conversation. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. And if you become an annual subscriber of my newsletter, you get a year free of a bunch of incredible products. Including Lovable, Replic, Bold, N8N, Linear Superhuman, D Script, Whisperflow, Gamma, Perplexity, Warp, Granola, Magic Patterns, Raycast, Chapier, D and Mobbin. Check it out at Lenny's Newsletter.com and click bundle.

2:17 With that, I bring you Nick Turley. This episode is brought to you by Orchis, the company behind Open Source Conductor, the orchestration platform powering modern enterprise apps and agentic workflows. Legacy automation tools can't keep pace. Siloed low-code platforms, outdated process management, and disconnected API tooling fall short in today's event-driven, AI-powered agentic landscape. Orcas changes this. With Orcas Conductor, you gain an agentic orchestration layer that seamlessly connects humans, AI agents, APIs, microservices, and data pipelines in real time at enterprise scale. Visual and code first development, built-in compliance, observability, and rock solid reliability ensure workflows evolve dynamically with your needs. It's not just about automating tasks, it's orchestrating autonomous agents and complex workflows to deliver smarter outcomes faster.

3:08 Whether modernizing legacy systems or scaling next-gen AI driven apps, Orcus accelerates your journey from idea to production. Learn more and start building at orcas.io slash Lenny. That's O R K E S dot IO slash Lenny. This episode is brought to you by Vanta, and I am very excited to have Christina Cassiopo, CEO and co-founder of Vanta, joining me for this. Very short conversation.

3:31 Great to be here. Big fan of the podcast and the newsletter. Vanta is a longtime sponsor of the show. But for some of our newer listeners. What does Vanta do, and who is it for? Sure.

3:42 So we started Vanta in twenty eighteen focused on founders. Helping them start to build out their security programs and get credit for all of that hard security work with compliance certifications like SOC Two or ISO twenty seven oh one. Today, we currently help over 9,000 companies, including some startup household names like Atlassian, Ramp, and Langchain, start and scale their security programs. And ultimately build trust by automating compliance, centralizing GRC, and accelerating security reviews.

4:12 That is awesome. I know from experience that these things take a lot of time and a lot of resources. And nobody wants to spend time doing this. That is very much our experience, but before the company and some extent during it. But the idea is with automation, with AI, with software, we are helping customers build trust with prospects and customers in an efficient way. And you know our joke, we started this compliance company, so you don't have to. We appreciate you for doing that. And you have a special discount for listeners. They can get a thousand dollars off banta at banta.com slash Lenny. That's V-A-N-T A.com slash Lenny.

4:46 For one thousand dollars off Anta. Thanks for that, Christina. Thank you. Nick.

4:56 Thank you so much for joining me and welcome to the podcast. Thanks for having me, Lenny. I already had a billion questions I wanted to ask you. And then you guys decided to launch GPT five the week that we're recording this. So now I have at least two billion questions for you. I hope you have I hope you have a lot of time. First of all, just congrats on

5:12 The launch. It's coming tomorrow, the day after recording this. Just uh congrats, how you feeling. I imagine this is an ungodly amount of work and Stress. How you doing? It's a busy week, but you know, we we've been working on this for a while, so it also feels really good to get it up. So by the time people hear this, they're gonna have their hands on GPT five and the newest Jet GPT. What's the simplest way to just understand what this is, what it unlocks, what people can do with it? Give us kind of the the pitch.

5:39 I'm so excited about GPT five. It uh I think for most people is going to feel like a a real step change. If you're the average chat GPT user and we have you know seven hundred million of them. Um this week. We uh you've probably been on GPT four O for, you know, a while.

5:55 You probably don't even think about the model that powers the product. And GPD five is is it just Categorically different. I'll talk about a lot of the specifics, but

6:04 You know, at the end of the day the vibes are good. At least we feel that way. We hope that users feel the same. Um and increasingly that is the thing that I think most people notice. Right. Um they don't look at the academic benchmarks, they don't look at Evaluations. They try the model and and see what it feels like. And just on that dimension alone. I'm so excited. I've been using it for a while.

6:24 But it is also, you know, the smartest, um, most useful And um Fastest frontier model. Um that we've Ever launched.

6:32 You know I'm pure smarts. One way to look at that is academic benchmarks. on many of the standard ones, um, whether or not it's math or reasoning or you know, just raw intelligence. This model state of the art. I'm especially excited about its performance on coding.

6:48 Um whether or not that's sweebench, which is a common benchmark, or Actually front end coding is really, really good. Um as well. And um That's an area where I I feel like there's a there's a true step change improvement in in in GPT five.

7:03 But really no matter how you sort of measure the smart it's it's it's quite remarkable and I think people are gonna feel the upgrade. Especially if they weren't using O three already. And you know the the second thing, um Yeah on smarts is it's just really useful.

7:17 Coding is one axis of utility. Whether or not you have coding questions or you're Vive coding an app. Um But he's also a really good writer. I write for a living.

7:27 Uh Internally, externally. I just Wrote a big blog post. Um That we publish Monday and you know this thing is like Such an incredible editor.

7:35 Um and and you know, compared to some of the the the the older models, it's just got it's got taste. Which I think is really exciting. And um to me that's like something that is truly useful um in in in my day to day. And um there's other a bunch of other areas, like it's it's state of the art on health, which is useful when you need it. But again, the the sort of the thing you can't really express in use cases or even

7:56 Yeah, in use cases or or data. is sort of the vibe of the model and it just feels a little bit more alive, a bit more human. In a way, that is it. Kind of hard to articulate until you try it. So Feel good about that. And yeah, as mentioned, it's faster.

8:07 Um it uh It thinks too, just like O three did. But you don't have to manually, you know, tell it to do that. It'll just dynamically decide to think when it needs to. Um and when it doesn't. Ends up feeling quite a bit faster than

8:23 Using a three did? And then you know. Maybe the thing that's most exciting is that we're making it available for free. And that's like one of those things that I feel like we can uniquely do at OpenAI because You know, many companies I think if they have a subscription model like us, they would gate it behind their paid plan and for us, you know.

8:39 If we can scale it, we will. And that just feels awesome. We did that with four O as well. So everyone's gonna be able to try GPT five uh tomorrow, hopefully. How long does something like this take? Like I don't know if there's a simple answer to this, but just how long have you guys been working on GPT five? We've been working on it for a while. Um you know you can kind of view GPT five as a culmination of a bunch of different efforts. Yeah, we had the reasoning tech, we had a more c classic post screening.

9:02 uh methodologies. Um and um therefore it's really hard to put a beginning on it. But you know, um it it really is kind of the end point of a bunch of different techniques that we've been for a while. Can you give us a peek into the vision?

9:16 For Where JPT is going, GPT in general is going. Like if you look at on the surface, it's just it's been kinda the same idea. with a much smarter brain for a long time. I'm curious where this goes long term.

9:27 So to to maybe back up a bit. Um Now you think of Chat GPT as this kind of ubiquitous product. Um again, about ten percent of the world population uses it every week. Um I think we have like five Million business customers now. Um it's like you know.

9:43 An established category in its own right. When we started We set out to build a super assistant. That's what we That's how we talked about it at the time.

9:51 In fact, the code base that we use is is called SA Server. Um It was it was supposed to be a hackathon codebase, um but you know, things things always turn out a little bit differently. And uh uh so so yeah, in some ways. That is still the vision.

10:05 The reason I don't talk about it more than I Yeah. do is because I think assistant is a bit limiting in terms of the mental model we're trying to create. You think of this like very personified human thing. Maybe utilitarian. Maybe uh, you know.

10:17 And and frankly, uh, you know, having an assistant is not particularly relatable to most people unless they're like in Silicon Valley and they're a manager or something like that. So it's imperfect, but like really what Yeah. we envision is is this entity that can help you with any task, whether or not that's at home. Or at work.

10:32 What school? Um really any Context and Uh it's essentially that you know. knows what you're trying to achieve. So you know, unlike ChatchBitch today.

10:40 You uh uh don't have to describe your problem in in in minute detail. 'Cause it already stands your overarching goals and has context on your life, et cetera. Um So you know. That's one thing that we're really excited about. Um, the sort of inverse of giving it more inputs. On your life is

10:57 Giving it more action space. So We're really excited to allow it to do. Um over time. What a smart A pathetic human with a computer could do for you.

11:06 Um and I think Yeah, the limit of the the types of problems that you can solve for people once you Um Is is very, very different than what you might be able to do in a chat bot today. So you know, that's more outputs and

11:20 I often think, Okay, you know, I'm a general intelligence. If I what would happen if I you know became Lenny's uh intern or something. Um and you know, I wouldn't be particularly effective despite, you know, having both of those attributes that I just mentioned. Um, and it's because, you know, um I think this idea of Building a relationship with this technology is also incredibly important. So

11:38 That's maybe the third piece that I'm excited about is Building a product that can truly get to know you over time. You saw us launch some of those things, you know, with uh improved memory earlier this year and that's just the beginning of what we're hoping to do. So that it really feels like your AI. So I don't know if super system is still the right um exact analogy, but

11:56 I think people will just think of it as their AI, um, and I think we can put one in everyone's pocket. And uh um help them solve real problems. Whether or not that's becoming healthy, whether or not that's you know, um starting a business, whether or not that's you know Just having a second. Can you money?

12:12 Um, there's so many different problems that you can help with people in their in their daily life and that's what motivates me. So an interesting uh kind of between the lines that I'm reading here is The vision is for to be an assistant for people, not to replace people. It feels like a really important Uh.

12:26 piece of the puzzle maybe just talk about that. Yeah, it's really scary to people. Um and I understand, you know, there's Decades of movies on AI that have a certain mental model. Kind of baked in.

12:37 And even if you just look at the technology today, once everyone I think has this moment where the I does something that was really deeply personal to them and you're like kinda thought, hey the A I can never do that. Yeah, for me it was like weird music theory things where I was like, Wow, this thing actually like

12:51 understands music better than I do, and that's like something I'm passionate about. And uh yeah. So so it it's naturally scary and I think the thing that's been really important to us, um For a long time is to Build something that feels like it Yeah, it's helpful to you, but you're in the driver's seat.

13:08 And that's even more important as this stuff becomes agendic, right? Um like the feeling of being in control. And that can be small things like Yeah. We built this watching what the AI is doing when it's in agent mode. Um it's not that like you actually are gonna watch it the whole time, but it gives you a mental model and makes you feel in control.

13:24 In the same way that when you're in a Waymo, you you get that screen for those of you who've tried. Wait. Yeah, you can see the other cars. It's not like you're gonna actually watch, but it gives you the sense that you know how this thing works and what's happening. Or we you know, we always check with you to confirm things. It's a little bit annoying, but it puts you in the driver's seat, which is which is um

13:40 Important. And For that reason, you know, we always view technology and the technology that we build as something that amplifies what you're capable of. Rather than replacing it? And uh that becomes important as the deck gets more powerful. Okay, so you mentioned the beginnings of chat GPT.

13:56 I was reading in a different interview. So you joined OpenAI. Chat GPT was kind of just this internal experimental project. that was basically a way to test GPT three point five. And then Sam Altman's just like, Hey, let me tweet about it, maybe see if people find this interesting. Yada yada yada. It's the most uh successful consumer product in history.

14:14 I think both in growth rate and users and revenue and just Absurd. Can you give us a glimpse into that early period before it became something everyone's obsessed with? Yeah. Um so we had decided that we wanted to do something consumer facing, I think, you know. right around the time that GPT four finished training. And it was actually

14:33 Uh mainly for a couple of reasons. You know, we already had a product out there, which is our developer product. That's actually what I came in Um To help with initially. And uh yeah, that has been amazing for the mission. In fact, it's grown up and how it's the open AI platform with

14:47 I don't know, four million developers, I think. But you know at the time It was you know early stage and and we were running into running into some constraints with it because um We There's two problems. One

14:57 you couldn't iterate very quickly because every time you would change the model, you'd break everyone's app. So it was really hard to try things. And then the other thing, um was that it was really hard to learn. Because the feedback we would get was like the feedback from the end user to the developer to us. So it was very disintermediated and

15:13 We were excited to make fast progress toward towards AGI and it just felt like We needed a more direct relationship with with consumers. So we were trying to figure out where to start and you know in classic open yeah fashion, especially back then. Um we put together a hackathon of enthusiasts of just

15:28 hacking on GPT four. To kind of see what awesome stuff we could create and maybe ship to users. And um Everyone's idea had was was some flavor of a super assistant. Like they were more specific ideas, like we had a meeting bot. That would call into

15:42 uh meetings and you know the vision was, you know, maybe we would like help help help it will help you run the meeting over time. We had a coding tool, which you know Uh Full circle now. Probably ahead of its time. Um and you know the the challenge was that with we tested those things, but every time we tested these more bespoke ideas.

15:59 People wanted to use it for all this other stuff because it's just a very, very generically powerful technology. So after a couple of months of prototyping We took that same. kinda crew of volunteers. And it was truly A volunteer group, right? We had like someone from the supercomputing team.

16:13 Who had built an iOS team uh iOS app before we had um someone you know on the research team who had written some back end code in their life. They they they were all part of this initial chat GPT. Team And we decided to ship something open ended because we just wanted a real use case distribution. Um, and this is a pattern with the I I think where you know you really have to ship To understand what is even possible and what people want.

16:35 Uh rather than being able to reason about that a priorite. So chat GPT came together at the end because we just wanted the learnings as soon as we could. And um we shipped it before before the holiday. thinking we would sort of come back. And get the data and then wind it down. And obviously that part turned out super differently because um

16:53 Um people really liked the product as is. Um so I remember sort of going through the motions of like, Oh man, dashboard's broken. Oh wait, people are liking it. I'm sure it's just you know going viral and and stuff is gonna die down to like Oh wow, people are retaining, but I don't understand why.

17:08 Um and then eventually we kind of like Yeah. fell into product development mode, but it was a little bit by accident. Wow, I did not know that uh ChatGPT emerged out of a hackathon project. Definitely the most successful hackathon.

17:21 I like to tell the story when we when we talk about uh when we d when we do our our hackathons because I really do want people to feel like they can ship their idea, and it's certainly been true in the past and we'll continue to make it true. If you don't want to share these things, but I wonder who that team was. The team's um largely still around. Some of the researchers working on GPT five actually, you know, they're We're always part of the the Chat GPT team. Um engineers are still around. Um Designer Um designers are still around.

17:47 I'm still here, I guess. So yeah, you've got the team um still running things, but obviously we've grown up tremendously and we've had to because you know, with scale comes responsibility and um Yeah. Um we're gonna hit a billion users. Soon. And you you kinda have to begin acting in a way that is appropriate. Um

18:04 Um to that scale. Okay, so let me spend a little time there. So Uh uh I don't know if this is a hundred percent true, but I believe it is that ChatGPT is the fastest growing, most successful consumer product in history. Also the most impactful on people's lives.

18:19 I feel like it's just part of the ether of society now. It's just my wife talks to it like it every question I have, I go to it, voice mode. My wife's just like let me check with check with Chat GPT. It's just such a part of our life now. And And I think it's still early. So many people don't even know what the hell is going on.

18:37 Just as someone leading this, how does just Do you ever just take a moment to reflect and Think about just like holy shit. I have to. It's

18:47 Quite humbling to get to run a product like that and um I have to binge myself. Very frequently. And I also have to sometimes sit back and let you know just think. Which is really hard when things are moving so quickly. Yeah.

19:00 I love setting fast pace. Um at at the company, but in order to do that with confidence, I r you know I need at least one day. Every week that I'm like entirely unplugged and I'm just thinking about, you know What what to do and process the week, et cetera. Um

19:14 And uh The other thing is. I've never ever worked on a product that is so empirical in its nature where If you don't stop and watch and listen.

19:29 To what people are doing. You're gonna miss so much. Like both on the utility and on the risks, actually, because Normally, you know, by the time you ship a product you you you uh No. You don't know if people are gonna like it. That's all always empirical, but you know what it can do.

19:45 With AI because I think so much of it is emergent. You actually really need to stop and listen after you launch something and then Yeah. Iterate on on on the things people are trying to do and then and and on on the things that aren't aren't quite working yet. So for that reason alone, I think it's very important to Yeah.

20:01 Take a break and and just watch what's going on. Okay, so you take a day off every week. Not off, okay, that's not the right way to put it. You take a day of of thinking time, deep work. I I need it. Yeah, yeah, yeah. And and um I need to hard unplug, you know, on a Saturday or something like that. On a Saturday But uh you know, it it's just not possible otherwise. It's this has been a giant marathon for three years now. Um like a sprint to marathon. Sprint marathon, that's right. Or interval training or something. I I don't know how to exactly describe the open air launch cadence, but you know.

20:33 Uh y you gotta you gotta you know. Set yourself up in a way that is sustainable. Even even at if even if this wasn't AI and it didn't have the interesting attributes that I just mentioned, I think you you would need to do that. But um especially with AI, it's important to go watch. So on along those lines, I talked to a bunch of people that work with you that work at OpenAI. Uh Joanne specifically said that uh urgency and pace.

20:52 are a big part of how you operate that that's just uh something you find really important to create urgency within the team constantly. Even when you are the fastest growing product in history Growing like crazy. Talk about just your philosophy on the importance of pace and urgency on teams.

21:07 Well, it it's nice of her to say that. Um Yeah, I I spent a lot of Two things. You know, with Chat G V T I m when we decided to do it. Yeah. prototyping for so long and I was just like, you know, in ten days we're gonna ship this thing and you know we did. So

21:23 That was like maybe. a moment in time thing where I just really wanted to make sure that we go learn something. Um but for Yeah, ever since then I I just spent so much time thinking about why Chat GPT became successful in the first place. And I think there was some element of just doing things.

21:38 Where you know there were many other companies that had Um technology. in the LM space that just never got shipped. Yeah, I just felt like you know, of all the things we could optimize for.

21:49 Learning as fast as possible. is incredibly important. So I just started rallying people around that. And that took different forms like for a while when we were I just Ran is like, you know, daily release sync and I had everyone he was required to make a decision in it, and we would just talk about what to do and to pivot from yesterday, et cetera. Obviously at some point that doesn't scale, but

22:06 I always felt like part of my role here obviously was like to think about, you know The direction of the product, but also To just set the pace. And the resting hard beat. Um for our teams.

22:15 And again, this is Anywhere but it's especially important when You know, the only way to find out what people like and y um and and what's valuable is to bring it into the

22:28 External world. Um So For that reason, I think. It's become a superpower of open AI.

22:34 And I'm glad that Joanne thinks I had some part in that, but it it's it really has taken the village. I love this phrase, the resting heart rate of your team. That's such a perfect metaphor of just the pace. Uh being equivalent to resting heart rate. I actually learned that uh at at Instacart when I when I showed up there because we were in the pandemic and it was um Kind of all hands on deck.

22:53 For a while there was this like Yeah, I think there was a company wide stand up. Um because we disbanded all teams where he's trying to keep the site up and for me Yeah, I I had been used to kind of taking my sweet time and just thinking really hard about things and that's important. But I really learned to hustle over there and um uh I think that's come in handy, um at OpenAir.

23:11 Okay, so along these same lines I asked Kevin Weal, your CPO, what to ask you and he said to ask you about uh this principle of is it maximally accelerated? Talk about that. Uh that's funny. Th we have a Slack emoji apparently for this now,'cause I used to say that. Now that now I try to like paraphrase uh

23:29 Sometimes I just Really wanna jump to the Yeah. to the punchline of like okay Why can't we do this now? Or why can't we do it tomorrow?

23:39 Um I think that Yeah, it it's a good way to cut through. A huge number of blockers. uh with the team and just instill especially if you come from a larger company, you know, at some point we started hiring people from from you know larger tech companies. I think they're used to Yeah, let's check check in on this in a week. Or it let's you know

23:57 Um, circle back next quarter to see if we can go on the on on on the plan. And I just Kind of as a thought exercise, oh, is like people asking like, Okay, if like this was the most important thing and you wanted to truly Maximally accelerated, what would you do? That doesn't mean that you go do that, but it's really a good forcing function for understanding what's critical path versus what, you know.

24:16 Can happen later. And I just always felt like Yeah. Execution is

24:22 incredibly important. These ideas they're they're everywhere. Everyone's talking about, you know Me personally, I, you know. You might have seen News on that, you know in in in you know, I I really think that execution is is is one of the most important things in the space and this is a tool.

24:36 So um It's funny that that became a meme. Um it's like a little pink slack emoji that people just put. On um Whatever they try to to force the question. I was going to ask a thing much of us, so it's a little pink. Is there something in there like Mac It's a comic stance emoji that says is this maximally accelerated? And so the kind of the culture there is when someone is working on something, the question the push is is this maximally accelerated? Is there a way we can do this faster? Is there anything we can unblock? Yeah.

25:03 We use that sparingly, right? Because it it has to needs to be appropriate to the context. Um there there's some things where you don't want to accelerate um as as as quickly as possible. Um because you you kind of want Process and We're very, very deliberate on that. Where your process is a tool and one of the areas where we have

25:21 An immense amount of process of safety. Uh because Yeah. A the stakes are already really high. Um, especially with these models, you know, GPT five pushes the frontier in so many different ways. But B, you kind of if you believe in the exponential, which I do and you know most people who work on this stuff do.

25:36 You have to play practice for a time where Yeah. You really, really need the process for sure, sure, sure. And that's why I think it's been really important to separate out, you know, the product development velocity, which has to be super high. From Okay, for things like frontier models there actually needs to be a a a rigorous process where you red team

25:54 You work on the system card. You get external input. Um and then you put things out with with confidence that it's gone through, you know. The right safeguards so Again, it's a nuanced concept, but I found it very, very useful when we needed um and

26:07 For everything product development, you're a debt on arrival, so it's it's important to get stuff out. We gotta open source as memes so that other teams can build on this approach. Absolutely. So Interestingly, with ChatGPT and it's not a surprise, but not only is it the fastest growing

26:23 Most successful consumer product ever. Retention is also incredibly high. People have shared these stats that one month retention is something like ninety percent. Six month retention is something like eighty percent. First of all, are these numbers accurate?

26:38 I'm obviously limited on what exactly I can share, um but it is true that our retention numbers are really exciting and That is actually the thing we we look at. You know, we We don't care at all how much time you spend in the product. Um, you know, in fact our incentive is just to solve your problem. And you know, if you really like the product, you'll subscribe. But you know, there's No incentive to keep you in the product.

26:59 Um for long, but we are obviously really, really happy if you know over the long run, you know, three month period, et cetera. You're still using this thing and For me this was always the elephant. In the room early on. Like, hey, this may be really cool product, but you know, is this really the type of thing that you come back to?

27:15 Yeah. It's been incredible to not just see strong retention numbers, but just see you know in in in improvement in retention over time. Um even as our cohorts become, you know, um less of an early adopter and more, you know, the the average person. So um Yeah. So th like that note is something that I don't think people truly understand how rare this is. Yeah. When a product The cohort of the case.

27:37 of users comes, tries it out, and then retention over time goes down and then it comes back up. People come back to it a few months later. And use it more. And that's it it's called a smiling curve or smile curve, and that's extremely rare. Yeah, yeah, yeah. No, this this some smiling going on. Um not just on the team. And um the yeah. I feel like I have to acknowledge that some of it is is not the product. I think people are actually just getting used to this technology and like

28:01 A really interesting way where I find and this is why the pro needs evolve too. That this idea of delegating to an AI it's not natural to most people. Not like you're going through your life and figure out what can I delegate like. certain sphere of Silicon Valley does that, you know, because they're like a self optimization.

28:16 mode and they're trying to delegate everything they can, but I think for most people in the world. It's actually quite unnatural and you really have to learn okay what What are my goals actually and what could it another intelligence helped me with. And I think that just takes time.

28:28 And people do figure it out once they've had enough time with the product. But then of course there's been tons of things that we've done in the product too. Whether or not it's making the core models better. Well or not it's Yeah, new capabilities like search. And personalization.

28:40 Um and and all that. uh kind of stuff or you know, um just standard growth work too, which we're starting to do. Yeah, th that stuff matters too, of course. So uh you might have you might be answering this question. Already, but let me just ask it directly.

28:55 People may look at this and be like, Okay, they're building this kind of layer on top of this godlike intelligence. Uh huh. Of course it will grow incredibly fast and retention will be incredible. What the heck does what are you guys actually doing that sits on top of the model that makes it grow so fast.

29:12 and retain so much. Is there something that has worked incredibly well? That has moved metrics significantly that you can share. I mean one thing we've learned, um I'll answer that question in a minute, but you know the the one thing we've learned with ChatGPT is that there really is no distinction between the model and the product. Like the model is the product. Um and therefore you need to iterate on it like a product. Uh and by that I mean if there's You obviously

29:33 You typically start by shipping something very open ended. Um, at least if you're open AI on I've blinded. That's kind of a a playbook. Um, but then you really have to look at what are people trying to do. Okay, they're trying to write, they're trying to code, they're trying to Get advice. They're trying to get recommendations.

29:48 You need to systematically improve on those use cases and that Is pretty similar. Obviously the methodology is a bit different, but the discovery is is is the same. You gotta talk to people, you gotta do data science. And you gotta try stuff and and get feedback.

30:03 Um, so that's like one chunk of work that we've been very consciously doing. Um is improving the model on the use cases people care about. And There's also such a thing as vibes as a I'm sure you're

30:15 You know, and that's one of the things that I'm excited about in GPT five is that the the vibes are really good. So that too is, you know, we have a a model behavior team and they really focus on, you know, what is the personality of this model and how, you know, how does it speak and talk. So does that kind of work. I would say that's maybe yeah. A third of the

30:32 Yeah. Retention. uh improvements that we see or so just roughly. And then I think another third is is is what I would call So product research capabilities. Um they're research driven for sure. They have a research component, but

30:44 They really new product. features or capabilities and like search is one example of that where you know If you Remember in the olden days. AK like you know, maybe twenty months ago or something.

30:54 You would talk to Chat GPT and be like, you know, how's my knowledge cut off? Yeah, or I can't answer that because that happened too recently or something like that. And Yeah, that is a type of capability that has been incredibly retentive. Um And um For good reason. It just allows you to do it. Do more with the product.

31:10 Personalization. Like this idea of Advanced memory. Where if things can really get to know you over time is another example. of a capability like that. You know, I think that's another good chunk.

31:20 And then you know, the third stuff is the stuff you would do in any product and those things exist too, you know, um like Not having to log in was a huge hit. Um because it removed a ton of the friction. And um I think we all we had this intuition from the beginning, but we n never got to it because we didn't have enough GPU or

31:36 Yeah. To really really really go do that. So, you know, there's the like to have traditional product work too. So I often think about it sort of as roughly a third, a third, a third, but really, you know, we're still learning and um we're planning to evolve the product a ton, which is Why I'm sure there's gonna be new levers.

31:51 Uh you mentioned something that I wanna come back to real quick. You said that the It was something like ten days. From hackathon to Sam tweeting about Chat GPT being live. The you know the hackathon happened much earlier and we were prototyping for a long time, but at some point we basically ran out of patients.

32:06 On you know. Uh and trying to, you know, build something more bespoke. And again, that was mostly because people always wanted to do all this other stuff, uh, whenever we tested it. So it was ten days from from when we decided we were gonna ship to when we shipped. Um and um Yeah.

32:22 The the research we've been testing for a long time, it was kind of an evolution of what we'd called instruction following, uh, which was the idea that, you know, instead of just completing the sentence. These models could actually follow your instructions. So if you said summarized this, it would actually do so. And The research had evolved from that into a chat format where we could do it multi-turn. So then that research took way longer than ten days and that kind of baking in the background.

32:44 But the you know the privatization of this thing um was very, very fast. Um and you know, m lots of things didn't make it in. Like I remember we didn't have history. Which of course was like the you know, first user feedback we got. The model had a bunch of You know. Shortcomings.

32:58 And it was so cool to be able to iterate on the model. The thing I just talked about, like treating them all as a product. was not a thing before chat GPT because we would chip in more like hardware where Yeah, there we there'd be a a release like GPT three and then we would start working on G P P four and these weird

33:12 Giant. big spend R and D projects that would take a really long time and you kind of The spec was whatever the spec was, and then you'd have to wait another year. And chat GPT really broke that down because we were able to make make uh iterative improvements to it, just like software.

33:26 And really My dream is that it would be amazing if we could just Ship daily or even Hourly. Like in software land, because you could just fix stuff, et cetera.

33:34 But there's of course all kinds of challenges in how you do that while you know keeping the personality intact while like not regressing other capabilities. So it's an open open research field to get there. That's such a good example of is it maximally accelerated? Okay, we're gonna ship that too. Holy moly. We've been talking about chat GPT. Clearly it's a kind of a Chat interface. Everyone's always wondering. Is chat the future of all of this stuff?

33:56 Interestingly, Kevin Weel had made this really profound point that has always stuck with me when he was on the podcast that Chat is a actually a genius interface for s building on a superintelligence. Because it's how we interact with Humans of all variety of intelligence. It scales from someone at the lower end to the to a super super smart person.

34:17 And so it's really valuable as a way to kind of Scale the spectrum. Uh Maybe just talk about that and just is chat the long term interface for Chat GPT. I guess it's called chat GPT. I feel like we should either drop the chat or drop the GP GPT at some point because it is a mouthful. Uh we're stuck with the name. Um but yeah, no matter what we do with that, yeah, it it uh

34:37 Um The product blah blah. I agree that there's something profound about Um natural language. It just really is.

34:46 the most natural war form of communicating. Um to humans and therefore It feels important that you should Yeah. Communicating.

34:54 With Your software in natural language. I think that's different from chat though. Yeah. I think chat was the simplest way. To put something to you know to ship

35:04 At the time. I'm baffled by how much it took off. Um as as a concept. I'm even more baffled by how many people have copied the paradigm rather than, you know. Trying out a different way of interacting with the I'm still hoping that will happen.

35:17 So I think natural language is here to stay, but this idea that it has to be a turn by turn chat interaction, I think, um is really limiting. Um and This is one of the reasons I don't love the super system analogy, even though we are Yeah.

35:31 You think that way. Then you're kinda Feel like you're talking to a person. But you know, and GPT five is amazing at at um making great front end applications. So I I don't see a reason why

35:42 You wouldn't have you know, AI is that you know can can render their own UI in some way. And you obviously want to make that predictable and feel good. But it feels limiting to me to think of The end all be all interface. as a chatbot, it it actually kinda feels dystopian almost, where like I don't want to use all my software through the proxy of some

35:58 interface. Like I love being in Figma. I love being in, you know Uh Google Docs. Those are all great products to me and they're not chatbots. So Um Yes on natural language, but no on chat is is where I would describe my my point of view. Um and I'm just hoping generally that we see more

36:15 sort of consumer innovation on how people interact with AI. Yeah, there's so many Possibilities. Yeah, you just gotta try stuff. That's why chat stuck is like you know, we just did it and peop peop people liked it. So I'm I'm hoping that um we s we see over there and we'll we'll try to do our part.

36:31 So you mentioned that you kind of like got stuck with this name chat GPT. Uh Maybe this is part of the answer, but I'm curious just are there any accidental decisions you guys made early on. that have stuck in have essentially become

36:44 History changing. There there there's so many and it's it's funny because you have like no time to think about them and then they end up being super consequential, you know, the day it was one. Yeah, we went Chat with G P P three point five to chat GPT the night before. Slightly better, but still really bad. What was it called before? It's gonna be chat with GBT three point five.

37:01 We we really didn't think it was gonna be a successful product. Like we were trying to actually be as as nerdy as we could about it because that's really what it was. It was like, you know, a research demo, not not a product. So We didn't think that was bad. But um Yeah, I I think that In the original release, you know, making it free.

37:17 Was a big deal. I I don't think we appreciate that because The uh GPT three point five model was in our API for, you know, at least six months. Prior to that.

37:26 I think anyone could have built. Something like this. Might not have been quite as good on the modeling side, but I think it would have taken off. So making it free and putting a nice UI on it. Very consequential in the way that you take for granted now.

37:38 And this is why I think that A distribution and B. the you know the interface. Or continued. continuously important even in in twenty twenty five.

37:47 The paid. Business which now is it's it's it's a it's a Giant business. Um both in you know the consumer space and in the enterprise space. The birth of that was just a turn away demand.

37:59 originally. Like it was not like you know, we brainstormed Oh, what is the best monetization model for AI it was really What is what monetization model has or what what mechanism would allow us to turn away people who are like you know, less serious than the people who are really trying to use it. And subscriptions just happen to have that property in it, yeah.

38:16 grew into a large business. Yeah, I think. Shipping really kind of funky capabilities. Before they were polished is another thing where Yeah, that feels like a tactical decision, but it became a playbook because we would learn so much. Like remember when we shipped code interpreter.

38:31 We learn so much after Uh we shipped it. Yeah, now it's known as I think data analysis and chat GPT or something like that. Just because we actually got real world use cases back that we could then optimize. So

38:43 I think there's been like a lot of decisions over over time that um Prove pretty consequential, but Yeah, we made them very, very quickly as as as we have to. So Um the tw the twenty dollars a month feels like an important part of this. Feels like everybody's just doing that now and that one actually I remember I had this like kind of Panic attack because we we really needed to launch subscriptions because at the time we we're we were taking the product down every time.

39:05 Um it was like I don't know if you remember, we had this like Fail Whale, there's like a little P E three generated poem. On it. So they had to get this out and uh they're calling up. Um Someone I greatly respect who's like, you know, incredible at pricing. Um and you know it's like what should I do? And like we talked a bunch

39:21 And I just ran out of time to to Incorporate most of that feedback. So what I did do is ship a Google form to Discord with like I think the four questions you're supposed to ask on how to price something. Yeah, exactly. Yeah, it literally had those four questions, and I remember distinctly

39:37 A you're not gonna price back. Um, and that's kind of how we got to twenty dollars. But B Uh the next morning there was like a press article. And like you won't believe the like ch four genius questions the Chat GT team

39:49 Asked to price their it was like If only you knew. So there's like something about building in this extreme public where people interpret so much more intentionality into what you're doing than you know might have actually existed at the time. But we got with the twenty

40:02 We're debating, you know Something slightly higher at the time. I often wonder what would have happened because so many other companies ended up to copying the twenty dollar price points and like do we like erase a bunch of market cap by pricing it this way. But

40:15 Ultimately I don't care because Like the more accessible we can make this stuff. The better. Yeah, I think. This is the price point that in Western countries Has been

40:24 um reasonable to a lot of people in terms of the value that they get back. And um Most importantly, we're able to push things down to the free tier um semi regularly, and we always do that when we can, um including with you to butt. So the survey just to give you the official name, the Van Weston Drop.

40:39 Survey. Uh is how you guys ended up pricing ChatGPT. It was the top Google result. This was before ChatGPT had realtor information. Otherwise it could have maybe price itself. But uh it was Discord plus Google Forum plus A blog post on that methodology that um got us there. So That is incredible. What a fun story. This is the survey that Revolvo at Superhuman com popularized in his first round articles. Yeah, yeah, yeah. That's right. That's right. Uh yeah. Definitely don't bring me on here as a pricing expert. I think you you you have got better people for that. Whether it was right or wrong, it is now the fastest growing

41:11 insane revenue generating business in the world. So Uh it wouldn't feel too bad. No, it worked out. Yeah. It worked out. Uh and by the way, I'm on the two hundred uh a month here. So there's clear uh room. Thank you. Thank you. Yeah, that the story of that one is is interesting too, because you know originally it

41:28 The purpose of the plus plan was to be able to ship First uptime and then be able to ship capabilities that we couldn't scale to everyone and at some point You got so many people in the plus tier that have just lost That Property.

41:40 Um, so the re the main reason we came up with the two hundred dollar tier is just we had so much incredible research that's actually really, really powerful. Um like you know, O three pro or tom you know, tomorrow GPT five pro. Um and just having a vehicle of shipping that to people who really, really care is exciting, even though it kinda violates The standard. Way a SaaS page should look.

41:59 Um it's like a little jarring too. See the see the Jennics job. So um um thank you for being a subscriber on that and thank you everyone else who's watching you subscribe to any tier. Um it's great. I'm just gonna throw a a fishing line into this pond of are there any other stories like this. You should have this incredible story of

42:17 Chat with GPT three point five being the original name. how you came up with pricing. Is there anything else? Enterprise is an interesting one too. Incredible adoption in the enterprise. And it's Sort of objectively crazy to try to take on building a developer business and a consumer business and a develop and and an enterprise business and and and all at once.

42:40 But Yeah, the story there is in like month one. Or or two. I it was like very clear that most of the usage was like kind of worky usage. Actually much more than today, where you've got so many like kind of consumers. Uh on the product and yeah. It's kinda

42:55 Sort of. transcended into pop culture, but at the time it was like you know, writing, coding Analysis, that kind of stuff. And uh we were pretty quickly in l you know, organically in like ninety eight percent of Fortune five hundred companies. In a way that I had seen maybe a Dropbox back when I you know that was my

43:10 Two jobs ago where we kind of had a similar story, and since then there's been More P L G companies. But the real reason we did enterprise, remember we were debating should we do Enterprise or should we launch an iOS app because that's how small the team was. Yeah, the reason they did yeah, did it is we were starting to get banned on companies because they all you know, felt, you know, rightfully or wrongly that you know the the

43:29 privacy and deployment story, et cetera, wasn't there. So it's just like man, we have to do something. We're gonna miss out on a generational opportunity to build a a a a a work product. And you know We've literally defined AGI as you know. outperforming most humans at economically valuable work or

43:44 I probably would do that, but you know I think um Think that's the way we put it. And um Um so it I feel like we had to be present there and it was a fairly Yeah. Quick decision at the time, but it's grown into an immense

43:56 uh business. We just hit five million um business subscribers. Up from three, I think, uh A month or two ago. So It is kind of the spin-off. that it's taking a life of its own that I'm really, really excited about. Um

44:09 Um for perhaps reason. That is a lot to be handling. Uh the platform, essentially the API. the consumer product, the fastest growing, most successful product in history. And also the B2B side, which is uh clearly a massive business. Uh do you have any kind of heuristics for how to make these trade offs, do all this at once and stay sane and be successful.

44:30 Uh that's a good question. And yeah, um first off, I don't run the developer stuff anymore. We found some way more competent. Uh Olivier to do that. Um and he's amazing. So I still look after the, you know Various forms of of of of chat, but yeah, they luckily don't have to make Make that trade off. OpenAI does and I I can get into that too, but um it keeps me a little bit more sane.

44:51 I will say that There are you kinda have to practice in two different ways when you're when you're building on this AI stuff. One is sort of working backwards from the model capabilities. And that is much more art than science, where I think you really need to look at what tech do we have. Available and what is like the most

45:08 awesome way to product uh producttize it. And If you applied to some sort of PM framework to that, I think you would If you have tech that's, you know. Um, for example, GPT five is

45:21 is really, really good at front end coding now. Thank you. I think we that means you've got to reprioritise and you gotta like actually bring that capability to life. Maybe that's, you know, uh making making Chat GPT better at at vibe coding and rendering you applications, maybe that's More like you know.

45:37 Leveraging the taste of the model to make the the UI more expressive. There's like a number of things we could do, right? But you kinda have to replan and reprioritize and that, you know is more important than any particular audience segmentation. It's really just looking at, you know, what Is the magic thing we have and how do you make it shine? Voice is a similar thing. It wasn't like

45:54 our customers need voice if they're begging for it or something like that. It's like Well. We figure out a way how to Yeah, to make these things Anything in and anything out.

46:02 What is like a creative, awesome way to productize that and then we can see what people do. So I think that's one chunk of it. But then the other chunk of it. Really is More like classic product management where you need to listen to customers and then when your customers are really different, that can be confusing.

46:18 Because Uh yeah, Chat GT is a very general purpose product. We see When you look at end users, there's actually an immense amount of overlap in terms of what they want. Like primitives like projects or um

46:30 Yeah, history s uh search or Um sharing Um c collaboration. Like all all those kind of things, they are actually very, very present whether or not you're talking to people at work or you're talking to people at home and school. The slightly different mechanics sometimes.

46:45 Um but They're they're largely similar investments that I think we can get a lot of mileage out of. And then there's enterprise specific work that we just have to do. Like you gotta do HIPAA, you gotta do Sacchu, you gotta do all those things if you wanna be a serious player. And those are just non negotiable. So

47:00 It's complex, as you correctly identified, um but it's kind of the the curse of working on a very open ended and powerful Um technology. Uh one analogy that that um someone at OpenAI I really respect sometimes is like we're kinda like Disney, where Disney has this like one

47:18 Kind of Creative IP. Um which is like their their their content and they have cruises and they have um uh, you know, uh theme parks and they have comics and they have all these different things. I think we have amazing models, but there's all these different ways that you could productize them and we kinda just

47:34 maximize the impact in um in all these different ways. As you were talking, I was thinking about how usually uh horizontal platforms that are just so general and can do so much take a long time to take off because people don't know what to do with them. They're not amazing at anything.

47:49 And this is an amazing counterexample where it took off immediately and everyone figured it out and then over time they figured out more and more. But I I think the reason why is because it just went live Talk about another consequential decision actually. You know, we were debating wait list, no wait list because we just really knew we couldn't scale the engineering systems and you know, the fact that there was No wait list, which

48:09 No open AI release had worked like that before. Yeah. And it'd be consequential because like you were able to watch what everyone else was doing. Live. So I think when you launch these things all at once for everyone, there really is a special moment. Where

48:22 You can see what other people are doing and learn from that. And a lot of that is actually out of product. There's these crazy TikTok posts that go viral and they have like two thousand use cases in the comments and I go through those in detail because it's not like I knew about those use cases either. Like they're they're very, very emergent.

48:38 And I just go through the comments and you know process because there's so much to learn and For that reason I think we get to escape the empty box problem a little bit. Because you know, the so much learning is happening out of product. Um as people are watching each other Either in I R L or uh or online.

48:55 That is so interesting because you you think about air table, you think about notion, all these companies they took like years to just build and craft and think and go deep on what it could be. It's like the the Caper Air table, which like, you know, they they had to do templates, they had to do um like all these kind of things of taking the horizontal product and making it Like use case. Driven? I mean compared to the like the instapot.

49:16 Um, which you know, there's recipes being shared on everywhere online. Like there's a kind of the whole ecosystem around it. I think We were really lucky with Chat GPT that that happened where There's just users sharing use cases with other users. Everywhere. Um and and therefore I I think you know

49:31 We we we we we kinda Got very lucky by by Yeah. Jumping jumping ahead. Um on on that journey. And it feels like a cord there is a Sam had all big following and everyone would pay attention to something he launched. So

49:45 That's a really interesting new strategy for launching horizontal product with a huge distribution channel just Launch it and see what see what comes up. Yeah, and of course I'm I'm actually really excited to take some of that into the product. I think there's there's the we shouldn't you you know. rest on the fact that there's so much out of product discovery happening. Like I actually think for the average Consumer it would be amazing if the product did a little bit more work on Really exposing to you what is possible.

50:06 I I still feel like Chat GPT feels a little bit like MS DOS. Uh we haven't built Windows yet. It will be obvious once we do. But yeah, there there there's something that feels a little bit like Like, imagine MS Dos like gone viral and you were just trying to like hack

50:19 Like Blue conversation starters onto it. That might have missed sort of the big picture in terms of how to really communicate. Affordances and value to people. Yeah. So I I think there's actually a ton more product work to do in addition to

50:30 Um yeah, just seeing use cases spread. Are you able to share just what you think that might look like, those Windows version of Chat GPT. I'll let you know when we figure it out. Um we're hiring. Um I think there's so many interesting product problems here. Okay, got it. Uh, by the way, I also love that TikTok was like their feedback.

50:47 Uh channel. Those comment threads are they're they're just so wild. And and and also the love that people have for it. Like the excitement with which they're sharing their product. I I I I I kinda feel like it's it it's it's special that people are so excited about to share what they're doing with your product. And um

51:04 I don't take that for granted either. This episode is brought to you by Post Hog, the product platform your engineers actually want to use. PostHog has all the tools that founders, developers, and product teams need, like product analytics, web analytics, session replays, heat maps, experimentation, surveys, LLM observability, air tracking, and more. Everything Posthoc offers comes with a generous free tier that resets every month. More than 90% of customers use Post Hoc for free.

51:32 You are gonna love working with a team this transparent and technical. You'll see engineers landing pull requests for your issues, and their support team provides code level assistance when things get tricky. Posthoc lets you have all your data in one place. Beyond analytics events, their data warehouse enables you to sync data from your Postgres database, Stripe, HubSpot, S3, and many more sources. Finally, their new AI product analyst, MaxAI, helps you get further faster. Get help building complex queries and setting up your account with an expert who's always standing by. Sign up today for free.

52:04 at posthog.com slash Lenny And make sure to tell them Lenny sent you. That's post HOG.com slash Lenny. How do you find emergent use cases these days? I imagine the volume is very high. Do you have kind of a trick for figuring out, Oh, here's a new thing we should really think about.

52:22 before I built the pride team, I actually built the data science team um because I it I was getting frustrated. I was talking to as many users as I could. In my calendar. Yeah, but weeks after Chat GPT was just fifteen minute user interview the whole week through.

52:36 And it was usually I stop doing interviews when I like can predict what the next person's gonna say. That's how I know I've talked enough users, but it just wasn't happening. Like I just kept getting new stuff. So Data is one way out. Where I think you you know, we we have conversation classifiers that without, you know Us having to look at the conversations. allow us to kind of figure out what are people talking about, what use cases are taking off, et cetera. And I think that's very, very helpful.

53:00 The quality of stuff is important for empathy. Even though you're never gonna get a wrap on like all the use cases people have, um, I still spend a huge amount of my time d doing that. And then Yeah. Things like those TikToks, um collections of threads. I think they're really, really useful and uh

53:17 Um it's just fun to watch people talk. to each other about the various use cases that they have. Is there kind of an a new emergent use case that you're excited about. Or is there like a really unusual use of chat GPT that you think about? That would be fun to share?

53:30 I mentioned this earlier, but I had always conceptualized Chat G B T as a a worky product, whether or not you're at home or you at at work. Like you I feel like yeah. Helping getting help with your tax is very similar to, you know, um the types of things you do at work or you know, planning a trip is actually very similar to Yeah, planning an event for work.

53:47 So I always felt like okay. This thing is gonna kinda be a productivity tool. And I think Something has happened over the last. Yeah, a few months.

53:56 Where That has begun to change and I really do think The fact that you have Consumers turn into this thing for day to day advice. Helping them like have better relationships, like the seeing like you know

54:10 people talk about how this thing like you know saved their marriage is like really exciting to me because like they yeah Process it to process their own emotions. Get feedback on their communication style. Body it.

54:21 talk to about like really difficult things. And that comes with a ton of responsibility and work that we have to do to make those things like life advice great. But it also is really, really important to me because you can't run a w away from those use cases. You have to run towards them and make them awesome. And um that's part of what we're trying to do. So That emergent behavior is really, really cool.

54:41 Yeah. More broadly. I am so excited about education I'm so excited about. Um health

54:49 Like I I think it would really be a waste if we didn't take the opportunity Oh. using chat G PT to really, really help people. And I think we've just begun to scratch the surface, um, on on that. So um there's many aspirational use cases that I want to make happen.

55:04 Along the lines, an interesting use case I've recently had. I feel like it's gonna be really helpful for uh couples that are disagreeing about something when they need like a third opinion. I just had this recently where my wife's like, you can't heat a whole thing that you're gonna only eat part of in the microwave but then put it back in the fridge. It's like what's the problem? I'll heat it up, I'll put it back in the fridge. And she's like, No, that's really dangerous. I'm like, Let's ask Chat G P T and the fact that she so trusts chat GPT now and relies on it throughout the day.

55:32 It's such a valuable third independent party that we can go to. Yeah, yeah, totally. And and and you know, the a lot of those micro interactions talk about like Interesting product work. Right. Those are micro interactions are important, right. Did it like definitively weigh in or did it help you guys think through you know, that that that disagreement and you know, um solve it on your own. I think those details actually matter a lot.

55:52 And it's where we're spending a bunch of time. Along those lines, there was this whole launch of the very sycophantic version of Chad GPT where it was just You are the best person in the world, everything you tell me is Amazingly correct.

56:06 Uh Are you able to tell us just what happened there? Yeah, we have you know, we we have all kinds of collateral um online because we really felt like we should overcommunicate on Hell.

56:16 We discovered it, what we did about it, et cetera. So I encourage people to check that out. Um we d have a whole retro. Um U on on that model release. But basically what happened is that we pushed out an update that Yeah. more likely to

56:30 Yeah. Tell you things that sound good in the moment. And um yeah, like you're you're totally right, you know, you you yeah. Should break up with your boyfriend or something like that. And yeah.

56:39 That's just really dangerous. And it's and we we took it more seriously than you even might expect because again Uh Current and technology levels. You can kind of laugh about it, maybe, is like, ah, this thing's always complimenting me, I thought it was just me. I saw those comments online.

56:55 But you know, it actually is is really important to make sure that um these models are optimized for the right things. And we have an immense I think luxury to have a mission that affords us to Really help people. A business model that

57:08 Does not incentivized Yeah, maximizing engagement. Um um you know, um or time spent in in the product. Right. So it's really important to us that you

57:19 feel like this product is helping you with your goals, whether or not that's your current goals or even your long term goals. And oftentimes, you know. Uh being extremely complimentary with the user isn't actually. In in service of that. So

57:32 We instilled new measurement techniques, like you know, whenever we put these models in contact with reality and we, you know learn about a problem, we actually go back and make sure we have good metrics for this stuff. So you know, we measure circumfits now. With every release to make sure we don't regress and can actually improve on that metric. Um GPD five is an improvement, which is really exciting for me.

57:51 But we have more work from there. Um And more broadly, it cause us to articulate our point of view actually. Spent a bunch of time on a blog post that we just published on Monday on what we're optimizing Chat GPT for and

58:04 It really is for your Yeah, to to to help you thrive, um um and achieve your goals, not to you keep you in the product. And um so there was a bunch of good outcomes from from from that incident. It's a good example of how contact with reality is not just important. For the use cases.

58:20 But also for learning what to avoid, because you would have never discovered this issue purely in a lab unless you actually heard from it. I am excited to read that blog post. Then I was gonna ask you this just like how you're gonna get feedback on it. Yeah. And yeah, I guess is there anything more they're just like how you 'Cause this tension is so difficult. Like

58:35 You know, helping people feel supported, but not just letting them believe everything they want to believe. Is there anything more you can share there or just try to find that middle ground? Incentives are important. Charlie Munger, maybe? Um I think that's where it came from, right? Yeah, I think that's very, very important. So I would take a good look at, you know

58:56 Our mission, our business model, the type of product we're trying to build. And you know, I I I really think that Yeah. I think.

59:07 And Vast majority of cases it makes you You leave it feeling better, not worse. And you've like you know, feeling like you're achieving something you're trying to trying to do. Yeah, so I think that those incentives really matter because it helps you reason about okay when there isn't

59:21 Behavior. In the wild. That's not good. Was that a bug? Or was that by design? Yeah, and w with Sikovitsi I can very much say that.

59:29 To us that's a bug. And then on You know the The forward looking work. There's so many yeah.

59:37 Kind of challenging. Scenario to get right. And you could easily run away. From from from from these use cases. Like you know the like yeah. You and your wife go into this thing.

59:49 um for you know input on a relationship um um uh question or like a dispute. You could very easily run away if you were totally risk avoidant and say Sorry, I can't help you with that. I think that's what most tech companies do when they hit a certain scale.

1:00:04 They run away from these use cases and I think it's a lost opportunity to help people. So We wanna run towards these use cases by making the model behavior really, really great. Um that can mean connecting you with external resources when you're struggling. That can mean

1:00:17 Not directly answering your question. But it's said giving you a Help framework. You know, in the case of like should I break up with my boyfriend? Chat should probably not answer that question for you, but it should help you think through that question.

1:00:29 In the way that a thoughtful companion Would So I think it's really important to do the work. Because I think the upside is immense. That is a really profound point you're making there.

1:00:39 That If most companies if they're If their users want to ask them something. risky like getting medical advice or should I break up with my partner or Or what should I do with this big problem I have?

1:00:50 I feel like we would have immense regret if you had a model that was state of the art on Health bench. You know a um Um GPT five is the state of the heart and you know. A bunch of these medical benchmarks, right?

1:01:01 And you didn't use that to help people. Like you just disable that use case because you wanted to like avoid all possible downside. I I think the duty is to make it awesome. Um and to do the work. Talk to experts.

1:01:12 Figure out how good it really is. Where it breaks down. Communicate that. And um you know, I I think this this technology is too important and has too much potential positive impact on people. to to run away from from um these high stakes use cases.

1:01:27 I fast forward to today. It's saving lives regularly. It's Probably saving relationships regularly. Such a consequential decision, which I imagine was made early on. Yeah, we're we're just at the beginning of of watching how this people th this this this stuff can transform people. Um it's incredibly democratizing.

1:01:44 If you compare, you know You roll out of this with the roll out of the personal computer. Right. Yeah, computers were like so scarce when they first came out. And this stuff is ubiquitous in a way where I th uh you you have access to A second opinion on

1:01:57 On medical stuff you have access to, you know, um Um a a relationship buddy, you have access to a personal tutor. On literally any topic that uh makes you curious. Uh it's really, really special that that that we get to do that. So um

1:02:13 Um unique point it is in history. Let me zoom out a bit. And talk about open AI. And

1:02:19 Just product in general. So you've worked at traditional, let's say traditional product companies, Dropbox, Instacart. Now you're at open AI. What's what's maybe the most counterintuitive lesson you've learned? And by building products from your time at OpenAI. Each time

1:02:34 Different. Maximally different job. Whenever I made a job change. Yeah, so Yeah, after Dropbox I was like craving a

1:02:42 real world product because it was just so different than working on Sass, et cetera. Uh and after Instacart I was was interesting, um and had, you know, this kind of like

1:02:55 sort of invoke the nerd in me. And you know. So I've always looked for things that are really different. And then Once I shut up. At these places I try to understand what makes that Place successful? Like what is truly the thing?

1:03:06 that they cracked and how he can lean in that into that even more. And I think I spent a lot of time thinking about this with open AI. Um especially after chat GPT. Before that, you know, it was kind of a mood point'cause we didn't really have

1:03:20 much revenue or products or anything that you know. Like that. And There's a f you know, a few things um that that that that come to mind that have driven Made decisions. Um

1:03:33 One is the empiricism, we talked about that a bit. The fact that you can only find out. By shipping. Um which is why I've maximum leaned into that and that's you know Huge part of why.

1:03:43 Uh we ship so much. Um one of them is that Yeah. Amazing ideas come from Anywhere.

1:03:50 Um there were thing about running a research lab is you really don't tell people what to research. Um that's not what you do. And we inherited that culture even as we become research and product company. So just letting people do things who have amazing ideas. Rather than sort of being the the gatekeeper or prioritizer of everything or something like that.

1:04:07 Um is been proven, you know, immensely. valuable to us and that's where much of the innovation comes from is Empowered, smart people on any function, really. Um so that was a good inheritance from

1:04:19 What I think made OpenAI successful and makes us successful. The interdisciplinariness of really making sure that you put research and engineering and design And probably together. Rather than treating them as silos.

1:04:33 Think that's the thing that has made us successful and that you see Come through in every product we ship. For shipping a feature and it doesn't get two X better as the model gets two X smarter. Probably not a feature we should be shipping. Um Yeah, not always true. You know, SOC two doesn't get better with uh, you know, uh threader bottles.

1:04:48 But you know, I think for many of the core capabilities, that's a good litmus test. So I've always found you really have to lean into why is this place successful and then maximally accelerate that, uh, so to speak, because um it's it's what allows you to Turn something that feels like an accident into something that is a repeatable uh label. So you talked about this kind of collaboration between researchers and product people.

1:05:11 And you've been at the beginning of chat GPT from day one to today, from zero to seven hundred million weekly active users. Not just registered users, weekly active users. How have you approached building out that team? Over time. What are the other inheritance of um Being in a research lab, is that you take recruiting really seriously.

1:05:31 That's something that You know AI labs now. Every person matters. But many tech companies that go through hypergrowth and they kind of lose their identity, they lose their you know, their talent bars, they they they just kinda have chaos. Um

1:05:46 So we've always had this tendency to run Relatively lean. So it is a small team. That is running chat GPT. Um I I take inspiration from WhatsApp. Where like you know, it was a very small team running a very global scale product.

1:06:01 Um and then the more importantly, I yeah, I you know. You have to treat Hiring A little bit more like executive recruiting and less like just pure pipeline recruiting where you really need to understand what is the gap you're trying to fill on each team. What is the specific skill set and how do you fill it?

1:06:16 Um to give you an example Yeah. I'm a product person at heart, but sometimes a team doesn't need a product person because like there's already someone doing that role. Like like you know, in many cases we have a really talented engineering leader who has amazing product sense, or we have a researcher. who has private ideas and then and My mind they can play that role.

1:06:34 And maybe we have something else missing. Um Instead, like maybe we need like a little bit more front end. Um or something like that. In other cases Uh maybe what you're missing is

1:06:44 incredible data scientists. So I really like to go through every single team. And figure out What is the skill sets that that team needs and how do you put it together from principles? Right than just assuming, hey, we're gonna do like Yeah.

1:06:57 A bunch of pipeline recruiting for all these different roles and then You know. People will find a team later. So so I think that's always felt really important to me. Um and it's the way that you keep your team really small. Yeah, super high throughput.

1:07:09 Also allows you to hire people who I think Ke Keith Rebois calls this like like barrels, I think. Um bear barrels and amis ammunition where he thinks I I think I think this comes from him, but I um The idea being that sort of The throughput of your depends on how many barrels you have, um, which is like people who can make stuff happen.

1:07:26 Yeah, I think you can hire um and then you can add am ammunition around them, um, which is the people helping those people. And I you know, I I think that's been really true for our recruiting too, where we try to maximize sort of the number of empowered people who can chip. Because that's how you have a small team and still get the ton done. So th those a couple of things. Um And uh

1:07:46 I spent a lot of time on like vibes too. with like each team because I think one of those things that is challenging when you Try to do research and product together is that the cultures Are different. People have different backgrounds.

1:07:59 And um I think to make that go super well. You need to spend time team building and making sure that people Have a huge amount of trust for each other's skill sets. um feel like they can think across their boundaries. Um like you know.

1:08:12 Um, I really believe that product is Everyone's job. For example, and and and for that reason the recruiting sort of doesn't stop when you the people are on the door. It actually starts because you have to, you know, start Making the teams awesome. Is there something you do with team building that would be fun to share, just like some you do to

1:08:28 Creative. I just love whiteboarding with teams. Like I just like like love getting into a Generative mindset. It breaks down everything. So that's the that's the thing that I I I try it's not particularly creative, but I find it to be Um a universal tool where the minute you can

1:08:42 Get people to Stop thinking about Yeah. What's my job versus the other person's job and more like, you know, we're all in a room like trying to crack something together, that is incredible. You mentioned this idea of

1:08:51 First principles. This came up actually when I talk to a lot of people about you. Is this something you're really big on? A lot of people talk about first principles. Most people are like, I don't really understand like or they think they're amazing at thinking from first principles. Is there something you can share of just what it actually looks like? to think from first principle is maybe an example that comes to mind where you really

1:09:12 Went to first principles and Came up with something unexpected. Yeah, this is not something I'd ever say about myself. It's nice that someone else would say it, but um you know. It's a mysterious thing. I yeah, I think you just really gotta

1:09:27 Get to ground truth on what you're really trying to solve. Like for example, w when as I mentioned with the recruiting thing. I'm not dogmatic that you have to have a product manager and an engineering manager and a designer or whatever. We're just trying to make an awesome team that can chip. So in that case, first principles means just really understanding

1:09:45 what we actually need and what we're missing rather than applying a previously um learn process or behavior. So yeah. I think that's a good example. Another good example of I think being first principles in this environment is is is you know

1:09:59 Does this feature need to be polished? Yeah, we get a lot of crap for the for for for the model choiser and I own it. Um I've tried to say that every ever to everyone who will listen. Um you know, for those who don't know model choosers is like giant drop down in the product that is like literally the anti pattern of any good product traditionally. But

1:10:17 You know, if you are actually Reason from scratch. Like is it better? Two wait until you got a product or to ship out something raw, even if it makes less sense and start learning and getting into people's hands. Um

1:10:32 I think a Company with a lot of process or a lot of just Yeah, learned behaviors. We'll make one call which is no we have like a quality bar when we ship and that's what we do. If your first principle's about it, I think

1:10:44 You're like You know what? We should chip. It's embarrassing, but that's strictly less bad than, you know, um not getting the feedback you wanted. So I think just approaching each scenario from Yeah. From scratch.

1:10:57 Is So important in this space. Because there is no analogy for what we're building. Like there's just you can't copy an existing thing. There is no Yeah.

1:11:06 Are we like an Instagram or are we like you know a Google or like a Like a you know productivity tool or something like that. I don't know. But you can learn from everywhere, but you have to do it from from from scratch. And I think that's why that trait um tends to Make someone effective at open AI and that's something we test for in our interviews too.

1:11:23 Theme keep coming up and I think it's just important to highlight something that you keep coming back to, which is this trade off of speed and polish and how in this space Speed is more important not just to stay ahead But to learn what the hell people actually want to do with this thing.

1:11:39 Is there anything more that you think people just may be missing about why they need to move So fast in the space of AI. Yeah, I mean the the boring answer would be Oh it's competitive and everyone's an AI and they're trying to Yeah, I compete each other. Yeah.

1:11:53 May be true, but that's not the reason that I believe this. I the the the reason really is that You're gonna Be polishing the wrong things. In the space. You absolutely should polish.

1:12:04 Yeah. Um Things like the model output, et cetera, but you won't know what to polish until after you ship. And I think that is Uniquely true. And

1:12:13 an environment where the properties of your product are emergent. Yeah, not knowable. Um In advance. Uh and I think but many people get that wrong because like the best product people tend to be crafts people. Um

1:12:23 And they have a traditional definition of craft. I also think it would be easy to, you know. Use all what I just said as an excuse not to eventually build a great product. So often tell By James, that shipping is just kinda one point on the journey towards awesomeness? And you should put pick that point.

1:12:41 Uh intentionally. Where it doesn't have to be the end. Um of of your iteration at all? It can be the beginning, but you better follow through. So we've been doing a bunch of work, especially over the last quarter of like really cleaning up the UI of Chat GPT.

1:12:55 Really excited to do the same for the sort of the response Layouts and formats next. Simply because once you know what people are doing, there's no excuse to not polish your product. Um it's just really In a world where you don't know yet, you might get very distracted. So It's situational. Again, you kind of have to be first principles about it, but I do think

1:13:14 Using velocity especially early on as a tool. Yep, actually th this has been said about consumer social, for example. This is it's not the first phase where people have said, Hey, you just gotta try ten things because you're probably gonna be wrong. So I I don't think this is Yeah, never existed before as a dynamic either. But I do think with AI, um it's it's it's important to internalize. And there's also an element of the models are getting or changing constantly and so you may not even realize what they're capable of, I imagine.

1:13:38 Totally. The models are changing and um Yeah, the the best way to improve them, whether or not you're a lab or actually just someone who's doing context engineering or or uh you know um Fine tuning a model, maybe. You need failure cases.

1:13:52 Real failure cases. T. Make these things better. The benchmarks are increasingly saturated. So really you need real world scenarios where your product or model is not actually doing the thing it was supposed to do.

1:14:05 And the only way you get that is by shipping. Because you get back to sort of use case distribution and you can make those things good. Um and and therefore, you know, it it's actually the best way to then go articulate to your team, especially your male teams. What did he climb on? Like, oh you know, people are trying to do X and the model's failing. In ways why. Now let's make those things really good.

1:14:23 This point about failure cases makes me think about something that both Kevin Wheeler. And Mike Krieger. Shared. Which is that evals are becoming a huge new skill that product people need to

1:14:35 get good at because so much of product building is now evals. Writing e bills. Is is there something there you wanna Share. My entire open ed journey has been this journey of Rediscovering.

1:14:45 Eternal product. Wisdom and principles. In like slightly new context. So I re I started writing evals before I knew what an eval was.

1:14:55 Because like I was just outlining sort of very clearly specified ideal behavior for various use cases. Until someone told me, Hey, you should make an eval. And I r realise there was this entire world Uh

1:15:07 Research, evaluation. Benchmarks that had nothing to do with the product that I was trying to make. And I was like, wow, this might be the lingua franca of how to communicate What um The product should be doing. To

1:15:19 people who do AI research. And that really clicked for me. And at the end of the day, it's not that different from The wisdom of You ought to articulate success before you do anything else. It's just a new mechanism for doing that. But you can do it in a spreadsheet. You can you do it

1:15:35 Anywhere and I Really wanna demystify it for people who heal that term. Like it's not some technical Magic. that you have to understand it's really just about articulating Success.

1:15:46 in a way that is maximally useful for for training blocks. Awesome. There's a I have a post coming out uh soon that gives you a very good uh how to for PMs of how to ready value. I would love to read it. Uh and I hope you disag you I hope you agree with it with what I just said because maybe there's deep to it. Yeah. Yeah. And now there's all these tools that make this easier for you. Totally. Okay. So this this basically backs up this point that this is just a a very important skill that product teams and builders need to get good at. Yeah, yeah. Okay. Just a few more questions. I know you have a lot going on today.

1:16:16 Um One is that this trend of chat GPT being a a big driver of growth. for s traffic to sites a tr uh for product For example Chat TPT is now uh driving more traffic to my newsletter than Twitter.

1:16:33 Which completely shocked me. I just was looking at my stats. I'm like, what the hell? This is not something I knew was coming. So just I guess thoughts on the future of this. How much how you think about just chat GPT driving growth and traffic to Product and sites. I'm really excited about it, um, because

1:16:50 Yeah. In the same way that I I find it d dystopian too. talk to everything through a chat bot. I also find it Dystopian to uh Yeah.

1:16:59 No have. Amazing new high quality content. out there. And um for that reason, you know, I talked a little bit earlier about Uh

1:17:08 search and how that solved like a really important user problem early on because you had this like knowledge cut off thing and you suddenly could talk About anything. Uh very obvious in retrospect. A it wasn't just a user problem, right? It's an ecosystem problem where like the original chat GBT It didn't have outlinks. It would just, you know.

1:17:24 um answer your question and it'll keep you in the product and Yeah, even if you wanted to keep reading or or go deeper. There was no way for us to Drive traffic back. to

1:17:34 uh the content ecosystem and I've been really excited about what we've been doing in Surge. Not just because it gives people more accurate answers, because it allows us to surface really high quality content. Like this podcast. to people um

1:17:46 Who wanna see it. Of course, there's so many interesting questions about Well in the sort of Google era, you know, there was the search engine optimization and there was a clear understood mechanisms of how to show up and get more traffic.

1:18:00 So I get a lot of questions from people, like what is the equivalent of that? The IRA. Yeah. If I'm Lenny and I want to like 10x the traffic to my podcast, you know, what do I actually need to do? Yeah, yeah, the truth is we don't have amazing answers there, um, simply because

1:18:13 The way it appealed to an AI model. Ideally is the same way that you would appeal to a um real user because the model's supposed to proxy the interests of the user and nothing else. At least you know that's how I want our product to work. And

1:18:27 For that reason. Yeah, my advice is super late, which is like make really high quality content. Um which you know is is is not as actionable as I think people making content would ideally. Like and I think this is why we have more work to do, because maybe there's a better mechanism or protocol. Um that we could come up with. But uh

1:18:42 I'm excited. This is driving meaningful uh traffic for you. And I hope that you know other other um people making great content start to feel this way because Again, it's a very neat scenario. There's two uh acronyms people have been using for this this specific skill of AI driven SEO. I think one is AEO, which is answer engine optimization, the other is GEO. Is that I I don't know. I forget the G one. General

1:19:05 Yeah, I don't know. Generative yeah, AI optimization. Do you have a favorite in of those two? No, no, I I I Try to shy away from these terms as in unless they I I'm not entirely sure if if yeah, if that should be a concept or not. Um Again.

1:19:21 I think Ideally. Chat GPT understands your goals. And therefore understands what content would be.

1:19:30 Um interesting to you. And The content creators job is to to yeah, um Share enough.

1:19:38 information and metadata about that content such that the uh model can make a User aligned decision. And therefore I'm I'm not sure if giving this thing a name and you know Making a thing is is is is what we should be doing or not. I'm very eager to learn um from folks making content uh about what this could look like because um again

1:19:56 Um We're we're still working through. Along these lines, another question uh people think about is you have GPTs, which are kind of these like uh GP custom GPT apps that you can build to answer very specific use cases. There's always this question of you gonna build kind of like an app store where I can plug in

1:20:13 my news my product into chat GPT, monetize that. Is there stuff there that you could talk about that might be coming someday? GBTs are cool. They're they're kind of ahead of their time in the sense that We built that kind of concept. Before You could really build very differentiated things.

1:20:29 Uh at least in the consumer space. Yeah. Um You're like learning TPT is gonna be pretty similar to what the model could already do out of the box. So it's mainly like a way of articulating a use case to people. Uh, but it doesn't have enough tools yet to make something. That

1:20:44 Feels like an app. Um, so to speak. Different in the enterprise, by the way, we're seeing a ton of adoption of GPTs there because just Every single company has very bespoke business processes and and problems, et cetera.

1:20:56 And it's a really, really useful tool there. They also have the unique data that they can hook up to these things that it can retrieve over. So We've seen a lot of success there. I think the idea of

1:21:07 Is The right one. Um and I and I think we're gonna figure out a good mechanism for it because When you have so much Capability packed into AI?

1:21:16 It feels really powerful to allow people to package that up. in ways that have a clear affordance, a clear use case and are differentiated from each other. I also would love it if you could start a business on Chat GBT. Like I think there really is a world where, you know As this thing hits. billion years of scale.

1:21:33 It can get you distribution, it can get you uh, you know. Started on making something in the same way that people built on the internet. And you know, there was entirely new businesses to be built. So I think we'll have more to share there in the future. GBTs was an early stab.

1:21:45 And I'm just excited to evolve the thinking there, um, as the models get. Good. Reach uh increases as well. Amazing. That is really cool. I'm really excited to see what you guys do there. Okay. Uh co a completely different direction. Something that I know about you is you studied philosophy.

1:22:00 In college. I did. Computer science and philosophy, right? A combo. Yeah, I started as a philosophy major um And and uh uh took one coding class because I really liked logic and

1:22:13 programming was similar was most similar to that. And then I fell in love with coding and then eventually computer science and I just kept doing more and more of it. But Until then I'd never really thought of myself as a technical person, so it was kind of a late discovery in my life, um that I'm very grateful for. What an incredible combination for someone leading this product. Just it's true. It is really coming in full circle in a way that I couldn't have predicted. Like the amount of questions you have to grapple with.

1:22:36 are truly Super interesting and philosophy isn't it's not a pr traditionally practical skill, but it does really teach you To think things through from scratch. And to

1:22:46 You know, articulate. A point of view. And I think That has come in handy numerous times. Is there a specific philosopher or school That has been most handy to you, or is there more just the jobs?

1:23:00 whether and why rational people can disagree. Um Which um, you know, also comes in handy when lot of people with very different values have opinions on your model behavior, or on w you know how things should work. Um so Um I really like you know.

1:23:14 Twentieth century analytical philosophers. Um it's it's kinda nerdy stuff. Uh but uh um Um, I don't know if I'm a favorite. Um Too many to count. Um But um that's the kind of stuff I like. Um and some of it ends up being quite analytical. Like you have like let P be this you know this theory of love and let Q be, you know This other theory of love and then you do some sort of

1:23:35 Symbolic manipulation. So It is just as much a like sort of brain Thought exercise as it is. Or is much more that than than practical. But it it taught me how to think in a way that continues to be pretty valuable.

1:23:48 Incredible. What a cool What a cool compo of skills and and background. Uh last question before we get to a very exciting lightning round. So you were a product leader at Dropbox, then Instacart, now you're

1:23:59 The PM of Arguably the most consequential product in history. How did you land in this role? What was the story of joining open AI and Taking on this work.

1:24:10 Every single career decisions I ever made, um, including my First one out of college was just Figuring out. Who who are the smartest people I know that I wanna like hang out with and learn from and can I work with them? And

1:24:25 I don't know how to make companies. I don't know how to really logically think through, you know, what space is gonna take off or something like that. But I just do I have a sense on people and um

1:24:37 Yeah. For Dropbox I, you know, f followed like the head teaching assistant for a class that I Uh was Ting and um you know for it's the car as well followed some of the Smartest product people I knew and for for open AI. Um

1:24:50 The person who I recruit who recruit me uh Joanne, uh I had messaged her about getting off the Dolly wait list and she said hey only if you interview here. So she like kinda turned it into like a reverse recruiting. thing. And you know, initially, honestly, I didn't know what I would do here because it was a research lab and As a private person.

1:25:08 Yeah, they said you know, don't worry. Um We'll figure it out and they were sort of being cagey. And I thought they were being cagey because it's open AI and they can't share anything. But they were being c because we we actually just didn't know yet. Um At the time, so I showed up and I Kinda did.

1:25:23 Everything. Under the sun. Yeah. definitely wasn't product. You know, it was like, you know I think my first task was like fix the blinds or something like that. And then yeah, I

1:25:32 started sending out NDAs for people because they were Needed some up operational help. And then Yeah, I started asking, wait, why am I sending out NDAs? Oh, so we could talk to users. And I was like, Talking to users, that sounds like the thing I know how to do. And I quickly stumbled into Doing product work. Um and then eventually you know.

1:25:49 leading a bunch of uh uh product work, but it was organic by just, you know Showing up and doing what had to be done. Um because again the the company I joined was not a product company. But Wow. This is such a good example of I don't know if you think of it this way, but when someone offers you a

1:26:05 Seat on a rocket ship, don't ask which seat. Uh, except I didn't know it was a rocket ship. I just thought it was I I kinda got nerd sniped, is what I would describe it as. Where like, you know, as I prepared for the conversation to get yeah. Oh, the Dolly wait list, really. Uh yeah, I I just started yeah. reading about the space and that, you know. peak the like philosophy brain and then also actually the

1:26:27 Computer science branders like we This is cool. And then I started reading all the academic papers. Of that era. And uh Yeah, I so I just

1:26:35 It was intellectual itch and and the people. But then I stayed for the product opportunity, obviously. I I you know. Post chat GPT when that took off realized that you know we'd built a rocket ship. Um Uh where we launched it while building it.

1:26:49 Uh maybe this is the analogy. Uh but I can't say that you know it felt like a hyped a job or um um or anything like that when I Light. So kind of a a lesson there is follow as you said, follow the smartest people you know. There's also just this thread of

1:27:06 Uh follow things that are interesting to you. Just you playing with Dolly led to this opportunity. Yeah, yeah. And actually that's something we still test for. Is a is curiosity is like a attribute that we think matters so much more than your ML knowledge. Um

1:27:20 I'm not making a comment on research hiring. I think you do need some ML knowledge, I'm afraid. But you know, on Like for product and engineering and design people and you know those kinds of functions, I actually think that if you are just Curious about the stuff works. It doesn't matter at all if you've never done it before. In fact, if you were to filter for people who've done it before, you'd have a very narrow filter.

1:27:39 Of very lucky people. Rather than necessarily the best people you can get. So Um I think we've scaled that. Certainly what got me here, but I think it's actually just generically been a good predictor of success at Open Yeah. Nick, I told you I had a a billion uh I said I had two billion questions to ask you. I feel like I've asked a lot. I feel like I still have a billion left, but I know You told me right after this you have a big GPT five check in that you gotta get to, so

1:28:01 We got a ship. We gotta ship it. Better ship now that this is recorded and we're putting this out. This is this is what the forcing this is the forcing function. Okay, so before we get to our very exciting lightning round, is there anything else that you want to share? Leave listeners with think is important to To share. I try to share a little bit about how I made decisions because I hope to

1:28:23 Yeah. I'm not that far out of school. I like relate a lot to people who are coming in the job market who are trying to figure out What do they do with their life right now? And I feel very confident that if you s surround yourself with people that give you energy

1:28:37 And if you follow The things you're actually curious about. That you're going to be successful in this era. So my you know Parting advice um to folks really is

1:28:49 Put yourself around good people. Um and Do the things you're actually passionate about because In a world where this thing can like, you know, answer any question, asking the right question is very, very important. Yeah, the only way to get yeah.

1:29:02 Um Learn how to do that is is to to yeah. Nurture your own curiosity. So um I uh it worked for me and um it's the one repeatable thing that I can Um

1:29:13 Sure. Everything else is luck. And this is counter to what a lot of people are doing right now, which is follow the money. Where can I make the most? How do I grow this thing and make a hundred million dollars? Like all these people that are getting these crazy offers were not. Planning to make a lot of money doing this.

1:29:27 It's quite interesting to see. that stuff play out because I think all these people entered Yeah, school for genuine reasons. They were like excited about the space. They were researching it. They were pursuing knowledge. And I'm happy that that's being rewarded.

1:29:41 Um I don't know what the rewards Especially in a post AGI. Um world. But I I just have a feeling that if you if you yeah you if you follow that advice, um it'll end up okay.

1:29:54 With that, Nick, we reached our very exciting lightning round. I've got five questions for you. Are you ready? Yep. What are two or three books that you find yourself recommending most to other people? In the product space, probably things like high output management or the design of everyday things or those kind of classic type things, because I think they're extremely applicable. And we talked about philosophy. I don't know. Is there a philosophy book you would be like here's the one to read if you're getting people. fun fun it's uh that is the type of thing I recommend. I don't think there's a practical reason. to to read that stuff, but I will nerd out about it with you. So um

1:30:30 Um at your own peril. Do you have a favorite recent movie or TV show you've really enjoyed, if you've had time to watch anything? I think you've gotta do a little bit of sci fi to Um you shouldn't copy any of it, but um I th I I think you you learn from it. So, I'm gonna go. regularly rewatch her and

1:30:46 West World. Severance was Severance was great. Um I think that's the stuff that, you know, when I have Time all.

1:30:55 I'll meddle with That is awesome. Uh I love that those are the two. Of all the sci fi movies, those are the ones you resonate most with and find most interesting and valuable. Mm yes, but that's probably my own limitation. Um so um I'm sure there's more to more to discover. By the way, have you read Fire Upon the Deep? Side notebook. Um Okay. Uh I don't know if you have time to read this book, but it's

1:31:14 I think you would love it. It's such a AI oriented sci fi space opera sort of book. Right. Yeah. Um okay. Is there a favorite Do you have a favorite product you recently discovered that you really love?

1:31:29 I actually don't. I am like at extreme capacity. Yeah. Yeah. Yeah.

1:31:37 API developers ask me, it's like, Hey, are you like, you know, cop gonna copy all of our products that are like I actually just do not have time to to Follow up. Yeah. Outside of open AI.

1:31:48 Because the pace here is is so so intense. So um Don't have good recs for you, I'm afraid. That's a really that's a comfort against answer, I think, to a lot of product companies. Okay, Nick does no time to even look at our stuff. Oh man. Okay. Do you have a favorite life motto that you find yourself

1:32:04 Using when things are tough. Sharing with friends or family that other few people find useful. Being the average of the you know, the the five the five people you you spend the most time with is is like a thing that really internalized. And both in my personal life, where there's like people who give me energy and who Yeah, lift me up and make me like a better person. Um

1:32:22 My fiance's one of those people, but you know, if there's many people in my life. But then there's also just like you know Um at work, there's the equivalent and again that's how I've made all the career decisions. It's like you know. Who do I want to learn from? So I apply that principle constantly.

1:32:35 Final question. Everybody I talked to told me that you are a very good jazz pianist. You have won competitions, I think you were planning to do this as your main thing and then you somehow took the side quest. Yeah, I chickened out uh the at the very last minute, but I was gonna I was gonna go to school for for music and um that's still my like Hopefully chapter two. Uh I love that that might still happen. Might still happen. Now I'm like I'm in some some some for fun bands. Um and we will kick from time to time. It's like that.

1:33:04 The one thing I can do when I'm otherwise Yeah. Super tired and can't can't can't think anymore because it it balances me out and In good ways. But uh yeah, hopefully I'll get to do more of it. Um

1:33:15 Um in the future. Is there any analogs between music and your job? Anything that you you find Yeah, actually. I I feel like I feel like You can think of software development as like Yeah.

1:33:27 Or being a product person as you could you could be a conductor of an orchestra or you could be in a jazz band. And I think of it as a jazz band. Where like Don't believe in the in the idea of everyone having this like set part

1:33:39 That they have to play. Um And me like kind of you know. Telling people when to play. I

1:33:46 I I love how you know in in jazz or like other forms of it provides music. You're kinda riffing off of each other and you listen to what one person played and then you like play something back. And I I think That great product development is like that. In the sense that ideas could come from anywhere. It shouldn't be a scripted process.

1:34:01 You should be like trying stuff out, having fun, having play. And in in in in what you do. So I use that analogy a lot. Mm.

1:34:10 For those who like music, it tends to resonate. Nick, I am so thankful that you made time for this. I know today is insane today. Tomorrow's gonna be even more insane for the entire world. They have no idea what's coming. Thank you so much for doing this. Two final questions, where can folks find you, if you want them to find you online. Where can folks find GPT five potentially? And then just how can listeners be useful to you?

1:34:30 Just use the product. You don't even have to pay. Um should be your default model. Starting tomorrow. Um yeah, just use it and Don't think about models anymore.

1:34:39 Uh unless you want to and you're a pro user, in which case you get all the little models. So um rest assured. And uh Useful Honestly, I I learned so m so much from people at large and chat GPT users, et cetera. So just keep doing your thing. I'm

1:34:55 Watching in learning and um I appreciate all the feedback. So I'm sure after we fix the model chooser, you guys will roast me for something else and take it. So uh keep it coming. Amazing. Nick, thank you so much for being here. Thanks for having me, Lenny. And good luck tomorrow.

1:35:10 Thanks. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show.

1:35:30 at Lenny's podcast dot com. See you in the next episode.

What you see above is a preview of the first minutes. One unlock costs 10 credits and covers this episode forever: full segment and word-level timestamps on this page, plus .txt, .srt, .vtt and word-level JSON downloads, as many times as you like.