Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google, and Amazon

Transcript from https://podmenti.com/t/8cb00789c4a7ce04

We worked on a guest post together that this really key insight that building AI products is
Very different from building non-AI products. Most people tend to ignore the non-determinism. You don't know how the user might behave with your product, and you also don't know how the LLM might respond to that. The second difference is the agency control trade-off. Every time you hand over decision-making capabilities to agentix systems, you're kind of relinquishing some amount of control on your
This significantly changes the way you should be building product. So we recommend building step by step. When you start small, it forces you to think about what is the problem that I'm gonna solve. In all this advancements of the AI one easy slippery slope is to keep thinking about complexities of the solution and forget the problem that you're trying to solve. It's not about being the first company to have an agent.
Among your competitors. It's about have you built the right flywheels in place so that you can improve over time. What kind of ways of working do you see in companies that build AI products successfully? I used to work with the CEO of now Rackspace. He would have this block every day in the morning, which would say catching up with AI four to six AM. Leaders have to get back to
to being hands on, you must be comfortable with the fact that your intuition might not be right. And you probably are the dumbest person in the room and you want to learn from everyone.
is gonna look like persistence is extremely valuable. Successful companies right now building in any new area. They are going through the pain of learning this, implementing this, and understanding what works and what doesn't work. Pain is the new mode.
Today my guests are Ashwarya Raganti and Kuriti Bottom.
Three T works on codecs at OpenAI.
And has spent the last decade building AI and ML infrastructure at Google and at Kumo. Ash was an early AI researcher at Alexa and Microsoft.
And has published over thirty five research papers.
Together, they've led and supported over fifty AI product deployments across companies like Amazon, Databricks, OpenAI, Google, and both startups and large enterprises.
Together, they also teach the number one rated AI course on Maven.
Where they teach product leaders all of the key lessons they've learned about building successful AI products. The goal of this episode is to save you and your team a lot of pain and suffering and wasted time trying to build your AI product.
Whether you are already struggling to make your product work or want to avoid that struggle, this episode is for you. If you enjoyed this podcast, don't forget to subscribe and follow to your favorite podcasting app or YouTube. It helps tremendously.
And if you become an annual subscriber of my newsletter, you get a year free of a ton of incredible products, including a year free of lovable, replit, bulk, gamma, N8M, Linear, Devon, Posthoc, Superhuman, Descript, Whisperflow, Perplexity, Warp Granola Magic Pattern, Dreycast, Chapter D Mobbed, and Stripe Atlas. Head on over to Lenny's Newsletter.com and click Product Pass.
With that, I bring you Eshwarya Raganti and Kiriti Bottom after a short word from our sponsors.
This episode is brought to you by Merge.
Product leaders hate building integrations.
They're messy.
They're slow to build, they're a huge drain on your roadmap.
And they're definitely not why you got into product in the first place.
Lucky for you, Merge is obsessed with integrations.
With a single API, B2B SaaS companies embed Merge into their product and ship 220 plus customer-facing integrations in weeks, not quarters.
Think of merge like Plat.
But for everything B to B Sass.
Companies like Meestrol AI, Ramp, and Drata use Merge to connect their customers' accounting, HR, ticketing, CRM, and file storage systems to power everything from automatic onboarding to AI ready data pipelines.
Even better, Merge now supports the secure deployment of connectors to AI agents with a new product, so that you can safely power AI workloads with real customer data.
If your product needs customer data from dozens of systems, merge is the fastest, safest way to get it. Book and attend a meeting at merge.dev slash Lenny, and they'll send you a$50 Amazon gift card.
That's merge.dev slash lenny.
This episode is brought to you by Strella, the customer research platform built for the AI era.
Here's the truth about user research.
It's never been more important or more painful.
Teams want to understand why customers do what they do. But recruiting users, running interviews, and analyzing insights takes weeks.
By the time the results are in
The moment to act has passed.
Strella changes that.
It's the first platform that uses AI to run and analyze in-depth interviews automatically.
Bringing fast and continuous user research to every team.
Strella's AI moderator asks real follow-up questions.
Probing deeper when answers are vague.
And services patterns across hundreds of conversations all in a few hours, not weeks.
Product, design, and research teams at companies like Amazon and Duolinga are already using Strela for Figma prototype testing, concept validation, and customer journey research, getting insights overnight instead of waiting for the next sprint.
If your team wants to understand customers at the speed you ship products.
Try strong. Run your next study.
At strella.io slash Lenny. That's S T R E L L A dot Io slash Lenny.

Ash and Kiriti, thank you so much for being here and welcome to the podcast.
Thank you, thank you for having us. Super excited for this.
Let me set the stage for the conversation that we're gonna have today.
So you two
Have built a bunch of AI products yourself.
You've gone deep with a lot of companies who uh
have built AI products, have struggled to build AI products.
Build AI agents.
You also teach a course on building AI products successfully.
that and you're kind of like on this mission to just reduce
Pain and suffering.
And failure.
uh that you constantly see people go through when they're building
AI products.
So to set a little just foundation for the conversation we're gonna have.
What are you seeing on the ground within companies trying to build
A I products. What's going well, what's not going well.
I think.
Twenty twenty five has been significantly different than twenty twenty-four. One, the skepticism has significantly reduced. Um, there were tons of leaders last year who probably thought this would be yet another crypto wave and kind of skeptical to get started.
And a lot of the use cases that I saw last year were more of Slapchat on your data, right? And that was, you know, um calling themselves an AI product.
And this year, a ton of companies are really rethinking their user experiences and their workflows and all of that, and really understanding that you need to
deconstruct and reconstruct your processes in order to have uh in order to build successfully products, right? And that's that's the good stuff.
The bad stuff is the execution is still all over the place. Um
Think of it, right? This is a three year old field. There are no play playbooks, there are no textbooks.
Um so you really need to figure out as you go. And the AI life cycle both
Pre deployment and post deployment.
is very different as compared to a traditional software life cycle. Um
And so
So a lot of old contracts and handoffs between traditional roles, like say PMs and engineers and data folks.
has now been broken. It's it's and
People are really getting adapted to this new way of working together and kind of owning the same feedback loop in a way, because.
Previously I feel like PMs and engineers and all of these folks had their own feedback loops to optimize. And now you need to be probably sitting in the same room. You're probably looking at agent traces together and deciding how your uh product should behave. So
it's a tighter form of collaboration. So companies are still kind of figuring that out. That's kind of
what I see um in my consulting practice this year.
So let me follow that thread. We worked on a guest post together that came out a few months ago.
And the thing that stood out to me most that stuck with me most after working on that post is
Yeah, that's really uh
key insight that building AI products is
Very different from building non AI products. And the thing that you're
big on getting across is there's two very big differences.
Talk about those two differences.
Yes, um and again I I wanna make sure that we
Drive home the right point. Um, there are tons of uh similarities of building AI systems and software systems as well.
But then there are some things that kind of fundamentally change the way you build software systems.
um versus AI systems, right? And one of them that most people tend to ignore is the non-determinism.
uh you're pretty much working with a non-deterministic API as compared to traditional software.
What does that mean and why does that have to affect us is in traditional software, you pretty much have a very well mapped
Is it should an engine or workflow?
Think of something like booking.com, right? You um you have an intention that uh you want to make a booking in San Francisco for two nights, etc.
Uh, the product has kind of been built so that your intention can be converted into a particular action and you kind of are clicking through a bunch of buttons, options, forms, and all of that, and you finally achieve your intention.
But now that layered and EI products has completely been replaced by
a very fluid um
interface which is mostly natural language.
Which means you
The user can literally come up with a ton of ways of saying or communicating their intentions, right? And that kind of changes a lot of things because now you don't know how your user is going to behave. That's on the input side. And the output is also that you're working with a non-deterministic, probabilistic API, which is your LLM.
And
LLMs are pretty sensitive to prompt phrasings and they're pretty much black boxes. So you don't even know how the output surface will look like, right? So this.
Um, you don't know how the user might behave with your product, and you also don't know how the LLM might respond to that. So you're now working with an input, output, and a process process.
And you don't understand all the three very well. You're trying to kind of anticipate behavior and build for it. And with agentic systems, this kind of gets even harder. And that's where we talk about the second difference, which is the agency control trade-off, right?
What we mean by that, and I'm kind of shocked, so many people don't talk about this. They're extremely obsessed with building autonomous systems, agents can that can do work for you. But every time you hand over decision making capabilities or autonomy to agentic systems.
You're kind of relinquishing some amount of control on your entry, right? And when you do that, you wanna make sure that your agent has um chained your trust or it is reliable enough that you can allow it to make decisions. And that's where we talk about this agency control trade-off, which is
If you give your AI agent or your AI system, whatever it is, more agency, which is the ability to make decisions, you're also
uh losing some control and you want to make sure that
the agent or the I system has earned
um that ability or has built up trust over time.
So just to summarize what you're sharing here.
Essentially people have been building product software products for a long time.
We're now in a world where the software you're building is one
non-deterministic can just do things differently. Like you know, as you said, you go to Booking.com, you find a hotel that's gonna be the same experience every time. You'll see different hotels, but
It's a predictable experience.
But they uh you can't
Predict that it's gonna be the exact same thing the thing that you
Uh planet to be every time.
And then the other is there's this trade off between agency and control.
How much will the AI do for you versus how much should the person still be in charge?
And the
What I'm hearing is the big point here is this significantly changes the way you should be building product. And we're gonna talk about
The impact on how
that product development lifecycle.
should change as a result.
Is there anything else you want to add there before we get into into that?
Yeah, it's definitely like one of the key points that uh this kind of distinction needs to exist in your mind, like when you're starting to build. For example, think about if your like objective is to hike uh half term in UCMT, right? You don't start hiking it every day, but you start, you know, training yourself for like, you know, in in
minor parts and then you slowly improve and then like you go to the end goal, right? I feel like that's ex extremely similar to what you want to build AI products.
In the sense that when you don't start with like agents with all the tools and all the context that you have in the company in day one and expect it to work or like you know even tinker at that level. You need to be deliberately starting in places where there is minimal impact and more human control. So that you have like a good grip of what are the current capabilities and what can I do with them.
And then slowly, you know, like lean into the more agency and lesser control. So this gives you that confidence that okay, I
Can know that okay, this is the particular problem that I'm facing, and the AI can solve this extent of it. And then like let me next thing through.
What context I need to bring in, what kind of tools I need to uh to this to improve the uh experience, right? So I feel like it's also uh it's a good and a bad thing in the sense that it's good that you don't have to see the complexity of the outside world of like, you know, all of this fancy AI uh agents force and feel like I cannot do that.
It's always everyone is starting from very uh
minimalistic structures and then evolving. And the second part is like it's also good the the bad thing is that as you are like, you know, trying to build this one click agents into your company, you don't have to be overwhelmed with this complexity. You can like slowly graduate. So that's extremely important. And we see this as a repeating pattern over and over.
Okay, so let's actually follow that throughout,'cause that's a really important component of how you recommend people build AI stuff.
A I stuff. A product, A agents, all the AI things.
Um, so sh give us an example of what you're talking about here, this idea of starting
Yeah, slow with agency and control and then moving kind of up the strong.
Yeah, for example, uh very important or like very prevalent uh application of uh AI agents is like customer support, right? Uh imagine like you are a company who has like a lot of customer support tickets. And why even imagine like OpenAF is the exact same thing when we were launching products.
And there was like a huge spike of uh support volume as like, you know, we launch successful products like Image and or uh you know, like GPT five and things like that. The kind of questions you get is different. The kind of like, you know, uh
problems that the customers bring to you is different. So it's not about just like dumping all the uh list of help center articles that you have into the agent. You kind of understand what are the things that you can build. And so initially the first step of it would be something like
Uh, you have your support agents, the human support agents, but you will be suggesting uh in terms of okay, this is what the AI thinks that is the right thing to do. And then you get that feedback loop from the humans that okay, this is actually a good suggestion for me in this particular case, and this is a bad suggestion. And then you can go back and understand, okay, uh this is what the
And drawbacks are this is where the blind spots are. And then how do I fix that? And once you get that, you can increase the autonomy to say that okay, I don't need to suggest the human, I'll actually show the uh show the answer directly to the customers up.
to the customer. And then we can actually add more complexity in terms of okay, uh I was only answering
questions based on health center articles, but now let me add new functionality like
I can actually issue refunds to the customers. I can actually raise feature requests with the engineering team and all of these things. So if you start all with all of this on day one, it's incredibly hard to control the complexity. So we recommend like, you know, building step by step and then increasing it.
Awesome. And you have a visual actually that
We'll share.
of what this looks like, but just to kinda mirror back what you're describing, this idea of start with
high control, low agency. In your c the example you gave is
The support agents just kind of
Giving suggestions.
is not able to do anything, the user
is in charge.
And then as that
Becomes
useful and you are confident it's doing the right sort of work.
you give it a little more agency and you kind of
pullback on the control the user has.
And then if that's starting to go well, then you give it more agency and the user
needs less control to control it.
Awesome.
I I think the higher level idea here is
Від АІ системс, і салабав бегаєр калібрицін.
It's incredibly impossible to predict upfront how your system behaves.
Now, what do you do about it? You make sure that you don't ruin your customer experience or your end user experience.
Um you keep that as is, but then remove the amount of control that the human has and
There is no single right way of doing it. You can decide how to constrain that autonomy, right? Um
a very I mean a different example of how you could constrain autonomy is
pre-authorization use cases.
Insurance pre-authorization is a very ripe use case for AI because uh clinicians spend a lot of time um pre-authorizing uh things like blood tests, MRIs, and things like that, right? And there are some cases which are more of low hanging fruits, for instance,
MRIs and blood tests because um as soon as you know patients' information, it's easier to approve that and AI could
Yeah.
do that versus something like an invasive surgery, et cetera, is more high risk. You don't want to be doing that autonomously. So you can kind of determine which of these use cases should go through that human and the loop layer versus which of the use cases AI can conveniently handle. And then all through this process you're also logging what the human is doing, right?
Because you want to build a flywheel, um
that you could use in order to improve your system.
Um so you're essentially um not ruining the user experience, not eroding trust, at the same time logging what humans would otherwise do so that you can continuously improve your system.
So let me let me give you a few more examples of this kind of progression that you recommend. And this the reason I'm spending so much time here is this is a really
Key part of your recommendation.
to help people build more successful AI products.
This idea of start.
slow with high control.
and low agency and then build up over time once you've build confidence that it's doing the right sort of work.
So a few more examples that you shared in your post that I'll just read.
So say you're building a coding assistant.
V one would be just suggest inline completion and boilerplate snippets.
V two would be generate larger blocks like tests or refactors for humans to review. And then V three is just apply the changes and open PRs autonomously.
And then another example is a marketing assistant. So B one would be draft emails or social copy, just like here's what I would do.
D two is build a multi step campaign and run the campaign.
And then long and V three is just launch it, A B test it, auto optimized campaigns across channels.
Awesome. Yeah.
And and again, just to summarize where we're at, just to give people the the the advice we've shared so far.
Uh
One is just
Important to understand AI products are different. They're non deterministic. And he pointed out and I forgot to actually mirror back this point.
Both on the in s on the input and the output.
The user experience is non deterministic. Like you people will see different things.
different outputs, different chat conversations, different maybe UI if it's designing the UI for you.
And also the output obviously is gonna be non deterministic, so that's a problem and a challenge.
And then uh I mean, if you think of it, it's also the most beautiful part of AI, which is
I mean, we're all much more comfortable talking than
following a bunch of buttons and all of that, right? So the bar to
using AI products is much lower because you can be as natural as you would be with humans. But that's also the problem, which is there are tons of ways we communicate.
Um and it's
you wanna make sure that that intent is rightly communicated and the right actions are taken because most of your systems are deterministic and you want to achieve a deterministic outcome.
uh but with non deterministic technology and that's where it gets a little messy.
Awesome. Okay. That's a I love I love the
The optimistic version of the
Why this is good.
Okay. And then the other piece is this idea of this trade off of autonomy versus control. When you're designing thing
And what I imagine what you're seeing is people try to jump to the idea like the
V three.
Immediately.
And that's when they get into trouble both.
probably a lot harder to build that and it's just doesn't work and then they're just like okay, this is a failure. What are we even doing?
Exactly. I feel there's like a bunch of things that you actually have to uh get confidence in before you get to V three. And it's it's easy to get overwhelmed that oh my AI agent is like doing these things wrong in like a hundred different ways. And you're not gonna actually tabulate all of them and fix it, right?
even though you've learned like, you know, how do you deal with the uh evaluation practices and stuff like that. If you're starting on the wrong spot,
you are actually going to have a hard time like you know correcting things from there. And when you start, uh
Small and when you start with a building like a
very minimalistic question with high human control and low agency. It also forces you to think about what is the problem that I'm gonna solve. Uh we we use this term called problem first.
And uh
To me, it was like obvious in the sense that yeah, I I do need to think about the problem, but it's incredible how well it s resonates with the people that in all this advancements of the AI that we're seeing, one easy slippery slope is to
Just keep thinking about uh complexities of the solution and not and forget the problem that you're trying to solve. So when you're tr trying to start at like a small at at a
smaller scale of autonomy, you start to really think about what is the problem that I'm trying to solve and how do I break it down into like a levels of autonomy that I can build later. So that is incredibly useful when like and we keep repeating this pattern over and over with everyone we talk to.
Mm.
And there's so many other benefits to
Uh limiting autonomy because
There there's just danger also of the thing doing too much for you.
And just messing up.
Your I don't know, your database sending out all these emails you never expected. Like there's like so many reasons this is a good idea.
Yep. I I recently read this paper from a bunch of folks at UC Berkeley.
Um basically Mate is Ahari uh Stoika and the folks at Databricks.

About seventy-four or seventy-five percent of the enterprises that they had spoken to, um, their biggest problem was reliability. And that's also why they weren't
a comfortable um deploying products to their end users and
building customer facing products because they just weren't sure or they just weren't um comfortable doing that and exposing their users to a bunch of these risks, right? And and that's also why they think a lot of AI products today have to do with productivity, because it's much low autonomy.
versus, you know, end to end agents that would replace workflows. Um and
Yeah, I love their work otherwise as well, but I think that's very in line with what um at least we're seeing at my startup as well. Okay. Very interesting.
There's an episode that'll come out before this conversation.
Where we go deep into another problem that this avoids
Which is around uh prompt injection and jailbreaking and just how big of a
Uh
ex risk that is for AI products where it's essentially an unsolved and unsolvable problem potentially. I'm not gonna go down that track, but that's yeah uh that's a pretty scary conversation we had that it'll be out before this conversation.
I think that'll be a huge problem once systems go mainstream. We're still so busy building AI products that we're not worried about security.
But it it will be um such a huge problem to kind of uh
Especially with this non deterministic API again, right? So you're
kind of stuck because um there are tons of instructions that
you could inject within your prompt and then
Yeah, it's it's going to be bad.
Okay. I uh let's actually spend a little time here,'cause it's actually really interesting to me and no one's talking about this stuff.
Which is like the conversation we had is just
It's pretty easy to get AI to trick to do stuff it shouldn't do.
And there's all these guardrail systems people put in place, but
turns out these guardrails aren't actually very good and you can always get around them.
And
To your point, as agents become more autonomous and robots
Uh it gets pretty scary that you could get AI to do things easy.
Shouldn't do.
I think this is definitely a problem, but I feel in the current spectrum of like customers adopting AI, the
the extent to which like you know companies can actually get advantage of AI or like improve their processes or like you know streamline the existing processes that they have. I feel it's in still in the very early stage. Like 2025 has been an extremely busy year for AI agents and customers trying to adopt AI.
But I feel the penetration is still not as much as you would actually get advantage out of it. So
with the right set of you know human in the dupe uh points in here. I feel we can actually avoid a bunch of these things and focus more towards like streamlining the processes. And I I I am more on the optimist side in the sense that like you need to try and adopt this before actually like trying to be only for highlighting the
negative aspects of like what could go wrong. So I I feel like strongly um that companies have this adopt this, they definitely like no company uh
At opening I we talk to is
has never been the case that oh AI cannot help me in this case. It has always been that oh there is this like set of things that it can uh optimize for me and then let me see how I can adopt it.
Sweet. I always like the optimistic perspective.
I'm excited to for you to listen to this and see what you think'cause it's really interesting and uh and to your point, there's a lot of things to focus on. It's one of one of many things to to worry about and think about.
Okay, let's get back on track here.
So we've shared a bunch of pro tips and important piece of advice.
Let me ask what other patterns.
And kind of ways of working.
Do you see in companies that do this well and teams that build AI
product successfully.
And then just what are the most common pitfalls people fall into? So we could just maybe start with
What are other people?
ways that companies do this well, build AI product successfully.
I almost
Think of it as like a
success triangle with three dimensions. It's never always technical.
Every technology problem is a people problem first.
And with companies that we have worked with, it's
These three dimensions, right? Like great leaders.
Good culture and technical prowess.
Um, with leaders itself, we work with a lot of companies uh for their AI transformation, training, strategy, and stuff like that.
And I feel like um
A lot of companies, the leaders have built in tuitions over ten or fifteen years and they're
kind of highly regarded for those intuitions but now with AI in the picture.
those intuitions will have to be relearned and
Teachers have to be vulnerable to do that, right? Um, I used to work with the CEO of now Rack Space.
the gen. So he would um
have this block every day in the morning, which would say catching up with AI four to six A.
And he would not have any meetings or anything like that. And that was just his time to pick up on the latest AI podcasts or information and all of that. And he would have um weekend white coding sessions and stuff like that.
So I think leaders have to get back to being hands on, and that's not because they have to be implementing these things, but more of
uh rebuilding their intuition because you must be comfortable with the fact that your intuition might not be right. Um and you
You probably are the dumbest person in the room and you want to learn from everyone.
Um, and that I've seen that being a very um distinguishing factor of companies that build products um which are successful because you're kind of bringing in that top down approach.
It's almost always impossible for it to be bottom up. You can't have a bunch of engineers go and
get buy in from the leader if they just don't trust in the technology.
Or if they have misaligned expectations about the technology, right? I've heard from so many
Folks who are building that.
leaders just don't understand the extent to which AI can solve a particular problem, or they just wipe out something and assume it's easy to take it to production. And you really need to understand the range of what AI can solve today, so that you can guide decisions within the company.
The second one is the culture itself, right? And
Again, I work with enterprises where
AI is not their main thing and they've
Um, they need to bring in AI into their processes just because a competitor is doing it, and just because it does make sense because there are use cases that are very ripe.
Then along the way, I feel a lot of companies have this culture of FOMO and you will be replaced, and those kind of things, and people get really afraid.
Subject matter experts are such a huge part of building AI products that work because you really need to consult them to understand how your AI is behaving or what the ideal behavior should be. But then I've spoken to a bunch of companies where the subject matter experts just don't want to talk to you because they think

S
As I mean, again, this comes from the leader itself. You want to build a culture of empowerment, of um augmenting AI into your own workflows so that, you know, you can 10x in what you're doing instead of saying that
you know, probably uh you'll be replaced if you don't adopt AI and stuff like that. So that kind of an empowering culture always helps. You want to make um your entire organization be in it together and make AI work for you instead of trying to, you know,
guard their own jobs, et cetera. And with AI, it's also true that it opens up a lot more opportunities than before. So you could have your employees doing a lot more things than before, and 10x their productivity.
Um and the third one is the technical part which we talk about, right? I think
Folks that are successful are incredibly obsessed about understanding their workflows very well and augmenting parts.
um that could be um um that could be ripe for AI versus the ones that might need human in the loop somewhere, et cetera. Whenever you're
uh trying to automate some part of a workflow.
It's never the case that you could
You could use an AI agent and that'll kind of solve your uh problems, right? It's always you probably have a machine learning uh model that's going to do some part of the job. You have deterministic code doing some part of the job. So you really need to be obsessed with understanding that workflow. So you can choose the right tool for the problem instead of being obsessed with the technology itself.
And um
Another pattern I see is also folks really understand this idea of
working with a non deterministic API, which is your LM
And
What that means is they also understand the I development lifecycle looks very different. And they iterate pretty quickly, which is can I um can I build something, iterate uh quickly in a way that it doesn't ruin my customer experience, at the same time gives me enough amount of data.
So that I can estimate behavior, right? So they build that flywheel very quickly.
As of today, it's not about being the first company to have an agent among your competitors. It's about have you built the right flywheels in place so that you can improve over time, right? When someone comes up to me and says, We have this one click agent, it's going to be deployed in your system and then
in in two or three days it'll start showing you significant gains.
I would almost be skeptical because it's just not possible. And that's not because the models aren't there, but because enterprise data and infrastructure is very messy and you need a bit to
even the agent needs a bit to understand um how these systems work.
There are very messy taxonomies everywhere, um
People
tend to do things like get customer data, we one, get customer data, we two, and these kind of things. And

Basically there's a lot of tech tet that you need to deal with.
So most of the times it
If you're obsessed with the problem itself and you understand your workflows very well, you will know how to improve your agents over time, instead of just slapping in an agent and assuming that it will work from day one. I probably will go as far to say that if someone's selling you one click agents, it's it's pure marketing. You don't want to buy into that. I would rather
go with a company that says, We're gonna build this pipeline for you and
that that will learn over time and kind of build a flywheel to improve than something that's gonna work out of the box. To replace any critical workflow or to um build something that can give you significant ROI easily takes four to six months of work, even if you have the best data layer and infrastructure layer.
Amazing.
There's a lot there that resonates so deeply with other conversations I've been having on the podcast.
One is just
For a company to be successful at seeing a lot of impact from AI, the founder CO has to be
Deep into it.
Uh, I had Dan Shipper on the podcast and they
work with a bunch of companies helping them adopt AI and he said that's the number one predictor of success.
Is the CO chatting with
Chat GPT, Cloud, whatever.
Uh more many times a day.
I love this example you gave at the Rack Space C as like catch up on AI news in the morning every day. I was imagining he'd be like chatting.
with like the chat bot versus uh
Like reading news.
With the kind of information you have as of today, you could just
Um I mean you wanna choose the right
um channels as well, because everybody has an opinion. So whose opinion do you want to bank on? I feel like having that
Good.
quality set of people that you're listening to really make sense. So he just has a list of two or three sources that he always looks at and and then he comes back with a bunch of questions and bounces it around with
a bunch of AI experts to see what they think about it. And I was part of that group, so I kind of know um about the questions that he comes up with. So it's pretty cool. I was like, Why are you doing so much? And then he says it trickles down into a bunch of decisions that we would take.
Okay, let me talk about uh another topic that's very uh it's been a hot topic on this podcast. It was a hot topic on Twitter for a while, evals.
Mm.

Obsessed with embells.
think they're the solution to a lot of problems in AI.
Lot of people think they're overrated, that way you don't need evals, you can just
Feel the vibes and you'll you'll be all right.
What's your take on
Evals, how far does that
Take people in solving a lot of the problems that you talk about.
In terms of like what is going on in the community, I I feel there's just this false dichotomy of like there's either
Evals is going to solve everything, or online monitoring or production monitoring is going to solve everything. And I find no reason to trust like one of the extremes in the sense that I will
entirely bank my application on this and
Or like that to solve the uh thing, right? So if you take a step back, uh
Think of what are evals. Evals are basically your
uh trusted product thinking or like your a knowledge about the product that is going into this uh set of data sets that you're going to build in the sense that this is what matters to me. Like this is the kind of problems that my agent should not do. And let me build a list of data sets so that I'm going to do well on those.
And in terms of production monitoring.
What you're doing doing there is uh you're deploying your application and then you're having this some sort of key metrics that actually communicate back to you on how customers are using your product. Like you could be deploying uh any agent, and like if the customer is giving a thumbs up for your interaction, you better want to know that.
So that is what production monitoring is going to do, right? And this production monitoring has existed
for products like in for a long time. Just that now with the AI agents, you need to be monitoring like a a l a lot more granularity.
It's not just the customer always giving you explicit feedback, but there is many implicit feedback that you can get. Uh for example in chat GPT, right? Like if you are uh liking the answer, you can actually give a thumbs up.
Or if you don't like the answer, sometimes customers don't give you thumbs down, but actually read regenerate the answer. So that isn't clear indication that the initial answer that you generated is not matting uh
My meeting the customers' expectation, right? So these are the kind of implicit signals you always need to think about. And that spectrum has been increasing in terms of production monitoring. Now let's come back to the initial topic of like, okay, it's is it evaluators or is it production monitoring? What does it matter? So
I feel again we go back to this problem first approach of
What is your what is it that you're trying to build? Like you're trying to build a reliable application for your customers that's not going to do a bad thing. Like it's always going to do the right thing. Or if it is doing a wrong thing, you are uh
You're basically alerted like very quickly, right?
So the I break this down into two parts. Like one is you like nobody goes into uh deploying an application without actually like you know just testing that.
This testing could be wipes or this testing could be okay, I have this like ten questions that it should not go wrong.
Any no matter what changes I make, and let me build this and let's call this an evaluation data set. Now let's say you build this, you deployed this, and then you figured, uh, okay, now I need to understand whether it's doing the right thing or not.
So
If you're a high uh high
uh throughput or like a high uh transaction customer, you cannot tactically sit and evaluate all the traces, right? You need some indication to understand what are the things that I should look at.
And this is where production monitoring comes into the picture that
You cannot predict your uh the ways in which your agent could be doing wrong, but all of these other implicit signals and explicit signals, those are going to communicate back to you.
What uh what are the traces that you need to look at. And that is where production monitoring helps.
And once you get these kind of traces.
You n need to examine what are the failure patterns that you're seeing in these uh different types of interactions. And is there something that I really care about that should not happen?
And if that kind of failure modes are happening.
Then I need to think about building an evaluation data set for it.
And okay, let's say I built an evaluation data set for my agent.
Trying to offer refunds.
Where
Explicitly have configured it not to.
So I built this evaluation data set and then like I made my changes in tools or prompts or whatever, and then I deployed the second version of the product, right? Now uh there is no guarantee that this is the only problem that you're gonna see.
You still need production monitoring to actually have like you know catch different kinds of problems that you might encounter. So I feel evaluation are important, production monitoring is important, but this notion of only one of them is going to solve things for you, that is uh completely dismissible, in my opinion.
All right. A very reasonable answer.
And the point here isn't uh it's not just as simple as do both, it's more that
There are different things to
Catch.
And one
approach won't catch all the things you need to be paying attention to.
Exactly.
Awesome.
I want to take two steps back and kind of
Talk about how much
weight the term Evals has had to take in the second
you know, half of twenty twenty five.
Because you go meet a data labeling company and they tell you our experts are writing evals.
And then uh you have all of these uh folks saying that PMs should be writing evals, they're the new PRDs.
And then you have folks saying that um
evals is pretty much everything, which is the feedback loop you're supposed to be building to improve your products.
No.
Step back as a beginner and kind of think
Like what are evals? Why is everyone saying evals? And these are actually different parts of the process. And nobody is wrong in the sense that yes, these are evals, but when a data labeling company is telling you that our experts are writing evals, they're actually referring to error analysis, or you know, experts just leading notes on what should be right.
lawyers and doctors write evals, that doesn't mean they're building LM judges or they're building this entire feedback loop. And when you say that a PM should be writing evals, doesn't mean they have to write an LM judge that's good enough for production. I think there's
There are also very prescriptive ways of doing this and plus one to K D, which is
You cannot predict upfront if you need to be building an LLM judge versus you need to be using ex um implicit signals from production monitoring, et cetera.
I think
Martin Fowler at some point had this term called semantic diffusion back in
the two thousands, um um, which kind of means that someone comes up with a term, everybody starts butchering it with their own definitions, and then you kind of lose the actual definition of it.
That is kind of what is happening to Evals or Agents or any word in AI as of today. Everybody kind of sees a different side to it, I guess.
Uh, but if you make a bunch of practitioners sit together and ask them, is it important to build a actionable feedback loop for AI products? I think all of them will agree. Now, how you do that really depends on your application itself.
When you go to complex use cases, it's incredibly hard to build LM judges because you see a lot of emerging patterns if you built a judge that would.
um, you know, test for verbosity or something like that. You
Turns out that you're seeing newer patterns that your LM charge is not able to catch and then you're just
Um, you just end up building too many evaluations. And at that point it just makes sense to, you know, look at your user signals, fix them, check if you've regressed and move on instead of actually building these charges. Um so it all depends, I think
one statement that every ML practitioner will tell you is it really depends on the context. Don't be obsessed with prescriptions, they're gonna change.
Uh that's such an important point, this idea that
Especially that eval just means
Many things to different people now. It's just like a term for so many things.
And uh
It it's complicated to just talk about evals when you're think when you see it as the stuff data labeling companies are giving you and things PMR, right?
And there's also benchmarks, people call benchmarks a little bit evals. It's like I I recently spoke to a client who told me we do evals. Yeah. And I was like, okay.
Can you show me your data set and say, No, we just checked LM Arena and artificial analysis. These are you know, independent benchmarks and we know that this model is the right one for our use case. And I'm like, You're not doing evals. That's not eval, it was a model evaluation. But yeah, now it's just confusing it even more.
Yeah.
Just like one more line of questioning here that I think uh
That's on my mind is the reason this became kind of a big debate is Claude Code.
the head of Clock Code, Boris, was like, nah, we don't do e balls on Cloud Code, it's all vibes.
What can you share, Kiriti, on Codex on the Codex team of how you approach e balls?
So
Codex we have like this.
a balanced approach of like, you know, you need to have e balls and you need to definitely listen to your customers. And uh I think Alex has been on your podcast recently and he's been talking about how you're extremely focused on building the right product, right? And
A part of a big part of it is basically listening to your customers.
And coding agents are extremely unique compared to agents for other domains, in the sense that these are actually built for customizability and these are built for engineers. So coding agent is not a product which is going to solve like these top five workflows or like
Top six workflows or whatever, right? It's meant to be customizable in multi different ways. And the
Implication of that is that your product is going to be used in different integrations and different kinds of tools and different kinds of things. So it gets really hard to build an evaluat for all kinds of interactions that
Your customers are going to use your product for right. But that said, you also need to understand that, okay, if I'm going to make a change, it's at least not going to like damage something that is really core to the product. So we have like evaluation for doing that. But the same time, we uh we take like extreme care on like understanding how the customers are using it. For example,
Uh we built this code review product recently and uh it has been gaining like an extreme amount of traction and uh I feel like many, many bugs in open AI as well as like even our external customers are getting caught with this. And now let's say if I'm making a model change to the code review or like
different kinds of uh uh RL mechanism that I trained with it. And uh now if I'm going to deploy it, I definitely do want to AP test.
and identify whether it's actually finding the right uh mistakes and are users how are users reacting to it.
And sometimes like if users do get annoyed by your like, you know, uh incorrect code reviews, they go to the extent of just switching off the product, right? So those are the signals that you want to look at and make sure that your new changes are doing the right thing. And it's extremely hard for us to, you know, uh think of these kind of scenarios beforehand.
And uh develop evaluation data sets for it. So I feel like there's a bit of both, like there's a lot of vibes and there's a lot of like customer feedback. Uh and we're super active on like the social media to understand if anybody's having certain types of problems and quickly that. So I feel it's a
It's a
Um how do I put this? It's it's like a domain of things that you do, yeah.
That makes so much sense. Okay, but I'm hearing codecs pro evals, but it's not enough.
You need to but also.
Uh and just watch customer behavior and feedback. And also there's some vibes, just like is this feeling good? Is this
As I'm using it.
Generating great code that I'm
Excited about that, we think is great.
I I don't think like if anybody is coming and seeing that like my I have this concrete set of evalu that I can like bet my life on and then I don't need to think about anything else, like it it's not gonna work. And every new model that we're gonna launch, we uh get together as a team and like, you know, test.
different things. Each each each each person is like concentrating on something else. And like we have this list of hard problems that we have and we throw that to the model and see how well they are progressing. So it's like
uh custom e vals for each engineer, you would say, and just like understand what the uh product is doing in its new model.
If you're a founder, the hardest part of starting a company isn't having the idea.
It's scaling the business without getting buried in back office work.
That's where Brex comes in.
Brex is the intelligent finance platform for founders.
With Brex, you get high limit corporate cards, easy banking, high yield treasury, plus a team of AI agents that handle manual finance tasks for you.
They'll do all the stuff that you don't want to do.
Like file your expenses, scour transactions for waste, and run reports, all according to your rules.
With Brexit's AI agents, you can move faster while staying in full control.
One in three startups in the United States already runs on Brent.
You can too.
And Brex dot
Oh.
We've been talking for almost an hour already and we haven't even covered your extremely powerful
software development workflow for building AI products that you develop that you teach in your course, that you basically combine all the stuff we've been talking about into
Of step by step approach to building
Yeah products.
You call it the continuous calibration, continuous development.
Framework.
Let's pull up a visual to show people what the heck we're talking about.
And then just walk us through what this is, how this works, how teams can
shift the way they build their AI products to
this approach to help them avoid a lot of pain and suffering.
Before we go about explaining um the life cycle, a quick story on why Kirija and I came up with this is because
Um, there are tons of um uh companies that we keep talking to that have the pressure from their competitors because they're all building agents. We should be building agents that are entirely autonomous and we
I did end up working with a few customers where we built these end to end agents and

Because you start off at a place where you don't know how the user might interact with your system and what kind of responses or actions the AI might come up with. It's really hard to fix problems when you have this really huge workflow, which is taking four or five steps, making tons of decisions. You're
You just you just end up debugging so much and then kind of hot fixing uh to the point where at it at a time we were building for a customer support um use case, which is what which is the example that we give in the newsletter as well. And we're to shut down the product because
We were doing so many hot fixes and there was no way we could um count all the emerging or uh emerging problems that were coming up, right? And
There's also quite some news online. Um, recently I think Air Canada had this thing where um one of their agents predicted
or hallucinated a policy um for a refund, which was not part of their original playbook and they had to go buy it because legal stuff. And there have been a ton of really uh scary incidents. And that's where the idea comes from, right? How can you build so that um you don't lose
customer trust, and you don't end up, or your agent or um AI system doesn't end up making decisions that are super dangerous to the company itself. At the same time build a flywheel so that you can improve your product as you go, right? And that's where we came up with this idea of continuous calibration, continuous development.
The idea is pretty simple, which is um we have this right side of the loop, which is continuous development.
uh sk
Capability and Cure A data essentially get a data set of what your expected inputs are and what your expected outputs should be looking at.
This is a very good exercise before you start building any AI product because many times you figure out that
a lot of the folks within the team are just not aligned on how the product should behave. And that's where your PMs can really give in a lot more information and your subject matter experts as well. So you have this data set that you know um your AI product should be doing really well on. It's it's not comprehensive, but it lets you get started.
And then you set up the application and then design the right kind of evaluation metrics.
And I intentionally use the term evaluation metrics, although we say evals because I just want to be very specific in what it is, because evaluation is a process. Evaluation metrics are dimensions that you want to focus on during the process, right? And then you go about deploying um run your evaluation metrics.
And the second part is the continuous calibration, which is
The part where you understand
What
Um
behavior you hadn't expected in the beginning, right? Because when you start
the development process, you have this data set that you're optimizing for, but more often than not, you realize that that data set is not comprehensive enough because users start behaving with your systems in ways that you did not predict. And that's where you want to do the calibration piece, right?
I've deployed my system. Now I see that there are patterns that I did not really expect, and your evaluation metrics should
give you some insight into that into those patterns, but sometimes you figure out that those metrics were also not enough. And you probably have new error patterns that you've not thought about. And that's where you analyze your behavior, spot error patterns.
You apply fixes for issues that you see, but you also design newer evaluation metrics to figure out that they are emerging patterns. And
That doesn't mean you should always
design evaluation metrics, there are some errors that you can just fix and not really come back to, uh, because they're very spot errors. For instance, there's a
there's a uh tool calling error just because your tool wasn't defined well and stuff like that. You can just fix it and move on, right? And this is pretty much how an AI product lifecycle would look like. But what we specifically also mention is while you're going through these iterations,
Try to think of lower
Agency iterations in the beginning and higher control iterations. What that means is.
constrain the number of decisions your AI systems can make and
um make sure that there are humans in the loop and then increase that over time because you're kind of building a flywheel of behavior and uh you're understanding what kind of use cases are coming in or how your users are using the system, right? And one example I think we give in the newsletter itself is um the customer support. This is a nice
Image that kind of shows how you can think of agency and control as two dimensions, and each of your versions keep on increasing the agency or the ability of your AI system to make decisions and lower the control as you go. And one example that we give is that of the customer support agent.
where you can break it down into three versions. The first version is just routing, which is is your agent able to classify and route a particular ticket to the right department?
And sometimes when you read this, you probably think, Is it so hard to just do routing? Why can't an agent easily do that? And
When you go to enterprises, routing itself can be a super complex problem.
Any retail company, any popular retail company that you can think of has hierarchical taxonomies.
most the times the taxonomies are incredibly messy. I have
worked in, you know, use cases where you probably have
taxonomy that says um
you know, some tax um some kind of hierarchy and then that shows
And then women's shoes and men's shoes all at the same layer where
idea you should be having shoes and then women's shoes and men's shoes should be sub uh you know classes, right? And then you're like, okay, fine, I could just merge that and you go further and you see that there's also another section on the shoes that says for women and for men. And it's just not aggregated. It's not uh fixed for some reason.
So if an agent kind of sees this kind of a taxonomy, what is it supposed to do? Where is it supposed to route? And a lot of the times we are not aware of these problems until you actually go about building something and understanding it, right? So um and when
These kind of problems, um or real human agents see these kind of problems, they know what to check next. Maybe they realize that.
the the node that says for women and for men that's under shoes was last updated in 2019, which means that it's just a dead node that's lying there and not being used. So they kind of know that, okay, we're supposed to be looking at a different node and stuff like that.
And I'm not saying agents cannot understand this or models are not capable enough to understand this, but there are really weird roles within enterprises that are not documented anywhere, and you want to
um make sure that the agents have all of that context instead of just throwing the problem at them, right?
Um, yeah, uh coming back to the versions we had, routing was one where you have really high control, because even if your agent routes to the wrong department, humans can take control and, you know, undo uh those actions.
Um, and along the way, you also figure out that you probably are dealing with a ton of data issues that you need to fix and, you know, um uh make sure that your data layer is good enough for the agent to function.
Uh, we do is what we said of a co-pilot, which is now that you've figured out routing works fine after a few iterations and you've fixed all of your data issues, you could go to the next step, which is
Can my agent provide suggestions based on some standard operating procedures that we have for the customer support agent, right? And it could just generate a draft that the human can make changes to. And when you do this, you're also logging human behavior, which means that how much of this draft was used by
the customer support agent or what was omitted. So you're actually getting error analysis for free when you do this because you're literally logging everything that the user is doing that you could then build back into your flywheel. And then we say
Suppose that once you've figured out that those drafts look good, and most of the times maybe humans are not making too many changes, they're using these drafts as is.
That's when you wanna go to your
end to end resolution assistant that could, you know, um
draft a resolution that could
uh saw the ticket as well, right? And those are the stages of agency where you start with low agency and then you go up high, right?
Um we also have this really nice table that we put together which is
What do you do at each version?
And what you learn that
can enable you to go to the next step and what information do you get that you can feed into the loop, right? When you're just doing your routing, you have better quality routing data, you also know what kind of prompts you need to be building to improve the routing system. Essentially you're figuring out your structure for context engineering.
and um building that flywheel that you want, right? And
While I go through this, I want to also be very clear that
Two things. One is
When you build with CCCD in mind, it doesn't mean that you've fixed the problem all for once. It's possible that you've probably gone through V3 and you see a new distribution of data that you've never previously imagined.
But um this is just one way to lower your risk, which is you get enough
information about how users behave with your system before going to a point of complete um autonomy.
And the second thing is um
You're also kind of um
building this um
you know implicit logging system uh
a lot of people come and tell us that oh wait, there are evals, right? Why do you need something like this?
The issue with just building a bunch of evaluation metrics and then having
um them in production is evaluation metrics catch only the errors that you're already away already aware of, but there can be a lot of emerging patterns that you understand.
Only after you put things in production, right? So for those emerging patterns, you're kind of creating um you know a low risk.
uh kind of a framework so that you could
understand user behavior and not really be in a position where there are tons of errors and you're trying to fix all of them at once. And
This is not the only way to do it. There are tons of different ways. You want to decide how you constrain your autonomy. It could be based on the number of actions that the agent is taking, which is what we do in this example. It could be based on topic. There's just some um domains where it's uh pretty high risk to make
a system completely autonomous for certain decisions, but for some other topics, it's okay to make them completely autonomous and depending on the complexity of the problem. And that's where you really want your product managers, your engineers and subject matter experts to align on how to build the system and continuously improve it.
The idea is just behavior calibration and not losing user trust as you do that behavior calibration, I guess. We'll link folks to this actual post if they want to go really deep. You basically go through all of these steps by step, a bunch of examples.
And the idea here is
As you said, that like the reason
Everything about what you're describing here is
about making it uh continuous and iterative and
kind of moving along this progression of
higher autonomy, less control.
And this idea of even calling continuous calibration, continuous development.
is communicating it's this kind of iterative process.
And just to be clear, this
This naming is kind of a
uh owed to uh CI
C I C D. Continuous integration, continuous deployment.
Sweet.
And the idea here is like that this is the version of that for AI.
Where instead of just like integrating to unit test and deploying constantly, it's
Uh running evaluation, looking at results, iterating on
on the metrics you're watching.
Figuring out where it's breaking and iterating on that.
Awesome.
Okay. So again, we'll point people to this post if they want to go deeper.
That was a a great overview. Is there anything else before we go into a different topic around this framework specifically that you think is important for people to know?
I think one of the most common questions we get is
How do I know if I need to go to the next stage or
If this is calibrated enough, right?
There's not really a rule book you can follow, but it's all about minimizing surprise, which means let's say you're calibrating every one or two days.
um, and you figure out that you're not seeing new data distribution patterns, your users have been pretty consistent with how they're behaving with the system, then the amount of information you gain is
Kind of very low, and that's when you know you can actually go to the next um stage, right? And it's all about the wipes at that point. Like do you know you're ready?
Um you're not receiving any new information.
But
Also it really helps to understand that
sometimes there are events that could completely uh
Mm.
you know, mess up the calibration of your system.
An example is um
GPD four or doesn't exist anymore, or it's going to be deprecated in APIs as well. So most companies that were using four or should switch to five, and five has very different properties. So that's where your calibration's off again. You want to go back and do this process again. Sometimes users start
Users start behaving with systems also differently over time, but user behavior evolves.
Even with consumer products, right? You don't talk to chat GPT the same way
you were talking, say, two years ago, just because you know the capabilities have increased so much. And
And also just people get excited when um, you know, these systems can solve
one task, they want to try it out on other tasks as well.
Uh, we built this system um for underwriters at some point, right? Uh underwriting is a painful task. There are agreements that are like
you know, uh you know, loan uh applications that are like thirty or forty pages. And the idea for this bank was to build a system that could help underwriters
pick policies and, you know, um information about the bank so that they could approve loans, right?
And
For a good three or four months, everybody was pretty impressed with the system. We had underwriters actually report gains in terms of how much time they were spending, et cetera.
And post three months, we realized that they were so excited with the product that they started asking very deep questions that we never anticipated. They would just throw the entire application document at the system and go, like, for a case that looks like this, what did previous underwriters do?
And
For a user, that just seems like a natural extension of what they were doing, but
the building behind it should significantly change. Now you need to understand what does for a case like this mean.
in the context of the loan itself. Is it referring to people of a particular, you know, income range, or is it referring to people in a particular GO and stuff like that? And then you need to pick up historical documents, analyze those documents, and then tell them, okay, this is what it looks like versus just saying that there's a policy X, Y, and Z and
you want to um you know look up that policy.
Um, so something that might seem very natural to a end user might be very hard to build as a product builder, and you see that user behavior also evolves over time, and that's when you know you you know that you want to go back and recalibrate.
What do you think is uh overhyped?
in the AI space right now, and even more importantly, what do you think is has underhyed?
I um
As I said, like super optimistic in different things that are going in AI. So I wouldn't say over five, but I feel kind of misunderstood.
Is the concept of multi agents.
Uh people have this notion of like uh I have this incredibly complex problem.
Now I'm going to break it down into hey, you're this agent, take care of this. You're this agent, take care of this. And now if I somehow connect all of these agents, they think they're the agent utopia. And it's never the case that
There are incredibly successful multi agent systems that are built, right? Like there's no doubt about that. But I feel a lot of it comes in terms of how are you limiting the
Uh
ways in which the system can go off tracks. And uh for example, like if you're building a supervisor agent and there are like sub agents that actually do the work for the superagent, supervisor agent.
That is a very uh successful pattern. But coming with this notion of I'm going to
divide the responsibilities based on functionality and somehow ex uh expect all of that to work together in
some sort of like gossip protocol. Uh that is like extremely uh
I misunderstood that you could do that. I don't think like
current uh ways of building and current like uh model capabilities are like right there in terms of like uh building those kind of applications. I feel that is kind of
Misunderstood than overrated.
Uh underrated I feel
It's hard to probably believe, but I still feel coding agents are underrated.
In the sense that I feel like you can go on Twitter and you can go on Reddit and you see a lot of chatter about coding agents, but talking to an engineer in like in any random company, uh, especially outside of Bay Area, you
you can see like the amount of impact this coding agents can create and the penetration uh is very low. So I feel like twenty twenty-five.
Uh and twenty twenty six is going to be like an incredible year for
optimizing all of these processes and I feel that is going to be creating a lot of value with AI.
That's really interesting on that first point. So the idea there is
Uh
you'll probably m be more successful building and using uh an agent.
That is able to do its own sub agent.
splitting of work versus like a bunch of
Say codex agents will you do this task, you do that task.
You can have agents to do these things and you as a human can orchestrate it, or you can have like one uh larger agent that is going to orchestrate all of these things. But letting the agents communicate in terms of peer-to-peer kind of protocol, and then especially uh doing this in a customer support kind of use case is incredibly hard to
control what kind of agent is replying to your customer. Because you need to shift your guardrails everywhere and things like that.
Yeah, okay. Uh great picks. Okay, Ash, what do you got?
Can I say evals? Will I be canceled? Which which bucket do they go? Overrated. Overrated. Okay, go go for it. You won't we won't let you get canceled. Uh just kidding. I think evals are misunderstood. They are important folks. I'm not saying they're not important. But I think just um
This um
I'm gonna keep
Um
chomping across tools and going to pick up and learn a new tool is overrated.
I I
still in old school and feel like you would need really need to be obsessed with the business problem you're trying to solve. AI is only a tool. I try to think of it that way.
of course you need to be learning about the latest and gre greatest, but don't be so obsessed with just building so quickly. Building is really cheap today. Um design is more expensive. Really thinking about your product, what you're gonna build, is it gonna really solve a pain point is
is what is
way more valuable today. And it will only become uh more true in the near future, right? So
really obsessing about your problem and design is underrated and just rote building is overrated, I guess.
Awesome.
Okay.
Uh similar sort of question.
From a a product.
Point of view.
What do you think the next year of AI is gonna look like?
Give us a vision of where you think things are gonna go by say by the end of twenty twenty six.
Yeah, I feel uh there's a lot of promise in terms of uh
This background agents are proactive agents.
who is like they are going to like basically understand your workflow even more.
Uh if you think if you think of like where is AI
failing to create value today. It's mainly about not understanding the context. And the reason that it's not understanding the context is it's not plugged into the right places where the actual work is happening, right? And as you do more of this,
you can give the agent mode of context and then it start to see the world around you and understand what is the
What are the set of metrics that you're optimizing for, or what are the kind of activities that you're trying to do? It is a very easy extension from there to actually gain more out of it and then let the agent prompt you back. Uh we already do this in terms of chat GPT pulse, which kind of gives you this daily update of things you might care about. And it's
It's very nice to actually have that like jog your brain up in terms of, oh, this is something that I haven't thought about. Maybe this is good. And now when you extend this to more complex tasks.
like a coding agent which says that like okay, I have fixed five of your linear tickets and here are the patches, just review them at the start of your day. So I feel that is going to be like extremely useful and I
See that as like a strong direction in which like products are going to build in twenty twenty six.
That's so cool. So essentially agents kind of anticipating
what you want to do and
getting going getting ahead of you and here's I've solved these problems for you.
Or I think this is gonna crash your site, maybe you should fix this thing right here.
Or I see the spike here and let's refactor a database.
Amazing.
What a world.
Okay.
Ash, what do you got?
I'm all in for multi modal experiences in twenty twenty six, I think
We have done quite some progress in twenty twenty five and
Um not just in terms of generation, but also understanding. Um
Until now I think LMs have been our most commonly used models, but
As humans we are
multimodal creatures, I would say. Like, um language is probably one of our last forms of evolution. As the three of us are talking, I think we're constantly getting so many signals. I'm like
Oh, Lenny's nodding his head, so probably I would go in this direction or
any sport, so let me start uh stop talking. So there's a
Chao
behind your chain of thought and you're constantly altering it with language that
dimension of expression is not explored as well. So if you we could build better multi-modal experiences, that would get us closer to human like um conversation richness.
And um yeah, I think um and just you will also just given the kind of models, there's a bunch of boring tasks as well, which are ripe for you if multimodal understanding gets better. There are so many handwritten documents and really messy uh PDFs that
cannot be passed even by the best of the models as of today.
And if it's possible there's
There'd be so much um um data that we can tap into.
Awesome. I just saw Demists from DeepMind, AI, Google, whatever they they call the whole org
Uh, talking about this where he's thinks that's gonna be a big part of where they're going, combining
The image model work.
the
L L M and also their world model stuff, Genie, I think is what it's called.
So that's gonna be a wild, wild time.
Okay.
Uh last question.
If someone wants to just get better at building AI products
What's just maybe one skill or maybe two skills that you think
They should lean into
develop.
I think we did cover a bunch of best practices for AI products, which is start small, try to
Catcher iteration
going well and build a flywheel and all of that.
But again, if you kind of look at it at at a ten thousand feet level for anybody building today, like I was saying, implementation is going to be
ridiculously cheap in the next few years. So really
Nail down your design.
your judgment, your taste and all of that.
Um
And in general, if you're building a career as well, I feel
For the past.

Your your former years say the first
Two three hours of
uh building your career is always focused on execution mechanics and all of that. And now we have AI that could help you ramp pretty quickly and post that
I mean, after a few years, I think everybody j everybody's job becomes about your taste, your judgment and kind of um uh, you know.
Uh.
What is uniquely you
I think nail down on that part and try to figure out how you can bring in that kind of a perspective.
Um, it doesn't have to mean that
You should be significantly old of have X um years of experience. We recently s hired someone and
We use this very popular app uh for tracking our tasks, right? And we've been using it for years and we pay a high subscription fee for it.
And this guy just came with his own white coded app to the meeting. He onboarded us to all of it, and he's like, okay, let's start using this.
And I think that kind of agency and that kind of ownership to really rethink experiences is what uh will set people apart. And I'm not being blind to the fact that white coded apps have high maintenance costs and maybe as we scale as a company, we have to replace it or we have to think of better approaches. But
given that we're a small size company now and just
I I was really shocked because I never thought of it. Um, if you've been used to working in a certain way, you associate a cost with building, and I feel like
folks who grew up in this age uh have a much lower cost associated in their mind. They just don't mind building something and going ahead with it. And that's they're also very um
enthusiastic to try out new tools. Um, that's also probably why AI products have this retention problem, because everybody's so excited about trying out these new tools and all of that. But essentially um having the agency and ownership and
I think it's also the a going to be the end of
The
busy work error, right? You can't be sitting in a corner doing something that doesn't move the needle for a company.
you really need to be thinking about, you know, end to end workflows, how you can bring in more impact. I think all of that'd be super important.
Mm. That reminds me I just had Jason Lemkitt on the podcast. He's um
Uh, very smart on sales, go to market or on disaster and
He replaced his whole sales team with agents. He had ten salespeople now he has one point two and twenty agents.
And
one of the agents it was just tracking everyone's updates to Salesforce and kind of
Uh
Updating it automatically for them based on their calls.
And one of the sales people
I was like, Okay, I'm I I quit.
And it turned out he wasn't really doing anything.
He was just sitting around.
And he's like, Okay, this will catch me. I gotta get out of here.
Yeah. So to your point about you can't it'll be harder to sit around and twiddle your thumbs, uh
Think is really right.
Yeah, I think to add on to that, I feel like persistence is also something that
is extremely valuable, especially given that anybody who wants to build something is the information is like at your fingertips even more than like the past decade, right? You can learn anything overnight and become that sort of like Iron Man kind of approach. So if you like
having that persistence and like going through the pain of like
learning this, implementing this and like understanding what works and what doesn't work. And as you're going through this like pain of like
developing multiple approaches and then solving the problem. I feel that is like going to be the real mode as an individual. Like I I I like to call it like pain is the new moat, but uh I feel that is ex exactly super useful to actually have this
in especially in like you know you're building these AI products.
Say more about this. I love this concept. Pain is the new moat.
Is there more there? Yeah, I feel as a company, I mean like successful companies right now building in any new area, they are successful not because they're first to the market or like they have this fancy feature that more customers are liking it. They went through the pain of understanding what are the set of non negotiable things.
And uh
trade them off exactly with like what are the features or like what are the model capabilities that they can use to solve that problem.
It it this is not a straightforward process, right? There's no textbook to do this, or like there's no straightforward way or like a known credit path to be here. So
A lot of this pain I was talking about is just like going through this
iteration of like, okay, let's try this. And if this doesn't work, let's try this. And that kind of knowledge that you built across the organization or across like your own experience lived experiences. I feel that
the that pain is what uh translates into the mood of the company, right? This could be like a product of evaluation that you build, and I feel that is the going to be the
Game changer.
That is awesome.
Coal into diamond.
Yes.
Okay.
Uh, I feel like we've done a great job helping people
Avoid some of the
Biggest issues.
People
consistently run into building AI products. We covered so many of the
pitfalls and the ways to actually do it correctly.
Before we get to our very exciting lightning round, is there anything else that you wanted to share? Anything else you want to leave listeners with?
Be obsessed with your customers, be obsessed with the problem. Um
Yeah, it's just to do that.
And um try to make sure that you're really understanding your workflows.
Eighty percent of so called AI engineers, AIPMs spend their time actually understanding their workflows very well. They're not building the fanciest and the
You know, most
Mm.
cool models or um workflows around it, they're actually in the weeds understanding their customers' behavior and data.
Um and
When software інженер
Who's never done AI before here's the term look at your data. I think it's a huge revelation to them. But
It's always been the case. You need to go there, look at your data, understand your users, and that's
going to be a huge differentiator.
It's a great way to close it. It's not the AI isn't the answer. It's
It's a tool to solve the problem.
With that, we have reached our very exciting lightning round. I've got five questions for both of you. Are you ready?
Yay, yes.
All right. So you can both answer'em, you can pick one which you want to answer, either way, up to you.
What are two or three books you find yourself recommending most to other people?
Uh, for me it's this book called When Breath Becomes Air, Lenny. Yeah, it was written by Paul Kalanity. I think he was um
Yeah, any Indian origin neurosurgeon who is diagnosed with lung cancer is.
And the whole book is his memoir.
and just is written after he was diagnosed.
And it's
It's really beautiful, especially because I read it during COVID, and all we ever wanted to do during COVID is stay alive. Um
There are bunch of really nice courts.
within the book as well, but I remember one of them, he was kind of arguing against a very popular quote by Socrates, which is
The unexamined life.
is not worth living or something like that. And
Which means you really need to be thinking about your choices, you need to, you know, understand your values, your mission, and all of that.
And um Paul says, if the unexamined life is not worth living, was the unlived life worth examining? Which means are you spending so much time just understanding your mission and purpose that you've forgotten to live and
I think it
Uh, everybody who's uh staying in the AI era and building and continuously going through this phase of reinventing themselves need to take a pause and.
Live for a bit, I guess. They need to stop evaling life too much. I was gonna say that. That's where my mind went. Generate some evals for your life.
Oh my God. We've gone too far.
Yep. Yeah. Yeah. That's my favorite book.
I I I like more of science fiction books, so I uh
Really like this three body problem series.
Uh it's like a three book series. It's
It's like it has elements of like grandet in science fiction, uh life.
outside earth and how it impacts like human decision making process. And it also has like elements of geopolitics and how
How much important or like valuable abstract sciences to human progress? And then that gets when that gets stopped, it's
It's not noticeable in everyday life, but it
It can ca cause like devastating effects. So I feel like AI helping in these areas, for example, is going to be like extremely crucial. And that book is like a nice example of what could happen otherwise. Completely agree. Absolutely. Love might be my favorite sci fi book, except
Or series even, and it's three. I have to read'em all three, by the way. I find that
It only got really good about one and a half books in. So if anyone's tried it and like what the heck is going on here.
Just keep reading and get to the middle of the second one and then it gets mind blowing.
Yes. Uh
If you love Sofai and you're in AI, you gotta read this book called A Fire Upon the Deep.
By
Uh Vernon Vinge.
Mm-hmm.
Check it out.
It's
Incredible.
Uh I saw Noah Smith on his newsletter recommend this book.
And there's like a whole there's like sequels to it, but this is the one that's
So incredible. And it's actually turns out it's about AGI and super intelligence and all these things. And it's just like so epic.
And no one's heard of it.
There you go. I'm giving you one back. Okay, next question.
What's a favorite recent movie or T V show that you've really enjoyed?
I started re watching Silicon Valley and
I think it's so true, it's so timeless. Everything is repeating all over again. Anybody who's watched it a few years ago should start re watching it and you'll
see that it's early similar to everything that's happening right now with the AI wave.
That's that's a good idea to rewatch it. I love that their whole business was like an algorithm to compress. Like a compression algorithm? It's like maybe a precursor to LMs in some small way.
No we got.
All right, Grity, what you got?
Uh I'm gonna digress and say lot a movie or a T V show, but there's this game I picked up recently called Expedition Thirty Three. Uh it has nothing to do with AI, but it's an incredibly, incredibly well made game in terms of
the gameplay or like the movie and the story and the music. Uh it's it's been amazing. I love that you have time to play games. That's a great sign. I love that.
I'm just imagining your
There's nothing else going on except just coding and
Yeah, it has been incredibly hard to I find time for that.
That's good. That's a good sign. I'm happy to hear this. Okay. What's a favorite product that you've recently discovered that you really love?
For me it's Whisperflow. I think I've been using it quite a bit and I didn't know
I needed it so much. Um
The best part is it's a conceptual
transcription tool, which means if you go to you know codex and start using Wispflow, it starts identifying variables and all of that.
And
It's so seamless in terms of transcription to instruction. You could say something like I'm so excited today, add three exclamation marks and it
seamlessly switches. It adds those three exclamation marks instead of, you know, writing add three exclamation marks. And I think it's pretty cool. Um if you're not using it, you should try it.
I'll do a plug, get Whisper Flow for free for an entire year.
Yeah.
For a year for free.
By becoming an annual subscriber of my newsletter.
That's how I got access to it, Lenny.
Like I think I I pitched this deal. I think people don't truly understand how incredible this is. They're like no way, this is real. It's real. And eighteen other products.
Ladiesproduct past dot com, check it out.
Moving up.
Awesome. Uh I actually am a stickler for productivity. I keep experimenting new CLI tools and like things which can uh make me faster.
Uh so I feel like a recast is been amazing. Uh I've discovered all this like new shortcuts that you can use to open different things, type in shortcut commands and things like that. And caffeinate is another thing that I've recently discovered from my teammates. It helps you like
Prevent Mac from sleeping. So you can run this really long codex task for like four or five hours locally. Let it build the thing, and then you can wake up and be like, Okay, this is good. I like this.
That's hilarious, that combo. Codex and caffeinate.
You guys you guys need to use the like build that it yourself, an open air version of that. Or the codex agent should just keep your Mac from sleeping. That's so funny. Uh by the way, Raycast, also part of Lenny's product past. One year free of Raycast.
Yeah.
We Link tell us these folks. These are actually our favorite. These are just two of nineteen products. No caffeinate, though. I don't know if that's even paid.
Okay, let's keep going.
Do you have a favorite life motto that you find yourself coming back to?
In work or in life.
For me, I think this is what my dad told me when I was a kid and it's always stuck, which is um
Um, they told it couldn't be done, but the fool didn't know it, so he did it anyway. I think be foolish enough to believe that you can do anything if you
put your heart to it. Especially now because
you have so much data at your hand that could be pointing towards the fact that you probably will be unsuccessful. How many podcasts made it to more than a thousand subscribers, or how many companies it mo hit more than one million and there's always
data to show you that you won't be successful, but sometimes just be foolish and go ahead with it.
That's great.
Yeah, for me, I uh am more of an overthinker. So I really like this quote from Steve Jobs that you can only connect the dots looking backwards.
So it's a lot of the times there are like numerous choices and you don't really know the optimal one to pick, but life's life works in ways that you can actually see back and be like, Oh, these are actually beautiful in terms of how I I would transition. So I feel like that is extremely useful in like, you know, keep moving forward, keep experimenting.
Final question.
Whenever I have two guests on the podcast at once, I like to ask this question.
What's something that you admire about the other person?
I think with Kitty um
It's about
He's
He's pretty calm and uh very grounded.
Um and he's always been my sounding board. I can throw a ton of ideas at him and he always comes up with
he he's able to anticipate the kind of issues that might um run into and he's
Extremely um
Kind and
lets his work speak instead of actually doing a lot of talking, I guess.
Well if I had to pick one, I think uh he's the most incredible husband.
Reveal little people know.
Yeah.
Yeah. We've been married for four years and been the most beautiful four years of my life.
Wow.
Okay. How do you follow that?
Yeah, it's super hard to follow that. I would say I
I'm extremely privileged in terms of working with like really smart people in great companies in the Silicon Valley. And I feel the unique thing that stands with Ashwarya.
across like any other uh smart folks I've worked on is like
She has this really amazing knack of teaching.
And like explaining something uh in a very understandable and easy to comprehend way. And that combined with persistence is like super useful, especially in this uh
fast moving AI world that we are in in the sense that there's so many new things coming up. It feels overwhelming, but when I hear her talk about like this is how you make sense of this entire thing. This is where it plugs in. I feel like oh that is so simple. Like I can also do that. So she empowers a lot of people by simplifying things and you know like
uh explaining things in the most understandable way. So I feel that is like an incredible quality.
How amazing. How sweet. I gotta do this all the time. I need more more yes yos. That was that was great.
Okay. Uh final questions working, folks. Find stuff that you're working on, find your online, talk about your share your course link.
And then just how can listeners be useful to you?
I write a lot on LinkedIn. Um
Um, so if you if you want to listen to pragmatists who've been in the weeds working on AI products and um what they're seeing.
you can uh follow my work. We also have a GitHub repository with about twenty K stars, and that repository is all about
good resources for learning AI. It's completely free.
And if you
um like what we spoke today, we also run a super popular course. We leave a link to it on building enterprise AI products. And the course is a lot about unlearning mindsets and
following like a problem first approach, uh instead of a tool first or a hype first approach.
Um, so you can check that out as well. And if you don't want to do the course, we write a lot, we give out a lot of free resources, we have free sessions, so make sure you follow our work.
Yeah, I would also add that I you can also find me on LinkedIn. Uh I don't like write a lot, I guess, but I'm super all excited to just talk to any complex product that you're building. And if you have thoughts on like how you can uh
Use coding agents to make your life better, or how what are the problems that you're seeing? Um always my DMs are open and like we can have a great discussion.
Awesome.
Well, Kiritse and Ash, thank you so much for being here.
Thank you so much. This was so much fun. So much fun.
By everyone.
Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app.
Also, please consider giving us a rating or leaving a review, as that really helps other listeners find the podcast.
You can find all past episodes or learn more about the show.
at Lenny's podcast dot com.
See you in the next episode.