Scott Wu - Building Cognition - [Invest Like the Best, EP.402]

0:00 I know firsthand how complex the tech stack is for asset management firms. And seemingly every new tool and data source makes the problem even worse, adding more complexity, more headcount, and more risk. Ridge line offers a better way forward, one unified platform that automates away the complexity across portfolio accounting. Reconciliation, reporting, trading, compliance, and more, all at scale. Ridge line is revolutionizing investment management, helping ambitious firms scale faster.

0:25 Operate smarter and stay ahead of the curve. See what Ridgeline can unlock for your firm. Schedule a demo at ridgeline.ai. Hello and welcome everyone. I'm Patrick O'Shaughnessy and this is Invest Like the Best. This show is an open ended exploration of markets, ideas, stories, and strategies that will help you better invest both your time and your money. Invest Like the Best is part of the Colossus family of podcasts, and you can access all our podcasts, including edited transcripts, show notes, and other resources to keep learning at joincolosis.com.

1:00 Patrick O'Shaughnessy is the CEO of Positive Sum. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of Positive Some. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of Positive Sum may maintain positions in the securities discussed in this podcast. To learn more, visit psum.vc.

1:29 My guest today is Scott Wu. Scott is the co-founder and CEO of Cognition, which is an applied AI lab that has created the first AI software engineer, which they call Devon. In just a year since founding cognition, Devin functions at the level of a junior software engineer. Capable of handling complete engineering workflows from bug fixing to submitting pull requests. Scott is a former competitive programming champion. And he describes the engineering field as simply the art of telling a computer what you want it to do.

1:56 Scott predicts AI will surpass the world's best competitive programmer within one to two years. And sees this technology not as replacing programmers, but as democratizing software creation. We discussed the bottleneck in software development, the future of AI in various industries, and the challenges of leveling up Devon. The implication of Scott's work and Devin itself is Is enormous.

2:17 Please enjoy my great conversation with Scott Will. Scott, since not everyone listening will know cognition intimately, can you just start by describing what the business is and does? It's quite young. It's made a lot of progress very, very quickly, which is gonna be a topic of our conversation in general. But maybe just start by describing for those listening What cognition does.

2:39 Yeah, absolutely. And thank you so much for having me. We got started actually just around one year ago. And we are focused on building AI to Automate software engineering. The product we're building is called Devon.

2:50 And Devin is the first fully autonomous software engineer. And what that means is that Devin is Basically able to do all of the parts of the software flow. And so you work with Devin the same way that you would work with the coworker. And Devin is in your Slack, Devin is in your GitHub, uses all of your tools and works in your local developer environment. And so you can just tag Devin, you can say, Hey, uh, Devin.

3:12 I've got this thug that's going on in the front end and you figure out what's going on. Devin will do not just the parts of writing the code, but actually all of the parts around it as well, which includes getting the local environment running, reproducing the bug itself, taking a look at the logs, figuring out what's going on, testing the bug, adding in some unit tests, and then making a pull request for you as the engineer to go review. What's the right way to think about If I think about Devin as an actual engineer its level of sophistication.

3:39 Maybe you could put it in like salary terms or something or years of experience terms or some other thing like help manage my expectations for how much I could have this thing do. Without Any human involved.

3:52 It's very much a junior engineer today. That's how we think but that's how we describe it. I would say right around when we got started high school C S student or something like that is probably fair. maybe six months ago, an intern, and I think today it's very much uh entry level junior engineer. There's gonna be a lot of tasks. I mean, for example, if if you wanted to say like

4:12 All right, I need you to re architect my entire code base, rewrite the whole thing, make it all ten X more efficient, and then I'll check in in like a week. That's probably not the right way to use Devin. Whereas actually a lot of it, it really does function. Turning every engineer into an engineering manager where you get to work with your own team of Devons and hand off tasks. And so a lot of these little things which you would want to delegate and spread off, you would say, hey, Devin, take a look at this, take a first pass.

4:36 maybe you have to fix a few things or maybe Devin asks a few questions that you have to answer or things like that. But the whole idea is that you're working asynchronously with your team of Devons while you're doing your own stuff or doing things separately. Should I think of Devin as a singular or a plural? The way that it's all set up is We bill by usage. So we have a plan. The plan is five hundred bucks a month for engineering teams by default. And then obviously as you use more, then we build more. And the whole idea with it, obviously, is that you can have your own army.

5:05 of Devons and each Devon is its own thing. And let's say you have five different bugs or issues or something that you're trying to work on today. A lot of our users do this where you just spin up five different devants. And each one is tackling a different one. And obviously they ask you as they have questions or put up code for you to review or things like that, but now you're actually jumping between your different Devons and tackling these tasks in parallel. It seems like

5:27 GPTs are GPTs, general purpose technologies. And one of the cool things about general purpose technologies is it's very hard to predict How the world will use them to build other things. Now that you've had Devin live for a while in private and now generally available. What has surprised you most about the ways in which people use it to do things?

5:47 A lot of the emergent use cases Are things that we wouldn't have predicted ourselves. One of the first things that companies that we work with have been Using Devin Four actually is a lot of these Migrations, replatforms. modernization and version upgrades, things like that.

6:02 For example, we work with bangs, we work with healthcare companies and many of these folks have Code that is Literally decades old. And they would love to be on the latest stack, on the latest technologies, make things faster, make things more efficient, it would make things more secure, it would make things easier for their developers to work on as well. But it's just such a pain as you can imagine if your code base is 30,000 files.

6:24 to actually go through and do this kind of modernization effort. And so it's the kind of thing that's very tedious. And it requires a little bit of active thought and effort to get into, but it's obviously there's just so much volume of it. And that was one of the first things where we really, really saw folks using Devin. a lot of our customers did their own internal studies of the usage and they were finding that every hour of an engineering's time. Using Devin was Equivalent to eight to twelve hours actually without

6:48 Devin. So what that looks like is you're just asking questions and reviewing the code rather than having to write the code yourself. And so some of these usage patterns where a lot of the kind of more routine tasks that needed to get done. Devin was just really able to do in one fell swoop.

7:02 Can you explain the key differences between those innovating in the IDE like a cursor or something that just make it much easier for a programmer to program. versus something like Devin where it feels more holistic, like I'm handing off a whole thing and coming back and getting a complete piece of work. We've been doing this call it this agent tech approach, like a full automation, ever since we got started. And the way that I describe it, I think the biggest Step function change is about

7:30 Asynchronous versus synchronous. So a lot of these code assistant tools to call that, which you mentioned a lot of great examples, really great tools, honestly. What happens obviously is you use language models to do an autocomplete or you have simple things where it writes certain files or functions for you or things like that. And if it writes the right things, you hit tab and confirm those or Confirm the suggestion that gets given. Saves you that time of typing. I think in general, what folks have seen with those kinds of technologies is they see something like a 10 to 20 improvement across the board. So

8:02 all of the different software work that they're doing, it makes them a ten to twenty percent faster engineer. Basically. By the way, engineering's huge. That's not a small number at all. But with Devin, what we kind of see is There is a certain level of task, obviously, that Devin is capable of taking on.

8:18 You're not gonna ask it to do the entire re architecting or something like that. But for the tasks that Devin can do. It's more like a ten X rather than a ten percent because it really is you just handing off The task to Devon. And you go ahead and do your own stuff, you work on other things, maybe you're running other Devons as well. And in the meantime, Devin is really just a co worker that you give in the task and works on that and gives you a full review and a and a full piece of code.

8:40 There's a lot of meaningful differences in the product experience that all stem from that. So Devon has to have the ability to test its own code to run things live to go and look up documentation to be able to like pull up websites and stuff by itself. you have to be able to talk to it in your GitHub or your Slack or your linear or whatever technologies you're using as a team to communicate with each other. It's quite a different product experience. And honestly, we've seen from folks that it's complimentary. We use the assistant suite of tools as well, and then we also use Devin.

9:08 A lot of folks use both. Because of where you sit in your own personal experience, I'd love to use the opportunity of chatting with you today to really talk about the past, present, and future of software engineering. I would love to start with the past. Maybe, you know, in my mind it's this growing abstraction layer where it gets simpler and simpler and you have to know less and less things down close to the bare metal itself of semiconductors to Do what you want to do in a computer system.

9:33 Maybe you can describe your perspective on the history so far to this point. of computer programming and like the major milestones that we've reached before we talk about Today and then the future. Well, I love the way that you actually just described it, which is I think software engineering, this whole time, programming or whatever we want to call this, has always been basically just the art of.

9:52 Telling the computer what you want it to do. That has been the goal. It is really that simple. And it's kind of funny, obviously, because the form factor of that has changed so much over time. And obviously like a long, long time ago, it was all about punch cards. Even before that was about these gas tubes or things like that. That was programming. And then there was assembly and assembly, that was programming. There is basic and there is C and These other languages and frameworks and technologies

10:15 We had cloud, we had mobile. A lot of big broad shifts. In the technology. I think the undercurrent that's going through this all is over the last Fifty to a hundred years.

10:26 We have been able to automate. So much of the processes that we need in our economy. And Automating that is essentially it boils down to being able to explain to the computer exactly what you want the computer to do for you. really, really fascinating to study a lot of the history of it because

10:42 I think often it actually was the case that many of these technologies At the time they came out, they were very interesting technology. And then there's a question of how do you make it useful? What is the use case? And then Sometimes it was kinda surprising. I mean, probably the most famous example that folks know about is the ENIAC. back in the forties, which was the first real

10:59 automated computer and the initial use case obviously that really got it there was missile trajectories. ballistics and all the physics that you want to calculate with that. If you want to be very, very accurate and doing ten digits of precision or something, you don't want to be doing that by hand. With punch cards, for example, the first real big thing that really caused that to be valuable was

11:17 The U Census. And that was the thing that needed to be automated. the first Macintosh, for example, like a lot of the initial use case actually was for graphic designers, actually. And then number two was actually the spreadsheet was the killer app that made it useful. With each of these, there was this technology that came about, and then it took some time to really make it useful.

11:34 We're seeing a very similar paradigm here where language models came out and obviously they took the world by storm. Chat GBT in November of 2022, almost exactly two years ago. It was the moment when everyone realized what was possible. And now we're seeing what are all the natural applications of that. And I think one of those obviously is software itself. Accelerating software itself is just this recursive thing where you get such massive continued gains. You I don't know what age you were, I think you were seventeen or something, won the competitive programmage. So literally you're the best programmer alive or at least in some major way?

12:08 At that time and maybe today. What distinguishes You hear about this idea of the ten X or Hundred X engineer. What are the qualities of those people that make them better than others.

12:19 In those competitions. What is the competitive frontier? What are you trying to best your opponents at. in that field. Well, thank you so much for the kind words.

12:28 There's these competitions, which are kind of more like algorithmic problems. There's software. But in all these things, I honestly think the soul of programming the whole way through, even as we're saying, it's changed in form factor so much. I think the soul of it is really just being able to understand deeply understand these Abstractions and just Work at this. abstract level. I think that's true in software today where

12:49 The soul of software engineering is the part of taking a customer's problem or your own product's problem or something like that and really just figuring out Okay, here is exactly the solution that I need to build. Here's how I'm gonna represent it and here's how I'm gonna architect it. Here's all the cases and the details that I need to be thinking about. Here's the flows, maybe that I'm gonna be setting up on my website. Here's the database schema that's going to make this all work and make this efficient, or something like that.

13:12 And the thing that's kind of funny about it, obviously, is That is the part that every engineer loves about software. You know, and it is also I think As you're saying, it's the thing that really distinguishes somebody from being a one X versus a ten X versus a hundred X. At the same time, it's probably only about 10% of what the average engine does, because the other ninety percent of their time obviously is. Debugging, implementing, setting up testing.

13:35 Dealing with their DevOps. figuring out what went down with their database, would that just crash or All these kind of various things which look a lot more like implementation than this creativity, but I would say even despite that, the creativity and the problem solving is actually really what defines programming.

13:50 In those competitions. How did you get better? Maybe just walking through the competitions themselves. How they were structured. what you did to try to win, how you won. I'm really interested in that chapter of your life.

14:01 The competition was called the IOI, the International Olympiad of Informatics. It was very much the Olympics of programming. So every country sends their top contestants. There's gold medals, silver medals, bronze medals. You qualify at your local level and then you make it to the team selection at your country and you qualify. And yeah, I mean, that was my entire life growing up, actually. I'm from Baton Rouge, Louisiana. There are not that many people who are into

14:24 Competitive programming in Baton Rouge, Louisiana. But I just really, really loved it. My older brother actually did it first. And I got super into it through him. And It was the number one focus of my life. For the first seventeen of my life, honestly.

14:36 At the core of it, it's really just problem solving. And you have code as the medium by which you build the solutions and all the things that you're designing around and so on, but the core of it is really whether you can Figure out. these algorithmic problems and understands. The right approaches. I wonder if you could sketch for us what

14:52 Your best guesses as to what will happen in this world in the next five to ten years. Computer programming. writ large. Obviously you're directly trying to affect that trajectory and what the answer is. But you also occupy this cool seat with maybe five or ten others.

15:07 At the very bleeding edge frontier. So not only are you pushing that frontier, but it probably gives you the chance to see a little bit further. As you look out, what do you see? What do you think? Maybe you could draw confidence intervals or something for us.

15:18 Best case, worst case or something. But if you think forward five years. Where might we get in terms of what's possible? with engineering. I mean, for one, you're asking about competitive programming. The best programmer in history is actually a guy named Ganadi Korokhevich. He's got from Belarus.

15:33 And I would imagine that we have AI that can beat Gannati. In the next one or two years. Being the best program in the world at this programming competition. I think that is going to happen. And I think it's going to be like an AlphaGo style moment where it's obviously like holy shit, like this is truly there. The tech today, honestly already, is folks have been putting out things

15:51 Google put out something on the IMO, the math Olympiad, and OpenAI put something out on the IOI, but they're already at a level where they're competitive. With some of the best. competitors in the world. Getting to number one is honestly not too far from that. I think what happens to software engineering as a whole

16:08 is I really come back to this point of what we're saying that All of programming it really is just about telling your computer what to do. a lot of this work that's happening with AI and AI agents as truly just The next generation of human computer interface. And so what I mean by that obviously is

16:22 If you think about a programming language, let's say Python, for example. Perhaps the most popular language out there. Python is great, it's beautiful and everything. But if you really just look at it, it is such a start compromise that you have to be making.

16:34 From both sides. From the human side, obviously you have to learn how to code. Ninety nine percent of people in the world. don't know how to cut it. And it takes years to learn all the little details and here's how you do like a lambda function, and here's how you do this little particular thing. And these are all things you have to learn just to be able to talk to your computer. And from the computer side, Python itself is notoriously actually slow and inefficient and so on.

16:56 even compared to other languages, but obviously compared to How fast you could truly run things on bare metal assembly if you knew this is exactly what I want to do. You can make all of the systems honestly. Ten or a hundred X faster. And the way I think about it is

17:11 Python is maybe the best language out there by a lot of people's definitions. And really it's this awkward compromise that we have to make because this is the only way that humans and computers are really communicating. Yeah. One of the great things I think that AI will really solve is essentially this translation problem where A human, any human.

17:28 doesn't have to be somebody who is trained in code or something can describe precisely what it is that they want to build. And the computer will build the efficient and feature complete version of that that actually just works and does all these things. It's gonna take some real time to get there, obviously. I think that's the future that we're headed towards. Meaning just to use a simple example.

17:47 I using natural language want a very, very specific application to stick on my phone or something. That's maybe within five years we'll be at a point that Me with no particular CS experience can get exactly what I want in a way that works with high fidelity.

18:03 without having to rely on human programmers. Do you think that's feasible? I think that's right. If I had to put down a number right now, it's obviously very hard to predict. I think five to ten years is like the reasonable realm for that. One point I would make on that. even the way that we talk about applications, we kind of assume obviously that there's some application that has to generalize.

18:22 One way to put it maybe Is Software engineering is so expensive today. It is so, so expensive. That the only software that you can actually build That's worth building is

18:33 Basically applications that reach millions of people. It's like okay, it was worth it to code Facebook because of how many people use the exact same software every day. YouTube or something or DoorDash or whatever. Now obviously there are somewhat more niche things, but there's honestly lots and lots of things out there where It would be better served with code. There's communities of

18:52 Call it a couple thousand people. There's little groups or little niches or things like that, there's personalized products even for one person. And so whatever your day to day chores are. They really are better served with code. That code can do all the things that you might need to do and just make all the execution of that much cleaner.

19:08 The reason we don't have that obviously is because There's not really anyone in the world For whom that makes sense to write custom software just for their own personal use case every time they want to do something new. But I think we will get to that point where you are just able to execute these things. And maybe it's code on the back end, but from the perspective of us, it's really, yeah, just Explaining to the computer exactly what we want and seeing that happen.

19:30 There's gonna be a lot of middle steps, but yeah. I'm curious what you think about one layer down or up, depending on how you think about it, which is the same question For agents specifically, and whether or not All of these systems are gonna cause applications themselves to be less common.

19:45 and agents we interact with through voice or through text or through other means. To do more and more of the things that we currently use. applications to do. Since you're building an agentic system. I'd love you to just teach us about agents.

19:58 Everyone's saying that now? Hear the word agent in every single call. I feel like that scene in Seinfeld talking about write off. So you don't really know what that means, do you? Tell us what agents means to you.

20:09 And why it's such an important construct in this coming age of technology. Like we were saying, GPT came out two years ago. Is huge surprise for everyone, and I think for a while All of the products that came out of that were basically what I would call these text completion products that work off of that. And by the way, it makes sense, obviously, because the native form of a language model is

20:28 Put in this input text, tell me what the output text is. But that's kind of where you saw every application at first. GPT itself is obviously like that. You had a lot of Q and A products. marketing copywriting, hey, I need you to write this. paragraph for me that explains X, Y, Z. Can you just put together a nicely written thing?

20:43 Boom, here's the output. Customer support. customers asking something and then it's like what would you say here? Here's all the information you need to know. Uh. code that was the same thing too, right? Where

20:53 The first big code products, I mean GitHub compilers probably the biggest of these. It's a text autocomplete where it's like here's the file so far. Predict me the next line. And if it is correct, I'll just hit tab and it'll obviously save me the time to type the line. It doesn't save me the time of thinking of the line because obviously it's still me that's deciding what's correct and what's not correct, but it does save the time of typing the line. And I think What we're now seeing and what everybody's talking about now with agents, although of course it's very early, is

21:18 This new paradigm where models can actually act and make decisions in the real world. There's lots to it and it's how humans do it too. It's not like a human just produces this perfect paragraph. Here's everything that I needed to give you, and then you're all done. A lot of it is. Going and interacting with the real world.

21:35 looking up websites or something, trying out things, seeing what works and what doesn't work. In the case of code obviously running the code and looking at the output or looking at the logs. And things like that. And I think maybe the simplest way to put it is

21:48 In all of these verticals, I think we can see this where basically Hey, having a legal Q and A bot is great, but The next step is to have a lawyer. having something that can answer questions about math problems or something that you have is great.

22:00 The next step is to have like a full teacher. And obviously in software it's the same thing where it's like having the text autocomplete for coding does make you faster yourself as an engineer. But What if you had a whole engineer that you're working with? That's kind of the core. of this agent work.

22:14 Maybe you can describe then how literally Devin is built. Behind this question I'm also curious about taking your experience as advice for other builders. who are using core foundation models as a key part of the thing that they're building. And building in such a way that is complimentary

22:30 to those underlying foundation models and won't just get subsumed by them. Everyone asks about this. I'm sure every VC asks every founder, like, how will O1 not just do this? So tell us about how Devon is built. And how you think about building with

22:45 And alongside these evolving foundation models. We are A hundred percent laser focused on solving all the problems along the way of software engineering. And the truth is software engineering is just very messy. My point with that is just that

22:58 There's this idea of like solving these math problems in a vacuum, for example. There's a lot of research that's going towards that. But there's also this entire thread between of How do you take that? IQ

23:08 and turn that into something that is actually meaningfully useful for software engineers in their day to day work. So there's a ton of stuff that needs to be done. on adapting to the capabilities for particular tooling and particular stack. And then there's also a ton of stuff that has to be done on the product side as well. I'll just give several examples of that. A simple one is

23:26 Obviously, Devin has to be plugged into all your tools. Solving problems in a vacuum is great, but Devon has to work in your GitHub. has to be working in your code base. It has to use your confidence docs or whatever it is. That's where the information is. It has to read the bugs on GR. And

23:40 On top of that, obviously, Devin needs to learn with the code base over time. And so even a really, really, really sharp software engineer, they join your company on day one. They're probably not gonna be all that productive, at least compared to somebody who's been there for five years and knows the entire code base in and out.

23:55 Devin needs to learn the same way. Of understanding. Hey. Here's what this file does. Here's what these functions are for. If I want to do the schema migration here is step one, two, three to get that done.

24:05 And all that. And then there's obviously all of the actual step by step decision making that goes into it, which is somewhat different. We were talking about fixing a bug. I think a very natural flow with fixing a bug is

24:16 Okay, I got a report for Buck. First thing I'm gonna do is I'm going to try and reproduce the bug myself. And see what's going on. Gotta go run all the code locally and try to make the bug happen. If the bug does happen.

24:27 I'm gonna go and take a look at the logs and see what the error was or what happened exactly. What the variables were. And then from there, I'm gonna go read the relevant files. Maybe you can make some edits, test things out, and whatever. And then I'm gonna go test it again and see if it works.

24:41 And if that all works, then I'm gonna add a unit test to make sure it doesn't fail again and whatever. And all this step by step decision making is a very different problem, obviously, from Text completing. What with somebody really smart? On the internet, say.

24:53 How do you use the tools available to you and make a sequence of decisions that maximizes your chance of success? I think the other question you asked was how as a builder to think about this in relation to obviously this steady stream of improving models. It's a really interesting one because a lot of other technologies that we've had have been a lot more step function.

25:12 Mobile phones came out and the first question was what are you gonna do when everyone has a smartphone in their pocket? There's in some ways the core of Uber or Airbnb or DoorDash or all these things. And you can kind of think about Okay, this is the technology shift and Let's think about what happens over the next coming years now that we have this ship. AI is a lot more of a very incremental thing.

25:31 Here's GPT three and three point five and four, and each little technology gets a little bit better over time. And part of what that implies, I think, is yeah, just really thinking about What are the models not gonna do by themselves over time. And a lot of the stack and a lot of the things that we care about obviously are around building.

25:49 capabilities and the product experience specifically for software. I have no doubt that the foundation models will continue to increase in IQ and increase in IQ and so on over time. But a lot of the detail of all of the tools that you work with. how you make decisions with all of that, even how you communicate with the user. What is the user interface where you're talking with Devin and giving feedback and seeing what Devin's up to and checking in and making sure nothing's wrong and all these things.

26:12 There's a lot, a lot of detail, obviously, in that flow. That really brings out the inherent IQ. To put it that way. We've always thought about it in that way of How do we make sure everything that we're doing is complementary with the model increases and by the way, complimentary with everything else.

26:26 The hardware's getting faster all the time. The whole stack is getting more efficient everywhere. And We have our part of the stack and we all have always thought about it as how do we do as well as possible and or part of the stack specifically for software. What improvements are you hoping for from the foundation model companies that would make Devin

26:43 more effective than it is today in the future. I should mention, by the way, at the foundation model labs are all super, super excited about Ancients. And it makes sense. I think one thing which is kinda interesting just practically is

26:55 Let's say you ask GPT a question, who is the fifth president or something. You ask one question, you get one answer, and that's a single model correct. And obviously there's a consumer pricing and everything, but if you're going by the EPI pricing that costs one millionth of a dollar or something like that. Something ridiculously low. Whereas if you're asking Devin, hey I've got this new feature that I need you to build down. Can you build it and test it and make sure it looks good?

27:16 Over the course of that, Devin's probably gonna be making hundreds if not thousands actually of decisions over the next Half an hour or something. While it's going and building that out. And it also just practically means it's actually just a lot more inference on GPUs.

27:29 And so for that reason, what we'll see over the next while honestly even the next one year, I think agents are gonna become a lot more widespread. And what we'll see actually is I think the dominant usage of these GPUs of these language models and everything will be for agents. Largely because Every single Asian query is just so much more magnitude of

27:48 underlying language model calls than a single QA query. And I think what happens next with that Which I think these foundation model labs are all thinking about. Is yeah, how do we build our

27:57 our models to be as optimized for agents. And so I think a lot of these things of working with multi turn traces being able to handle long context outputs, for example, like context is a big thing that people are spending a lot of time thinking about. And with an agent, again, it's like

28:12 Do you have half an hour of decisions, obviously it helps a lot to know what's been going on in the last half hour. So some of these particular technologies that will really help with that. How do you use Devin? in your own building. Of

28:25 Devin. We use a lot of Devon actually when we're building Devon. It really is this new workflow, a very asynchronous workflow. And so I'll just give an example. We do all of our communications in Slack. We have a few channels. We have a crashes channel where every production crash gets reported. We have

28:41 a front end channel where we talk about any like front end feature requests or maybe like, hey, let's switch the flow around this way, or hey, this design is a little bit off or things like that. Front end. Things that come up. We have a bugs channel where users are filing bug reports of hey, this thing didn't work or something, or this thing broke or whatever. In all of these channels. Number the one thing we do every time something gets report is we tag down.

29:02 Okay. You can just go through the channel. Hey, the thing. Hey, somebody requested this. Add dev. Hey, we just had a crash. Add devices. Devin's not perfect. Devin's a junior engineer and often does need a lot of guidance and I would say probably about fifty percent of the time Devin's output on one of these is just like a pull request code that's straight out of the box, ready to merge.

29:22 And the other 50% time, obviously you can make the call of okay, do I want to talk with Devin and coach Devin through this. Why wanna Go and make some changes myself and just touch up whatever needs to be fixed, or do I want to just go start from scratch if it's especially far off or whatever. But having this be part of our workflow. You had to report the bug anyway. And so you might as well just

29:40 tag on and at Devon at the end and then you get a first pass for free. And so a lot of workflows like this Lot of it is just thinking about how to put it into the asynchronous workflows. A similar one honestly is If one of our engineers is building some big new project

29:53 And they're going and implementing things, or building the IDE, for example. This happens all the time. As an engineer, you're gonna go through the code and you're gonna see all these little things that are like, man, why did we write it this way? This is so messy, or like this thing is technically correct, but it is not the most stable way to do this, or lots of little things. You get caught up in a lot of these little things as you're building your bigger projects. It does distract and it takes away your focus. And what we do now, obviously, is

30:16 You just kick off Devin from the IDE for each one of these and it's like hey Can you go refactor this one real quick? Okay. Can you change this to be type safe? Hey, order this in a different way so that there's not going to be a race condition or something. And you're doing all these little fixes and you're kicking them off async.

30:30 And then you're doing your own project in the meantime. In both these cases, a lot of it is really just figuring out these asynchronous flows. It seems like part of the answer too in the near term, you said earlier that maybe ten percent of the engineer's time is spent on what a friend of mine calls positive engineering.

30:44 building things that create a part. Ninety percent might be on Make sure things work. More almost like a mechanic than a creative, productive engineer. So does that mean that in the near term what Devin and other systems like it are gonna help us do is eat that ninety percent and keep the program we're focused on the ten percent.

31:02 And then what happens beyond that? If we beat the World's best programmer in two years. Why do we need programmers? Will there be programmers? What will they do? So I think it's exactly right. Eating the ninety percent of all the miscellaneous implementation and debugging and iteration and whatever. I think what happens then is honestly.

31:20 You just get to do ten X more than ten percent. And I think the cool thing about it, we should say it outright. Obviously, there's a lot of fear in AI about what's gonna happen and what happens next, but I think one of the really beautiful things about software is There really is ten times more software to go, right? If we just had ten times the capacity.

31:35 Over the last however many years we've always been bottlenecked by supply. If you talk to any engineering team, I haven't met a single team who's said Yeah, we build everything that we wanted to build those extra software list to do. Every team out there has twenty projects that they want to do. They have to pick three. to take on this week because

31:51 There's just so few software engineers and software just takes so long to build. I honestly think we're gonna see a lot of this Jevons paradox where we're just gonna be building more and more and more software as it gets easier. It's what we've seen already, but maybe to a somewhat lesser scale. Building software now versus building it in the eighties. Probably is about ten X easier or so.

32:08 And this shift will happen a lot quicker, obviously, but Over that forty years. We've gotten a lot more programmers and we've built a lot more software. And I think we're gonna see a shift that looks like that as well. think in the long term, I think the soul of software engineering again is

32:21 Each person getting to decide. What are the problems that I want to solve and what is exactly the solution that I want to build? AI is not magic. Obviously it's not gonna go and solve all your problems for you. But it will give you the power to specify exactly what the solution is that you want to have and just be able to have that happen and build that out.

32:38 Do you think in general that all this means that business people need to start thinking harder about demand and distribution. If we've been supply constrained. And that's about to go away. Does that mean that the people that are gonna win in the future are those with better demand and distribution?

32:53 I think in software especially. We have always been capped by supply and not demand. Obviously, there are a lot of things that are more fixed in terms of the demand. I think there's a lot of interesting questions of what happens to the economy with this and a lot of interesting second order effects. The simplest way to put it, where does all the money in tech go? Where does the money at venture capital What does it go into and

33:12 It is primarily software engineers. So when you're getting ten X more bang for the buck for every software engineer that you're bringing on. It does change the landscape a lot of how much people can build with products and how much people can do. And then I think down the line you see a lot of pretty interesting effects with businesses. I see a few different categories of businesses.

33:29 Brightly. If we're being I think the toughest category to be in is business that's heavily reliant on switching costs and lock it. Lot of businesses out there, software businesses that The reason they make all the revenue that they make is because it is so painful to switch to the other product or the other technology or something like that.

33:46 If Devin is just able to do all that implementation and migration and those costs go to zero. Obviously it's gonna be a lot more about having the better product. That's going to be a real shift, I think, that companies are going to deal with. At the same time, it's not a nihilist view. You know, there are a lot of things that will be. Just as strong, if not stronger. Network effects, I think, will always be golden. I think data and personalization is going to be even more powerful. So

34:07 a business that owns the infrastructure, for example, or a business that has all the personal relationships and the data and so on, being able to serve for each person like an individualized product that is truly optimized just for them instead of this clunky thing that kind of works for everybody and touches all the millions of people, but is not the perfect product for anybody, I think a lot of those businesses will be able to do very well. What kinds of questions are becoming most common when you spend your time talking to the senior most people at the big established technology companies? There's definitely a lot of folks asking. How do I think about my engineering headcount and how do I work with this?

34:42 I think we're very much in the paradigm where people like to say this. A I won't replace you, but someone using AI better than you might. And so a lot of it is how do you get smart and how do you make sure you're on top of the latest technologies and you're ready for the shift, we're sure. Honestly gonna quicker than most people imagine. And getting your whole team set for that and getting things built for that. I think there's also a special case actually with these technology businesses themselves.

35:05 folks who are building SaaS products or things like that, where we talk so much about agents. I actually think one of the kind of interesting product changes is that a lot of these companies are actually going to be serving agents in addition to humans. A lot of great businesses out there that are basically meant for this collaboration, coordination, a lot of interfacing we talked about companies like Slack or GitHub or

35:26 Data dog or I Lasian or things like that, where there's a lot of work and compiling that they do. And also a lot of it is about the interface with how humans interact. I think the way I think about it is right now it's Those are really valuable tools for agents to be using as well.

35:40 Sure, you can imagine this nihilist view where an agent is just building everything on its own from scratch, but The truth is there's a reason that we look at logs the way that we look at logs. The reason that we set up our documents the way that we set up our documents and it does just make it easier. It makes the problem easier of getting the context that you need. So what'll happen actually is we're gonna have a lot of agents who are using these products as well. So businesses that are in that space, a lot of folks are actually thinking about

36:05 How do I make my product ready for My new wave of agent customers and in addition to all the human customers. I was talking to Martin Casado about these clining cost curves that technology creates. And the first one that's clearly happened already in AI is like the cost of marginal content creation has started its march towards zero. in all different forms of media.

36:26 Seems like With you And what you're building, there's that same curve story happening in the cost of software itself. How do you build a software business in that? reality. If I fast forward

36:36 ten years or for something sufficiently long. Why should Devon cost anything? Why won't there just be like an open source Devin that Does everything for me and What do I need to pay for software if we're entering an era

36:49 of marginal cost of creating software being very close to zero. I mean, I think right now Devin is really the only one out there in the market. That's doing that, but As you're saying, I'm sure there will be lots and lots of competitors. Pretty soon. Even and folks that want to do similar things in code.

37:04 Is great for the consumer. We were just talking about the ten X engineers and things like that. The truth is that Every level of decision making quality and every level of skill that you can have in your software engineer actually really does have this exponential reward curve and it makes a big difference. And one of the things that we're getting, obviously, is this ability to really, really personalize Devin to each of their customers. And so like Devin out a lot of our customers today, when we work with Devin, for example, it's just like Devin knows the entire code base in and out. Devin wrote half the files in the code base. And so there's all these details in it. Obviously, it's not just the lines of code, it's the decision making that went into it and thought process and so on that Devin has internalized and working with the entire engineering team.

37:41 And it really is that trade off where it it is about here's an engineer who's Maybe smart and sharp and stuff, but It's starting day one on your code base today, versus Here's an engineer who's been with your code base for years and years and understands every file and

37:54 has built all these things and knows exactly what your engineers mean when they ask certain things, and there's a real trade off there. It comes down to these things where it's kind of like pure technology switching costs, where it's migrating all your files to some new thing or moving to some different platform. Yeah, I think that is gonna get cheaper and cheaper over time. At the same time, obviously if you have the level of personalization, if you have the network effects or whatever, it's not about the cost switch. It's really just about you're able to provide a better product. Maybe say a bit about pricing. This is such an interesting question. Seems like

38:25 someone at OpenAI put their finger in the air and decided twenty bucks was a reasonable price to charge. And a whole business ecosystem has spun up and that seems to be the default pricing. You made a very different pricing choice. It's five hundred bucks a month opening. I just released the two hundred dollar a month option. So we're marching upward, but Talk me through how you thought about And think about pricing in this new world.

38:48 Yeah, we're gonna see more of that, obviously. I mean, I think for us It's funny, we had a lot of questions with our launch, for example. Questions of like how much do we anthropomorphized Devin? How do we explain the structure of it, and so on. But I really do think it is a pretty different paradigm where With agents. It does come down to usage for better or for worse.

39:06 There's reasons for both the inputs and the outputs for that. But the inputs like we're saying, I mean It is just more expensive to run. That is the case. You're doing more queries and more work. There's more GPU compute that goes into each job or each task or something like that. But I think with the outputs as well, obviously, when you're starting to take on tasks. There are real end to end tasks.

39:27 It's just much cleaner and much clearer to measure. the concrete impact that something like this has. I think the proxy that we essentially want to have is value based pricing, where it's basically We want to make a lot of the projects or the bugs or the features or whatever that you're building 10x cheaper to build. And so we want the cost of Devon to run on those to be 10x less.

39:46 It's obviously hard to have a perfect proxy for that, but the proxy that we have essentially is usage. We have a unit called ACUs, agent compute units, and it's basically all the decisions that Devin makes, the box that Devin uses, the code that Devin runs, or things like that. And it roughly corresponds to basically for every hour of Devon's work that you're doing. Again, it depends on different machines, but you're paying something like eight to twelve bucks or something like that. Obviously Devin can do a lot in our the setup is such that we want it to be ten X cheaper than having to do those yourself.

40:14 But I think that is gonna be broadly the new paradigm when we start to have AIs that are doing tasks instead of answering questions. Any thoughts on this big fun almost has become like a cocktail. Party style. question around whether or not we've achieved AGI or not and whether or not that's a question That

40:31 You care about? I'm curious what you think about this one. I'm in camp number two on that. We already have AGI. Someone said this in our share in twenty seventeen, you ask if we have AGI, the answer's no. In twenty twenty four, you ask if we have AGI, the answer is.

40:43 Well, you gotta define AGI and depends on this and that. I think there is a somewhat recursive definition of AGI, obviously. People like to say, Oh, it's doing eighty percent of human labor. It's like well, if you could do that, then the humans would be doing the twenty percent, so it wouldn't be the eighty percent anymore. There's some recursiveness, but the way I would say it practically is I think there's gonna be a tale of things where we will push the capabilities, we'll push the capabilities and for a while there's gonna be lots of little things that are like, Well, actually humans are still better at this one thing. And then we'll solve that thing. And then people will be like, Oh, well, humans actually can still do this one thing. And we obviously have a lot of pride as humans in our own species of

41:19 Wanting to find the things that make us unique and so on. But I think that like practically What matters for the world, obviously, is Is the AI good enough and is it

41:30 Built into products and distributed well enough. That it's actually affecting the world and affecting the economy and affecting people's day to day lives. In some sense that question matters more than Is it doing ninety five percent or ninety six percent or ninety seven percent of you know the human sport flow?

41:45 It's more a question of are humans actually out there. doing twenty five times more because of the AI technology. And I think that actually is Much more of a practical question. And in some sense it's like we're talking about the IQs and things like that. The math problems that these base AIs can solve already, honestly.

42:01 Shockingly good. And among humans would be like in the upper echelons of humans, if not quite number one in the world yet. And I think the biggest distinction is actually not whether we are number one in the world or only top zero point one percent human or something like that. The distinction is a bit more of Is that level of intelligence actually used? by every human out there in the world and done in a way that's actually given them a lot of value. That's going to be the big shift that actually gets us to the massive GDP growth, massive efficiency gains, quality of life, whatever you want to call it.

42:32 Investors mostly are obsessing over scaling loss because it just has such big implications for public equities, for companies building new products in the private markets. And so that's where they tend to focus. If you are training your attention on what's going on in the foundation model world, What do you spend your time wondering about asking your friends about, learning about where's your eye trained in that world. Yeah, my hot take on this too is I think scaling laws are a little bit of a myth.

42:58 I think we have made continued progress on these models and pushed the capabilities and things like that. And there's been a lot of new technologies, obviously, that we figured out. And there's way better post training and there's some of this stuff obviously that's going on reinforcement learning and things like that. Sure, there is some return to scale, but I think a lot of what we're seeing is basically it is also the introduction of new technologies that are making this more efficient or allowing it to grow with scale beyond that. And so people sometimes like to talk about scaling laws as if it you just sit back and as long as you got the hardware for it, it's just gonna get better and better and better. And I think really obviously we have needed more and more hardware, but it's also been innovations that have actually gotten us there.

43:35 I think practically with foundation models. We always obviously want to have a pulse on what's going on and what things are going on next. We work very closely with all the foundation model labs. Um we do evaluation of early models with them. We do custom training of models with them, things like that. It's like we're saying it is an incremental thing. These things will get better.

43:54 And a lot of the question as a decision maker is to basically just have a very clear sense on What are the capabilities that will get better? And what are the either the capabilities or the product or the human experience of it that will always need to complement these. for you to deliver something of value to your customers. I'm really curious to learn a bit about the formative experiences for you as a business person. Obviously, you had we've talked a lot about formative experiences as a technologist, building cognition and also IOI and these other competitions for the first part of your life.

44:26 I'm sure everyone's seen the video of you when you're twelve or something winning math competitions. Obviously, you've had some pretty cool formative experiences. What have those been on the business side? For example, I'm interested to hear about your experience with Lunch Club and what you were building, why you're building it. and what that experience or that chapter of your business career taught you. Yeah, Lunch Club was my first company. I dropped out of school actually to start the company.

44:49 It was a great experience, first of all. It was a AI for professional networking. We made millions of meetings happen and it was super fun team and product to build. I would say one of the big things, honestly, that I learned is Sounds simple, but it really does come down to the fundamentals. I feel like the common pieces of startup wisdom that everybody talks about.

45:07 In some sense they're so obvious that it's not obvious. It's not even worth mentioning if people feel that way, where it's like, go as fast as you can. Everybody says that. Everybody knows that. Never compromise on recruiting. Stay as close to your customers as possible. Focus on building a product that people love and want.

45:22 These are almost tautologically true. But the thing is. The reality is there's always another gear. that you can kick into with your fundamentals. And that's how I've really thought about it now, having gone to see it in simple examples like everybody says you can go fast. In our case, we incorporated the company in January. We were building the first versions of the product.

45:42 We got it out to the first group of initial users in February. We did a launch in March. We raised big round in April. We did a big partnership with Microsoft in May. While we were doing that, we got on a lot of enterprise customers. We grew the team. No matter how fast you're going, and every startup in the valley obviously thinks they're going quite fast. It's still worth it to really push for even faster.

46:03 And no matter how close you are to your customers and how deeply you understand their problems, you're probably not gonna regret spending even more time with them and understanding that deeper. And no matter how high your bar is for hiring, you're probably not going to regret trying to go for folks that are even higher still. And I think that is the biggest lesson I've learned with startups. The competitive landscape. Does it feel like this big exciting green field, or does it feel like

46:27 battle lines are being drawn and Insanely competitive for every available opening. And you know, sharp elbows. Characterized what the playing field. feels like right now.

46:39 You all work out of this amazing house that I've been to. It's such a cool environment. Which feels so start up y And yet the next day you're with Microsoft. The contrast here of Real true startups. in startup houses and raw cultures that are

46:54 Emerging They're interacting like you are with the biggest players in the world of technology inside of a few months. It's just like a bizarre set of circumstances. How would you Describe the playing field to an outsider. I think it's still very greenfield. I mean the technology itself is so early.

47:10 We're like one or two years in in some sense to all of this stuff really being valuable and and getting there. One thing That I think about a lot is Software's just so big. We obviously know that, but there's so many facets of software and so many details and So many different things that can be done and can be built.

47:27 And so one way that I think about it is each of the use cases that we're talking about, there are enormous and really, really meaningful businesses that are built in Just do that use case. A lot of folks are talking about AI for Data observability, for example. DataDog is a great business. It's making billions of revenue. It's something like a fifty or sixty billion dollar market cap.

47:45 to doing very, very well and growing quickly. And as you can imagine, there's so much to do with AI for data observability in some form. Incident response is a huge thing that's only gonna get bigger and bigger. And there's a lot of great businesses in the space, like PG is a great business, but AI is going to make these things again, orders of magnitude bigger. It's true with testing, it's true with modernization programs. It's true. Migration. A lot of these big guys like Microsoft, Amazon, Google or anyone.

48:08 The amount of effort they have to go through to migrate customers onto their platform, even today, is enormous. It's billions and billions of dollars every year. So all of these are just such enormous spaces. And there's still so many questions even about the product form factor. And as the technology gets better and better and better, the kinds of use cases that we'll be able to support as a space are only going to get bigger. What do you think the world is overestimating and underestimating right now about your whole world about the

48:36 automatability of software engineering. The speed of it is probably a big one. One thing I always think about If we're just looking all the way across AI, for example. I think agents in general are gonna really work everywhere.

48:48 Code is one of the first for a lot of reasons, but I would almost argue one of the first agents, if you want to call it that, actually was self driving cars. That's an actual AI that's making decisions in the real world and taking in feedback and iterating with that. People are familiar with the story of self driving cars where I actually lived in Mountain View in twenty fourteen. I would see the Waymo driving around all the time. I never got a driver's license. I had just gotten started working at the time.

49:12 I'm not gonna need one. They're basically there. I'm sure I'll be able to use it. We did finally get there. I I think there's still a lot of room to grow, obviously, but it did take a lot of time. I think the difference between self driving and code and Want to see a lot of other applications I think we'll see over the next year is

49:26 Thing with self trim, you really, really need to have it be like ninety-nine point nine nine nine nine nine percent. sufficiency and there's not a world where it's like uh Or there is, but it's not as big of a difference where it's like human and machine combined can do a lot more together. And I think that's people's typical example is like, okay, you know, there's gonna be all the edge cases to figure out and all that, which is true, by the way. But the nice thing with software, for example, is there actually is just like a really great way to do human plus machine and do a lot more.

49:53 That's the case with Dev. Dev is not the ninety-nine point nine nine percent solution. It looks a lot more like the twenty fourteen Waymo, or honestly, probably even a little earlier than that. But the thing is. Having Devin take a first pass or having Devin send you the code and you take a look and give a review or whatever it is like that, where Devin is doing the 90%. If you're doing one nine instead of six nines. It's still super, super useful. It actually does save you ninety percent of the time. Whereas you're not gonna get in a car that is only ninety percent consistent.

50:22 And so I think there's gonna be a real form factor difference and how soon we're gonna see it actually really transform practical software. Not just because of the capabilities, but also just because The product space is better set up for that. What was the lesson that

50:36 Lunch Club taught you about The world of entrepreneurship. Was it something around market, was it something around product? If you think back on that experience

50:46 What's the big takeaway and how are you applying that learning to what you're doing at cognition? Probably the biggest one that stands out, honestly, is Sometimes it's easier to solve a bigger problem than it is to solve a smaller problem. Which is fine, you know, people folks talk about this. If you're going after something that's truly enormous and it's the thing and it's the most exciting thing.

51:07 You'll be able to bring together a group of people who's all really excited to make it happen and is really passionate and dedicated to pushing the frontier. in a way that actually sometimes make it easier. Our team, for example. Our team's about twenty people right now. Of those twenty, I think fourteen of them actually have been founders themselves before.

51:26 And it's kind of an interesting one where we've all done companies and honestly, it's a really, really talented team and a lot of these folks. would get a blank check to raise money if that's what they wanted to do and they wanted to do their own thing. But a lot of it is just How do we come together? And build something that's really great. I think that's one of the big lessons for me. What has been the most

51:44 difficult or stressful moments so far in the one year cognition journey. Oh, there's been a lot of that. Let's do a few then. I love these. One of my favorites.

51:55 All these launches, you know how it is with startups, obviously. Things are so last minute. So we had the GA launch this week, and then we obviously had our initial product launch back in March. And for both of those. The few days leading up to the launch were some of the most stressful, but just the most memorable experiences and our launch in March. It's huge. There's so much you have to do.

52:15 to get into it. And obviously there's The product itself and testing with early users and making sure everything is truly working the way that it's supposed to, getting the info ready for it. setting everything up. And then there's obviously all these other things with filming the video, thinking about the blog post, thinking about the content, getting things out there, working with customers to get testimonials and things like that. And our March launch actually, we were doing all that in addition to We're doing a fundraise right before the launch as well, actually. And so we were doing that. We had candidates who were in the pipeline, pretty meaningful candidates that we were working on and figuring out and everything during that as well. And so that weekend was a very fun time. That much was actually

52:51 There's an article that went out by Ashley Vance, the Bloomberg article about our company. And that was the article, obviously, that marked the launch. It was supposed to go out. at six AM Tuesday. We ask them to push. Just a little bit later. We got it to be nine AM Tuesday instead of six AM. We needed those extra few hours. We weren't done filming.

53:09 the video for the launch as of that night at one or two AM. We were still recording lines and And then there was all the blog posts and putting that all together. It's a crazy weekend. Yeah. Do not get very much sleep that night, basically. It's just a really fun moment to get together and really do everything together. It's one of the reasons actually I think that us all just living and working in a house together is great, is because there's something about the shared experience. Yeah. Seeing the late night energy and everything. I feel like it wouldn't be quite the same at an office. And yeah, this launch as well, you know, actually we were all in Utah.

53:39 Which was kinda hilarious actually last minute decision. It's like okay, we're gonna use that. And we got on a plane that day. We found a cabin that would fit us all. got into that and we were all just working on the launch and it was the same story always. Even having done this we got no wiser and so we were still filming at two AM the night before and putting things together for the launch and getting all the content ready and getting the info ready. And obviously in this case it's a full GA thing too. So We gotta be ready to handle ten X alone or even more than that.

54:06 So all these kinds of things that we had to do to get ready. And those are some of the funnest moments, honestly. I'm just extremely interested in launching things. I think there's so much rolled up into it. that it's like business encompassed or something. Can you tell me the best and worst part about each of those two launches? In retrospect.

54:22 Honestly, the experience with the team is the best part. Just getting to build it together and do it together with the team. It really is the soul of Startups. Yeah. The worst, I mean, there's scares and stuff that happen all the time.

54:34 The night before it's just like hey, we did our load testing. This shit didn't work at all. All the info went down, there's no way we're gonna be able to handle the load. We were expecting to handle this many concurrent Devon sessions per minute and this did not work at all. And the last minute things we have to do. The onboarding flow, for example, last minute changes that we had to make. My co founder Steven actually put out a post talking about how we ship the whole thing where you could do a Slack connect with us and work

54:57 directly with our team. That got started about one a night before the launch and ships the morning of the launch. And Devin built a lot of that because obviously we needed the extra bandwidth from Devin. In the moment. Every single thing that you're optimizing for is how do we make this not? incredibly embarrassing for us all. And you know, sometimes you just have to put it out there and do your best.

55:18 How do you think about something like O one Pro Mode and people using that? To do coding. versus Devin. Yeah, I think these tools are great. And I think a lot of companies are thinking about code and are focused on code. And like we're saying, code is such a big thing. There are a lot of cases where, yeah, sure. It's like you want to build something, a single file of code out of the box, and you want to think through particular problems or whatever.

55:40 And have that be a really great single file of code or something. Where yeah, you can just go to Oone Pro and consult it or an AI stack overflow, if you will, where it's like you can paste in an error. And it'll give you its best guess from what it has.

55:53 Here's what I think the error is. With Devin, there's a ton of just these iterative flows that make a lot of natural sense. And so for example If it's not just writing a single file and that's all. But it's like

56:04 Hey, I need you to work in my code base and build a new feature and plug it into all the existing stuff and test it out to make sure that it works, then obviously all of those iterative steps, those are the things that Devin is doing for you. And similarly. If it was a debugging thing like a AI stack overflow thing, for example. Actually one of our first Devin moment I wanna say is

56:23 We started building this as more like research type technology. And just try to understand multi turn agents. iterative decision making and stuff. And eventually it made sense to build into the Devon product. But one of the first things that really convinced us is we were trying to set up our database for our own work when we're setting up MongoDB and anyone who's done it knows what this flow looks like basically where it's just Give you a set of commands, you try it and then it doesn't work, and then you read the error and you copy and paste it and you try to figure out you Google what's going on.

56:50 You find something and you run that. And then you run into a different error. You do that ten times until the thing works. Everybody setup is always a little bit different and there's always these little details that will get you. In this case, we'd spent a while trying to set him ongo DB and like he wasn't working. And so we just gave it to Dev and we're like, Hey. Can you just please get this working with me? And then Devin did it. And that was the thing.

57:09 So what we were saying earlier. It's not just a stack overflow where it just gives you the one answer of, Oh, you ran to this error. Oh, maybe you should try this. Like Devin sees the error. And says, okay, let me see what's going on. Let's check what ports are open. Let me see what the schema of this database is. Let me see if the socket connection is actually set up. It's going and running all those commands to go look at that. And then it uses that information to decide, okay, what's the next thing I'm gonna try? And then it tries that.

57:31 And if it run into a new error, then it's able to debug that. The integrated autonomous flow is I think the biggest difference. How hard is it? To build that. I'm not technical enough to ask the best version of this question.

57:43 But if I think about part of the secret sauce being the iterative agentic nature of the whole thing. What are the hardest parts about building that capability. The obvious one is the model capability itself.

57:56 The models are roughly like here's what a smart person on the internet might say. But if you're now thinking, Okay, I'm doing a step by step process to make decisions to maximize my chance. Not now, but after a hundred steps that I'll have solved the issue, then it's obviously a pretty different thing that you're solving for. And so a lot of it goes into the capabilities. But I think beyond that, too, there's so much with the infra and with the whole experience that gets quite involved, as you can imagine. And so for example, Devin has its own.

58:22 Machine that it works in. And a lot of the features that you'd want to support, for example, like rolling back state of the machine or dealing with hot loading these quickly and being able to spin up a Devon instance with a new machine immediately. being able to go and navigate through all the previous decisions and go back and forth between these different things. Or the ability to work with different Devons in parallel or things like that. Yeah, there's a lot of infrastructure problems that go into that as well.

58:45 And I think it just comes down to this thing where Yeah. Software engineering is so messy. And so it really is just so different from doing problems in a vacuum and Building all of these.

58:56 high touch components with the real world where you're actually able to do these things in the real world, you're actually able to use all your integrations, you're actually able to maintain your own machine state or work within your Git checkout or all these things. A lot of the mess of that is the part that I think that really comes out from us just really focusing only on software engineering. If you had to put your most sci fi

59:18 lenses on for the future of technology in the world. What are some things that you maybe talk about with your team just for fun that don't seem feasible yet, but might be feasible in the future that get you the most excited. My co founder Walden has this great line, which I've always really loved, which is

59:35 We've been playing in Minecraft survival mode for so long, and now it's gonna be time for Minecraft creative mode. Describe what that means for those that don't have ten year old boys. Yeah, yeah. And so basically Minecraft it's a sandbox open world game and it's meant to be a simulation of life. You can go and grow food, you can go and forage, you can mine, you can do all these different things, you can build stuff. It's kind of funny because there's there's two major modes. There's one, which is survival mode, which is you are actually just have to deal with the enemies that come at you. You have to go build your own shelter and make sure you have enough food to stay alive and all these things. And there's obviously creativity within that things that you could do, but you are also constrained by how hungry your own character is. Or whether you're being attacked or something like that. Whereas Minecraft creative mode is basically a mode where

1:00:17 you're able to put together whatever you want. It's not like I didn't find any iron ore today. And so I'm not gonna be able to build this thing. It's like I wanna build this thing. I wanna try this new thing. I wanna set up this really cool thing. There's folks who have set up entire computers and computer programs in Minecraft, which is insane. It's basically just Focusing on the ideas and the imagination rather than the execution.

1:00:38 It's not about okay, now I'm gonna have to go spend another two hours going and collecting gold ore because I wanna build this thing. It's like I know exactly what I want to build and now I have the freedom to just make that come to life. That's honestly I I think the big dream of AI as a whole is to be able to free up humans. The point of AI isn't to go and just solve all of our problems for us and leave us with nothing.

1:00:59 I think it's to give us the power that or the kinds of problems that we want to solve or the things that we want to build to give us the ability to actually just turn that into reality. Yeah, it's so funny to imagine I'll use my own experience when I visited you You asked I think the demo is incredibly powerful. So you asked me to come up with an application idea. I think it took eight minutes to build and we watched it do it. Watching the iterative nature of it feels like uh

1:01:21 real aha moment because it really does look like it's in hyper speed doing something that a person would do as it builds something. But maybe the more profound Realization I had was It built.

1:01:32 the thing that I described in natural language. But it wasn't until I saw the thing The little Stupid application that I made out. That I realized

1:01:42 why that idea was bad. When I said the idea it sounded good, and when I saw the thing, I knew why it was bad. And So I would say The thing that excites me is

1:01:54 Not that I can make my ideas. But that I can get better ideas. Which I think is Pretty underrated. part of this whole thing that just like a computer program is gonna be faster now.

1:02:07 Anyone with ideas is gonna be better and faster because they can see the thing. And see why it stinks. Does that resonate? That's a great point. It's very much the idea of turning ten percent of time that you get to spend on your ideas until a hundred percent of time where whereas before

1:02:20 You gotta get a team together, they're gonna go and build the whole thing, and you spend a couple of days or something building this whole thing, and then you get to do your next step of idea iteration. And by the way, Devin's still honestly a lot of room to grow and it's gonna take a lot of progress, honestly, to get to this future. But but to get to a point where it's basically just You get to do the ideation work. You get to iterate against that. You get to do the new idea, you get to do the new idea, and so on. And instead of these ninety percent implementation with the little 10% chunks in between, you just really just get to spend all your focus on the ideas. I'm super excited about that.

1:02:48 It seems like one of the defining aspects of cognition. Is the degree of friendship and trust between the leaders is extremely unusual. Maybe you can describe the origin of that.

1:03:00 But when you're moving fast, trust is really valuable. And I'm just curious to hear about that side of your building experience so far. Pretty much all of us on the team have known each other for years and years and years. Steven Wald and Russell. I've known him these guys for a long time and the whole crew as well.

1:03:15 I had always wondered about okay, is it gonna be awkward to work with your friends? There's lots of things that come up. And if anything, it's honestly the other way around where it's easier to have tough conversations because you know, obviously. We were friends before all this. All of this that we're doing is coming from a place of we are just trying to take care of each other and make something great together. when you both know that's true and when that undertone is always there, it's a lot easier to just call out stuff like, Hey, like I think this is totally wrong or we need to change this thing right now, or things like that.

1:03:43 What in your experience talking to investors Present company included. What do people not ask you about the product, the company, or the vision that they should ask you about. Yeah, it's a good question. I think one thing that's definitely different.

1:03:58 Talking about it versus feeling it is that folks sometimes talk about or ask about really is just kind of the experience with the team. I think The day to day it's Getting the product out there and seeing what folks are saying about it and talking with our users and things like that, obviously But ourselves, I think.

1:04:14 Just how we do it and how we build it as a team is truly Special, I would say. I think we have Some of the smartest and most ambitious and most capable people and yet somehow they're All such low ego people as well.

1:04:28 I think that is truly the That's the soul of it for me is the twenty of us. in a house together trying to do something meaningful. It's a big difference in talking about it versus Really seeing it live. And so One of the things that we

1:04:40 often do as we're really pushing with it is having folks At the house. Just come by and see what it's like. It's gonna be one AM or two AM and stuff, and people are still building and there's a lot of discussions and stuff that's moving quickly and everything. And that's probably the thing about building the company that I'm most great for.

1:04:57 What uncertainty. interests you the most. Yeah. That's outside of your control. Interesting. Yeah.

1:05:05 One of the uncertainties I think I should think a lot about is Broad adoption. It seems very clear to me now that the technology will get there, or maybe already has gotten there in a lot of cases to be extremely impactful. And I think the rate at which we see it happening in different industries is going to be pretty fascinating. We can both think of things like medicine or law or things like that, where you can certainly imagine it will take time to do that. And then there's obviously just a whole world of distribution. I will call it a win where everyone in Baton Ridge, Louisiana is actually having a significantly better life because of the work that AI is doing.

1:05:40 It's a different curve everywhere, obviously. I think we're lucky in software that Developers are just kind of inherently curious and always want to try to do something. But I do think that rates of adoption, it's gonna be a real question with this technology because I think The pace of progress is gonna be at such that

1:05:53 technology in some places gonna be way ahead of reality. When we were first together, you did this insane Car trick. Are you willing to do it for us? Yeah it's Tell us the origins of this trick.

1:06:05 Yeah. So this is a card trick. So I wouldn't even call it a card trick, it's more of a card game. And so Growing up, I always used to play this game called Twenty Four. It's a little math game. And I think it actually helps a lot with just getting the fluency.

1:06:18 In math. I think it's one thing to learn, okay, here's addition, here's multiplication, whatever, which is obviously the first building block. But also a lot of it is just this creativity and this ability to reason with numbers and experiment with numbers. And so I'll I'll just show what the game looks like. And so the idea is you have four cards and they're four numbers. By the way, Jack is eleven, queen is twelve, king is thirteen. So it would be a four, seven, eight, and twelve. Are the four numbers that we're dealing with here.

1:06:43 And the idea with this is You can use the four numbers in any order in any combination. You got to add, subtract, multiply, divide any of these. And your goal is to Put together an equation that uses all four of these and makes the number twenty four.

1:06:56 And so here what I would do is Twelve minus eight is four. Four times seven is twenty eight. And twenty eight minus four. Is twenty four.

1:07:04 And so you can see how it uses a lot of this creativity where it's like hmm, messing around this number Four. Twenty-four is a multiple of four. And so if I could just make a six from the other three, then maybe I could do something. Turns out that you can't quite do that, but you can make a seven and subtract another four out. Things like that. Where you're playing with these ideas. And so at these math camps and programming camps, we used to play this game all the time. And eventually we got to the point where you learn all the combinations of 24. There's only about thirteen hundred total combinations. That you have to know. Not all of them are possible, but for the ones that are possible, you know them. And so what we did next was we did a thing where you take six cards and you're making 163.

1:07:38 And so I'll just steal another six parts. And so in this case, for example, what I would do is eleven times two is twenty two. Forty two minus three is nineteen. Six plus ten is sixteen. Sixteen times nine is a hundred and forty four and a hundred forty four plus nineteen. It's a hundred sixty three.

1:07:55 And so as you can imagine, it's a lot of fun. We we play this at the math camp during the team selection training for the US team. Basically in between contests, we would just be playing this game all the time and doing this. And eventually you get to the point where one sixty three, you have a lot of the patterns down and you see things. And so then the next thing that we did was we do eight cards and we make the current year. And so now it's two thousand twenty four. And so let's just go ahead and do that. So Thirteen minus twelve is one.

1:08:20 This is actually relatively easy. I'm just gonna take one times one times one and we still got a one. And thirteen plus ten is twenty three. Twenty three times eleven is two hundred and fifty three. Two hundred and fifty three times eight. Is two thousand twenty four, which times one is still two thousand twenty four. Incredible. And eventually you learn all the patterns with that. And so obviously the next fun thing that we have to do is we have to do a custom number. How many digits? Four digits? Yeah, yeah.

1:08:42 Let's do seven five three two. Seven, five, three, two. Okay, so it's a little bigger, so we're gonna do nine cards, okay? Okay. So Eleven times seven is seventy seven. Seventy seven minus two is seventy five.

1:08:54 We got two tenth here, which is a nice hundred. And so that makes it seventy five hundred. And then from here, nine times four is thirty six, three plus one is four, thirty six minus four is thirty two. So seventy five hundred. Plus thirty two, seventy five, thirty two. Yeah. Pretty amazing.

1:09:09 I don't know how you do it. What are you doing? What is the method behind calculation? Or is it multiple methods? A lot of it is intuition, I guess, that you build over time. with one sixty three, for example, a lot of these little things Where I know okay. Eleven times fifteen is one sixty five. So if I can make an eleven into fifteen and then I can make a two left over, then I could do one sixty five minus two.

1:09:31 And then I'm searching for that. And so based on the numbers I see, I'm solving for something like that. And it depends a lot on the patterns that you get and everything. It's actually I think great game for kids. This is my favorite game as a three year old kid or whatever, playing twenty four with the four cards. Because a lot of it it really pushes the number sense. It's not just a question of can you add two numbers, but can you really reason around that and like you know what that means and be creative with the numbers that you have, which I think is the core skill that you really want to learn, not just the mechanical part. I think what you're building

1:10:01 is so cool to be because effectively what you're doing downstream is unlocking the creativity of a lot more people than currently have the ability to build things through computer science and programming. And I just think that's an incredibly Cool and Amazing thing for the world. I hope you and others like you are successful in doing so.

1:10:18 When I do these interviews, I ask everyone the same traditional closing question. What is the kindest thing that anyone's ever done for you? Mm. The number one thing that I come to throughout my life is honestly just all the mentors that I've had and how much effort they truly put in.

1:10:33 even as a kid, as you can imagine, like training for the US T and for the international thing. It was funny because I never had a coach. I never had formal this is the person, but I did have a couple of people and one in particular who just really invest in me and believed in me. And you know, he was the one who really just pushed I was a high school kid. I had a lot of other things that I wanted to do besides just doing math all the time. And he were always pushing like you gotta practice, you gotta do this thing. Every time it would get something wrong, you'd be like, Yeah, you can't make that mistake. Here's how we're gonna do these things. And all the time that you put in is

1:11:02 Everything honestly comes from that. I feel like I'm really only the sum of all the mentorship that I've received and I think the same is true in startups. I can think of a few people and again I can think of one in particular who's really just coached me through a lot of this and guided me through a lot of these things and

1:11:17 been the one to give me the harsh criticism and the feedback and Hit me with the things that matter and push me to do better and to do more and People often think of kindness as sacrifice. And I do think actually that often a lot of the greatest kindness is about building more together and doing more together. Scott, thank you so much for your time.

1:11:35 Thank you so much, man. If you enjoyed this episode, check out Join Colossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.

What you see above is a preview of the first minutes. One unlock costs 10 credits and covers this episode forever: full segment and word-level timestamps on this page, plus .txt, .srt, .vtt and word-level JSON downloads, as many times as you like.