Aravind Srinivas - Building An Answer Engine - [Invest Like the Best, EP.363]

0:00 I know firsthand how complex the tech stack is for asset management firms. And seemingly every new tool and data source makes the problem even worse, adding more complexity, more headcount, and more risk. Ridge line offers a better way forward, one unified platform that automates away the complexity across portfolio accounting. Reconciliation, reporting, trading, compliance, and more, all at scale. Ridge line is revolutionizing investment management, helping ambitious firms scale faster.

0:25 Operate smarter and stay ahead of the curve. See what Ridgeline can unlock for your firm. Schedule a demo at ridgeline.ai. Hello and welcome everyone. I'm Patrick O'Shaughnessy and this is Invest Like the Best. This show is an open ended exploration of markets, ideas, stories, and strategies that will help you better invest both your time and your money. Invest like the best is part of the Colossus family of podcasts, and you can access all our podcasts, including edited transcripts, show notes, and other resources to keep learning at joincolossis.com.

1:00 Mm. Patrick O'Shaughnessy is the CEO of Positive Sum. All opinions expressed by Patrick and podcast guests are solely their own opinions and do not reflect the opinion of positive some. This podcast is for informational purposes only and should not be relied upon as a basis for investment decisions. Clients of positive sum may maintain positions in the securities discussed in this podcast.

1:23 To learn more, visit psum.vc. Mm. My guest today is Arabin Sri Nivas. He is the founder and CEO of Perplexity. A startup that he describes as an answer engine, built from scratch with AI.

1:38 has set out for perplexity to become the most powerful answer engine backed by up to date sources. He helps me pick apart the technology Describing the behind the scenes of what it takes to build perplexity to reach its potential. and compete along the likes of Google and OpenAI. Our conversation goes deep into programming this kind of infrastructure, the competition around latency.

1:58 and constructing a business model around deep learning. There's so much on the horizon, so please enjoy my conversation with Ervin Shrinevas. I thought a fun place to begin would be A game I like to play with people, which is I call it the one minute bio. I'd love to hear the one minute summary of your life. Up until the founding of perplexity, just to set the context for all that we'll talk about.

2:23 Yeah. I grew up in India. From a pretty humble background. was focused a lot on engineering and programming right from the beginning. Studied in one of the ITs. Got really excited about AI.

2:35 Because I happen to do a Course on machine learning. Deep Mind published a paper on training AIs to play Atari games. That got me to be resourceful. I got

2:46 Consumer gaming GPU cards. from other people on my lab. created my lab desk for them so that I could stay at the hostel and work on Training these neural nets. And

2:58 In general, when I look back, I've always realized I've been good at making deals. But Traditionally you would consider me a nerd just excited about research and programming. My research there got me into Berkeley. I got to do more exciting AI research in Berkeley. that got noticed by institutions like Open AI and DeepMind.

3:17 Which got me more exposure to the cutting edge. And At Deep Mine, I had a pretty crappy apartment. When I was an intern, so I just mostly stayed in the office. And I used to stumble upon books like how Google Works there.

3:30 And in that, like I got really excited about entrepreneurship. And that got me into thinking a lot of starting a company to work on like a hard problem. Little did I realize I would literally go on to work on search itself. But That was Fate Love's Irony sort of thing, where the people you're excited about you Taking on them.

3:49 I worked at OpenAI and my entrepreneurial ambition exceeded what I could do as an individual contributor there. So I left and started perplexity. Initially as a small project to just work on searching over Twitter and LinkedIn and things like that. And as we kept seeing the progress, we just expanded our ambition to just searching over the entire internet. Providing a

4:11 Much better experience by giving answers rather than links. What is your theory of good deal making? Create a win win situation. And don't be greedy. There's a show called Succession.

4:24 Yeah, great show. Their Logan Roy advises. It's not a deal unless you really screw over the other person. But in Silicon Valley at least. Long term. Deal making is the best deal. When I think about you, I picture that

4:38 Seen in Star Wars where Luke Skywalker is flying the little tiny plane into the Death Star, and you're the plane and the Death Star is Google or something. Google's been considered One of the most unassailable Motes in all of business. And the product's been ubiquitous forever. We've all used it. Many times a day for decades now.

4:57 Tell me about how you conceive of the components Of a great search product. To go up against school, you obviously have a theory of this is what great search should be. And maybe Google has gotten away from that. Just talk us through what great is to you.

5:11 What do you think about search? When I think about the word great, I'm reminded about Steve Jobs. Insanely great. You don't want to be just great, you want to be insanely great. Look, search has always been a hack. Ten blue links was always a hack. To get us information. But

5:28 Not needed anymore. When we can more or less answer your question directly. So that is the great experience. And what is insanely great then? Insanely great is The AI doesn't even

5:42 Let you struggle. To articulate a good question. This is the next part. Of course you still have to Make the first part work reliably, accurately. You know, address the long tail of mistakes.

5:55 But assume that that's gonna get solved with a good amount of engineering. The next part actually to get into insanely great territory is Make it so easy to even ask a question. Why are very few people good podcasters? Why are very few people good interviewers? Because asking good questions is not a skill that most people have.

6:14 Everybody in the world is curious. Curiosity is unlimited, unbounded. But not all curiosity in every individual. can be precisely articulated into good Interesting.

6:28 questions that elicit the most from an interesting mind. And AI is an interesting mind. It is a very knowledgeable mind. It is access to basically All the world's knowledge and In an instant. But it's up to you to harness the power. Now

6:42 Why do we require humans to be great prompt engineers? Why do you want to sell that vision? Open AI version of oh AIs are amazing. You guys figure out how to be good at using them. But what would Steve Jobs do? Steve Jobs will be like

6:57 Bring the power Of these amazing AI is to the mere mortal. In fact he's used the word mere model many of these emails. Our job is to bring the joy of personal computing.

7:09 To mere mortals. Those who can see it. Before others, but democratize the joy. That's what we want to do for knowledge. Bring the joy of learning to everybody. And so that's what we wanna address.

7:22 Either helping people ask questions or start with some dumb version of the question and Help the AI refine it for you. Um profile you enough that Suggests interesting questions to ask.

7:36 And how the knowledge Feed. of questions that you just see because you asked questions about those topics before. And just Every single day make you some X percent smarter. That's what we wanna do.

7:49 It's really cool to think about the sequencing to get there. We've had search engines, like you said, it's a hack to get to answers. You're building what I think of today as an answer engine. I type something in, you just give me the answer directly. with great citation and all this other stuff we'll talk about. And the vision you're articulating is this question engine. anticipate the things that I want to learn about and give them to me beforehand. And I'd love to build up towards that. So

8:11 Maybe starting with the answer engine. Explain to us how it works. Maybe you could do this via the timeline of how you've built the product or something, but what are the components? What is happening behind the scenes when I type something into perplexity. either a question or a search query or whatever. Walk us through in some detail the actual goings on behind the scenes.

8:31 In terms of how the product works itself. Yeah. So when you type in a question into perplexity. The first thing that happens is It first reformulates the question. It tries to

8:43 Understand the question better. expands the question in terms of adding more suffixes or prefixes to it to make it more well formatted. So it speaks to the question engine part. And then after that It goes and pulls so many links from the web. That are relevant to this reformulated question.

9:03 There are so many paragraphs in each of those links. It takes only the relevant paragraphs from each of those links. And then an AI model, we typically call it large language model. It's basically a model that's been trained to predict the next word on the internet and fine tune for being good at summarization and chats. That AI model Looks at all these chunks of knowledge snippets that you've

9:25 Surface from the Important or relevant links. And takes only those parts that are relevant to answering your query. And gives you a very concise four or five sentence answer.

9:37 But also With references. Every sentence has a reference to which web page or which chunk of knowledge it took from which web page. And puts it at the top in terms of sources. That gets you this uh nicely formatted rendered answer. Sometimes in markdown bullets, or sometimes just generic paragraphs.

9:55 Sometimes it has images in it. But A great answer with references of citations so that If you want to dig deeper, you can go and visit the link. If you don't want to just read the answer and ask a follow up, you can engage in a conversation. Both modes of usage are encouraged and allowed.

10:11 What percent of users end up clicking beneath the summarized answer into a source web page? At least ten percent. So ninety percent of the time They're just satisfied with what you give them. Depends on how you look at it.

10:25 Do you want it to be hundred percent of the time, people always click on a link. That's a traditional Google. And you wanna be a hundred percent of the time where people never click on links. That's chat GPT. We think a sweet spot is somewhere in the middle. People should click on links sometimes to go do their work there. Let's say you're just booking a ticket.

10:41 You might actually want to go away. Experior or something. Let's say you're deciding where to go first. You don't need to go away and read all these SEO blogs and get confused on what you want to do. You first make your decision independently. With this

10:55 Research body that's helping you decide. And once you finish your research and you have decided, then that's when you actually have to go out and do your actual action. Of booking your ticket. That way I believe there's a nice sweet spot of one product providing you both the navigational search experience as well as the And surengine experience together. And that's what we strive to be doing.

11:16 Can you talk about the relevant pieces of third party technology that make something like this possible and how you blend them with your own technology. So we could talk about Retrieval augmented generation here. We could talk about the various lineup of open AI versus Cloud versus Google and how you think about the underlying LLMs that you can use in tap. I'd love to talk about how you think about that as a business person, too. I hear a lot of chat GPT rapper or something like this. Tell us about the stack that you use.

11:45 And don't be afraid to get technical super smart audience. I'm just curious how these things all tie together and work in combination. First of all, we started off with two million dollars in funding. When you have two million dollars in funding You have no job trying to build infrastructure yourself.

12:01 Yeah. Your only goal is to validate If you have a product that people want to use on a day to day basis, or at least a weekly basis. And get enough traction and awareness among users.

12:14 And then Think about building infrastructure that allows you to scale from where you got to ten X, a hundred X, more. No That Is the level to which we were ambitioning at the start.

12:27 And we decided to be a rapper. We want to Do things that allow you to get the product out as quickly as possible. So we decided to be a wrapper. We connected Bing API with GPT three point five API and launch perplexity.

12:41 Anybody could have done that. Except Once you do it, that's when the game begins. The game begins after you got some excitement and some users are using your product even after your initial hype. And you get the sustain usage. Now That's when you say, Hey

12:57 Look, this thing I'd a wrap Together. Is not gonna scale. Sure, Open AI is gonna continue making three point five more scalable and Bing has already done decades of work to make their search engine API scalable.

13:10 But the orchestration layer. that takes both of these together and handles so many queries and can handle any outage in any of these APIs. Requires you to build infrastructure yourself. And that's when we started building infrastructure ourselves. When 10,000 people were on the site at once, when Jack Dorsey tweeted about us. Even though it's a wrapper, it went down.

13:31 Because we don't have the right rate limits with Open AI, we don't have the right rate limits with Bing, or chat GPT goes down frequently. There are all sorts of issues. AWS servers go down. That's when you start to actually build all the foundation layer of your infrastructure and back end.

13:46 to support a more scalable product. And when you start doing that, you keep encountering newer issues every time. And every time you solve the newer issues, your infrastructure keeps getting more and more robust. And after a year, if you look back, you're like, Oh damn, this is impossible how far we have come. Like we would have never imagined we could have built such a sophisticated backend. When we started off with just two or three people.

14:12 Can you explain from an insider's perspective and someone building an application on top of these incredible new technologies? What you think the future might look like, or even what you think the ideal future would be. For how many different LLM providers there are, how specialized they get. scale the primary answer so there's only going to be a few of them.

14:32 How do you think about all this and and where you think it might go? It really depends on who you're building for. If you're building for consumers, you do wanna build a scalable infrastructure. Because you do want to ask many consumers to use your product. If you're building for the enterprise You still want a scalable structure.

14:49 No, it really depends. Are you building for the people Within that company? who are using your product, like say you're building an internal search engine, you only need to scale to the size of the largest organization. Which is like maybe hundred thousand people. And not all of them will be using your thing at one moment. You're decentralizing it, you're gonna

15:08 keep different servers for different companies and you can uh elastically decide what's the level of truth good you need to offer. But then if you're Solving another enterprise's problem. Where that enterprise is serving consumers and you're helping them do that.

15:24 you need to build scalable infrastructure indirectly at least. For example, open AI. Their APIs are used by us, other people to serve a lot of consumers. So unless they solve that problem themselves, they're unable to Help other people solve that problem. Same thing with AWS. So that's one advantage.

15:41 you have of actually having a first party product that your infrastructure is helping you serve. And By doing that, by forcing yourself to solve that hard problem Whatever you build can be used by others as well.

15:55 Amazon built AWS first for Amazon. And because Amazon dot com requires very robust infrastructure. that can be used by so many other people. And so many other companies emerg by building on top of AWS. Same thing happened with Open AI. They needed robust infrastructure to serve the GPD three developer API and

16:14 Chat GPT as a product. Then they can now support other companies that are building on top of them. So it really depends on what's your end goal and who you're trying to serve and what's the scale of your ambition.

16:28 Say a click more about how you view the relative merits. of this, what seems like just a pure arms race that's happening. This context window gets longer, this latency gets lower, the sophistication goes up, the generations of models are dizzying. How do you build the business? in a way that wins no matter what happens at all these companies or however many LLMs there are to make it forward compatible.

16:50 with a rapidly changing environment. I think there's no solution. You obviously need to have a good engineering team and people who are very nimble and can use and learn new things pretty quickly. One thing I would say very inspired by Jeff Bezos.

17:06 is the end user doesn't care what models you're using or what indexes you're using. All they want is a great product experience. None of your users are gonna say, Hey, Arvin, one year from now, I want Your product to be slower. Or hey, one year from now I want your product to be less accurate. One year from now I want your product to be rendering the answer in these huge paragraphs. I don't want a better format. They're not gonna say that they only want these three things to keep getting better, and they don't care how you achieve it.

17:36 So the way we think about this particular question is whatever helps us get there. Ideally it's something we build ourselves because usually for speed you have to build stuff in house. If you rely on others At some point you'll hit the limits of How much you can speed up.

17:55 On the other hand, if you're like building the back in in house You can go even further in terms of making speed a priority for you. Let me give you one example. There are people who build on the Flutter or React native stack so that there's one single code base for both mobile platforms, iOS and Android. But because you choose to do that.

18:15 To minimize engineering overhead for you. The apps might get slower for the end user. More unusable, more large in terms of Memory that it consumes on the Device.

18:26 Things like that, and just makes for a worse user experience. On the other hand, if you build natively If you directly go to Swift UI and build your app. The apps can be feel a lot faster, snappier. Consume less memory.

18:40 And allow you to take more advantage of the native components. Render more natively. Earlier we used to render The answer on perplexity. By using a render on the web? And using the web view to render on the app.

18:53 But then if you render more natively on Swift UI and build components for that. It feels even snappier, even better. Small, small optimizations like these, if you keep shipping them every few weeks, the user loves it. And everyone loves a fast, snappy, accurate, reliable app. And then that increases your retention, your word of mouth, and grows your now a better company than before.

19:14 So I genuinely think backwards from what the user wants and try to do whatever it takes to get there. When I think about the history of the product, which I was a pretty early user of the first thing that pops to my mind is that it solves this hallucination problem, which has become less of a problem, but early on Everyone just didn't know how to trust these things and you solved that. You gave citations, you can click through the underlying web pages, et cetera.

19:37 I'd love you to walk through what you view the major timeline product milestones have been. of perplexity dating back to its start. The one I just gave could be like one example. There is this

19:49 possibility, but there was a problem and you solved it. At least that was my perception as a user. What have been the major milestones as you think back on the product and how it's gotten better. I would say the first major thing we did is really make the product a lot faster. When we first launched the latency for every query was seven seconds. That we actually had to

20:11 speed up the demo video to put it on Twitter so that it doesn't look embarrassing. And one of our early friendly investors, Daniel Gross, who co-invests a lot with Nat Friedman, he was one of our first testers before we even released a product. And he said, you guys should call it a submit button for a query. It's almost like you're submitting a job and waiting on the cluster to get back. It's that slow. And now we are widely regarded as the fastest chatbot out there.

20:38 Some people even Come and ask me, why are you only as fast as ChatGPD? Why are you not faster? And little did they realize that ChatGP doesn't even use the web by default. It only uses it on the browsing mode on Bing. So for us to be as fast as Chat GPT already tells you that

20:55 In spite of doing more work to go pull up links from the web, read the chungs. pick the relevant ones and use that to give you the answer with sources and a lot more work on the rendering. Despite doing all the additional work, if you're managing an end to end latency as good as Chat GPT. That shows we have like even a superior backend to them. So I'm most proud about the speed at which we can do things today compared to when we launched. The accuracy has been constantly going up.

21:22 Primarily two things. One is We keep expanding our index and like keep improving the quality of the index. From the beginning we knew all the mistakes that previous Google competitors did, which is Obsess about the size of your index and focus less on the quality. So we decided from the beginning we would not obsess about

21:40 The size. Size doesn't matter in index, actually. What matters is the quality of your index. What kind of domains are important for AI chatbots and question answering and knowledge workers? That is what we care about. So that decision ended up being right. The other thing that has helped us improve the accuracy

21:58 Was training these models to be focused on hallucinations. When you don't have enough information in the search snippets. Try to just say I don't know instead of making up things. LMs are conditioned to always be helpful.

22:11 Always try to serve the user's query despite what it has. Access to may not be even sufficient to answer the query. So that part took some reprogramming, rewiring. You gotta go and change the weights. You can't just solve this with prompt engineering. So we have spent a lot of work on that.

22:27 The other thing I'm really proud about is getting our own inference infrastructure. So when you have to move outside the open AI models to serve your product Everybody thinks, Oh, you just train a model to be as good as GPT and you're done. But reality is OpenEI's mode is not just in the fact that they have trained the best models. But also that they have the most

22:47 cost efficient scalable infrastructure for serving this on a large scale consumer product like ChatGPT. That is itself a separate layer of mode you can build, tech mode you can build. And so we are very proud of our inference team how fast high throughput, low latency infrastructure we've built for serving our own LLMs. We took advantage of the open source revolution, Lama, and Mistral, and took all these models, trained them to be very good at being great answer bots.

23:14 And serve them ourselves on GPU so that We get better margins on our product. So all these three layers, both in terms of speed, through actual product backend orchestration. accuracy of the AI models and serving our own AI models. We've done a lot of work on all these things.

23:32 Can you explain the early discussions that you and your team had about How to set up the business model. Because while the ten blue links is fundamentally broken user experience, maybe from this point forward, it did play very nicely with an amazing ad based business model. This is a different business model like ChatGPT. You can use a version for free or you can pay for a better version. But I'm really curious.

23:53 How you consider the various different business models. What were the business models that you almost did but didn't do. How do you think about the trade offs of the business model that you chose? This is all happening in real time. We're trying to figure out how to build companies and products around this new technology. And I'd love to hear the early story of how you made those decisions. Honestly, I had no idea of business models when we were initially growing. Of course, investors were all you're growing, keep focused on the growth, don't worry about making money.

24:22 When you're growing, people are willing to support you to keep growing bigger and see how far it can grow before deciding what is the business model to stick to. So until we got to late hundreds of thousands of queries a day, we were not even concerned about making money. But at one point we were really interested to know. Hey. Are you guys getting all this usage because People want to Use

24:46 Some chat CPT alternative when it's down. Or some free GPT four usage. We had like, you know, ten queries a day of GPT four or something at one point. Are you guys just getting all this usage because of um people not wanting to pay for open AI or when open AI is down.

25:02 But you don't actually have real product market fit. So that was a question that we were asking ourselves. And it was a valid question. So How to best answer this question?

25:13 You create a subscription of your product. That has the same Pricing as chat GPT plus. You charge it the exact same thing, twenty dollars a month. And See how many people convert to paying users.

25:27 They cannot be just paying for GPT four because They're getting that in open AI as well. And they cannot just be paying for browsing alone because they're also getting that on open AI. So despite that, why are they coming and paying for you? They come and pay for you because they like your product experience. They like what you offer. And so that needed to be known, or else there's no point raising another round to scale this thing further up and build them.

25:50 Business. 'Cause when you don't really have true product market fit and you're just a subsidy then you shouldn't be raising more money. There's no real PMF there. And that was What we evaluated quickly. We put in a subscription plan, we saw how many people converted to paying users. We hardly even tried to convert them. So

26:09 Despite that fact. a lot of people chose to convert and start paying. And and the business was growing really fast. We just said okay, the subscription model works. It's not the ultimate motto. I don't think that's the final piece. And uh

26:24 Profits that are gonna be generated in the AI chatbot sector. But it's a good start. Everyone is doing that. OpenAI started it. We're doing it. Google's also trying that. Microsoft's trying that. So it's a good start. What's like something that You know, potentially it could be billion dollars in revenue for us. It's already billions of dollars in revenue for open AI.

26:44 Let's start there. And also s when you have sufficient scale of usage, try to think of what advertisements too. What does advertisement in this new medium look like? How would it even work? How would you not compromise the quality of the answer? How would you not compromise the quality of the citations?

27:01 Despite that, how can you help creators of content reach more consumers. So that's an interesting challenge to figure out. So we will We'll also work on that. And expect the others to also keep thinking about these things. And the other businesses like APIs.

27:16 We have our own online L APIs, which is basically An LM that has no knowledge cut off. So it's always live, up to date, real time information, unlike the GPT APIs. And we are slowly expanding access to it and letting other people build on it. For example, rabbit.

27:34 devices are using those APIs. We're also partnering with other devices. Browsers like Arc are using those APIs. So it's small steps towards also building a developer or enterprise focused version of the business. Which can make use of all the infrastructure we built. Similar to how we use OpenEIs infrastructure.

27:53 How does that work? So if you think about it in simple terms, opening I is training this huge model up to a date. And it's using information available up to that date to train the thing, that's the knowledge cut off. How do you continue to have something that valuable? That is up to date, including today's dates to web pages. How does that actually get built? How do you build that?

28:13 It's the same thing as a product. You ask a query and it goes and pulls pages from our index and then uses signals from the web to rank it. And then gets you back the answer that uses Knowledge from these. The bits that I pulled up. in the form of a concise paragraph. So it's whatever happens in the product, it's the same thing.

28:31 Except it's been black boxed to you as an API and then you just send in your request and you get a completion. And you can make it a chat assistant too, so that you can create products that enable this conversational answer engine experience. And people have built WhatsApp assistants using that. ask a question grab it, it can hit the API and give you the answer. So we can do a lot more. The reason our APIs are valuable is because nobody else offers this level of speed and accuracy for an end to end search plus LLM experience.

29:02 Can you expand on Index. You've referenced that a few times for those that haven't built one or haven't thought about this. Just explain that whole concept and the decisions that you've made. And you already mentioned quality versus size, but just explain what it means to build an index, why it's so important, et cetera. Yeah, so what does an index mean? It's basically a Copy of the

29:23 The web has so many links. And you want a cache. You want a copy of all those links. In a database. So a URL and the contents in that URL. Now

29:35 The challenge here is new links are being created every day on the web. And also existing links keep getting updated on the web as well. New sites keep getting updated. So you gotta periodically refresh them. The URL needs to be updated in the cache with a different version of it.

29:51 Similarly, you gotta keep adding new URLs to your index, which means you gotta build a crawler. And then how you store a URL, the contents in that URL also matters. Not every page is native HTML anymore. The web is upgraded a lot. rendered in JavaScript a lot. And every domain has custom ways to render the JavaScript. So you gotta build parsers.

30:13 So you gotta build a crawler, indexer, parser. And that Together makes up for a great index. Now the next step comes to retrieval. Which is

30:24 Now that you have this index Every time you hear a query Which links do you use? And which paragraphs in those links do you use?

30:33 Now that is the ranking problem. How do you figure out what is relevance? Relevance and ranking. And once you retrieve those chunks Like the top few chunks relevant to a query that the user is asking. That's when the AI model comes in. So this is the retrieve part. Now the generate part. That's why it's called Rack Retrieve and Generate.

30:50 So once you retrieve the relevant chunks from the huge index that you have, the AI model will come and read those chunks and then give you the answer. Doing this ensures that You don't have to keep training the AI model to be up to date. What you want the AI model To do is to be intelligent.

31:08 To be a good reasoning model. Think about this is when you were a student, I'm sure you would have written an open book exam, open notes exam in school or High school or college. What do those exams test you for? They don't test you for road learning. So it doesn't give an advantage to the person who has the best memory power.

31:26 It gives advantage to the person who has read the concepts. Can immediately query the right part of the notes. But the questions require you to think on the fly as well. That's what we want to design systems. It's very different philosophy from Open AI, where Open AI wants is one model.

31:43 That's so intelligent, so smart, you can just ask it anything, it's gonna tell you. We rather want to build a small, efficient model. That's smart, capable, can reason on facts that it's given on the fly. and disambiguate different individuals with different names or say if there's not sufficient information, not get confused about dates, when you're asking something about the future, say that must not yap. These sort of corner cases handle all of this with good reasoning capabilities.

32:09 Yep. Have access to all of the world's knowledge in an instant through a great index. And if you can do both of these together into an orchestrated with great latency and user experience. You're creating something extremely valuable. So that's what we want to build.

32:23 If you think about the history of the business so far and every episode of what you've had to build, What stands out in your memory as the most difficult period or thing that was built. What would you least want to go back and live through again in terms of its difficulty and stress? Well, I started working in deep learning in two thousand fourteen. And we were not even right uh doing deep learning in Python at the time.

32:46 Everything was done with C plus plus and Cuda. I was using this framework in deep learning called Cafe that literally where if you have to build a different architecture outside of the traditional CNNs. You had to go and write those layers in C plus plus and CUDA. Recompile the library again. Because everything's in C plus plus it has to be compiled. And after you get a compiled object.

33:09 You write the new neural net using those layers. Create a proto buff file and just Rewrite all the data layers again, and then launch shops. It was a nightmare. Most of the CURA drivers would have to be reinstalled again for

33:23 different GPU cards. There was no standardization. So You would probably spend hundreds of hours just installing CUDA and installing these libraries, changing the layers, changing the libraries. that the amount of patience and willpower you needed to still do all this Uh to succeed in your research was just crazy.

33:43 But it's good. It's a good proxy to test if somebody As a good engineer or not, because Usually people give up very fast. I didn't give up. And of course life got a lot easier once Python based symbolic languages came like Tiano. from Montreal and then TensorFlow from Google. Intensor flow is a pain in the ass too because debugging it was really hard.

34:04 Every time you got something wrong, you had to actually change the graph and not be able to print any intermediate things. This is the core AI deep learning stuff that Was such a pain. When we used to work with and

34:18 I would not want to go back to those days, honestly. How would you explain the feeling of the transformer coming online and what that was like to experience as an engineer. How would you explain what a transformer unlocked? to a person that's less technical.

34:31 Yeah, so what the transformer primarily did is it just made the description length of the architecture of a neural net so minimal. It's very homogenous architecture. Until the transformer you would have A recurrent layer, a convolutional layer, and a bunch of hidden layers.

34:50 You would have to be sophisticated to know the right combinations of them. It's almost like you're cooking a meal, but You have to get the right. Mixtures of so many different parts. that any mistakes anywhere could just cost you so much.

35:03 What the transformer did is One simple model But just two layers. Attention. Matmos attention, matmos alternating each other. That.

35:13 It's the same layer repeated again and again. You just had to decide three or four hyperparameters. And that's it. So It is Lower the barrier to entry.

35:23 You don't have to be a sophisticated neural network expert. to design architectures anymore. Instead the work went more into getting the data right. The architectural problem was solved. You just gotta literally scale it up in terms of layers. A number of hidden dimensions.

35:41 But that's it. More work was spent on getting the tokenizer right, the data right, how the word is converted into the vocabulary, what parts of the internet you're scraping. quality of the data. Which do you leave out? Which do you train on? How do you ablate for

35:57 What do you evaluate on in terms of how do you know the model is good? So that created Oh, the different set of skill sets. Were more like physics. PhDs.

36:08 who had that rigor and experimentation. Less background and ML. To come and have a big advantage right now. And that is the core skill set of the Anthropic team. There's a company called Anthropic. They used to work at OpenAI. Their CEO, Dario Amory, he's actually a physics guy.

36:25 Physics PhD. But he became incredibly skillful leading teams like this because of his background. And he hired people like that. He hired people who were having physics background to come work with him. And they built a whole GPT three Testing for Q capabilities.

36:41 I bleeding clearly at the smaller scale. Forecasting, scaling loss. They brought in this new discipline there. That's what has led to most of the breakthroughs that we see in ChatGPT and all the stuff. Like nobody launches a hundred million dollar run YOLO. You cannot do that. It's most likely gonna fail.

36:58 It's not like how people on Twitter talk, Whoa why is Google not doing this? Why are they not taking all the data that they have and launching a huge model and just Destroying open AI. 'Cause you cannot do that if you just Put Lot of data and

37:12 Moto's gonna be confused. It's gonna look at so much that it's not gonna learn any one thing properly. So there is a sci towards figuring out the right data mixes at smaller scale. Forecasting what will happen if you scale it up. And then rigorously launching larger and larger runs.

37:29 And I believe that It's less about being a great transformer designer. And more about being a great data expert and experimenter. Do you think that the transformer Architecture is here to stay and will remain the dominant

37:41 tool or architecture for a long time. This is a question that everybody ask in the last Six years. Or seven years since Saturday's Transformer came. Honestly.

37:52 Nothing has changed. The only thing that has changed is the transformer became a mixture of experts model. Where there are multiple models and not just a single model. But The core self attention model architecture has not changed. And

38:08 People say there are shortcomings, the quadratic attention Complexities there. But any solution to that incurs cost somewhere else to Most of the people not aware that majority of the computation in a large transformer like GPT three or four.

38:24 It's not even spent on the attention layer. It's actually spent on the matrix multiplies. So if you're trying to focus more on the quadratic part, you're incurring cost in the matrix multiplies, and that's actually the bottleneck in the larger scaling. So honestly, it's very hard to make an innovation on the transformer that can have a material impact at the level of GPT four. complex cost of training those models. So I would bet more on innovations auxiliary layers like retrieval augmented generation. Why do you want to train a really large model?

38:56 When you don't have to memorize all the facts on the internet. When you literally have to just be a good reasoning model. Nobody's gonna value Patrick for Knowing all facts. They're gonna value you for being an intelligent person. Fluid intelligence. If I give you something very new that nobody else has any experience in.

39:13 Are you well positioned to learn that skill fast and start doing it really well? When you hire a new employee, what do you care about? Do you care about how much they know about something, or do you care about whether you can give them any task and they would still Get up to speed and do it. Which employee would you value more? So that's the sort of intelligence that we should take into these models and that requires you to think more on the data. What are these models training on? Can we make them train on something else than just memorizing all the words on the internet?

39:41 Can we make reasoning emerge in these models through a different way? And that might not need innovation on the transformer. That might need innovation more on what data you're throwing at these models. Similarly, another layer of innovation that's waiting to happen is the architecture like sparse versus dense models. Clearly a mixture of experts is working. GPT four is a mixture of experts. Mixed Ral is a mixture of experts. Gemini 1.5 is a mixture of experts. So Even there it's not.

40:07 One model for coding, one model for reasoning and math, one model for history. that depending on your input, it's getting routed to the right model. It's not that sparse. Every individual token is routed to a different model. But it's happening every layer. So it's still spending a lot of compute. How can we create something that's actually

40:26 hundred humans in one company. So the company itself as an aggregate is so much smarter. We're not created the equivalent by the model layer. More experimentation on the sparsity. and more experimentation on how we can make reasoning emerge in a different way. is likely to have a lot more impact.

40:42 Then thinking about what is the next transformer. I'm curious then to think about bottlenecks in two ways. So Bottlenecks specific to perplexity and what it wants to build. And your perception of what the bottlenecks are in AI writ large. If you had to answer for both, what do you think the number one bottleneck to progress is?

41:01 In both those cases. I would say for us perplexity, the main bottleneck today is Just getting reasoning to emerge in smaller models. If that happens, the cost per query This is gonna go down tremendously.

41:16 If you don't need GPT four for being accurate. A rough statistic. It's not actually Rigorous. Let's say a model like GPT three point five or

41:26 a mixed trial fine tune version of that that's matches three point five gets eight out of ten queries or something like Seven out of ten queries, no hallucinations. And GPT four will get ninety nine or a hundred. The accuracy rate is so much better in the long tail.

41:42 Now if I can get a three point five or mixed trial model to be as good as four. And hallucinations. Which is basically connected to reasoning capabilities. When you don't have enough information, just say no. Did you use it? Then that makes a tremendous impact.

41:56 I no longer need to serve a large model anymore. And the service can be run way more profitably. That sort of a skill is lacking in smaller models, and that's connects to the first point I made about making how do you train models in a different way so that The most reasoning capable model shouldn't necessarily be the largest model. That hasn't happened yet.

42:15 And if that correlation breaks, I think it'll have a huge impact. The other impactful scenario In general for the field, not just specific to perplexity is Synthetic data. What happens when all the data on the Internet

42:30 Saturated. We've trained on all of it. that every new data set that's being created on the internet doesn't add a lot of value to it's very marginal. How can you make these models create the next generation of the data for themselves. And recursively improvement.

42:44 I'm not talking about scenarios where these models are going to go rogue and start thinking for themselves and take over humanity or something. Very simple. experiment where GPT five or designed by GPT four largely instead of human annotators. Now this will have an impact because I spend a lot on human annotation for hallucinations. I don't have to. I can have the model do it. I can have a smarter model do it for me.

43:10 So I can spend less and get data annotated faster. I can make improvements on my core models that are sold on production much faster because an AI can look at million queries, figure out what's wrong, annotate what is wrong, and tell my smaller AI model to train on them, and I can finish the training run in a week. Instead of doing it over a month. That way my users get to feel the product getting better much faster.

43:35 Accumulate more users that way. And I get more data, the improvements on the product can be tremendously faster. So I think b both of these will have a lot of impact just not just on us, any other startup. Synthetic data and reasoning smaller models. You talked about how in search

43:51 speed, latency, and accuracy are obviously two things that you focused on a ton. I'd love to talk about When you think they'll become customized. more context aware of who I am relative to like the next perplexity user, how that happens. And also when they become more agenti, when they can actually start doing stuff for me.

44:09 'Cause if you think about the broken ten links, you could argue the answer engine's broken too though. the end of the day I just want to have an idea and have a action happen. How do you think those next two components of context awareness. And

44:24 Agent behavior. Might start to find their way into models like yours. First, let's start with context awareness. Maybe we call it personalization. More hyper personalized versions of perplexity.

44:36 Let's achieve very simple things. Location. Gender. Age. Will already solve a lot of personalization for you.

44:45 For what's worth it. People think you need to literally put all of your activity on the prompt. For what it's worth, these models get confused when you throw a lot of information at them. People advertise long contacts a lot, but The more you throw at these models in the context

45:00 The more confused they get in terms of what to focus on. So Personalization can be done. When you know exactly what to retrieve. From your past. And focus more on those.

45:12 highest order bits like location, gender, and demographics. And create a much better experience that's more uh catered to you than the average user. I think we can already do this this year. And we we focus on doing that. The second part.

45:27 Agentic versions of perplexity. We believe that's likely to happen very fast. Let's say there's three parts Towards taking an action. You do your research. You make your decision and then you take your action.

45:38 We are doing the first part pretty well, allowing you to do your research. The decision you're still exercising, do you want to decide? And the final part, the action you just want to task the AI as if it was your executive assistant. That's what you want to do today. No, you can go a step further and say, I don't even want to do

45:56 Decision decision. Let the AI decide everything for me. Let the AI do the research for me. Let the AI act for me. I just want to have no agency I just want a student to Watch Netflix all the time. AI is like will work for me. Have meals show up for me. So I think the second part.

46:12 Seems more dystopian. I don't want that to happen, though if people want that to happen, it will likely happen. I think the first part we can work towards that once we have models that are better reasoners than GPT four. Today I can confidently claim that GPT four is not there yet to be a good action bot. That is the biggest reason why

46:32 The GPT plugin store failed. because it cannot handle all these different APIs calls together at once. And therefore didn't work. When you want an action bot to work. It needs to chain a lot of decisions together.

46:47 And handle corner cases. Now why do we still need executive assistance? The reason you need them is because sometimes You're scheduling something with somebody and they might not have availability for the availability you have, and you might want to move some things around because you might want that to happen that week itself. These kind of thinking. and corner cases you want to be able to handle.

47:08 And you don't want the AI to keep coming, Hey Patrick, that guy's busy, what about this? You want your assistant to think and act on your behalf. So that you're able to focus on other things. Now we don't have the AIs that can do this today. GPT four cannot do this today. Maybe four point five can, maybe five can, I don't know.

47:25 When that model is available, definitely we will also add more agent experiences in our product. We are not capable of training those models today. We don't have the budget. We don't have the computer and the talent to do that. I think somebody has to show The proof of existence of such models. And then we can figure out how to get there.

47:43 And until then we have our jobs right in front of us to just reduce hallucinations and improve the research part. You you mentioned the word talent there, which feels like such an important Topic to cover. What is the talent in AI right now as a leader of a business that obviously wants to and needs to recruit

48:02 Awesome talent. This is certainly the most actively fast moving, exciting area in technology today. It's attracting lots of really talented people, but I'm sure you're supply constrained. There's not enough great AI talent out there. So Yeah, just describe the talent war. What's it like? How do you participate in it? Anything you can share would be really interesting.

48:20 Yeah, I would say that If you wanna compete on pre training a large model like GPT four. Anthropic Claude. Mistral.

48:31 Llama. Gemini. That's it. Maybe Elon's XAI. A lot of potential, but not done anything major there yet.

48:41 But that's it. That's over. Everybody who has some chops at doing large scale training Lot of rigorous scaling law analysis. Data experimentation. Are working in these six companies today. Competing for

48:56 Building a competitor in a number seven. You just not only have to raise a lot of money and give these people a big cluster You also have to hire the talent away from them. Zero sum at this point. And people don't want to leave because when you don't have anything

49:12 When they have peers to work with. And when they already have a great experimentation stack and existing models to bootstrap from. For somebody to leave it's a lot of work. You have to offer such amazing incentives and immediate availability of compute. And we're not talking of small compute cluster here.

49:30 I tried to hire somebody from Meta, very senior researcher. And you know what they said? They said Come back to me when you have ten thousand H one hundredths. And you know what? Ten thousand eight hundred billions of dollars. over a five to ten year period.

49:44 Why would I have it? And then how do I create it? Also, by the way, it's not just about having the money to buy these clusters. You need to make it available. Today. And there is a supply chain problem.

49:55 Most of the GPUs are getting booked out one year in advance or two years in advance. So even if you have the money, you have to wait. By the time you waited and got the money and booked the cluster and got it. the guys that are working here have already made the next generation model. And they're like, look.

50:12 The world has changed. I'm already in the next generation. I'll come when the next version of the model is finished training. This time you'll come back to me when you have twenty thousand eight hundreds. So It's a game that Is being played by seven people right now, six people.

50:27 Largely for a five, I would say. My hope is that it gets commoditized. It's not a Hope in vain. There is some good reasons to why it could happen.

50:39 AWS. Ah sure. T C P All have incentive to commoditize this. And be the biggest winners of all these models, actually.

50:48 More than open AI or Anthropic or Mostral. So the cloud service providers And in video, of course. All of them have incentive to commoditize these models. And how so many other businesses make use of these models.

51:04 Until there were a lot of profits. Than just a few people. eating all the profits. That way they get to win the most. So my hope is that since the cloud service providers are the ones bankrolling all these companies, directly or indirectly. They will put it on their clouds.

51:20 For enterprises. And then enterprises can take them and post train them. I'm talking about a different kind of training. The first type of training I've talked about is pre-training. But pre training alone is not enough. You cannot take a model and just put it into a product and do nothing. it'll still fail at a lot of consumer use cases, customer use cases. You have to post train them.

51:40 And address the long tail of issues you get. Serving a product. Now there you actually have A huge advantage. If you have a lot of users.

51:50 Because you have the data fly. So if you have a lot of users and establish the data fly wheel and establish all the tooling and evals to constantly make use of your existing data sets to improve your product. And build that machine, that flywheel.

52:05 and accumulate enough users and a brand, you have tremendous advantage to create a lot of value. And we are focused on that. And that talent doesn't need to be as sophisticated or scarce. as the pre training talent. This talent you can hire people who want to get into AI from other industries.

52:24 Like crypto or like e commerce. And teach them. And there are so many resources out there. They are fast learners, they even can learn themselves. And they can add a lot of value that way. Proplexy is one of the companies doing that. I'm sure there'll be many more companies there.

52:39 If you were to oversimplify it. and ask the investor community what is stopping them from funding more AI application companies, not infrastructure, not LLMs. Companies more like yours. I think they would say, Well we're worried about defensibility. We don't know

52:56 how these companies control their margins and control churn, et cetera, over time if they're very reliant on underlying infrastructure. I'm I'm not saying this is my opinion, I'm just saying A common take. What would you say to the people that want to build

53:11 AI applications as they think about their business model, their defensibility, their sustainable differentiation What advice would you give to other entrepreneurs that want to be successful building apps. Using AI.

53:24 I would say that this is gonna be an issue until you're a monopoly in your sector. Honestly. I have thought about this too and I've expressed my frustration to a lot of people. I'm getting asked the same question that I was asked when I had 10x fewer users. When this is ever gonna stop. Will it stop when I have another 10X more? And the answer that guy gave a pretty successful billionaire entrepreneur was

53:46 No. You'll still be asked. You'll continue to be asked. So get used to this. And you'll only stop getting ask when you are the number one.

53:56 And then what you'll be asked is Oh. Look at these smaller guys trying to do the same thing. When are you gonna crush them? So it's always gonna be a thing.

54:06 I would say The best answer is always This is a hard market. There are big players, but what we are offering is this. Look at what people users are saying. Trust the users that's signal. It is not Unfair for investors to feel

54:20 Like they might be making a mistake. Look at what happened in the Sora text to video release that opening edit. Until then runway ML and Pico were every investor wanted to Peace and those companies.

54:33 Nolik oh What should I do? Because until then their mindset was perplexity is such a dumb investment to make. Because they're directly in the text. Interface. And Ipen AI is not focused on alternate modalities like voice or video

54:49 Therefore I'll go and invest and start doing that. And now does that reasoning apply anymore? I would rather do enterprise chat GPT as an investment because OpenAI doesn't care about Enterprise and then they come and release Chat GPT for Enterprise. Everything has competition and I think at one point you just gotta realize that for one company to be doing so many projects at once, like OpenAI is doing today.

55:13 Definitely not all of them are gonna succeed. Even if they do succeed. Their success doesn't mean your failure. They're trying to create as much value for themselves. Same thing with Microsoft, same thing with Google.

55:26 For you, the only thing you can focus on is fast execution. And be the best in your sector. If those two things are not true anymore, if someone else is kicking your ass there, then you have to be worried. But that is true, anything you do, you're always going to have competition on anything that can create billions of dollars in revenue. Because for these guys, billion dollars in revenue is what they care about. If you're focused on building something that is only gonna be hundreds of millions of dollars in revenue, you're just focusing on building a billion dollar company.

55:54 Then you don't take absolute venture funding. Do it yourself. Try to be bootstrap like how the mid journey guy is doing it. There's a great quote that I've seen you post, which is the successful warrior is the average man with laser like focus, which is a great Bruce Lee quote. Sums up everything you just said. There's one more code, by the way. I don't fear the man who practiced ten thousand kicks once. I fear the man who practiced One kick ten thousand times. And that's what you're trying to do.

56:18 If you have only one thing to protect. You go out of your way to Be the best at it. If you apply that quote to your own experience. The laser like focus piece.

56:28 What have you not done? in order to stay focused. What have been some things that you maybe would love to have tried Or gone and done, and maybe I'll do them in the future, but things that you've actively said no to in order to stay on That laser like focus. Well, we never did image generation.

56:44 We have image generation as part of the answer. If somebody wants to enrich the answer. But not as a chat experience where someone can just Type in a prompt and get an image. And then We've never done free form chat.

57:00 Everybody said, Why don't you just support both modes, like Bing does. Dude, I'm just trying to build a search product here. Everyone's like AIs are not meant for information, man. Hallucination is a feature. You should build products where hallucination is a feature, not a bug. When you're building AI chatbots for search, hallucination is a bug. So you're doomed.

57:20 You should take advantage of the Hallucination being a feature, you should build something like character AI, you should build a GPT store. You should have a travel perplexity, like uh health or like shopping. You should have a store, how it looks on character AI or you should go click on one agent and talk to it. And you should allow people to create their own agents too. And you should work on agents. You should allow people to book restaurants.

57:41 You should go enterprise. You should allow internal search. Honestly. I give a lot of credit to one of my co founders, Johnny Ho. He's actually Running our product.

57:53 Division basically. Um He was a competitive very successful quantitative algorithmic trade or two before starting perplexity. I would give him even more credit.

58:05 at saying no to things than even myself. Because sometimes I'm also tempted How that founder Experimentation energy. Um But the good thing is I'm not very arrogant.

58:17 So when somebody that's Done a lot more thinking about it than me. Says no, we should not do this. Tend to trust their advice. And Of course there are some times I

58:27 not trust despite the saying that we have to go and do this thing. Has happened. But most of the time I Do listen to the the Steve Jobs code of I'm as proud of saying the things I said no to.

58:40 As I am of the things I chose to do. If you think about the Future now. And where this all Might go.

58:49 I'd love you to paint the biggest possible picture for us. In search specifically, not for AI large. Could go any direction. But for you and what you want to build. Or in twenty thirty or whatever, pick your date.

59:01 What gets you the most excited? What potential future states get you the most excited? I would say that disrupting Search categories where We are currently clicking on a lot of links. And having a lot of commercial intent there.

59:17 Would be insanely amazing. And I think it's possible to do that. We'll be working hard on that. So right now perplexity is associated with this amazing knowledge assistant that you use for like fact checks, trivia, learning about things, digging deeper. But more like a research body.

59:34 That experience needs to expand to the average consumer search categories, shopping and travel and insurance and legal and medicine. But that's huge. amount dollars of advertising thrown at Google for these categories, that Google has zero incentive to make these categories good on Gemini. Even if they want it.

59:54 And that's what we want to go and disrupt. What gets you the most worried? But I would be lying if I said I wasn't worried of Open AI was trying to do similar things to us. Brobato and all that is great, but Let's be honest.

1:00:07 I'm definitely worried about chat GPT. Trying to go more in the direction of search. And The only thing that we have going for us is our speed and accuracy and our UX. That is very

1:00:19 much better than what they have, at least regarded by many users that way. Even if it's not my opinion. If they chose to focus more, it's gonna be more like competition there and In that case, the differentiation is gonna come from us executing even faster and better. Because unlike a big company, If you call them a big company, they're actually pretty fast.

1:00:38 So we have to be even faster. So that is one thing I do think about. Unlike what most people say, I'm not really worried about Google. Not because they cannot execute on this. They actually are way better engineers and researchers than us. It is their own business model and Yeah, it's counter positioning thing. Yeah. You've just raised a round from some well known investors and individuals

1:00:59 So I'm sure you have talked to lots of investors that invest in this sort of stuff. If you think about all of those experiences What do you think investors understand best? And Least

1:01:09 about your kind of company. Right now. I think what they don't understand, at least a lot of them is How hard it is to actually create a product like this. Their mental models are all

1:01:22 In the uh Vertical SAS era. Where They found companies that We went more verticalized.

1:01:30 And found customer lock in effects and then Succeeded as a business. What they fail to understand is verticalization might actually be the wrong strategy in AI. Because One

1:01:42 generic bot that can do many things is a lot more valuable to the end user. When people think of AIs They think of the most generic way of interacting, which is natural language powered Whereas when you are going verticalized, there are always going to be certain query categories you cannot handle. But you cannot instruct a human user

1:02:02 To only interact in a constrained way. You have to design the product that way. It takes a lot of product design and verticalization on the Product clear. To still let humans interact in the most freeform way.

1:02:15 yet have a constrained experience. Nobody has succeeded at this. And further, people don't realize that incumbents in the verticals can just sprinkle an AI chatbot within that app. And have all the other layers like product The adapt and Other support like

1:02:31 databases, existing vendors on that one single platform already going for them. Most of the things are handle. So That is one thing that took me a lot of time to explain to people in the beginning. Only one particular investor Said this to me without even me having to explain this.

1:02:47 Mark Andreessen. Mark Henderson talked to me in January two thousand twenty three. And he said. All I'll tell you is

1:02:57 When Google came out There was so much of investor frenzy. In investing in Google for a vertical. You don't know any of those companies today because they all shut down. But they got a lot of funding.

1:03:10 So Don't do that mistake. Everybody's gonna tell you to make perplexity a vertical. So that In their mind.

1:03:18 They're investing in something safe. But If you do that, you're doomed. You better go all the way and go all in. Or you just don't do this company.

1:03:29 I was like, damn, finally one guy. Said exactly what I was thinking. And happens to be the guy who pioneered the browser. So I got a lot of courage from his advice today. From the outside, it's so interesting to watch a company like yours just as a user and it's been a blast to use it and see it progress. Is there anything else that we haven't talked about that you think

1:03:50 would be the most surprising to the non builders out there, that whether that's investors or users or people that aren't actively building Is there anything surprising? about the state of things today that You think we should talk about that we haven't? This is not surprising if you

1:04:08 put in more thought into this, but A lot of people are worried about competition and modes, ensuring they don't get destroyed by open AI. Some amount of thought there is very useful. I'm not Discarding that at all? But if that is this only way you make your decisions on what to do for your company or your product.

1:04:30 You're likely to fail because These guys will do everything under the sun if it's actually valuable worth doing. So as a startup, your only job is to figure out if there's something you can do that has not been done yet already. And that can deliver value to the world.

1:04:45 Like that is truth. Startups are all about finding A truth vector. If it is truth and if it can generate value, there's no reason the existing players don't want to do the same thing too. They will also try. The way you establish your modes continue to execute faster.

1:05:01 Make sure that it's not easy for the other companies to do because it takes a lot of non trivial work. And keep going. On the other hand, if you're like, hey, I wanna build a sales AI co-pilot because open AI is not gonna go start vertical. And Google doesn't care, Microsoft doesn't care. The Salesforce would do it.

1:05:20 So hard. These are things that HubSpot would do it. So just don't be so naive in the way you think about strategy and modes and don't spend so much time in the beginning of the company thinking about strategy. Try to iterate. We made a lot of mistakes. We built sexus sequels, the dumbest idea you can work on, honestly, because nobody even writes Sequo. Eighty percent of the SQL that actually makes money for Snowflake or Databricks.

1:05:44 It's not even being written. It's just power B I generated or already written queries that are constantly periodically running. When you start thinking about these Charging someone based on consumption rather than

1:05:55 Writing new queries. Text or SQL is such a bad idea. Because once the same sequel is being written, you're not making any money out of it. So usually when you s try to think On white paper, a great idea or a whiteboard.

1:06:09 It doesn't happen. Very few people that build companies of that nature and I would say probably all the existing big players. We're all built with iteration and trying out things and stumbling upon something awesome. And then building the strategy around it. Build execution muscle first. Don't try to be a great strategist right away.

1:06:29 Build something. Make sure it has some traction. Get the muscle that You can keep iterating. And then you deserve the right to strategize.

1:06:38 This has actually been said by this other guy called Frank Slootman as well. The Snowflake CEO. Yes, this whole Line in his book Amp it up where You only deserve the right to strategized once you have earned the track record of execution.

1:06:53 And I strongly believe in that. Really, really interesting closing thought. I'm so appreciative of you letting us behind the scenes here into building one of these things very actively. I'm sure it's been stressful and all consuming and very fun and very interesting. I ask everyone that I interview the same traditional closing question, what is the kindest thing that anyone's ever done for you? We were going through the S V B incident? You remember?

1:07:16 the bank collapse. Yes. I was supposed to be on vacation that weekend and I was not. Yeah, we were going through that and I was very stressed and a lot of people checked on me during that time and Nat Friedman just came and said I'll give you the money, don't worry, I'll take care of your payroll. That was very Nice of him to do that at that time.

1:07:35 Him and Daniel obviously have done an amazing job of supporting This ecosystem. Pretty cool. Yeah, exactly. I had no idea because I was actually in Redmond at the time doing some company visit and this whole thing was going on. I was in the airport before I could get on the flight. I tried to wire the money out to my personal account, in fact, because I didn't even have another account. And I check with my lawyer, this is okay. He said this is very emergency just get the money out somewhere, man. Doesn't matter. And Nad and Dan were like, Don't worry, even if that money goes out, we'll fund you. Amazing.

1:08:04 Well, thank you so much for your time and for a great conversation. Thank you, Pat. Um If you enjoyed this episode, check out Join Colossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.

What you see above is a preview of the first minutes. One unlock costs 10 credits and covers this episode forever: full segment and word-level timestamps on this page, plus .txt, .srt, .vtt and word-level JSON downloads, as many times as you like.