Google Part III: The AI Company

0:00 I went and looked at a studio, well, a little office that I was gonna turn into a studio nearby. But it was not good at all. It had drop ceiling so I could hear the guy in the office next to me. You would be able to hear him talking on episodes. Third co host. Third co host. Is it Howard? No, it was like a lawyer. They seemed to be like talking through some horrible problem that I didn't want to listen to, but I could hear every word.

0:22 Does he want millions of people listening to his conversation? Alright. Alright. Let's do a podcast. Let's do a podcast. Who got the truth? Is it you, is it you, is it you Who got the truth now?

0:40 Is it you, is it you, is it you Me down Straight! Another story on the way

0:49 Welcome to the fall twenty twenty five season of Acquired, the podcast about great companies and the stories and playbooks behind them. I'm Ben Gilbert. I'm David Rosenthal. And we are your hosts. Here's a dilemma.

1:01 Imagine you have a profitable business. You make giant margins on every single unit you sell. And the market you compete in. is also giant. One of the largest in the world, you might say.

1:14 But then on top of that, lucky for you You also are a monopoly in that giant market with ninety percent share. And a lot of lock in. And when you say monopoly, monopoly As defined by the US government.

1:27 That is correct. But then imagine this. In your research lab. Your brilliant scientists come up with an invention. This particular invention, when combined with a whole bunch of your old inventions by all your other brilliant scientists,

1:42 turns out to create the product that is much better for most purposes than your current product. So You launch the new product based on this new invention, right? Right. I mean, especially because out of pure benevolence Your scientists had published research papers about how awesome the new invention is

1:59 And lots of the inventions before also So now there's new startup competitors quickly commercializing that invention. So of course, David. You change your whole product to be based on a new thing, right? Uh, this sounds like a movie.

2:13 Yes, but here is the problem. You haven't figured out how to make this new incredible product anywhere near as profitable. As your old giant cash printing business. So maybe You shouldn't launch that new product.

2:29 David, this sounds like quite the uh dilemma to me. Of course, listeners, this is Google today, and in perhaps the most classic textbook case of the innovator's dilemma. Ever. The entire AI revolution that we are in right now.

2:43 is predicated by the invention of the transformer. out of the Google Brain Team in twenty seventeen. So think open AI and chat GPT, Anthropic, NVIDIA hitting all time highs, all the craziness right now. depends on that one research paper published by Google in twenty seventeen. And consider this.

3:02 Not only did Google have the densest concentration of AI talent in the world ten years ago that led to this breakthrough. But today They just about the best collection of assets that you could possibly ask for. They've got a top tier AI model with Gemini.

3:17 They don't rely on some public cloud to host their model. They have their own in Google Cloud that now does fifty billion dollars in revenue. That is real scale. They're a chip company with their tensor processing units or TPUs. Which is the only real scale deployment of AI chips in the world besides NVIDIA GPUs. Maybe AMD, maybe, but these are definitely the top two. Somebody put it to me in research that

3:43 If you don't have A foundational frontier model. Or You don't have

3:51 An AI chip. You might just be a commodity in the AI market. And Google is the only company that has both. Google still has a crazy bench of talent and despite ChatGPT becoming kind of the Kleenex of the era. Google does still own the text box. The single one that is the front door to the internet for the vast majority of people.

4:11 Any time anyone has intent to do anything online. But the question remains What should Google do strategically? Should they risk it all and lean into their birthright to win in artificial intelligence? Or while protecting their gobs of profits from search.

4:27 Hamstring them. as the AI wave passes them by. But perhaps first, we must answer the question. How did Google get here, David Rosenthal? So listeners today. We tell the story of Google.

4:41 The AI company. Woo. You like that, David? Was that good? I love it. Did you hire like a Hollywood script writing consultant without telling me? I wrote that a hundred percent myself with no AI. Thank you very much. No AI. Listeners, if you want to know every time an episode drops, vote on future episode topics or get access to corrections from past episodes, check out our email list. That's acquired.fm slash email. Come talk about this episode with the entire Acquired community in Slack after you listen. That's acquired.fm slash Slack.

5:12 Speaking of the acquired community, we have an anniversary celebration coming up. We do. Ten years of the show. We're gonna do an open Zoom call with everyone to celebrate. Kinda like how we used to do our L P calls back in the day with L Ps. And we are gonna do that on October twentieth, twenty twenty five, at four PM Pacific time. Check out the show notes for more details.

5:37 If you want more acquired, check out our interview show, ACQ two. Our last interview was super fun. We uh sat down with Toby Lutke. The founder and CEO of Shopify. about how AI has changed his life and where he thinks it will go from here. So search ACQ2 in any podcast player. So with that, this show is not investment advice. David and I may have investments in the companies we discuss, and this show is for informational and entertainment purposes.

6:00 Only. David. Google. The AI company. So Ben, as you were alluding to in that

6:06 Fantastic intro. Really? You're really up in the game again. If we rewind ten years ago. From Today. Before the transformer paper comes out.

6:17 Uh Of the following People. As we've talked about before, we're Google employees.

6:24 Founding. Chief Scientist of Open AI. who along with Jeff Hinton and Alex Koshewski had Done the seminal.

6:34 A I work on Alexnet and just Published that a few years before all three of them were Google employees. As was Dario Amade. D.

6:43 Founder of Anthropic. Andre Carpathy, chief scientist at Tesla until recently. Andrew Ng, Sebastian Thrun. Nom Shazir, all the deep mind folks, Demis Asabis, Shane Leg, Mustafa Suleiman, Mustafa. Now, in addition to in the past having been a founder of Deep Mind. Runs AI at Microsoft.

7:02 Basically Every single person Of note. In AI. worked at Google, with the one exception of Jan Lacoon who worked it.

7:11 Facebook. Yeah. It's pretty difficult to trace A big AI lab now. back and not find Google in its origin story.

7:20 Yeah, I mean the analogy here is it's almost as if at the dawn of the computer era itself. A single company like say IBM. had hired Every single person who knows how to code.

7:31 So it would be like, you know, if anybody else wants to write a computer program, oh, sorry, you can't do that. Anybody who knows how to program works at IBM. This is how it was with AI and Google in the mid 2010s. But Learning how to program a computer wasn't so hard that people out there couldn't learn how to do it. Learning how to be an AI researcher.

7:48 significantly more difficult. Right. It was the stuff of very specific PhD programs with a very limited set of advisors. And A lot of infighting in the field of where the direction of the field was going, what was legitimate versus what was crazy heretical religious stuff. Yeah.

8:06 So then yes, the question is how do we get to this point? Well, it goes back to the start of the company. I mean, Larry Page always thought of Google as an artificial intelligence company. And in fact, Larry Page's dad was a computer science professor. And had done his PhD at the University of Michigan in machine learning and artificial intelligence, which was not a popular field in computer science back then.

8:27 Yeah, in fact a lot of people thought specializing in AI was a waste of time because so many of the big theories from thirty years prior to that Had been kind of disproven at that point, or at least people thought they were disproven. And so

8:43 It was frankly contrarian for Larry's dad to spend his life and career and research work in AI. And that rubbed off on Larry. I mean If you squint Page rank, the page rank algorithm that Google was founded upon.

8:57 Is a statistical method. You could classify it as part of AI within computer science. And Larry, of course, was always dreaming much, much bigger. I mean there's the quote that we've said before on this show. in the year two thousand, two years after Google's founding When Larry says artificial intelligence would be the ultimate version of Google.

9:16 If we had the ultimate search engine, it would understand everything on the web. It would understand exactly what you wanted, and it would give you the right thing. That's obviously. Artificial intelligence. We're nowhere near doing that now. However, we can get incrementally closer, and that is basically what we work on here. It's always been an AI company.

9:34 Yep. And that was in 2000. Well One day In either late two thousand or early two thousand one, the timelines are a

9:43 A Google engineer named Georges Herrick. Is talking over lunch. With Ben Gomes famous Google Engineer, who I think would go on to lead search. And a relatively new engineering hire.

9:55 Named Gnome Shazir. Now George was one of Google's first ten employees, incredible engineer, and just like Larry Page's dad. He had a PhD in machine learning from the University of Michigan. And even when George went there, it was still a relatively rare

10:11 contrarian subfield within. Computer science. So the three of them are having lunch. And George says offhandedly to the group that he has a theory from his time as a PhD student. Yeah.

10:23 Compressing data. Is actually Technically equivalent to Understanding it. And the thought process is

10:32 If you can take a given piece of information And make it smaller. Stor it away and then later. reinstantiate it in its original form. The only way that you could possibly do that

10:45 is if whatever force is acting on the data actually understands what it means because you're losing information. Going down to something smaller. And then recreating the original thing. It's like a kid in school. You learn something in school, you read a long textbook. You store the information in your memory, then you take a test to see if you really understood the material.

11:03 And if you can recreate the concepts, then you really understand it. Which kind of foreshadows. Big LMs today. are like compressing the entire world's knowledge into some number of terabytes that's just like the smash down little vector set.

11:18 Little, at least compared to all the information in the world. But It's kinda that idea, right? You can store all the world's information in an AI model in something that is like kind of incomprehensible and hard to understand. But then if you Uncompress it, you can kind of bring knowledge back to its original form. Yep, and these models

11:36 Demonstrate. Understanding, right? Eh, do they? That's the question. That's the question. They certainly mimic understanding. So

11:45 This conversation is happening, you know, this is twenty five years ago. And Gnome, the new hire, the you know, young buck. He sort of stops in his tracks and he's like Well. If that's true, that's really profound. Is this in one of Google's micro kitchens?

11:59 This is in one of Google's micro kitchens. They're having lunch. Where did you find this, by the way? A twenty five year old Uh, this is in in the Plex. This is like a small little passage in Stephen Levy's great book that's been a source for all of our Google episodes, in The Plex. There's a small little throw away passage in here about this,'cause this book came out before Chat GPT and AI and all that. So Gnome kinda latches on to

12:19 Sure. And keeps vibing over this idea and over the next couple of months the two of them decide In the most googly fashion. That they are just gonna stop working on everything else and they're gonna go work on this idea.

12:33 On language models and compressing data. And can they generate machine understanding? with data. And if they can do that, then that would be good for Google. I think this coincides With that period in two thousand one when Larry Page fired all the managers in the engineering organization. And so everybody was just doing whatever they wanted to do. So there's this great quote.

12:55 From George in the book. A large number of people Thought it was a really bad thing. For Nome and I to spend our talents on, but Sanjay Gemawat, Sanjay of course being Jeff Dean's famous prolific coding partner.

13:10 Thought it was cool. So George would posit the following argument to any doubters that they came across. Sanjay thinks it's a good idea. And no one in the world is as smart as Sanjay. So why should Gnome and I accept your view that it's a bad idea? Oh, if you beat the best team in football, are you the new best team in football no matter what? Yeah.

13:32 So All of this ends up taking Nomen George deep down the rabbit hole of probabilistic models for Natural language. Meaning.

13:41 For any given sequence of words that appears on the internet. What is the probability for another specific sequence of words to follow. This should sound pretty familiar for anybody who knows about LLM's work today. Oh, kinda like a next word predictor. Yeah, or a next token predictor, if you generalized it. Yep. So the first thing that they do with this work

14:02 is they create the did you mean Spelling correction in Google search. Oh that came out of this? That came out of this. No created this. So this is huge for Google because obviously it's a bad user experience when you mistype a query and then Need to type another one. But

14:21 It's a tax to Google's infrastructure, because every time these mistyped queries are going, well, Google's infrastructure goes and Serves the results to that query. That are Useless and immediately overwritten with the new one. Right.

14:33 And it's a really tightly scoped problem where you can see like oh wow, eighty percent of the time that someone types in God Groomer, oh, they actually mean dog groomer and they retype it. And If it's really high confidence, then you actually just correct it without even asking them and then ask them if they want to opt out instead of opting in. It's a great feature and it's sort of a great first

14:53 use case for this in a very narrowly scoped domain. Totally. So They get this way and they keep working on it, no man short. And they end up

15:01 Creating a f fairly large from using large in quotes here, you know, for the time. Language model. That they call it. Affectionately.

15:11 Phil. The probabilistic hierarchical inferential learner. These AI researchers love creating their uh Acronyms. They love their word buttons. Yeah. Yep. So fast forward to two thousand three. And Susan Wajiski and Jeff Dean are getting ready to launch Ad sense.

15:32 They need a way to understand the content of these third party web pages, the publishers, in order to run The Google Ad corpus against them. Well Phil is the tool that they use to do it. Hm.

15:46 I had no idea that language models were involved in this. Yeah. So Jeff Dean. Borrows. Phil. and famously uses it to code up his implementation of AdSense in

15:57 A week. 'Cause he's Jeff Dean. And Boom. Adsense. I mean, this is billions of dollars of new revenue to Google.

16:05 Overnight. 'Cause it's the same corpus of Ads better add words that are search ads. That they're now serving on third party pages. They just massively expanded the inventory for

16:15 the ads that they already have in the system. Thanks to Phil. Thanks to Phil. All right. This is a moment where we gotta stop and just give some Jeff Dean facts.

16:24 Jeff Dean is gonna be the through line of this episode of Wait, how did Google pull that off? How did Jeff Dean just go home and over the weekend rewrite some entire giant distributed system and figure out all of Google's problems. Back when Chuck Norris facts were big, Jeff Dean facts became a thing internally at Google. I just want to give you some of my favorites. The speed of light in a vacuum used to be about thirty five miles per hour. Then Jeff Dean spent a weekend optimizing physics. So good. Jeff Dean's pin is the last four digits of pie.

16:55 Only Googlers would come up with these. Yes. To Jeff Dean, NP means no problemo. Oh yeah. I've seen that one before. I think that one's my favorite. Yes. Oh man. So so good. Also a wonderful human being who we spoke to in research and was very, very helpful. Thank you, Jeff. Yes.

17:16 So Language models. Definitely work. Definitely gonna drive a lot of value for Google. And they also

17:24 Fit. pretty beautifully into Google's mission to organize the world's information and make it universally accessible and Useful. If you can understand the world's information and compress it and then Recreate it.

17:36 Yeah, that fits the mission, I think. I think that checks the box. Absolutely. So Phil gets so big. That apparently by the mid two thousands Phil is using fifteen percent of Google's entire data center infrastructure.

17:49 And I assume a lot of that is Ad sense ad serving, but also did you mean and all the other stuff that they start using it. For within Google. So uh Early natural language systems computationally expensive.

18:01 Yes. So okay, now mid two thousands, fast forward to two thousand seven. Which is a very Very big ear. For the purposes of our story.

18:11 Google Had just recently launched. The Google Translate product. This is the era of all the great, great products coming out of Google that we've talked about. Maps and Gmail and Docs and all the Wonderful things that Chrome and Android are gonna come later. They had like a ten year run where they basically launched

18:28 everything you know of at Google except for search. Truly in a ten year run. And then There were about ten years after that from twenty thirteen on. Where they basically didn't launch any new products that you've heard about until we get to Gemini, which is this fascinating thing, but this

18:44 Oh three to twenty thirteen era. Was just So rich with hit after hit after hit. Magical. And so one of those products was Google Translate, you know, not the same

18:54 level of user base or perhaps impact on the world as Gmail or maps or whatnot. But still a magical, magical product. And the chief architect For Google Translate.

19:06 incredible machine learning PhD named Franz Och. So Franz had a background in natural language processing and machine learning, and that was his PhD. He was German, he got his PhD in Germany. At the time. DARPA. The Defense Advanced Research Projects Agency, division of the government.

19:25 had one of their famous challenges going. For Machine translation. So Google and France, of course, enters this. And Franz builds

19:36 An even larger Language model. That blows away the competition in this year's version of the DARPA challenge. This is either two thousand six or two thousand seven. Get a

19:47 Astronomically high blue score for the time. It's called the Bilangual Evaluation Understud is the sort of algorithmic benchmark for judging the quality of translations. At the time. higher than anything else possible.

20:01 So Jeff Dean? hears about this and the work that France and the Translate team have done and it's like This is great. This is amazing. Uh when are you guys gonna ship this in production? Oh, I heard the story. So Jeff and Noam talk about this on the Dwar Cash podcast. Yes. That episode is so, so good.

20:18 And Franz is like. No, no, no, no, Jeff, you you don't understand. This is research. This isn't for the product. We can't ship this model that we built. This is a N gram language model.

20:30 Grams are like number of words in a cluster. And we've trained it. On a corpus of two trillion words. From the Google search index. This thing is so large

20:42 It takes it twelve hours to translate a sentence. So the way the Dartpa challenge worked in this case was You got a set of sentences on Monday. And then you had to submit your machine translation of those set of sentences by Friday. Plenty of time for the servers to run. Yeah. They were like, okay, so we have whatever number of hours it is from Monday to Friday. Let's use as much compute as we can to translate these couple sentences. Hey, learn the rules of the game and use them to your advantage. Exactly. So

21:12 Jeff Dean being the engineering equivalent of Chuck Norris. He's like hmm Let me see your code. So Jeff goes and parachutes in and works with the

21:22 Translate team for a few months. And he re architects The algorithm. to run on the words and the sentences in parallel instead of Sequentially.

21:32 translating a set of sentences or a set of words in a sentence. You don't necessarily need to do it in order. You can break up the problem into different pieces, work on it independently. You can parallelize it.

21:44 And you won't get a perfect translation, but you know, imagine you just translate every single word. You can at least go translate those all at the same time in parallel, reassemble the sentence and like Mostly understand what the initial meaning was. Yep. And as Jeff knows very well because he and Sanjay basically

22:00 Built it with Ursholza. Google's infrastructure is extremely parallelizable. Distributed, you could break up workloads into little chunks, send them all over the various data centers that Google has. Reassemble the projects, return that to the user. They are the single best company in the world at parallelizing workloads across CPUs.

22:20 across multiple data centers. CPUs. We're stalking CPUs here. Yeah. And Jeff's work. With the team.

22:28 Get that. average sentence translation time down from twelve hours To one hundred milliseconds. And so then they ship it in Google Translate. And it's amazing. This sounds like a Jeff Dean fact. Well, you know, it used to take twelve hours and then Jeff Dean took a few months with it, now it's off a hundred milliseconds. Right, right, right, right, right. So

22:48 This is the first large large in quotes here language model. Used in production in a Product at Google. They see how well this works. Like uh.

22:59 Maybe we could use this for other things. Like Predicting search queries as you type. That might be interesting. You know, and of course. A crown jewel of Google's business that also might be interesting application for this. The ad quality score for AdWords.

23:18 Is Literally the predicted click through rate. On You can see how an LM

23:27 that is really good at ingesting information. understanding it and predicting things based on that. Might be really useful for calculating Add quality for Google. Yep.

23:38 which is the direct translation to Google's bottom line. Indeed. Okay. So obviously all that is great on the Language.

23:46 Model. Front. I said two thousand seven was a big year. Also In two thousand seven.

23:53 Begins the sort of momentous intersection. Of several computer science professors. On

24:02 The Google. Campus. So in April of two thousand seven, Larry Page hires Sebastian Thrun. From Stanford. To come to Google.

24:12 And work first part time and then Full time. On Machine learning applications. Sebastian was the head of sale at Stanford, the Stanford Artificial Intelligence Laboratory.

24:24 Legendary AI laboratory. That was Big in the sort of first wave of AI back in the sixties, seventies, when Larry's dad was active in the field. Then actually shut down for a while and then had been restarted and re energized here in the early two thousands and

24:40 Sebastian was The leader. The head of Sale. Funny story about Sebastian, the way that he actually comes to Google

24:47 Sebastian was kind enough to speak with us to prep for this episode. I didn't realize It was basically an aqua hire. He and some I think it was grad students were in the process of starting a company, had term sheets from Benchmark and Sequoia. Yes. And Larry came over and said, What if we just acquire your company before it's even started in the form of signing bonuses? Yes.

25:08 Probably a very good decision on their part. So Sale, this group within the C S department. At Stanford. Not only had Some of the

25:18 most incredible, most accomplished. professors and PhD AI researchers in the world. They also had this stream of Stanford undergrads that would come through And work there. as researchers while they were working on their C S degrees or symbolic system degrees or you know, whatever it was that they were doing as Stanford undergrads.

25:37 One of those people Was Chris Cox. Who's the chief product officer at Meta. Yeah. That was Kinda how he got his start in All of this and AI and obviously Facebook and Meta are gonna come.

25:50 Back into the story here in a little bit. Wow. You really can't make this up. Another undergrad who passed through sale while Sebastian was there. Was a young Freshman and sophomore.

26:01 Who would later Drop out of Stanford to start A company. That went through Y Combinator's very first Batch.

26:09 In summer two thousand five. I'm on the edge of my seat. Who is this? Any guesses? Uh, Dropbox, Reddit. I'm trying to think who else was in the first batch. Oh no, but way more on the nose for this episode.

26:23 The company was a failed local mobile social network. Oh Sam Altman, looped. Sam. All That's amazing. He was at sale at the same time. He was at sale. Yep. That's not a grad researcher.

26:40 Wow. Wild, right? We told you that It's a very small set of people that are all doing all of this. Man, I miss those days. Sam presenting at the WWE C with Steve Jobs on stage with the double pop collar. Right. Different time in tech. The double popped collar. That was amazing. That was a vibe. That was a moment. Oh man.

27:00 All right. So April two thousand seven, Sebastian comes over from sale. Into Google Sebastian thread. One of the first things he does

27:08 over the next set of months. is a project called Ground Truth. For Google Maps. Which is essentially Google Maps.

27:16 It is essentially Google Maps. So before ground truth, Google Maps existed as a product. But they had to get all the mapping data from a company called Tele Atlas. I think there were two, they were sort of a duopoly. Navtech was the other one. Yeah, Navtech and Tele Atlas. But it was this like Kind of crappy. source of truth map data that everyone used and you really couldn't do any better than anyone else'cause you all just used the same data. Yeah.

27:38 It was Not that good. And it cost a lot of money. Tell Atlas and Navtec were multi billion dollar companies. I think maybe one or both of them were public at some point and got acquired, but

27:50 Lot of money. Lotta revenue. Yeah. And Sebastian's first thing was street view, right? So he already had the experience of orchestrating this fleet of all these cars to drive around and take pictures. Yes. So then.

28:01 Coming into Google. Ground truth is this sort of moonshot type project. To recreate All the tele atlas data.

28:10 Mostly from their own photographs. Of streets from Street View. And they incorporated some other data. There was like census data they used. I think it was forty something data sources to bring it all together, but ground truth was this Very ambitious effort to Create

28:24 New maps from Whole Cloth. Yep. And just like All of the AI and AI enabled projects within Google that we're talking about here. Works very, very well. Very quickly. Huge win.

28:35 Well Especially when you hire a thousand people in India to help you uh sift through all the discrepancies in the data and actually hand draw all the maps. Yes, we are not yet in an era of a whole lot of AI automation. So on the back of this win with ground truth, Sebastian starts lobbying to Larry and Sergey.

28:52 Hey, we should do this a lot. We should bring in A I professors, academics, I know all these people. into Google part time. They don't have to be full time employees. let them keep their posts in academia, but come here and work with us on Projects for our products.

29:08 They'll love it. They get to see their work used by millions and millions of people. We'll pay them. They'll make a lot of money. They'll get Google stock. And they get to stay. Professors at their academic institutions Win win win. Win win win. So as you would expect, Larry and Sergey are like, Yeah, yeah, yeah, that's a good idea. Let's do that. More of that.

29:25 So in December of two thousand seven Sebastian brings in a Relatively. Little known.

29:33 Machine learning. Professor. From the University of Toronto. Name Jeff. Hinton.

29:39 to the Google campus to come and give a tech talk. Not yet hiring it, but come give a tech talk to, you know, all the folks. At Google. And talk about Some of the new work, Jeff, that you and your PhD and postdoc students there at the University of Toronto are doing on

29:56 Blazing new paths with neural networks. And Jeff Hinton, for anybody who doesn't know the name now very much known as the godfather of neural networks and really the godfather of kind of the whole direction that AI went in. Modern AI. He was kind of a fringe academic.

30:13 Yeah. At this point in history, I mean neural networks were Not a respected sub tree of AI. No, totally not.

30:21 And part of the reason is There had been a lot of hype. thirty, forty years before around neural networks. That just didn't pan out. So it was effectively, everyone thought disproven and certainly backwater.

30:35 Yeah. Ben, do you remember from our NVIDIA episodes my favorite piece of trivia about Jeff Hitton. Oh yes, that his grandfather, great grandfather was George Boule.

30:47 Yeah. He is the great great grandson of George and Mary Boole. Who invented Boolean algebra and Boolean logic. Which is hilarious now that I know more about this because That's the basic building block of symbolic logic of

31:01 defined deterministic computer science logic. And The hilarious thing about neural nets is it's not. It's not symbolic AI. It's not I feed you the specific instructions and you follow a big if then. Tree.

31:15 It is non deterministic. It is the opposite of that field. Which actually just underscores again how sort of heretical this branch of machine learning and computer science was. Right. So

31:25 Ben, as you were uh Neural network's not a new idea. And had all of this great promise in theory. But in

31:35 Practice Just took too much computation to do multiple layers. You could really only have a single or maybe small single digit number of layers. in a computer neural network up until this time.

31:48 But Jeff and his former Postdoc, guy named Jan Lacoon. start evangelizing within the community. Hey, if we can Find a way to have

31:59 multi layered deep layered neural network, something we call deep learning. We could actually realize the promise here. It's not that the idea is bad, it's that The implementation.

32:11 Which would take a ton of compute to actually do all the math, to do all the multiplication required to propagate through layer after layer after layer of neural networks to sort of detect and understand and store patterns. If we could actually do that. A big multi-layered neural network. would be very valuable and possibly could work.

32:33 Yes. Here we are now in two thousand seven, mid two thousands, Moore's Law has increased enough that You could actually start to try to test some of these stuff. Theories. Yeah.

32:43 So Jeff comes and he gives this talk at Google. It's on YouTube. You can go watch it. We'll link to it in the show notes. This is incredible. This is an artifact of history sitting there on YouTube. And Sebastian, Jeft D, and all the other folks who are talking about they get Very, very, very excited.

33:01 Because they've already been doing stuff like this with Translate and the language models that they're working with. That's not using Deep neural networks that Jeff's working on. So here's this whole new architectural approach. That if they could get it to work. would enable these

33:17 models that they're building to Work way better. more sophisticated patterns, understand the data better. Very, very promising. Again, kind of all in theory at this point. Yeah.

33:28 So Sebastian Throne Brings Jeff Hinton into the Google fold after this tech talk, I think. first as a consultant over the next couple of years and then This is amazing. Later Jeff Hinton technically becomes an intern at Google. Like that's how they get around the part time, full time policies here.

33:48 Yep. He was a summer intern in somewhere around twenty eleven, twenty twelve. And mind you, at this point, he's like sixty years old. Yes. So in the next couple of years after two thousand seven here. Sebastian's concept of bringing these computer science, machine learning academics into Google as contractors or part time or interns. Basically letting them keep their academic posts and work on

34:12 big projects for Google's products internally. Goes so well that by Late two thousand nine. Sebastian and Larry and Sergey decide, hey, we should just start A whole new division.

34:24 Within Google. And it becomes Google X. The moonshot factory. The first project within Google X, Sebastian

34:33 leads himself. Ooh, David, don't say it. Don't say it. I won't say the name of it. We will come back to it later. But for our purposes for now The second project. Would be

34:42 Critically important. Not only for our story, but to the whole world. Everything in AI. Changing the entire world. And that second project. It's called Google.

34:54 Brain. All right listeners. Now is a great time to talk about a new partner of ours here on Acquired, Lagora. The agentic operating system that is redefining how the world's best legal teams work. Yep. It's sort of obvious that AI is gonna completely change the legal industry. I bet most of you listening have dropped a contract into some sort of AI chatbot out there. Ligora took that insight and asked the question, what if you really built something with that power from the ground up for the legal industry?

35:25 So the founders did exactly what great founders do. operate with obsessive customer focus. They embedded inside a massive law firm. For months. They sat with the lawyers just watching how the work really gets done.

35:40 And that's how you get features that customers love, like tabular review, where you drop in a folder of hundreds of contracts and it pulls every key term into a grid a lawyer can actually work with. Legor's Bet Here is interesting, since it lets each lawyer handle more complexity, any given person can increase the quality of their work and do higher value work. And this means that the pie can grow even as each individual task takes less time. And they recently launched Lagora Agent, offering greater intelligence and performance. The agent lets lawyers set an objective. Then it can handle the planning and the execution and delivery of the final product. Legal teams get to maintain full control and transparency since they're still involved where judgment is required. And Lagora works where you already work. You can use it within Microsoft Word while redlining or drafting. The early Lagora numbers essentially speak for themselves. When they have a head to head pilot with their top competitor, they win seventy percent of the time. Legora now has over a hundred thousand lawyers. on the platform from twelve hundred legal teams in fifty countries.

36:45 And crazily, they went from one million to a hundred million in ARR. About. Eighteen months. Mm.

36:53 truly insane numbers. And that is the real test. Plenty of things demo well, but the question is whether a busy associate actually reach for it during crunch time, or whether a partner trusts it before going into a conversation with a major client. If your legal team wants to check it out, whether you're a law firm or you're in-house at a company, you can learn more at Lagora.com slash acquired. And just tell him that Ben and David sent you. All right, David.

37:19 So Google Brain. So When Sebastian left Stanford full time and join Google full time.

37:27 Course somebody else had to take over. Sale. And the person who did is a Another computer science professor. Brilliant guy named Andrew Ng.

37:35 This is like all the hits. All the hits. This is all the AI hits on this episode. So what does Sebastian do? He recruits Andrew to come part time. Start spending a day a week on the Google campus. And this coincides right with the start of X. And Sebastian formalizing.

37:53 This division. So one day in twenty ten, twenty eleven time frame. Andrew's spending his day a week on the Google campus and he bumps into who else? Jeff Dean. And

38:04 Jeff Dean is telling Andrew about what? he and Franz have done with language models and what Jeff Hinton is doing in deep learning. Of course Andrew knows all this, and Andrew's talking about what he and Sale are doing at Stanford. And they decide. You know The time might finally be right.

38:20 To try and take a real Big swing on this within Google. And build a Massive. Really large.

38:29 Deep learning model. In the On highly paralyzable. Google infrastructure.

38:37 And when you say the time might be right, Google had tried twice before and neither project really worked. They tried this thing called Brains on Borg. Borg is sort of an internal system that they use to run all of their infrastructure. They tried the Cortex project, and neither of these really worked. So there's a little bit of scar tissue in the sort of research group at Google of Are large scale neural networks actually gonna work for us on Google infrastructure? So

39:03 The two of them, Andrew Eng and Jeff Dean, pull in Greg Carado. who is a neuroscience PhD and amazing researcher who was already working at Google. And in two thousand eleven The three of them.

39:15 Launch. The second official project within X. Probably enough call. Google Brain. And the three of them get to work building a

39:24 Really, really Big Deep. Neural. Network.

39:29 model. And if they're gonna do this, they need a system to run it on. You know, Google is all about taking this sort of frontier research. And then doing the architectural and engineering system to make it actually run. Yes. So

39:43 Jeff Dean is working on this. system on the infrastructure. Name. The infrastructure. Dist belief.

39:52 Which of course is a pun. Both on the Distributed nature of the system and also on, of course, the word disbelief because Noah thought it was gonna work. Most people in the field thought this was not gonna work. And most people in Google thought this was not gonna work. And here's a little bit on why, and it's a little technical, but follow me for a second.

40:11 All the research from that period of time pointed to the idea that you needed to be synchronous. So all the compute needed to be sort of really dense, happening on a single machine with really high parallelism. Kind of like what GPUs do. that you really would want it all sort of happening in one place. So it's really easy to kind of go look up and see, hey, what are the computed values for Everything else in the system before I take my next move.

40:35 What Jeff Dean wrote with Dist belief Was the opposite. It was distributed across a whole bunch of CPU cores. And potentially all over a data center or maybe even in different data centers. So in theory, this is really bad because it means you would need to be constantly waiting around on any given machine for the other machines to sync their updated parameters before you could proceed.

40:57 But instead The system actually worked asynchronously. without bothering to go and get the latest parameters from other cores. So you were sort of updating parameters on stale data. You would think that wouldn't work.

41:10 The crazy thing is It did. Yes. Okay. So you've got disbelief. What do they do with it now?

41:17 They want to do some research. So they try out can we do cool neural network stuff. And what they do. In a paper that they submitted in twenty eleven, right at the end of the year.

41:29 Is And I'll just give you the name of the paper first. Building high level features using large scale unsupervised learning. But everyone just calls it the cat paper. The cat paper. You talk to anyone at Google, you talk to anyone at AI, they're like, Oh yeah, the cat paper. What they did

41:46 Was They trained a large nine layer neural network. To recognize cats. from unlabeled frames of YouTube videos Using sixteen thousand CPU cores on a thousand different machines.

42:01 And listeners, just to like underscore how seminal this is. We actually talked with Sundar in prep for the episode. And he cited seeing the cat paper come across his desk as one of the key moments that sticks in his brain. in Google Story.

42:15 Yeah. A little later on they would do a T G I F where they would present. the results of the cat paper and you talk to people at Google like that DJF. Oh my God, that's when it all changed. Yeah, it proved that large neural networks could actually learn meaningful patterns. without supervision and without labeled data.

42:34 And not only that. It could run. on a distributed system that Google built. to actually make it work on their infrastructure. And that is a huge unlock of the whole thing. Google's got this big infrastructure asset.

42:47 Can we take this theoretical computer science idea that the researchers have come up with and use dyst belief to actually run it on our system. Yep. That is the amazing technical achievement here. That is Almost secondary.

43:01 to the business impact of The cat paper. I think it's not that much of a leap. To say that the cat paper Led to probably hundreds of billions of dollars.

43:14 Of revenue. Generated by Google and Facebook and by dance over the next Decade. Definitely pattern recognizers in data.

43:23 So UT had a big problem. At this time. which was that people would upload these videos and there's tons of videos being uploaded to YouTube.

43:33 But people are really bad at describing what is in the videos that they're uploaded. And YouTube is trying to become more of a destination site, trying to get people to watch more videos, trying to build a feed, increase dwell time, et cetera, et cetera. And the problem is the recommender is trying to figure out what to feed and it's only just working off titles and descriptions that people were writing about their own videos. Right. And whether you're searching for a video or they're trying to figure out what video to recommend next.

44:02 They need to know what the video's about. Yep. So the cat paper proves That you can use This technology, a deep neural network.

44:11 Running on disbelief. To go inside of the videos in the YouTube library and understand what they were. About And use that data.

44:22 To then figure out what videos to serve to people. If you can answer the question cat or not a cat, you can answer a whole lot more questions too. Here's a quote from Jeff Dean about this. We built a system that enabled us to train pretty large neural nets through both model and data parallelism. We had a system for unsupervised learning on ten million randomly selected YouTube frames, as you were saying then. It would build up unsupervised representations based on trying to reconstruct the frame.

44:48 from the high level representations. We got that working and training on 2000 computers using sixteen thousand cores. After a little while that model was actually able to build a representation at the highest neural net level. where one neuron would get excited by images of cats. It had never been told what a cat was.

45:07 But it had seen enough examples of them in the training data of head on facial views of cats that that neuron would then turn on for cats and not much. Else. It's so crazy. I mean, this is the craziest thing about unlabeled data, unsupervised learning. That A system can learn what a cat is without ever being

45:25 Explicitly told what a cat is. And that there's a cat neuron. Yeah. And so then there's a iPhone neuron and a San Francisco Giants neuron and all the things that YouTube recommends. Not to mention porn filtering, explicit content filtering.

45:40 Not to mention. Copyright identification and enabling revenue share with copyright holders. Yeah. This leads to Everything in YouTube. Basically puts YouTube on the path to

45:50 today becoming the single biggest property on the internet and the single biggest media company In the planet. This kicks off a ten year period from Twenty twelve, when this happens. Until chat GPT on November thirtieth, twenty twenty two.

46:06 When AI is already shaping the human existence for all of us and driving hundreds of billions of dollars of revenue. It's just in the YouTube feed. And then Facebook borrows it and they hire Jan Lacoon and they start Facebook AI Research.

46:20 And then they bring it into Instagram. And then TikTok and Bite Dance take it. And then it goes back to Facebook and YouTube with reels and shorts. This is the Primary way that humans on the planet

46:30 spend their leisure time for the next ten years. This is my favorite David Rosen thaulism. Everyone talks about twenty twenty two onward as the AI era, and I love this point from you that Actually for anyone that could make good use of a recommender system and a classifier system, basically a company with a social feed, the AI era started in twenty twelve. Yes, the AI era started in twenty twelve. Mm.

46:52 Part of it was the cat paper. The other part of it. Was what Jensen and NVIDIA. always calls the big bang moment.

47:01 For AI. Which was Alex Nett. Yes. So we talked about Jeff Hinton. Back.

47:08 Uh the University of Toronto. He's got two Grad students. who he's working with in this era.

47:15 Alex Krashewski. And Ilya. Sutskeeper. Co founder and chief scientist of Open AI. And the three of them.

47:25 Jeff's Deep neural network. Ideas and algorithms. To create an entry.

47:34 for the Famous image net. Competition. In computer science. This is Fay Fey Lee's thing from Stanford. It is a annual

47:44 Machine vision. Algorithm. Competition. And What it was was Fey Fey had assembled

47:51 Of fourteen million. Images. That were hand labeled. Famously.

47:58 She used Mechanical Turk on Amazon, I think, to Get them all hand labeled. Yes, I think that's right. And so then the competition was what team can write the algorithm. That without looking at the labels, so just seeing the images.

48:10 could correctly identify The largest percentage. The best algorithms that would win the competitions year over year. We're still getting more than a quarter of the images wrong. So like seventy five percent success rate. Great.

48:24 Way worse than a human. Can't use it for much in a production setting when quarter of the time you're wrong. So then the two thousand twelve competition. Along comes Alex Snap. It's error rate. was fifteen percent.

48:37 Still high, but a ten percent leap from the previous best being a twenty five percent error rate. all the way down to fifteen in one year. A leap like that had never happened before. It's forty percent better than the next best. Yes. On a relative basis. Yes.

48:54 And why is it so much better, David? What did they figure out that would create a four trillion dollar company in the future? So what Jeff and Alex and Ilya did. Is they knew Like we've been talking about all episode, that deep neural networks had all this potential. And Wars Law advanced enough that you could

49:15 Use CPUs to Create a few layers. They had the aha moment. We re architected.

49:24 This stuff. Not to run On CPUs. But to run on a whole different Class. Of computer chips.

49:34 that were by their very nature highly, highly, highly Parallelizable. Video game. Graphics cards. Made.

49:44 by the leading company in the space at the time. NVIDIA. Not obvious at the time. And especially not obvious that this highly advanced cutting edge.

49:54 Academic computer science research. That was being done on supercomputers, usually. That was being done on supercomputers with incredible CPUs. would use these toy video game cards. That retail for thousand dollars. Yeah, less at that point in time, a couple hundred bucks. So The team. In Toronto.

50:12 They go out to like the local Best Buy or something. They buy two NVIDIA. G Force GTX five eighties. Which were NVIDIA's top of the line.

50:22 Gaming cards. At the time. The Toronto team rewrites their neural network algorithms in CUDA. Nvidia's programming language. They train it on these two off the shelf GTX five eighties.

50:36 And This Is how They achieve their deep neural network. And do forty percent better.

50:43 than any other entry in the image net. Competition. So when Jensen says that this was the big bang moment of artificial intelligence. A he's right. This shows everybody that

50:53 Holy crap, if you can do this with two off the shelf GTX five eighties. Imagine what you could do with more of them or with specialized chips. And B This event is what sets NVIDIA on the path from a somewhat struggling PC gaming accessory maker.

51:10 To the leader of the AI wave and the most valuable company in the world today. And this is how AI research tends to work, is there's some breakthrough that gets you this big step change function. And then there's actually a multi year process of optimizing from there, where you get these kind of diminishing returns curves on breakthroughs. Where the first half of the advancement happens.

51:32 All at once. And then the second half takes many years after that to figure out. But it's rare and amazing and Must be so cool. When you have an idea. you do it and then you realize oh my God, I just found the next giant leap in the field. It's like I unlocked the next level to use the video game analogy. Yes. I leveled up.

51:51 So After Alex Nett. The whole computer science. World is a buzz. People are starting to stop doubting neural networks at this point. Yes.

52:01 So After Alex Nep. The three of them from Toronto. Jeff Hinton, Alex Kashevsky, and Ilya Sitzgiver. Do the natural thing.

52:10 They start a company. Called DNN research, deep neural network research. This company does not have any products. This company has AI researchers.

52:18 Who just won a big competition. And Predictably, as you might imagine. It gets acquired by Google. Almost immediately.

52:27 Oh Are you intentionally shortening this? That's what I thought the story was. Oh, it is not immediately. Oh okay. There's a whole crazy thing that happens. Where

52:37 The first bit is actually from Baidu. Oh, I did not know that. So Baidu offers twelve million dollars. Jeff Henton doesn't really know how to value the company and doesn't know if that's fair. And so he does what any academic would do to best determine the market value of the company.

52:56 He says. Thank you so much. I'm gonna run an auction now and I'm gonna run it in a highly structured manner. We're Every time anybody wants to bid The clock resets and there's another hour where anybody else can submit another bit.

53:11 No way. So I didn't know this. This is crazy. He gets in touch with everyone that he knows from the research community who is now working at a big company who he thinks Hey. This would be a good place for us to Do our research. That includes Baidu.

53:26 That includes Google. That includes Microsoft. And there's one other Facebook, of course. Two year old startup.

53:34 Oh wait, so it does not include Facebook? It does not include Facebook. Think about the year. This is twenty. Twelve? So Facebook's not really in the AI game yet.

53:47 They're still trying to build their own. AI lab. Yeah, yeah,'cause Yan Lacoon and Fairwood start in twenty thirteen. Is it Instagram? Nope.

53:55 It is the most important part of the end of this episode. Wait, well, it can't be Tesla. 'Cause Tesla is older than that. Nope. Well, hoping I wouldn't get founded for years.

54:04 Wow. Okay, you really got me here. What company slightly pre dated OpenAI? doing effectively the same mission. Oh.

54:15 Of course. Of course. Hiding in plain sight. Dmind. Wow. Deep mind, baby. They are the fourth bidder in a four way auction for DNN research. Now of course

54:28 right after the bidding starts, Deep Mind has to drop out. They're a startup. They don't actually have the cash to be able to buy. Yeah. Didn't even cross my mind'cause my first question was like where the hell would they get the money?'Cause they had no money. But Jeff Hinton already knows and respects Demis. Uh even though he's just doing this at the time startup called Deep Mind. That's amazing. Wait, how is Deep Mind in the auction but Facebook is not? Isn't that wild? That's wild. So the timing of this

54:54 Is concurrent with the it was then called nips, now it's called NURIPS. Conference. So Jeff Hinton actually runs the auction.

55:04 From his hotel room at the Harris Casino in Lake Tahoe. Oh my god, amazing. So the bids all come in and we gotta thank Cade Metz, the author of Genius Makers, great book on the whole history of AI, that we're actually gonna reference a lot in this episode. The bidding goes up and up and up. At some point Microsoft drops out, they come back in, told you DeepMind drops out. So it's Baidu and Google really going at the end. And finally at some point

55:30 the researchers look at each other and they say, Where do we actually want to land? We wanna land at Google. And so they stop the bidding at forty four million dollars and just say, Google This is more than enough money. We're going with you. Wow. I knew it was about Forty million dollars. I did not know that whole story.

55:46 It's almost like Google itself and you know the Dutch auction IPO process. Right? How fitting. That's kind of a perfect DNA, yes. Well. And the three of them were supposed to split it thirty three each. And Alex and Ilya go to Jeff and say, I really think you should have a bigger percent. I think you should have forty percent and we should each have thirty, and that's how it ends up breaking down.

56:06 Ah, wow. What a team. Well. That leads to the three of them. Joining.

56:12 Google Brain. Directly. And Turbocharging, everything going on there. Spoiler alert.

56:18 A couple years later Astro Teller who would take over running Google X after Sebastian Throne left. He would get quoted in the New York Times in a profile of Google X. that the gains to Google's core businesses in search and ads and

56:34 Uh. From Google Brain. Have Way more than funded. All of the other bets that they have made.

56:42 within Google X and throughout the company over the years. Uh it's one of these things that if you make something a few percent better that happens to do tens of billions of dollars or hundreds of billions of dollars in revenue. You find quite a bit of loose change in those couch cushions. Yes, quite quite a bit of loose change. But That's not where the AI history ends within Google. There is another

57:03 Very important piece. of the Google. A I story. That is an acquisition. From outside of Google, the AI equivalent.

57:12 Of Google's acquisition of YouTube. What we talked about in a minute ago. Deep mind. All right, listeners, now is a great time to tell you about a longtime friend of the show, Vanta. AI has scrambled the whole security picture.

57:26 It used to be that you proved that you were secure once a year on audit or a static PDF, then everyone would nod and you're done. But in an AI first world, that doesn't hold up anymore. Yep, your risk surface changes every week now. A vendor turns on an AI feature or someone writes in a new model without telling IT. And your posture is different than it was last week, let alone at your last audit.

57:49 Banta's own research found that around seventy percent of companies have this quote unquote shadow AI running with no security review at all. Right. And that's where Vanta comes in. They're the leading agentic trust platform, meaning they've built the thing that closes the gap. And the way that they close that gap is Vanta Agent.

58:07 Think of it as a GRSC engineer, that's governance, risk, and compliance, except that it's software and it doesn't sleep. It finds the issues, drafts the fixes, and cuts the time that you'd spend on vendor assessments in half. In half. Which is exactly why more than sixteen thousand companies today run on Vanta. Companies like Ramp, Cursor, and Snowflake. All stay audit ready and catch the risks that crop up between audits across every vendor.

58:35 Every AI tool. The whole environment. And that's the real value. Trust has to be continuous now, which is why Vanta automates your security, your compliance, and the work to earn and prove trust. We're huge fans of Vanta over here, and literally hundreds of acquired listeners have become Vanta customers at their companies over the years.

58:53 So you can get$1,000 off Vanta at vanta.com slash acquired. That's V-A-N-T-A.com slash acquired for a thousand dollars off. And just tell them. That Ben and David sent you. All right, David. Deep mind.

59:08 I kinda like your framing. The YouTube Of AI. The YouTube of AI for Google. They bought this thing for we'll talk about the purchase price, but It's worth

59:18 What, five hundred billion dollars today? I mean, this is as good as Instagram or YouTube in terms of greatest acquisitions of all time. Hundred percent. So I remember when this deal happened.

59:30 Just like I remember when the Instagram deal happened.'Cause the number was big at the time. It was big. But I remember it for a different reason. It was like when Facebook bought Instagram, like Oh my God, this is Wow, what a tectonic shift in the landscape of Tech. In January twenty fourteen I remember reading on TechCrunch this

59:49 Random news. Right. You're like deep what? That Google is spending A lot of money to buy something in London that I've never heard of. That's Working on Artificial intelligence?

1:00:02 Question mark? Right. This really illustrates how outside of mainstream tech AI was at the time. Yeah. And then you dig in a little further and you're like

1:00:11 This company Doesn't seem to have any products. And It also doesn't even really say anything on its website about what Deep mind.

1:00:20 Is it a Quote and quote. Cutting edge artificial intelligence company. Wait, did you look this up on the Wayback Machine? I did. I did. Oh, nice. to build general purpose learning algorithms For simulations

1:00:34 E commerce. And games. This is twenty fourteen. This does not compute, does not register. Simulations, e commerce, and games. It's kind of a random spattering of Exactly. It turns out though.

1:00:48 Not only was that description. Of what Deep Mind was fairly accurate. This Company. And this purchase of it by Google.

1:00:57 Was the Butterfly flapping its wings equivalent moment. That directly leads to Open AI. Chat GPT.

1:01:06 Anthropic. And basically everything. Certainly Gemini that we know Gemini directly. in the world of AI today. And probably X AI given Elon's involvement. Yeah, of course X AI. In a weird way, it sort of leads to Tesla self driving too. Carpathy. Yeah.

1:01:23 Definitely. Okay. So what is the story here? Deep Mind was founded. In twenty ten.

1:01:29 By a Neuroscience PhD Name Demis Hasabas. Who previously started a video game company? Oh yeah.

1:01:39 And a postdoc. Named Shane Leg. At University College London. And a third co founder. Who was one of

1:01:49 Demis's friends. From growing up. Gustafa Suleiman. This was unlikely, to say the least. This would go on to produce

1:01:58 A night. and Nobel Prize winner. Yes. So Demus.

1:02:05 The CEO. Was a childhood chess prodigy turned video game developer. Who When he was Age seventeen.

1:02:14 In nineteen ninety four. He had gotten accepted to University of Cambridge. But he was too young. And the university told him, Hey

1:02:24 Take a you know, gap year. Come back. He decided that he was gonna go work at a video game developer at a video game studio called Bullfrog Productions for the year. And while he's there, he created The game theme park.

1:02:36 You remember that. It was like a theme park version of Sim City. This was a big game. This was Very commercially successful.

1:02:44 Roller Coaster Tycoon would be sort of a clone of this that would have many, many sequels over the years. Oh, I played a ton of that. Yeah. It sells fifteen million copies. In the Mid nineties.

1:02:56 Wow. Wild. Then after this he goes to Cambridge, studies computer science there. After Cambridge, he gets back into gaming. Found another game studio called Elixir, that would ultimately fail.

1:03:09 And then he decides, you know what, I'm gonna go get my PhD in neuroscience. And that is how Demis ends up at university. College. London. There he meets Shane Leg. Who's there as a postdoc.

1:03:21 Shane is a self described at the time. member of the lunatic fringe in the AI community. In that He believes This is

1:03:34 Two thousand eight, nine, ten. He believes that AI is going to get more and more and more powerful every year. And that

1:03:43 It will become so powerful. Then it will become more intelligent than humans. And Shane is one of the people who actually popularizes the term artificial general intelligence, AGI. Oh, interesting. Which of course

1:03:58 Lots of people talk about now, and approximately zero people were afraid of that. I mean, you had like the Nick Bostrom type folks, but Very few. people were thinking about superintelligence. Or the singularity or anything like that. For what it's worth.

1:04:13 Not Elon Musk. He's not included in that list because Demis would be the one Who Tell Zelon. About this. Yes. We'll get to it.

1:04:23 So Demis and Chain. Hit it off. They pull in Mustafa, Demis's childhood friend, who is himself extremely intelligent. He had gone to the University of Oxford and then dropped out, I think, at age nineteen to Do other start up y type stuff.

1:04:37 So the three of them decide to start a company. Deepmind, the name of course being a reference to Deep learning. Jeff Hinton's work and everything coming out of the University of Toronto. And the goal that the three of these guys have of

1:04:48 actually creating an intelligent mind. With deep learning. Jeff and Illy and Alex aren't really thinking about this yet. As we said, this is lunatic fringe type stuff. Yes.

1:04:59 Alexnet, the cat paper, that whole world Is about Better classifying data. Can we better sort into patterns? It's a giant leap from there to say, Oh, we're gonna create intelligence.

1:05:10 Yes. I think probably some people Almost almost certainly at Google, we're thinking. Oh, we can create narrow intelligence that'll be better than humans at certain tasks.

1:05:20 I mean a calculator is better than humans at certain tasks. Right. But I don't think too many people were thinking, Oh, this is gonna be general intelligence smarter than humans. Right.

1:05:31 They decide on the tagline for the company. is gonna be solve intelligence and use it to solve everything else. Ooh, I like it. I like it. Yeah, yeah. I mean they're they're they're good marketers too, these guys. So There's just one problem.

1:05:46 Two Do what they want to do. Money. Just say it. Money is the problem. Money is the problem. For lots of reasons, but Even more so than uh any other given startup in the twenty ten era. It's not like

1:06:02 They can just go Spin up an AWS instance and like build an app and deploy it to the app store. They want to Build really, really, really, really, really big.

1:06:14 Deep learning neural networks that requires Google size levels of compute. Well it's interesting it actually They don't require that much funding yet. The AI of the time was go grab a few GPUs. We're not training giant LLMs. That's the ambition eventually.

1:06:31 But right now what they just need to do is raise a few million bucks. But who's gonna give you a few million bucks when there's no business plan, when you're just trying to solve intelligence? You need to find some lunatics. It's a tough sell to VCs. Except for the exact thing. As you say they need to find some lunatics. Oh, I chose my words carefully, didn't you? Yeah. We use the term lunatic in uh It's endearing. Yeah. Most endearing possible way here, given that

1:06:57 They were all basically right. So In June twenty ten. Demis and Chain managed to get invited to the Singularity Summit.

1:07:06 In San Francisco, California.'Cause they're not raising money for this in London. Yeah, definitely not. I think they tried for a couple months and learned that that was not gonna be a viable path. Yes. The summit, the singularity summit.

1:07:18 Organized by Ray Kurzweil. Uh future Google employee, I think, chief futurist, noted futurist. Eliezer Yudkowski. And Peter Teal.

1:07:32 Yes. So Demis and Chain are uh excited. About getting this invite. Like

1:07:38 This is probably our one chance. To get funded. But we probably shouldn't just Walk in guns blazing and say Peter, can we pitch you? Yeah.

1:07:47 So They finagle their way into Demis. getting to give a talk on stage. at the summit.

1:07:55 Always the hack. Like this is great. This is gonna be the hack. The talk is going to be our pitch to Peter. And Founders Fund. Peter has just started Founders Fund at this point. Obviously. member of the PayPal Mafia. Very wealthy. I think he had a big Roth IRA at this point is the right way to frame it. Big Roth IRA. That he had invested in Facebook, first investor in Facebook.

1:08:16 He is the perfect target. They architect the presentation. at the summit to be a pitch directly to Peter, essentially a thinly veiled pitch. Shane has a quote.

1:08:27 in Parmi Olson's great book, Supremacy, that we used as a source for a lot of the Steve Mind story. Yeah, and Shane says, We needed someone crazy enough to fund an AGI company. Somebody who had the resources not to sweat a few million and liked super ambitious stuff. They also had to be massively contrarian because every professor that he would go talk to would certainly tell him, absolutely do not even think about funding this.

1:08:53 That Venn diagram sure sounds a lot like Peter Thiel. So They show up at the conference. Demis is gonna give the talk. Goes out on stage, he looks out into the audience.

1:09:03 Peter is not there. Turns out Peter wasn't actually that involved in the conference. No, he's a busy guy. He's a co founder, co organizer, but is a busy guy. Yes. Guys like shoot. Oh we missed our chance.

1:09:17 What are we gonna do? And then Fortune Turns in their favor. They find out. That Peter is hosting an after party.

1:09:25 That night. At his house in San Francisco. They get into the party. Demis. Seeks out Peter and he's like Devin's very, very, very smart.

1:09:34 As anybody who's ever listened to him talk would immediately know. It's like rather than just Pitching Peter. Head on. We come about this.

1:09:43 Obliquely. He starts talking to Peter about Chess because he knows, as everybody does, that Peter Tiel loves chess. And Demis. had been the second highest ranked player in the world.

1:09:54 As a teenager in the under fourteen category. Good strategy. Great strategy. The man knows his chest moves. So Peter's like Hmm.

1:10:03 I like you. You seem smart. What do you do? And Demis explains he's got this. A GI startup, they were actually here. He gave a talk on stage is part of the conference. People are excited about this. And Peter says, Oh

1:10:15 Okay, all right. Come back to Founders Fund tomorrow and give me the pitch. So they do, they make the pitch, it goes well. Foundry's fun. Leads.

1:10:23 Deep minds seed round of about two million dollars. My how times have changed for AI company seed rounds these days. Oh yes. Imagine Leading Deep mind seed around with. Less than two million dollar check.

1:10:37 And Through Peter and Founders Fund. They get introduced. Hey, Elon, you should meet this guy. To another member of the PayPal Mafia, Elon Musk. Yes.

1:10:49 So It's teed up in a pretty Low key way. Hey Elon. You should meet this guy, he's smart, he's thinking about Artificial intelligence. So

1:10:57 Elon says, Great, come over to SpaceX, I'll give you the tour of the place. So Demis comes over for lunch and a tour of the factory. Of course Demis thinks it's Very cool. but really he's trying to reorient the conversation over to artificial intelligence.

1:11:10 And I'll read this great excerpt from an article in The Guardian. Musk told Hasabas his priority was getting to Mars as a backup planet in case something went wrong here. I don't think he'd thought much about AI at this point. Hasabas pointed out a flaw in his plan. I said, what if AI was the thing that went wrong here?

1:11:29 then being on Mars wouldn't help you because if we got there, then it would obviously be easy for an AI to get there through our communication systems or whatever it was. He hadn't thought about that. So he sat there for a minute without saying anything. Just sort of thinking. Hmm.

1:11:44 That's probably true. Shortly after, Musk too became an investor in DeepMind. Yes. Yes, yes. I think it's crazy that

1:11:54 Demis is sort of the one that woke Elon up to this idea of We might not be safe from the AI on Mars either. Right, right. I hadn't considered that. So uh this is the first time the bit flips for Elon of We really need to figure out a safe secure AI for the good of the people. That sort of

1:12:14 Seed being planted in his head. Yeah. Which of course. Is what Deep Minds Ambition is.

1:12:20 We are Here doing research for the good of humanity like scientists in a peer reviewed way. Yep. I think all that is true. Also

1:12:30 In the intervening months to year After This meeting between Demis and Elon and Elon Investing in D Mind. Elon also starts to get Really, really

1:12:43 Excited and convinced about the capabilities of AI. In the near term. And specifically the capabilities of AI for Tesla.

1:12:52 Yes. Like with everything else. in Elon's world. Once the bit flips And he becomes interested.

1:12:59 he completely changes the way he views the world, completely sheds all the old ways and actions that he was taking and it's all about What do I most do to embrace this new world view that I have. And other people have been working on for a while already by this point.

1:13:15 A I Driving Cars. Yep. That sounds like it would be a pretty good idea for Tesla.

1:13:22 Does. So Elon. Starts. trying to recruit as many AI researchers as he possibly can and machine

1:13:31 Vision and machine learning experts. Into Tesla. And then Alexnet happens. And man, Alex Nets. Really, really.

1:13:39 Really good at Identifying. Classifying images and Cat videos on YouTube and the YouTube recommender feed. Well Is that really that different from

1:13:49 A live feed of video from a car that's being driven and understanding what's going on there. Can we process it in real time and look at differences between frames? Perhaps controlling The car? Not all that different.

1:14:02 So Elon's excitement Channeled initially through Deep minded demus. About AI and AI for Tesla. Starts.

1:14:11 Ratching up. Big time. Yeah. Meanwhile, back in London. Deep mind is

1:14:16 Getting to work, they're hiring researchers. They're getting to work on models. They're making some vague noises about products to their investors. Maybe we could do something in shopping, maybe something in gaming like the description on the website. At the time of acquisition. Said.

1:14:34 But mostly what they really, really want to do is just build these models and work on intelligence. And then One day. In late twenty thirteen. They get?

1:14:43 A call. From Mark Zuckerberg. He wants to buy the company. Mark

1:14:50 Has woken up. To everything that's going on at Google. After Alex Net. and what AI is doing for social media.

1:14:59 Femendations. At YouTube. The possibility of what it can do. At Facebook and for Instagram. He's gone out.

1:15:06 And recruited. Yan Lacoon. Defendant. Old postdoc, who's together with Jeff, one of the sort of godfathers of AI and deep learning. And really popularized the idea of convolutional neural networks. the next hot thing in the field of AI at this point in time.

1:15:21 And so with Jan They have created fair. Facebook AI research. Which is a Google Brain rival within Facebook.

1:15:30 And remember who the first investor in Facebook was, who's still on the board. Peter Thiel and is also the lead investor in Deep Mine. Where do you think Mark learned about Deep Mind? Peter Teal. Was it do you know for sure that it was from Pierre? No, I don't know for sure, but like how else could Mark have learned about this startup in London?

1:15:46 I've got a great story of how Larry Page found out about it. Oh, okay. Well we'll get to that in one second. So Mark calls and offers to buy the company. And There are various rumors of how much Mark offered. But according to Parmi Olsen in her book Supremacy.

1:16:02 The reports are that it was up to eight hundred million dollars. Company with no products and a long way from ADI. That squares with what Cade Metz has in his book that the founders would have made about twice as much money from taking Facebook's offer versus taking Google's offer. Yeah. So

1:16:18 Demis, of course, takes this news to the investor group. Which by the way Is kind of against everything the company was founded on.

1:16:27 the whole aim of the company and what he's promised the team is that Deep mind is gonna stay independent. do research, publish in the scientific community. We're not gonna be sort of captured and told what to do by the whims of a capitalist institution.

1:16:40 Yep. So definitely some deal point negotiating that has to happen with Mark and Facebook. If this offer is gonna come through. But Mark is so desperate at this point, he is open to these very large deal point negotiations. such as Jan Lacoon gets to stay in New York. Jan Lakoon gets to stay operating his lab at NYU. Yan Lakoon is a professor. He's flexible on some things.

1:17:02 Turns out Merc is not flexible on letting Demis keep control of DeepMind. If he buys it. Demis sort of argued for w we need to stay separate and carved out and we need this independent oversight board with these ability to intervene, if the mission of Deep mind is no longer being followed and Mark's like

1:17:20 No. You'll be a part of Facebook. Yeah. And you'll make a lot of money. So as this negotiation is going on. Of course the investors in Deep Mind get wind of this.

1:17:33 Elon Finds out about what's going on. He immediately calls up Demis and says I will buy the company right now. with Tesla stock. This is late twenty thirteen, like early twenty fourteen.

1:17:46 Tesla's market cap is about twenty billion dollars. So Tesla stock from then to today is about a 70X run up. Demist and Shane and Mustafa are like Wow, okay, there's a lot going on right now. But to your point

1:18:04 They have the same issues with Elon and Tesla. That they had. With Mark. Elon wants them to come in. And work on.

1:18:13 Autonomous. Driving. For Tesla. They don't want to work on autonomous driving. Right.

1:18:18 Or at least exclusively. At least exclusively. Yep. So Then Demis gets A third call.

1:18:26 Larry Page. Do you want my story of how Larry knows about the company? I absolutely want your story of how Larry knows about the company. All right, so this is still early in Deep Minds life. We haven't progressed all the way to this acquisition point yet. Apparently.

1:18:40 Elon Musk is on a private jet with Luke Nosick, who's another member of the PayPal Mafia and an angel investor in Deep Mind. And they're reading an email from Demis. with an update about a breakthrough that they had where DeepMind AI figured out a clever way to win at the Atari game breakout. Yes. And the strategy it figured out with no human training. was

1:19:02 that you could bounce the ball up around the edges Of the Bricks. And then without needing to intervene, it could bounce around along the top and win the game faster without you needing to have a whole bunch of interactions with the paddle down at the bottom.

1:19:16 They're watching this video of how clever it is. And flying with them. On the same private plane. is Larry Page. Of course, because

1:19:25 Elon and Larry used to be very good friends. Yes. And Larry is like w wait, what are you watching? What company is this? And that's how he finds out. Wow. Yes.

1:19:37 Elon must have been So angry about all this. And the crazy thing is This kinship between Larry and Demis is I think the reason why the deal gets done at Google.

1:19:50 Once the two of them get together. They are like peas in a pod. Larry has always viewed Google. As an AI company. Yep. Demus.

1:19:59 Of course. Deep mind so much as an AI company that he doesn't even want to make any products until they can get To AGI. And Demis In fact, we should share with listeners, Demis told us this when we were talking to him to prep for this episode.

1:20:12 Just felt like Larry got it. Larry was completely on board with the mission of everything that Deep Mind was doing. And There's something else. Very convenient. About Google.

1:20:23 They already have brain. So Larry doesn't need Demis and Shane and Mustafa. And deep mind to come. Work on products.

1:20:31 Within Google. Right. Brain is already working on products within Google. Demis can really believe Larry when Larry says, nah, stay in London. Keep working on intelligence. Do what you're doing. I don't need you to come work on products within Google. Brain is like actively going and engaging with the product groups, trying to figure out, hey, how can we deploy neural nets into your product to make it better? That's like their reason for being. So They're happy to agree to this. And it's working.

1:20:57 brain and neural nets are getting integrated into search, into ads, into Gmail, into Everything. It is the perfect. Home. For Deep Mind.

1:21:07 Home away from home, shall we say. Yes. And and There's a third reason why Google's the perfect fit for Deep Mind. Infrastructure.

1:21:15 Google has all the compute infrastructure. You could ever want. Right there on tap. Yes, at least with CPUs. So far.

1:21:22 Yes. So How's the deal actually happen? Well, after buying DNN research, Alan Eustace, who David you spoke with, right? Yep. was Google's head of engineering at the time. He makes up his mind That's the thing.

1:21:33 He wanted to hire All the best. deep learning research talent that he possibly could. I need a clear path to do so. A few months earlier.

1:21:41 Larry Page held a strategy meeting. On an island in the South Pacific, in Cade Metz's book, it's an undisclosed island. Of course he did. Larry thought that deep learning was gonna completely change the whole industry. And so he tells his team this is a quote

1:21:55 Let's really go big. Which effectively gave Alan a blank check to go secure all the best researchers that he possibly could. So in twenty thirteen he decides I'm gonna get on a plane in December before the holidays and go meet Deep Mind. Crazy story about this. Jeff Hinton, who's at Google at the time.

1:22:11 had a thing with his back where he couldn't sit down. He either has to stand or lay. And so a long flight across the ocean is not doable. But he needs to be there as a part of the diligence process. You have Jeff Hinton, you need to use him to figure out if you're gonna buy a deep learning company. And so Alan Eustace decides

1:22:29 He's gonna charter a private jet. And He's gonna build this crazy custom harness rig so that Jeff Hinton won't be sliding around when he's lay on the floor during takeoff and landing. Wow. I was thinking the first part of this, I'm pretty sure Google has plans. They could just get into Google Play. For whatever reason this was a separate charter. But it's not solvable just with a private plane. You need also a harness.

1:22:54 Right. And Alan is the guy who Set the record for Jumping out of the world's highest Was it a balloon? I actually don't know. The highest free fall jump that anyone has ever done, even higher than that Red Bull stunt a few years before.

1:23:10 So he's like very used to designing these custom rigs for airplanes. He's like, Oh, no problem. You just need a bed and some straps. I jumped out of the atmosphere in a scuba suit. I think we'll be fine. That is amazing. So they fly to London, they do the diligence, they make the deal. Demis has true kinship with Larry.

1:23:27 And It's done. Five hundred and fifty milyen US dollars. There's an independent oversight board that is set up. to make sure that the mission and goals of Deep Mind are actually being followed.

1:23:39 And this is an asset that Google owns today. That again, I think is worth Half a trillion dollars if it's independent. Do you know?

1:23:48 What other member of the PayPal Mafia. gets put on. The Ethics Board. After the acquisition. Reed Hoffman? Reed Hoffman. Has to be.

1:23:56 Given the open AI tie later. We are gonna come back to read in just a little bit here. Yes. So after the acquisition It goes very well very quickly. Famously the data center cooling thing happens where

1:24:09 Deep mind carved off some part of the team to go and be an emissary to Google and look for ways to use DeepMind. And one of them is around data center cooling. Very quickly, July of 2016, Google announces a forty percent reduction in the energy required to cool data centers. I mean

1:24:28 Google's got a lot of data centers, a 40% energy reduction. I actually talked with Jim Gao, who's a friend of the show and actually led a big part of this project. And I mean it was just the most obvious application of neural networks inside of Google. right away.

1:24:43 Pays for itself. Yeah, imagine that paid for the acquisition pretty quickly there. Yes. David, should we talk about AlphaGo on this episode? Yeah, yeah, yeah. I watched the whole documentary that Google produced about it. It's awesome. This is actually something that you would enjoy watching, even if you're not researching a podcast episode and you're just looking to pull something up and spend an hour or two.

1:25:02 I highly recommend it. It's on YouTube. It's the story of how DeepMind, post acquisition from Google, trained a model to beat the world go champion. At go. And I mean everyone in the whole Go community coming in thought there's no chance this guy Lee C doll is so good that there's no way that an AI could possibly beat him.

1:25:22 It's a five game thing and just won the first three games straight. I mean Completely cleaned up. And with inventive new creative moves that no human has played before. That's sort of the big crazy takeaway. There's a moment in one of the games right where it makes a move of people, Is that a mistake? Yeah, move thirty seven. Yeah, yeah.

1:25:41 And then a hundred moves later it plays out and that it was like completely genius. And humans are now learning from Deep Minds. strategy. of playing the game. and discovering new strategies.

1:25:54 A fun thing for acquired listeners who are like, why is it Go? Go is so complicated. Compared to chess, chess has 20 moves that you can make at the beginning of the game in any given turn. And then mid game, there's like 30 to 40 moves that you could make. Go on any given turn has about two hundred. And so if you think combinatorily

1:26:13 the number of possible configurations of the board. is more than the number of atoms in the universe. That's a great demis quote, by the way. Yeah. And so he says, even if you took all the computers in the world and ran them for a million years as of twenty seventeen, that wouldn't be enough compute power to calculate all the possible variations. So

1:26:32 It's cool because it's a problem that you can't brute force. you have to do something like neural networks and there is this white space to be creative and explore. And so it served as this amazing breeding ground. for watching a neural network be creative against a human. Yep. And of course it's totally in with

1:26:51 Demis' background and The DNA of the company playing games. Your demis was chess champion and Then after go, then they Play Starcraft, right? Oh, really? I actually didn't know that. Yeah, that was the next game that they tackle was

1:27:05 Starcraft, a real time strategy game against an opponent. And that'll um Opponent. Yeah.

1:27:13 In open AI. All right listeners. Now is a great time to thank our longtime friend of the show, ServiceNow. If you are running a large enterprise, AI agents are likely spread across every team, and deploying them is uh no longer the hard part. Yeah. The hard part is knowing what permissions they have, what employees are using them for, or what decisions AI is making.

1:27:36 AI security for an enterprise at scale is not a small concern. Like the risks Are real. Exactly. And the challenge with AI is governing it, securing it, measuring it, and making sure that it actually delivers value. That is why Service Now built the AI control tower. Yep. AI control tower gives enterprises a single place to see, manage, govern, and optimize AI across the entire business. And it works with Any AI, not just theirs.

1:28:03 Every device on your network, every permission across every system. Every AI agent visible and secure in one place. And ServiceNow can do this because they've spent more than twenty years building the operational backbone of the enterprise, the workflows, governance, approval, security controls, and institutional knowledge that power how work actually gets done across IT, H, customer service, finance, and security. ServiceNow already runs more than a hundred billion workflows annually and trillions of transactions for more than eighty five percent of the Fortune five hundred. So when companies need a place to govern AI at enterprise scale, they're building on a platform at the center of how their business already operates. And in a future, that isn't going to be one AI, it's going to be thousands of AI agents working across every function of the company. But the question is,

1:28:51 Who's managing them all? So if you're trying to turn AI ambition into real business outcomes and make it work safely, securely at scale, Go check out service now.com slash acquired and tell'em that Ben and David sent you. Alright, David, so What are the second order effects of Google buying DMind?

1:29:09 Well There's one person who is really, really, really upset about this. Maybe two people if you include. Mark Zuckerberg, but Mark tends to play his cards a little closer to the vest.

1:29:21 Of course. Elon Musk is. Barry. Upset. about this acquisition.

1:29:27 When Google buys Deep Mind out from Under him, Elon goes Ballistic. As we said, Elon and Larry had always been Very close. And now

1:29:37 Here's Google who Elon has already started to sour on a little bit as he's now trying to hire AI researchers. And you've got Alan Eustace flying around the world sucking up all of the AI researchers into Google. And Elon's invested in Deep Mind

1:29:53 wanted to bring Deep Mind into his own AI team at Tesla and Gone out from under him. So This leads

1:30:04 In Silicon Valley's history. Organized In the summer of twenty fifteen. At the Rosewood Hotel on Sand Hill Road. Of course, where else would you do a dinner? In Silicon Valley. But the Rosewood.

1:30:19 By two of the most leading figures. In the valley at the time. Elon Musk. And Sam Altman. Sam.

1:30:28 Of course, being president of Y Combinator. Yeah. The time. So what is the purpose of this dinner? They are there.

1:30:38 To make a pitch. To all of the AI researchers. That Google And to a certain extent Facebook. Have sucked up and basically created this duopoly.

1:30:49 Status. Oh. Again. Google's business model and Facebook's business model, these feed recommenders. Or these classifiers.

1:30:57 Turn out to be unbelievably valuable so they can It's funny in hindsight saying this. Paid Tons of money to these people. Tons of money. Like millions of dollars. Take them out of academia.

1:31:09 and put them into their dirty capitalist research labs inside the companies. Selling advertising. Yes. How dirty could you be? And the question and the pitch.

1:31:21 That Elon and Sam. have for these researchers gathered at this dinner is What? Would it take? to get you out of Google.

1:31:31 For you to leave. And the answer they go around the table from Almost everybody is Nothing. You can't. Why would we leave? We're getting paid

1:31:41 Way more money than we ever imagined. Many of us get to keep our academic Positions and affiliations. And we get to hang out. Here at Google with each other. With each other. Iron sharpens iron. These are the some of the best minds in the world getting to do cutting edge research with enormous amount of resources and hardware at their disposal.

1:32:01 It's amazing. It's the best infrastructure in the world. We've got Jeff Dean here. There is nothing you could tell us. Yeah. Would cause us to leave. Google.

1:32:13 Except there's one person. Who Is intrigued. And To quote from an amazing wired article at the time by Cade Metz.

1:32:21 Who would later write Genius Makers, right? Yep, exactly. Quote is the trouble was So many of the people most qualified to solve these problems were already working for Google. And no one at the dinner was quite sure that these thinkers could be lured into a new startup, even if Musk and Maltman were behind it. But

1:32:37 One key player. was at least open to the idea of jumping ship. And then there's a quote from that key player. I felt like there were risks involved. But I also felt like it would be a very interesting thing to try. It's the most Ilya quote of all time.

1:32:52 The most Ilya quote of all time because that person was Ilya. Set's keeper. Course. Of AlexNet. And DNN research and Google.

1:33:02 And About to become founding chief scientist. Of open AI. So the pitch that Elon and Sam are making to these researchers is Let's start a new

1:33:13 nonprofit AI research lab where we can do all this work out in the open. You can publish. free of the forces of Facebook and Google and independent of their control. Yes, you don't have to work on products. You can only work on research.

1:33:28 You can publish your work. It will be open. It will be for the good of humanity. All of these incredible advances This intelligence that we believe is to come. Will be

1:33:40 For the good of everyone, not just for Google. And Facebook. And for money researchers, it seemed too good to be true. So they basically weren't doing it because they didn't think anyone else would do it. It's sort of an activation energy problem. Where once Ilya said, Okay, I'm in. And once he said I'm in, by the way. Google came back with a big counter, something like double the offer.

1:34:00 And I think it was delivered from Jeff Dean personally, and Ilia said, Nope, I'm doing this. That was Massive. for getting the rest of the top researchers to go with him. And it was nowhere near all of the top researchers who left Google to do this, but it was enough.

1:34:14 It was a group of seven or so researchers. Who left Google? And joined Elon and Sam and Greg Brockman from Stripe. Who came over? To create

1:34:25 Open AI. 'Cause that was the pitch. We're all gonna do this in the open. And that's totally what it was. It totally is what it was.

1:34:32 And the stated mission of Open AI was to quote advance digital intelligence. in the way that is most likely to benefit humanity as a whole. Unconstrained by a need to generate Financial.

1:34:46 Return. Which is fine. As long as the thing that you need to fulfill your mission. doesn't take tens of billions of dollars.

1:34:56 Yes. So here's how they would fund it originally. There was a billion dollars pledged. Yes. And that came from famously Elon Musk. Sam Altman.

1:35:07 Read Hoffman. Jessica Livingston, who I think most people don't realize was part of that initial tranche. And Peter Teel. Yep.

1:35:16 Founders fund, of course, would go on to put massive amounts of money into open AI itself later as well. The funny thing is It was later reported that a billion dollars was not actually collected, only about a hundred and thirty million of it. was actually collected to fund this non profit. And for the first few years

1:35:33 That was plenty. for the type of research they were doing, the type of compute they needed. Paying salaries to the researchers. Not as much as they could make at Google and Facebook, but still. million or two million dollars for these folks.

1:35:47 Right. And yeah, so that really worked until it really didn't. Yeah. So David, what were they doing in the early days? Well.

1:35:55 In our first days it was All hands on deck recruiting and hiring researchers and There was the initial crew that came over And then pretty quickly after that in early twenty sixteen, they get a

1:36:06 Big. Big win. Leaves Google. Comes over.

1:36:13 Joins Elliot and Crew. At OpenAI. Dream team, you know, assembling here. And was he on Google Brain before this? He was on Google Brain, yep. And he, along with Ilya, would run large parts of open AI for the next couple of years.

1:36:29 Before of course leaving to start. Anthropic. But We're still a couple of years away from Anthropic, Claw Hud.

1:36:37 Chat CPT. Gemini everything today. For at least the first Year or two. Basically the plan at OpenAI is

1:36:46 Let's look at what's happening at Deep Mind. And show the research community that we can do as a new lab. do the same incredible things that they're doing and maybe even do them better. Is that why it looks so game like and game focused? Yes. Yes. So they started building models to play games. Famously the big one that they do is Dota Two, Defense of the Ancients Two.

1:37:08 The uh massively online battle arena video game. They're like, All right, well Deep mind, you're playing Starcraft. Well, we'll go play Dota Two. That's even More complex, more real time. And similar to the emergent properties of Go the game would devise unique strategies that you wouldn't see humans trying. So it clearly wasn't

1:37:27 humans coded their favorite strategies and rules in. It was emergent. Yep. They did other things. They had a product called Universe, which was around training computers to play thousands of games from Atari games to open world games like Grand Theft Auto. They had something where they were teaching a model how to do a Rubik's Cube.

1:37:47 And so it was A diverse set of projects. That didn't seem to Coalesce around. one of these is gonna be the big thing.

1:37:56 Yep. It was Research stuff. It was what Deep Mind was doing. Yeah. It was like a university research.

1:38:01 It was like D mind. And if you think back to Elon being an investor in Deep Mind, being really upset about Google acquiring it out from under him. Makes sense.

1:38:12 And I think Elon deserves a lot of credit for having his name and his time attached to OpenA at at the beginning. A lot of them. big heavy hitter recruiting. Was Elon throwing his weight behind this, I'm willing to take a chance.

1:38:26 Absolutely. Okay, so that's what's going on over at OpenAI, doing a lot of deep mind like stuff. Bunch of projects. Not one single obvious big thing they're coalescing around. It's not

1:38:38 Chat GPT time. Let's put it that way. Let's go back to Google. Cause last we sort of checked in on them. Yeah, they bought Deep Mind, but they had their talent rated and I don't want you to get the wrong impression about where Google is sitting just because some people left to go to open AI. So back in twenty thirteen. When Alex Khrushchevsky arrives at Google with Jeff Hinton and Iliaskever,

1:38:58 He was shocked to discover. that all their existing machine learning models were running on CPUs. People had asked in the past for GPUs, since machine learning workloads were well suited to run in parallel, but Google's infrastructure team had pushed back and said the added complexity and expanding and diversifying the fleet. Let's keep things simple.

1:39:16 That doesn't seem important for us. We're a CPU shop here. Yes. And so to quote from Genius Makers, in his first days at the company, he went out and bought a GPU machine. This is Alex. from a local electronics store, stuck it in the closet down the hall from his desk, plugged it into the network, and started training his neural networks on this lone piece of hardware. Just like he did in academia.

1:39:37 Except this time Google's paying for the electricity. Obviously one GPU was not sufficient, especially as more Googlers wanted to start using it too. And Jeff Dean and Alan Eustace had also come to the conclusion that disbelief While amazing. had to be re architected to run on GPUs and not CPUs. So spring of twenty fourteen rolls around.

1:39:58 Jeff Dean and John Giandra, who we haven't talked about this episode. Yeah, J G. Yes, you might be wondering, wait, isn't that the Apple guy? Yes, he went on to be Apple's head of AI. who at this point in time was at Google and oversaw Google Brain twenty fourteen. They sit down to make a plan For how to actually formally put GPUs into the fleet.

1:40:19 of Google's data centers, which is a big deal. It's a big change. But they're seeing enough. reactions to neural networks that they know to do this. Yeah. After Alex Knight is just a matter of time. Yeah. So they settle on a plan to order forty thousand GPUs.

1:40:34 From NVIDIA? Yeah, of course. Who else are you gonna order'em from? For a cost of a hundred and thirty million dollars. That's a big enough price tag that the request gets elevated to Larry Page. who personally approves it, even though finance wanted to kill it. Because

1:40:51 future of Google is deep learning. As an aside. Let's look at NVIDIA at the time. This is a giant Giant order.

1:40:59 Their total revenue was four billion dollars This is one order for 130 million. I mean, NVIDIA's primarily a consumer graphics card company at this point. Yes. And their market cap is$10 billion. It's almost like Google gave NVIDIA a secret.

1:41:17 that hey, not only does this work in research like the image net competition, but neural networks are valuable enough to us as a business. to make a hundred plus milliн dollar investment in right now No questions asked. We gotta ask Jensen about this at some point. This had to be

1:41:33 A tell. Mm-hmm. This had to really give NVIDIA the confidence, oh, we should way forward invest on this being a giant thing in the future. So all of Google wakes up to this idea.

1:41:45 they start really putting it into their products. Google Photos happened, Gmail starts offering typing suggestions. David, as you pointed out earlier, Google's giant AdWords business. started finding more ways to make more money with deep learning. In particular, when they integrated it, they could start predicting what ads people would click in the future. And so Google started spending hundreds of millions more on GPUs on top of that hundred and thirty million.

1:42:10 but very quickly paying it back from their ad system. So it became more and more of a no-brainer to just buy as many GPUs as they possibly could. But Once neural net started to work. Anyone using them, especially at Google Scale, kinda had this problem. Well now we need to do giant amounts of matrix multiplications.

1:42:29 Anytime anybody wants to use one. The matrix multiplication are effectively how you do that propagation through the layers of the neural network. So you sort of have this problem. Yes, totally. There's the Inefficiency of it.

1:42:42 But then there's also the business problem of Wait a minute. It looks like we're just gonna be shipping hundreds of millions, soon to be billions of dollars over to NVIDIA every year for the foreseeable future. Right. So there's this amazing moment. Right after Google rolls out speech recognition, their latest use case for neural nets.

1:43:01 Just on Nexus phones. 'Cause again, they don't have the infrastructure to support it on all Android phones. It becomes a super popular feature and Jeff Dean does the math. And figures out if people use this for I don't know, call it three minutes a day. And we roll it out to all billion Android phones.

1:43:18 We're gonna need twice the number of data centers. That we currently have across All of Google. Just to handle it. Just for this feature. Yeah. There's a great quote where Jeff goes to Erz Holzal.

1:43:30 And goes. We need another Google. Or, David, as you were hinting at. The other option is we build a new type of chip.

1:43:42 Customized for just Our particular Use case. Yeah. Matrix multiplication, tensor multiplication, a

1:43:50 Tensor processing unit, you might say. Yes. Wouldn't that be nice? So conveniently, Jonathan Ross, who's an engineer at Google, has been spending his twenty percent time at this point in history working on an effort involving FPGAs. These are essentially expensive but programmable chips. that yield really fantastic results.

1:44:09 So they decide to create a formal project to Take that work, combine it with some other existing work, and build a custom ASIC or an application specific integrated circuit. So enter David, as you said, the tensor processing unit. Made

1:44:23 Just For neural networks. that is far more efficient from GPUs at the time. With the trade off that You can't really use it for anything else. It's not good for graphics processing. It's not good for lots of other GPU workloads.

1:44:36 Just Matrix multiplication. And just neural networks. But it would enable Google to scale their data centers without having to double their entire footprint.

1:44:46 So the big idea behind the TPU, if you're trying to figure out like what was the core insight. They use reduced computational precision. So we take numbers like four thousand five hundred and eighty six point eight two seven two. And round it just to four thousand five hundred eighty six point eight. Or maybe even just four thousand five hundred eighty six with nothing after the decimal point.

1:45:06 And this sounds kinda counterintuitive at first. Why would you want Less. Precise. rounded numbers for this complicated math. The answer is efficiency.

1:45:15 If you can do the heavy lifting in your software architecture. or what's called quantization to account for it. You can store information as less precise numbers. then you can use the same amount of power and the same amount of memory and the same amount of transistors on a chip to do far more calculations per second. So you can either spit out answers faster or use bigger models.

1:45:36 The whole thing is quite clever behind the TPU. Mm. The other thing that has to happen with the TPU is it needs to happen now. Because It's very clear speech to text is a thing. It's very clear some of these other use cases at Google. Yeah.

1:45:49 Demand for all of this stuff that's coming out of Google Brain is Through the roof. Immediately. Right. And we're not even two LLMs yet. It's just like everyone sort of expects some of this whether it's computer vision in photos or speech recognition, like it's just becoming a thing that we expect. And it's gonna flip Google's economics upside down if they don't have it.

1:46:07 So The TPU was designed, verified, built, and deployed into data centers in fifteen months. Wow. It was not like a research project that could just happen over several years. This was like a hair on fire problem that they launched immediately. One very clever thing that they did was A, they used the FPGAs as a stopgap.

1:46:26 So even though they were like too expensive on a unit basis, they could get'em out as a test fleet and just make sure all the math worked before they actually had the ASICs. Printed it. I don't know if it was a T S M C but you know. Fabb and ready. The other thing they did is They fit the TPU into the form factor of a hard drive.

1:46:42 So it could actually slot into the existing server racks. You just pop out a hard drive and you pop in a TPU without needing to do any physical re architecture. Wow. That's amazing. That's the most Googly infrastructure story since the cork boards. Exactly. Also, all of this didn't happen in Mountain View. It was at a Google satellite office in Madison, Wisconsin. Whoa.

1:47:06 Yes. Why Madison, Wisconsin? There was a particular professor out of the university and There was a lot of students that they could recruit from and Wow. Yeah, I mean it was probably them or Epic. Where are you gonna go work?

1:47:19 Yeah. Wow. They also then just kept this a secret. Right. Why would you tell anybody about this? Because it's not like they're offering these in Google Cloud, at least at first. And why would you want to tell the rest of the world what you're doing? So the whole thing was a complete secret for at least a year before they announced it at Google I.O.

1:47:37 So Really crazy. The other thing to know about the TPUs is they were done in time. for the Alpha Go match. So that match ran on a single machine with four TPUs in Google Cloud.

1:47:48 And once that worked, obviously that gave Google a little bit of extra confidence to go really Really reproduction. So That's the TPU. V one, by all accounts, was not great. They're on V seven or V eight now, it's gotten much better. TPUs and GPUs look a lot more similar than they used to.

1:48:05 then they've sort of adopted features from each other. But today. Google, it's estimated has two to three million TPUs. For reference, NVIDIA shipped people don't know for sure, somewhere around four million GPUs last year.

1:48:18 So people talk about AI chips Like it's this just oh one horse race with NVIDIA. Google has like an almost NVIDIA scale. Internal thing making their own ships at this point for their own and for Google Cloud customers.

1:48:31 The TPU is a giant deal in AI in a way that I think a lot of people don't realize. Yep. This is one of the great ironies and Maddening things to open AI and Elon Musk is the Open AI gets founded in twenty fifteen with the goal of Hey, let's shake all this talent.

1:48:49 out of Google and level the playing field. And Google just accelerates. Right. They also build TensorFlow. That's the framework that Google Brain built to enable researchers to build and train and deploy machine learning models.

1:49:02 And they built it in such a way that it doesn't just have to run on TPUs. It's super portable without any rewrites to run on GPUs or even CPUs too. So this would replace The old dist belief system. And kinda be there. internal and external framework for enabling

1:49:18 ML researchers going forward. So Somewhat paradoxically, during these years after the founding of OpenAI Yes. Some amazing researchers are getting siphoned off from Google and Google Brain.

1:49:32 But Google Brain is also firing on all cylinders during this time frame. Delivering on the business purposes for Google left and right. Yes. And pushing the state of the art forward in so many areas. And then in twenty seventeen A paper.

1:49:47 Gets published. From eight researchers on the Google Brand team. Kinda quietly. These eight folks Were

1:49:54 Obviously very excited. about the paper and what it described and the implications of it. And they thought it would be very big. Google itself

1:50:04 Oh Cool. This is like the next iteration of our language model work. Great. Which is important to us. But Are we sure this is

1:50:13 The next Google No. No. There are a whole bunch of other things we're working on that Seem more likely to be.

1:50:20 The next Google. But This paper and its publication. What gave OpenAI the opportunity to do it.

1:50:29 To grab the ball and run with it and build the next Google. Because this is the transformer paper. Okay, so where did the transformer come from? Like what was the latest thing that language models had been doing at Google. So coming out of the success of Franz Ochs' work on Google Translate.

1:50:47 And the improvements that happened there. In like the late two thousands ish, two thousand seven. Yeah, mid to late two thousands. They keep iterating on Transly. And then once Jeff Hidden comes on board and Alexnet happens. They switch over to a neural network based language model.

1:51:03 Four. Translate. Which was dramatically better and like a big crazy cultural thing because you've got these researchers Parachuting in, again led by Jeff Dean, saying

1:51:15 I'm pretty sure our neural networks can do this way better than the classic methods that we've been using for the last ten years. What if we take the next several months? And do a proof of concept.

1:51:28 They end up throwing away the entire old code base and just completely wholesale switching to this neural network. There's actually this great New York Times magazine story that ran in twenty sixteen about it. And I remember reading the whole thing with my jaw on the floor, like Wow, neural networks are a Big effing deal.

1:51:44 And this was the year before the transformer paper would come out. Before the transformer paper. Yes. So they do the rewrite of Google Translate. Make it based on recurrent neural networks, which were state of the art. At that point in time.

1:51:57 And it's a big improvement. But As teams within Google Brain and Google Translate keep working on it. There's some limitations. And in particular, a big problem was that they quote unquote

1:52:08 Forgot. Things. Too quickly. I don't know if it's exactly the right analogy, but you might say in sort of like today's transformer world speak, you might say that their context window was Pretty short.

1:52:19 as these language models progressed through text. They needed to sort of remember everything they had read. So that when they need to change a word later or come up with the next word. They could

1:52:31 Have a whole memory of the body of text to do that. So one of the ways that Google tries to improve this. is to use something called long short term memory networks or

1:52:45 L STMs is the acronym that people use for this. And basically what LSTMs do. is they create a persistent or long Short term memory. You gotta use your brain a little bit here. For the model.

1:53:00 So that it can keep context as it's going through a whole bunch of steps. And people were pretty excited about LSTMs at first. People are thinking like, Oh L S TMs are What are gonna take

1:53:12 language models and large language models mainstream. Right. And indeed, in twenty sixteen. they incorporate it into Google Translate. These LSTMs.

1:53:21 It reduces the error rate by sixty percent. Huge jump. Yeah. The problem with LSTMs though They were effective.

1:53:29 But they were Very computationally intensive. And they didn't Parallelize that great. All the effort that are coming out of Alexnet and then the TPU project of parallelization. This is the future. This is how we're gonna make

1:53:44 A I really work. LSTMs are a bit of a roadblock here. Yes. So A team within Google Brain.

1:53:53 Start searching. For a better architecture That's also has the attractive properties of LSTMs. That it doesn't forget context too quickly.

1:54:04 But can parallelize and scale. Better to take advantage of all these new architectures. Yes. And a researcher named Jakob Oscaret

1:54:14 had been toying around With the idea of Broadening the scope of quote unquote attention. Mm. language processing.

1:54:23 What if rather than focusing on The immediate words. Instead. What if you told the model. Hey.

1:54:31 Pay attention to the entire corpus of text. Not just the next few words. Look at the whole thing. And then based on that entire context and giving your attention to the entire context. Give me a prediction of what the next translated word. Should be.

1:54:48 Now by the way This is actually how professional human translators translate. Text. You don't just go word by word. I actually took a translation class in college, which was really fun.

1:54:58 You read the whole thing of the original in the original language. you get and understand the context of what the original work is And then you go back and you start to translate it. With the entire context of the passage in mind. Mm.

1:55:13 So it would take a lot of computing power for the model to do this. But It is extremely parallelizable. So Jakob starts collaborating with a few other people. on the brain team, they get excited about this.

1:55:27 They decide that they're gonna call this new technique. The transformer. Because one That is literally what it's doing. It's taking in a whole chunk of information. Processing, understanding it.

1:55:39 And then transforming it. And the They also love transformers as kids. That's not not why they named it the transformer. And it's taking in the giant corpus of text and storing it in a compressed format, right? Yeah. I bring this up.

1:55:54 Because that is exactly how you pitched the micro kitchen conversation with Noam Shazir. In two thousand, two thousand and one, seventeen years earlier. Who is a co author on this paper? Yes, well, so speaking of Gnome Shazir. He learns about this project.

1:56:10 And he decides Hey, I've got some experience with this. This sounds pretty cool. LSTMs definitely have problems. This could be promising. I'm gonna jump in. And work on it with these guys. And it's a good thing he did.

1:56:24 Because before Gnome joined the project They had a working implementation of the transformer. But it wasn't actually producing any better results than LSTMs. Nome joins the team. Basically pulls a Jeff Dean.

1:56:39 Rewrites the entire code base from scratch. And when he's done. The transformer now crushes. The L S T M Based.

1:56:49 Google Translate solution. And it turns out That the bigger they make The model. The better the results get.

1:56:57 It seems to scale. really, really, really well. Stephen Levy wrote a piece in Wired about the history of this. And there are all sorts of quotes from the other members of the team just littered all over this piece with things like

1:57:09 Noam is a magician. Nome is a wizard. Noam took the idea and came back and said. It works now.

1:57:18 Yeah. And you wonder why Noam and Jeff Dean are the ones together working on the next version of Gemini now. Yes. No and Jeff Deen uh definitely two Ps in a pot here. Yes. So

1:57:30 We talked to Greg Carrado from Google Brain. one of the founders of Google Brain. And It was a really interesting conversation'cause he underscored how elegant the transformer was. And he said it was so elegant that people's response was often

1:57:43 This can't work. It's too simple. Transformers are barely a neural network architecture. Right. It was another Big

1:57:51 Change. From the Alex net. Jeff Hinton lineage neural networks. Yeah. It actually has changed the way that I look at the world,'cause he pointed out that in nature, this is Greg

1:58:02 The way things usually work is the most energy efficient way they could work. Almost from an evolution perspective. The the most simple elegant solutions are the ones that survive. because they are the most efficient with their resources.

1:58:18 And you can kinda port this idea over to computer science too. that he said he's developed a pattern recognition inside of the research lab. to realize that you're probably onto the right solution when it's really simple and really efficient versus a complex idea. Mm-hmm.

1:58:35 It's very clever. It's I think it's very true. You know how when you sit around you have a thorny problem and you debate and you whiteboard and you come up with all and then you're like, Oh my God. Oh my God, it's so simple. And that ends up being the right answer. Yeah, there's an elegance. to the transformer.

1:58:48 Yes. And that other thing that you touched on there, this is the beginning of the modern AI. Just feed it more data. The famous piece The Bitter Lesson by Rich Sutton wouldn't be published until twenty nineteen.

1:59:03 For anyone who hasn't read it, it's basically We always think as AI researchers are we're so smart and our job is to come up with another great algorithm. but effectively in every field from language to computer vision. To Chess.

1:59:15 You just figure out a scalable architecture and then the more data wins. Just these infinitely scaling. More data, more compute. Better results. Yes. And this is really the start of when that starts to be like Oh, we have found the scalable architecture that will go.

1:59:31 at so far for I don't know close to a decade. Of just More data in, more energy, more compute, better results. So the team And no.

1:59:41 Like Yo, this thing has a lot. A lot of potential. This is more than better translate. We can really apply this. Yeah, this is gonna be more than better Google Translate.

1:59:51 The rest of Google though Definitely slower. To wake up to the potential. They build some stuff. Within a year they build Burt. The large language model.

2:00:02 Yes, absolutely too. It is a False narrative out there that Google did nothing with the transformer after the paper was published. They actually did a lot. In fact, Burt was one of the first LLMs. Yes. They did a lot with transformer based large language models. After the paper came out. What they didn't do

2:00:20 Was treat it as a wholesale technology platform change. Right. They were doing things like Bert and uh Mum, this other model. you know, they could work it into search results quality. And I think that did meaningfully move the needle, even though Google wasn't bragging about it and talking about it. They got better at query comprehension. They were working it into the core business.

2:00:37 Just like every other time. Google Brain came up with something great. Yeah. So And perhaps

2:00:43 One of the greatest decisions ever for value to humanity. And maybe one of the worst corporate decisions ever. For Google. Google allows this group of eight researchers to publish the paper.

2:00:56 Under the title Attention is all you need, obviously a Nod to the classic Beatlesong about love. As of today in twenty twenty five. This paper has been cited over a hundred and seventy three thousand times in other academic papers. making it currently the seventh most cited

2:01:18 Paper of the twenty first century. And I think all of the other papers above it on the list have been out. Much longer. Wow. And

2:01:27 Also, of course, within a couple of years. All eight. authors of the Transformer paper. had left Google to either start or join AI startups, including

2:01:39 Open AI. Brutal. And of course No I'm starting character AI. Which

2:01:45 What do we call it? A acquisition? He would end up back at Google via some strange licensing and IP and hiring agreement. On the few billion dollars order. Very, very expensive mistake on Google's part. It is fair to say that twenty seventeen begins the five year period of

2:02:02 Google not sufficiently seizing the opportunity that they had created. With the transformer. Yes. So speaking of seizing opportunities, what is going on at OpenAI during this time? And does anyone think the transformer's a big deal over there? Yes, yes they did.

2:02:17 But here's where history gets really, really crazy. Right. After Google publishes the transformer paper. In September of twenty

2:02:26 Seventeen. Elon gets really, really fed up. With what's going on. At OpenAI.

2:02:34 There's like seven different strategies. Are we doing video games? Are we doing competitions? What's the plan? What is happening here, as best as I can tell All you're doing is just trying to copy deep mind. Meanwhile.

2:02:48 I'm here building SpaceX and Tesla. Self driving is becoming more and more clear as critical to the future of Tesla. I need AI researchers here and I need great AI advancements to come out. To help what we're doing at Tesla.

2:03:05 Open AI isn't cutting it. So he makes An ultimatum. To say um And the rest of the open AI board.

2:03:12 He says. I'm happy. to take full control of open AI. And we can merge this into Tesla. I don't even know how that would be possible.

2:03:22 to merge a nonprofit into Tesla. But in Elon Lan, if he takes over as CEO of OpenAI, it almost doesn't matter. We're just treating it as if it's the same company anyway, just like we do with the deals with all of my companies. Right. Or He's out completely, along with all of his funding. And

2:03:41 Sam and the rest of the board are like No. And as we know now, they're sort of calling capital into the business. It's not like they actually got all the cash up front. Right. So they're only a hundred and thirty million ish into the billion dollars of commitment. They don't reach a resolution. And by early twenty eighteen, Elon is

2:03:59 Out. Along with him the main source of open AIs. Funding. So either

2:04:06 This is just A really, really, really bad misjudgment. By Elon. Or the sort of panic that this throws open AI

2:04:17 into is the catalyst. That makes them reach. for the transformer and say, All right. We gotta figure things out. Necessity's the mother of invention. Let's go for it.

2:04:29 It's true. I don't know if during this personal tension between Elon and Sam. if they had already decided to go all in on Transformers or not. Because the thing you very quickly get to If you decide.

2:04:43 Transformers, language models, we're going all in on that. You do quickly realize you need a bunch of data. You need a bunch of compute. You need a bunch of energy and you need a bunch of capital.

2:04:53 And so if your biggest backer is walking away. The three D chess move is Oh, we gotta keep him because we're about to pivot the company. And we need his capital for this big pivot we're doing. The four D chess is

2:05:06 If he walks away, maybe I can turn it into a for profit. company And then raise money into it and eventually generate enough profits to fund this extremely expensive new direction we're going in.

2:05:22 I don't know which of those it was. Yeah. I don't know either. I suspect The truth is it's some of both. Yes. But either way, how nuts is it?

2:05:30 that A, these things happen at the same time. And B the company wasn't burning that much cash and then they decided to go all in on We need to do something so expensive that we need to be a for profit company. In order to actually achieve this mission'cause

2:05:45 It's just gonna require hundreds of billions of dollars for the far foreseeable future. Yeah. So in June of twenty eighteen Open AI releases a paper.

2:05:55 Describing how they have Taken The transformer. And developed a new approach. Of pre training them.

2:06:04 On Very large amounts. Of general text on the internet. And then fine tuning. That general pre training.

2:06:13 They also announced that they have Trained and run. The first proof of concept model. Of this approach.

2:06:22 Which they are calling G P T One. Generatively pre trained transformer. Version one.

2:06:32 Which We should say is right around the same time as Burt and right around the same time as another large language model based on the transformer out of here in Seattle, the Allen Institute. Yes. Indeed. So

2:06:44 It's not as if this is heretical and a secret. Other AI labs, including Google's own, is doing it. But From the very beginning, open AI seemed to be taking this more seriously given

2:06:56 the cost of it would require betting the company if they continued down this path. Yeah. Or betting the nonprofit. Betting the entity. Yes. We're gonna need some new terminology here. Yes. So Elon's just walked out the door. Where are they gonna get the money for this? Sam turns to one of the other board members of OpenAI. Read Hoffman.

2:07:15 Read. Just a year or so earlier. had sold LinkedIn to Microsoft. And Reed is now on the board. Of Microsoft.

2:07:25 So Reed says, Hey, why don't you come talk to Satya about this? Do you know where he actually talks to Satya? Oh I do. Oh I do. In July of twenty eighteen. They set a meeting. For Sam Altman and Satya Nadella. To sit down.

2:07:39 While they're both at the Allen and Company. Sun Valley Conference. In Sun Valley, Idaho. It's perfect. And while they're there, they hash out a deal. For Microsoft.

2:07:49 To invest. One billion dollars. Into open AI. In a combination of both cash and Azure cloud credits.

2:07:58 And in return Microsoft will get access to open AIs. Technology get an exclusive license to open AI's technology. For use in Microsoft's. Products.

2:08:09 And the way that they will do this is open AI, the nonprofit. We'll create a captive for profit entity called OpenAI LP. Controlled by The nonprofit OpenAI Inc. And Microsoft will invest into the captive for profit entity.

2:08:24 Reed Hoffman joins the board of this new structure along with Sam Ilya Greg Brockman. Adam D'Angelo and Tasha McCay. And thus the modern

2:08:34 Open AI. For profit, non profit. Question mark? Is created. Even today here in twenty twenty five.

2:08:44 is created. This is like the complete history of AI. This is not just the Google AI episode. Well, these things are totally inextricable. And I was just gonna say, this is the Google Part three episode. Microsoft.

2:08:56 They're back. Microsoft is Google's mortal enemy. Yes. That In our first episode on the founding of Google and search and then in the second episode on Alphabet and all the products.

2:09:07 That they made the whole strategy at Google was always about Microsoft. They finally beat them. On every single front. And here they are showing up again saying, What was Satya's line? We just want to see them dance. I think the line that would come a couple years later is we want the world to know that we made Google dance.

2:09:29 Oh, man. But This is all still pre Chad GPT. This is just Sam lining up the financing he needs for what appears to be a very expensive scaling exercise they're about to embark on. with GPT two and onward.

2:09:45 Cup. And this is the right time to talk about Why? From open AI's perspective, Microsoft is the absolute Perfect partner.

2:09:54 It's not just that they have a lot of money. Although that helps. I mean that helps. That helps a lot. More important than money.

2:10:03 They have a really, really great Public cloud. Azure. Yes. Open AI is not gonna go

2:10:09 buy a bunch of NVIDIA GPUs and then build their own data center. Here at this point and twenty eighteen. That's not the scale of company that they are. They need a cloud provider.

2:10:19 in order to actually do all the compute that they want to do. If they were back at Google and these researchers were doing it, great, then they have all the infrastructure. But OpenAI needs to tie themselves to someone with the infrastructure. And there's

2:10:32 Basically only two non Google options. They're both in Seattle and hey one of them in Microsoft is really interested. Also has a lot of cash. It seems like a great Partnership.

2:10:47 That's true. I wonder if they did talk to AWS at all about it. 'Cause I think this is a crazy Easter egg. I hesitate to say it out loud, but I think AWS was actually in the very first investment with Elon.

2:11:00 In open AI. Oh, wow. And I don't know if it was in the form of credits or what the deal was, but I'd seen it reported a couple places. That AWS actually was in that. nonprofit round. Yeah, in the uh Nonprofit funding the donations too.

2:11:15 Yes. The early open AI. Anyway, Microsoft OpenAI, they end up tying up A match made in heaven. Satya and Sam are on stage together talking about how this amazing partnership and marriage has come together. And There.

2:11:30 Off to model training. Yeah. And this paves the way. For the GPT era. Of open AI.

2:11:38 All right listeners. Now is a great time to talk about one of our favorite companies, StatSig. Yes, there is a reason why the best product teams rely on Statsig, whether they are iterating on their core product features or shipping AI powered experiences at scale. Yeah. In the crazy speed of today's AI world. Shipping fast is just table stakes now.

2:12:01 It's basically trivial to build and deploy your app constantly. The real advantage is how quickly you learn what changes actually created value for customers and how fast you can use that signal to guide what you ship next. This is where StatSIG comes in. It brings experimentation, feature flags, and product analytics into one unified system so teams can ship safely, test rigorously, and directly link what they changed. to how users actually behaved. So if you want to make learning your competitive advantage, whether you're building new AI experiences or just evolving your existing core product, go to statsig.com slash acquired to get started. All right.

2:12:40 So what are we in GPT two? Is that what's being trained right here? Yes. D P T two. This was the first time I heard about it. Data scientists.

2:12:48 Around Seattle we're talking about this cool right. So after the first Microsoft partnership, the first billion dollar investment. In twenty nineteen, OpenAI releases GPT two. Which is still early, but Very promising.

2:13:02 That can do. A lot of things. A lot of things, but it required an enormous amount of creativity on your part. You kinda had to be a developer. to use it. And if you were a consumer There was a very heavy load.

2:13:16 put on you. You had to go write a few paragraphs. And then paste those few paragraphs. into the language model and then it would suggest a way to finish what you were writing based on the source paragraphs. But it wasn't

2:13:29 Interactive. Yes. It was not A chat interface. There was no interface essentially for it. It was an API. But it can do things like obviously translate text. I mean Google's been doing that for a long time, but GPT two, you could do stuff like

2:13:44 Make up a fake news headline and give it to GPT two and it would write a whole article. You would read it and you'd be like, uh sounds like it was written by a bot. Yeah. But again, there was no front door to it for normal people. You had to really be willing to wait in the muck to use this thing. So then. The next year, in June of twenty twenty.

2:14:05 GPT Three comes out. Still no front door, you know, user interface to the model. But it's very good. GPT two

2:14:15 Showed the promise of what was possible. GPT three It's starting to be in the conversation of can this thing pass the Turing test? Oh yeah. You have a hard time distinguishing between Articles that

2:14:29 GPT wrote and articles that humans Right. It's very good. And there starts to be a lot of hype around this thing. And so even though consumers aren't really using it, the broader awareness is that There's something interesting on the horizon. I think the number of

2:14:44 AI pitch decks that VCs are seeing is starting to tick up. Around this time. As is the NVIDIA stock price. Yes. So Then in the next year in the summer of

2:14:55 twenty twenty one. Microsoft Releases GitHub. Copilot?

2:15:03 Using GPT three. This is the first Not just Microsoft. product that comes out with GPT. Baked into it.

2:15:10 First productization. Product anywhere. Yeah. First productization of GPT. Yes, of any open AI technology. Yeah. It's big. This starts a massive change in how Software gets written in the world. Slowly then all at once. It's one of these things where at first just a few

2:15:26 software engineers and there was a lot of whispers of how cool is this? It makes me a little bit more efficient. And now you get all these comments like seventy five percent of all companies' code is written with AI. Yep. So after that Microsoft invests another Two billion dollars in open AI, which seemed Like a lot of money at the time.

2:15:42 So that takes us To the end of twenty twenty one. There's an interesting kind of context shift that happens around here. Yeah, the bottom falls out on tech stocks. Crypto

2:15:54 The broader markets really everyone suddenly goes from risk on to risk off. And part of it was war in Ukraine, but a lot of it was interest rates going up. And Google gets hit really hard.

2:16:07 The high water mark was November nineteenth of twenty twenty one. Google was right at two trillion dollars of market cap. About a year after that slide began. They were worth a trillion dollars.

2:16:20 Nearly a fifty percent drawdown. Wow. So towards the end of twenty twenty two, leading up to The launch of Chat GPT. People I think are starting to realize Google Flow.

2:16:33 They're slow to react to things. It feels like they're uh old crusty company? Are they like the Microsoft? Of the two thousands. where they haven't had a breakthrough product in a while. People are not bright on the future of Google.

2:16:48 And then chat GPT comes out. Yeah. Wow. Which means if you were bullish on Google back then and contrarian, you could have invested at a trillion dollar market cap. Which is interesting, like in October of twenty one

2:17:01 the market was saying that the forthcoming AI wave will not be a strength for Google. Or maybe what it was saying is we don't even know anything about a forthcoming AI wave'cause people are talking about AI, but they've been talking about VR and they've been talking about crypto and they've been talking about all this frontier tech and like that's not the future at all. This company just feels slow and unadaptive.

2:17:20 And slow and unadaptive at that point in history, I think would have been a fair characterization. They had an internal chatbot, right? Yes, they did. All right. So Before we talk about chat GPT Google had a chat bot.

2:17:33 So Gnomeshir. Incredible. Engineer re architected the transformer, made it work, one of the lead authors of the paper.

2:17:42 storied career within Google has all of this sway, should have all of this sway within the company. After the transformer paper comes out. He and the rest of the team are like Guys, we can use this for a lot more than Google Translate. And in fact, the last paragraph of the paper Are you about to read the Transformer paper? Yes, I am.

2:18:02 We are excited about the future of attention based models and plan to apply them to other tasks. We plan to extend the transformer To problems involving input and output modalities. Other than text. And to investigate large inputs and outputs such as images.

2:18:20 Audio. And video. This is in the paper. Wow. Google obviously does not.

2:18:26 Do any of that. For quite a while. Nome though immediately starts advocating to Google leadership. Hey. I think this is gonna be so big.

2:18:36 The transformer. Yeah. We should actually consider just throwing out the search index. And the ten blue links model. And go all in on transforming all of Google into one

2:18:48 Giant. Transformer model. And then Gnome actually goes ahead And builds A chat bot.

2:18:56 interface to a large transformer Model. Is this Lambda? This is Before Lambda. Mina.

2:19:05 Is what he calls it. And There is a chat bot in the like late teens, twenty twenty time frame that Nomas built within Google. That Arguably is pretty close to chat.

2:19:17 GPT Now. It doesn't have any of the post training safety that Chat GPT does, so It would go off the rails. Yeah, someone told us that you could just ask it who should die. And it would come up with names for you of people that should die.

2:19:31 It was not a shippable product. It was a very raw Not safe, not post trained. Chatbot and bottle. Right. But

2:19:42 It existed within Google. And they didn't ship it. And Technically, not only did it not have post training, it didn't have RLHF either. This very core component of the models today, the reinforcement learning with human feedback that chat GPT I don't know if it had it.

2:19:58 in three, but it did in three point five and it did for the launch of chat GPT. Realistically it wasn't launchable, even if it was an open AI thing,'cause it was so bad. But a company of Google stature certainly could not take the risk. So strategically they have this working against them. But aside from the strategy thing, there's two business model problems here.

2:20:18 One If you're proposing drop the ten blue links and just turn Google.com into a giant AI chat bot. Revenue drops. When you provide direct answers to questions

2:20:29 versus showing advertisers And letting people click through to websites. That upsets the whole Apple cart. Obviously they're thinking about it now, but Until twenty twenty one.

2:20:39 That was an absolute non starter. To suggest something like that. Two There were legal risks. Of sitting in between publishers and users. I mean, Google at this point had spent decades fighting the public perception.

2:20:51 and court rulings that they were disintermediating publishers from readers. So there was like a very high bar internally culturally to clear if you were gonna do something like this. Even those info boxes that popped up. that took until the twenty teens to make it happen.

2:21:08 Those really were mostly on non monetizable queries anyway. So any time that you were gonna say Hey, Google's gonna provide you an answer instead of 10 blue links. You had to have um Bulletproof case for it.

2:21:21 Yeah. And it was also a Brand promise and trust issue too. Consumers Trusted.

2:21:30 Google so much. For us even today, you know, when I'm doing research for acquired We need to make sure we get something right. I'm going to Google. I look something up in Claude. Yeah. It gives me an answer. I'm like, that's a really good answer. And then I verify by searching Google that I can find those facts too, if I can't click through the sources on Claude. That's my workflow. Which sort of sounds funny today, but it's important. If you're gonna propose replacing the ten blue links.

2:21:54 With a chat bot, you need to be Really damn sure that it's gonna be accurate. Yes. And In twenty twenty, twenty twenty one, that was definitely not the case. Arguably

2:22:05 Still isn't the case today. And There also wasn't A compelling reason to do it. Because

2:22:13 Nobody was really asking for this product. Right. No m new And people in Google knew that you could make a chatbot interface to a transformer based LLM. And that was a really compelling product. The general public didn't know. Open AI didn't even really know. I mean GPT was out there.

2:22:34 Do you know the story of the launch of Chat GPT? Well, I think I do. I have it in my notes here. All right, so they've got GPT three point five. It's becoming very, very useful. Yeah, this is late twenty twenty two. They've got three point five. But there's still this problem of How am I supposed to actually use it? How does it productized? And

2:22:52 Sam just kind of Says. We should make a chat bot. That seems like a natural interface for this. Can someone just make a chat? And within like a week Internally.

2:23:03 Turn calls to the chat GPT three point five API. into a product where you're just chatting with it. And every time you kick off a chat message, it just calls GPT 3.5 on the API. And that turns out to be This magic product.

2:23:18 I don't think it's not. They expected it. I mean servers are tipping over. They're working with Microsoft to Try to get more compute. They're cutting deals with Microsoft in real time to try to get more investment, to get more Azure credits, or get advances on their Azure credits. In order to handle the incredible load in November of twenty twenty two that's coming in of people wanting to use this thing.

2:23:39 They also just throw up a paywall randomly. Because They thought that the business was gonna be an API business. They thought that the projections We're all about how much revenue they were gonna do.

2:23:49 through B2B licensing deals. And then they just realize, oh, there's all these consumers trying to use this. Put up a paywall to at least dampen the most expensive use of this thing so we can kinda offset the costs or Slow the roll out.

2:24:02 Right. This isn't uh Google search, you know, eighty nine percent gross margin stuff here. Right. So they end up having incredibly fast revenue takeoff just from the quick stripe paywall that they threw up

2:24:15 Over a weekend. To handle all the demand. So To say that OpenAI had any idea what was coming. would also be completely false. They did not get that this would be the next big consumer product when they launched it.

2:24:27 Ben Thompson loves to call open AI the accidental consumer tech company, right? Yes. It was definitely accidental. Now there is actually another slightly different version of the motivation for Launching the chat. Is this the Dario interface? Yeah, the Dario and Anthropic version. So Anthropic was working on what would become Claude.

2:24:51 And rumors were out there and people had open AI got wind of like, Oh hey, Anthropic and Dario or Working on a chat interface. we should probably do one too, and if we're gonna do one

2:25:04 We should probably launch it before they launch theirs. So I think that had something to do with the timing. But again, I don't think anybody Including OpenAI, realized. What was gonna happen, which is Ben you alluded to it, but uh

2:25:17 Give the actual numbers. On November thirtieth, twenty twenty two. Basically Thanksgiving. OpenAI launches. A research preview.

2:25:27 Of an interface to The new GPT three point five called Chat GPT. That morning, on the thirtieth, Sam Altman tweets, Today we launched Chat GPT. Try talking with it here. And then a link to chat dot openai.

2:25:42 Done. Come. Within a week. Less than a week, actually, it gets one million users. Bye.

2:25:48 The end of the year, so you know, one month later. December. Thirty first. twenty twenty two. It has thirty million users.

2:25:57 By the end of the next month. by the end of January twenty three, so two months after launch. It crosses one hundred million. registered users the fastest product In history

2:26:11 To hit that milestone. Completely insane. Completely insane. Before we Talk about what that unleashes within Google. Which is the famous code red.

2:26:22 To rewind a little bit back to know him in the chat bot. within Google Mina. Google does keep working on Mina. They

2:26:31 develop it into something called lambda. Which is also a chat bot. Also internal. I think it was a language model. At this point in time they still differentiated between the underlying model

2:26:44 Brand name. and the application name. Yes, Lambda was The model and then there also was a chat interface too. Lambda.

2:26:52 That was internal for Google use. Only. Nome is still advocating to leadership. We gotta release this thing. He leaves in twenty twenty one. And founds a chatbot company, Character AI.

2:27:06 That still exists to this day. And they raise A lot of money. as you would expect. And then Google ultimately in twenty twenty four After Chat GPT launches.

2:27:17 pays two point seven billion dollars, I think, to do uh licensing deal with character AI, the net of which Gnome comes back to Google. Yeah, I think Larry and Sergey were like Uh

2:27:29 If we're gonna compete Seriously. We kinda need no back and Blank check to go get him. Yeah.

2:27:36 So Throughout twenty twenty one, twenty twenty two, Google's working on The Lambda model And then the chat interface to it. In May of twenty twenty two, they do release

2:27:47 Something that is available to the public called AI Test Kitchen, which is a AI product test. test area where people can play around with Google's internal AI. Products including the Lambda. Chat interface. Yeah.

2:28:01 In all fairness, predates chat GPT. Do you know what they do to nerf Chat so that it doesn't go too far off the rails. This is amazing. No. For the version of Lambda Chat that is in

2:28:12 AI test kitchen. They stop all conversations after five turns. So you can only have five turns of conversation with the chat bot and then it's just And we're done for today. Thank you. Goodbye. Oh wow. And the reason they did that was for safety of like, you know if The more turns you had with it, the more likely it would start to go off the rails.

2:28:31 And honestly. It was a fair concern. I mean this thing Was not for public consumption. And if you remember back a few years before, Microsoft released Tay. Which was this crazy racist chat bot.

2:28:44 Yeah, they launched it as a Twitter bot, right? And it was going off the rails on Twitter. This was in twenty sixteen, I think. Right, maximal impact of badness. Yeah. And so despite Google all the way back in twenty seventeen, Sundar declared we are an AI first company. is being understandably very cautious. in real public AI launches, especially on consumer facing things. Yep.

2:29:08 And as far as anyone else is concerned before chat GPT They are an AI first company and they're launching all this amazing AI stuff. It's just within the vector of their existing products. Right. So TET TPD comes out, becomes the fastest product in history to a hundred million users.

2:29:25 It is immediately obvious. To Sundar, Larry, Sergei, all of Google leadership that this is an existential threat to Google. Chat GPT is a better user experience to do the same job function that Google search does. And

2:29:39 to underscore this. So if you didn't know it in November of twenty two, you sure knew it by February of twenty three. Because Good old Microsoft, our biggest scariest enemy. Oh yeah. announces a new Bing powered by Open AI and Satya has a quote. It's a new day for search.

2:29:57 The race starts today. There's An announcement of a new AI powered search page. He says we want to rethink. what search was meant to be in the first place.

2:30:08 In fact, Google's success in the initial days came by reimagining what could be done in search, and I think The AI era we're entering gets us to think about it. This is the worst possible thing that could happen to Google. That

2:30:23 Now Microsoft can actually challenge Google on their own turf. Intent on the internet. With a legitimately different, better differentiated product vector.

2:30:36 Not what Bing was trying to do copycat. This is the full leapfrog and they have the technology partnership to do it. Or so everybody thinks at the moment. Oh my God. Terrifying. This is when Satya says the quote in an interview around this launch with Bing. I want people to know. That we made Google dance.

2:30:56 Oh boy. Well, hey, if you come at the king You'd best not miss. Right. And this big launch kinda misses. Yes. So What happens in Google?

2:31:08 December twenty twenty two. Even before the Bing Launch, but after. The chat GPT moment. Seniority is a code red within the company. And what does that mean?

2:31:18 Up until this point. Google and Sundar. And Larry and everyone. Had been thinking about AI. As a sustaining innovation.

2:31:28 Mm Clay Christensen's terms. This is great for Google. This is great for our products. Look at all these amazing things that we're doing. It further entrenches incumbents. It further is entrenching our lead in all of our already leading Products.

2:31:43 We can deploy more capital in a predictable way to either drive down costs or make our product experiences that much better than any startup could make. Yeah, more monetized that much better. All the things. Once ChatGPT comes out.

2:31:59 On a dime overnight. AI shifts from being a sustaining innovation To a disruptive. Innovation. It is now an existential threat.

2:32:09 And many of Google's Strengths from the last. ten, fifteen, twenty years of all the AI work that's happened in the company. Are now liabilities.

2:32:19 They have A lot of existing castles to protect. That's right. They have to run everything through a lot of filters. Before they can decide if it's a good idea to

2:32:28 Go try to Out, open AI, open AI. Yeah. So this code red That soon dark.

2:32:34 Issues to the company. is actually a huge moment. Because what it means and what he says is We need to Build.

2:32:44 And ship. Real Native AI products. A S A P This is actually what you

2:32:52 The textbook response to a disruptive innovation as the incumbent You need to not. bury your head in the sand and you need to say Okay, we need to like

2:33:02 Actually go. build and ship products that are comparable. to these disruptive innovators. And you need to be laser operationally

2:33:12 in all the details to try and figure out Where is it that the new product Is actually cannibalizing our old product. And where is it that the new product can be complimentary?

2:33:24 and just lean into all the ways in which you can be complimentary in all the different little scenarios. And really what they've been trying to do, this ballet from twenty twenty two onward. is protect the growth of search. While also creating the best AI experiences they can. And so it's very clever the way that they do AI overviews for some, but not all queries.

2:33:47 And they have AI mode for some, but not all users. And then they have Gemini, the full AI app. But They're not redirecting Google dot com to Gemini. It's this like very delicate dance of protecting the existing franchise while also building a hopefully non-cannibalizing as much as we can new franchise. Yep. And you see them.

2:34:08 really going hard in I think. building leading products in non search cannibalizing categories like video. Right. VO three or nano banana, these are things that

2:34:20 Don't in any way. cannibalize the existing franchise. They in fact use some of Google's strength, all the YouTube training data and stuff like that. Yeah. So What happens next? As you might expect

2:34:31 It gets worse before it gets better. Code red goes out December twenty twenty two. Bard, baby. Launch Bard. Oh boy. Well, Even before that. January twenty three. When

2:34:44 Open AI hits a hundred million registered users for Chat GPT. Microsoft announces they are investing another ten billion dollars in open AI. And says that they now own forty nine percent. of the for profit entity. Incredible.

2:35:00 In and of itself. But then now think about this from the Google lens of Microsoft, our enemy. They now Arguably own Obviously in retrospect here. They don't own open AI. But it seems at the time like, Oh my God, Microsoft might now own.

2:35:16 Open AI, which is our first true existential threat in our history as a company. Not great, Bob. So then February twenty twenty three, the big integration launches. Uh Satya has the quote about wanting to make Google dance. Meanwhile, Google is scrambling internally to launch AI products.

2:35:32 As fast as possible. So the first thing they do is they take the Lambda model and the chatbot interface to it. They rebrand it as Bard. They ship that publicly. And they release it. Immediately.

2:35:45 February twenty twenty three. Ship it publicly. Available GA to anyone. Which maybe was the right move, but God, it was a bad product. It was really bad. I didn't know the term at the time, R L H F, but it was clear it was missing

2:35:59 A component of Some magic that chat GPT had. This reinforcement learning with human feedback. Where you could really tune The Appropriateness, the tone, the voice.

2:36:10 the sort of correctness of the responses. It just wasn't there. Yeah. So To make matters worse.

2:36:18 In the launch video. For Bard. A video. This is a choreographed pre recorded video where they're showing Conversations with Bard. Bard gives an inaccurate factual response to one of the queries that they include.

2:36:34 in the video. This is one of the worst keynotes in history. After The bared launch and this keynote, Google stock drops eight percent. On that day. And then like we were saying, once the actual product comes out it becomes clear it's just not good. Yeah.

2:36:49 And it pretty quickly clear it's not just that the chatbot isn't good, it's the model isn't good. So In May. They replace Lambda with a new model from the brain team called Pom.

2:37:01 But it's still clearly behind Not only GPT three point five, but in March of twenty twenty three, OpenAI comes out with GPT four. Which is even better.

2:37:13 You can access that now through Chat GPT. And here is where Sundar makes two Really, really big decisions. Number one. He says,

2:37:23 We cannot have two AI teams within Google anymore. We're merging Brain and deep mind. into one entity called Google Deep Mind.

2:37:33 Which is a giant deal. This is in full violation. of the original Deal terms. Of bringing it.

2:37:42 Deep mind in. Yep. And the way He makes it work. is he says Demis, you are now CEO of the AI division of Google. Google Deep Mind.

2:37:53 This is all hands on deck. And You and deep mind. are gonna lead the charge, you're gonna integrate with Google Brain. And

2:38:01 We need to change. All of the past. Ten years of culture around building and shipping AI products within Google. To further illustrate this When alphabet became alphabet.

2:38:14 They had all these separate companies. But things that were really core to Google, like YouTube actually stayed a part of Google. DeepMind was its own company. That's how separate this was. They're working on their own models.

2:38:27 In fact, those models are predicated on reinforcement learning. That was the big thing that DeepMind had been working on the whole time. And so Reading in between the lines. It's Sundar looking at his two AI labs and going Look, I know you two don't actually get along that well.

2:38:43 But look, I don't care that you had different charters before. I am taking the responsibility of Google Brain and giving it to DeepMind and DeepMind is absorbing the Google Brain team. I think that's what you should sort of read into it because as you look at where the models went from here.

2:38:59 They kinda came from deep mind. Yep. There's a little bit of interesting backstory to this too. So Mustafa. Suleiman, the third co founder of Deep Mind.

2:39:09 At some point before this. He became like the head of Google AI policy or something? He had already shifted over to Brain and to Google. He stayed there for a little while. And then he ended up getting close with

2:39:25 Who else? Reed Hoffman. Remember, Reed is on the ethics board for Deep Mind. And Mustafa and Reed. Leave. And Go found

2:39:35 Inflection. AI. Which fast forward now into twenty twenty four after the Absolute insanity. that goes down at open AI.

2:39:45 In Thanksgiving twenty twenty three when Sam Altman gets fired over the weekend during Thanksgiving. And then Brought back by Monday when all the team threatened to quit and go to Microsoft. Can't wait for this year. They love Thanksgiving.

2:40:02 After all that, which certainly strains the Microsoft relationship. Remember again, Reed is on the board at Microsoft. Microsoft does one of these acquisition type deals. With Inflection AI

2:40:15 And brings Mustafa in. as the head of AI. For Microsoft. Crazy. Wild, right? Just wild. Crazy turn of events.

2:40:24 Okay, so That first big decision. that Sundar makes is unifying. Deep mind and brain. That was Huge.

2:40:32 Equally big, He says I want You guys.

2:40:39 And we're just gonna have one model. That is going to be the model for all of Google. Internally. For all of our AI products externally. It's gonna be called Gemini.

2:40:50 No more different models. No more different teams. Just one model for everything. This is also a huge deal. It's a giant deal.

2:41:00 And it's twofold, it's push and it's pull. It's saying, Hey, if anyone's got a need for an AI model, you gotta start using Gemini. But two, it's actually kind of the Google Plus thing where they go to every team and they start saying Gemini has our future. You need to start looking for ways to integrate Gemini into your product.

2:41:16 Yes. I'm so glad you brought up Google Plus. This came up with a few folks I spoke to in the research. Obviously this is all playing out. Real time. But the point a lot of people at Google made is

2:41:28 The Gemini situation is very different than the Google Plus situation. This is a technical thing. A. Which has always been Google's wheelhouse. But B, even more importantly. This is the

2:41:40 Rational Business. Thing to do. in the age of these huge models. Even for a company like Google.

2:41:48 There are massive scaling laws To models. The more data you put in, the better it's gonna get, the better all the outputs are gonna be. And Because of scaling laws

2:41:59 You need your models to be as big as possible in order to have the best performance possible. If you're trying to maintain multiple models within a company You're repeating Multiple huge costs. To maintain huge models.

2:42:14 You definitely don't want to do that. You need to centralize on just one model. Yeah, it's interesting. There's also something to read into Where at first it was the Gemini model. underneath the Bard product. Bard was still the consumer name.

2:42:28 Then at some point they said, No, we're just calling it all Gemini and Gemini became the user facing name also. This Pulls in. My Quintessence from the alphabet episode.

2:42:40 I know it's a little bit We will. But with Google saying we're actually gonna name the consumer service the name of the AI model. They're sort of admitting to themselves

2:42:51 This product is nothing but Technology. There isn't Productiness. to do on top of it. It's just like Gmail.

2:42:59 Gmail was technology. It was fast search. It was lots of storage. It was use it in the web. The productiness wasn't particular the way that like Instagram was all about the product. Gemini the model, Gemini the chat bot says We're just exposing our amazing breakthrough technology to you all. and you get to interface directly with it.

2:43:19 Anthropologically looking from afar, it kind of feels like it's that principle at work. I totally agree. I think it's actually a really important Branding point. And sort of rallying point too.

2:43:31 Google and Google culture to do this. Right. All right, so this is all the stuff going on in Google. twenty twenty three ish. in AI.

2:43:40 Before we catch up to the present I have a whole other branch of alphabet. That has been a real bright spot. Four.

2:43:48 AI. Can I go there? Can I take this off ramp, if you will. Can you uh take the wheel, so to speak? May I take the wheel. May I investigate another bet. Yeah. Please tell us the Waymo story.

2:43:59 Awesome. So we gotta rewind back. All the way to two thousand and four. The DARPA. Grand challenge. Which was created as a way to spur research.

2:44:09 into autonomous ground robots for military use. And actually what it did for our purposes here today is create the seed talent. For the entire self driving car revolution. twenty years later. So the competition itself is really cool. There is a 132 mile race course. Now, mind you, this is 2004.

2:44:28 In the Mojave Desert. that the cars have to race on. It is a dirt road. No humans are allowed to be in or interact with the cars. They are monitored. A hundred percent remotely.

2:44:40 And the winner gets One million dollars. One million dollars. Which was a break from policy. Normally these are grants, not prize money. So This needs to be authorized by an act of Congress. The one million dollars eventually felt comical. So the second year they raised the pot to two million dollars. It's crazy thinking about what these researchers are worth today, that that was the prize for the whole thing.

2:45:04 So the first year in two thousand four went fine. There were some amazing tech demonstrations on these really tight budgets, but ultimately Zero of the one hundred registered teams finish the race. But the next year in two thousand and five was the real special year. The progress that the entire industry made in those first twelve months from what they learned.

2:45:24 is totally insane. Of the twenty three finalists that were entering the competition, Twenty of them. Made it past the spot. where the furthest team the year before had made it.

2:45:35 The amount that the field advanced in that one year is insane. Not only that. Five of those teams actually finished. All hundred and thirty two miles. Two of them were from Carnegie Mellon.

2:45:47 And one was from Stanford. led by a name that all of you will now recognize. Sebastian Thrawn. Mm-hmm. Indeed. This is Sebastian's origin story before Google.

2:45:59 Yeah. As we said, Sebastian was kind enough to help us with prep for this episode, but I actually learned most of this from watching a twenty year old Nova documentary. that is available on Amazon Prime Video. Thanks to Brett Taylor for giving us the tip on where to find this documentary. Yes. The hot research tip. So what was special about this Stanford team?

2:46:18 Well, one, there's a huge problem with noisy data that comes out of all of these sensors. You know, it's in a car in the desert getting rocked around. It's in the heat, it's in the sun. So common wisdom and what Carnegie Mellon did was to do as much as you possibly can on the hardware to mitigate that. So things like custom rigging and gimbal and giant springs to stabilize the sensors. Carnegie Mellon would essentially Buy a hummer. and rip it apart and rebuild it from the wheels up. We're talking like welding and

2:46:46 Real construction on a car. The Stanford team did the exact opposite. They viewed any new piece of hardware as something that could fail. And so in order to mitigate risks on race day, They used all commodity cameras and sensors that they just mounted on a nearly unmodified Volkswagen. So they only innovated in software.

2:47:04 And they figured they would just kind of come up with clever algorithms to help them clean up the messy data later. Very googly, right? Very googly. The second thing they did. was an early use of machine learning to combine multiple sensors.

2:47:18 They mounted laser hardware on the roof, just like what other teams were doing. And this is the way that you can measure texture and depth of what is right in front of you. And the data, it's super precise. But you can't drive very fast because you don't really know much about what's far away, since it's this fixed field of view. It's very narrow. Essentially you can't answer that question of How fast can I drive? Or is there a turn coming up? So on top of that, the way they solved it was

2:47:46 They also mounted a regular video camera. That camera can see a pretty wide field of view, just like the human eye, and it can see all the way to the horizon, just like the human eye. And crucially It can see color. So what it would do

2:47:59 This is like Really clever. They would use a machine learning algorithm. In real time. In two thousand and five.

2:48:07 This computer is like sitting in the middle of the car. They would overlay the data from the lasers on top. On to the camera feed. And from the lasers you would know if the area right in front of the car was okay to drive or not. Then the algorithm would look up in the frames coming off the camera overlaid.

2:48:23 What color? That safe area was And then extrapolate by looking further ahead at other parts of the video frame to see where that safe area extended to. So you can figure out your safe path through the desert. That's awesome.

2:48:37 It's so awesome. I'm imagining like a Dell PC sitting in the middle of this car in two thousand five. It's not far off. In the email that we send out, we'll share some photos of it. It could then drive faster with more confidence and it knew when turns were coming up. Again, this is real time onboard the camera, two thousand and five.

2:48:56 is wild on that tech. So ultimately both of these bets worked and the Stanford team won in super dramatic fashion. They actually passed one of the Carnegie Mellon teams autonomously through the desert. It's like this big dramatic moment. in the documentary. So you would kinda think So then Sebastian goes to Google and builds Waymo.

2:49:16 No. As we talked about earlier. He does join Google through that crazy Please don't raise money from benchmark in Sequoia and we'll just hire you instead. But

2:49:25 He goes and works on Street View and Project Ground Truth and co founds Google X. David, as you were alluding to earlier. This project chauffeur that would become Waymo. is the first project.

2:49:38 Inside Google X. And I think the story, right, is that Larry came to Sebastian and was like, Yes Yo, that self driving car stuff. Like, do it. And Sebastian was like, No, come on, that was a DARPA challenge. And Delari's like, No, no, you should do it. He's like, No, no, no, that won't be safe. There's people running around cities. I'm not just gonna put multi ton

2:49:57 Killer robots on roads. And go and potentially harm people. And Larry finally comes to him and says Why?

2:50:04 What is the technical reason that this is impossible? And Sebastian goes home. has a sleep on it, and he comes in the next morning and he goes, I realized what it was. I'm just afraid.

2:50:16 Such a good moment. So they start He's like there's not a technical reason As long as we can take all the right precautions and hold a very high bar on safety Let's get to work. So

2:50:29 Larry then goes, Great, I'll give you a benchmark. So that way you know if you're succeeding. He comes up with these ten stretches of road in California that he thinks will be very difficult to drive. It's about a thousand miles. And the team starts calling it the Larry One Thousand. And it includes driving to Tahoe.

2:50:45 Lombard Street in San Francisco, Highway One to Los Angeles. The Bay Bridge. This is the bogey. Yep. If you can autonomously drive these stretches of road

2:50:57 Pretty good indication that you can probably do anything. Yeah. So they start the project in two thousand and nine, within eighteen months, this tiny team I think they hired I don't know, it's like a dozen people or something.

2:51:08 They've driven thousands of miles autonomously. And they managed to succeed. in the full Larry one thousand. Within eighteen months. Totally.

2:51:18 Unreal. How fast. They did. And then also totally unreal. How long it takes after that to productize and create

2:51:28 The Waymo that we know today. Right. It's like the first ninety nine percent and then the second ninety nine percent that takes 10 years. Yeah. Self-driving is one of these really tricky types of problems where it's surprisingly easy to get started, even though it seems like it would be an impossible thing, but then there's Edge cases everywhere, weather, road conditions, other drivers, novel road layouts. night driving. So it takes this massive amount of work. for a production system to actually happen.

2:51:54 So then the question is what business do we build? What is the product here? And there was what Sebastian wanted, which was highway assist. Sort of the lowest stakes. Most realistic

2:52:04 Let's make a better cruise control. There's what Eric Schmidt wanted. Which is crazy. He proposed Oh, let's just go by Tesla and that'll be our starting place.

2:52:13 And then we'll just put all of our self driving equipment on all the cars. David, do you know what it would have cost to buy Tesla at the time? I think At the time that negotiations were taking place between Elon and Larry and

2:52:26 Google. This was in the depths of the Model S. Production scaling woes. I think Google could have bought the company for five billion dollars. That's what I remember.

2:52:36 It was three billion. Three billion dollars. Oh my goodness. Obviously that didn't happen, but what a crazy alternative history that could have been. Right. I mean, I think if that had happened Deepmind would not have gone down in the same way and probably OpenAI would not have gotten founded.

2:52:52 Who? That's probably right. I think that is Obviously unprovable. Right, the counterfactuals that we always come up with on this show, you can't know.

2:53:01 Yeah. Seems more likely than not to me. That at a minimum open AI would not exist. Right. So then there was what Larry wanted to do.

2:53:10 Option three. Build robo taxis. Yeah. And ultimately. That is at least right now.

2:53:17 What they would end up doing. So We could do a whole episode about this journey, but we will just hit some of the major points for the sake of time. The big thing to keep in mind here Neither Google nor the public really knew if self-driving was something that could happen in the next two years.

2:53:33 from any given point or take another ten. And just to illustrate it, for the first five years of Project Chauffeur It did not use deep learning. At all. They did the Larry one thousand without

2:53:46 any deep learning and then win another three and a half years. Well. That's crazy. Yeah. And yeah, totally illustrates. You never know. How far away

2:53:56 The end goal is. And this is a field that comes from The only way progress happens is through these series of breakthroughs. And you don't know A, how far the next breakthrough is, because at any given time, there's lots of promising things in the field, most of which don't work out. And then B, when there is a breakthrough,

2:54:12 Actually, how much lift that will give you over existing methods. So anytime people are forecasting, oh, and AI we're going to be able to do XYZ in X years. It's a complete fool's errand. Even the experts don't know. Here are the big milestones. Twenty thirteen, they started using convolutional neural nets. They could identify objects, they got much better perception capabilities. This twenty thirteen, twenty fourteen period is when Google found religion around deep learning. So this is like

2:54:36 right after the 40,000 GPUs rolled out. So they've actually got Some hardware to start doing this on now. Twenty sixteen they've seen enough technology proof that they think Let's commercialize this. We can actually spin this out into a company. So Waymo becomes its own.

2:54:51 subsidiary inside of Alphabet. It's no longer a part of Google X anymore. Twenty seventeen, obviously the Transformer comes out. they incorporate some learnings from the transformer. Especially around prediction and planning.

2:55:03 March of twenty twenty, they raise three point two billion dollars from folks like Silver Lake Canada Pension and Investment Board, Mubadala. And recent Horowitz and of course The biggest check, I think. Alphabet and I think they're always the biggest check because Alphabet is still the majority owner. even after a bunch more fundraises.

2:55:21 In October of twenty twenty. They launched the first public commercial no human behind the driver's seat thing. In Phoenix. It's the first in the world. This is eleven years after succeeding in the Larry One Thousand. And this is nuts.

2:55:36 I had given up at this point. I was like, that's cute that Waymo and all these other companies are trying to do self driving. Seems like it's never gonna happen. And then They actually were doing a large volume of rides safely with consumers and charging money for it in Phoenix. Then they bring it to San Francisco where

2:55:53 For me and lots of people in San Francisco, it is a huge part of life in the city here now. It's Amazing. Yeah, every time I'm down, I love taking them. They're launching in Seattle soon, I'm pumped. Interestingly, they don't make the hardware, so they use a Jaguar vehicle. Yep. That

2:56:09 From what I can tell you. Is Only in Waymo's. Like I don't know if anybody else drives that Jaguar or if you can buy it. But they're working on a sort of van next. They have some next generation hardware.

2:56:21 For anyone who hasn't taken it, it's an Uber, but with no driver. And that launched in June of twenty four. Along the way there, they raised their quote unquote series B, another two point five billion. Then after the San Francisco roll out, they raised their quote unquote series C five point six billion. This year, in January

2:56:41 They were reportedly doing more in gross bookings than lift in San Francisco. Um I totally believe it. I mean it is the number one option in San Francisco. That I and everybody I know to always goes to for Right.

2:56:55 It's like try to get away, Mo. If there's not a Waymo available anytime soon, then go down the stack. Like we're living in the future and how quickly we fail to appreciate it. Yeah. And what's cool, I think for people who it hasn't come to their city and is not part of their lives yet. It's not. Just

2:57:11 That it's a cool experience to not have a driver behind the like pretty quickly that just Fades. It's actually a different experience. Mm.

2:57:19 So If I need to go somewhere with My older daughter. I don't mind hailing a Waymo, bringing the car seat, installing the car seat in the Waymo and driving with my daughter. And she loves it. We call it a robot car and she's like, A robot car, I'm so excited. Huh. I would never do that with an Uber. That's interesting.

2:57:37 To my dog. Whenever I need to go with my dog, like it's super awkward to hail an Uber and be like, Hey, I got my dog, you know, can the dog come in it? Not a big deal. With A Waymo. And then when you're in town. Yeah, we can actually have sensitive conversations in the car. You can have phone calls. It really is a different experience.

2:57:54 Yeah. That's so true. Yeah, so may as well catch up to today. They're operating in five cities. Phoenix, San Francisco, LA Austin and Atlanta. They have hundreds of thousands of paid rides every week.

2:58:06 They've now driven over a hundred million miles with no human behind the wheel, growing at two million every week. There's over ten million paid rides across two thousand vehicles in the fleet. They're gonna be opening a bunch more cities in the U Us next year. They're launching in Tokyo, their first international city. Slowly and then all at once. I mean, that's kind of the lesson here.

2:58:27 The technology They really continued with that multi-sensor approach all the way from the DARPA Grand Challenge. Camera, LIDAR, they added radar. And actually they use audio sensing as well. And their approach is basically any data that we can gather is better because that makes it safer.

2:58:43 So they have thirteen cameras, four LIDAR, six radar, and the array of external microphones. This is obviously way more expensive of a solution. than what Tesla is just doing with cameras. But Waymo's party line is they believe it is the only path to full autonomy to hit the safety bar and regulatory bar that they're aiming for.

2:59:01 Yeah. It seems like a really big line in the sand for them anytime you talk to somebody in that organization. Yeah, and look as a regular user of Both products, so you know. happy owner and driver of a model Y in addition to regular Waybo user.

2:59:15 At least with the current instantiation of full self driving on my Tesla. Vastly different products. Full self driving. on my model Y is great. I use it all the time on the freeway. But

2:59:27 I would never not pay attention. Whereas every time I get in a Waymo It's almost like Google search, right? It's like I just trust that Oh, this is going to be completely and totally safe. And I'm sitting in the back seat and I can totally tune out. I think I trust my model Y FSD more than you do, but I get what you're saying. And frankly, regulatorily you are required to still pay attention in Tesla and not in the Waymo.

2:59:50 The safety thing is super real, though. I mean, if you look at the numbers Over a million Motor vehicle. crashes cause fatalities every year, or there's over a million fatalities. In the US alone.

3:00:02 Over forty thousand deaths occur per year. So if you break that down, that's a hundred and twenty. Every day. It's like Yeah.

3:00:10 giant cause of death. Yes. The study that Waymo just released last month showed that they have ninety-one percent fewer crashes with serious injuries or worse. compared to the average human driver. Even controlled.

3:00:24 for the fact that Waymo's right now are only driving on city surface streets. So they controlled it apples to apples. With human driving data and it's a ninety one percent reduction. in those serious either fatality or serious things.

3:00:38 Why aren't we all talking about this all the time every day? This is gonna completely change the world in a giant cause of death. Yeah. So While we're in Waymo Land, what do you think about doing some quick analysis? Great.'Cause I've been scratching my head here of

3:00:51 What is this business? Then I promise we'll go back to the rest of Google AI and catch up to today. It is super expensive to operate, especially at early scale.

3:01:02 The training is high, the inference is high, the hardware is high. Excer, et cetera, et cetera. Also The operations are expensive. Yes. And in fact they're experimenting some cities, they actually outsource the operations. So the fleet

3:01:15 is Managed by There's a rental car company in Texas that manages it, or they've partnered, I believe, with Lyft and with Uber and different so they're trying all sorts of O and O versus uh partnership models to operate it. Yeah. And the operations are like

3:01:30 These are electric cars. They need to be charged. They need to be cleaned. They need to be returned to depots. They need to be checked out. They need to have sensors replaced. So the question is What is the potential market opportunity. How big could this business be? And there's a few different ways you could try to quantify it.

3:01:47 one total market size thing you could do is try to sum the entire auto maker market cap today. And That would be two and a half trillion. globally, if you include Tesla or one point three trillion without

3:02:01 But Waymo's not really making cars, so that's probably the wrong way to slice it. You could look at all the ride sharing companies today, which might be a better comp because that's the business that Waymo is actually in today. That's on the order of three hundred billion, most of which is Uber. Yeah.

3:02:16 That's addressable market cap. Today with ride sharing. Waymo's ambitions though are bigger than that. They wanna be in the cars that you own. They wanna be in long haul trucking.

3:02:27 So They believe they can grow the share. of transportation because there's blind people that could own a car. There's elderly people who could get where they need to go on their own without having a driver, that sort of thing. So

3:02:40 the most squishy, but I think the most interesting way to look at it is What is the value from all of the reduction in accidents. 'Cause that's really what they're doing. It's a product

3:02:52 to replace accidents with non accidents. I think that's viable, but I I again I would say as a regular user of the product, it is a Different and expanding product to human ride share. So your argument is whatever number I come up with. Four. reducing accidents, it's still a bigger market than that because there's additional value created in the product experience itself. Yeah. Scoping just to ride share.

3:03:16 Now that we have Waymo in San Francisco I use Waymo in scenarios where I would never use an Uber or a Lyft. Yeah. Make sense. So here's the data we have.

3:03:27 The C D C released a report saying deaths from crashes in twenty twenty two in the US Resulted in four hundred and seventy billion dollars in total costs, including medical costs and the cost estimates for lives lost. Which is crazy that the CDC has some way of putting the cost on human life, but they do.

3:03:44 So if you reduce crashes 10 X, which is what Waymo seems to be seeing in their data, at least for the serious crashes, That's over four hundred and twenty billion dollars a year. in total costs that we would save as a nation. Now it's not Totally apples to apples. I recognize this, but

3:03:59 That cost savings. Is more. than Google does today in revenue in their entire business. You could see a path to a Google sized opportunity for Waymo as a standalone company. just through this analysis, as long as they figure out a way to get

3:04:14 cost down to the point where they can run this as a large and profitable business. Yeah. It is a Incredible.

3:04:23 twenty plus year success story. Within Google. The way I want to close it is The investment so far actually hasn't been that large when you consider this opportunity. They have burned somewhere in the neighborhood of ten to fifteen billion dollars. That's sort of why I was listing all the investments to get to this point. Jump change compared to foundational models.

3:04:43 Dude, also, let's just keep it scoped in this sector. That's One year of Uber's profits. Wow. Seems like a good bet.

3:04:53 I used to think this was like some wild goose chase. It now looks Really, really smart. Yeah. Totally agree.

3:05:00 Also that costs ten to fifteen billion. is the profits that Google made last month. Google.

3:05:11 Well Speaking of Google. Should we catch us up to Today with Google AI. Yes.

3:05:17 So I think where you were is the Gemini launch. So Soon dark. Makes these Two decrees.

3:05:25 mid twenty twenty three. One, we're merging Brain. And deep mind. into one team. For AI within Google.

3:05:33 And two, we're gonna standardize on one model. The future Gemini. And Deep mind. You go build it and then

3:05:43 Everybody in Google, you're gonna use it. Not to mention apparently Sergey Brand is like now back as an employee working on Gemini. Yes. Employee number Got his new badge back. Yeah, got his badge back. So Once Sundar makes these decisions Jeff Dean and Oriel Vinyalis from Brain.

3:06:05 go over and team up with the Deep Mind team and they start working on Gemini. I'm a believer now, by the way. You got Jeff Dean working on it, I'm in. If you got Jeff Dean on it, it's probably gonna work. If you weren't a believer, yeah, wait till I'm gonna tell you next. Once they get Noam back, when they do the deal with character AI, bring him back into the fold.

3:06:23 Noins the Gemini team and Jeff and Noam are the two Co technical leads. For Gemini now. So Let's go. Let's go. So they actually Announced this

3:06:36 Very quickly. Uh the Google IO keynote in May twenty twenty three. They announce Gemini, they announce the plans. They also

3:06:47 Launch AI overviews in search. First as a labs product and then later. That becomes just Standard for Everybody using Google search?

3:06:56 Which is crazy, by the way. The number of Google searches that happen Is unfathomably large. I'm sure there's a number for it, but just think about That's about the highest level of computing scale that exists.

3:07:07 other than like high bandwidth things like streaming. But just think about the instances of Google searches that happen. They are running. On LLM inference. On all of those.

3:07:17 Or at least as many as they're willing to show AI overviews on, which I'm sure is not every Quiry, but many of you a subset. Yeah. But still a large, large number of Google search. I mean, I see them all the time. Yeah. This is really Google. immediately deciding to operate at

3:07:31 A I speed. I mean chat GPT happened November thirtieth. twenty twenty two. We're now in May twenty twenty three. All of these decisions have been made, all of these changes have happened and they're announcing things at IO. And they're really flexing the infrastructure that they've got. I mean the fact that they can go like oh yeah, sure, let's do inference on every query. We're Google, we can handle it. So

3:07:52 A key part of this new Gemini model that they announced in May twenty twenty three. is it's gonna be multimodal. Again, this is one model for everything. Text, images, video, audio, one model.

3:08:04 They Release it. For early public access in December.

3:08:10 twenty twenty three. So also crazy. Six months. They build it, they trade it, they release it. That is amazing. Wild.

3:08:18 February twenty twenty four, they launched Gemini one point five. With a one million token context window. much, much larger context window than any other model. On the market. Which enables all sorts of new use cases. There's all these people who were like, Oh, I tried to use AI before, but it couldn't handle my XYZ use case.

3:08:37 Now they can. Yep. The next year, February twenty twenty five, they released Gemini two point oh. March of twenty twenty five, one month later. They launched Gemini.

3:08:48 Two point five pro in Experimental. mode and then that goes GA in June. This is like NVIDIA pace, how often they're shipping. Yeah, seriously.

3:08:58 And also in March of twenty twenty five. They launch AI mode. So you can now switch over on Google.com to Chatbot mode. And they're split testing.

3:09:08 auto-opting some people into AI mode to see what the response is. This is the golden goose. Yeah. The elephant is tap dancing here. Yep. Then there's all the

3:09:19 other AI products that they launch. So Notebook L M Comes out. During this period. AI generated. Podcast.

3:09:27 Which Does that sound like us to you? The number of texts that we got when that came out of This must be trained on acquiring. I do know that a bunch of folks on the notebook LM team are acquired fans, so

3:09:41 I don't know if they trained on us. And then there's the video, the image stuff, VO three, Nano Banana. Genie three that just came out recently. Genie, this is insane. This is a World builder. Based on prompts and videos. Yeah.

3:09:55 You haven't actually used it yet, right? You watch that hype video. Yeah, I watched the video. I haven't actually used it. Yeah. I mean, if it does that, that's unbelievable. It's a real time generative

3:10:05 World Builder. world builder. Yeah. You look right and it invents stuff to your right. I mean you combine that with like a Vision Pro hardware. You're just living in a fantasy land. So they announced there are now four hundred and fifty million monthly users of

3:10:21 Gemini. Now that includes everybody who's accessing Nano Banana. Yeah, I can't believe this stat. This is insane. Even with recently being number one in the app store. It still feels hard to believe. Google's saying it, so it must be true, but I just wonder what are they counting as use cases of the Gemini app?

3:10:39 Right. Certainly. Everybody who's using Nano Banana is using Gemini. But is it counting AI overviews or is it counting AI mode? Or is it counting something where I'm like accidentally like Meta said. that crazy high number of people using meta AI. That was complete garbage. That was people searching Instagram who accidentally hit

3:10:58 a llama model that made some things happen and they were like oh go away. I actually am just looking for a user. Is it really four hundred and fifty million or is it Four hundred fifty million. Yeah. Good question.

3:11:08 Either way, going from Zero. crazy impressive in the amount of time that they have done. Especially given revenues at an all time high. They seem to so far be

3:11:19 at least in this squishy early phase, able to figure out how to keep the core business going. Wow. Doing well. as a competitor in the cutting edge of AI. Yeah. And

3:11:30 To foreshadow a little bit to we're gonna do a bull and bear here in a minute. As we talked about in our alphabet episode, Google does have a history of navigating platform shifts. Incredibly well in the transition of mobile. It's true.

3:11:44 Definitely a rockier start here in the AI platform shift. Much rockier, but Hey, look, I mean If you were to lay out a recipe for

3:11:55 How to respond, given the rocky start. Better slate of things than what they've done over the last Two years. Yeah.

3:12:04 All right. Should I give us the snapshot of the business today? Oh yeah, also by the way. the federal government decided they were a monopoly. And then decided not to do anything about it because of AI.

3:12:17 Yeah, so between the time when we shipped our alphabet episode and here with our Google AI episode. Or our uh Part two and part three for those who prefer simpler naming schemes. Yeah.

3:12:29 There was A US versus Google antitrust case. The judge. First ruled that Google Was

3:12:38 a monopoly in internet search. And then did not come up with any material remedies. I mean there are some But I would call them immaterial.

3:12:47 They did not need to spin off chrome. And they did not need to stop sending tens of billions of dollars to Apple and others. In other words, yes, Google's a monopoly and the cost of doing anything about that. would have too many downstream consequences on the ecosystem. So we're just gonna let them keep doing What they're doing.

3:13:05 And one of the reasons that the judge cited Of why they weren't gonna really take these actions. is because of the race in AI. that because tens of billions of dollars of funding have gone into companies like

3:13:18 open AI. And anthropic. And perplexity. Google essentially has this new war to fight and we're gonna leave it to the free market to do its thing. where it creates viable competition on its own.

3:13:30 And we're not gonna hamstring Google. Personally. I think this argument is a little bit silly. I mean, none of these AI companies are generating net income and just because they've raised a huge amount of money. it doesn't mean that will last forever. They'll all burn through their existing cash in a pretty short period of time. And if the spigots ever dry up, Google doesn't have any self-sustaining competition right now, whether in their old search business or in AI. It is all

3:13:54 dependent. on people believing that the opportunity is so large that they keep pouring tens of billions of dollars into these competitors. Yeah. Plenty of other folks have made the sort of glib comment, but There's merit to it of

3:14:07 Hey, as flat footed as Google was when ChatGPT happened. If the outcome of this is They avoid a Microsoft level distraction and damage to their business from a U S Federal court monopoly.

3:14:23 Judgment. Worth it. Well, there's a funny meme here that you could draw. You know that meme of someone pushing the domino and it knocking over some big wall later? Yeah. There's the domino of

3:14:34 Ilya leaving Google. to start open AI. And the downstream effect is Google is not broken up. Yeah, right, exactly. It actually saves Google. It actually saves Google. It's totally wild.

3:14:49 Totally well. All right, so here's the business today. Over the last twelve months, Google has generated three hundred and seventy billion. Dollars. in revenue.

3:14:58 On the earning side. They've generated a hundred and forty billion over the last twelve months. Which is more profit. than any other tech company. And the only company in the world with more earnings.

3:15:11 Is Saudi Aramco. Let's not forget. Google is the best business ever. And we also made the point at the end of The alphabet episode. Even in the midst of all of this AI era and everything that's happened over the last

3:15:27 Ten years the last five years. Google's core business. has continued to grow. Five X since the end. of our alphabet episode in twenty fifteen, twenty sixteen.

3:15:39 Yeah. market cap, Google surge past their old peak of two trillion and just hit that three trillion mark. earlier this month. They're the fourth most valuable company in the world behind NVIDIA, Microsoft, and Apple. It's just crazy.

3:15:55 On their balance sheet. Actually, I think this is pretty interesting. I normally don't look at balance sheet as a part of this exercise, but it's useful, and here's why. In this case. They have 95 billion in cash and marketable securities. And I was about to stop there and make the point, wow, look at how much cash and resources they have. I'm actually surprised it's not more. So it used to be a hundred and forty billion in 2021. And over the last four years. they've massively shift from this mode of accumulating cash.

3:16:20 to deploying cash. And a huge part of that has been the capex of the AI data center build out. So they're very much playing offense in the way that Meta, Microsoft, and Amazon are. in deploying that CapEx. But

3:16:34 The thing that I can't quite figure out is the largest part. Of That was actually Buybacks. And they started paying dividend.

3:16:43 So If you're not a finance person, the way to read into that is Yes, we still need a lot of cash for investing in the future of AI and data centers. But we still actually had way more cash than we needed and we decided to distribute that to shareholders.

3:16:57 Yeah. That's crazy. Best business of all time, right? That illustrates What a crazy business their core search ads business is if they're saying

3:17:07 The most capital intense race in business history is happening right now. We intend to win it. Yeah. And

3:17:15 We have tons of extra cash lying around on top of what we think Plus a safety cushion. for investing in that CapEx race. Yeah. Yes.

3:17:26 Wow. So There are two businesses. That are worth looking at here.

3:17:32 One is Gemini to try to figure out what's happening there. And two is a brief history of Google Cloud. I wanna tell you the cloud numbers today, but it's probably worth actually understanding how did we Get here on cloud.

3:17:42 Yep. First on Gemini. because this is Google and they have I think the most obfuscated financials of any of the companies we've studied. They anger me the most. in being able to hide the ball in their financial statements. Of course we don't know Gemini specific revenue.

3:17:57 What we do know is there are over a hundred and fifty million paying subscribers. to the Google one bundle. Most of that is on a very low tier. It's on like the five dollar a month, ten dollar a month. The AI stuff kicks in on the twenty dollar a month tier where you get the premium AI features, but I think that's a very small fraction of the hundred and fifty million today. Yeah, I think that's what I'm on.

3:18:18 But two things to note. One, it's growing quickly. That Hundred and fifty million is growing almost fifty percent year over year. But two is Google has A subscription bundle.

3:18:29 that 150 million people are subscribed to. And so I've kinda had it in my head. That AI doesn't have a future as a business model.

3:18:37 that people pay money for, that it has to be ad supported like search. But hey. That's not Nothing. That's like a That's almost half of America. I mean, how many subscribers does Netflix have? Netflix is in the hundreds of millions. Yeah. Spotify is now

3:18:52 A quarter billion, something like that. Yep. We now live in a world. where there are real scaled consumer subscription services. I owe this insight to Shashir Moroto.

3:19:05 We chatted actually last night. 'Cause I name dropped him in the last episode and then he heard it and so we reached out and we talked And That's made me do a one eighty. I used to think if you're gonna charge for something your total addressable market shrunk by ninety to ninety nine percent.

3:19:18 But He kinda has this point that If you build a really compelling bundle. And Google has the digital assets Oh my goodness. YouTube Premium

3:19:29 NFL Sunday ticket. Yes. Stuff in the play store. YouTube music all the Google One storage stuff. They could put AI in that bundle and figure out through clever bundle economics.

3:19:41 A way to Make a paid AI product that actually reaches a huge number of paying subscribers. Totally. So We really can't figure out how much money Jim and I

3:19:50 makes right now. Probably not profitable anyway. So what's the point of even analyzing it? Yeah. But okay. Tell us the cloud story. So

3:19:58 We intentionally did not Include cloud in our alphabet episode. Google Part two, effectively. Google Part two, yes. Because It is a

3:20:09 New product and now very successful one within Google that was started during the same time period as all the other ones that we talked about during Google Part two. But it's so strategic for AI. Yes.

3:20:23 Yeah. Is A lot more strategic now in hindsight. than it looked when they launched it. So

3:20:29 Just quick background on it. It started as Google App Engine. It was a way in two thousand and eight for People to quickly spin up. a backend for a web or soon after a mobile app.

3:20:41 it was a platform as a service, so you had to do things in this very narrow Googly way. It was very opinionated. You had to use this SDK, you had to write it in Python or Java, you had to deploy exactly the way they wanted you to deploy. It was not a thing Where they would say, Hey developer

3:20:58 You can do anything you want, just use our infrastructure. It was opinionated. super different than what AWS was doing at the time and what they're still doing today, which the whole world eventually realize was right. Which is cloud should be infrastructure as a service. Even Microsoft pivoted Azure to this reasonably quickly.

3:21:16 Where it was like, You want some storage, we got storage for you. You want a VM, we got a VM for you. You want some compute, you want a database. We gotcha. Fundamental building blocks. So eventually Google launches their own infrastructure as a service in twenty twelve, took four years. They launched Google Compute Engine that they would later rebrand Google Cloud Platform. That's the name of the business today. The knock on Google.

3:21:37 is that they could never figure out how to possibly interface with the enterprise. Their core business they made Really great products for people to use. that they loved polishing. They made them all as self serve as possible. And then the way they made money was from advertisers And let's be honest.

3:21:53 There's no other choice but to use Google search. Right. It didn't necessarily need to have a great enterprise experience for their advertising customers'cause they were gonna come anyway. Right. So they've got this

3:22:04 Self serve experience. Meanwhile, the the cloud is A knife fight. These are commodities. All about the enterprise. It's the lowest possible price, and it's all about enterprise relationships and clever ways to bundle and being able to deliver a full solution. You say solution, I hear gross margin. Yes. But yes, so Google

3:22:24 out of their natural uh habitat in this domain. And early on, they didn't want to give away any crown jewels. They viewed their infrastructure as this is our secret thing. we don't want to let anybody else use it. And the best software tools that we have on it that we've written for ourselves, like Big Table or Borg, how we run Google, or Distbelief These are not services that we're making available on Google Cloud. Yep. These are competitive advantages. Yes. And then

3:22:50 They hired the former president of Oracle. Thomas Carrion. Yes. And everything Kinda changed.

3:22:57 So twenty seventeen. Two years before he comes in, they had four billion dollars in revenue. Ten years into running this business. Twenty eighteen. their first very clever strategic decision. They launch Kubernetes.

3:23:09 The big insight here is Is if we make it More portable. for developers. to move their applications to other clouds.

3:23:18 The world is kinda wanting multi-cloud here. Right. We're the third place player. We don't have anything to lose. Yes. So we can offer this tool a kind of counterposition against AWS and Azure. We shift the developer paradigm to use these containers. They orchestrate on our platform, and then you know, we have a great service to manage it for you. It was very smart. So this kind of becomes one of the pillars of their strategy is you want multi-cloud, we're gonna make that easy and you can sure choose AWS or Azure too. It's gonna be great. So David, as you said,

3:23:47 The former president of Oracle, Thomas Curian. Is hired. in late twenty eighteen. You couldn't ask for her. A better person who understands the needs of the enterprise.

3:23:58 than the former president of Oracle. This shows up in revenue growth. right away. In twenty twenty, they crossed thirteen billion in revenue, which was nearly tripling in three years. They hired like ten thousand people into the go to market organization. I'm not exaggerating that, and that's on a base of a hundred and fifty people when he came in, most of which

3:24:19 were seeded in California, not regionally distributed throughout the world. The funniest thing is Google kind of was A cloud company all along. They had the best engineers building this amazing infrastructure. Right. They had the products. They had the infrastructure. They just didn't have the go to market organization. Right.

3:24:36 And the productization was all like Googly. It was like for us, for engineers. They didn't really build things that let enterprises build the way they wanted to build. This all changes. Twenty twenty two They hit twenty six billion in revenue. Twenty twenty three, they're like a real viable third cloud. They also flipped to profitability in twenty twenty three.

3:24:56 And today They're over fifty billion dollars in annual revenue run rate. It's growing thirty percent year over year. They're the fastest growing of the major cloud providers five X in five years. And it's really three things.

3:25:09 It's Finding religion on how to actually serve the enterprise. It's leaning into this multi-cloud strategy and actually giving enterprise developers what they want. And three. AI has been such a good

3:25:21 Tailwind. for all hyperscalers, because these workloads all need to run in the cloud, because it's giant amounts of data and giant amount of compute and energy. But In Google Cloud, you can use TPUs Which they make a ton of

3:25:35 And everyone else is desperately begging NVIDIA for allocations to GPUs. So if you're willing to Not use CUDA and build on Google Stack.

3:25:46 They have an abundant amount of TPUs for you. This is Why we saved cloud for this episode. There are two aspects of Google Cloud.

3:25:57 That I don't think they first saw back when they Started the business with App Engine. But are hugely strategically important to Google today. One. Is

3:26:07 Just simply that Cloud is the distribution mechanism for AI. So If you want to play an AI today, you either need to have a great application A great model.

3:26:19 A great ship. All right, great. Cloud. Google. Is

3:26:24 Trying to have all four of those. Yes. There is no other company. That has I think More than one.

3:26:34 I think that's the right call. Think about the big AI players. Nvidia Chips. Kinda has a cloud, but not really. They just have chips. The best chips and the chips everyone wants, but chips. And then you just look around the rest of the big tech companies.

3:26:46 Meta right now. Only an application. They're completely out of the race for the frontier models at the moment. We'll see what their hiring spree yields. You look at

3:26:55 Amazon. Infrastructure. They have application. Maybe I don't actually know if Amazon.com I'm sure it benefits from L L Ms in a bunch of ways. Mainly it's cloud.

3:27:04 Yes, cloud. And cloud leader. Microsoft. Cloud. It's just cloud, right? They make some models, but I mean they've got applications, but

3:27:13 Yeah. Cloud. Cloud. Apple? Nothing. Nothing. A M D Just chips?

3:27:20 Yep. Open AI. Model. Anthropic. model.

3:27:26 Yep. Yeah. These companies don't have their own data centers. They are like making noise about making their own chips, but Not really.

3:27:32 And certainly not at scale. Google has scale data center. Scale chips. scale usage of model. I mean even just from Google.com queries now on AI overviews. And

3:27:43 Scale applications. Yes. Yeah. They have Uh Of the pillars.

3:27:49 of AI. And I don't think any other company has more than one. And they have the very most net income dollars to lose. Right. So then there's the chip side specifically of this. If Google didn't have a cloud It wouldn't have a chip business. It would only have an internal chip business.

3:28:04 The only way Yeah. External. Companies, users, developers, model researchers could use TPUs. would be if Google had a cloud to deliver them, because there's no way in hell that Amazon or Microsoft are going to put TPUs from Google in their clouds.

3:28:20 We'll see. We'll see, I guess. I think within a year it might happen. There are rumors already that some Neo Clouds in the coming months are gonna have TPUs. Mm, interesting. Nothing announced. But TPUs are likely gonna be available in Neo Cloud soon. Which is an interesting thing. Why would Google do that? Are they trying to build an NVIDIA type business where they make money selling chips?

3:28:42 I don't Think so. I think it's more that they're trying to build an ecosystem around their chips the way that CUDA does. And you're only gonna credibly be able to do that if your chips Harb.

3:28:54 Accessible and Anywhere that someone's running their existing workloads. Yeah. be very interesting if it happens. And you know, look, you may be right. Maybe there will be TPUs in AWS or Azure someday.

3:29:07 But I Don't think they would have been able to start. There. If Google didn't have a cloud and there weren't any way for developers to use TPUs and start Wanting to be used.

3:29:18 Would Amazon or Microsoft be like? Uh, you know, all right, Google. We'll take some of your TPUs, even though no developer out there uses them. Right. All right.

3:29:29 Well with that. Let's move into analysis. I think we need to do bull and bear. on this one. You have to this time. Gotta bring that back. For these episodes in the present, it seems like we need to paint the possible futures. Yes. Bringing back Bull and Bear. I love it. We'll do playbook.

3:29:44 Powers. Quintessence. Bring it home. Perfect. All right. So

3:29:50 Here's my set of bull cases. Google has distribution. to basically all humans as the front door to the internet. They can funnel that however they want. You've seen it with AI overviews.

3:30:01 You've seen it with AI mode. even though lots of people use chat GPT for lots of things. Google's traffic, I assume, is still essentially an all time high and it's a default behavior. Yeah.

3:30:12 Powerful. So That is a bet on implementation that Google figures out how to execute and build a great business out of AI, but It is still theirs to lose.

3:30:23 Yeah. And they've got a viable product. It's not clear to me that Gemini is Any worse than open AI or Anthropics products. No, I completely agree. This is a value creation, value capture thing. The value creation is there in spades. The value capture mechanism is still TBD.

3:30:39 Yeah. Google's old value capture mechanism. is one of the best in history. So that's the issue at hand. Let's not get confused that it's not like a good experien. It's a great experience. Yeah, yeah.

3:30:49 Yeah. Okay. So we've talked about the fact that Google has all the capabilities to win an AI and it's not even close. Foundational model. Chips. hyper scalar All this with self sustaining funding.

3:31:02 I mean that's the other crazy thing is you look at The clouds have self sustaining funding. NVIDIA has self sustaining funding. None of the model makers have self sustaining funding. So they're all dependent on external capital. Yeah.

3:31:14 Google is the only model maker who has self sustaining funding. Yes. Isn't that crazy? Yeah. Basically all the other Large scale usage foundational model companies are effectively startups. Yes.

3:31:27 And Google's is funded by a money funnel so large that they're giving extra dollars back to shareholders for fun. Yeah. Again, we're in the bull case. Well when you put it that way. Yeah. A thing we didn't mention, Google has incredibly fat pipes connecting all of their data centers.

3:31:44 After the dot com crash in two thousand, Google bought all that dark fiber for pennies on the dollar. And they've been activating it over the last decade. They now have their own private backhaul network between data centers. No one has infrastructure like this. Yep.

3:31:59 Not to mention. That serves YouTube. They're fat pipes. Which in and of itself is its own bullcase for Google in the future.

3:32:09 That's a great point. Yeah. Ben Thompson had a big article about this yesterday at the time of recording. Yeah. That was like a mega bull case that Ben Thompson published this week that It was an interesting point. A text based internet. is kinda the old internet. It's the first instantiation of the internet because we didn't have much bandwidth.

3:32:27 the user experience that is actually compelling. Is Video. High resolution video. Everywhere all the time. The YouTube internet.

3:32:38 Right. And not only can they train models on really the only scale source of UGC. media across long form and short form. But They also

3:32:49 have that as the number two search engine, this massive destination site. So they previewed things like you'll be able to by AI labeled or AI determined things that show up in videos. And if they wanted to. They could just go label every single product in every single video and make it all instantly shoppable.

3:33:06 Doesn't require any human work to do it. They could just Do it. and then run their standard ads model on it. That was a mind expanding piece that Ben published yesterday. Or I guess if you're listening to this a few weeks ago about that.

3:33:17 And then there's also all the Video. AI applications that They've been Building like

3:33:24 Flow and VO. What is that gonna do? for generating videos for YouTube that will Increase engagement and add dollars for you too.

3:33:34 Yeah. Yep. They still have an insane talent bench. Even though you know they've bled talent here and there and lost people. They have also shown they're willing to spend billions for the right people and retain them.

3:33:47 Unit economics. Let's talk about the unit economics of chips. Everyone is paying NVIDIA. Seventy five, eighty percent gross margins. Implying something like a four or five X markup.

3:33:58 on what it costs to make the chips. A lot of people refer to this as the Jensen tax or the NVIDIA tax. Uh you can call it that, you can call it good business, you can call it pricing power, you can call it scarcity of supply, whatever you want. But That is True.

3:34:12 Anyone who doesn't make their own ships is paying a giant giant premium to Nvidia. Google has to still pay some margin to their chip hardware partner, Broadcom, that handles a lot of the work to actually make the chip. interface with T SM C

3:34:26 I have heard that Broadcom has something like a fifty percent margin. when working with Google. on the TPU versus NVIDIA's eighty percent. But that's still a huge difference to play with.

3:34:38 A fifty percent gross margin from your supplier or an eighty percent gross margin from your supplier is the difference between A two X markup. in a five X markup. Yeah, I guess that's right.

3:34:49 When you frame it that way, it's actually a giant difference. of the impact to your cost. So you might wonder. appropriately. Well, are chips actually the big part of the cost of like the total cost of ownership of running one of these data centers or training one of these models.

3:35:04 Chips are the main driver of the cost. They depreciate very quickly. I mean, this is at best a five year depreciation because of How fast. We are

3:35:13 pushing the limits of what we can do with chips. the needs of next generation models. How fast TSMC is able to produce. I mean, even that is ambitious, right? If you think you're gonna get five years of depreciation on AI chips five years ago

3:35:29 We were still two years away from chat GPT. Right. Or think about what Jensen said at um we were at GTC this year. He was talking about Blackwell. He said

3:35:38 something about Hopper and he was like, eh, you don't want Hopper. My sales guys are gonna hate me, but like you really don't want hot at this point. I mean these were the H one hundreds. This was the hot chip just when we were doing our most recent NVIDIA episode. Yes. Things move quickly. Yes.

3:35:52 So I've seen estimates that over half the cost of running an AI data center Is the chips and the associated depreciation. the human cost that R D is actually a pretty high amount because hiring these AI researchers and all the software engineering.

3:36:07 Call it. twenty five to thirty three percent. The power is actually a very small part. It's like two to six percent. So when you're thinking about the economics of doing what Google's doing It's actually incredibly sensitive.

3:36:19 to how much margin are you paying your supplier. In the chips. 'Cause it's the biggest cost driver of the whole thing. Mm-hmm. So I was sanity checking some of this with Gavin Baker, who's the partner at Atreides Management to prep for this episode. He's like a great public equitior who's studied this space for a long time. We actually interviewed him at the NVIDIA GTC pregame show.

3:36:41 And he pointed out normally, like in historical technology eras It hasn't been that important to be the low cost producer. Google. didn't win because they were the lowest cost search engine. Apple didn't win because they were the lowest cost. You know, it's not what makes people win.

3:36:56 But This era. might actually be different. 'Cause these AI companies don't have eighty percent margins the way that we're used to

3:37:05 In the technology business, or at least in the software business. At best, these AI companies look like fifty percent gross margins. So Google being definitively the low cost provider of tokens. Because

3:37:17 They operate all their own infrastructure, and because They have access to low markup hardware. it actually makes a giant difference and might mean that they are the winner. in producing tokens for the world. Very compelling bill case there.

3:37:32 That's a weirdly winding analytical bullcase, but it's kinda the if you want to really get down to it. They produce tokens. Yep. I've got One more.

3:37:42 bullet point to add to the bookcase for Google here. Everything that we talked about In part two. The alphabet episode.

3:37:51 Oh Of the other products within Google, Gmail, Maps. Docs, Chrome, Android. That is all. Personalized data about you.

3:38:01 That Google owns. That they can use. To create personalized AI products. For you. That nobody else has.

3:38:09 Another great point. So really the question to close out the bull case is Is AI a good business to be in compared to search? Search is a great business to be in. So far AI is not.

3:38:20 But in the abstract Again, we're in the bull case, so I'll give you this. It should be. With traditional web search, you type in Two to three words. That's the average query length. And I was talking to Bill Gross and he pointed out that in AI chat.

3:38:34 You're often typing twenty plus words. So There should be an ad model that emerges. And Ad rates should actually be dramatically higher because you have perfect precision. Right. You have even more intent.

3:38:48 Yes, you know the crap out of what that user wants. So you can really decide to target them with the ad. or not. And AI should be very good at targeting with the ad. So it's all about figuring out The user interface the mix of paid versus not, exactly what this ad model is, but in theory, even though we don't really know what the product looks like now, it should actually lend itself very well to monetization.

3:39:10 Yep. And since AI is such a Amazing transformative experience. All these interactions that were happening in the real world or weren't happening at all, like answers to questions and being on a time spent.

3:39:23 is now happening in these AI chats. So It seems like the pie is actually bigger. for digital interactions. Than it was. in the search era. So again, monetization should kind of

3:39:34 Increase because the piece. increases there. Yeah. And then you've got the bull case of Waymo could be its own Google size business. I was just thinking that. Yeah.

3:39:43 That's scoping all of this to A replacement to the search market. Waymo and potentially other applications of AI beyond The traditional search market. Could add to that.

3:39:56 Right. And then there's the like galaxy brain bull case, which is if Google actually creates AGI. None of this even matters anymore. And like, of course, it's the most valuable thing. That feels out of the scope for An acquired episode. It's disconnected. Yes, agree.

3:40:11 Bear case. So far, this is all fun to talk about, but then the product shape of AI has not lent itself well to ads. So despite more value creation. There's way less value capture. Google makes something like four hundred ish dollars per user per year just based on some napkin math.

3:40:29 In the US. That's A free service that everyone uses. And they make four hundred ish. dollars a year.

3:40:37 Who's gonna pay four hundred dollars a year? For access to AI. It's a very thin slice of the population. Some people certainly will, but not every person in America. Some people will pay 10 million. But Right. So

3:40:51 If you're only looking at the game on the field today. I don't see the immediate path. Two value capture.

3:41:00 And think about when Google launched in nineteen ninety eight. It was only two years before they had AdWords. They figured out an amazing value capture mechanism instantly. Very quickly. Yeah.

3:41:10 Another Spare case. Think back to Google launch in nineteen ninety eight. It was

3:41:16 Immediately obviously the superior product. Yes. Definitely not the case today. No. There's four five great products.

3:41:24 Google's Dedicated AI offerings and chat bot was Initially the Immediately obviously inferior product. And now it's arguably on par with several others.

3:41:35 Right. They own ninety percent of the search market. I don't know what they own of the AI market, but It ain't ninety percent. Is it twenty five percent? I don't know, but at steady state it probably will be something like twenty five, maybe up to fifty percent. But

3:41:48 This is gonna be a market with several big players in it. So Even if they monetized each user as great as they monetize it in search. They're just gonna own way less of them. Yeah. Or at least it certainly seems that way right now.

3:42:00 Yes. AI might take away the majority of the use cases of search. And even if it doesn't, I bet it takes away a lot of the highest value ones. Mm-hmm. If I'm planning a trip.

3:42:12 I'm planning that in AI. I'm no longer searching on Google for things that are gonna land expedia ads in my face. Or health, another huge vertical. Hey, I think I might have something that reminds me of misothelioma. Is it that or not? Right. Oh, where are you gonna put the lawyer ads? Maybe you put'em there, maybe it's just an ad product thing, but These are very high value queries former searches that those feel like some of the first things that are getting siphoned off to AI. Yeah. Any other bear cases?

3:42:42 I think the only other bare case I would add is that They have the added challenge now of being the incumbent this time around. And People and the ecosystem isn't necessarily rooting for them in the way that people were rooting for. Google

3:42:57 When they were a startup. And in the way that people were still rooting for Google In the mobile transition. I think The startups

3:43:06 Have more of the hearts and minds. These days. Right. So I don't think that's quantifiable. But is Just gonna make it all a little

3:43:15 harder path to row this time around. Yeah. You're right. They had this incredible PR and public love tailwind the first time around. Yep. And part of that's systemic too. Like all of tech and all of big tech is just

3:43:28 generally more out of favor with the country in the world now than it was ten or fifteen years ago. Where's more important? It's just big infrastructure. It's not underdogs anymore. Yeah.

3:43:38 And that affects. The open AIs and the anthropics and the startups too. But I think to a lesser degree. Yeah, they had to start behaving like big tech companies really early in their life. compared to Google. I mean, Google gave a Playboy interview during their quiet period of their IPO.

3:43:55 Times have changed. Well, I mean, given all the drama at OpenAI, I I don't know that I characterize them as uh acting like a mature company. Fair. Fair. Company, entity, whatever they are. Yes. Yeah. Point taken. Well, I worked most of my playbook into the story itself. So you wanna do power?

3:44:15 Yeah, great. Let's move on into Power. Hamilton Helmers Seven Powers analysis of Google here in the AI era. And the seven powers are scale economies, network economies, counter positioning, switching costs, branding, quartered resource, and process power.

3:44:30 And the question is which of these enables a business to achieve persistent differential returns. What entitles them to make greater profits than their nearest competitor sustainably. Normally we would do this on the business all up. I think for this episode we should try to scope it to AI products. Yes, agree. Usage of

3:44:51 Gemini AI mode and AI overviews versus a competitive set of anthropic open AI perplexity grok. Meta I Et cetera. Scale economies, for sure.

3:45:05 Even more so in AI than traditionally in tech. Yeah, they're just way better. I mean, look, they're amoritizing the cost of model training across every Google search. I'm sure it's some super distilled down model that's actually happening for AI overviews. But Think about

3:45:21 How many inference tokens. are generated for the other model companies and how many inference tokens are generated by Gemini. they just are amoretizing that fixed training cost over a giant Giant.

3:45:35 Amount of inference. Th I saw some crazy charts. We'll send it out to email subscribers. In april of twenty four. Google

3:45:43 was processing ten trillion tokens across all their surfaces. In April of twenty five That was almost five hundred trillion. Wow. That's a fifty x increase in one year.

3:45:58 Of the number of tokens. That they're vending out across Google services through inference. And between April of twenty five and June twenty five it went from A little under five hundred trillion.

3:46:09 One quadrillion tokens, technically nine hundred and eighty trillion. But they are now,'cause it's later in the summer, definitely sending out Maybe even multiple quadrillion tokens. Wow.

3:46:23 Wow. So among all the other obvious scale economies things of amortizing all the costs of their hardware. they are amortizing the cost of training runs over a massive amount of value creation. Yeah. Scale economies. Must be.

3:46:38 The biggest one. I find switching costs to be relatively low. I use Gemini for some stuff, then it's really easy to switch away. That probably stops being the case when it's personal AI to the point that you're talking about integrating with your calendar and your Mail and all that stuff. Yeah, the switching costs

3:46:54 Have not. Really? come out yet in AI products. Although I expect they will. Yes. They have within the enterprise for sure.

3:47:04 Yeah. Network economies. I don't think if anyone else is a Gemini user, it makes it better for me because they are sucking up the whole internet, whether anyone's participating or not. Yeah.

3:47:15 Agree. I'm sure AI companies will develop network economies over time. I can think of ways it could work, but yeah, right now, no. And Arguably for the foundational model companies. Can't.

3:47:27 Think of obvious reasons right now. Where does Hamilton put distribution? Because that's a thing that they have right now that no one else has, despite ChatGPT having the Kleenex brand. Google distribution is still unbelievable. I don't know. Is that a cornered resource? Cornered resource, I guess. Yeah. Definitely of that. Yeah, Google search is a cornered resource, for sure.

3:47:46 Certainly don't have counter positioning. They're getting counter positioned. Yeah. I don't Think they have process power. Unless they were like coming up with the next transformer or

3:47:56 Reliably. But I don't think we're necessarily seeing that. There's great research being done at a bunch of different labs. Branding they have. Yeah. Branding is a funny one, right? Well, I was gonna say it's a little bit to my

3:48:08 Bear case point about They're the incumbent. It cuts both ways, but I think it's net positive. Yeah, probably. For most people they trust Google. Yeah, they probably don't trust these who knows AI companies, but I trust Google. I bet that's actually stronger than any downsides, as long as they're willing to still release stuff on the cutting edge.

3:48:27 Yeah. So to sum it up, it's scale economies is the biggest one. It's branding. And it's a cornered resource. And potential for switching costs in the future.

3:48:36 Yep. Sounds right to me. But it's telling that it's not all of them. You know, in search it was like very obviously all of them or most of them. Yep. Quite telling. Well, I'll tell ya after

3:48:46 Hours and hours. spending multiple months learning about this company. My quintessence. When I boil it all down. is just that this is the most fascinating example of the innovator's dilemma.

3:48:57 Ever. I mean Larry and Sergey Control the company. They have been quoted repeatedly. saying that they would rather go bankrupt

3:49:05 And lose an AI. Will they really? If AI Isn't as good a business. as search.

3:49:12 And it kinda feels like of course it will be. Of course it has to be. It's just because of the sheer amount of value creation. But if it's not. And there are choosing between two outcomes. One is fulfilling our mission.

3:49:27 Accessible. And useful. And having the most profitable tech company in the world. Which one wins.

3:49:36 'Cause if it's just the mission. they should be way more aggressive on AI mode than they are right now and full flip over to Gemini. It's a really hard needle of thread. I'm actually very impressed. and how they're managing to currently protect the core franchise. But

3:49:51 It might be one of these things where it's being eroded away at the foundation in a way that just somehow isn't showing up. In the financials yet. I don't know. Yeah. I totally agree.

3:50:03 And in fact Perhaps influenced by you. I think my quintessence is a version of that. Two. I think if you look

3:50:12 At all the big. Tech companies. Google as unlikely as it seems given how things started is is probably doing the best job of trying to thread the needle. With AI?

3:50:27 Right now. And that is Incredibly commendable. Two. Sundar and their leadership.

3:50:34 They are making hard decisions. Like We're unifying Deep mind and brain. We're consolidating and standardizing on one model.

3:50:43 And we're gonna ship this stuff. Real fast. While at the same time not making Rash decisions. It's hard. Rapid but not rash, you know. Yes.

3:50:55 And Obviously we're still in early innings. of all this going on. And we'll see in ten years where it all ends up.

3:51:03 Yeah. Being tasked with being the steward of a mission is And the steward of a franchise. with public company shareholders. is a hard dual mission.

3:51:14 And Sundar and the company is handling it remarkably well. Especially given where they were five years ago. And I think This will be one of the most fascinating examples in history to watch it play out. Totally agree.

3:51:28 Well Thus concludes Our Google series. For now. Yes.

3:51:34 Alright, let's do some carve outs. All right. Let's Do subcarbots. Well, first off

3:51:40 We have a uh Very Very fun announcement. To share with you all. The NFL called us.

3:51:47 We're going to the Super Bowl, baby. Acquired is going to the Super Bowl. This is So cool. It's the craziest thing ever. The NFL.

3:51:56 is hosting a innovation summit. The week at the Super Bowl, the Friday before Super Bowl Sunday. The Super Bowl is going to be in San Francisco this year in February. And so it's only natural. coming back to San Francisco with the Super Bowl that the NFL should do an innovation.

3:52:13 Summit. Yeah. And we're gonna host it. That's right. So the Friday before

3:52:18 There's gonna be some great on stage interviews and programming. most of you, you know, we can't fit. millions of people in a tidy auditorium in San Francisco. the week of the Super Bowl when every other venue has Tons of stuff too.

3:52:32 So there'll be an opportunity to watch that streaming online. And as we get closer to that date in February. We will make sure that you all know a way that you can tune in and watch the uh MCing, interviewing and festivities at hand Super Bowl week. It's gonna be An incredible, incredible day.

3:52:50 Leading up to An incredible Sunday. Yes. Well speaking of sport My carve out is I finally went and saw F one.

3:52:59 It is great. I highly recommend anyone go see it, whether you're an F one fan or not. It is just Beautiful cinema. Amazing Did you see it in the theater or I did see it in the theater, yeah.

3:53:09 Wow unfortunately missed the IMAX window, but It was great. It was my first time being in a movie theater in a while and whether you watch it at home or whether you watch it in the theater, I recommend the theater, but It's gonna be a great surround sound experience wherever you are. Oh I haven't been to the movie theater since the Eras tour.

3:53:24 Ah. Which I think is just more about the current state of my family life with two young children. Yes. My second one, so you're gonna laugh. is the travel pro suitcase.

3:53:37 Uh This is the brand that pilots and flight attendants use, right? Maybe I think I've seen some of them use it. Usually they use something higher end, like a Briggs and Riley or A Tumi or you know, travel pro is not the most high end suitcase. But I bought two really big ones.

3:53:53 For some international travel that we were doing with my two year old toddler. And I must say They're robust the wheels glide really well they're really smooth they have all the features you would want They're soft shell, so you can like really jam it full of stuff.

3:54:08 But it's also a thick amount of protection. So even if you do jam it full of stuff, it's probably not gonna break. This is approximately the most budget suitcase you could buy. I mean, I'm looking at the big honkin international checkbag version. It's four hundred and sixteen dollars on Amazon right now.

3:54:25 I've seen it cheaper. They have great sales pretty often. Everything about this suitcase. checked lots of boxes for me and I completely thought I would be the person buying the Ramoa suitcase or the Something very high end. And this is just perfect.

3:54:41 So I think I may be investing in more travel pro suitcases. More travel pro. Nice, nice. Well, I mean, hey, look, for family travel, you don't want nice stuff. Yeah, I mean I bought it thinking like I'll just get something crappy for this trip. But it's been great. I don't understand why I wouldn't have a full lineup of travel pro gear. So this is my like Budget pick gone right that I highly recommend for all of you. I love how uh Quiet is turning into the wire cutter here.

3:55:08 That's it for me today. Great. All right. I have To Carve outs. I have one carve out and then I have a

3:55:15 update in my ongoing Google Carve Out Saga. But first my actual car own. It is The Glue Guys podcast. Oh, it's great.

3:55:26 Those guys are awesome. So great. Our buddy Ravi Gupta, partner at Sequoia, and his Buddy, Shane Badier. the former basketball player and Alec Smith, the former quarterback for the Forty Niners and the Kansas City Chiefs and the Redskins.

3:55:41 Their dynamic is so great. They have so much fun. Half of their episodes like us are just them. Then half of their episodes are with guests. Ben and I, we went on it a couple of weeks ago. That was really fun. When we were on it. We are talking about this dynamic of

3:55:55 Some episodes do better than others and pressure for episodes and whatnot and The guys brought up this interview they did with a guy named Wright Thompson. And they said, like, look, this is an episode. It's got like five thousand listens. Nobody's listened to it. It's so good. And

3:56:11 The Mentality that We have about it is Not that we're embarrassed that Nobody listened to it.

3:56:17 It's that we feel sorry for the people who have not yet listened to it because it's so good. I was like, that is the way to think about your episode. So here you are. You're giving everyone the gift of giving everyone the gift because I then I was like, all right, well, I gotta go listen to This episode. Ray Thompson. I didn't know anything about him before I'd probably read his work in magazines over the years without realizing it. He's the coolest dude. He

3:56:41 has the same accent as Bill Gurley. So listening to him sounds like listening to Bill Gurley, instead of being a VC Only wrote about sports and basically dedicated his whole life to understanding the mentality and psychology of athletes and coaches. It's so cool. It's so cool. It's a great episode. Highly, highly, highly recommend. All right. Legitimately I'm cuing that up right now.

3:57:04 That's my car out. And then my Ongoing family. Video gaming. Saga.

3:57:12 In Google Part One I said I was debating between the switch two and the steam deck. That's right. First you got the Steam Deck because you decided your daughter actually wasn't old enough to play video games with you, so you just got the thing for you. The update was I went with the Steam Deck for that reason. I thought if it's just for me, it would be more ideal.

3:57:31 I have an update. You also got a switch. Uh no, not yet. Okay, okay. But the most incredible thing happened. My daughter notice this device that appeared in our house that

3:57:43 Dad plays every now and then. And we were on vacation. And I was playing the Steam Deck and she was like, What's that? Well, let me tell you And I was playing, I've been playing This Really cool indie.

3:57:56 old school style RPG called Sea of Stars. It's like a Chrono Trigger style, Super Nintendo style RPG. I'm playing it. My daughter comes up, she's like, Can I watch you play? And I'm like Hell yeah, you can watch me play. I get to play video games and you sit here and snuggle with me and like you know amazing. I get to play video games and call it parenting.

3:58:18 Then It gets even better. Probably like Two weeks ago. We're playing

3:58:24 And she's like Hey Dad, can I try? I'm like Absolutely you can try. I hand her. The Steam Deck.

3:58:30 And It was the most incredible experience, one of the most incredible experiences I've had as a parent because she doesn't know how to play video games. And I'm watching her learn how to like use a joystick and hit the button. Supervised learning. Yeah, yeah, yeah. Supervised learning. I'm telling her what to do. And then Within

3:58:48 Two or three nights she got it. She doesn't even know how to read yet, but she figured it out. I'm watching her in real time. And so now The last week It's turned to mostly she's playing. And I'm like

3:59:00 Helping her asking questions of like Well what do you think you should do here? Like, you know, should you go here? I think this is the goal. I think this is where it's So so fun. So I think I might actually pretty soon End up getting a switch so that we can play

3:59:16 You know, together on the switch. Right. But unintentionally the Steam Deck was the gateway drug. For my soon to be four year old daughter. That's awesome. There you go.

3:59:25 Parent of the year right there, getting to play video games and Honey, I got it. I'll I'll take it. Oh yeah, I got it. I got it. All right, well, it's there is we have lots of Thank you. to make for this episode. We talked to so many folks who were instrumental in helping put it together.

3:59:42 As always, all of our sources for this episode are linked. In The show notes. Yes. First.

3:59:48 Steven Levy at Wired and his Great. Classic book on Google in the Plex, which is been an amazing source for all three of our Google episodes. Definitely go buy the book and read that.

4:00:00 Also to Parmi Olson at Bloomberg for her book Supremacy about Deep mind and open AI, which was a main source for this episode. And I guess also decayed Mets, right? For genius makers. Yeah. Yeah. Great book. Our research thank yous.

4:00:14 Max Ross. Liz Reed, Josh Woodward, Greg Carrado, Sebastian Thrun. Anna Patterson, Brett Taylor, Clay Bevore. Demis Asabis, Thomas Kirrion, Sun Dark Chai. Mm.

4:00:27 Special thank you to Nick Fox. So Is the only person we spoke to for all three. Google episodes for research. We got the hat trick.

4:00:36 Yeah. To Arvin Navarotnam at Worldly Partners for his great write up on Alphabet linked in the show notes. To Jonathan Ross? original team member on the TPU. And today the founder and CEO of Grok. That's Grok with a Q.

4:00:50 Making chips for inference. To the Waymo folks, Dmitry Doglov and Suzanne Filion. to Gavin Baker from Atreides Management, to MG Siegler. Writer at Spy Glass. MG is just one of my favorite

4:01:03 Technology writers and pundits. OG Tech Crunch Rider. That's right. To Ben Eidelson for being a great thought partner on this episode and his excellent recent episode on the Step Change podcast on the history of data centers. I highly recommend it if you haven't listened already. It's only episode three for them of the entire podcast and they're already getting

4:01:22 I don't know, thirty, forty thousand listens on it. I mean, this thing is Taken off. Amazing. Dude, that's way better than we were doing on episode three. It's way better than we were doing. And if you like Acquired, you will love the Step Change podcast, and Ben is a dear friend. So Highly recommend checking it out.

4:01:37 to Korai Kovakchalu from the Deep Mind team building the core Gemini models. Shashir Moroda, the CEO of Grammarly, formerly ran product at YouTube, to Jim Gow. CEO of Phaedra and former Deep Mind team member. Cha think Puttagunta, partner at Benchmark, Dwarcash Patel. for helping me think through some of my conclusions to draw and to Brian Lawrence from Oak Cliff Capital.

4:01:59 for helping me think about the economics of AI data centers. If you like this episode, go check out our episode on the early history of Google and the 2010s with our alphabet episode, and of course. our series on Microsoft and NVIDIA. After this episode, go check out ACQ2 with Toby Lutke, the founder and CEO of Shopify. And come talk about it with us in the Slack at acquire.fm slash slack.

4:02:22 And don't forget, our 10th anniversary celebration of Quired. We are gonna do a Open Zoom call, an LP call, just like the days of your with anyone, listeners, come join us on Zoom. It's gonna be on October twentieth. At four PM

4:02:38 Pacific time. Details are in the show notes. And with that listeners. We'll see you next time. See you next time.

4:02:45 Who got the truth? Is it you, is it you, is it you Who got the truth now? Oh.

What you see above is a preview of the first minutes. One unlock costs 10 credits and covers this episode forever: full segment and word-level timestamps on this page, plus .txt, .srt, .vtt and word-level JSON downloads, as many times as you like.