WEBVTT - Goldman Sachs CIO on How the Bank Is Actually Using AI 0:00:02.680 --> 0:00:19.919 Bloomberg Audio Studios, Podcasts, radio News. Hello and welcome to 0:00:19.960 --> 0:00:23.600 another episode of the Odd Blots podcast. I'm Tracy Alloway. 0:00:23.320 --> 0:00:24.520 And I'm Joe Wisenthal. 0:00:24.800 --> 0:00:28.920 Joe, what's been your favorite chat GPT or claude prompt 0:00:29.040 --> 0:00:29.400 so far? 0:00:31.320 --> 0:00:33.040 You know, it's funny because I have a lot of 0:00:33.040 --> 0:00:37.040 fun with them, and also I use them for serious things. 0:00:37.040 --> 0:00:40.560 So I'll like upload conference call transcripts and say, tell 0:00:40.600 --> 0:00:44.440 me what this company said about labor market indicators or 0:00:44.520 --> 0:00:46.880 something like that, and that'll be extremely useful for that. 0:00:47.159 --> 0:00:49.519 Wait, do you actually find that more efficient than just 0:00:49.520 --> 0:00:52.720 doing a word search for like labor or working? I don't. 0:00:52.760 --> 0:00:54.760 I hate uploading stuff because you can only do it 0:00:54.760 --> 0:00:55.840 in like fragments. 0:00:56.120 --> 0:01:01.960 No, what, Tracy, Oh, let me, I'll show you how prompt? Okay, No, 0:01:02.160 --> 0:01:05.040 I get a lot of professional use out of the 0:01:05.160 --> 0:01:08.080 various AI tools, but I also, you know, have a 0:01:08.120 --> 0:01:11.039 lot of fun with them. And there's even a song 0:01:11.160 --> 0:01:13.480 and I'm not going to say which one that I wrote. 0:01:13.880 --> 0:01:16.840 I didn't use the lyrics. No, I did not like 0:01:16.880 --> 0:01:18.800 because it's very good. Wait what did you use? 0:01:18.920 --> 0:01:21.080 Did he give you an actual melody? What happened? 0:01:21.160 --> 0:01:21.280 No? 0:01:21.560 --> 0:01:24.720 So there was a song that I liked, okay, and 0:01:24.840 --> 0:01:28.600 the song title sort of rested upon a pun okay, 0:01:28.959 --> 0:01:31.800 and so I asked chat GPT to come up with 0:01:31.920 --> 0:01:38.040 another song that sort of like had a similar twist 0:01:38.240 --> 0:01:40.880 based on the headline of that song. I needed basically 0:01:40.920 --> 0:01:43.000 a song prompt idea. 0:01:43.240 --> 0:01:45.800 This opens up a whole can of worms. No, this 0:01:45.920 --> 0:01:48.600 is actually the perfect segue into what we're going to 0:01:48.680 --> 0:01:52.400 talk about today, because for you and I, using something 0:01:53.040 --> 0:01:56.400 like a chat GPT, we don't really have the same 0:01:56.520 --> 0:02:01.800 concerns that a proper company or large which corporation would have, Like, 0:02:01.840 --> 0:02:04.640 it doesn't really matter to us if the answer is wrong. 0:02:04.720 --> 0:02:06.680 I mean, ideally you would like it to be correct, 0:02:06.680 --> 0:02:10.000 but if I'm just asking some silly question, it doesn't 0:02:10.000 --> 0:02:12.560 really matter what chat gpt spits out at me. And 0:02:12.680 --> 0:02:16.119 also copyright kind of doesn't matter, so we don't care 0:02:16.320 --> 0:02:18.400 what it spits out in terms of who owns it, 0:02:18.440 --> 0:02:21.040 and also we don't care what we're putting in in 0:02:21.160 --> 0:02:23.920 terms of who owns that. That's right, But if you 0:02:23.960 --> 0:02:27.919 are a company you are thinking about generative AI very differently. 0:02:28.280 --> 0:02:30.040 I just want to say one thing, which is. 0:02:29.960 --> 0:02:32.200 That your defense Okay, defend yourself. 0:02:32.240 --> 0:02:35.320 No, No, I'm not even trying to defend myself. If I upload, say, 0:02:35.760 --> 0:02:38.000 you know, the McDonald's earning transcript, and I say, what 0:02:38.040 --> 0:02:40.720 does McDonald say about the labor market, then there's some quote. 0:02:41.040 --> 0:02:43.440 I always go back and check that that quote is 0:02:43.520 --> 0:02:45.799 actually in there. So I do very good, you know, 0:02:45.840 --> 0:02:49.040 I'm not just blindly relying on it. I do also 0:02:49.320 --> 0:02:52.120 do my own work and everything. But yeah, it's very true. 0:02:52.160 --> 0:02:54.480 Like so I can say I get a tremendous amount 0:02:54.480 --> 0:02:56.840 of use from chat, GPT or Claude or whatever, and 0:02:56.880 --> 0:03:00.600 it is very useful to me. But it makes mistakes sometimes, 0:03:00.880 --> 0:03:03.920 and if you think about deploying AI in the sort 0:03:03.960 --> 0:03:08.720 of enterprise world, then maybe like a one percent mistake 0:03:08.840 --> 0:03:11.799 raid or a one percent hallucination or you ever want 0:03:11.800 --> 0:03:14.960 to call them, is just completely unacceptable and a level 0:03:15.000 --> 0:03:19.880 of risk that makes it almost unusual for professional purposes. 0:03:19.919 --> 0:03:22.480 Absolutely. And of course the other thing with AI is 0:03:22.639 --> 0:03:25.880 there is still this ongoing, very heated debate about how 0:03:25.919 --> 0:03:29.200 transformational it's actually going to be. So you and I 0:03:29.400 --> 0:03:32.160 are using it as you know, a productivity hack in 0:03:32.240 --> 0:03:35.880 some cases, or maybe to generate song lyrics or even 0:03:36.280 --> 0:03:41.720 songs in some cases, but what is the true use 0:03:41.840 --> 0:03:44.480 case for this particular technology. There's still a lot of 0:03:44.480 --> 0:03:47.800 debate about that, and so I'm very pleased to say 0:03:47.840 --> 0:03:50.400 we do, in fact have the perfect guest. We're going 0:03:50.440 --> 0:03:54.280 to be speaking to someone who is implementing AI at 0:03:54.280 --> 0:03:57.080 a very, very large financial institution. We're going to be 0:03:57.080 --> 0:04:02.080 speaking with Marco Urgenti, the chief information officer at Goldman Sachs. Marco, 0:04:02.160 --> 0:04:03.760 thank you so much for coming on of thoughts. 0:04:04.200 --> 0:04:05.200 Thank you for having me. 0:04:05.600 --> 0:04:08.680 Marco tell us what a chief information officer does at 0:04:08.680 --> 0:04:11.920 Goldman Sachs. Whenever I see CIO, I always think chief 0:04:11.920 --> 0:04:15.240 investment officer, as it's very confusing. Yeah, so what does 0:04:15.280 --> 0:04:16.640 the other CIO do? 0:04:17.480 --> 0:04:19.880 So last week I was in Italy visiting my mother. 0:04:20.200 --> 0:04:23.320 She's eighty three, and she obviously doesn't know much about 0:04:23.360 --> 0:04:26.800 technology or banking, and so she said, what do you 0:04:26.880 --> 0:04:29.080 do with Coleman? And I said, you know, I just 0:04:29.120 --> 0:04:31.320 tried to simplify. I say, make sure that the printers 0:04:31.320 --> 0:04:36.880 don't run out of And interestingly, the CIO job has 0:04:36.920 --> 0:04:40.359 been traditionally associated with the word it. 0:04:41.080 --> 0:04:42.360 Okay and it. 0:04:42.640 --> 0:04:45.200 I tell you, talk to any technologist, they don't want 0:04:45.200 --> 0:04:46.400 to be classified as IT. 0:04:47.320 --> 0:04:49.880 Right, because those are you associated with those are the 0:04:49.920 --> 0:04:51.960 people who like, see if the ethernet cable with. 0:04:51.960 --> 0:04:54.159 Those are the ones who tell you that those that 0:04:54.600 --> 0:04:56.480 you know, I mean, I have a lot of respect 0:04:56.520 --> 0:04:59.000 for it, but generally you go to the IT department 0:04:59.000 --> 0:05:02.440 when something doesn't work, okay, And so it's very back 0:05:02.520 --> 0:05:06.400 office and something that attracted me to this job. I've 0:05:06.440 --> 0:05:08.080 been here for five years and this is the first 0:05:08.120 --> 0:05:10.240 time that I do like a CIO job. Before I 0:05:10.279 --> 0:05:12.880 was doing more like, you know, creating technology, et cetera, 0:05:12.920 --> 0:05:15.200 and service. I can talk about that, but is the 0:05:15.240 --> 0:05:17.080 fact that the role of a CEO has actually changed 0:05:17.160 --> 0:05:21.599 quite a bit, and now it's about really asking the question, 0:05:21.800 --> 0:05:26.760 you know, how do we implement technology in order to 0:05:26.960 --> 0:05:30.480 achieve our strategic objectives and actually to be differentiated, And 0:05:30.520 --> 0:05:33.520 it's really sitting at the strategic table of the firm. 0:05:33.560 --> 0:05:33.880 Okay. 0:05:34.760 --> 0:05:37.440 So today we live in a world where obviously a 0:05:37.480 --> 0:05:39.279 lot of the things that we want to do, or 0:05:39.360 --> 0:05:42.400 every company wants to do, are really kind of determined 0:05:42.400 --> 0:05:45.599 by how good you are at technology. And so I 0:05:45.640 --> 0:05:48.080 think the role of the CIO has changed quite a bit. 0:05:48.200 --> 0:05:50.680 And now, you know, I would define it as in general, 0:05:51.279 --> 0:05:54.159 defining the technology strategy of a firm and also making 0:05:54.160 --> 0:05:56.520 sure that you have the right culture in the engineering 0:05:56.560 --> 0:05:58.240 team in order to execute on that. 0:05:58.600 --> 0:06:00.880 What's the day to day look like? Like, what's the 0:06:00.920 --> 0:06:03.360 typical day you get into the office and then what. 0:06:03.320 --> 0:06:03.599 Do you do? 0:06:04.120 --> 0:06:06.880 Well? I mean I get into the office, and I generally, 0:06:07.160 --> 0:06:09.440 like everybody else, you know, I talk to people every 0:06:09.520 --> 0:06:11.599 day all day, and so I talk to people. You know, 0:06:11.640 --> 0:06:13.640 we have a bunch of meetings one after the other. End. 0:06:13.640 --> 0:06:16.600 I have teams coming to me with either regularly scheduled 0:06:16.600 --> 0:06:19.800 meetings or meetings that have been requested to discuss a 0:06:19.839 --> 0:06:23.440 certain topic. And you know, we just go through is 0:06:23.480 --> 0:06:27.159 there a whiteboard? Well right now in the age of Zoom, 0:06:27.360 --> 0:06:29.880 I guess still. You know, we have a globally distributed 0:06:29.880 --> 0:06:31.479 team and so a lot of our people are not 0:06:31.600 --> 0:06:33.880 in the same office, and so we use virtual whiteboards 0:06:33.920 --> 0:06:36.640 like everybody else. But I would say, you know, one 0:06:36.680 --> 0:06:39.280 of the things that I tried to do while joining Golma, 0:06:39.320 --> 0:06:41.960 which was part of sort of the cultural agen that 0:06:42.160 --> 0:06:48.760 was emphasizing the importance of narratives and written world versus 0:06:48.800 --> 0:06:51.920 you know, PowerPoint and talking. Okay, so, which is kind 0:06:51.920 --> 0:06:54.000 of what I learned that Amazon over the years. Okay, 0:06:54.080 --> 0:06:56.880 all right, w I was a REDWS and one of 0:06:56.880 --> 0:06:59.200 the things you learned there as soon as you join Amazon, 0:06:59.279 --> 0:07:03.680 in any part of Amazon, like the first few meetings 0:07:03.680 --> 0:07:07.160 are kind of shocking because nobody talks. Everybody starts reading. 0:07:07.560 --> 0:07:10.559 You start reading for like sometimes thirty minutes or forty 0:07:10.600 --> 0:07:14.480 five minutes, and if you're the author of the document, 0:07:15.000 --> 0:07:18.000 you're just sitting there basically, and you just try to 0:07:18.080 --> 0:07:20.760 look at people's faces and understand what they think about 0:07:20.800 --> 0:07:22.840 your document. And sometimes, you know, if you're with Jeff 0:07:22.880 --> 0:07:25.720 Bezos or others, you know, at that time it can 0:07:25.760 --> 0:07:29.480 be pretty pretty terrifying. And so this kind of shift 0:07:29.760 --> 0:07:34.240 from a culture of people talk, people comment on a PowerPoint, 0:07:34.280 --> 0:07:37.720 and the discussion sometimes get you know, driven by who 0:07:37.720 --> 0:07:40.520 has the stronger personality versus, you know, who has the 0:07:40.520 --> 0:07:43.600 greatest ideas. One of the things that I try to 0:07:43.720 --> 0:07:45.280 change is that a lot of the meetings that we 0:07:45.360 --> 0:07:49.120 do today actually start the same way by reading a document. 0:07:49.880 --> 0:07:51.920 So I now read a lot of documents like I 0:07:52.000 --> 0:07:54.240 used to in Amazon. You know, I would say maybe 0:07:54.400 --> 0:07:56.800 thirty forty percent of the meeting are starting that way, 0:07:57.520 --> 0:08:00.280 and I think people love it because it breaks the 0:08:00.320 --> 0:08:02.560 barrier of language for someone like me, that English is 0:08:02.600 --> 0:08:05.720 obviously not my first language, breaks the Sometimes some of 0:08:05.760 --> 0:08:08.040 the people are more shy than others, et cetera. So 0:08:08.040 --> 0:08:11.120 people see that as a mechanism for inclusion. So back 0:08:11.160 --> 0:08:14.400 to your question, let's say thirty forty percent of my 0:08:14.520 --> 0:08:17.840 meetings actually now start by us reading a document together 0:08:17.920 --> 0:08:19.920 and then commenting on that and making decisions. 0:08:20.000 --> 0:08:22.800 Can I just say, Tracy, I've always thought more meetings 0:08:23.000 --> 0:08:24.920 you should start with just reading. Because you go to 0:08:25.000 --> 0:08:27.920 you hear like a quarterly call or a FED event, 0:08:28.280 --> 0:08:30.680 and someone just reads out of prepared text. It's like, 0:08:30.880 --> 0:08:33.080 just let everyone read it and just jump straight into like, 0:08:33.160 --> 0:08:34.400 let everyone do the reading first. 0:08:34.440 --> 0:08:35.320 You don't need someone. 0:08:35.160 --> 0:08:38.440 Standing up there talking about what's on a written piece 0:08:38.480 --> 0:08:39.240 of paper somewhere. 0:08:39.240 --> 0:08:43.880 Anyway, I agree that we could reduce the time of meetings. Yes, okay, 0:08:43.880 --> 0:08:47.400 So speaking of meetings and the decision making process, then 0:08:47.760 --> 0:08:52.160 talk to us about how Goldman Sachs decided to approach 0:08:52.520 --> 0:08:56.320 generative AI. What was the decision making process? Like there 0:08:56.440 --> 0:08:59.680 the development process, and you know, we'll get to what 0:08:59.720 --> 0:09:02.720 you're developing, but like, how did you initially approach it? 0:09:03.160 --> 0:09:07.679 So I think our initial approach was really to realize 0:09:07.880 --> 0:09:10.480 that there were so many more things that we didn't 0:09:10.520 --> 0:09:13.080 know compared to the things that we knew, because it's 0:09:13.120 --> 0:09:15.760 a really new thing, and even for companies like us 0:09:15.760 --> 0:09:19.240 that have been working on machine learning and traditionally I 0:09:19.400 --> 0:09:23.920 for literally decades, this felt like a very different thing. 0:09:24.400 --> 0:09:26.920 What sort of timeframe are we talking about? Like, was 0:09:26.960 --> 0:09:29.760 there a sort of like big realization that this is 0:09:29.800 --> 0:09:31.199 something that we need to focus on. 0:09:31.679 --> 0:09:35.000 Yes, because I was lucky enough that I got into 0:09:35.280 --> 0:09:40.480 the very very early version of GPT, even before it 0:09:40.520 --> 0:09:44.079 was called chat GIBT. So the very first version was 0:09:44.200 --> 0:09:49.440 essentially completing a sentence. It wasn't even allowing you to 0:09:49.440 --> 0:09:52.440 do interactive chat. You would just paste a text and 0:09:52.520 --> 0:09:55.240 that will just complete that text. And so I started 0:09:55.240 --> 0:09:56.920 to do that with a bunch of stuff, and then 0:09:56.960 --> 0:09:59.439 I was saying that the quality which this will continue 0:10:00.400 --> 0:10:03.200 was pretty much indistinguishable with the part that you actually 0:10:03.200 --> 0:10:05.800 put in that. And so we started to obviously talk 0:10:05.880 --> 0:10:08.559 between ourselves but also among other people in the industry, 0:10:08.600 --> 0:10:12.560 and we all realized very soon that this would be 0:10:12.880 --> 0:10:15.880 something very different, but be also something that could have 0:10:15.920 --> 0:10:18.120 a pretty profound impact in what we do. Because at 0:10:18.160 --> 0:10:21.599 the end of the day, we are a purely digital business. 0:10:21.760 --> 0:10:24.200 We don't bend metal, we don't you know, like use 0:10:24.280 --> 0:10:26.960 high temperatures. We don't really have physics. So it's all 0:10:27.000 --> 0:10:29.679 about how we service our clients. It's all about how 0:10:29.720 --> 0:10:33.000 smart we are. It's all about how we can process 0:10:33.160 --> 0:10:36.839 incredible amount of information. It's all about, you know, how 0:10:36.880 --> 0:10:40.200 we analyze data in a very sometimes opinionated way. We 0:10:40.280 --> 0:10:43.160 form our own views on the market, we form our 0:10:43.280 --> 0:10:47.000 views of investments, et cetera. And so given that this 0:10:47.240 --> 0:10:52.880 AI showed very early sign of being able to synthesize 0:10:53.000 --> 0:10:57.719 and summarize very complex set of information but also identify patterns, 0:10:58.320 --> 0:11:00.679 we thought that could be something that we definitely need 0:11:00.720 --> 0:11:03.920 to pay attention to. So given that, one of the 0:11:03.960 --> 0:11:06.640 things that we decided to do very early on was 0:11:06.840 --> 0:11:09.920 to put a structure and I can say that more 0:11:09.960 --> 0:11:13.120 about that, put a structure around this so that we 0:11:13.160 --> 0:11:17.640 could experiment but in a sort of safe and controlled way. 0:11:18.080 --> 0:11:21.760 Right, So you decided to develop your own Goldman Sachs 0:11:21.880 --> 0:11:26.199 AI model versus you know, use a chat, GPT or 0:11:26.320 --> 0:11:27.640 clod or getting something off the show. 0:11:27.679 --> 0:11:30.679 Actually, initially we kind of thought about that, but then 0:11:30.800 --> 0:11:34.120 very quickly. We decided that our time was spent much 0:11:34.160 --> 0:11:37.960 better with using existing models, which by the way, we're 0:11:37.960 --> 0:11:41.920 iterating really really quickly, but then put them in a 0:11:41.960 --> 0:11:44.960 condition so that they would be safe to use and 0:11:45.000 --> 0:11:48.240 also they would actually give us the most reliable information, 0:11:48.360 --> 0:11:51.960 because taken as they are, you can't just drop a 0:11:52.040 --> 0:11:55.560 model in an environment like Goldman and then, like you know, 0:11:55.640 --> 0:11:57.960 to your earlier point of a one percent in accuracy, 0:11:58.200 --> 0:12:02.800 zero point one percent in accuracy completely an acceptable class. 0:12:03.520 --> 0:12:06.520 There are a lot of potential issues related to you know, 0:12:06.559 --> 0:12:09.240 what data has it been used to train? And you know, 0:12:09.280 --> 0:12:12.240 there is a lot of uncertainty with regards to you know, 0:12:12.320 --> 0:12:15.079 like what are the boundaries between what you can safely 0:12:15.160 --> 0:12:17.560 use and what you can And so what we decided 0:12:17.600 --> 0:12:23.080 to do was instead to build a platform around the model. 0:12:23.160 --> 0:12:25.120 So think of that almost as if you had a 0:12:25.200 --> 0:12:28.840 nuclear reactor. You know that now you have invented fission 0:12:28.920 --> 0:12:30.880 or fusion, and there is a lot of power that 0:12:30.920 --> 0:12:33.280 can be generated from that, but then you need to 0:12:33.320 --> 0:12:36.080 contain it and direct it in a certain way. And 0:12:36.080 --> 0:12:40.280 so we build this GSAI platform, which essentially takes a 0:12:40.360 --> 0:12:43.200 variety of models that we select, puts them in the 0:12:43.240 --> 0:12:47.840 condition of being completely segregated and completely secluded and completely 0:12:47.880 --> 0:12:51.800 safe from an information a security standpoint. Abstract some of 0:12:51.840 --> 0:12:54.559 the ways to use the model, so that our developers 0:12:54.559 --> 0:12:58.360 can use the models interchangeably, and then creates a set 0:12:58.400 --> 0:13:03.480 of standardized way, for example, improve the accuracy using retrieval, 0:13:03.520 --> 0:13:09.199 a granted generation, access external or internal data sources, applying 0:13:09.960 --> 0:13:13.160 entitlement so that someone is on the private side, you know, 0:13:13.160 --> 0:13:15.160 I've got to see different information that someone is on 0:13:15.200 --> 0:13:18.240 the public side. And then on top of that, build 0:13:18.320 --> 0:13:21.920 a developer environment so that people will very easily be 0:13:22.000 --> 0:13:25.760 able to embed that AI in their own applications. So 0:13:25.880 --> 0:13:28.880 imagine this, we got a great engine and we decided 0:13:28.920 --> 0:13:30.320 to build a great car around that. 0:13:45.960 --> 0:13:47.440 What are you putting in the model? 0:13:47.440 --> 0:13:49.840 Because I have to imagine at a bank like Goldman, 0:13:50.080 --> 0:13:51.600 you know, you have a lot of data, but you 0:13:51.679 --> 0:13:54.720 must have just an extraordinary amount of unstructured data. There's 0:13:54.800 --> 0:13:59.880 conversations that bankers have with clients. There's other sort of meeting, 0:14:00.000 --> 0:14:02.320 the meetings you have, and there's words that are said 0:14:02.400 --> 0:14:05.840 during that meeting that could be synthesized in some way. 0:14:06.280 --> 0:14:11.200 In these early iterations, you know, I upload a conference 0:14:11.200 --> 0:14:12.960 called transcript and I ask a question, what do you 0:14:13.040 --> 0:14:16.040 upload it? What is the unstructured data that you have 0:14:16.800 --> 0:14:19.240 or the questions or these yeah, what are you what 0:14:19.280 --> 0:14:22.240 are you putting into it from your reams of knowledge 0:14:22.240 --> 0:14:23.280 that you must have internally. 0:14:23.720 --> 0:14:26.200 So one of the first things that we did was 0:14:26.680 --> 0:14:30.320 use the platform and the models to extract information from 0:14:30.520 --> 0:14:34.280 publicly available documents. That's kind of the safest way public 0:14:34.320 --> 0:14:36.720 filing all the case or the queues and you know, 0:14:36.760 --> 0:14:40.600 and obviously earnings, and put our bankers in a condition 0:14:40.720 --> 0:14:45.480 to be able to ask very very sophisticated multi dimensional 0:14:45.600 --> 0:14:50.960 questions around what was reported, cross refit with previous reports, 0:14:51.320 --> 0:14:55.560 cross refit with any announcement, any earnings, called transcripts, all 0:14:55.640 --> 0:14:57.880 things that are out there but just are difficult to 0:14:57.880 --> 0:15:00.920 bring together. And so that as a involved into a 0:15:01.000 --> 0:15:04.600 tool that physically we use and we're rolling it out 0:15:04.680 --> 0:15:08.520 right now as an assistant to our bankers so that 0:15:08.680 --> 0:15:11.360 they can you know, service their client or answer client 0:15:11.480 --> 0:15:14.720 questions or even their wrong questions. In a time there 0:15:14.760 --> 0:15:17.280 is a fraction of what you used to take even 0:15:17.400 --> 0:15:21.720 generate documents that then can be you know, shared the 0:15:21.760 --> 0:15:23.720 clients and so on and so forth. And obviously we 0:15:23.800 --> 0:15:27.360 always have as a rule, like when you drive a 0:15:27.400 --> 0:15:29.920 car that has some autonomous capability, that you always keep 0:15:30.000 --> 0:15:31.840 the hands on the wheel. Our rule is that there 0:15:31.840 --> 0:15:33.720 always needs to be a human in the loop. Okay, 0:15:34.200 --> 0:15:37.240 And so the way that works is actually interesting because 0:15:37.320 --> 0:15:40.720 we found out that you can't just shove something into 0:15:40.720 --> 0:15:42.960 a model and then pretend that the model is going 0:15:43.000 --> 0:15:47.560 to give you the answer right away. Why well, because models, 0:15:47.600 --> 0:15:51.800 by themselves, you know, they essentially apply a stochastic or 0:15:51.840 --> 0:15:54.560 a statistical way to understand what is the next world 0:15:54.560 --> 0:15:57.240 that they need to say. So, no matter how good 0:15:57.560 --> 0:16:00.440 is the material that you put in, there's always going 0:16:00.480 --> 0:16:03.160 to be some level of variability. There is almost like 0:16:03.240 --> 0:16:06.000 the intersection between the documents that you insert and what 0:16:06.200 --> 0:16:09.160 is I call it like the shadow of all the 0:16:09.200 --> 0:16:11.720 knowledge of all the things that the model has seen before. 0:16:12.520 --> 0:16:15.280 And so we really perfected this. You know, there are 0:16:15.280 --> 0:16:19.680 two techniques that are widely used to improve the accuracy 0:16:19.680 --> 0:16:23.920 of the answers. One is working on the way those 0:16:24.000 --> 0:16:28.960 models represent knowledge, which is called embeddings technically, and the 0:16:29.000 --> 0:16:31.760 concept of embeddings by the way, everybody talks about embeddings, 0:16:31.760 --> 0:16:34.720 but then for very few people actually it took me 0:16:34.760 --> 0:16:38.320 a while to understand that well. And embedding is simply 0:16:38.520 --> 0:16:41.920 a way for the model to parameterize and create a 0:16:42.040 --> 0:16:45.040 description of what they're seeing. So if I see a phone, 0:16:45.080 --> 0:16:47.280 for example, in front of me, the embeddings of a 0:16:47.320 --> 0:16:50.920 phone could be it's a piece of electronic Yes, one, 0:16:51.000 --> 0:16:54.840 it's definitely a piece of electronics. It's edible. Zero. You 0:16:54.880 --> 0:16:56.840 can't really eat it, you know, And then you have 0:16:56.880 --> 0:17:00.760 all these parameters. Is almost like twenty questions. I give 0:17:00.800 --> 0:17:02.720 you all these questions and then you finally understand that 0:17:02.800 --> 0:17:04.960 it's a phone, and that's what the embeddings is almost 0:17:04.960 --> 0:17:08.000 like the twenty questions of the reality instead of twenty 0:17:08.080 --> 0:17:11.479 is like twenty twenty thousands. And then you have DRAG, 0:17:11.560 --> 0:17:14.560 which is the retrieval augmented generation, which is actually interesting 0:17:14.600 --> 0:17:18.720 because you tell the model that instead of using its 0:17:18.760 --> 0:17:20.919 on internal knowledge in order to give you an answer, 0:17:20.960 --> 0:17:23.640 which sometimes, as I said, is like a representation of reality, 0:17:23.680 --> 0:17:26.840 but it's often not accurate, you point them to the 0:17:26.920 --> 0:17:30.520 right sections of the document that actually is more likely 0:17:30.560 --> 0:17:33.280 to answer your question. Okay, and that's the key. It 0:17:33.320 --> 0:17:35.480 needs to point to the right sections and then you 0:17:35.520 --> 0:17:38.560 get the citations back. So that took a lot of effort. 0:17:39.040 --> 0:17:41.880 But we're using that in many many cases because then 0:17:41.920 --> 0:17:45.399 we expanded the use case from purely like banker assistant 0:17:45.440 --> 0:17:48.840 in a way to more like okay, document management. You know, 0:17:48.880 --> 0:17:53.280 we process millions of documents. Think of that credit confirmation 0:17:53.600 --> 0:17:59.439 implements confirmation. Every document has a task called entity strauction. 0:17:59.560 --> 0:18:02.639 So you need to extract stuff from the document and 0:18:02.680 --> 0:18:05.320 then digitize it and then model it in a certain way. 0:18:05.880 --> 0:18:09.600 And so the use of general TVii there does a 0:18:09.680 --> 0:18:14.600 great job at extracting information. And this is an interesting 0:18:14.640 --> 0:18:19.760 concept because you don't have to actually tell a fixed pattern. 0:18:20.000 --> 0:18:22.520 You can just say, give a lot of examples, and 0:18:22.560 --> 0:18:24.840 then the AI will figure out from that pattern. One 0:18:24.840 --> 0:18:27.520 of my favorite example is the following. Let's say that 0:18:27.600 --> 0:18:31.480 my phone number is five five three two one three 0:18:31.640 --> 0:18:35.439 h five oh, and someone writes in the document instead 0:18:35.440 --> 0:18:39.600 of with zero rights an oh. Okay. You can test 0:18:39.640 --> 0:18:42.840 yourself even with GPT, if you give a number with 0:18:42.920 --> 0:18:45.480 an O instead of zero, and you ask GPT, what's 0:18:45.800 --> 0:18:50.320 likely wrong with this entity? GPT is gonna tell you, well, 0:18:50.960 --> 0:18:53.520 it looks like a phone number that is an all, 0:18:53.600 --> 0:18:56.280 which general is not in phone numbers. Most likely this 0:18:56.320 --> 0:19:00.600 is the correct phone number. Now, nobody has written software 0:19:01.040 --> 0:19:04.080 to do a pattern match in there. And imagine if 0:19:04.119 --> 0:19:06.800 in the tradition, in traditional way of doing antity instruction, 0:19:06.920 --> 0:19:10.280 there were developers that were writing rules. They were saying, okay, numbers, 0:19:10.480 --> 0:19:13.560 it needs to be ten digits and blah blah blah. 0:19:13.760 --> 0:19:16.000 The AI figures. 0:19:15.560 --> 0:19:18.520 Out their own rules. 0:19:17.960 --> 0:19:20.600 That are the most likely. So this is the key thing. 0:19:20.760 --> 0:19:24.240 It has common sense. And that common sense when you're 0:19:24.359 --> 0:19:28.840 dealing with millions of documents that contain all bunch of 0:19:28.960 --> 0:19:32.320 ways that you must might have written those things, and 0:19:32.400 --> 0:19:34.439 imagine the complexity of all the rules that you need 0:19:34.480 --> 0:19:37.560 to write. And every bank has the same problem. This 0:19:37.720 --> 0:19:42.680 simplifies things tremendously because it's able to figure out what's 0:19:42.800 --> 0:19:48.240 most likely by itself. And so that thing evolved into 0:19:48.560 --> 0:19:52.080 a tremendous time saving for everybody in the bank that 0:19:52.119 --> 0:19:54.639 has to do with the workflow documents. And so that 0:19:55.119 --> 0:19:57.359 was a very interesting finding that we did early on. 0:19:57.440 --> 0:20:02.119 And so again to summarize more, those are raw material 0:20:02.240 --> 0:20:05.800 of intelligence. You know you need to somehow direct them, 0:20:05.840 --> 0:20:07.960 you need to guide them, you need to instruct them, 0:20:07.960 --> 0:20:10.040 you need to put them in an environment that actually 0:20:10.080 --> 0:20:12.000 gets the most out of that, and that's what we've 0:20:12.000 --> 0:20:12.800 been focusing on. 0:20:13.119 --> 0:20:16.240 So going back to the analogy that you used previously, 0:20:16.280 --> 0:20:18.800 this idea of a nuclear reactor and sort of building 0:20:18.840 --> 0:20:22.400 the containment casing or the protective casing around it. I 0:20:22.400 --> 0:20:26.359 imagine one of the complications of being Goldman Sachs and 0:20:26.400 --> 0:20:29.960 working with AI is that you're a regulated financial entity. 0:20:30.560 --> 0:20:35.480 How does that added complexity affect your use of AI. 0:20:35.640 --> 0:20:39.680 Are there additional data considerations or additional infosec considerations. 0:20:40.240 --> 0:20:44.520 I think that's a great question, because obviously we live 0:20:44.560 --> 0:20:47.040 in a regulated world, and in fact, I have to 0:20:47.040 --> 0:20:49.880 tell you that in this case, regulation actually helps us 0:20:50.000 --> 0:20:53.520 think through all the possible unknown now, something that, as 0:20:53.520 --> 0:20:56.080 I said, is something that is still largely something that 0:20:56.119 --> 0:20:59.600 nobody really completely understands. And so what we did was 0:20:59.680 --> 0:21:03.240 to put but governance around the usage of the models 0:21:03.280 --> 0:21:05.760 and also governance with regards to the use cases that 0:21:05.800 --> 0:21:09.119 we can implement on the models. Every bank has a 0:21:09.160 --> 0:21:12.639 function called model risk, which, in the traditional sense, a 0:21:12.720 --> 0:21:18.040 model is any decision or any algorithm that is running 0:21:18.080 --> 0:21:21.040 automatically to do for example, pricing or you know, there 0:21:21.119 --> 0:21:24.400 is a lot of that tradition in every bank risk calculation, etc. 0:21:24.760 --> 0:21:27.920 So that's the traditional model risk. We use that very 0:21:27.960 --> 0:21:31.000 well established pattern. That is also you know, that has 0:21:31.040 --> 0:21:35.280 its own second and third line like controls and supervision 0:21:35.840 --> 0:21:38.840 also to validate what we do on the AI side. 0:21:38.880 --> 0:21:41.600 So there is a governance part which we really set 0:21:41.680 --> 0:21:44.359 up very early on. We have an AI committee that 0:21:44.400 --> 0:21:47.479 looks at the business case should we do this? And 0:21:47.520 --> 0:21:50.840 then we have an AI control and risk committee that 0:21:50.880 --> 0:21:52.720 looks at, okay, how are we going to do that? 0:21:52.840 --> 0:21:54.840 And then the two of them need to actually come 0:21:54.880 --> 0:21:57.719 together before we can release a use case. And then 0:21:57.760 --> 0:21:59.879 of course we did a lot of work with regards 0:21:59.880 --> 0:22:05.080 to the let's say accuracy lineage and in a way, 0:22:05.200 --> 0:22:08.040 the way you connect the output to where does the 0:22:08.119 --> 0:22:10.840 data come from and who can actually see that what 0:22:10.920 --> 0:22:14.159 we call entitlements, and we did that in lockstep with 0:22:14.160 --> 0:22:17.320 the regulators, so that I think, you know, you know, 0:22:17.359 --> 0:22:18.920 in a world, I think we put a sort of 0:22:19.000 --> 0:22:22.399 what we like to call responsible AI first since the 0:22:22.480 --> 0:22:24.680 very beginning, and it really helped us. The fact that 0:22:24.760 --> 0:22:28.080 you know, we embedded all those controls into a single platform. 0:22:28.160 --> 0:22:31.320 This is how our people use AI inside the inside goal. 0:22:31.760 --> 0:22:33.919 This is something I'm really interested in just from a 0:22:33.960 --> 0:22:36.800 technical perspective, But can you talk a little bit more 0:22:37.000 --> 0:22:41.000 about that interoperability aspect. So you have a pool of 0:22:41.119 --> 0:22:43.960 data that is gold pins that you presumably don't really 0:22:44.000 --> 0:22:46.720 want to share with outside entities, So how do you 0:22:46.760 --> 0:22:50.520 plug that into an AI model if you're working with 0:22:50.680 --> 0:22:52.920 you know, Chat, GPT or clod or something like that. 0:22:53.160 --> 0:22:55.399 So there are two ways that we do that. We 0:22:55.560 --> 0:22:59.639 use the sort of a large proprietary models in a 0:22:59.680 --> 0:23:02.719 way that we worked with Microsoft, we work on Google. 0:23:02.760 --> 0:23:07.680 We have very strong partnerships, so that essentially there are 0:23:07.760 --> 0:23:11.040 controls that guarantee that nobody has access to the data 0:23:11.080 --> 0:23:13.680 that we put into the model, that the data leaves 0:23:13.760 --> 0:23:17.679 no side effects, so it's not saved anywhere, it's the 0:23:17.680 --> 0:23:21.560 only stays in memory. The model is completely stateless, meaning 0:23:21.640 --> 0:23:24.199 that the state of the model doesn't change after the 0:23:24.280 --> 0:23:26.159 data comes through, so there is no training, there is 0:23:26.200 --> 0:23:29.520 nothing down on that data. And also that operator access 0:23:29.960 --> 0:23:32.720 meaning who can actually access the memory or those machines 0:23:32.960 --> 0:23:35.879 is restricted and controlled and needs to be agreed with us. 0:23:36.040 --> 0:23:39.720 So imagine secure in putting a vault around those models. 0:23:39.760 --> 0:23:44.560 But even then, what's really really sort of secret, source, proprietory, etc. 0:23:44.920 --> 0:23:48.160 We like to use also different approach to use open 0:23:48.200 --> 0:23:52.439 source models that we can run on our own environment. Okay, 0:23:53.320 --> 0:23:55.439 and we like a lot of open source models. I 0:23:55.480 --> 0:23:58.040 have to say that. One we particularly like Islama and 0:23:58.080 --> 0:24:01.359 actually Lama tree and Lama tree point one especially as. 0:24:01.280 --> 0:24:02.560 No one developed by Facebook. 0:24:03.040 --> 0:24:06.800 Oh yeah, so they recently announced Lama three point one, 0:24:07.320 --> 0:24:10.080 which has a version that is four hundred and five 0:24:10.119 --> 0:24:14.320 million billion parameters. So it's pretty large and it seems 0:24:14.359 --> 0:24:17.320 to be performing. You know, the gap with those big 0:24:17.359 --> 0:24:20.600 fundational models is now very very narrow. So for that, 0:24:20.760 --> 0:24:23.080 we run it in our own sort of a private cloud, 0:24:23.200 --> 0:24:26.920 call it that way, with GPUs that we own, and 0:24:26.960 --> 0:24:29.640 that we train it with data that stays in that environment. 0:24:29.680 --> 0:24:32.000 So imagine that. You know, our approach is okay, there 0:24:32.040 --> 0:24:34.960 is a sort of arrating of sensitivity of this data. 0:24:35.680 --> 0:24:38.960 Every data needs to be protected. Therefore we use those 0:24:39.000 --> 0:24:42.960 safeties all throughout regardless. But then for the super super 0:24:42.960 --> 0:24:45.560 super secret stuff, you know, we like to do it 0:24:45.600 --> 0:24:47.000 in our own embod. 0:24:46.760 --> 0:24:49.239 Since you're talking about building your own environment, and this 0:24:49.320 --> 0:24:51.600 is something we've talked a lot about on the podcast. 0:24:52.000 --> 0:24:57.239 Hardware constraints, energy constraints, things like that, how does that 0:24:57.359 --> 0:25:01.040 manifest in your world some of these physical, real world 0:25:01.119 --> 0:25:06.119 constraints to building out the compute platform at Goldman sax Well. 0:25:06.200 --> 0:25:09.199 Initially we thought maybe we can host those GPUs in 0:25:09.240 --> 0:25:13.000 our own data centers, and then immediately you run into 0:25:13.000 --> 0:25:15.840 considerations such as a first of all, they develop a 0:25:15.880 --> 0:25:18.679 lot of heat. Secondly, they consume a lot of power. 0:25:19.160 --> 0:25:21.720 Tree there is a decent chance that they might fail 0:25:21.880 --> 0:25:24.439 because you know, of all those considerations if you're not 0:25:24.920 --> 0:25:29.680 properly addressed. And then d they need very special for example, 0:25:29.680 --> 0:25:32.280 interconnect and high speed bandwidth between them. And so the 0:25:32.359 --> 0:25:35.199 decision what we ended up doing is actually to have 0:25:35.280 --> 0:25:38.640 them hosted into some of the hyperscalers that we use, 0:25:39.200 --> 0:25:42.560 but use them in their own virtual private clouds. So 0:25:42.640 --> 0:25:46.639 those racks are basically only ours. And if you're asking 0:25:46.640 --> 0:25:49.280 me the more general question, which is, hey, where is 0:25:49.320 --> 0:25:52.600 the world going with regards of that? Okay, so right 0:25:52.640 --> 0:25:57.320 now there are two really rapidly competing forces. One is 0:25:57.400 --> 0:26:01.000 pushing towards more and more consumption and one is pushing 0:26:01.000 --> 0:26:03.560 for more and more optimization. Okay, and I can talk 0:26:03.600 --> 0:26:06.959 about that for a couple of minutes. For the more consumption, 0:26:07.440 --> 0:26:10.119 I mean, really the two dimensions for scaling a model 0:26:10.720 --> 0:26:12.959 is one of the most important. One is obviously the 0:26:13.000 --> 0:26:16.000 size of the prompt or the context. Okay, and there 0:26:16.040 --> 0:26:19.160 is pretty good evidence that the larger the context, which 0:26:19.200 --> 0:26:21.479 is really like the memory of those models, and the 0:26:21.480 --> 0:26:23.680 more you can get out in terms of the ability 0:26:23.680 --> 0:26:27.480 to reason on your data. That has already gone up 0:26:27.520 --> 0:26:30.679 from thousands to tens of thousands to now millions. And 0:26:30.720 --> 0:26:33.240 there is a prediction, you know, you heard some very 0:26:33.280 --> 0:26:36.439 prominent people saying that there could be the trillion prompt 0:26:36.480 --> 0:26:40.160 and the power scales quadratically with the prompt, so that 0:26:40.320 --> 0:26:43.280 points to a consumption of energy and GPU power which 0:26:43.359 --> 0:26:46.040 is going to continue to raise exponentially. At the same time, 0:26:46.560 --> 0:26:51.400 we've seen great results with optimization techniques such as quantitization, 0:26:51.960 --> 0:26:55.280 reducing from sixteen bits to eight bit to four bit precision, 0:26:56.000 --> 0:27:00.520 having even smaller models using what's called window that which 0:27:00.560 --> 0:27:03.120 means that you know that you can only pay more 0:27:03.160 --> 0:27:05.760 attention to some of the parts of the context intell 0:27:05.840 --> 0:27:08.760 of all of it, and so you need a smaller one. 0:27:08.920 --> 0:27:11.640 And so I'm seeing those two kind of going into 0:27:11.680 --> 0:27:13.919 two opposite directions. It's going to be very interesting to 0:27:13.920 --> 0:27:17.040 see how that evolves. I would say for the short term. 0:27:17.520 --> 0:27:20.680 I see that definitely that trend is going to continue 0:27:20.720 --> 0:27:23.600 to go up. And one of the things that fascinates 0:27:23.640 --> 0:27:26.800 me the most is that from one version to another, 0:27:27.520 --> 0:27:31.800 the most striking difference is the ability to reason and 0:27:31.840 --> 0:27:36.000 the ability to actually come up with logical step by 0:27:36.080 --> 0:27:40.840 step instructions or step by step chains of thought of 0:27:40.880 --> 0:27:44.280 what the output is going to be. So we decided, okay, 0:27:44.320 --> 0:27:45.639 first of all, we need to get access to the 0:27:45.680 --> 0:27:49.000 most powerful GPUs, secondary we need to host them into 0:27:49.000 --> 0:27:52.920 an environment that actually allows for the most optimal functioning 0:27:52.960 --> 0:27:55.160 in terms of bandit, in terms of power consumption, etc. 0:27:56.359 --> 0:27:58.399 And then at the same time, we've been focusing a 0:27:58.480 --> 0:28:00.840 lot on optimizing the algorithm so that you know, we 0:28:00.880 --> 0:28:03.120 can really got we could really get the most out 0:28:03.160 --> 0:28:03.359 of that. 0:28:04.160 --> 0:28:06.119 Just to press you on this point, what are the 0:28:06.160 --> 0:28:10.200 conversations actually like with cloud providers at the moment when 0:28:10.200 --> 0:28:14.200 you're trying to get more compute or more space, more racks, whatever. 0:28:14.680 --> 0:28:17.000 Is it maybe different for you because you were at AWS. 0:28:17.040 --> 0:28:18.879 Maybe you can just call someone up there and be like, 0:28:18.960 --> 0:28:22.959 we would like some more servers, or have you found 0:28:23.000 --> 0:28:26.359 yourselves at times maybe limited in what you can do 0:28:26.600 --> 0:28:28.320 by the amount of power available to you. 0:28:29.640 --> 0:28:31.200 Well, I wish that would be the case, but I 0:28:31.480 --> 0:28:34.880 cannot just pick up the phone and get whatever I want. 0:28:35.000 --> 0:28:37.919 But I think so far. I mean obviously because we 0:28:38.000 --> 0:28:40.920 are a really good client of those companies in general, 0:28:41.160 --> 0:28:43.720 but also because we've been very selective in the use 0:28:43.800 --> 0:28:46.200 cases that we put in production. I have to say, 0:28:46.240 --> 0:28:48.880 like I said before, think about that, if you look 0:28:48.880 --> 0:28:52.640 at the consumption of resources today, those who consume more 0:28:52.680 --> 0:28:55.360 resources are people that actually do the training of their 0:28:55.360 --> 0:29:00.880 own models. Okay, and it initially everybody was trying to 0:29:00.880 --> 0:29:03.400 do full training from scratch, which was taken like the 0:29:03.480 --> 0:29:06.720 absolutely if that's one hundred, we do fine tuning, which 0:29:06.800 --> 0:29:10.040 is adaptation of existing models that could be one to 0:29:10.120 --> 0:29:12.920 one hundred or less in terms of consumption or resources. 0:29:12.920 --> 0:29:15.280 So because of the techniques they were using, and because 0:29:15.280 --> 0:29:18.120 of the fact that we decided to really focus on 0:29:18.200 --> 0:29:21.840 fine tuning or RAG versus full training, we haven't really 0:29:22.120 --> 0:29:24.640 hit any caps. And also have to be honest, you know, 0:29:24.720 --> 0:29:28.479 we bought our GPUs pretty well early, so probably there 0:29:28.520 --> 0:29:31.360 wasn't as much craziness as there is today, and so 0:29:31.400 --> 0:29:47.160 that's turned out probably to be a good idea. 0:29:48.560 --> 0:29:49.760 You know, in videos huge. 0:29:49.880 --> 0:29:52.360 Everyone would like to have some of in Video's market 0:29:52.440 --> 0:29:55.800 cap be their market cap. I have offering some cheaper product. 0:29:56.200 --> 0:30:00.400 We interviewed some guys who have a semiconductor started that's 0:30:00.440 --> 0:30:04.000 just going to be LLLM focused startups. We know that Google, 0:30:04.080 --> 0:30:08.000 for example, has TPUs their own chips. Can you envision 0:30:08.040 --> 0:30:12.040 as a roadmap some alternative where GPUs are not the 0:30:12.200 --> 0:30:14.320 dominant hardware for AI? 0:30:14.640 --> 0:30:17.200 Well, that's literally like you know the trillion dollar question. 0:30:17.240 --> 0:30:18.520 Yeah, well that's I'm asking you. 0:30:18.600 --> 0:30:21.040 Yeah, but I'm not an analyst and I'm just a technogy. 0:30:21.040 --> 0:30:23.480 Remember I'm the guy that makes sure that I. 0:30:23.440 --> 0:30:26.000 Would say, you're probably a better person to ask than 0:30:26.040 --> 0:30:28.200 an analyst because you're actually the one who's going to 0:30:28.200 --> 0:30:29.440 be making So I'm. 0:30:29.560 --> 0:30:31.640 Okay, so they're going to ask it to you. So 0:30:32.240 --> 0:30:35.400 you have to distinguish between There are actually two dimensions 0:30:35.440 --> 0:30:37.440 that we need to consider. One is training and the 0:30:37.440 --> 0:30:41.720 other one is inferenced. Okay, that's the first dichotomy. For training. 0:30:42.200 --> 0:30:45.959 At the moment, there's most likely nothing better than GPU's okay, 0:30:46.080 --> 0:30:50.400 because when you train a model, the software or Pythons 0:30:50.480 --> 0:30:53.720 or whatever framework needs to see all your GPUs as one. 0:30:54.320 --> 0:30:58.040 As a cluster, and it's not just the GPU itself, 0:30:58.080 --> 0:31:00.360 but it's the what Nvidia has been doing a great 0:31:00.440 --> 0:31:03.680 job at is actually to make them work in unison 0:31:04.160 --> 0:31:07.400 with the virtualization software called Kuda, which runs on and 0:31:07.560 --> 0:31:11.320 video GPUs, which is a extraordinary piece of software and 0:31:11.440 --> 0:31:15.480 it became the standard for that. And also because you know, 0:31:15.600 --> 0:31:19.000 the performance premium that you have on those GPUs when 0:31:19.000 --> 0:31:22.840 you're trying to train those incredibly large models is something 0:31:22.880 --> 0:31:25.360 that you really really want. And so the training part, 0:31:25.520 --> 0:31:27.360 I'm pretty sure that it's going to be dominated by 0:31:27.440 --> 0:31:30.200 GPUs for a while. But then you know, as those 0:31:30.200 --> 0:31:34.760 models get used, obviously the pendulum swings towards inference, which 0:31:34.800 --> 0:31:36.920 is the actual Now you have a model which is 0:31:36.920 --> 0:31:38.880 a bunch of weights and you just need to calculate 0:31:38.880 --> 0:31:43.240 a bunch of matrix multiplications on that. I think accelerators 0:31:43.280 --> 0:31:47.000 and specialized chips are actually going to have a really 0:31:47.040 --> 0:31:49.680 big role to play. So you may imagine that you 0:31:49.760 --> 0:31:52.880 go from a world where everybody builds the cars and 0:31:52.920 --> 0:31:55.240 not too many people drive the cars to a world 0:31:55.240 --> 0:31:57.560 where most people are going to drive cars. And then 0:31:57.600 --> 0:32:00.880 there is another two dimensions, which is models that are 0:32:00.920 --> 0:32:04.720 hosted by the client and models that are hosted by 0:32:04.880 --> 0:32:08.680 a hyperscale. So, as you know today, I can take 0:32:08.720 --> 0:32:10.720 a model like Lamma, I can put it in my 0:32:10.800 --> 0:32:13.920 own environ, I can run it on a MacBook, or 0:32:13.920 --> 0:32:16.040 I can run it in my own data center and 0:32:16.080 --> 0:32:19.800 with my own GPUs. And given that I'm used to GPUs, 0:32:20.240 --> 0:32:22.160 given that those are the ones that we can buy, 0:32:22.240 --> 0:32:25.280 given that Kuda is what developers know, etc. I'm most 0:32:25.320 --> 0:32:27.280 likely going to use that. That's a good part for 0:32:27.440 --> 0:32:30.560 Nvidia for that. But then there is another way to 0:32:30.640 --> 0:32:33.720 use those models, which is to have someone host them 0:32:33.760 --> 0:32:36.840 for me and I just access them to an API. 0:32:37.280 --> 0:32:40.760 That's what services like Amazon Bedrock does. You basically choose 0:32:40.760 --> 0:32:42.920 your own model and then you serve it through them. 0:32:43.160 --> 0:32:46.040 When you do that, you don't really know what's underneath. 0:32:46.440 --> 0:32:48.120 You don't know if it's a VP, or if it 0:32:48.160 --> 0:32:51.120 is an accelerator, if it is Amazon's own chips or 0:32:51.320 --> 0:32:54.840 Google's own chips, etc. So now the real question, that's 0:32:54.840 --> 0:32:58.040 why the trillion dollar question is are most people going 0:32:58.120 --> 0:33:03.160 to use those models through hosted environments where the hyperscaler 0:33:03.200 --> 0:33:04.920 will have a lot of freedom with regards to what 0:33:05.000 --> 0:33:07.920 they use underneath, and most likely they will vertically integrate 0:33:08.600 --> 0:33:11.400 or are they going to use them you know, themselves 0:33:11.560 --> 0:33:13.720 in a more more like you know, in a self 0:33:13.760 --> 0:33:16.760 service way, And in that case it's less likely that 0:33:17.040 --> 0:33:20.960 those accelerators are going to dominate. We currently are in 0:33:20.960 --> 0:33:23.520 a sort of a you know, balanced way because we 0:33:23.600 --> 0:33:25.680 have our own that we use like I described, and 0:33:25.720 --> 0:33:28.440 also we use you know, the hosted models. And so 0:33:28.720 --> 0:33:31.160 where is this going to go? It's hard to say, 0:33:31.240 --> 0:33:34.360 because I think it depends on the evolution of the models, 0:33:34.400 --> 0:33:36.400 and it depends which models are going to be made 0:33:36.400 --> 0:33:39.080 available as an open source that you can actually host yourself. 0:33:39.880 --> 0:33:42.040 And I think right now one of the greatest questions 0:33:42.160 --> 0:33:45.480 is are the open source models are going to be 0:33:45.600 --> 0:33:49.160 in absolutely on parer alternative to the to the hosted model, 0:33:49.200 --> 0:33:53.120 to the to the foundational proprietary models, and that given 0:33:53.160 --> 0:33:55.960 Glama three point one, that answer seems to be more likely. 0:33:56.120 --> 0:33:59.280 Yes, I had a question about this actually, which is 0:33:59.440 --> 0:34:03.440 do you think Wall Street's attitudes towards open source have 0:34:03.640 --> 0:34:06.280 changed over time? And the reason I ask is because 0:34:06.320 --> 0:34:09.799 nowadays it seems like a fact of life. Everyone uses 0:34:09.920 --> 0:34:12.520 open source, whether you're a Goldman or somewhere else. But 0:34:12.600 --> 0:34:16.480 I remember, you know, like back in as recently as 0:34:16.719 --> 0:34:20.480 like twenty twelve. I remember Deutsche Bank had like this 0:34:20.600 --> 0:34:25.640 open source project called the Loadstone Foundation, where they were like, oh, 0:34:25.680 --> 0:34:29.000 we should all stop wasting our own resources developing our 0:34:29.040 --> 0:34:31.279 own code and our own software. We should all pool 0:34:31.320 --> 0:34:34.239 our resources together and do open source. And they had 0:34:34.239 --> 0:34:38.239 to actually lobby. It was unsuccessful ultimately, but they were 0:34:38.239 --> 0:34:40.840 trying to get all the banks on Wall Street to 0:34:40.880 --> 0:34:44.479 work together for open source. Nowadays, it seems like there's 0:34:44.520 --> 0:34:47.040 been this significant cultural shift, it's not even a question. 0:34:47.719 --> 0:34:51.080 So in general, my direction, my guide as to you know, 0:34:51.120 --> 0:34:55.080 my team is a don't build anything unless you have to. 0:34:57.000 --> 0:34:59.480 Don't think that just because you're a smart person you 0:34:59.520 --> 0:35:02.439 can build software better than anybody else. Maybe you can, 0:35:03.080 --> 0:35:05.200 but it's a good thing that we focus on building 0:35:05.200 --> 0:35:09.080 things that are actually differentiating for us. And then I 0:35:09.120 --> 0:35:11.480 think the use of open source software, which we very 0:35:11.520 --> 0:35:15.839 much endorse, is also really good hedge with regards to 0:35:16.200 --> 0:35:19.320 you know, which vendors to use, because it really heavily 0:35:19.360 --> 0:35:23.800 reduces the vendor lock in. Of course, open source software, 0:35:24.000 --> 0:35:26.800 as you know, is a tremendous long tail. There's millions 0:35:26.840 --> 0:35:29.520 of that, and so I think there are best practices 0:35:29.560 --> 0:35:33.200 around the use of open source, and those best practices are, 0:35:33.400 --> 0:35:35.360 you know, like you know you need to run reviews 0:35:35.360 --> 0:35:38.280 on open source tech or tech risk reviews or security 0:35:38.320 --> 0:35:41.799 reviews or anything as I've almost built it yourself. And 0:35:41.840 --> 0:35:46.759 then secondly tending to concentrate on the larger, very well 0:35:46.840 --> 0:35:50.279 supported by the community type of open source. And so 0:35:50.840 --> 0:35:53.360 my philosophy is yes to open source, but then you 0:35:53.480 --> 0:35:57.000 need to own it in in truest way because you 0:35:57.040 --> 0:35:59.600 are actually going to be generally the one that actually 0:35:59.719 --> 0:36:02.520 needs to support that as or really building knowledge around that. 0:36:02.680 --> 0:36:05.279 And now you can ask AI to run the code 0:36:05.360 --> 0:36:06.200 for you and check it. 0:36:06.239 --> 0:36:09.120 For yeah, okay. That of course leads to probably what 0:36:09.440 --> 0:36:12.560 if you ask everybody where did you get so far? 0:36:13.040 --> 0:36:16.280 The biggest bank for the back for AI? Most CIOs 0:36:16.280 --> 0:36:19.360 are going to tell you on developer productivity. And I 0:36:19.400 --> 0:36:21.880 think it's something that for us was the first project 0:36:21.880 --> 0:36:23.960 that we actually expanded at scale. I have to say 0:36:23.960 --> 0:36:27.240 that today virtually every developer in Goma SACS is equipped 0:36:27.239 --> 0:36:30.040 to with generative coding tools, and you know we have 0:36:30.080 --> 0:36:33.200 twelve thousand of that. So we didn't enable yet the 0:36:33.239 --> 0:36:36.840 ones that are using our own proprietary language called slang, 0:36:36.920 --> 0:36:39.720 but everybody else has an AI tool and the resulso 0:36:39.800 --> 0:36:41.279 be pretty extraordinary. 0:36:41.440 --> 0:36:43.360 How do you measure that? What are what are some numbers? 0:36:43.440 --> 0:36:44.440 Or how would you describe the right? 0:36:44.560 --> 0:36:48.000 So we measure it according to a number of metrics, 0:36:48.000 --> 0:36:51.040 such as the time that it takes from let's say 0:36:51.040 --> 0:36:53.400 when you start the sprint, when you actually commit the code, 0:36:53.680 --> 0:36:56.000 or when you complete your task. We measure it by 0:36:56.120 --> 0:36:58.680 number of commits, meaning how many times you actually put 0:36:58.680 --> 0:37:01.720 code into production. We measure it by a number of defects, 0:37:01.760 --> 0:37:04.880 which in this case is like, for example, deployment related errors. 0:37:05.040 --> 0:37:08.319 So there are more like velocity and quality metrics. At 0:37:08.360 --> 0:37:14.000 the same time, we have seen a wide range ranging 0:37:14.040 --> 0:37:18.479 from ten to forty percent productivity increase. I would say 0:37:18.480 --> 0:37:22.880 that today we are probably on average seeing twenty percent. Now, 0:37:23.040 --> 0:37:25.880 developers don't spend one hundred percent of their time coding. 0:37:26.640 --> 0:37:29.040 They maybe spend fifty percent of their time coding. So 0:37:29.480 --> 0:37:31.759 your question is what are they doing with half of 0:37:31.840 --> 0:37:35.160 their times where there is a lot of other activities 0:37:35.239 --> 0:37:39.440 such as documenting code, such as doing deployment, doing deployment scripts, 0:37:39.480 --> 0:37:41.799 doing you know, buntio tests, et cetera, et cetera. So 0:37:41.840 --> 0:37:45.880 what's called generally the software development life cycle. Okay, and 0:37:45.960 --> 0:37:49.120 so we see net of ten percent. But then the 0:37:49.160 --> 0:37:51.320 cool thing is that those AIS and the things that 0:37:51.360 --> 0:37:55.000 we're building around that are starting to go beyond coding. 0:37:55.440 --> 0:37:57.799 They're starting to help you write the right tests, write 0:37:57.800 --> 0:38:01.360 the right documentation. They are even figure out algorithms or 0:38:01.400 --> 0:38:06.719 even for example, reducing or minimizing the likelihood of deployment 0:38:06.800 --> 0:38:10.520 issues writing deployment scripts for you. So as that expands, 0:38:10.800 --> 0:38:12.719 we're going to be closer to one hundred percent, and 0:38:12.760 --> 0:38:14.920 therefore we're going to be closer probably to twenty percent, 0:38:15.000 --> 0:38:17.120 which you know, for an organization of our side, is 0:38:17.160 --> 0:38:18.680 a pretty massive efficiency play. 0:38:18.800 --> 0:38:20.839 Can I ask a question about hiring developers? So I've 0:38:20.840 --> 0:38:23.759 probably read one hundred articles over the years about Wall 0:38:23.800 --> 0:38:26.640 Street competing with tech companies to hire developers, like, oh, 0:38:26.680 --> 0:38:28.360 they got a ping pong Lloyd Blank. 0:38:28.120 --> 0:38:30.040 Fine used to say, they are a technology company. 0:38:30.160 --> 0:38:32.279 Yeah, you gotta have your ping pong tables and your 0:38:32.320 --> 0:38:34.640 free lunches and let people are sneakers and I have 0:38:34.719 --> 0:38:37.560 all that stuff. But now it seems with AI, there's 0:38:37.680 --> 0:38:41.320 a number of people interested in who are truly believing 0:38:41.400 --> 0:38:44.080 that within a few years they might build the digital 0:38:44.120 --> 0:38:47.719 god that's ten thousand times smarter than any human, and 0:38:47.800 --> 0:38:50.719 that they approach the task with messianic fervor. And I 0:38:50.719 --> 0:38:53.799 imagine it, right if you're at Goldman and you're trying 0:38:53.840 --> 0:38:57.360 to help a banker answer a question to a client 0:38:57.520 --> 0:39:00.600 about something in the chemical industry, like maybe that's not 0:39:00.680 --> 0:39:03.120 like the thing that gets you out of bed the way, 0:39:03.280 --> 0:39:06.520 sort of like metaphysical realms about what is the nature 0:39:06.560 --> 0:39:09.879 of consciousness and things like that that people talk. Does 0:39:09.920 --> 0:39:13.200 that present any challenges or anything when trying to hire 0:39:13.480 --> 0:39:15.080 talented a developers. 0:39:15.560 --> 0:39:19.960 I think developers love to solve real problems. And one 0:39:19.960 --> 0:39:22.520 of the things also that attracted me in the first place, 0:39:23.080 --> 0:39:24.800 Not that it matters, but I'm saying, you know, I 0:39:24.880 --> 0:39:28.160 tell you my own personal experience is that working in 0:39:28.160 --> 0:39:31.920 a technology company is absolutely fantastic, but you're always like 0:39:31.960 --> 0:39:35.000 one step removed from the business or from the application. 0:39:35.160 --> 0:39:36.799 So I have to you know, let's say you are 0:39:36.840 --> 0:39:40.160 the bank and I'm the technology company. I need to 0:39:40.239 --> 0:39:41.920 sell you a tool that then you're going to use 0:39:42.040 --> 0:39:45.200 to run your business or improve your business. We are 0:39:45.280 --> 0:39:48.520 kind of one degree of separation. Less I were right 0:39:48.560 --> 0:39:52.760 there in a digital business there is fast, huge amounts 0:39:52.760 --> 0:39:55.920 of data, huge amounts of flaws, immediate results, and that's 0:39:56.000 --> 0:40:00.360 kind of addictive. And so developers, especially when a AIS 0:40:00.400 --> 0:40:02.839 are starting to do all those magical things that we're 0:40:02.840 --> 0:40:06.320 talking about, you know, they can see the impact on 0:40:06.360 --> 0:40:08.960 the business right away, and then I think is kind 0:40:09.000 --> 0:40:10.799 of attracting a lot of people. In fact, that there 0:40:10.880 --> 0:40:13.480 is more and more people that are moving into the 0:40:13.560 --> 0:40:20.279 industries oil and gas, transportation, chemical, medical, finance because you know, 0:40:20.320 --> 0:40:22.640 this is new and there's nothing more exciting than seeing 0:40:22.640 --> 0:40:24.680 it in action. And so there is so much action 0:40:24.840 --> 0:40:27.640 going on that I think is actually really really interesting. 0:40:27.680 --> 0:40:29.680 I think another question that maybe you haven't asked me, 0:40:29.680 --> 0:40:31.640 but it's kind of part of this question, is what 0:40:31.760 --> 0:40:34.080 kind of developers? How is the profession of being a 0:40:34.120 --> 0:40:35.400 developers actually changed? 0:40:35.480 --> 0:40:37.719 Oh wait, I had a related question. It's not quite 0:40:37.800 --> 0:40:40.680 that question, but you can certainly answer that too. But Okay, 0:40:41.040 --> 0:40:45.000 to my knowledge, Goldman Sachs doesn't have a job title 0:40:45.160 --> 0:40:49.920 specifically with the words prompt engineer in it. So, looking 0:40:50.000 --> 0:40:53.799 at the impact of AI on your business overall, is 0:40:53.880 --> 0:41:00.640 AI a net hiring positive or a net hiring negative 0:41:01.239 --> 0:41:03.280 for gold Men's employees overall? 0:41:05.120 --> 0:41:07.560 Well, meaning, are we going to hire more or less development? 0:41:07.600 --> 0:41:10.000 Yeah, it doesn't lead to more jobs because you're doing 0:41:10.080 --> 0:41:13.400 more things and productivity increases. Or does it lead to 0:41:13.440 --> 0:41:16.080 fewer jobs because now you can automate a bunch of stuffs. 0:41:16.120 --> 0:41:18.440 Well, listen, there is so many things that we would 0:41:18.480 --> 0:41:20.920 like to do if we had more resources that I 0:41:20.960 --> 0:41:23.920 think this is going to be leading to more things 0:41:23.920 --> 0:41:26.600 that we can do. You know, some people tell me sometimes, 0:41:26.600 --> 0:41:29.080 so you're gonna maybe hire less or have less developers. 0:41:29.719 --> 0:41:32.240 I don't know. I've been in it quote and quote 0:41:32.280 --> 0:41:35.279 for like literally almost forty years, and I've never ever 0:41:35.360 --> 0:41:39.560 seen that go down. But I've seen inflection points where 0:41:39.840 --> 0:41:42.600 you can actually get developers to do way more and 0:41:42.760 --> 0:41:46.960 worry about way less. There is not related to a 0:41:47.040 --> 0:41:49.759 business outcome, and so I think it's more like how 0:41:49.800 --> 0:41:52.879 the profession is going to change. In my opinion, we're 0:41:52.920 --> 0:41:56.880 going to be less low level and more Hey, I 0:41:56.920 --> 0:41:59.440 need to really understand the business problem. Hey, I really 0:41:59.520 --> 0:42:02.360 need to think outcome driven. Ay, I need to have 0:42:02.400 --> 0:42:04.480 a crisp mental model and I need to be able 0:42:04.480 --> 0:42:06.719 to describe it in words. So the profession is going 0:42:06.800 --> 0:42:10.160 to change, and there are tasks that I think are 0:42:10.239 --> 0:42:14.520 so repetitive that the automation of those is actually going 0:42:14.600 --> 0:42:18.120 to help developers, you know, really kind of feeling really 0:42:18.160 --> 0:42:21.239 really connected with the business and with the strategy, and 0:42:21.239 --> 0:42:24.000 that will attract people that are generally curious, that are 0:42:24.040 --> 0:42:27.239 generally interested in understanding what we actually do. So the 0:42:27.239 --> 0:42:30.160 focus kind of shifts from the how to do what 0:42:30.520 --> 0:42:33.080 and to the why, which is really kind of the heart. 0:42:33.640 --> 0:42:35.799 Or think of this evolution of technology over the years 0:42:35.800 --> 0:42:38.600 from the back office of it, which doesn't even know 0:42:38.600 --> 0:42:40.440 what you're doing, but as long as your monitor is 0:42:40.440 --> 0:42:44.520 actually working to hey, I'm actually able to take a 0:42:44.520 --> 0:42:47.400 business problem and break it down into pieces that then 0:42:47.440 --> 0:42:50.440 even an AI can write code for. So to your 0:42:50.440 --> 0:42:54.759 specific question, I think this might maybe potentially for some 0:42:54.800 --> 0:42:57.399 companies are going to try to realize some of those 0:42:57.440 --> 0:43:01.319 efficiencies by curbing the growth or even sometimes reducing it. 0:43:01.719 --> 0:43:05.520 For companies like us that are extremely competitive, for companies 0:43:05.520 --> 0:43:08.160 that have lots of ambition, this race at the end 0:43:08.160 --> 0:43:09.600 of the day, and I think we're going to go 0:43:09.719 --> 0:43:12.279 for you know, trying to get even more out of 0:43:12.320 --> 0:43:14.919 our developers and actually like you know, trying to turn 0:43:14.960 --> 0:43:18.040 them more into something that makes them feel super super connected. 0:43:18.080 --> 0:43:18.919 To the business. 0:43:19.640 --> 0:43:23.480 What about non developer roles, non tech roles, And you know, again, 0:43:23.520 --> 0:43:26.200 I guess a company like Goldman doesn't have you know, 0:43:26.200 --> 0:43:29.120 probably a lot of like low level customers support things 0:43:29.160 --> 0:43:31.000 for in a window is like oh, I need to 0:43:31.160 --> 0:43:33.960 change my plane ticket, et cetera. But you know, a 0:43:34.000 --> 0:43:38.440 lot of modern work is essentially just answering somebody's basic question. 0:43:39.040 --> 0:43:41.480 Are the roles within a bank that are going to 0:43:41.560 --> 0:43:44.880 either fundamentally change or go away due to sort of 0:43:45.280 --> 0:43:46.880 agentic or generative AI. 0:43:48.000 --> 0:43:50.800 I think a lot of the work there is about 0:43:51.120 --> 0:43:56.279 content production or content summarization will actually be streamlined quite 0:43:56.320 --> 0:44:00.359 a bit, like, for example, taking an earnings report, making 0:44:00.400 --> 0:44:02.880 it into ten different sauces in order to wear for 0:44:02.960 --> 0:44:05.960 different channels of distribution. Here's the one for internal people, 0:44:05.960 --> 0:44:07.640 here's the one for the client, here's the one for 0:44:07.680 --> 0:44:10.960 the website, et cetera, et cetera. Imagine the creation of 0:44:11.000 --> 0:44:13.480 pitch books for clients where you take ten plates, you 0:44:13.480 --> 0:44:15.480 put a bunch of data, you go out and do research, 0:44:15.560 --> 0:44:17.680 you take logos, you take this, you take that. There 0:44:17.760 --> 0:44:21.480 is a lot of that machinery and factory, which you know, 0:44:21.480 --> 0:44:24.120 we have thousands of people doing that I'm sure there's a. 0:44:24.080 --> 0:44:26.799 Lot of junior analyst who would be maybe glad to 0:44:26.880 --> 0:44:29.160 hear that some of making a pitchbox is going to 0:44:29.239 --> 0:44:29.440 be on. 0:44:29.760 --> 0:44:31.640 But I think that's a good thing. It takes away 0:44:31.719 --> 0:44:33.880 some of the toil. And so I think at the 0:44:33.960 --> 0:44:36.960 end of the day, listen right now, have you noticed 0:44:36.960 --> 0:44:40.480 that everything is kind of converging to words and concepts, 0:44:40.600 --> 0:44:43.040 no matter if you're a developer, if you're a knowledge worker, 0:44:43.320 --> 0:44:47.640 those jobs are candle colliding. And I'm absolutely developers have 0:44:47.719 --> 0:44:51.880 seen that first. Why well, because it's a low hanging fruit. 0:44:51.960 --> 0:44:54.400 The developers deal with the vocabulary. There is no fifty 0:44:54.440 --> 0:44:57.480 thousand words. There's like two three hundred keywords for language, 0:44:57.480 --> 0:44:59.480 and so of course that works extremely well, and of 0:44:59.480 --> 0:45:01.879 course that's the first thing to go. But I think 0:45:01.920 --> 0:45:05.359 eventually the knowledge worker is going to be, you know, 0:45:05.480 --> 0:45:07.400 the one that is really benefit and no matter if 0:45:07.440 --> 0:45:09.839 you are a developer or or or if you are 0:45:10.080 --> 0:45:11.960 working on a pitch book, or if you're working on 0:45:11.960 --> 0:45:14.440 a summarization of a meeting or the action items, or 0:45:14.480 --> 0:45:16.920 you're working on a strategy, et cetera, et cetera. And 0:45:16.960 --> 0:45:21.640 I think overall this will elevate the quality of the work, 0:45:21.800 --> 0:45:24.920 which then everybody says a happy worker or a happy 0:45:24.920 --> 0:45:27.799 developer is a productive developer. I think you're happy when 0:45:27.800 --> 0:45:29.840 you're actually doing something that allows you to do your 0:45:29.920 --> 0:45:33.480 best work. And I'm hoping that if AI allows all 0:45:33.520 --> 0:45:36.440 of us to do more of our best work, I 0:45:36.480 --> 0:45:38.280 think it's going to be, you know, probably the biggest 0:45:38.320 --> 0:45:39.319 effect that we can have. 0:45:39.719 --> 0:45:41.319 I know, we just have a couple more minutes. So 0:45:41.400 --> 0:45:44.320 one very quick question, what makes a good prompt? 0:45:45.160 --> 0:45:49.239 Well, believe it or not. Empathy. You need to be empathic, 0:45:49.280 --> 0:45:51.080 and you need to be gentle, and you need to 0:45:51.120 --> 0:45:54.080 be kind, and you need to kind of, you know, just. 0:45:55.560 --> 0:45:57.040 Like empathetic. 0:46:00.120 --> 0:46:01.760 She makes fun of me for how empathement. 0:46:01.840 --> 0:46:04.320 You know, I've said, it's very sweet that you say. 0:46:04.960 --> 0:46:07.480 You need to take the AI literally by the hand 0:46:07.560 --> 0:46:09.600 and take it where you want to go. And I 0:46:09.680 --> 0:46:11.759 tell you that, you know, one of my interesting, more 0:46:11.800 --> 0:46:15.080 interesting experience with prompts is the following. You know how 0:46:15.120 --> 0:46:17.520 hard it is to get an AI to say I 0:46:17.560 --> 0:46:21.120 don't know. It's almost impossible. You're always going to get 0:46:21.120 --> 0:46:24.080 an ass And so one time I decided I want 0:46:24.120 --> 0:46:26.120 to get it to the point, and so I had 0:46:26.160 --> 0:46:30.160 to navigate the prompt and the AI to understand that 0:46:30.239 --> 0:46:33.040 it was safe and okay to say I don't know. 0:46:33.800 --> 0:46:36.000 And so then at the end I prompted it, what's 0:46:36.040 --> 0:46:39.840 the capital of you know, the United States? Okay? And 0:46:39.880 --> 0:46:42.400 then I said that you know, what's the weather going 0:46:42.440 --> 0:46:44.200 to be like tomorrow? And I got an answer, and 0:46:44.200 --> 0:46:46.200 then I said what's the weather going to be in 0:46:46.239 --> 0:46:50.000 a year, and it's simply I don't know. And then 0:46:50.200 --> 0:46:53.200 at one point, you know, I even decided what to say. 0:46:53.239 --> 0:46:55.719 It's like, is there a role for humans in a 0:46:55.840 --> 0:46:57.400 world of a eye? 0:46:59.400 --> 0:46:59.960 I don't want to know? 0:47:04.400 --> 0:47:07.640 Okay, Well, everyone's going to be off on chat GPT 0:47:07.840 --> 0:47:09.560 now trying to get it to say I don't know. 0:47:09.800 --> 0:47:12.440 Marco Argenti from Goldman Sachs, thank you so much. That 0:47:12.480 --> 0:47:12.920 was good fun. 0:47:12.960 --> 0:47:14.520 Thank you, Johing, thank you so much. 0:47:14.520 --> 0:47:28.959 Thank you so much, Joe. That was a lot of fun. 0:47:29.000 --> 0:47:30.600 And I have to say I do not make fun 0:47:30.600 --> 0:47:32.640 of you for saying please and thank you to Chat GPT. 0:47:32.880 --> 0:47:35.240 I have I'm going to repeat it. I've said it's endearing, 0:47:35.400 --> 0:47:38.560 it's very sweet, and I've tried to follow your example. 0:47:38.600 --> 0:47:40.160 And I now I don't say thank you because I 0:47:40.239 --> 0:47:42.239 usually move on to the next question. But I do 0:47:42.280 --> 0:47:42.800 say please. 0:47:43.080 --> 0:47:44.359 I've heard this though. 0:47:44.400 --> 0:47:46.400 It's funny that you said that, because I actually have 0:47:46.680 --> 0:47:50.640 heard this that there does seem to be quantitative evidence 0:47:51.200 --> 0:47:54.000 that words like please and thank you, et cetera do 0:47:54.360 --> 0:47:59.600 actually improve really well, yeah, mad Buseegan, who you know 0:47:59.640 --> 0:48:02.640 we've known on Twitter forever, has posted about this. So 0:48:03.400 --> 0:48:05.880 there's a good reason to do it besides just the 0:48:06.000 --> 0:48:08.360 habit the all entities you talk to, you should be 0:48:08.360 --> 0:48:09.160 in the habit of flight. 0:48:09.239 --> 0:48:11.960 Oh yeah, that was your argument, right, yeah, yeah, yeah, Okay, 0:48:11.960 --> 0:48:13.520 Well I thought that was fascinating. 0:48:13.600 --> 0:48:14.000 Yeah. 0:48:14.040 --> 0:48:16.399 We've been talking a lot about AI and the sort 0:48:16.400 --> 0:48:19.800 of potential use cases and the chips that are driving 0:48:19.840 --> 0:48:21.839 the technology and things like that, but it was nice 0:48:21.840 --> 0:48:25.279 to hear from someone who's actually making the purchasing decisions, yes, 0:48:25.360 --> 0:48:27.719 and implementing them at a large institution. 0:48:28.440 --> 0:48:29.040 Absolutely. 0:48:29.040 --> 0:48:31.880 That was probably one of my favorite AI conversations we 0:48:31.960 --> 0:48:36.320 had for precisely that reason, because it was interesting hearing 0:48:36.400 --> 0:48:39.399 him talk about this idea that right now, like these 0:48:39.440 --> 0:48:43.080 open source models, particularly like the latest version of LAMA, 0:48:43.239 --> 0:48:46.920 is getting really close to sort of the core proprietary models. 0:48:47.160 --> 0:48:50.879 That was striking the fact that he sees, perhaps particularly 0:48:50.920 --> 0:48:55.479 on the inference side of model usage, an opportunity for 0:48:55.600 --> 0:48:58.400 greater use of different types of hardware. 0:48:58.440 --> 0:49:01.480 Also very interesting, that's right, And we're so used to 0:49:01.520 --> 0:49:04.560 talking about the massive amounts of power and energy that 0:49:04.600 --> 0:49:07.200 AI will consume, and we you and I have had 0:49:07.280 --> 0:49:09.480 a lot of conversations about how we're going to power 0:49:09.600 --> 0:49:13.279 all these servers and things. But what's gotten far less 0:49:13.360 --> 0:49:17.800 attention is just optimizing the way you use AI such 0:49:17.840 --> 0:49:20.160 that you don't need to consume as much power, So 0:49:20.280 --> 0:49:23.840 maybe doing less training, leaving training to the big like 0:49:23.960 --> 0:49:26.800 hyperscalers or whatever, and then just doing the inference. 0:49:27.080 --> 0:49:28.840 In the end, it's going to be both, right, because 0:49:28.880 --> 0:49:31.439 in the end, like there's both, it's going to happen. 0:49:31.440 --> 0:49:35.160 People are gonna find algorithmic techniques and Marco described some 0:49:35.280 --> 0:49:39.440 of them to lessen the sort of pressure and stress 0:49:39.480 --> 0:49:42.280 that you're putting on the hardware, but of course that's 0:49:42.360 --> 0:49:44.920 just going to mean you're going to use it more. 0:49:45.040 --> 0:49:46.799 And then also people are going to have to solve 0:49:46.800 --> 0:49:49.719 the power consumptions. That kind of like all of economic 0:49:49.800 --> 0:49:53.400 history in general, in which we're always finding new ways 0:49:53.440 --> 0:49:56.440 to get more out of the same you know, gigajewel 0:49:56.640 --> 0:49:59.680 of energy but also using more energy at the same time. 0:49:59.760 --> 0:50:00.240 Yeah. 0:50:00.280 --> 0:50:02.600 Absolutely, well, shall we leave it there. 0:50:02.680 --> 0:50:03.359 Let's leave it there. 0:50:03.520 --> 0:50:06.360 This has been another episode of the aud Thoughts podcast. 0:50:06.440 --> 0:50:09.760 I'm Tracy Alloway. You can follow me at Tracy Alloway. 0:50:09.360 --> 0:50:12.239 And I'm Jill Wisenthal. You can follow me at the Stalwart. 0:50:12.480 --> 0:50:16.000 Follow our producers Carman Rodriguez at Carman Ermann dash O, 0:50:16.040 --> 0:50:19.560 Bennett at Dashbot, and kel Brooks at Kelbrooks. Thank you 0:50:19.600 --> 0:50:22.720 to our producer Moses ONEm and from our Odd Lots content. 0:50:22.760 --> 0:50:25.879 Go to Bloomberg dot com slash od loss. We have transcripts, 0:50:25.880 --> 0:50:28.440 a blog, and a newsletter, and you can chat about 0:50:28.440 --> 0:50:30.759 all of these topics in our discord where we even 0:50:30.800 --> 0:50:34.160 have an AI channel. Great stuff in their discord dot 0:50:34.200 --> 0:50:35.440 gg slash Odlins. 0:50:35.719 --> 0:50:38.200 And if you enjoy Odd Loots, if you like our 0:50:38.280 --> 0:50:41.960 continuing series of AI conversations, then please leave us a 0:50:42.000 --> 0:50:45.880 positive review on your favorite podcast platform. And remember, if 0:50:45.920 --> 0:50:48.279 you are a Bloomberg subscriber, you can listen to all 0:50:48.320 --> 0:50:51.360 of our episodes absolutely ad free. All you need to 0:50:51.400 --> 0:50:54.840 do is connect your Bloomberg account with Apple Podcasts. In 0:50:54.960 --> 0:50:57.080 order to do that, just find the Bloomberg channel on 0:50:57.160 --> 0:51:00.760 Apple Podcasts and follow the instructions there. Thanks for listening.