WEBVTT - Transformations in AI: why foundation models are the future 0:00:02.120 --> 0:00:05.880 Hey, it's Jacob Goldstein for Smart Talks with IBM. Last 0:00:05.920 --> 0:00:08.280 year I had the pleasure of sitting down with doctor 0:00:08.360 --> 0:00:13.400 David Cox, VP of AI Models at IBM Research. We 0:00:13.480 --> 0:00:17.200 explored the fascinating world of AI foundation models and their 0:00:17.239 --> 0:00:21.960 revolutionary potential for business automation and innovation. When we first 0:00:22.000 --> 0:00:25.319 aired this episode, the concept of foundation models was just 0:00:25.400 --> 0:00:29.280 beginning to capture our attention. Since then, this technology has 0:00:29.400 --> 0:00:33.479 evolved and redefined the boundaries of what's possible. Businesses are 0:00:33.520 --> 0:00:36.960 becoming more savvy about selecting the right models and understanding 0:00:36.960 --> 0:00:40.360 how they can drive revenue and efficiency. As I listened 0:00:40.360 --> 0:00:43.279 back to the conversation, it was interesting to reflect on 0:00:43.320 --> 0:00:47.120 some new developments and ideas that have emerged, and many 0:00:47.159 --> 0:00:50.320 of these we will continue to explore throughout the season, 0:00:50.840 --> 0:00:52.920 like how to play an active role in choosing the 0:00:52.920 --> 0:00:56.440 best model for your needs. Whether you're a longtime listener 0:00:56.520 --> 0:00:58.920 or tuning in for the first time, I'm certain you'll 0:00:58.920 --> 0:01:01.800 find doctor Cox's in sites as thought provoking as ever. 0:01:02.360 --> 0:01:05.679 Thanks as all ways for joining us. Now let's dive in. 0:01:07.440 --> 0:01:11.240 Hello, Hello, Welcome to Smart Talks with IBM, a podcast 0:01:11.240 --> 0:01:16.720 from Pushkin Industries, iHeartRadio and IBM. I'm Malcolm Gladwell. Our 0:01:16.760 --> 0:01:21.440 guest today is doctor David Cox, VP of AI Models 0:01:21.480 --> 0:01:26.480 at IBM Research and IBM Director of the MIT IBM 0:01:26.560 --> 0:01:30.800 Watson AI Lab, a first of its kind industry academic 0:01:30.920 --> 0:01:36.000 collaboration between IBM and MIT focused on the fundamental research 0:01:36.400 --> 0:01:41.160 of artificial intelligence. Over the course of decades, David Cox 0:01:41.240 --> 0:01:45.880 watched as the AI revolution steadily grew from the simmering 0:01:45.920 --> 0:01:49.760 ideas of a few academics and technologists into the industrial 0:01:49.840 --> 0:01:54.160 boom we are experiencing today. Having dedicated his life to 0:01:54.240 --> 0:01:57.640 pushing the field of AI towards new horizons, David has 0:01:57.680 --> 0:02:01.600 both contributed to and presided over many of the major 0:02:01.720 --> 0:02:07.320 breakthroughs in artificial intelligence. In today's episode, you'll hear David 0:02:07.320 --> 0:02:12.440 explain some of the conceptual underpinnings of the current AI landscape, 0:02:12.600 --> 0:02:17.520 things like foundation models, in surprisingly comprehensible terms. I might add, 0:02:17.800 --> 0:02:20.959 we'll also get into some of the amazing practical applications 0:02:20.960 --> 0:02:24.000 for AI in business, as well as what implications AI 0:02:24.120 --> 0:02:27.640 will have for the future of work and design. David 0:02:27.680 --> 0:02:31.400 spoke with Jacob Goldstein, host of the Pushkin podcast What's 0:02:31.440 --> 0:02:35.720 Your Problem, A veteran business journalist Jacob has reported for 0:02:35.760 --> 0:02:38.560 The Wall Street Journal, the Miami Herald, and was a 0:02:38.560 --> 0:02:44.240 longtime host of the NPR program Planet Money. Okay, let's 0:02:44.240 --> 0:02:45.120 get to the interview. 0:02:47.480 --> 0:02:49.280 Tell me about your job at IBM. 0:02:49.840 --> 0:02:53.160 So, I wear two hats at IBM. So one, I'm 0:02:53.200 --> 0:02:56.040 the IBM Doctor of the MIT IBM Watson They Lab. 0:02:56.560 --> 0:02:59.600 So that's a joint lab between IBM and MIT where 0:02:59.600 --> 0:03:02.079 we we try and invent what's next in AI. It's 0:03:02.080 --> 0:03:05.000 been running for about five years, and then more recently 0:03:05.040 --> 0:03:08.040 I started as the vice president for AI Models, and 0:03:08.120 --> 0:03:12.400 I'm in charge of building IBM's foundation models, you know, 0:03:12.560 --> 0:03:15.080 building these these big models, generative models that allow us 0:03:15.080 --> 0:03:17.320 to have all kinds of new exciting capabilities in AI. 0:03:17.840 --> 0:03:19.760 So, so I want to talk to you a lot 0:03:19.840 --> 0:03:23.240 about foundation models, about genitive AI. But before we get 0:03:23.280 --> 0:03:26.359 to that, let's just spend a minute on the IBM 0:03:26.600 --> 0:03:31.920 MIT collaboration. Where did that partnership start, How did it originate? 0:03:33.080 --> 0:03:35.880 Yeah? So, actually, it turns out that MIT and IBM 0:03:36.120 --> 0:03:39.200 have been collaborating for a very long time in the 0:03:39.240 --> 0:03:43.280 area of AI. In fact, the term artificial intelligence was 0:03:43.360 --> 0:03:47.000 coined in a nineteen fifty six workshop that was held 0:03:47.040 --> 0:03:49.200 at Dartmouth. It was actually organized by an IBM or 0:03:49.280 --> 0:03:52.600 Nathaniel Rochester, who led the development of the IBM seven 0:03:52.600 --> 0:03:55.840 O one. So we've really been together in AI since 0:03:55.880 --> 0:04:00.720 the beginning, and as AI kept accelerating more and more 0:04:00.760 --> 0:04:04.360 and more, I think there was a really interesting decision 0:04:04.360 --> 0:04:07.080 to say, let's make this a formal partnership. So IBM 0:04:07.120 --> 0:04:08.960 in twenty seventeen and now, so it'll be committing close 0:04:08.960 --> 0:04:11.800 to a quarter billion dollars over ten years to have 0:04:11.920 --> 0:04:15.760 this joint lab with MIT, and we we located ourselves 0:04:15.840 --> 0:04:18.000 right on the campus and we've been developing very very 0:04:18.080 --> 0:04:20.359 deep relationships where we can you know, really get to 0:04:20.400 --> 0:04:23.479 know each other work shoulder to shoulder, conceiving what we 0:04:23.480 --> 0:04:26.120 should work on next, and then executing the projects. And 0:04:26.200 --> 0:04:30.159 it's really, you know, very few entities like this exist 0:04:30.640 --> 0:04:33.600 between academia industry. It's been really fun of the last 0:04:33.640 --> 0:04:34.920 five years to be a part of it. 0:04:35.560 --> 0:04:37.080 And what do you think are some of the most 0:04:37.080 --> 0:04:40.600 important outcomes of this collaboration between IBM and MIT. 0:04:42.000 --> 0:04:44.680 Yeah, so we're really kind of the tip of the 0:04:44.720 --> 0:04:49.640 sphere for for IBM's the I strategy, So we're really looking, 0:04:49.880 --> 0:04:52.680 you know, what's coming ahead and you know in areas 0:04:52.720 --> 0:04:56.480 like Foundation Models, you know, as the field changes, MIT 0:04:56.640 --> 0:04:59.279 people are interested in working on you know, faculty, students 0:04:59.279 --> 0:05:01.440 and staff are interesting working on what's the latest thing, 0:05:01.480 --> 0:05:04.479 what's the next thing. We at IBM Research are very 0:05:04.560 --> 0:05:06.640 much interested in the same. So we can kind of 0:05:06.640 --> 0:05:09.599 put out feelers, you know, interesting things that we're seeing 0:05:09.640 --> 0:05:12.400 in our research, interesting things we're hearing in the field. 0:05:12.400 --> 0:05:14.839 We can go and chase those opportunities. So when something 0:05:14.839 --> 0:05:17.880 big comes, like the big change that's been happening lately 0:05:17.880 --> 0:05:20.440 with Foundation Models, we're ready to jump on it. That's 0:05:20.440 --> 0:05:23.440 really the purpose, that's that's the lab functioning the way 0:05:23.480 --> 0:05:26.640 it should. We're also really interested in how do we 0:05:26.680 --> 0:05:29.760 advance you know AI that can help with climate change 0:05:30.000 --> 0:05:32.960 or you know, build better materials and all these kinds 0:05:32.960 --> 0:05:35.640 of things that are you know, a broader aperture sometimes 0:05:35.640 --> 0:05:38.240 than what we might consider just looking at the product 0:05:38.279 --> 0:05:40.919 portfolio of IBM, and that that gives us again a 0:05:40.960 --> 0:05:43.159 breadth where we can see connections that we might not 0:05:43.200 --> 0:05:46.039 have seen otherwise. We can you know, think things that 0:05:46.080 --> 0:05:48.680 help out society and also help out our customers. 0:05:49.480 --> 0:05:53.920 So the last whatever six months, say, there has been 0:05:53.960 --> 0:05:59.640 this wild rise in the public's interest in AI right, 0:05:59.680 --> 0:06:03.120 clearly coming out of these generative AI models that are 0:06:03.160 --> 0:06:07.279 really accessible, you know, certainly chat GPT language models like that, 0:06:07.320 --> 0:06:10.400 as well as models that generate images like mid Journey. 0:06:11.000 --> 0:06:14.480 I mean, can you just sort of briefly talk about 0:06:13.920 --> 0:06:17.920 the breakthroughs in AI that have made this moment feel 0:06:18.000 --> 0:06:21.480 so exciting, so revolutionary for artificial intelligence. 0:06:22.560 --> 0:06:27.279 Yeah. You know, I've been studying AI basically my entire 0:06:27.320 --> 0:06:29.440 adult life. Before I came to IBM, I was a 0:06:29.440 --> 0:06:32.000 professor at Harvard. I've been doing this a long time, 0:06:32.240 --> 0:06:34.520 and I've gotten used to being surprised. It sounds like 0:06:34.560 --> 0:06:37.640 a joke, but it's serious, Like I'm getting used to 0:06:37.680 --> 0:06:41.599 being surprised at the acceleration of the pace again. It 0:06:41.680 --> 0:06:44.560 tracks actually a long way back. You know. There's lots 0:06:44.600 --> 0:06:47.160 of things where there was an idea that just simmered 0:06:47.839 --> 0:06:51.200 for a really long time. Some of the key math 0:06:51.680 --> 0:06:55.239 behind the stuff that we have today, which is amazing. 0:06:55.960 --> 0:06:58.880 There's an algorithm called back propagation, which is sort of 0:06:58.920 --> 0:07:01.880 key to training networks that's been around, you know, since 0:07:01.920 --> 0:07:05.760 the eighties in wide use, and really what happened was 0:07:06.080 --> 0:07:09.800 it simmered for a long time and then enough data 0:07:10.040 --> 0:07:13.679 and enough compute came. So we had enough data because 0:07:14.320 --> 0:07:17.520 you know, we all started carrying multiple cameras around with us. 0:07:17.520 --> 0:07:20.320 Our mobile phones have all, you know, all these cameras 0:07:20.400 --> 0:07:22.840 and this. We put everything on the Internet, and there's 0:07:22.840 --> 0:07:25.360 all this data out there. We caught a lucky break 0:07:25.360 --> 0:07:27.640 that there was something called the graphics processing unit, which 0:07:27.920 --> 0:07:29.920 turns out to be really useful for doing these kinds 0:07:29.960 --> 0:07:32.480 of algorithms, maybe even more useful than it is for 0:07:32.560 --> 0:07:36.640 doing graphics. They're great graphics too. And things just kept 0:07:36.720 --> 0:07:39.920 kind of adding to the snowball. So we had deep learning, 0:07:40.360 --> 0:07:43.960 which is sort of a rebrand of neural networks that 0:07:44.040 --> 0:07:46.360 I mentioned from from the eighties, and that was enabled 0:07:46.400 --> 0:07:50.080 again by data because we digitalized the world, and compute 0:07:50.080 --> 0:07:52.480 because because we kept building faster and faster and more 0:07:52.520 --> 0:07:55.760 powerful computers, and then that allowed us to make this 0:07:55.760 --> 0:07:59.440 this big breakthrough. And then you know, more recently, using 0:07:59.520 --> 0:08:03.600 the same building blocks, that inexorable rise of more and 0:08:03.640 --> 0:08:08.160 more and more data that the technology called self supervised learning. 0:08:08.640 --> 0:08:13.360 Where the key difference there in traditional deep learning, you know, 0:08:13.400 --> 0:08:16.040 for classifying images, you know, like is this a cat 0:08:16.120 --> 0:08:19.360 or is this a dog? And a picture those technologies 0:08:19.800 --> 0:08:23.120 require super visions, so you have to take what you 0:08:23.200 --> 0:08:24.560 have and then you have to label it. So you 0:08:24.600 --> 0:08:25.920 have to take a picture of a cat, and then 0:08:25.960 --> 0:08:28.640 you label it as a cat, and it turns out that, 0:08:28.800 --> 0:08:31.000 you know, that's very powerful, but it takes a lot 0:08:31.000 --> 0:08:33.920 of time to label gats and to label dogs, and 0:08:34.360 --> 0:08:36.280 there's only so many labels that exist in the world. 0:08:36.679 --> 0:08:40.240 So what really changed more recently is that we have 0:08:40.320 --> 0:08:42.800 self supervised learning where you don't have to have the labels. 0:08:42.800 --> 0:08:45.360 We can just take unannotated data. And what that does 0:08:45.400 --> 0:08:48.480 is it lets you use even more data. And that's 0:08:48.520 --> 0:08:52.120 really what drove this this latest sort of rage. And 0:08:52.120 --> 0:08:54.280 then and then all of a sudden we start getting 0:08:54.320 --> 0:08:58.199 these these really powerful models. And then really this has 0:08:58.240 --> 0:09:03.040 been simmering techechnologies, right, this has been happening for a 0:09:03.080 --> 0:09:07.280 while and progressively getting more and more powerful. One of 0:09:07.280 --> 0:09:11.560 the things that really happened with CHATGBT and technologies like 0:09:12.000 --> 0:09:15.079 stable Diffusion and mid Journey was that they made it 0:09:15.640 --> 0:09:18.319 visible to the public. You know, you put it out 0:09:18.360 --> 0:09:20.600 there the public can touch and feel, and they're like, wow, 0:09:20.880 --> 0:09:24.480 not only is there palpable change, and wow this you know, 0:09:24.520 --> 0:09:26.000 I can talk to this thing. Wow, this thing can 0:09:26.080 --> 0:09:28.959 generate an image. Not only that, but everyone can touch 0:09:29.000 --> 0:09:33.240 and feel and try. My kids can use some of 0:09:33.280 --> 0:09:38.720 these AI art generation technologies. And that's really just launched, 0:09:38.800 --> 0:09:42.040 you know. It's like a propelled slingshot at us into 0:09:42.360 --> 0:09:44.400 a different regime in terms of the public awareness of 0:09:44.400 --> 0:09:45.239 these technologies. 0:09:45.920 --> 0:09:49.040 You mentioned earlier in the conversation foundation models, and I 0:09:49.080 --> 0:09:50.920 want to talk a little bit about that. I mean, 0:09:50.960 --> 0:09:54.360 can you just tell me, you know, what are foundation 0:09:54.600 --> 0:09:57.320 models for AI and why are they a big deal? 0:09:58.520 --> 0:10:02.520 Yeah? So this termoundation model was coined by a group 0:10:02.520 --> 0:10:06.360 at Stanford, and I think it's actually a really apt 0:10:06.480 --> 0:10:09.680 term because remember I said, you know, one of the 0:10:09.679 --> 0:10:13.160 big things that unlocked this latest excitement was the fact 0:10:13.160 --> 0:10:17.040 that we could use large amounts of unannotated data. Could 0:10:17.080 --> 0:10:18.440 we could train a model. We don't have to go 0:10:18.480 --> 0:10:22.000 through the painful effort of labeling each and every example. 0:10:22.559 --> 0:10:24.800 You still need to have your model do something you 0:10:24.800 --> 0:10:27.000 wanted to do. You still need to tell it what 0:10:27.040 --> 0:10:28.600 you want to do. You can't just have a model 0:10:28.640 --> 0:10:31.160 that doesn't, you know, have any purpose. But what a 0:10:31.200 --> 0:10:35.000 foundation models that provides a foundation, like a literal foundation. 0:10:35.280 --> 0:10:37.360 You can sort of stand on the shoulders of giants. 0:10:37.360 --> 0:10:40.079 You could have on these massively trained models and then 0:10:40.120 --> 0:10:42.280 do a little bit on top. You know, you could 0:10:42.480 --> 0:10:44.560 use just a few examples of what you're looking for 0:10:45.360 --> 0:10:47.520 and you can get what you want from the model. 0:10:48.040 --> 0:10:50.080 So just a little bit on top now gets to 0:10:50.200 --> 0:10:52.240 the results that a huge amount of effort used to 0:10:52.280 --> 0:10:54.240 have to put in, you know, to get from the 0:10:54.320 --> 0:10:56.360 ground up to that level. 0:10:56.640 --> 0:11:00.440 I was trying to think of of an analogy for 0:11:00.679 --> 0:11:03.720 sort of foundation models versus what came before, and I 0:11:03.760 --> 0:11:06.200 don't know that I came up with a good one, 0:11:06.240 --> 0:11:07.959 but the best I could do was this. I want 0:11:07.960 --> 0:11:10.920 you to tell me if it's plausible. It's like before 0:11:11.000 --> 0:11:14.520 foundation models, it was like you had these sort of 0:11:14.600 --> 0:11:17.800 single use kitchen appliances. You could make a waffle iron 0:11:17.840 --> 0:11:20.360 if you wanted waffles, or you could make a toaster 0:11:20.520 --> 0:11:23.079 if you wanted to make toast. But a foundation model 0:11:23.160 --> 0:11:25.720 is like like an oven with a range on top. 0:11:25.800 --> 0:11:27.559 So it's like this machine and you could just cook 0:11:27.640 --> 0:11:29.480 anything with this machine. 0:11:30.120 --> 0:11:34.600 Yeah, that's a great analogy. They're very versatile. The other 0:11:34.720 --> 0:11:37.280 piece of it, too, is that they dramatically lower the 0:11:37.400 --> 0:11:40.560 effort that it takes to do something that you want 0:11:40.600 --> 0:11:43.600 to do. And sometimes I used to say about the 0:11:43.640 --> 0:11:45.600 old world of AI, would say, you know, the problem 0:11:45.640 --> 0:11:49.400 with automation is that it's too labor intensive. U sounds 0:11:49.440 --> 0:11:50.400 like I'm making a joke. 0:11:50.640 --> 0:11:55.160 Indeed, famously, if automation does one thing, it substitutes machines 0:11:55.280 --> 0:11:58.520 or computing power for labor. Right, So what does that 0:11:58.600 --> 0:12:02.880 mean to say AI or automation is too labor intensive? 0:12:03.360 --> 0:12:05.360 It sounds like I'm making a joke, but I'm actually serious. 0:12:05.559 --> 0:12:08.800 What I mean is that the effort it took the 0:12:08.880 --> 0:12:12.719 old regime to automate something was very, very high. So 0:12:12.920 --> 0:12:15.800 if I need to go and curate all this data, 0:12:15.840 --> 0:12:19.040 collect all this data, and then carefully label all these examples, 0:12:19.440 --> 0:12:23.400 that labeling itself might be incredibly expensive and time. So 0:12:23.760 --> 0:12:26.360 and we estimate anywhere between eighty to ninety percent of 0:12:26.400 --> 0:12:29.240 the effort it takes to feel then AI solution actually 0:12:29.360 --> 0:12:32.959 is just spent on data so that that has some consequences, 0:12:33.240 --> 0:12:38.600 which is the threshold for bothering. You know, if you're 0:12:38.600 --> 0:12:40.800 going to only get a little bit of value back 0:12:41.040 --> 0:12:43.280 from something, are you going to go through this huge 0:12:43.280 --> 0:12:46.800 effort to curate all this data and then when it 0:12:46.800 --> 0:12:49.240 comes time to train the model. You need highly skilled 0:12:49.240 --> 0:12:53.280 people expensive or hard to find in the labor market. 0:12:53.440 --> 0:12:54.959 You know, are you really going to do something that's 0:12:55.000 --> 0:12:56.920 just a tiny, little incremental thing. Now, you're going to 0:12:56.960 --> 0:13:01.000 do the only the highest value things that warrant level 0:13:01.400 --> 0:13:01.959 because you have. 0:13:01.960 --> 0:13:05.960 To essentially build the whole machine from scratch, and there 0:13:06.000 --> 0:13:08.560 aren't many things where it's worth that much work to 0:13:08.600 --> 0:13:11.600 build a machine that's only going to do one narrow thing. 0:13:12.040 --> 0:13:15.000 That's right, and then you tackle the next problem and 0:13:15.080 --> 0:13:17.400 you basically have to start over. And you know, there 0:13:17.400 --> 0:13:20.240 are some nuances here, like for images, you can pre 0:13:20.280 --> 0:13:22.800 train a model on some other tasks and change it around. 0:13:22.800 --> 0:13:25.760 So there are some examples of this, like non recurring 0:13:25.880 --> 0:13:28.480 cost that we have in the old world too, But 0:13:28.480 --> 0:13:31.040 by and large, it's just a lot of effort. It's hard. 0:13:31.320 --> 0:13:35.600 It takes, you know, a large level of skill to implement. 0:13:36.400 --> 0:13:39.199 One analogy that I like is, you know, think about 0:13:39.200 --> 0:13:41.320 it as you know, you have a river of data, 0:13:41.679 --> 0:13:45.080 you know, running through your company or your institution. Traditional 0:13:45.080 --> 0:13:47.600 AI solutions are kind of like building a dam on 0:13:47.600 --> 0:13:51.080 that river. You know. Dams are very expensive things to build. 0:13:51.400 --> 0:13:55.679 They require highly specialized skills and lots of planning, and 0:13:55.880 --> 0:13:57.560 you know you're only going to put a dam on 0:13:57.960 --> 0:14:00.719 a river that's big enough that you're gonna get enough 0:14:00.800 --> 0:14:03.199 energy out of it that it was worth trouble. You're 0:14:03.200 --> 0:14:04.559 gonna get a lot of value out of that dam. 0:14:04.679 --> 0:14:06.280 If you have a river like that, you know, a 0:14:06.360 --> 0:14:09.960 river of data, but it's actually the vast majority of 0:14:10.160 --> 0:14:12.520 the water you know in your kingdom actually isn't in 0:14:12.559 --> 0:14:16.600 that river. It's in puddles and greeks and bablet bricks, 0:14:16.640 --> 0:14:20.080 And you know, there's a lot of value left on 0:14:20.120 --> 0:14:22.640 the table because it's like, well, I can't there's nothing 0:14:22.720 --> 0:14:24.520 you can do about it. It's just that that's too 0:14:25.480 --> 0:14:28.600 low value. So it takes too much effort, so I'm 0:14:28.640 --> 0:14:30.200 just not going to do it. The return on investment 0:14:30.560 --> 0:14:33.120 just isn't there, So you just end up not automating 0:14:33.160 --> 0:14:35.960 things because it's too much of a pain. Now what 0:14:36.000 --> 0:14:38.440 foundation models do is they say, well, actually, no, we 0:14:38.480 --> 0:14:41.680 can train a base model a foundation that you can 0:14:41.720 --> 0:14:43.360 work on that we don't we don't care. We don't 0:14:43.400 --> 0:14:45.240 specify what the task is ahead of time. We just 0:14:45.280 --> 0:14:48.400 need to learn about the domain of data. So if 0:14:48.440 --> 0:14:51.320 we want to build something that can understand English language. 0:14:51.640 --> 0:14:54.920 There's a ton of English language text available out in 0:14:54.960 --> 0:14:59.040 the world. We can now train models on huge quantities 0:14:59.040 --> 0:15:02.480 of it, and then it learned the structure, learned how 0:15:02.640 --> 0:15:05.200 language you know, good part of how language works on 0:15:05.240 --> 0:15:07.600 all that unlabeled data. And then when you roll up 0:15:07.600 --> 0:15:10.560 with your task, you know, I want to solve this 0:15:10.560 --> 0:15:13.720 particular problem, you don't have to start from scratch. You're 0:15:13.720 --> 0:15:17.200 starting from a very very very high place. So that 0:15:17.280 --> 0:15:19.560 just gives you the ability to just you know, now, 0:15:19.600 --> 0:15:22.440 all of a sudden, everything is accessible. All the puddles 0:15:22.440 --> 0:15:25.160 and greeks and babbling brooks and kettlepons, you know, those 0:15:25.200 --> 0:15:29.960 are all accessible now. And that's that's very exciting. But 0:15:30.040 --> 0:15:32.520 it just changes the equation on what kinds of problems 0:15:32.640 --> 0:15:33.840 you could use AI to solve. 0:15:33.960 --> 0:15:39.400 And so foundation models basically mean that automating some new 0:15:39.520 --> 0:15:42.760 task is much less labor intensive. The sort of marginal 0:15:42.840 --> 0:15:45.840 effort to do some new automation thing is much lower 0:15:45.840 --> 0:15:49.120 because you're building on top of the foundation model rather 0:15:49.200 --> 0:15:53.560 than starting from scratch. Absolutely, so that is that is 0:15:53.680 --> 0:15:57.240 like the exciting good news. I do feel like there's 0:15:58.080 --> 0:16:01.240 a little bit of a countervailing idea that worth talking about. Here, 0:16:01.280 --> 0:16:03.400 and that is the idea that even though there are 0:16:03.440 --> 0:16:08.000 these foundation models that are really powerful that are relatively 0:16:08.040 --> 0:16:10.640 easy to build on top of, it's still the case, 0:16:10.720 --> 0:16:13.960 right that there is not some one size fits all 0:16:14.080 --> 0:16:17.680 foundation model. So you know, what does that mean and 0:16:17.800 --> 0:16:20.280 why is that important to think about in this context? 0:16:20.880 --> 0:16:24.680 Yeah, so we believe very strongly that there isn't just 0:16:24.800 --> 0:16:27.640 one model to rule them all. There's a number of 0:16:27.720 --> 0:16:30.720 reasons why that could be true. One which I think 0:16:30.760 --> 0:16:34.800 is important and very relevant today is how much energy 0:16:35.120 --> 0:16:39.880 these models can consume. So these models, you know, can 0:16:39.920 --> 0:16:45.360 get very, very large. So one thing that we're starting 0:16:45.400 --> 0:16:48.120 to see or starting to believe, is that you probably 0:16:48.160 --> 0:16:53.280 shouldn't use one giant sledgehammer model to solve every single problem, 0:16:53.440 --> 0:16:55.400 you know, like we should pick the right size model 0:16:55.440 --> 0:16:58.240 to solve the problem. We shouldn't necessarily assume that we 0:16:58.280 --> 0:17:02.840 need the biggest, baddest model for every little use case. 0:17:03.320 --> 0:17:05.520 And we're also seeing that, you know, small models that 0:17:05.560 --> 0:17:09.760 are trained like to specialize on particular domains can actually 0:17:09.800 --> 0:17:13.600 outperform much bigger models. So bigger isn't always even better. 0:17:13.680 --> 0:17:16.280 So they're more efficient and they do the thing you 0:17:16.320 --> 0:17:17.960 want them to do better. As well. 0:17:18.480 --> 0:17:21.760 That's right. So Stanford, for instance, a group of Stanford 0:17:21.800 --> 0:17:24.919 trained a model. It is a two point seven billion 0:17:24.960 --> 0:17:28.080 parameter model, which isn't terribly big by today's standards. They 0:17:28.080 --> 0:17:30.359 trained it just on the biomedical literature, you know, this 0:17:30.400 --> 0:17:32.800 is the kind of thing that universities do. And what 0:17:32.840 --> 0:17:36.320 they showed was that this model was better at answering 0:17:36.400 --> 0:17:38.920 questions about the biomedical literature than some models that were 0:17:39.440 --> 0:17:43.159 one hundred billion parameters, you know, many times larger. So 0:17:43.320 --> 0:17:45.880 it's a little bit like you know, asking an expert 0:17:46.320 --> 0:17:49.600 for help on something versus asking the smartest person, you know, 0:17:50.240 --> 0:17:53.040 the smartest person you know, maybe very smart, but they're 0:17:53.040 --> 0:17:56.399 not going to be expertise. And then as an added bonus, 0:17:56.440 --> 0:17:58.399 you know, this is now a much smaller model, it's 0:17:58.480 --> 0:18:00.919 much more efficient to run. We aren't know, you know, 0:18:00.960 --> 0:18:04.760 it's cheaper. So there's lots of different advantages there. So 0:18:05.040 --> 0:18:08.280 I think we're going to see at tension in the 0:18:08.320 --> 0:18:11.600 industry between vendors that say, hey, this is the one, 0:18:11.800 --> 0:18:14.159 you know, big model and then others that say, well, actually, 0:18:14.440 --> 0:18:16.960 you know, there's there's you know, lots of different tools 0:18:16.960 --> 0:18:19.000 we can use that all have this nice quality that 0:18:19.040 --> 0:18:21.680 we outligned at the beginning, and then we should really 0:18:21.680 --> 0:18:23.200 pick the one that makes the most sense for the 0:18:23.560 --> 0:18:24.280 task at hand. 0:18:25.560 --> 0:18:29.960 So there's sustainability basically efficiency. Another kind of set of 0:18:29.960 --> 0:18:32.239 issues that come up a lot with AI A are 0:18:32.440 --> 0:18:36.240 bias hallucination. Can you talk a little bit about bias 0:18:36.480 --> 0:18:38.720 and hallucination, what they are and how you're working to 0:18:39.119 --> 0:18:40.240 mitigate those problems. 0:18:40.640 --> 0:18:43.479 Yeah, so there are lots of issues still as amazing 0:18:43.520 --> 0:18:46.440 as these technologies are, and they are amazing, let's let's 0:18:46.480 --> 0:18:48.960 be very clear, lots of great things we're going to 0:18:49.080 --> 0:18:52.880 enable with these kinds of technologies. Bias isn't a new problem, 0:18:53.240 --> 0:18:57.840 so you know, basically we've seen this since the beginning 0:18:57.880 --> 0:19:00.760 of AI. If you train a model on data that 0:19:01.200 --> 0:19:03.320 has a bias in it, the model is going to 0:19:03.359 --> 0:19:07.920 recapitulate that bias when it provides its answers. So every time, 0:19:08.119 --> 0:19:10.639 you know, if all the text you have says, you know, 0:19:10.680 --> 0:19:13.760 it's more likely to refer to female nurses and male scientists, 0:19:13.800 --> 0:19:15.879 then you're going to you know, get models that you know. 0:19:15.960 --> 0:19:19.040 For instance, there was an example where a machine learning 0:19:19.040 --> 0:19:23.440 based translation system translated from Hungarian to English. Hungarian doesn't 0:19:23.480 --> 0:19:26.760 have gendered pronouns English does, and when you ask them 0:19:26.800 --> 0:19:29.119 to translate, it would translate they are a nurse to 0:19:29.560 --> 0:19:32.520 she is a nurse, translate they are a scientist to 0:19:32.600 --> 0:19:35.680 he is a scientist. And that's not because the people 0:19:35.720 --> 0:19:38.520 who wrote the algorithm were building in bias and coding 0:19:38.560 --> 0:19:40.120 in like, oh, it's got to be this way. It's 0:19:40.119 --> 0:19:42.359 because the data was like that. You know, we have 0:19:42.480 --> 0:19:46.920 biases in our society and they're reflected in our data 0:19:46.960 --> 0:19:50.600 and our text and our images everywhere. And then the 0:19:50.640 --> 0:19:53.760 models they're just mapping from what they've seen in their 0:19:53.800 --> 0:19:56.600 training data to the result that you're trying to get 0:19:56.600 --> 0:19:59.280 them to do and to give, and then these biases 0:19:59.320 --> 0:20:04.240 come out. So there's a very active program of research 0:20:04.600 --> 0:20:06.600 in you know, we we do quite a bit at 0:20:06.600 --> 0:20:10.320 IBM research and my T but also all over the 0:20:10.359 --> 0:20:13.040 community and industry and academia trying to figure out how 0:20:13.080 --> 0:20:16.800 do we explicitly remove these biases, how do we identify them, 0:20:17.119 --> 0:20:18.960 how do you know, how do we build tools that 0:20:19.000 --> 0:20:21.119 allow people to audit their systems to make sure they 0:20:21.119 --> 0:20:23.760 aren't biased. So this is a really important thing. And 0:20:23.920 --> 0:20:27.159 you know, again this was here since the beginning, uh, 0:20:27.440 --> 0:20:31.560 you know, of machine learning and AI, but foundation models 0:20:31.560 --> 0:20:34.840 and large language models and generative AI just bring it 0:20:34.840 --> 0:20:37.600 into sharper even sharper focus because there's just so much 0:20:37.680 --> 0:20:41.000 data and it's sort of building in, baking in all 0:20:41.040 --> 0:20:44.520 these different biases we have, so that that's that's absolutely 0:20:45.040 --> 0:20:47.840 a problem that these models have. Another one that you 0:20:47.880 --> 0:20:51.800 mentioned was hallucinations. So even the most impressive of our 0:20:51.880 --> 0:20:55.720 models will often just make stuff up. And you know, 0:20:55.920 --> 0:20:58.919 the technical term that the field has chosen is hallucination. 0:20:59.480 --> 0:21:02.439 To give you an example, I asked chat tbt to 0:21:02.720 --> 0:21:06.480 create a biography of David Cox's IBM, and you know, 0:21:06.720 --> 0:21:09.439 it started off really well, you know, identifying that I 0:21:09.480 --> 0:21:11.800 was the director of the mt IBM, Watson may and 0:21:11.800 --> 0:21:14.200 said a few words about that, and then it proceeded 0:21:14.200 --> 0:21:18.760 to create an authoritative but completely fake biography of me. 0:21:18.800 --> 0:21:21.320 Where I was British, I was born in the UK, 0:21:22.680 --> 0:21:25.640 I went to British university, you know universities in the UK. 0:21:25.720 --> 0:21:27.320 I was professor the authority. 0:21:27.400 --> 0:21:30.960 Right, it's the certainty that that is weird about it, right, 0:21:30.960 --> 0:21:34.240 It's it's dead certain that you're from the UK, et cetera. 0:21:34.840 --> 0:21:37.879