1 00:00:13,960 --> 00:00:17,239 Speaker 1: Welcome to tech Stuff. This is the inside View. I'm 2 00:00:17,280 --> 00:00:19,440 Speaker 1: os Vloschen here with Cara Price. 3 00:00:19,840 --> 00:00:23,319 Speaker 2: Hello, so as I'm very curious to know more about 4 00:00:23,320 --> 00:00:25,000 Speaker 2: the story you've brought me this week, since it's a 5 00:00:25,040 --> 00:00:27,440 Speaker 2: topic we discussed a lot on this podcast. 6 00:00:27,840 --> 00:00:31,320 Speaker 1: Yes, so today I've got a story about AI in healthcare, 7 00:00:31,760 --> 00:00:35,960 Speaker 1: specifically AI and diagnosis. I spoke with doctor Matthew Lungren, 8 00:00:35,960 --> 00:00:39,040 Speaker 1: who is the chief Scientific officer for Microsoft Health and 9 00:00:39,080 --> 00:00:43,320 Speaker 1: Life Sciences, about this blog post that Microsoft recently published 10 00:00:43,600 --> 00:00:47,440 Speaker 1: with the title the Path to Medical Superintelligence. 11 00:00:47,880 --> 00:00:50,840 Speaker 2: Do I want to know what medical superintelligence is? It's 12 00:00:50,880 --> 00:00:54,639 Speaker 2: more big than just regular intelligence. But I actually heard 13 00:00:54,640 --> 00:00:57,760 Speaker 2: about this study. It was everywhere, and if I remember correctly, 14 00:00:57,840 --> 00:01:00,520 Speaker 2: it was that the AI were better at diagnosing than doctors. 15 00:01:00,600 --> 00:01:04,160 Speaker 1: Right, Yeah, that's right, In fact, four times better. There 16 00:01:04,240 --> 00:01:07,240 Speaker 1: was a headline in Time magazine which really says it all. 17 00:01:07,800 --> 00:01:12,240 Speaker 1: Microsoft's AI is better than doctors are diagnosing disease. Special 18 00:01:12,240 --> 00:01:14,679 Speaker 1: shout out here to Elliot Fishman, who's our old friend. 19 00:01:14,880 --> 00:01:18,000 Speaker 1: He's a professor of radiology at Johns Hopkins and he 20 00:01:18,080 --> 00:01:22,039 Speaker 1: runs this fascinating email group that discusses new developments in AI. 21 00:01:22,760 --> 00:01:24,880 Speaker 1: Matthew Lunger and I are both members of this group, 22 00:01:25,240 --> 00:01:28,120 Speaker 1: and Matthew is also one of the authors of the study. 23 00:01:28,360 --> 00:01:29,880 Speaker 2: What kind of doctor is Doctor Lungren? 24 00:01:30,240 --> 00:01:33,440 Speaker 1: Like Elliott Fishman our friend, he's a radiologist by training 25 00:01:33,520 --> 00:01:36,280 Speaker 1: and has a public health background. He was hired at 26 00:01:36,280 --> 00:01:41,400 Speaker 1: Stanford where he started using machine learning to analyze large 27 00:01:41,480 --> 00:01:43,200 Speaker 1: data sets. Here's Matthew. 28 00:01:43,640 --> 00:01:46,520 Speaker 3: Eventually my lab grew into a very large AI center 29 00:01:46,560 --> 00:01:49,880 Speaker 3: at Stanford, which bridged the computer science department in the 30 00:01:49,880 --> 00:01:54,280 Speaker 3: medical school and kind of saw translation of newest techniques 31 00:01:54,320 --> 00:01:58,920 Speaker 3: into healthcare applications accelerate. Taking that work further, I went 32 00:01:59,000 --> 00:02:03,520 Speaker 3: to Microsoft on sabbatical at Microsoft Research and realized that 33 00:02:03,640 --> 00:02:07,400 Speaker 3: a very similar opportunity was there in big tech if 34 00:02:07,440 --> 00:02:10,800 Speaker 3: you could start to connect the latest technology to problems 35 00:02:10,800 --> 00:02:12,799 Speaker 3: in healthcare. And so that's how I came to be here, 36 00:02:12,800 --> 00:02:14,679 Speaker 3: and that's kind of what I still do all day. 37 00:02:15,160 --> 00:02:17,120 Speaker 1: And Matthew is also one of the authors of the 38 00:02:17,120 --> 00:02:18,040 Speaker 1: Microsoft study. 39 00:02:18,480 --> 00:02:22,240 Speaker 3: I believe that the human expert plus these expert systems 40 00:02:22,280 --> 00:02:24,639 Speaker 3: together will ultimately deliver better care. 41 00:02:25,040 --> 00:02:25,680 Speaker 4: No matter what. 42 00:02:25,560 --> 00:02:29,480 Speaker 3: Profession you're in, there's always a gray haired person that has, 43 00:02:29,680 --> 00:02:31,440 Speaker 3: you know, in some sense, seen it all and kind 44 00:02:31,440 --> 00:02:34,080 Speaker 3: of compressed that into their brain and their pattern matching 45 00:02:34,120 --> 00:02:37,200 Speaker 3: in a way that is just faster than folks that 46 00:02:37,240 --> 00:02:39,920 Speaker 3: don't have as much experience. And that's true anywhere, but 47 00:02:40,000 --> 00:02:42,799 Speaker 3: certainly in medicine, right. I think that the assistance or 48 00:02:42,880 --> 00:02:46,360 Speaker 3: ability of AI to now sort of connect dots in 49 00:02:46,440 --> 00:02:51,680 Speaker 3: ways that maybe can achieve that wisdom or that experience 50 00:02:52,160 --> 00:02:53,400 Speaker 3: and bring that to the surface. 51 00:02:53,760 --> 00:02:55,160 Speaker 4: It's kind of an unprecedented time. 52 00:02:55,480 --> 00:02:59,359 Speaker 1: The only exceptional performance I four times better than human doctors. 53 00:03:00,040 --> 00:03:02,160 Speaker 1: One of the things I found most interesting about the 54 00:03:02,200 --> 00:03:06,040 Speaker 1: study was that it wasn't just one single AI model 55 00:03:06,120 --> 00:03:09,240 Speaker 1: doing a diagnosis. It was a whole team of AI 56 00:03:09,320 --> 00:03:11,920 Speaker 1: models that were able to talk to each other in 57 00:03:12,000 --> 00:03:15,280 Speaker 1: order to count with hypotheses, order tests, and ultimately count 58 00:03:15,280 --> 00:03:16,120 Speaker 1: with a diagnosis. 59 00:03:16,600 --> 00:03:20,360 Speaker 2: So multiple AI models seems a little bit unfair. 60 00:03:20,240 --> 00:03:22,800 Speaker 1: Yes, and in fact we talked about this. The doctors 61 00:03:22,800 --> 00:03:25,760 Speaker 1: in the study were not allowed to call specialists to 62 00:03:25,800 --> 00:03:28,720 Speaker 1: help them with their diagnosis, but the ais were allowed 63 00:03:28,760 --> 00:03:31,160 Speaker 1: to talk to each other. So doctors are not going 64 00:03:31,160 --> 00:03:32,600 Speaker 1: to be made obsolete anytime soon. 65 00:03:32,680 --> 00:03:34,520 Speaker 2: Well good, because I have a physical coming up and 66 00:03:34,560 --> 00:03:37,760 Speaker 2: I don't need four AI models being like, well, this 67 00:03:37,800 --> 00:03:39,520 Speaker 2: girl got real big this year. 68 00:03:40,880 --> 00:03:43,400 Speaker 1: Now, as you and I already know, people are already 69 00:03:43,520 --> 00:03:47,560 Speaker 1: using AI regularly to diagnose themselves. In fact, I think 70 00:03:47,600 --> 00:03:51,440 Speaker 1: more than ten percent of the overall CHATCHBT traffic is 71 00:03:51,480 --> 00:03:54,800 Speaker 1: around medical stuff. This is not always music to the 72 00:03:54,840 --> 00:03:57,480 Speaker 1: ear of doctors, so it was interesting to look at 73 00:03:57,520 --> 00:04:00,200 Speaker 1: an example where this is actually an AI build built 74 00:04:00,360 --> 00:04:03,320 Speaker 1: for doctors and to work with doctors rather than patient facing. 75 00:04:03,800 --> 00:04:06,200 Speaker 1: And the other interesting thing for me, which we talk 76 00:04:06,280 --> 00:04:09,160 Speaker 1: about with Lunger, which we'll get to, is how this 77 00:04:09,280 --> 00:04:13,560 Speaker 1: idea of multiple ais talking to each other can simulate 78 00:04:13,640 --> 00:04:17,120 Speaker 1: the experience of the best hospital systems in the US 79 00:04:17,360 --> 00:04:20,800 Speaker 1: for people who otherwise might not have access to these 80 00:04:20,839 --> 00:04:22,000 Speaker 1: panels and experts. 81 00:04:22,279 --> 00:04:24,920 Speaker 2: I can't wait to hear what you learned from him. 82 00:04:25,240 --> 00:04:28,159 Speaker 1: Well, here's the rest of my conversation with doctor Matthew Lungren. 83 00:04:28,920 --> 00:04:31,920 Speaker 1: So you're a trained doctor, and I want to start 84 00:04:31,960 --> 00:04:34,880 Speaker 1: with the basics, which is diagnosis. I'm not sure when 85 00:04:34,920 --> 00:04:38,200 Speaker 1: the last time you made a diagnosis on a patient was, 86 00:04:38,520 --> 00:04:40,880 Speaker 1: but I'd love to hear from you as a doctor. 87 00:04:41,320 --> 00:04:43,040 Speaker 1: What is the process of diagnosis? 88 00:04:43,320 --> 00:04:46,680 Speaker 4: Yeah, I mean it depends quite a bit on the specialty. 89 00:04:46,720 --> 00:04:50,680 Speaker 3: But as most people know, the classic image of a physician, 90 00:04:50,760 --> 00:04:52,960 Speaker 3: right is to speak with the. 91 00:04:52,880 --> 00:04:54,760 Speaker 4: Patient, kind of do a Sherlock Holmes kind of thing. 92 00:04:54,760 --> 00:04:56,840 Speaker 3: Everyone's seen the shows like House and Things are kind 93 00:04:56,839 --> 00:04:58,960 Speaker 3: of sensationalized sort of the approach. 94 00:04:58,960 --> 00:05:01,200 Speaker 4: But really there's a lot of unknowns that you have 95 00:05:01,240 --> 00:05:01,720 Speaker 4: to tease out. 96 00:05:01,800 --> 00:05:01,880 Speaker 2: Right. 97 00:05:01,960 --> 00:05:04,200 Speaker 3: You have to interview the page, you have to obviously 98 00:05:04,320 --> 00:05:07,120 Speaker 3: interpret labs and other information, and you have to start 99 00:05:07,160 --> 00:05:10,400 Speaker 3: to narrow things down and order appropriate tests. Try not 100 00:05:10,440 --> 00:05:13,240 Speaker 3: to chase too many what we call the zebras, but 101 00:05:13,600 --> 00:05:15,520 Speaker 3: keep those in mind in case you're dealing with one, and. 102 00:05:15,680 --> 00:05:18,160 Speaker 1: The zebra would be the classic House episode. 103 00:05:17,839 --> 00:05:20,039 Speaker 3: Right, yeah, right, Well every House episode is a zebra, 104 00:05:20,080 --> 00:05:22,960 Speaker 3: which actually has some relationship to the study we're going 105 00:05:23,040 --> 00:05:26,520 Speaker 3: to talk about today. But in general, it's more common 106 00:05:26,560 --> 00:05:29,920 Speaker 3: to have an uncommon presentation of a common disease than 107 00:05:30,000 --> 00:05:32,839 Speaker 3: in a common presentation of an uncommon disease, if that 108 00:05:32,880 --> 00:05:33,560 Speaker 3: makes sense. 109 00:05:33,920 --> 00:05:38,600 Speaker 1: Right, right, right, And this kind of relationship between AI 110 00:05:38,680 --> 00:05:41,440 Speaker 1: and doctors has been going on for a few years. 111 00:05:41,760 --> 00:05:45,040 Speaker 1: I remember reading a great piece in the Niyoka about 112 00:05:45,200 --> 00:05:48,840 Speaker 1: how one of the challenges for AI was that the 113 00:05:48,839 --> 00:05:52,920 Speaker 1: best doctors can't actually tell you in words why they're 114 00:05:52,920 --> 00:05:54,200 Speaker 1: good at making diagnoses. 115 00:05:54,320 --> 00:05:55,600 Speaker 4: That's right. It's interesting. 116 00:05:55,640 --> 00:05:58,640 Speaker 3: I think there are things that humans have, many cotton 117 00:05:58,680 --> 00:06:01,000 Speaker 3: adiases that are well undo and I think you know, 118 00:06:01,120 --> 00:06:05,760 Speaker 3: keeping that in check while also trying to leverage the 119 00:06:05,800 --> 00:06:08,359 Speaker 3: information in front of you not be affected by the 120 00:06:08,400 --> 00:06:10,640 Speaker 3: case you just saw or something you just heard at 121 00:06:10,640 --> 00:06:14,200 Speaker 3: a conference, or an error that you experienced years ago 122 00:06:14,240 --> 00:06:18,040 Speaker 3: that's still impacting the way that you think about diagnoses. 123 00:06:18,080 --> 00:06:21,760 Speaker 3: And I think those biases have been well published and 124 00:06:21,839 --> 00:06:25,559 Speaker 3: discussed at nauseum in healthcare, but we're kind of dealing 125 00:06:25,560 --> 00:06:28,760 Speaker 3: with this new human plus AI dance. 126 00:06:29,120 --> 00:06:33,560 Speaker 1: That's fascinating. Yeah. I mean I actually slipped and fell 127 00:06:33,640 --> 00:06:37,080 Speaker 1: down a few stairs at the weekend and bashed my 128 00:06:37,080 --> 00:06:39,160 Speaker 1: head slightly on one of the stairs, and then didn't 129 00:06:39,160 --> 00:06:41,080 Speaker 1: feel very well, and I was like, I wonder if 130 00:06:41,080 --> 00:06:43,800 Speaker 1: I could be concussed. So I did a selfie and 131 00:06:43,839 --> 00:06:45,800 Speaker 1: sent it to check GPT and it said my eyes 132 00:06:45,839 --> 00:06:48,800 Speaker 1: look fine. So I actually, if I'd been more wired, 133 00:06:48,800 --> 00:06:50,159 Speaker 1: I would have gone to the doctor. But there's a 134 00:06:50,200 --> 00:06:51,440 Speaker 1: kind of a duck side to that as well. 135 00:06:51,520 --> 00:06:53,400 Speaker 3: Yeah, I mean I think it sounds like you did okay, 136 00:06:53,400 --> 00:06:55,760 Speaker 3: But I would say that the old saying in healthcare 137 00:06:55,920 --> 00:06:57,880 Speaker 3: during the particularly the rise of the Internet, right, which 138 00:06:57,920 --> 00:07:00,279 Speaker 3: is kind of the other similar kind of technology logic 139 00:07:00,320 --> 00:07:03,840 Speaker 3: advancement that impacted healthcare. We used to say to our patients, 140 00:07:04,400 --> 00:07:07,160 Speaker 3: you know, your Google search does not replace our medical degree, right, 141 00:07:07,160 --> 00:07:09,480 Speaker 3: And that wasn't meant to be a condescending but it 142 00:07:09,560 --> 00:07:11,120 Speaker 3: was just sort of like we had to sort of 143 00:07:11,520 --> 00:07:13,960 Speaker 3: pull them back from the abyss of going down a 144 00:07:14,000 --> 00:07:17,240 Speaker 3: rabbit hole and every ache and pain was immediately terminal cancer, right, 145 00:07:17,240 --> 00:07:19,960 Speaker 3: that kind of But today it's different. It sort of 146 00:07:19,960 --> 00:07:24,240 Speaker 3: reference the experience you just mentioned that's happening everywhere. In fact, 147 00:07:24,280 --> 00:07:27,800 Speaker 3: the recent open Ai launch of GPD five, they spent 148 00:07:27,960 --> 00:07:32,720 Speaker 3: fifteen minutes talking with a patient who went through a 149 00:07:32,840 --> 00:07:35,880 Speaker 3: very difficult battle with cancer and worked with the model 150 00:07:35,880 --> 00:07:40,360 Speaker 3: herself and was able to have very complex medical jard 151 00:07:40,360 --> 00:07:44,000 Speaker 3: and explain to her in plain English, was able to 152 00:07:44,040 --> 00:07:47,120 Speaker 3: help her with questions to ask the position. And as 153 00:07:47,120 --> 00:07:50,360 Speaker 3: someone who still practices and sees patients today, I have 154 00:07:50,400 --> 00:07:52,720 Speaker 3: to say my patients are better informed than maybe ever 155 00:07:52,840 --> 00:07:56,520 Speaker 3: and it's kind of changing the bar with this classic 156 00:07:56,600 --> 00:07:59,600 Speaker 3: information asymmetry problem where the patient has to kind of 157 00:07:59,680 --> 00:08:02,160 Speaker 3: keep up up with the technical speak and all the 158 00:08:02,160 --> 00:08:03,960 Speaker 3: information that we spend decades learning. 159 00:08:04,400 --> 00:08:06,600 Speaker 4: It feels like there's almost a better playing field. 160 00:08:06,600 --> 00:08:08,679 Speaker 3: So I can have this conversation with my patient almost 161 00:08:08,680 --> 00:08:10,520 Speaker 3: at a peer level, is right, and then we can 162 00:08:10,560 --> 00:08:14,040 Speaker 3: go through the care journey together. I'm extremely excited about 163 00:08:14,040 --> 00:08:15,560 Speaker 3: that prospect. 164 00:08:15,880 --> 00:08:17,880 Speaker 1: Taking a couple of steps back, I mean, you mentioned 165 00:08:17,960 --> 00:08:21,240 Speaker 1: you've been in and around this since twenty twelve, twenty thirteen. 166 00:08:22,240 --> 00:08:24,520 Speaker 1: Why do people want to use AI medicine. 167 00:08:24,600 --> 00:08:28,560 Speaker 3: Well, it's an incredibly challenging discipline and it has only 168 00:08:28,720 --> 00:08:31,680 Speaker 3: become more so maybe in the last ten or fifteen years. 169 00:08:32,640 --> 00:08:34,320 Speaker 3: One of the things that is going on is that 170 00:08:34,400 --> 00:08:39,439 Speaker 3: information is doubling roughly every ninety days medical information. That 171 00:08:39,760 --> 00:08:41,800 Speaker 3: trend has been going on for a really long time. 172 00:08:41,840 --> 00:08:45,960 Speaker 3: And what does publication of papers, publication of papers, new therapies, 173 00:08:46,120 --> 00:08:49,240 Speaker 3: new guidelines, all these things keep stacking up, right, And 174 00:08:49,280 --> 00:08:52,679 Speaker 3: so just because you've been through medical school and training, right, 175 00:08:52,720 --> 00:08:54,760 Speaker 3: we have lots of systems in place to help us 176 00:08:54,800 --> 00:08:57,880 Speaker 3: continue our education. But really the reaction to that has 177 00:08:57,920 --> 00:09:01,480 Speaker 3: been to sub in some cases sub sub specialize. So 178 00:09:01,520 --> 00:09:04,640 Speaker 3: to give you an example, I am a diagnostic radiologist, 179 00:09:04,640 --> 00:09:08,079 Speaker 3: so that's the bigger specialty, and then I specialize in 180 00:09:08,200 --> 00:09:11,640 Speaker 3: interventional radiology, which is an image guid to procedures basically, 181 00:09:12,080 --> 00:09:14,720 Speaker 3: and then I am further specialized in pediatric version of that. 182 00:09:15,080 --> 00:09:17,440 Speaker 3: So that's like a Russian nesting doll of specialties. And 183 00:09:17,480 --> 00:09:21,000 Speaker 3: you see that throughout healthcare. And that is partly due 184 00:09:21,040 --> 00:09:25,120 Speaker 3: to the complexity of care that's required for some patients, 185 00:09:25,160 --> 00:09:29,120 Speaker 3: but also it's due to the information tidle wave and 186 00:09:29,200 --> 00:09:31,600 Speaker 3: being able to hold all that in a human mind 187 00:09:32,000 --> 00:09:34,679 Speaker 3: right with all of our limitations, and so AI, I 188 00:09:34,760 --> 00:09:38,280 Speaker 3: think at least the work that we've been doing here 189 00:09:38,480 --> 00:09:43,680 Speaker 3: is starting to provide a counter narrative to needing to 190 00:09:43,720 --> 00:09:46,480 Speaker 3: be sub subspecialized in order to be able to manage 191 00:09:46,520 --> 00:09:48,800 Speaker 3: information and take really good care of your patients across 192 00:09:48,840 --> 00:09:52,760 Speaker 3: a wide variety of complex diagnoses. And I think that 193 00:09:52,760 --> 00:09:55,040 Speaker 3: that's really where the excitement is. I think right now 194 00:09:55,120 --> 00:09:59,360 Speaker 3: is can I use this system to augment my ability 195 00:09:59,360 --> 00:10:00,120 Speaker 3: to care for PAYP. 196 00:10:00,920 --> 00:10:05,360 Speaker 1: And why isn't AI more ubiquitous in medicine? And what 197 00:10:05,760 --> 00:10:08,680 Speaker 1: has been integration challenge up until now, Well. 198 00:10:08,480 --> 00:10:11,160 Speaker 3: There's a whole podcast just on that odds, I would say, 199 00:10:11,200 --> 00:10:14,439 Speaker 3: but the short version is that we have been an 200 00:10:14,480 --> 00:10:20,480 Speaker 3: incredibly skeptical discipline it's skeptical of new technology and at 201 00:10:20,480 --> 00:10:24,120 Speaker 3: the same time extraordinarily risk averse for good reason, right, 202 00:10:24,320 --> 00:10:28,800 Speaker 3: we require significant evidence, right to change the way we practice. 203 00:10:29,240 --> 00:10:31,680 Speaker 3: We have you know, as you know, clinical trials take 204 00:10:31,920 --> 00:10:35,199 Speaker 3: years and years, and some still fail, actually many fail, 205 00:10:35,320 --> 00:10:37,720 Speaker 3: and we accept that as the system that keeps our 206 00:10:37,760 --> 00:10:40,400 Speaker 3: patients safe and keeps us on the cutting edge. I 207 00:10:40,440 --> 00:10:44,120 Speaker 3: think in terms of just the technical mechanics of adoption, 208 00:10:44,360 --> 00:10:46,719 Speaker 3: we have a very rigid system in the software two 209 00:10:46,760 --> 00:10:49,880 Speaker 3: world that is changing. What's so again, what's so exciting 210 00:10:49,880 --> 00:10:53,439 Speaker 3: about this is that again any physician can pull out 211 00:10:53,480 --> 00:10:56,440 Speaker 3: their cell phone and interact with this cutting edge AI 212 00:10:56,559 --> 00:10:59,520 Speaker 3: without needing to have you know, three four year long 213 00:10:59,600 --> 00:11:03,120 Speaker 3: cycles of integration with software. Right, and it's just the 214 00:11:03,160 --> 00:11:04,880 Speaker 3: early days, but as of the trends that we're saying, 215 00:11:05,240 --> 00:11:05,520 Speaker 3: just to. 216 00:11:05,480 --> 00:11:09,000 Speaker 1: Take a step back, I guess the classic model of 217 00:11:09,440 --> 00:11:14,160 Speaker 1: measuring AI performance versus doctor performance was to present a 218 00:11:14,160 --> 00:11:18,120 Speaker 1: hard problem or a hard diagnostic conundrum and ask for 219 00:11:18,160 --> 00:11:21,280 Speaker 1: an answer and measure answer versus answer. How is that 220 00:11:21,280 --> 00:11:22,240 Speaker 1: different to what you've done? 221 00:11:22,520 --> 00:11:25,480 Speaker 4: Yeah, well it's even less precise than that. 222 00:11:25,559 --> 00:11:28,400 Speaker 3: So that the way up until now, at least for 223 00:11:28,520 --> 00:11:31,400 Speaker 3: large language models, when people talk about they have medical capabilities, 224 00:11:32,160 --> 00:11:35,320 Speaker 3: they were actually using medical examination questions. 225 00:11:35,320 --> 00:11:37,520 Speaker 4: So there's a question stem and then there's a multiple 226 00:11:37,600 --> 00:11:38,320 Speaker 4: choice answer. 227 00:11:39,080 --> 00:11:41,680 Speaker 3: That's not medicine, but it is how we you know, 228 00:11:41,800 --> 00:11:45,640 Speaker 3: qualify our humans, right, human physicians to be granted a 229 00:11:45,679 --> 00:11:48,080 Speaker 3: medical license, so that we think we kind of use 230 00:11:48,120 --> 00:11:50,600 Speaker 3: that for a long time as a as a surrogate 231 00:11:50,679 --> 00:11:52,360 Speaker 3: or a bell weather, But it wasn't. 232 00:11:52,480 --> 00:11:54,480 Speaker 1: Could it pause a test to be a doctor rather 233 00:11:54,520 --> 00:11:57,360 Speaker 1: than could it actually be effective at acting as a doctor. 234 00:11:57,440 --> 00:12:00,280 Speaker 3: That's interesting, right, And we were able to show very 235 00:12:00,280 --> 00:12:03,840 Speaker 3: early on with GPD four that these models outperform positions 236 00:12:03,840 --> 00:12:06,240 Speaker 3: on these multiple choice tests. But there's all kinds of 237 00:12:06,280 --> 00:12:10,080 Speaker 3: caveats there. Is that really medicine? Has it seen some 238 00:12:10,160 --> 00:12:11,959 Speaker 3: of that data and it's training assuredly? 239 00:12:12,080 --> 00:12:14,319 Speaker 4: Yes? Right? And is that useful? 240 00:12:14,360 --> 00:12:18,120 Speaker 3: I think those questions came up now in practice, it's 241 00:12:18,480 --> 00:12:22,000 Speaker 3: estimated that ten to twenty percent of AI interactions with 242 00:12:22,040 --> 00:12:27,200 Speaker 3: these common chatbots like GPT are around a medical use case. 243 00:12:27,200 --> 00:12:29,360 Speaker 3: So we know that there's someone is getting value out 244 00:12:29,400 --> 00:12:31,520 Speaker 3: of that somewhere, right, and we see it with our 245 00:12:31,520 --> 00:12:33,360 Speaker 3: own eyes. So how do we bridge the gap to 246 00:12:33,400 --> 00:12:37,439 Speaker 3: something a slightly more realistic in terms of not giving 247 00:12:37,440 --> 00:12:39,160 Speaker 3: you all the information up front, just like we would 248 00:12:39,360 --> 00:12:42,240 Speaker 3: in real healthcare. One of the principal thoughts around the 249 00:12:42,240 --> 00:12:45,880 Speaker 3: study was is there a way to take advantage of 250 00:12:45,880 --> 00:12:50,520 Speaker 3: the incredible capabilities that these models have in medical diagnosis. 251 00:12:49,880 --> 00:12:53,760 Speaker 4: And knowledge but also push it a bit further. 252 00:12:53,880 --> 00:12:56,400 Speaker 3: And not have it kind of just be a question 253 00:12:56,440 --> 00:12:59,240 Speaker 3: answering machine. And so we thought, can we kind of 254 00:12:59,280 --> 00:13:01,679 Speaker 3: have several versions of the model kind of act as 255 00:13:01,720 --> 00:13:04,400 Speaker 3: different humans or this is that idea of an agent, 256 00:13:04,760 --> 00:13:07,840 Speaker 3: and give them jobs. One job is to look at 257 00:13:07,840 --> 00:13:10,960 Speaker 3: the economics of the tests that you're trying to order. 258 00:13:11,000 --> 00:13:15,360 Speaker 3: One is to question your next decision point. So the 259 00:13:15,360 --> 00:13:17,559 Speaker 3: information isn't just in and out with one model, it's 260 00:13:17,600 --> 00:13:20,160 Speaker 3: actually in and out through a system of models. And 261 00:13:20,200 --> 00:13:22,120 Speaker 3: we showed that no matter what model you use, whether 262 00:13:22,160 --> 00:13:24,880 Speaker 3: it's Google's model, whether it's open the Eyes model, whether 263 00:13:24,880 --> 00:13:28,520 Speaker 3: it's an open source model, it improves that diagnostic capability 264 00:13:28,600 --> 00:13:32,240 Speaker 3: on these extraordinarily challenging diagnostic tests. 265 00:13:32,640 --> 00:13:35,320 Speaker 1: So you had ten co authors on this study, and 266 00:13:36,000 --> 00:13:38,080 Speaker 1: you know, as we talked about when it was released, 267 00:13:38,240 --> 00:13:40,600 Speaker 1: took the world by storm, and so, I mean, how 268 00:13:40,600 --> 00:13:44,160 Speaker 1: did you go about designing the study and what was 269 00:13:44,200 --> 00:13:46,800 Speaker 1: the hypothesis and what have you found? 270 00:13:47,200 --> 00:13:50,520 Speaker 3: So this was a cross Microsoft collaboration, but harsh and Noori, 271 00:13:50,559 --> 00:13:53,000 Speaker 3: who is the lead on this, really wanted to say, 272 00:13:53,240 --> 00:13:55,360 Speaker 3: you know, we have a lot of evidence that these 273 00:13:55,400 --> 00:13:59,559 Speaker 3: models perform well for these standardized tests, and then we 274 00:13:59,600 --> 00:14:03,320 Speaker 3: see the real world situation where that's not how people present. 275 00:14:03,400 --> 00:14:05,880 Speaker 3: They don't show up with hey, these are all my tests, 276 00:14:05,920 --> 00:14:07,320 Speaker 3: these are all my problems, and these are the four 277 00:14:07,440 --> 00:14:10,000 Speaker 3: choices of what I may have right. And then taking 278 00:14:10,120 --> 00:14:13,079 Speaker 3: what are essentially some of the most difficult questions out 279 00:14:13,120 --> 00:14:15,880 Speaker 3: of New England Journal and structuring them in a way 280 00:14:16,520 --> 00:14:20,120 Speaker 3: that requires a model to ask for more information or 281 00:14:20,200 --> 00:14:21,000 Speaker 3: order tests, just. 282 00:14:20,960 --> 00:14:21,840 Speaker 4: Like a physician would. 283 00:14:22,640 --> 00:14:25,080 Speaker 3: The hypothesis was that that would be interesting and of itself, 284 00:14:25,200 --> 00:14:27,080 Speaker 3: but then what if we also put humans through that 285 00:14:27,120 --> 00:14:32,680 Speaker 3: same system. In other words, here's the first step headache, Okay, 286 00:14:32,880 --> 00:14:33,680 Speaker 3: what do you do next? 287 00:14:33,680 --> 00:14:33,880 Speaker 1: Well? 288 00:14:33,920 --> 00:14:35,520 Speaker 4: Do I need to ask more questions? Do I need 289 00:14:35,520 --> 00:14:36,880 Speaker 4: to order a test, et cetera, et cetera. 290 00:14:37,520 --> 00:14:40,160 Speaker 3: One of the really brilliant outcomes here was by having 291 00:14:40,240 --> 00:14:43,320 Speaker 3: that system of agents as opposed to just the single model, 292 00:14:43,720 --> 00:14:48,240 Speaker 3: allowed us to have a more realistic understanding of the capabilities. 293 00:14:48,240 --> 00:14:50,560 Speaker 3: In other words, if I wanted to know the answer, 294 00:14:50,600 --> 00:14:52,960 Speaker 3: and I'm a chatbot, my answer could be, let's order 295 00:14:53,000 --> 00:14:56,080 Speaker 3: every single test that there is, and that would probably 296 00:14:56,080 --> 00:14:56,920 Speaker 3: get you the right answer. 297 00:14:57,160 --> 00:14:58,040 Speaker 4: Is that feasible? 298 00:14:58,560 --> 00:14:58,760 Speaker 2: No? 299 00:14:59,160 --> 00:14:59,320 Speaker 4: Right? 300 00:14:59,400 --> 00:14:59,480 Speaker 2: Ye? 301 00:15:00,000 --> 00:15:04,000 Speaker 3: So forcing it to think about resources cost of the 302 00:15:04,240 --> 00:15:07,120 Speaker 3: care actually found a very interesting what we would call 303 00:15:07,200 --> 00:15:13,040 Speaker 3: the pride or frontier of capability underconstrained resources. So they 304 00:15:13,040 --> 00:15:16,600 Speaker 3: were actually getting to an incredible diagnoses very very accurately, 305 00:15:17,240 --> 00:15:20,440 Speaker 3: but also cost efficiently, and that was really one of 306 00:15:20,440 --> 00:15:21,960 Speaker 3: the biggest takeaways from this work. 307 00:15:22,960 --> 00:15:25,680 Speaker 1: Can you just to make it more concrete for our listeners, 308 00:15:25,720 --> 00:15:28,560 Speaker 1: can you kind of set up one of these cases 309 00:15:28,800 --> 00:15:32,720 Speaker 1: as though an episode of House Dare I say, and 310 00:15:32,760 --> 00:15:35,920 Speaker 1: then what the human doctors did and what the AI 311 00:15:36,400 --> 00:15:38,840 Speaker 1: agents did, and then how you compare that performance. 312 00:15:39,320 --> 00:15:42,040 Speaker 3: Let's just say it was someone that had easy bleeding 313 00:15:42,120 --> 00:15:44,640 Speaker 3: that unexpected. They were brushing their teeth and they started 314 00:15:44,640 --> 00:15:46,560 Speaker 3: bleeding and it was kind of unusual, and they noticed 315 00:15:46,600 --> 00:15:48,440 Speaker 3: that they were getting a lot of bruising, and there's 316 00:15:48,480 --> 00:15:50,520 Speaker 3: just a certain battery of tests. I think that was 317 00:15:50,560 --> 00:15:53,960 Speaker 3: pretty comparable on both sides in terms of what they ordered. 318 00:15:54,320 --> 00:15:55,840 Speaker 3: But taking continued to. 319 00:15:55,840 --> 00:15:58,000 Speaker 1: Be what the AI ordered and what than human doctors. 320 00:15:57,680 --> 00:15:59,280 Speaker 4: Are human and AI pretty much right. 321 00:15:59,360 --> 00:16:01,480 Speaker 3: So the first few steps, I think a lot there 322 00:16:01,520 --> 00:16:05,080 Speaker 3: was a lot of similarity, which is expected. Where we 323 00:16:05,080 --> 00:16:08,680 Speaker 3: started to see early diversions was because of that agent setup. 324 00:16:09,040 --> 00:16:11,880 Speaker 3: Humans did kind of jump to more advanced tests more quickly, 325 00:16:11,880 --> 00:16:15,320 Speaker 3: more expensive tests, and that was interesting because the models 326 00:16:15,320 --> 00:16:17,040 Speaker 3: were able to kind of get to the next step 327 00:16:17,440 --> 00:16:19,720 Speaker 3: with a battery of less expensive tests. So we thought 328 00:16:19,720 --> 00:16:21,360 Speaker 3: that was a kind of an interesting starting to see 329 00:16:21,360 --> 00:16:24,000 Speaker 3: some divergence. And then, to be fair to the humans, 330 00:16:24,840 --> 00:16:27,680 Speaker 3: they're still kind of handcuffed. In other words, they're just 331 00:16:27,760 --> 00:16:31,600 Speaker 3: getting text feedback as they're interacting with the system, whereas 332 00:16:31,960 --> 00:16:34,680 Speaker 3: when I'm with a patient, I'm seeing them, I'm able 333 00:16:34,720 --> 00:16:37,640 Speaker 3: to kind of take some cues, I'm able to examine them. 334 00:16:37,640 --> 00:16:40,240 Speaker 3: So there was some limitations there, but then the less 335 00:16:40,240 --> 00:16:43,280 Speaker 3: once it got to the stage where you had a 336 00:16:43,280 --> 00:16:47,280 Speaker 3: differential diagnosis, so a list of likely things, more often 337 00:16:47,320 --> 00:16:49,720 Speaker 3: than not, the model was ranking them in a much 338 00:16:49,760 --> 00:16:53,440 Speaker 3: more data driven order that ultimately led to the correct 339 00:16:53,440 --> 00:16:56,560 Speaker 3: diagnosis much more quickly. Whereas you know, as us you 340 00:16:56,600 --> 00:16:58,600 Speaker 3: would with humans, with these limitations, you're kind of going 341 00:16:58,600 --> 00:17:02,040 Speaker 3: in some rabbit holes, you're maybe not ordering them in 342 00:17:02,120 --> 00:17:04,240 Speaker 3: the best order, and so you're kind of going down 343 00:17:04,280 --> 00:17:07,120 Speaker 3: other paths that end up increasing the time or expense 344 00:17:07,200 --> 00:17:08,920 Speaker 3: or potentially leading to the rown diagnosis. 345 00:17:15,720 --> 00:17:19,040 Speaker 2: After the break, how the multi agent system the diagnostic 346 00:17:19,200 --> 00:17:21,880 Speaker 2: orchestrator actually works stay with us. 347 00:17:38,240 --> 00:17:42,560 Speaker 1: I put the study through chet GPT describe the diagnostic 348 00:17:42,680 --> 00:17:45,720 Speaker 1: orchestrator as like a virtual team of five doctors, each 349 00:17:45,760 --> 00:17:49,520 Speaker 1: with a different role. One less possible illnesses, one chooses 350 00:17:49,520 --> 00:17:53,680 Speaker 1: the best tests, one plays devil's advocate, one watches the budget, 351 00:17:53,760 --> 00:17:56,359 Speaker 1: and one checks the quality of everything. The team talks 352 00:17:56,400 --> 00:17:58,440 Speaker 1: it out step by set, but decides what to do next. 353 00:17:58,520 --> 00:18:00,600 Speaker 1: Is that is that a fair summary? That's exactly right? 354 00:18:00,640 --> 00:18:03,000 Speaker 1: And you can have infinite numbers of those agents. 355 00:18:03,040 --> 00:18:05,560 Speaker 3: I think these five were just kind of a scratching 356 00:18:05,560 --> 00:18:08,439 Speaker 3: the surface of what's possible. I will say just quickly 357 00:18:08,480 --> 00:18:11,320 Speaker 3: that I was incredibly happy to see that the curmudgeon 358 00:18:11,359 --> 00:18:13,679 Speaker 3: agent we called it, or the Devil's advocate agent was 359 00:18:13,720 --> 00:18:17,280 Speaker 3: helpful because you get into these group things situations, and 360 00:18:17,320 --> 00:18:20,399 Speaker 3: it's kind of fun to watch a model argue with 361 00:18:20,520 --> 00:18:24,720 Speaker 3: other models about some of the decisions being made in 362 00:18:24,840 --> 00:18:28,280 Speaker 3: questioning the steps. So where the models fall short today 363 00:18:29,320 --> 00:18:32,640 Speaker 3: is outside of the text domain. And what I mean 364 00:18:32,680 --> 00:18:37,320 Speaker 3: by that is models are incredibly good at understanding medical 365 00:18:37,320 --> 00:18:41,320 Speaker 3: concepts as their communicated in text form, but when you 366 00:18:41,359 --> 00:18:44,280 Speaker 3: get into the images and genomics and waveforms and all 367 00:18:44,280 --> 00:18:46,240 Speaker 3: the other types of ways that we take care of 368 00:18:46,320 --> 00:18:51,720 Speaker 3: our patients, the models are vastly underperforming humans. And a 369 00:18:51,760 --> 00:18:53,520 Speaker 3: good example of that is if I needed to look 370 00:18:53,520 --> 00:18:56,400 Speaker 3: at a chest sexuray in one of these diagnostic steps 371 00:18:57,000 --> 00:18:58,840 Speaker 3: and the model had to interpret the chess sector, it 372 00:18:58,840 --> 00:19:01,520 Speaker 3: couldn't read the report actually had to look at the image, 373 00:19:01,880 --> 00:19:04,080 Speaker 3: it would fall short and fail nine times out of ten. 374 00:19:04,560 --> 00:19:07,159 Speaker 3: So we know that that's a significant gap. But on 375 00:19:07,200 --> 00:19:11,080 Speaker 3: the other hand, most healthcare right eighty percent of physician 376 00:19:11,400 --> 00:19:15,320 Speaker 3: or patients interaction with their healthcare systems involve some kind 377 00:19:15,359 --> 00:19:20,840 Speaker 3: of other information like a ECG or a biopsy path 378 00:19:21,040 --> 00:19:25,600 Speaker 3: slide right or a MRI for example. So I'm hoping 379 00:19:25,600 --> 00:19:28,639 Speaker 3: to see agents that have those competencies included into the mix, 380 00:19:29,320 --> 00:19:31,200 Speaker 3: or we can start to really get to a place 381 00:19:31,240 --> 00:19:35,640 Speaker 3: where the diagnostic environment meets how we're testing the systems. 382 00:19:36,160 --> 00:19:39,199 Speaker 1: There was a study last year which I was fascinated by. 383 00:19:39,280 --> 00:19:44,120 Speaker 1: Wish is that AI diagnosis in this study was better 384 00:19:44,200 --> 00:19:47,119 Speaker 1: than human plus AI. In other words, I was a study, 385 00:19:47,119 --> 00:19:49,399 Speaker 1: and you would assume, or you would hope, that a 386 00:19:49,400 --> 00:19:51,760 Speaker 1: doctor using AI would be better than just an AI 387 00:19:51,800 --> 00:19:56,280 Speaker 1: diagnosis alone. But in fact, the human plus AI model 388 00:19:56,400 --> 00:19:59,439 Speaker 1: was worse than the pure AI model. And one of 389 00:19:59,440 --> 00:20:02,239 Speaker 1: the conclusions from this was maybe that the doctors what 390 00:20:02,240 --> 00:20:04,120 Speaker 1: didn't want to listen to what AI was telling them. 391 00:20:04,119 --> 00:20:06,479 Speaker 1: But I mean, did you see that study and did 392 00:20:06,480 --> 00:20:07,240 Speaker 1: it give you pause? 393 00:20:07,680 --> 00:20:10,520 Speaker 3: For more than a decade we've been kind of dealing 394 00:20:10,560 --> 00:20:14,359 Speaker 3: with this unexpected result. This goes all again, all the 395 00:20:14,400 --> 00:20:16,440 Speaker 3: way back to the earliest days of applying at least 396 00:20:16,440 --> 00:20:19,760 Speaker 3: some of the powerful deep learning systems in healthcare, we 397 00:20:19,960 --> 00:20:23,840 Speaker 3: have consistently seen that, in other words, in whatever set 398 00:20:23,920 --> 00:20:26,640 Speaker 3: up the AI, if you just leave it alone, typically 399 00:20:26,640 --> 00:20:28,920 Speaker 3: does better than the human plus THEI or. 400 00:20:28,840 --> 00:20:29,640 Speaker 4: The human alone. 401 00:20:29,880 --> 00:20:34,719 Speaker 3: Now is that a indictment on the human ability or 402 00:20:34,760 --> 00:20:36,840 Speaker 3: is that more of a Have we set this up 403 00:20:36,880 --> 00:20:40,200 Speaker 3: in a way that either doesn't favor the real world, 404 00:20:40,640 --> 00:20:44,119 Speaker 3: or have we not figured out the ideal human computer 405 00:20:44,200 --> 00:20:46,639 Speaker 3: interaction or how we should be What task should we 406 00:20:46,680 --> 00:20:48,840 Speaker 3: be offloading to the system versus the task that we 407 00:20:48,880 --> 00:20:51,280 Speaker 3: should be collaborating with the system on I think that's 408 00:20:51,320 --> 00:20:54,560 Speaker 3: really where the exploration is that I'm interested in, because 409 00:20:54,600 --> 00:20:59,920 Speaker 3: I still hold out hope and sort of some sense 410 00:21:00,240 --> 00:21:03,480 Speaker 3: of self preservation, but that there is a future where 411 00:21:03,520 --> 00:21:06,720 Speaker 3: the two are better. Just how to offload what job 412 00:21:07,440 --> 00:21:11,240 Speaker 3: and in what sort of system that ultimately becomes. Maybe 413 00:21:11,320 --> 00:21:14,880 Speaker 3: it's five agents, maybe it's ten, maybe it's a thousand. 414 00:21:15,080 --> 00:21:17,119 Speaker 3: You know, we don't know the answer yet. We're just 415 00:21:17,119 --> 00:21:20,639 Speaker 3: barely scratching the surface. But in three years time, I 416 00:21:20,680 --> 00:21:23,920 Speaker 3: expect this to be fairly common, that clinicians of all 417 00:21:23,960 --> 00:21:28,080 Speaker 3: types will be working alongside and or even consulting with 418 00:21:28,119 --> 00:21:30,160 Speaker 3: some of these systems for their care their patients. 419 00:21:30,480 --> 00:21:33,040 Speaker 1: And what is the adoption rate today? I mean, how 420 00:21:33,119 --> 00:21:36,240 Speaker 1: far what would need to happen for this, you know 421 00:21:36,440 --> 00:21:38,680 Speaker 1: paper that you've written in the system that you developed 422 00:21:38,680 --> 00:21:42,840 Speaker 1: to be widely deployed in US or global healthcare. 423 00:21:42,680 --> 00:21:45,960 Speaker 3: In a very practical sense, there is a lot of 424 00:21:46,000 --> 00:21:50,399 Speaker 3: regulation around this, and regulation requires very rigorous study and 425 00:21:50,440 --> 00:21:52,600 Speaker 3: evidence and real world deployment, all the things that you 426 00:21:52,600 --> 00:21:55,439 Speaker 3: would expect right if you're you know, care team is 427 00:21:55,600 --> 00:21:57,520 Speaker 3: using some of these things to take to take care 428 00:21:57,520 --> 00:22:00,720 Speaker 3: of you and your health problems generating that evidence, working 429 00:22:00,720 --> 00:22:03,760 Speaker 3: with policy makers, trying to figure out exactly what evidence 430 00:22:04,520 --> 00:22:07,320 Speaker 3: would get to the point where we can say definitively 431 00:22:07,359 --> 00:22:10,879 Speaker 3: this is at standard of care or beyond and it 432 00:22:10,960 --> 00:22:13,720 Speaker 3: should be used and here's how you use it. Those 433 00:22:13,720 --> 00:22:17,040 Speaker 3: are very mechanical, but they're very important. It may also 434 00:22:17,160 --> 00:22:20,479 Speaker 3: require a change in how we approach the regulation of 435 00:22:20,520 --> 00:22:24,400 Speaker 3: medical software because these kinds of systems are challenging our 436 00:22:24,440 --> 00:22:27,240 Speaker 3: traditional software that we have used for decades in healthcare. 437 00:22:27,320 --> 00:22:28,520 Speaker 4: Right, they're very different. 438 00:22:28,560 --> 00:22:32,520 Speaker 3: They're non deterministic, they have moments of brilliance and moments 439 00:22:32,560 --> 00:22:35,360 Speaker 3: of you know, stupidity. I should say, right, you've seen 440 00:22:35,400 --> 00:22:38,159 Speaker 3: these kind of things, and so how do we actually 441 00:22:38,200 --> 00:22:41,760 Speaker 3: design a system where it's safe, effective, and actually improving outcomes? 442 00:22:41,800 --> 00:22:44,199 Speaker 4: And that's ultimately the evidence we have to generate. 443 00:22:44,640 --> 00:22:48,240 Speaker 1: Yeah, I mean beyond stupid mistakes. How do you see 444 00:22:48,280 --> 00:22:51,479 Speaker 1: the risks here? I mean we're seeing this research around 445 00:22:51,560 --> 00:22:54,600 Speaker 1: you know the problems of cognitive offloading with AI, some 446 00:22:54,600 --> 00:22:57,560 Speaker 1: suggestions that if you use AI too much you become 447 00:22:57,640 --> 00:23:01,120 Speaker 1: dumba and deskill yourself. I mean, is there a risk 448 00:23:01,200 --> 00:23:03,960 Speaker 1: of de skilling doctors? Like what are some of the 449 00:23:04,840 --> 00:23:08,800 Speaker 1: maybe intangible but nonetheless medium time risks that we should 450 00:23:08,800 --> 00:23:09,720 Speaker 1: be considering here. 451 00:23:10,160 --> 00:23:13,080 Speaker 3: What they refer to a skill atrophy as real, and 452 00:23:13,400 --> 00:23:16,200 Speaker 3: we've seen this in various other disciplines too. I think 453 00:23:16,840 --> 00:23:22,160 Speaker 3: it will also require a shift in how we think 454 00:23:22,240 --> 00:23:25,720 Speaker 3: and perform our knowledge work jobs. And in one way 455 00:23:25,760 --> 00:23:27,480 Speaker 3: this has been sort of looked at is via the 456 00:23:27,520 --> 00:23:31,119 Speaker 3: idea of meta cognition. So rather than you having to 457 00:23:31,119 --> 00:23:34,760 Speaker 3: be the central source of decision making, are there things 458 00:23:34,760 --> 00:23:37,760 Speaker 3: that you can manage? So the imagine you managing a 459 00:23:37,840 --> 00:23:41,679 Speaker 3: team of a these agents. You have a goal, but 460 00:23:41,760 --> 00:23:45,360 Speaker 3: you're offloading some of the cognitive tasks to those agents. 461 00:23:45,720 --> 00:23:49,199 Speaker 3: Those are some of the early discussions around it. But 462 00:23:49,760 --> 00:23:54,639 Speaker 3: I fundamentally believe that everyone that's in a knowledge work 463 00:23:54,720 --> 00:23:58,960 Speaker 3: industry or role will have to rethink how that role 464 00:23:59,359 --> 00:24:01,359 Speaker 3: evolves in the future. And this is kind of that 465 00:24:01,400 --> 00:24:03,600 Speaker 3: first step, at least for us in the healthcare space, 466 00:24:03,640 --> 00:24:07,200 Speaker 3: which is that do you need to memorize all these 467 00:24:07,200 --> 00:24:09,680 Speaker 3: facts or do you just need to be able to 468 00:24:09,760 --> 00:24:12,520 Speaker 3: have the right judgment and know which where the models 469 00:24:12,800 --> 00:24:14,640 Speaker 3: are good and not good, and be able to fill 470 00:24:14,640 --> 00:24:16,399 Speaker 3: those gaps and manage it like you would a team, 471 00:24:16,440 --> 00:24:20,000 Speaker 3: like any manager would know one oh one, this person's 472 00:24:20,000 --> 00:24:21,879 Speaker 3: good at this, they have some weaknesses here, right, I'm 473 00:24:21,920 --> 00:24:24,119 Speaker 3: going to sign this task to them versus this. 474 00:24:24,280 --> 00:24:24,520 Speaker 4: You know. 475 00:24:25,000 --> 00:24:28,880 Speaker 3: That's that metacognition world that where I think we're rapidly 476 00:24:28,880 --> 00:24:30,960 Speaker 3: heading towards, and healthcare is going to have to figure 477 00:24:31,000 --> 00:24:32,440 Speaker 3: out a way to do that as well. 478 00:24:32,960 --> 00:24:37,200 Speaker 1: Yeah, I mean the other question around deployment is conflict 479 00:24:37,240 --> 00:24:40,480 Speaker 1: of interest. Right, So the previous research I've seen is 480 00:24:40,520 --> 00:24:44,200 Speaker 1: all around AI versus human doctors. But this element you've 481 00:24:44,200 --> 00:24:47,000 Speaker 1: added is to cost as well. It's not just outperforming, 482 00:24:47,040 --> 00:24:50,479 Speaker 1: but it's outperforming at less costs in terms of tests. 483 00:24:50,560 --> 00:24:54,320 Speaker 1: Is a really interesting element, but it adds the potential 484 00:24:54,440 --> 00:24:56,320 Speaker 1: for major conflict of interests on both sides. 485 00:24:56,400 --> 00:24:56,520 Speaker 2: Right. 486 00:24:56,560 --> 00:24:59,920 Speaker 1: So for example, I'm British, grew up with the NH 487 00:25:00,520 --> 00:25:03,600 Speaker 1: and one of the consistent themes of the NHS was 488 00:25:04,119 --> 00:25:09,920 Speaker 1: death panels. Are there bureaucrats deciding when people should die? 489 00:25:09,400 --> 00:25:12,439 Speaker 1: What is the appropriate level of care to give to 490 00:25:12,520 --> 00:25:15,000 Speaker 1: people to prevent them from dying, given that it's a 491 00:25:15,080 --> 00:25:17,720 Speaker 1: drain on a public budget. That's in the UK. Here 492 00:25:17,720 --> 00:25:21,159 Speaker 1: in the US we have these for profit healthcare model 493 00:25:21,240 --> 00:25:24,879 Speaker 1: where there is an incentive which if you're insured do 494 00:25:24,920 --> 00:25:28,159 Speaker 1: you sometimes worry about that your position or healthcare system 495 00:25:28,720 --> 00:25:32,120 Speaker 1: is pushing you through numerous medical procedures because ultimately it's 496 00:25:32,119 --> 00:25:34,560 Speaker 1: a profit center and you may not actually need them. 497 00:25:34,960 --> 00:25:38,640 Speaker 1: So how do you begin to grapple with those problems 498 00:25:38,680 --> 00:25:40,120 Speaker 1: when you think about a system like this. 499 00:25:41,200 --> 00:25:43,840 Speaker 3: These are problems that have existed even as you know 500 00:25:43,960 --> 00:25:49,480 Speaker 3: before before AI, and I think that the responsibility for 501 00:25:50,000 --> 00:25:53,199 Speaker 3: those of us kind of generating the evidence and the capabilities, 502 00:25:53,200 --> 00:25:56,520 Speaker 3: and kind of displaying the rationale behind how these things 503 00:25:56,560 --> 00:26:00,200 Speaker 3: work or don't work and where they work is a 504 00:26:00,200 --> 00:26:04,199 Speaker 3: conversation entirely from both the economics and the cultural societal 505 00:26:04,720 --> 00:26:07,200 Speaker 3: aspects of just how we deliber care. I think it 506 00:26:07,240 --> 00:26:09,639 Speaker 3: wouldn't be a controversial statement to say that, at least 507 00:26:09,640 --> 00:26:13,880 Speaker 3: from the US perspective, that our healthcare system is not ideal. Right, 508 00:26:14,240 --> 00:26:16,320 Speaker 3: And that's true whether you're in a capitated system, a 509 00:26:16,320 --> 00:26:20,080 Speaker 3: fee for service system, or a government based system like 510 00:26:20,119 --> 00:26:22,919 Speaker 3: the VA. I have to hope that the better angels 511 00:26:23,160 --> 00:26:26,040 Speaker 3: prevail here. But I agree with you and share your 512 00:26:26,080 --> 00:26:30,000 Speaker 3: concerns that in the wrong hands, or with the various 513 00:26:30,320 --> 00:26:33,840 Speaker 3: misalignments that happen in these systems at all different levels, 514 00:26:34,600 --> 00:26:38,240 Speaker 3: we could end up causing some disruption in a way 515 00:26:38,280 --> 00:26:39,560 Speaker 3: that we aren't hoping to see. 516 00:26:39,560 --> 00:26:42,040 Speaker 1: In the end, it was an interesting blog post that 517 00:26:42,160 --> 00:26:46,520 Speaker 1: you wrote around cancer care, and it al really struck 518 00:26:46,600 --> 00:26:48,560 Speaker 1: me because I mean, I don't want to put the 519 00:26:48,600 --> 00:26:51,520 Speaker 1: words into your mouth, but as I read it, if 520 00:26:51,560 --> 00:26:53,840 Speaker 1: you are one of the lucky few who gets to 521 00:26:53,880 --> 00:26:56,960 Speaker 1: go to one of the great cancer centers like M 522 00:26:57,040 --> 00:27:00,680 Speaker 1: d Anderson when you're sick with cancer and have access 523 00:27:00,760 --> 00:27:04,520 Speaker 1: to these cross field panel of experts who, as you 524 00:27:04,560 --> 00:27:08,159 Speaker 1: mentioned earlier, sub or sub sub or even subsub sub specialized, 525 00:27:08,920 --> 00:27:12,240 Speaker 1: you have a measurably better outcome, whereas in fact, most 526 00:27:12,240 --> 00:27:15,560 Speaker 1: people in the US and certainly almost all people practically 527 00:27:15,600 --> 00:27:20,040 Speaker 1: speaking globally don't have access to these cancer centers. Talk 528 00:27:20,080 --> 00:27:23,080 Speaker 1: about that and about this idea of the multi agentic 529 00:27:23,560 --> 00:27:26,639 Speaker 1: AI and how it sort of reflects or refracts what 530 00:27:26,680 --> 00:27:28,800 Speaker 1: we've been talking about with the diagnosis piece. 531 00:27:29,040 --> 00:27:31,720 Speaker 3: Yeah, this was a very important So, yeah, thanks for 532 00:27:31,720 --> 00:27:33,399 Speaker 3: bringing this up. I think a lot of people don't 533 00:27:34,000 --> 00:27:35,919 Speaker 3: know some of the inner workings of healthcare where some 534 00:27:35,960 --> 00:27:38,240 Speaker 3: of the really big bottlenecks are in terms of getting 535 00:27:38,280 --> 00:27:40,720 Speaker 3: the best possible outcome, and one of them is in 536 00:27:40,800 --> 00:27:43,960 Speaker 3: cancer care, as you're pointing out where some of the 537 00:27:44,280 --> 00:27:48,240 Speaker 3: leading centers, and in particularly larger cities, they have the 538 00:27:48,320 --> 00:27:53,000 Speaker 3: ability to bring specialists from all different disciplines together to 539 00:27:53,080 --> 00:27:56,359 Speaker 3: discuss the patient's care, and that's called the tumor boarder 540 00:27:56,480 --> 00:28:00,919 Speaker 3: multidimary tumor board. The reason that not everyone can do 541 00:28:01,000 --> 00:28:03,320 Speaker 3: that is not just because they don't maybe have that 542 00:28:03,400 --> 00:28:06,560 Speaker 3: specialist in house, but also because of the massive amount 543 00:28:06,600 --> 00:28:09,520 Speaker 3: of prep time it takes to gather all the information. 544 00:28:10,080 --> 00:28:12,640 Speaker 3: It's not just the patient's data that you have to gather. 545 00:28:12,760 --> 00:28:16,080 Speaker 3: You have to gather what clinical trials are new and 546 00:28:16,119 --> 00:28:18,800 Speaker 3: availed in this patient eligible for, what does the latest 547 00:28:18,840 --> 00:28:21,960 Speaker 3: literature say, And someone has to go through all that information, 548 00:28:22,480 --> 00:28:25,159 Speaker 3: prepare that and then present it to a group. And 549 00:28:25,240 --> 00:28:27,879 Speaker 3: what we found from the ASCO, which is the large 550 00:28:28,600 --> 00:28:32,080 Speaker 3: society in cancer care in the US, was that it 551 00:28:32,200 --> 00:28:33,760 Speaker 3: takes between two and a half and three and a 552 00:28:33,800 --> 00:28:37,920 Speaker 3: half hours of preparation time per patient, and some centers 553 00:28:38,000 --> 00:28:40,880 Speaker 3: run thousands of these tumor boards a year, and those 554 00:28:40,920 --> 00:28:43,240 Speaker 3: are the ones that have the most resources and certainly 555 00:28:43,240 --> 00:28:47,800 Speaker 3: the most access. The idea of AI, fundamentally for me, 556 00:28:47,880 --> 00:28:50,160 Speaker 3: and the reason I'm in this field is that I 557 00:28:50,200 --> 00:28:55,720 Speaker 3: want to democratize that experience for everyone, increase the access. 558 00:28:56,120 --> 00:28:57,840 Speaker 3: So no matter where you live or mate matter what 559 00:28:57,840 --> 00:29:00,280 Speaker 3: you do for a living, should that same level of 560 00:29:00,880 --> 00:29:03,800 Speaker 3: precision when it comes to your healthcare. And so this 561 00:29:04,000 --> 00:29:06,320 Speaker 3: was that first step to that. I do think this 562 00:29:06,360 --> 00:29:09,160 Speaker 3: is going to continue to evolve back to our conversation 563 00:29:09,280 --> 00:29:13,360 Speaker 3: around managing a team of experts as your primary physician, 564 00:29:13,440 --> 00:29:17,160 Speaker 3: could I call on a team of expert agents to 565 00:29:17,280 --> 00:29:19,360 Speaker 3: help walk through some of the things that we might 566 00:29:19,480 --> 00:29:21,800 Speaker 3: not be considering in our fifteen minutes we have together 567 00:29:21,880 --> 00:29:24,840 Speaker 3: once every six months or whatever that looks like. I'm 568 00:29:24,960 --> 00:29:28,480 Speaker 3: very hopeful that given the right circumstances in the way 569 00:29:28,520 --> 00:29:31,880 Speaker 3: the technology is progressing, we're going to get to a place, 570 00:29:31,920 --> 00:29:35,200 Speaker 3: I think, in a perfect world at least where the 571 00:29:35,240 --> 00:29:38,680 Speaker 3: access for every patient is equivalent to those who may 572 00:29:38,680 --> 00:29:40,440 Speaker 3: have access to the best resources. 573 00:29:41,320 --> 00:29:44,120 Speaker 1: I mean, you mentioned that twenty percent of all AI 574 00:29:44,280 --> 00:29:46,480 Speaker 1: search is or op to twenty percent of a AI 575 00:29:46,560 --> 00:29:50,520 Speaker 1: searches are around medicine, which is fascinating. I didn't know that. 576 00:29:51,160 --> 00:29:53,960 Speaker 1: But there are of course other people who don't want 577 00:29:54,000 --> 00:29:56,680 Speaker 1: AI in the healthcare settings, or who worried about their 578 00:29:56,760 --> 00:30:00,440 Speaker 1: human docture or their primary care physician being replaced by 579 00:30:01,160 --> 00:30:03,840 Speaker 1: an unfeeling machine. What do you say to them? 580 00:30:04,040 --> 00:30:06,040 Speaker 3: It's interesting, I think going all the way back to 581 00:30:06,080 --> 00:30:09,440 Speaker 3: the earliest days of search, that stat was still about 582 00:30:09,440 --> 00:30:13,160 Speaker 3: the same. Up to twenty percent of Internet searches were 583 00:30:13,600 --> 00:30:17,479 Speaker 3: healthcare related. And we're seeing two interesting trends. One from 584 00:30:17,520 --> 00:30:19,920 Speaker 3: the economists that showed that these searches that are going 585 00:30:19,960 --> 00:30:22,800 Speaker 3: on today in the typical search engines, the one that's 586 00:30:22,840 --> 00:30:27,800 Speaker 3: going down the fastest is healthcare. Isn't that interesting because 587 00:30:28,200 --> 00:30:29,080 Speaker 3: where are people going? 588 00:30:29,120 --> 00:30:31,960 Speaker 4: Then? Well, they're probably going to the models. So I 589 00:30:32,000 --> 00:30:33,080 Speaker 4: actually push back on that. 590 00:30:33,120 --> 00:30:36,520 Speaker 3: I think that most people want to be educated about 591 00:30:36,560 --> 00:30:38,960 Speaker 3: their medical condition, and they want to be they want 592 00:30:38,960 --> 00:30:41,959 Speaker 3: to feel safe and free to ask questions about their 593 00:30:42,000 --> 00:30:48,280 Speaker 3: own healthcare and essentially infinitely patient knowledgeable sort of oracle environment. 594 00:30:48,400 --> 00:30:50,920 Speaker 3: And again we're not there yet, so I don't want 595 00:30:50,920 --> 00:30:54,720 Speaker 3: to make that claim. But I even me, I put 596 00:30:54,760 --> 00:30:57,360 Speaker 3: my data into these models and ask questions about it, 597 00:30:57,360 --> 00:30:59,680 Speaker 3: and I walk away sometimes learning something or at least 598 00:31:00,160 --> 00:31:03,040 Speaker 3: what I should be asking my physicians. So again, would 599 00:31:03,080 --> 00:31:05,760 Speaker 3: I rather do that than any healthcare Not me personally, 600 00:31:06,080 --> 00:31:08,920 Speaker 3: I do want to have that relationship with my physicians, 601 00:31:08,920 --> 00:31:11,120 Speaker 3: but I also want to walk in much more knowledgeable, 602 00:31:11,560 --> 00:31:13,360 Speaker 3: so I feel like we're on a pure level when 603 00:31:13,360 --> 00:31:15,600 Speaker 3: we're speaking about my care decisions. 604 00:31:16,440 --> 00:31:17,800 Speaker 4: Matt, thank you, Thank you so much. 605 00:31:17,800 --> 00:31:43,600 Speaker 2: As that's it for this week for tech Stuff. I'm 606 00:31:43,680 --> 00:31:44,760 Speaker 2: Cara Price and. 607 00:31:44,640 --> 00:31:47,520 Speaker 1: I'm as Valosha And this episode was produced by Eliza Dennis, 608 00:31:47,560 --> 00:31:50,720 Speaker 1: Tyler Hill and Melissa Slaughter. It was executive produced by 609 00:31:50,760 --> 00:31:54,920 Speaker 1: me Cara Price and Kate Osborne for Kaleidoscope and Katria 610 00:31:55,040 --> 00:31:59,000 Speaker 1: Novelle for iHeart Podcast. The Engineer is Behit Fraser and 611 00:31:59,120 --> 00:32:02,840 Speaker 1: Jack Insley mixed this episode. Kyle Murdoch wrote our theme song. 612 00:32:03,320 --> 00:32:05,920 Speaker 1: Please do rate, review and reach out to us at 613 00:32:05,920 --> 00:32:08,480 Speaker 1: tech Stuff Podcast at gmail dot com. We love hearing 614 00:32:08,520 --> 00:32:08,800 Speaker 1: from you.