WEBVTT - Can AI Tutors Help Kids Learn? Khan Academy Thinks So 0:00:15.356 --> 0:00:22.916 Pushkin. There's a good chance you've heard of con Academy 0:00:23.316 --> 0:00:26.596 about how Salcon started out more than a decade ago 0:00:26.716 --> 0:00:30.836 tutoring his young cousins and then started posting simple tutoring 0:00:30.916 --> 0:00:34.116 videos on YouTube, like how to solve a quadratic equation, 0:00:34.476 --> 0:00:37.516 that kind of thing. He founded con Academy in two 0:00:37.596 --> 0:00:40.116 thousand and eight, and it grew into this big thing 0:00:40.356 --> 0:00:44.076 beloved by Bill Gates and the Ted talk set. Today, 0:00:44.436 --> 0:00:47.596 con Academy has tens of millions of monthly users and 0:00:47.716 --> 0:00:51.316 provides not just those YouTube videos, but thousands of practice 0:00:51.356 --> 0:00:55.396 questions across math and science and the humanities, along with 0:00:55.476 --> 0:00:59.036 software that monitors students progress. All this has been in 0:00:59.116 --> 0:01:02.876 service of a big idea using technology to give everyone 0:01:03.236 --> 0:01:06.316 the huge benefit that Salcon's cousins got from one on 0:01:06.316 --> 0:01:10.396 one tutoring. But in spite of Salza, the con Academy 0:01:10.596 --> 0:01:13.516 has not been able to fully deliver on that idea. 0:01:13.636 --> 0:01:16.556 The technology just hasn't been there to match what a 0:01:16.636 --> 0:01:20.436 human can do. But now sal thinks that may be 0:01:20.516 --> 0:01:24.556 about to change. Earlier this year, con Academy launched something 0:01:24.556 --> 0:01:28.036 they call Conmego, an AI tutor built on top of 0:01:28.156 --> 0:01:37.036 GPT four. I'm Jacob Goldstein, and this is what's your problem. 0:01:37.196 --> 0:01:40.236 My guest today is Sal Con. He's the founder and 0:01:40.356 --> 0:01:44.276 CEO of con Academy. Sal's problem is this, how do 0:01:44.316 --> 0:01:47.716 you use AI to bring students the benefit of working 0:01:47.756 --> 0:01:51.476 with a human tutor. Sal says his real work with 0:01:51.556 --> 0:01:54.116 AI started last summer when he heard from the co 0:01:54.156 --> 0:01:56.196 founders of the company that built. 0:01:55.876 --> 0:02:02.476 GPT About a year ago. Summer of twenty twenty two, 0:02:03.236 --> 0:02:06.036 I get an email from Sam Altman and Greg Brockman, 0:02:06.076 --> 0:02:09.556 the founders of open AI, who I knew, you know, 0:02:09.596 --> 0:02:11.916 I bumped into them at different events and things like that, 0:02:12.396 --> 0:02:14.756 and they said, hey, you know, we're working on our 0:02:14.756 --> 0:02:17.756 next model. If we would love to talk to you 0:02:17.756 --> 0:02:19.756 all about a few, you know, things that we might 0:02:19.796 --> 0:02:23.476 be able to collaborate on. I was skeptical that it 0:02:23.516 --> 0:02:27.476 would really have any implications for con Academy, but as 0:02:27.516 --> 0:02:29.996 a nerd, I said, Oh, this must be GPT four 0:02:30.356 --> 0:02:32.436 that they must be working on. And I have a 0:02:32.476 --> 0:02:34.876 lot of respect for Greg and Sam what they had 0:02:34.876 --> 0:02:37.836 already accomplished, so I was like, yeah, just I'm just curious. Yeah, 0:02:38.036 --> 0:02:41.396 let's meet is Actually at that point hadn't finished training 0:02:41.836 --> 0:02:44.956 what would eventually be GPT four, but they were about 0:02:44.956 --> 0:02:48.516 two weeks away from it, but they said, look, we 0:02:48.596 --> 0:02:52.196 think this is going to be the model, the generative 0:02:52.236 --> 0:02:55.396 AI that really opens people's minds to what's possible here. 0:02:56.436 --> 0:02:58.036 And because of that, it's going to be exciting and 0:02:58.036 --> 0:03:00.756 a little bit scary, and so we want to launch 0:03:01.116 --> 0:03:04.276 with some social positive use cases. The first one that 0:03:04.316 --> 0:03:07.436 came to our mind is education, and the first organization 0:03:07.476 --> 0:03:09.996 that came to our mind is Kin Academy, because y'all. 0:03:09.956 --> 0:03:12.236 Want they want you to be the warm, fuzzy, happy 0:03:12.236 --> 0:03:13.116 face of AI. 0:03:13.516 --> 0:03:15.796 Yeah, to see if we'd be interested to see and 0:03:16.036 --> 0:03:18.676 they want you know, they don't want to just be optics. 0:03:18.756 --> 0:03:21.956 I think they generally want AI to have warm, fuzzy applications. 0:03:22.076 --> 0:03:26.116 I believe, of course, the dark and sordid applications that. 0:03:26.556 --> 0:03:30.476 They clearly do, right, They clearly recognized the downside. 0:03:30.116 --> 0:03:33.436 Yeah, exactly. But the other reason they actually reached out 0:03:33.476 --> 0:03:36.436 to us and I connected the dots later, they said, 0:03:36.436 --> 0:03:37.756 you know, we really want it to be good at 0:03:37.756 --> 0:03:42.076 academic things like That's where GPT three it really had 0:03:42.156 --> 0:03:45.996 no solid handle on knowledge, and we think GPT four 0:03:46.036 --> 0:03:49.396 will and y'all have a lot of items we were 0:03:49.396 --> 0:03:51.076 just talking about how deep of an item bank we 0:03:51.116 --> 0:03:54.316 have it academy, across subjects and grades. And they were 0:03:54.356 --> 0:03:57.076 particularly interested in ap bio, which I later learned from 0:03:57.116 --> 0:04:00.356 Bill Gates. It was because when he saw GPT three 0:04:00.756 --> 0:04:03.556 he said he essentially told him, look, this is cute, guys, 0:04:03.836 --> 0:04:05.716 but I'll be impressed if this could pass an ap 0:04:05.836 --> 0:04:09.836 biology exam. So I think Greg and Sam literally said, Okay, 0:04:10.516 --> 0:04:13.796 you know, Bill Gates, that's a pretty good benchmark. Can 0:04:13.836 --> 0:04:15.996 this pass the ap bile exam? So I think that's 0:04:16.036 --> 0:04:17.756 the other thing. They said, hey, can we use some 0:04:17.796 --> 0:04:20.076 of your items to test to evaluate the model. 0:04:20.476 --> 0:04:24.116 So they wanted you have like whatever, thousands of questions 0:04:24.156 --> 0:04:27.476 about all these different subjects kind of test questions, test 0:04:27.636 --> 0:04:30.716 like questions about ap biology and calculus and lots of 0:04:30.756 --> 0:04:33.716 other subjects, and they wanted to train the model on 0:04:33.756 --> 0:04:35.916 your questions because you have the questions and the answers 0:04:35.956 --> 0:04:39.076 and exactly what you would want if you were training 0:04:39.436 --> 0:04:44.276 an AI model on sort of high school college level knowledge. 0:04:44.596 --> 0:04:46.676 Well, it's interesting out of the gate, they were actually 0:04:46.676 --> 0:04:49.076 more interested in just on the evaluation side. So once 0:04:49.116 --> 0:04:51.116 you produce a model, you want to see how good 0:04:51.116 --> 0:04:53.796 it is so you literally make it take questions and 0:04:53.836 --> 0:04:56.716 see how many gets right. So I was like, yeah, 0:04:56.756 --> 0:05:00.196 you know, maybe you know, I wasn't sure if this 0:05:00.276 --> 0:05:02.436 was really going to be something we could use, but 0:05:02.476 --> 0:05:04.036 I still wanted to see. I still wanted to see 0:05:04.036 --> 0:05:06.676 the demo when I was available. So two weeks later 0:05:06.676 --> 0:05:08.516 they said, oh, we finished training, we'd love to show 0:05:08.556 --> 0:05:11.436 you a demo. And I remember that meeting they put 0:05:11.516 --> 0:05:14.236 up and you know, this isn't that mind blowing the 0:05:14.276 --> 0:05:16.676 folks now because they've gotten used to this. But in 0:05:16.756 --> 0:05:20.756 summer of twenty twenty two, we were on a video 0:05:20.796 --> 0:05:23.836 conference and they on Greg's screen. He showed an ap 0:05:23.916 --> 0:05:26.036 bio question and he said, sah, what do you think 0:05:26.036 --> 0:05:27.916 the answer is? And I like read the question was 0:05:27.956 --> 0:05:31.596 about osmosis. I'm like, oh c. And then they asked 0:05:31.636 --> 0:05:33.676 the AI. I said yeah, see, I'm like, okay, that's 0:05:33.756 --> 0:05:36.196 kind of cool. And then I said, well, you know, 0:05:36.236 --> 0:05:39.476 ask it why is? The answer explained it very well, 0:05:39.516 --> 0:05:41.836 and that's when I was like, wow, that is interesting. 0:05:41.876 --> 0:05:43.796 I said, well, why aren't the other choices? The answer 0:05:44.316 --> 0:05:47.716 explained it very well. I'm like okay, yeah, this is getting. 0:05:47.836 --> 0:05:50.476 Kind of wow. Right, that's still like I know, it's 0:05:50.476 --> 0:05:53.516 not really cognition. Maybe it doesn't understand, but something is 0:05:53.556 --> 0:05:56.876 happening there, right, something something well. 0:05:56.876 --> 0:05:59.916 I think it's starting to make all of us start 0:05:59.916 --> 0:06:02.596 to parse words, like we start to realize that like 0:06:03.316 --> 0:06:08.716 sentience and cognition and intelligence are not necessarily the in fact, 0:06:08.716 --> 0:06:11.596 they aren't same thing, and we can separate these things. 0:06:11.596 --> 0:06:13.916 Something can be intelligent. And I think it's going to 0:06:13.996 --> 0:06:16.636 challenge our words, our semantics for some of these things. 0:06:16.796 --> 0:06:19.116 But then I said, hey, can it can write another 0:06:19.156 --> 0:06:20.636 question like this? And it did it, and I'm like 0:06:20.636 --> 0:06:22.876 that's a pretty good question. And then I said write 0:06:22.876 --> 0:06:25.636 ten more questions like this, and it did it. And 0:06:26.276 --> 0:06:27.396 you know, at the moment. 0:06:27.276 --> 0:06:28.996 At your moment where you were like, oh, no, AI 0:06:29.076 --> 0:06:31.596 is coming from my job, I kind. 0:06:31.356 --> 0:06:34.436 Of thought that a little bit. But then what really 0:06:34.556 --> 0:06:37.556 opened my mid They're like, oh, well, would you want 0:06:37.636 --> 0:06:39.956 access to it and to at least try it out 0:06:39.996 --> 0:06:42.396 see how you might be able to use it. And 0:06:42.476 --> 0:06:44.276 I was like, yeah, I want access. I want to 0:06:44.276 --> 0:06:47.356 try out this thing, and so myself and actually eventually 0:06:47.356 --> 0:06:49.756 our whole organization got under a non disclosure agreement with 0:06:49.796 --> 0:06:53.076 open ai back back in August, and we started testing 0:06:53.116 --> 0:06:55.036 it in that first weekend, and I remember it was 0:06:55.076 --> 0:06:55.476 myself in. 0:06:55.476 --> 0:06:57.076 The August twenty twenty two. 0:06:57.556 --> 0:07:03.116 Yes, myself, our Chief Learning Officer Christen, and our CTO 0:07:03.236 --> 0:07:05.916 Paul were the first people to have access to and 0:07:05.956 --> 0:07:08.116 I think we even got access to it before many 0:07:08.116 --> 0:07:11.476 folks that open Ai access to it, and we were 0:07:11.516 --> 0:07:13.836 just playing around with it, and we were also slacking 0:07:13.836 --> 0:07:16.156 with the open AI team. We're like, hey, you have 0:07:16.156 --> 0:07:17.716 any ideas. We're trying to get it to do this 0:07:17.876 --> 0:07:19.796 or we got it closed, but it's still and they 0:07:19.796 --> 0:07:21.516 were giving us some really good tips and by the 0:07:21.596 --> 0:07:24.676 end of that weekend we had it being able to 0:07:24.916 --> 0:07:29.556 take on personas and modeling you know, pretty good tutor behavior. 0:07:30.196 --> 0:07:32.196 And that's when it really when I started saying, Okay, 0:07:32.196 --> 0:07:35.316 this is a this is a game changer. This is 0:07:35.356 --> 0:07:39.276 science fiction has talked about artificially intelligent tutors forever, most 0:07:39.316 --> 0:07:43.436 famously Diamond Age Young Ladies. Illustrated primer Neil Stevenson wrote 0:07:43.476 --> 0:07:45.996 about this in the nineteen nineties about you know, an 0:07:46.036 --> 0:07:50.236 intelligent tutor being able to educate essentially. I mean, this 0:07:50.396 --> 0:07:52.916 book was set in the not too far future and 0:07:52.956 --> 0:07:55.956 it educates all these young girls in China living on 0:07:56.036 --> 0:07:58.316 barges were orphans and then they take over the world 0:07:58.436 --> 0:08:01.236 because this tutor was able to empower them so much so. 0:08:01.436 --> 0:08:06.116 Sidebar Neil Stevenson amazingly prescient, right like he's the like 0:08:06.796 --> 0:08:10.476 president guy for this, he's the president guy for for crypto, 0:08:10.836 --> 0:08:12.396 right like it's weird, right. 0:08:12.716 --> 0:08:14.796 Oh, he's pretty good. And he wrote, you know, diamond 0:08:14.836 --> 0:08:17.356 Age was nineteen ninety four, so yeah, you know, almost 0:08:17.356 --> 0:08:19.076 as an aside, it was a tablet app so he 0:08:19.116 --> 0:08:23.476 was prescient for tablet and mobile. But but that's when 0:08:23.516 --> 0:08:24.876 when I when I saw that it could take on 0:08:24.956 --> 0:08:29.316 personas and act as a tutor and not superficially but 0:08:29.356 --> 0:08:31.436 actually do some things that are I would view like 0:08:31.516 --> 0:08:34.396 actually quite thoughtful as a tutor, I said, Wow, we 0:08:34.556 --> 0:08:36.076 I think we're at the cusp of something here. And 0:08:36.076 --> 0:08:37.276 I didn't know if it was going to happen in 0:08:37.276 --> 0:08:39.916 like three months or three years, but I'm like, we've 0:08:39.996 --> 0:08:41.476 got we've got to work on this. 0:08:41.876 --> 0:08:44.476 Just to pause there and kind of go broader, Like 0:08:44.836 --> 0:08:48.436 the big idea behind con Academy the whole time, right 0:08:48.516 --> 0:08:51.556 since its inception whatever what more than a decade ago 0:08:51.636 --> 0:08:54.636 now is the power of a tutor right, that like 0:08:54.716 --> 0:08:58.876 a tutor is profoundly valuable, and that's sort of empirically clear, right. 0:09:00.076 --> 0:09:03.076 That's exactly right. It all got started back in two 0:09:03.116 --> 0:09:05.796 thousand and four with me tutoring cousins just on the side, 0:09:06.316 --> 0:09:09.596 one cousin Nada needed help, and then I started tutoring brothers. 0:09:09.676 --> 0:09:12.356 Wordspreads in my family, free tutoring is going on, and 0:09:12.836 --> 0:09:15.116 I saw it with my own cousins just on an 0:09:15.116 --> 0:09:17.956 anecdotal level, that everyone that I was tutoring, I was 0:09:17.996 --> 0:09:19.796 able to put in, you know, thirty forty minutes a 0:09:19.876 --> 0:09:22.116 day with all of them collectively, it was it was 0:09:22.236 --> 0:09:25.076 dramatically accelerating them. And so to a large degree, when 0:09:25.116 --> 0:09:29.076 I started making exercises and software for my cousins and 0:09:29.076 --> 0:09:32.236 then eventually making videos for my cousins that obviously many 0:09:32.236 --> 0:09:35.036 many more folks ended up using, I was always in 0:09:35.036 --> 0:09:37.876 the mindset of how can I help scale my tutoring, 0:09:38.236 --> 0:09:40.996 How can I start making that tutoring a little bit 0:09:40.996 --> 0:09:43.636 more self service so that my cousins and eventually other 0:09:43.676 --> 0:09:46.476 people could help themselves. So that's been the journey of 0:09:46.516 --> 0:09:48.916 coin Academy for the last I mean it's been almost 0:09:49.836 --> 0:09:52.876 it's been nineteen years. Since I started tutoring my cousins, 0:09:53.636 --> 0:09:57.916 and so this technology held the potential to take it 0:09:57.956 --> 0:09:58.596 that much further. 0:09:58.796 --> 0:10:01.436 Yeah, so you're playing with this thing. It's very early. 0:10:01.876 --> 0:10:04.436 The world doesn't even know about GPT for yet. What 0:10:04.436 --> 0:10:05.916 what are you doing? What are you working on? Within 0:10:05.996 --> 0:10:06.636 con Academy. 0:10:06.676 --> 0:10:08.876 At this point we started to see, Wow, this could 0:10:08.916 --> 0:10:10.916 just could really work, This could be really powerful, and 0:10:10.916 --> 0:10:14.076 we started figuring out ways to minimize some of the 0:10:14.156 --> 0:10:17.996 rough spots of Jenai, like hallucinations where it can make 0:10:18.076 --> 0:10:21.636 up things like it's making math errors, which is obviously 0:10:21.676 --> 0:10:23.756 a problem if we want it to be a math tutor. 0:10:24.436 --> 0:10:26.996 So we started going working through that. At the same time, 0:10:27.036 --> 0:10:30.236 we started having some pretty intense debates inside of our organization. 0:10:31.356 --> 0:10:33.836 Roughly speaking, half the organization was like, this is the 0:10:33.836 --> 0:10:38.356 most important technological advancement ever or at least in our lifetimes. 0:10:38.636 --> 0:10:40.476 We've got to go all in on this, like it's 0:10:40.796 --> 0:10:42.796 our duty. And then the other half of the team, 0:10:43.116 --> 0:10:46.196 not that they disagreed, but they said, look, kind academy 0:10:46.236 --> 0:10:49.636 stands for a lot it and we're there to help students, 0:10:49.676 --> 0:10:52.636 and if we go out there with something that's either 0:10:52.716 --> 0:10:56.916 not well baked or it leads to something that's suspicious 0:10:57.036 --> 0:11:00.316 or shady or just scary for folks. Because JENAI could 0:11:00.316 --> 0:11:02.836 be scary, it's going to be bad for us. So 0:11:02.876 --> 0:11:05.916 we started having those debates. But within a couple of months, 0:11:06.116 --> 0:11:10.356 as we were just kept prototyping, well, two things happened. 0:11:10.636 --> 0:11:12.596 One is I think a consensus. I don't know if 0:11:12.636 --> 0:11:15.756 it was a consensus, but I started to drive alignment 0:11:15.916 --> 0:11:19.036 around the idea like, these these fears and these risks 0:11:19.036 --> 0:11:21.676 that people are articulating are real. We should not ignore them. 0:11:22.236 --> 0:11:26.316 But there's ways to turn them into features that actually 0:11:26.636 --> 0:11:30.156 not only mitigate risks, but they actually can be enhancements. So, 0:11:30.196 --> 0:11:33.516 for example, we said, well, what if on what eventually 0:11:33.516 --> 0:11:37.116 would be konmego we didn't call it. Then, then all 0:11:37.156 --> 0:11:40.116 the sessions of a student who's under eighteen are recorded 0:11:40.116 --> 0:11:42.956 and accessible by parents or teachers. What if we have 0:11:43.036 --> 0:11:45.676 a second AI that moderates the conversations, and if the 0:11:45.676 --> 0:11:50.156 conversations go anywhere shady can actively notify the parent or teacher. 0:11:50.516 --> 0:11:53.036 What if our AI doesn't, It doesn't just give you 0:11:53.076 --> 0:11:55.516 the answer. Even before you know this was before chat 0:11:55.556 --> 0:11:57.436 GPT came out, we knew that this could be used 0:11:57.436 --> 0:12:00.076 for cheating that the raw technology could. We said, we're 0:12:00.076 --> 0:12:02.676 not going to do that, but it can, it can 0:12:02.716 --> 0:12:05.356 support you. So we started thinking through all of these 0:12:05.676 --> 0:12:08.236 what you know, we aren't going to make We aren't 0:12:08.276 --> 0:12:11.796 going to use information to train the AI, at least 0:12:11.796 --> 0:12:14.036 in this stage where people aren't sure how it might go. 0:12:14.356 --> 0:12:17.276 We weren't going to let any personally identifiable information go 0:12:17.356 --> 0:12:20.956 between the student and the artificially intelligent model. And so 0:12:20.956 --> 0:12:22.316 in a lot of ways, it's like we're gonna put 0:12:22.316 --> 0:12:24.916 more safeguards on jen AI than frankly there exist on 0:12:24.956 --> 0:12:28.316 the Internet, and when kids are just randomly on the Internet. 0:12:28.036 --> 0:12:30.116 That's a pretty low bar. You definitely want to be 0:12:30.196 --> 0:12:32.436 higher than random, let's be honest. 0:12:32.636 --> 0:12:34.316 Yeah, we made a much higher because we knew this 0:12:34.396 --> 0:12:36.596 was gonna kind of you know, people were going to 0:12:36.636 --> 0:12:37.516 have mixed feelings about it. 0:12:37.516 --> 0:12:39.036 A lot of ways it could go wrong. There's a 0:12:39.036 --> 0:12:39.756 lot of ways. That's right. 0:12:40.276 --> 0:12:42.756 That's right. Then, I would say the other big thing 0:12:42.796 --> 0:12:46.076 that happened end of November, chat GPT comes out, and 0:12:46.676 --> 0:12:51.116 that now that captured everyone's imagination and we all remember 0:12:51.196 --> 0:12:54.076 those first few weeks and months where people were you know, 0:12:54.076 --> 0:12:58.636 everyone was getting on chat GPT and taking screenshots and 0:12:58.636 --> 0:13:00.556 putting it on social media of what it was doing. 0:13:00.916 --> 0:13:02.876 And it was doing some amazing things. It was doing 0:13:02.956 --> 0:13:05.516 some very imperfect things too, the hallucinations, the math, theres, 0:13:05.556 --> 0:13:09.556 et cetera. I was worried initially because the narrative and 0:13:09.716 --> 0:13:13.516 education immediately became this thing as error prone, this thing 0:13:13.596 --> 0:13:17.556 can be used to cheat. It's the end of term papers, homework, 0:13:17.596 --> 0:13:21.996 et cetera. School district started banning chat GPT. I was like, 0:13:22.036 --> 0:13:24.156 oh no, we're working so much on this, and GPT 0:13:24.236 --> 0:13:26.356 four is so much better, and what was eventually going 0:13:26.396 --> 0:13:29.356 to become conmego was so much better, Like I hope 0:13:29.356 --> 0:13:32.676 the baby doesn't get thrown out with the bathwater. In hindsight, 0:13:33.116 --> 0:13:36.636 that was a good thing because it made us even 0:13:36.676 --> 0:13:39.596 internally say, look, the genie's out of the bottle. Now 0:13:39.636 --> 0:13:41.636 it's how we use the genie, Like we're going to 0:13:41.676 --> 0:13:43.716 be a force for good. Hopefully we have to work 0:13:43.756 --> 0:13:46.236 feverishly to show that it can be used well. And 0:13:46.276 --> 0:13:48.636 then I think by the time GPT four came out 0:13:48.676 --> 0:13:52.436 and Conmego came out with it in March, the education 0:13:52.636 --> 0:13:56.516 system and frankly society had a chance to process it 0:13:56.556 --> 0:13:59.356 and they had come around like, well, the technology isn't bad. 0:13:59.396 --> 0:14:01.916 We just need reasonable guardrails and we just need tools 0:14:01.916 --> 0:14:04.596 that are built for the actual use cases. And then 0:14:04.596 --> 0:14:06.396 we were able to show up with like you mean 0:14:06.516 --> 0:14:08.716 like this, you mean like con migo, and then most 0:14:08.756 --> 0:14:11.796 people and yes, exactly like that. And now we're seeing, 0:14:12.196 --> 0:14:15.756 I mean honestly, more schools and districts and parents want 0:14:15.796 --> 0:14:18.076 it for their students and their children than we can 0:14:18.116 --> 0:14:18.836 currently handle. 0:14:19.116 --> 0:14:23.916 So you mentioned some of the guardrails you built into it, 0:14:24.036 --> 0:14:26.956 you know, not accepting personal information, having a second AI 0:14:27.116 --> 0:14:32.036 monitoring for anything shady, not just giving answers. There's another 0:14:32.156 --> 0:14:35.996 piece of the process of building con Migo that I've 0:14:36.156 --> 0:14:39.316 heard you discuss elsewhere that is really interesting to me. 0:14:39.476 --> 0:14:47.036 And that's that's about having the AI sort of think 0:14:47.076 --> 0:14:52.036 about its answers, think about its responses. Right, I probably 0:14:52.076 --> 0:14:54.676 haven't articulated that well, but you can so tell me 0:14:54.716 --> 0:14:55.756 about that piece of it. 0:14:56.836 --> 0:15:00.116 As you can imagine, one of the hardest things to 0:15:00.196 --> 0:15:03.676 resolve was that even GPT four, which is dramatically better 0:15:03.716 --> 0:15:06.836 than GPT three point five at math, it was making 0:15:06.996 --> 0:15:10.796 an uncomfortable number of math errors, especially when it came 0:15:10.836 --> 0:15:14.876 to being in the tutoring use case where let's say 0:15:15.116 --> 0:15:18.116 there's an algebra problem and I the student take a 0:15:18.116 --> 0:15:21.036 step and maybe I distribute it, distribut use the distributive 0:15:21.076 --> 0:15:24.916 property and correctly will it recognize it? Will it not? 0:15:25.236 --> 0:15:27.916 How will it provide that feedback to the student? And 0:15:27.956 --> 0:15:30.756 out the box, it wasn't doing it that well. It 0:15:30.796 --> 0:15:36.076 was making mistakes a lot, unacceptably often. Then we had 0:15:36.076 --> 0:15:36.956 a it's. 0:15:36.796 --> 0:15:39.436 Really bad if the tutor is getting the problem wrong. 0:15:40.956 --> 0:15:45.236 No, tutor, Yeah, that's not acceptable. And then an open 0:15:45.276 --> 0:15:49.356 AI researcher gave us an idea, which is, instead of 0:15:49.516 --> 0:15:54.356 just having the AI respond immediately to the student, instead, 0:15:54.716 --> 0:15:57.676 what if you were to have the AI essentially on 0:15:57.756 --> 0:16:00.236 its own, not show this part to the student, but 0:16:00.436 --> 0:16:03.556 generate what it thinks could have been reasonable responses for 0:16:03.636 --> 0:16:09.316 the student HU and then use that plus the conversation 0:16:09.356 --> 0:16:11.956 with the student to then respond to the student. 0:16:13.076 --> 0:16:15.956 Yeah, and we did this thingly next level, right, like 0:16:15.996 --> 0:16:19.676 a teacher would always be doing that, but there's no 0:16:19.716 --> 0:16:22.916 reason to think that just a raw large language model 0:16:22.956 --> 0:16:23.636 would be right. 0:16:24.356 --> 0:16:27.276 That's right, But you know what's interesting about this. We 0:16:27.676 --> 0:16:30.156 did it and it dramatically improved the math. And then 0:16:30.196 --> 0:16:31.836 we started tweaking it more and more, and we got 0:16:31.836 --> 0:16:33.716 it better and better. Now we do a bunch of 0:16:33.996 --> 0:16:36.516 fancy stuff along along those lines to get it a 0:16:36.556 --> 0:16:39.756 lot better. But then we realize, to your point, that's 0:16:39.836 --> 0:16:42.556 exactly what a teacher would do. So that's what con 0:16:42.596 --> 0:16:44.596 Migo does. It works on it on its own and 0:16:44.676 --> 0:16:47.676 its own scratch paper, so to speak. Compares the student's 0:16:47.756 --> 0:16:51.476 response if they got something different. Con Migo doesn't assume 0:16:51.516 --> 0:16:54.716 the student is wrong. It says, because con Migo can 0:16:54.756 --> 0:16:57.396 sometimes be wrong even on its own work, It says, hey, 0:16:57.436 --> 0:17:00.476 I got something different, can you explain your reasoning? And 0:17:00.516 --> 0:17:03.396 then when the student explains the reasoning, that's really good 0:17:03.436 --> 0:17:06.276 for large language models. It's actually able to understand that. 0:17:06.676 --> 0:17:09.116 And what's interesting, not only is that very pedagogically good, 0:17:09.876 --> 0:17:12.316 we've been getting a strong a lot of feedback from 0:17:12.316 --> 0:17:15.316 students that they really appreciate that type of an interaction. 0:17:15.356 --> 0:17:19.356 It eerily feels human that this thing you were so 0:17:19.476 --> 0:17:21.876 used to computers being so perfect, and it's like you're wrong, 0:17:21.996 --> 0:17:24.396 you're right here, you know, that's what a robot would do, 0:17:24.596 --> 0:17:26.316 but no, this is what a tutor would do. It's like, hey, 0:17:26.316 --> 0:17:27.916 I didn't get the same thing. Let's work through it together. 0:17:27.956 --> 0:17:29.436 Let's see who made the mistake. 0:17:32.196 --> 0:17:36.276 In a minute, how conmego compares to GPT today and 0:17:36.436 --> 0:17:38.436 what it might be like a few years from now. 0:17:46.036 --> 0:17:48.076 When I was getting ready for this interview, I played 0:17:48.076 --> 0:17:51.116 around with con Migo and chat GPT. I put them 0:17:51.196 --> 0:17:53.676 side by side in different tabs in my browser, and 0:17:53.716 --> 0:17:57.156 I asked each one a basic calculus question, how do 0:17:57.196 --> 0:17:59.876 you take a derivative? This is a thing that I 0:17:59.916 --> 0:18:02.156 knew how to do a long time ago, but I 0:18:02.196 --> 0:18:05.036 forgot also a long time ago, so it seemed like 0:18:05.076 --> 0:18:08.476 a good test, and it was striking how different the 0:18:08.556 --> 0:18:13.236 response is were GVT gave me something like the Wikipedia 0:18:13.396 --> 0:18:16.356 entry about derivatives, a bunch of text with some rules, 0:18:16.596 --> 0:18:20.276 some equations, and then I tried it on conmego, and 0:18:20.396 --> 0:18:22.876 first it asked me what I knew, and I said, 0:18:22.876 --> 0:18:26.756 I knew algebra. And then it explained one rule, a 0:18:26.836 --> 0:18:29.796 sort of first rule, the power rule for finding derivatives, 0:18:29.836 --> 0:18:31.996 and it gave me a problem to try, and I 0:18:32.036 --> 0:18:34.236 got that problem wrong, and it sort of asked me 0:18:34.716 --> 0:18:36.596 what I was thinking, and I tried it again and 0:18:36.636 --> 0:18:39.316 I got it right. And then Konnigo asked if I 0:18:39.356 --> 0:18:41.596 wanted to try another problem, and I said yes, and 0:18:41.636 --> 0:18:44.316 it gave me another practice problem. So clearly this is 0:18:44.316 --> 0:18:48.436 a very different experience than the plain vanilla chat GPT. 0:18:49.156 --> 0:18:51.196 But it was still a little bit awkward and a 0:18:51.196 --> 0:18:53.636 little bit hard to follow. And when I closed the 0:18:53.676 --> 0:18:57.076 tab and came back, it didn't remember our previous conversation. 0:18:57.356 --> 0:19:00.916 So overall, very good, but not quite there yet. I 0:19:00.956 --> 0:19:03.596 asked sal if that seemed about right to him. 0:19:03.876 --> 0:19:05.556 Yeah, I think that's pretty accurate. I think where it 0:19:05.596 --> 0:19:09.116 is today, it's like if you really wanted to learn 0:19:09.236 --> 0:19:11.916 how to take a derivative, or if you're learning calculus 0:19:11.956 --> 0:19:13.876 for the first time, I would say, go to the 0:19:13.876 --> 0:19:17.396 calculus course on kon Academy, start watching some of those videos, 0:19:17.476 --> 0:19:19.596 do those practice And what con migo is going to 0:19:19.596 --> 0:19:22.996 be really good at is if those videos, or the 0:19:23.036 --> 0:19:25.676 practice problems, or the articles we have or the hints 0:19:25.676 --> 0:19:30.596 we provide, there's still some itchy conceptual thing you're not getting. 0:19:31.476 --> 0:19:35.516 Con Migo's really good at trying to unlock that one 0:19:35.676 --> 0:19:36.796 conceptual thing. 0:19:37.236 --> 0:19:39.516 Like what I think really kind of a narrow problem, 0:19:39.556 --> 0:19:41.556 like you're almost there, but you just need like a 0:19:41.596 --> 0:19:45.036 little one more explanation or a different kind of explanation 0:19:45.156 --> 0:19:45.996 or something. 0:19:46.156 --> 0:19:49.316 Exactly, or there's just some conceptual dimension that maybe the 0:19:49.356 --> 0:19:52.316 video didn't address that you're curious about or you want 0:19:52.316 --> 0:19:54.996 to connect it. You're learning about entropy and chemistry, and 0:19:54.996 --> 0:19:57.356 you're like, I've heard this word in computer science, how 0:19:57.356 --> 0:20:00.156 are they similar? Con Migo is great for that. I 0:20:00.156 --> 0:20:04.876 don't think conmego by itself is a place where you 0:20:04.916 --> 0:20:08.396 would just say, start being my calculus tutor for the 0:20:08.436 --> 0:20:11.316 next year year and you're gonna work through calculus with me. 0:20:11.716 --> 0:20:14.716 So that's that's where you are today. Where do you 0:20:14.716 --> 0:20:18.436 think you're going to be in a year or five years? 0:20:18.636 --> 0:20:22.356 Yeah, one year, I'm quite confident we're going to have 0:20:22.636 --> 0:20:25.436 I mean it might be six months memory. So kind 0:20:25.436 --> 0:20:26.796 of migo is going to be able to know about 0:20:26.796 --> 0:20:29.756 previous conversations. And memory isn't just even about that, it's 0:20:29.756 --> 0:20:31.796 also about being able to report back to teachers, so 0:20:31.836 --> 0:20:33.956 the teachers can say, Hey, con Migo, what have you 0:20:34.076 --> 0:20:36.716 been working on with my students? Have you noticed any 0:20:36.756 --> 0:20:40.076