1 00:00:02,819 --> 00:00:05,050 Speaker 1: You're listening to a CNA podcast. 2 00:00:06,679 --> 00:00:09,720 Speaker 1: So hey everyone, welcome to another episode of Deep Dive 3 00:00:09,720 --> 00:00:13,000 Speaker 1: with Oelli and myself Steven. today we're talking about 4 00:00:13,430 --> 00:00:15,869 Speaker 1: AI or rather the use of AI when it comes 5 00:00:15,869 --> 00:00:18,120 Speaker 1: to doing your homework. Are you an AI fan? Do 6 00:00:18,120 --> 00:00:21,299 Speaker 1: you use it for work, not homework, but work? 7 00:00:21,510 --> 00:00:24,709 Speaker 2: OK, who doesn't use it nowadays unless you are like 8 00:00:24,709 --> 00:00:27,750 Speaker 2: living under a rock, but everyone uses, you know, large 9 00:00:27,750 --> 00:00:31,750 Speaker 2: language models to a certain extent, right, to improve writing. 10 00:00:31,790 --> 00:00:33,299 Speaker 2: I mean, in our line of work, we use it 11 00:00:33,299 --> 00:00:37,939 Speaker 2: to refine the original sentences that I came up with. 12 00:00:37,990 --> 00:00:38,189 Speaker 2: I 13 00:00:38,189 --> 00:00:42,068 Speaker 1: like I basically to learn and to work. 14 00:00:42,303 --> 00:00:44,752 Speaker 1: Efficiently. It helps me do what I do. 15 00:00:45,482 --> 00:00:48,713 Speaker 2: Yeah. So we can imagine a lot of students from 16 00:00:48,713 --> 00:00:51,202 Speaker 2: the Higher Institutes of learning, they're also doing that and 17 00:00:51,202 --> 00:00:53,842 Speaker 2: as a matter of fact, recently, right, there was this 18 00:00:53,842 --> 00:00:56,283 Speaker 2: issue that blew up in the news. It's when NTU 19 00:00:56,283 --> 00:01:00,092 Speaker 2: accused 3 students from the School of Social Sciences of 20 00:01:00,402 --> 00:01:03,923 Speaker 2: academic fraud. It's a very, very harsh term to think 21 00:01:03,923 --> 00:01:06,782 Speaker 2: about it, saying that they basically use generative AI tools 22 00:01:06,782 --> 00:01:10,962 Speaker 2: in their assignments and were given zero marks for that. 23 00:01:11,066 --> 00:01:14,615 Speaker 1: Yeah, 0. That's a big blow. Of course, the students 24 00:01:14,615 --> 00:01:17,184 Speaker 1: are upset with that and they're fighting back. They're asking 25 00:01:17,185 --> 00:01:20,095 Speaker 1: the university to justify why they're doing that, uh, you know, 26 00:01:20,176 --> 00:01:22,816 Speaker 1: and trying to understand just, well, where do you draw 27 00:01:22,816 --> 00:01:24,875 Speaker 1: the line? When is it OK and when is it 28 00:01:24,875 --> 00:01:27,896 Speaker 1: not OK, especially when it comes to students who are 29 00:01:27,896 --> 00:01:30,606 Speaker 1: still in school. And today we have some people who 30 00:01:30,606 --> 00:01:34,095 Speaker 1: will help us understand that issue better. So let's welcome 31 00:01:34,096 --> 00:01:37,574 Speaker 1: Associate Professor Ben Leong. He's the director of AI Center 32 00:01:37,575 --> 00:01:39,666 Speaker 1: for Educational Technologies at NUS. 33 00:01:40,510 --> 00:01:43,180 Speaker 1: Hello, uh, it's a pleasure to be here and I'm 34 00:01:43,180 --> 00:01:46,018 Speaker 1: happy to, to, you know, discuss this issue, which is 35 00:01:46,019 --> 00:01:49,139 Speaker 1: quite close to heart. Jeremy Su, he's a co-founder of 36 00:01:49,139 --> 00:01:52,370 Speaker 1: Next AI and graduated from Neon Pay just last year. 37 00:01:52,559 --> 00:01:55,459 Speaker 2: Yes, happy to be here. Thanks for having me. I 38 00:01:55,459 --> 00:01:56,419 Speaker 2: think this is gonna be really interesting. 39 00:01:56,889 --> 00:01:59,279 Speaker 2: So OK, let's just start off with a very big 40 00:01:59,279 --> 00:02:02,919 Speaker 2: picture question, right? Help us understand how is AI being 41 00:02:02,919 --> 00:02:06,239 Speaker 2: used both by teachers and schools. So Jeremy from a 42 00:02:06,239 --> 00:02:08,320 Speaker 2: student's point of view because you just graduated, right? So 43 00:02:08,320 --> 00:02:10,359 Speaker 2: when it comes to the students, how they actually use it, 44 00:02:10,399 --> 00:02:12,478 Speaker 2: they use it for quite a wide range of things, right? 45 00:02:12,758 --> 00:02:15,880 Speaker 2: There is the obvious writing my essays, doing my homework, 46 00:02:15,960 --> 00:02:18,059 Speaker 2: that's a very, very standard use case. We've had it 47 00:02:18,059 --> 00:02:21,149 Speaker 2: since years ago, even before Chad GPT because we have 48 00:02:21,203 --> 00:02:24,792 Speaker 2: For example, GBT 3 original models 2, which were all 49 00:02:24,792 --> 00:02:28,702 Speaker 2: pretty good autocomplete models, these were used by different students to, 50 00:02:28,712 --> 00:02:31,703 Speaker 2: for example, write their essays or kind of refine certain sentences. 51 00:02:31,912 --> 00:02:33,513 Speaker 2: And then right now people are using it for more 52 00:02:33,513 --> 00:02:36,272 Speaker 2: creative things, so things like research, they're able to kind 53 00:02:36,272 --> 00:02:39,012 Speaker 2: of trigger an AI, have it go online, search for 54 00:02:39,013 --> 00:02:42,063 Speaker 2: a bunch of different information, uh, pull it together, synthesize it, 55 00:02:42,072 --> 00:02:44,432 Speaker 2: and then eventually create some sort of report and so 56 00:02:44,431 --> 00:02:45,431 Speaker 2: they don't need to do like. 57 00:02:45,785 --> 00:02:48,186 Speaker 2: work anymore, right? That's a big part. I think specifically 58 00:02:48,186 --> 00:02:50,305 Speaker 2: for me, what I've seen is because more from the 59 00:02:50,305 --> 00:02:53,065 Speaker 2: creative side. So in, for example, media school, people are 60 00:02:53,065 --> 00:02:56,285 Speaker 2: using it in art, people are using it to actually edit. 61 00:02:56,505 --> 00:02:58,065 Speaker 2: So people are using it in many different ways which 62 00:02:58,065 --> 00:03:00,865 Speaker 2: are not conventionally, I think what most people think of 63 00:03:00,865 --> 00:03:02,984 Speaker 2: when they think of AI in the academic sense. It 64 00:03:02,985 --> 00:03:03,546 Speaker 2: just reports and 65 00:03:03,546 --> 00:03:03,906 Speaker 2: essays. 66 00:03:04,106 --> 00:03:06,584 Speaker 1: Ben, I'll ask you, so at the university again, you know, 67 00:03:06,656 --> 00:03:09,225 Speaker 1: you are facing this, many students will also be using 68 00:03:09,225 --> 00:03:12,136 Speaker 1: AI and AI is of course a lot better now. 69 00:03:12,225 --> 00:03:14,255 Speaker 1: It can literally write the essay for you. 70 00:03:14,669 --> 00:03:17,089 Speaker 1: So at which point do you say, OK, that's all right, 71 00:03:17,258 --> 00:03:19,698 Speaker 1: at which point do you say no. OK, so I 72 00:03:19,699 --> 00:03:23,739 Speaker 1: think it's it's a complex question, right, really depends on 73 00:03:23,740 --> 00:03:26,179 Speaker 1: the discipline that you know that we're dealing with. So 74 00:03:26,179 --> 00:03:29,418 Speaker 1: I teach computer science and mostly programming. So in programming, 75 00:03:29,538 --> 00:03:31,460 Speaker 1: you know, you write code and I think my colleagues 76 00:03:31,460 --> 00:03:33,859 Speaker 1: who have students write essays, they have a different kind 77 00:03:33,860 --> 00:03:36,339 Speaker 1: of requirement. It may be harder for us to detect 78 00:03:36,339 --> 00:03:37,649 Speaker 1: plagiarism for coding. 79 00:03:38,132 --> 00:03:40,432 Speaker 1: Because the fact is that AI is really good at 80 00:03:40,432 --> 00:03:42,822 Speaker 1: generating good code and the code looks kind of like 81 00:03:44,292 --> 00:03:47,373 Speaker 1: standard to be fair, right? Pages is not a new thing. 82 00:03:47,473 --> 00:03:48,352 Speaker 2: So universities 83 00:03:48,352 --> 00:03:51,182 Speaker 2: that use these AI detectors, right, like turn it in 84 00:03:51,182 --> 00:03:53,962 Speaker 2: and all that, does that detect like the code? So, 85 00:03:55,152 --> 00:03:57,832 Speaker 1: so the truth is that we have had very good 86 00:03:57,832 --> 00:04:01,513 Speaker 1: AI code detectors in the in the past before AI, right, and. 87 00:04:01,576 --> 00:04:03,416 Speaker 1: The way I was going to explain, ever since I 88 00:04:03,416 --> 00:04:07,436 Speaker 1: start teaching, we have caught students cheating by copying their friends' code, 89 00:04:07,925 --> 00:04:10,085 Speaker 1: and generally what happens is we catch them because they 90 00:04:10,085 --> 00:04:12,686 Speaker 1: write bad code, not because they write good code. I 91 00:04:12,686 --> 00:04:14,445 Speaker 1: mean write good code, you have two clever people writing 92 00:04:14,445 --> 00:04:16,126 Speaker 1: good code and look the same. It's actually there's some 93 00:04:16,126 --> 00:04:18,565 Speaker 1: standard ways of writing code. So the general way to 94 00:04:18,565 --> 00:04:21,485 Speaker 1: catch people writing copying code is that they have a 95 00:04:21,485 --> 00:04:24,005 Speaker 1: friend who writes bad code and wrong code, and you 96 00:04:24,005 --> 00:04:26,605 Speaker 1: have make the same mistake, right? And the best part is. 97 00:04:26,928 --> 00:04:29,470 Speaker 1: It came from somewhere that you don't know where it 98 00:04:29,470 --> 00:04:31,469 Speaker 1: came from, right? So, so there's no way to kind 99 00:04:31,470 --> 00:04:34,350 Speaker 1: of prove that, in fact, no easy way to prove 100 00:04:34,350 --> 00:04:37,309 Speaker 1: because code is 2 + 2 is 4. So when 101 00:04:37,309 --> 00:04:39,299 Speaker 1: you write good code, good code is good code, as 102 00:04:39,299 --> 00:04:42,269 Speaker 1: in it should all look pretty much the same, right? Correct. 103 00:04:42,309 --> 00:04:43,790 Speaker 1: I mean, so there are many ways to do the 104 00:04:43,790 --> 00:04:45,849 Speaker 1: same thing, but there is, you're right, there's a certain 105 00:04:45,850 --> 00:04:50,320 Speaker 1: style that good code, right? So so my question then becomes, 106 00:04:50,709 --> 00:04:51,510 Speaker 1: what's wrong with that? 107 00:04:52,040 --> 00:04:54,609 Speaker 1: Because they're getting you the the result that they need 108 00:04:54,609 --> 00:04:56,459 Speaker 1: and in the future when they work, why would they 109 00:04:56,459 --> 00:04:58,178 Speaker 1: bother going through it themselves when they can use the 110 00:04:58,178 --> 00:05:00,320 Speaker 1: air to write it? That's an excellent question, right? So, 111 00:05:00,350 --> 00:05:02,260 Speaker 1: so then you need to understand what we're doing in 112 00:05:02,260 --> 00:05:04,659 Speaker 1: school here, right? There are two things that we are 113 00:05:04,660 --> 00:05:05,140 Speaker 1: trying to do. 114 00:05:05,570 --> 00:05:07,849 Speaker 1: Number one is that we're trying to help the students 115 00:05:07,850 --> 00:05:11,640 Speaker 1: learn something, OK, and let me blunt, I mean, homework 116 00:05:11,640 --> 00:05:13,369 Speaker 1: is there for a reason, right? I mean, if I 117 00:05:13,369 --> 00:05:16,169 Speaker 1: give you homework, frankly it's, it's going to cost us work, right, 118 00:05:16,250 --> 00:05:17,729 Speaker 1: because someone has to grade it. Of course you can 119 00:05:17,730 --> 00:05:20,049 Speaker 1: outsource to AI, but frankly, you know, if we don't 120 00:05:20,049 --> 00:05:21,928 Speaker 1: give issue homework, then we do less. But the homework 121 00:05:21,928 --> 00:05:24,329 Speaker 1: is there to the homework. No, no, no, no, there 122 00:05:24,329 --> 00:05:27,329 Speaker 1: are these things called formative assessments and there are these 123 00:05:27,329 --> 00:05:30,529 Speaker 1: things called summative assessments. So formative assessments is to help 124 00:05:30,529 --> 00:05:33,339 Speaker 1: you learn something. So you're going through the struggle to 125 00:05:33,339 --> 00:05:35,130 Speaker 1: learn the stuff, right? 126 00:05:35,225 --> 00:05:38,054 Speaker 1: Now, so the problem with AI in this regard is 127 00:05:38,053 --> 00:05:41,424 Speaker 1: that it allows the students to kind of like avoid 128 00:05:41,424 --> 00:05:44,225 Speaker 1: this this struggle, right? And they avoid learning, right? I mean, so, 129 00:05:44,234 --> 00:05:46,375 Speaker 1: so frankly, if I give you a question, they get 130 00:05:46,375 --> 00:05:49,445 Speaker 1: the result they get the results and so they're not, OK, 131 00:05:49,494 --> 00:05:51,875 Speaker 1: at some level they are short change themselves because their 132 00:05:51,875 --> 00:05:55,334 Speaker 1: parents paid good money for them to come to. OK, 133 00:05:55,415 --> 00:05:57,454 Speaker 1: so that's number one. Number 2, there's also a very 134 00:05:57,454 --> 00:06:00,054 Speaker 1: important uh aspect of this assessment thing which is that 135 00:06:00,053 --> 00:06:02,695 Speaker 1: there's this thing called a submitive assessment, but universities have 136 00:06:02,695 --> 00:06:04,894 Speaker 1: a role to play in the economy, right? 137 00:06:05,220 --> 00:06:09,380 Speaker 1: Because ultimately these kids graduate and they get jobs. OK, 138 00:06:09,559 --> 00:06:11,480 Speaker 1: so let me ask a question, right? If there's no 139 00:06:11,480 --> 00:06:14,839 Speaker 1: reason for us to give the first class honors, and 140 00:06:14,839 --> 00:06:17,269 Speaker 1: second class and so forth, we actually have a role 141 00:06:17,269 --> 00:06:20,558 Speaker 1: in society, right, to sort of like grade the students, right, 142 00:06:20,640 --> 00:06:22,920 Speaker 1: so that employers have an easier time with the hiring. 143 00:06:23,290 --> 00:06:26,709 Speaker 1: Right, so if everybody gets an A, then it's debatable 144 00:06:26,709 --> 00:06:29,549 Speaker 1: whether or not this is still a kind of function 145 00:06:29,549 --> 00:06:31,779 Speaker 1: of the university, but I'm certain that if we give 146 00:06:31,779 --> 00:06:34,000 Speaker 1: every student A's, I think that would be and there's 147 00:06:34,000 --> 00:06:38,010 Speaker 1: somehow you see, OK, the whole process as you mentioned 148 00:06:38,010 --> 00:06:40,450 Speaker 1: is for the student to go through the learning journey 149 00:06:40,450 --> 00:06:43,320 Speaker 1: so that when they face other situations in the real 150 00:06:43,320 --> 00:06:45,700 Speaker 1: world that AI maybe hasn't encountered yet. 151 00:06:46,178 --> 00:06:48,059 Speaker 1: Then they can help solve it. So you're teaching them 152 00:06:48,059 --> 00:06:50,390 Speaker 1: how to think, basically, right? But now with the AI 153 00:06:50,390 --> 00:06:52,219 Speaker 1: doing everything for them, they miss out on this. 154 00:06:52,660 --> 00:06:53,059 Speaker 2: I've got a 155 00:06:53,059 --> 00:06:55,049 Speaker 2: question as well that a lot of people are turning 156 00:06:55,049 --> 00:06:56,529 Speaker 2: to your large language models 157 00:06:56,529 --> 00:06:56,540 Speaker 1: right 158 00:06:56,540 --> 00:06:59,899 Speaker 2: to learn stuff. Let's say you go into like GPT 159 00:06:59,899 --> 00:07:00,980 Speaker 2: or Gemini and whatnot. 160 00:07:01,738 --> 00:07:04,219 Speaker 2: As you are learning stuff, right, it picks up your 161 00:07:04,220 --> 00:07:08,019 Speaker 2: learning style, it becomes individualized, so to speak. So then 162 00:07:08,019 --> 00:07:10,380 Speaker 2: does that mean that the role of a teacher becomes 163 00:07:10,380 --> 00:07:14,299 Speaker 2: so teachers be left behind, I'm trying to say because the, yeah, exactly, 164 00:07:14,339 --> 00:07:16,660 Speaker 2: they already used to learning from GBT. So when the 165 00:07:16,660 --> 00:07:19,859 Speaker 2: teacher is trying to teach them, it becomes increasingly difficult 166 00:07:19,859 --> 00:07:20,899 Speaker 2: for teachers to teach 167 00:07:20,899 --> 00:07:22,200 Speaker 2: students. 168 00:07:24,730 --> 00:07:25,660 Speaker 1: I just have my private tutor 169 00:07:25,660 --> 00:07:26,019 Speaker 1: at home 170 00:07:26,019 --> 00:07:27,820 Speaker 2: when it comes to this, I think there are a 171 00:07:27,820 --> 00:07:28,299 Speaker 2: few points. 172 00:07:28,444 --> 00:07:30,884 Speaker 2: So the personalized learning pathway is always going to be 173 00:07:30,884 --> 00:07:33,324 Speaker 2: better than the generalized one because it's simply it's going 174 00:07:33,324 --> 00:07:35,084 Speaker 2: to be a better fit. What we've seen is that 175 00:07:35,084 --> 00:07:38,123 Speaker 2: motivated individuals who are trying to pick up some new 176 00:07:38,123 --> 00:07:39,843 Speaker 2: skills or they're trying to accomplish a task or if 177 00:07:39,843 --> 00:07:42,204 Speaker 2: they're even trying to do less work in school, they 178 00:07:42,204 --> 00:07:44,523 Speaker 2: will turn to these AI models and for you mentioned 179 00:07:44,523 --> 00:07:46,523 Speaker 2: large language models and all sorts of different models, they 180 00:07:46,523 --> 00:07:49,194 Speaker 2: will turn to these tools to try and accomplish their task. Now, 181 00:07:49,204 --> 00:07:52,123 Speaker 2: the thing is, I think many people think that just 182 00:07:52,123 --> 00:07:54,234 Speaker 2: because you use AI, you know, people become dumber, people 183 00:07:54,234 --> 00:07:55,123 Speaker 2: don't think about it anymore. 184 00:07:55,278 --> 00:07:58,868 Speaker 2: And, you know, I'm seeing Reddit students are saying that 185 00:07:58,868 --> 00:08:02,097 Speaker 2: it's rare nowadays to see people think on their own 186 00:08:02,097 --> 00:08:05,507 Speaker 2: because they're sending their assignments to GPT before even thinking 187 00:08:05,507 --> 00:08:08,066 Speaker 2: the devil is in the details, how are they sending 188 00:08:08,067 --> 00:08:11,388 Speaker 2: it in, right? So the degree of specificity that you 189 00:08:11,388 --> 00:08:13,988 Speaker 2: would need to include in a prompt in an instruction 190 00:08:13,988 --> 00:08:17,347 Speaker 2: requires that you identify your own thinking and label and 191 00:08:17,347 --> 00:08:19,828 Speaker 2: actually describe step by step exactly how you would go 192 00:08:19,828 --> 00:08:22,088 Speaker 2: about attacking this task. So if I were to just 193 00:08:22,291 --> 00:08:24,611 Speaker 2: GBT write this essay for me. You're not going to 194 00:08:24,611 --> 00:08:26,052 Speaker 2: get that good of an essay. If you say write 195 00:08:26,052 --> 00:08:27,932 Speaker 2: and style of, you know, so and so, it does 196 00:08:27,932 --> 00:08:30,652 Speaker 2: a little better. If I wanted to think really in depth, 197 00:08:30,731 --> 00:08:33,841 Speaker 2: you know, go through this point, uh, articulate this particular argument, 198 00:08:34,091 --> 00:08:36,481 Speaker 2: quote an example here, and make sure it makes sense 199 00:08:36,481 --> 00:08:38,170 Speaker 2: in the grand scheme of things, and then go step 200 00:08:38,171 --> 00:08:40,331 Speaker 2: by step. You end up with these prompts that are 201 00:08:40,331 --> 00:08:43,372 Speaker 2: pretty normal. I think in industry to write 34 paragraphs 202 00:08:43,372 --> 00:08:48,892 Speaker 2: about the right questions are you actually learning anything? 203 00:08:49,005 --> 00:08:49,995 Speaker 2: What is learning then? 204 00:08:50,635 --> 00:08:51,356 Speaker 1: I 205 00:08:51,356 --> 00:08:53,716 Speaker 1: think you are. So it's like having when we first 206 00:08:53,716 --> 00:08:56,885 Speaker 1: had Google as a resource, how do you use it 207 00:08:57,155 --> 00:08:58,966 Speaker 1: and use it effectively, right? So you can use it 208 00:08:58,966 --> 00:09:00,324 Speaker 1: the wrong way, you can use it the right way. 209 00:09:00,385 --> 00:09:02,236 Speaker 1: Same thing with the AI. You can use it to 210 00:09:02,236 --> 00:09:05,265 Speaker 1: make whatever you're doing better and to learn more and 211 00:09:05,265 --> 00:09:07,106 Speaker 1: understand better, or you can just be lazy and just 212 00:09:07,106 --> 00:09:08,745 Speaker 1: chuck it the whole problem and let it do everything 213 00:09:08,745 --> 00:09:10,796 Speaker 1: for you. The question is how are you using the 214 00:09:10,796 --> 00:09:13,315 Speaker 1: AI and at which point can you say it's OK 215 00:09:13,315 --> 00:09:14,955 Speaker 1: to use it this way, but it's not OK to 216 00:09:14,955 --> 00:09:15,744 Speaker 1: use it that way. 217 00:09:16,099 --> 00:09:18,840 Speaker 2: Yeah, how do we differentiate misuse and misjudgment? 218 00:09:19,099 --> 00:09:21,599 Speaker 1: I think there are the two things I mentioned that matter, right? 219 00:09:21,640 --> 00:09:24,169 Speaker 1: Number 1 is the learning. Number 2 is the fairness, right, 220 00:09:24,330 --> 00:09:26,450 Speaker 1: because the trouble now is that some students use AI 221 00:09:26,450 --> 00:09:29,729 Speaker 1: and they get better grade other students who don't use AI, uh, 222 00:09:29,770 --> 00:09:32,939 Speaker 1: it's not unfair. I'm like just need to switch on 223 00:09:32,940 --> 00:09:34,069 Speaker 1: your computer, that's all. 224 00:09:35,169 --> 00:09:35,750 Speaker 1: Too lazy to 225 00:09:35,750 --> 00:09:39,309 Speaker 1: do that. The problem here is that there's there's a 226 00:09:39,309 --> 00:09:41,960 Speaker 1: class where the prophet said that you shall not use AI. OK, 227 00:09:42,229 --> 00:09:44,109 Speaker 1: so you have people who are this, this man looks 228 00:09:44,109 --> 00:09:46,299 Speaker 1: like he looks like he's a man, right. 229 00:09:47,250 --> 00:09:50,010 Speaker 1: I'm wrong, but suppose it's an honest man, right, and 230 00:09:50,010 --> 00:09:52,640 Speaker 1: you dishonest man and this honest one actually, you know, 231 00:09:52,729 --> 00:09:54,959 Speaker 1: gets a better grade, then then there's something wrong, right, 232 00:09:55,119 --> 00:09:57,520 Speaker 1: because then we have a society which actually, you know, 233 00:09:57,559 --> 00:09:59,520 Speaker 1: rewards people who are dishonest. Is that something you want 234 00:09:59,520 --> 00:10:02,960 Speaker 1: in Singapore? I think it's, is that the fault also 235 00:10:02,960 --> 00:10:05,359 Speaker 1: of the way they're being tested. The professor may be 236 00:10:05,359 --> 00:10:08,169 Speaker 1: a bit naive to say you cannot use AI, but 237 00:10:08,169 --> 00:10:09,819 Speaker 1: in this day and age, you know what I mean? OK, 238 00:10:10,450 --> 00:10:12,239 Speaker 1: let's set it aside, but, but I think if you 239 00:10:12,239 --> 00:10:14,320 Speaker 1: ask me what what is the issue here, it is 240 00:10:14,320 --> 00:10:15,839 Speaker 1: the issue of fairness, OK. 241 00:10:16,284 --> 00:10:19,314 Speaker 1: So to address this issue of fairness, if, if the 242 00:10:19,315 --> 00:10:22,744 Speaker 1: professor says you can't use AI, people should not use AI, right? Now, 243 00:10:22,755 --> 00:10:24,635 Speaker 1: in the case where by first process that I don't 244 00:10:24,635 --> 00:10:26,474 Speaker 1: care you'll use AI, then yeah, go ahead, but the 245 00:10:26,474 --> 00:10:29,064 Speaker 1: trouble will be a problem, right, which is that then, uh, 246 00:10:29,275 --> 00:10:31,353 Speaker 1: what's the whole point, right, because you get pretty much 247 00:10:31,354 --> 00:10:33,794 Speaker 1: the same thing and worse still, we still, you know, 248 00:10:33,835 --> 00:10:37,194 Speaker 1: there are different models, paid models, unpaid models, and suppose 249 00:10:37,195 --> 00:10:39,794 Speaker 1: this guy is rich, right? He buys the advantage and 250 00:10:39,794 --> 00:10:41,875 Speaker 1: he may get advantage we could argue the kids who 251 00:10:41,875 --> 00:10:44,114 Speaker 1: have tuition versus those who do not also have an 252 00:10:44,114 --> 00:10:45,155 Speaker 1: unfair advantage, right. 253 00:10:45,859 --> 00:10:47,979 Speaker 1: Well that's true. That's true, but, but I think that's 254 00:10:47,979 --> 00:10:50,169 Speaker 1: the way the real world works. That's how, how far 255 00:10:50,169 --> 00:10:52,530 Speaker 1: you want to bring that. But but ultimately there is 256 00:10:52,530 --> 00:10:53,690 Speaker 1: the issue of fairness 257 00:10:53,690 --> 00:10:57,210 Speaker 2: the fairness point in that case, is it even a 258 00:10:57,210 --> 00:10:58,869 Speaker 2: good proxy of the real world because the real world 259 00:10:58,869 --> 00:11:01,119 Speaker 2: is not really fair. People have to learn. I think 260 00:11:01,119 --> 00:11:03,140 Speaker 2: the whole point of having the school and having that 261 00:11:03,140 --> 00:11:03,580 Speaker 2: environment is. 262 00:11:03,724 --> 00:11:06,534 Speaker 2: To simulated almost a 1 to 1 mimic of the 263 00:11:06,534 --> 00:11:08,135 Speaker 2: real world and it trains them to operate in the 264 00:11:08,135 --> 00:11:10,215 Speaker 2: real world. Does it help them learn in that case? 265 00:11:10,375 --> 00:11:10,544 Speaker 1: That's an 266 00:11:10,544 --> 00:11:13,255 Speaker 1: excellent philosophical question. But, but so but to address these 267 00:11:13,255 --> 00:11:15,054 Speaker 1: two issues. The first issue is this, right? It is 268 00:11:15,054 --> 00:11:17,164 Speaker 1: that the learning part, I think the learning part is real. 269 00:11:17,294 --> 00:11:19,744 Speaker 1: There are students that are really not getting, I mean, 270 00:11:20,294 --> 00:11:23,414 Speaker 1: what I'm seeing, again, this is not scientific. I actually 271 00:11:23,414 --> 00:11:25,214 Speaker 1: put up some of the grant to learn to study 272 00:11:25,215 --> 00:11:27,214 Speaker 1: this thing more carefully, but based on what I'm seeing 273 00:11:27,215 --> 00:11:29,525 Speaker 1: on the ground today, there's a small growth students, OK, 274 00:11:29,614 --> 00:11:31,054 Speaker 1: who are using AI very effectively. 275 00:11:31,340 --> 00:11:34,539 Speaker 1: They are actually moving ahead faster, better than before. OK, 276 00:11:34,580 --> 00:11:37,140 Speaker 1: so you compare them today, the same same students to 277 00:11:37,140 --> 00:11:38,819 Speaker 1: those in the past, these guys will do better, and 278 00:11:38,820 --> 00:11:42,460 Speaker 1: that's the rest. I really think the rest are actually 279 00:11:42,460 --> 00:11:47,939 Speaker 1: getting dumber. I don't know for using AI they are 280 00:11:47,940 --> 00:11:49,098 Speaker 1: actually not OK, so 281 00:11:49,099 --> 00:11:49,340 Speaker 2: you are 282 00:11:49,340 --> 00:11:53,460 Speaker 2: saying that because they have lost their authenticity, their creative juices, 283 00:11:53,500 --> 00:11:54,700 Speaker 2: they are counting on. 284 00:11:55,210 --> 00:11:55,349 Speaker 1: Right, 285 00:11:56,940 --> 00:11:59,270 Speaker 1: so we're just saying, so do I get this project 286 00:11:59,270 --> 00:12:00,750 Speaker 1: and just chuck the whole thing to AI and say 287 00:12:00,750 --> 00:12:03,150 Speaker 1: give me an answer, or do I understand the project, 288 00:12:03,429 --> 00:12:05,549 Speaker 1: understand what are the 5 tools I need, then go 289 00:12:05,549 --> 00:12:07,909 Speaker 1: to AI and say, help me figure out what are 290 00:12:07,909 --> 00:12:09,819 Speaker 1: these 5 tools. So there are different ways of using 291 00:12:09,820 --> 00:12:12,239 Speaker 1: that right? That's not how the human psychology works. That's 292 00:12:12,239 --> 00:12:13,150 Speaker 1: not how the kids think, right? 293 00:12:13,450 --> 00:12:15,319 Speaker 1: So, so what I'm telling you on the ground is 294 00:12:15,320 --> 00:12:18,960 Speaker 1: that there's a small group, and they are actually moving ahead, right? And, 295 00:12:19,159 --> 00:12:21,679 Speaker 1: and actually, to be honest, we don't fully understand how 296 00:12:21,679 --> 00:12:24,520 Speaker 1: the kids are using AI today because it's quite new, 297 00:12:24,599 --> 00:12:27,479 Speaker 1: like 2 years, OK. We don't watch over them 24/7, right? And, 298 00:12:27,539 --> 00:12:30,450 Speaker 1: and they are doing some some things that way. 299 00:12:31,659 --> 00:12:34,179 Speaker 1: Really kind of wild, there are some ways but then 300 00:12:34,179 --> 00:12:37,299 Speaker 1: the question becomes, is it then, are we then testing 301 00:12:37,299 --> 00:12:41,289 Speaker 1: the students the wrong way? Should we also evolve in the, 302 00:12:41,419 --> 00:12:42,099 Speaker 1: you know, how we 303 00:12:42,099 --> 00:12:45,619 Speaker 2: analyze more face to face um sort of discussion about 304 00:12:45,619 --> 00:12:47,020 Speaker 2: certain topic or 305 00:12:47,020 --> 00:12:47,340 Speaker 1: excellent 306 00:12:47,340 --> 00:12:49,109 Speaker 1: point. There are two kinds of disciplines. There's my kind 307 00:12:49,109 --> 00:12:53,299 Speaker 1: of disciplines and there's the poor humanities people. The difference 308 00:12:53,299 --> 00:12:54,179 Speaker 1: is this, what do we do? 309 00:12:54,700 --> 00:12:56,590 Speaker 1: We lock them in the exam hall. We lock them 310 00:12:56,590 --> 00:12:58,719 Speaker 1: in a lab, but I, I do have some colleagues 311 00:12:58,719 --> 00:13:00,760 Speaker 1: who are in humanities whereby the grading is all done 312 00:13:00,760 --> 00:13:03,289 Speaker 1: by essays. Uh, they are struggling and, and I, I, 313 00:13:03,320 --> 00:13:05,780 Speaker 1: I feel bad because I don't really have a good 314 00:13:05,780 --> 00:13:06,599 Speaker 1: solution for them because 315 00:13:06,599 --> 00:13:10,359 Speaker 2: you are in humanities, right? I've been operating in tech 316 00:13:10,359 --> 00:13:10,919 Speaker 2: for 5 years. 317 00:13:11,530 --> 00:13:13,520 Speaker 2: So do you discuss among your friends, OK, what is 318 00:13:13,520 --> 00:13:16,070 Speaker 2: the best way, you know, to go around this problem, 319 00:13:16,400 --> 00:13:18,400 Speaker 2: to ask these large language models, how can we sort 320 00:13:18,400 --> 00:13:20,939 Speaker 2: of get around the system? I mean, I think it's 321 00:13:20,940 --> 00:13:22,840 Speaker 2: it's interesting you bring that up because I think many 322 00:13:22,840 --> 00:13:26,030 Speaker 2: people now use the conversational models as almost that partner. 323 00:13:26,440 --> 00:13:30,119 Speaker 2: So instead of asking friends, they just ask the models 324 00:13:30,119 --> 00:13:32,750 Speaker 2: will probably answer better. I think on the point of 325 00:13:32,960 --> 00:13:35,400 Speaker 2: essentially essays being a poor measure, it's it's a horrible measure. 326 00:13:35,559 --> 00:13:37,159 Speaker 2: Essays are the new MCQs. 327 00:13:37,669 --> 00:13:39,710 Speaker 2: Because what we learned about language from all the years 328 00:13:39,710 --> 00:13:43,569 Speaker 2: of research in AI is basically that language given the rules, uh, 329 00:13:43,590 --> 00:13:46,580 Speaker 2: given the syntax, given the semantics, the meaning of each word, 330 00:13:46,830 --> 00:13:48,549 Speaker 2: if you just apply a few rules and you keep 331 00:13:48,549 --> 00:13:51,549 Speaker 2: in mind all the exceptions, you can algorithmically generate anything 332 00:13:51,549 --> 00:13:53,750 Speaker 2: you want and that's what all these models do. So 333 00:13:53,750 --> 00:13:57,109 Speaker 2: it's basically standardized testing, it just doesn't look like standardized 334 00:13:57,109 --> 00:13:58,830 Speaker 2: testing when when you give an essay. 335 00:13:59,440 --> 00:14:02,130 Speaker 2: 100% agree, it's a horrible, it's horrible to test it. 336 00:14:02,460 --> 00:14:04,819 Speaker 2: I know you're not really in the humanities because that's 337 00:14:04,820 --> 00:14:07,979 Speaker 2: where it seems like the where it's questionable that a 338 00:14:07,979 --> 00:14:11,099 Speaker 2: very gray area you see. So therefore, I mean, can 339 00:14:11,099 --> 00:14:13,369 Speaker 2: we come to the conclusion that something has to change? 340 00:14:13,500 --> 00:14:15,059 Speaker 2: Because right now if you ask a lot of students, right, 341 00:14:15,099 --> 00:14:18,858 Speaker 2: it's not clear what exactly is the demarcation, you know, 342 00:14:18,979 --> 00:14:21,260 Speaker 2: like can I use AI? I can use AI, but 343 00:14:21,260 --> 00:14:22,140 Speaker 2: to what extent? 344 00:14:22,599 --> 00:14:24,130 Speaker 2: You know, how do I know? And now there's a 345 00:14:24,130 --> 00:14:26,109 Speaker 2: lot of anxiety among the students as well because I 346 00:14:26,109 --> 00:14:27,969 Speaker 2: don't want to. I know people that don't use AI 347 00:14:27,969 --> 00:14:28,849 Speaker 2: because they're scared 348 00:14:28,849 --> 00:14:32,369 Speaker 1: that I think that's wrong because these are the real 349 00:14:32,369 --> 00:14:35,289 Speaker 1: world resources, they're available to everyone. Why are we saying 350 00:14:35,289 --> 00:14:35,890 Speaker 1: don't use it? 351 00:14:36,239 --> 00:14:36,979 Speaker 1: But in the real world, 352 00:14:37,710 --> 00:14:40,510 Speaker 2: you want to be labeled risk being labeled academic fraud 353 00:14:40,510 --> 00:14:41,109 Speaker 2: because I wouldn't 354 00:14:41,109 --> 00:14:46,349 Speaker 1: want that in a sense has failed to evolve and 355 00:14:46,349 --> 00:14:48,169 Speaker 1: adapt to what is happening in the real 356 00:14:48,169 --> 00:14:48,390 Speaker 1: world. 357 00:14:48,500 --> 00:14:50,309 Speaker 2: Or maybe they're not catching up fast. 358 00:14:50,809 --> 00:14:52,750 Speaker 1: I try to address that point, right? So, so I 359 00:14:52,750 --> 00:14:54,669 Speaker 1: think I've explained to you the two concerns we have 360 00:14:54,669 --> 00:14:56,869 Speaker 1: in school, right? But, but the problem is this, right, 361 00:14:56,989 --> 00:14:58,950 Speaker 1: we've always had this complaint from students that 362 00:14:59,190 --> 00:15:00,880 Speaker 1: Uh, like I said, we lock them up in, in, 363 00:15:00,890 --> 00:15:03,010 Speaker 1: in the lab, right, to do a coding exam, right? 364 00:15:03,130 --> 00:15:04,510 Speaker 1: They're like, what the heck? I mean, you know, in 365 00:15:04,510 --> 00:15:06,090 Speaker 1: real life I get to go to GitHub and then 366 00:15:06,090 --> 00:15:09,229 Speaker 1: I copy I'm better, right? OK, so that's a very 367 00:15:09,229 --> 00:15:11,570 Speaker 1: good night now. But you see, the whole point of 368 00:15:11,570 --> 00:15:14,280 Speaker 1: school actually is that we're trying to train them to 369 00:15:14,409 --> 00:15:16,270 Speaker 1: to learn some things, OK, and 370 00:15:16,979 --> 00:15:19,280 Speaker 1: I don't know why people think that coding is easy lah, 371 00:15:19,400 --> 00:15:22,150 Speaker 1: but I must tell you it is not that easy, OK? 372 00:15:22,599 --> 00:15:25,280 Speaker 1: And the fact is this, I actually don't use AI 373 00:15:25,280 --> 00:15:27,719 Speaker 1: because sadly AI doesn't write as well as I can write, 374 00:15:27,799 --> 00:15:31,109 Speaker 1: and AI does not write code, uh, doesn't write code 375 00:15:31,320 --> 00:15:33,840 Speaker 1: the way I can write code either, right? So, OK, 376 00:15:33,960 --> 00:15:36,159 Speaker 1: so for coding, actually sometimes I might use AI because 377 00:15:36,159 --> 00:15:38,679 Speaker 1: it's just convenient because the truth is that we are 378 00:15:38,679 --> 00:15:40,520 Speaker 1: trying to teach the students to a certain level, OK. 379 00:15:40,820 --> 00:15:44,359 Speaker 1: And and the AI today, while it may seem like 380 00:15:44,359 --> 00:15:48,320 Speaker 1: magical people, they are doing things that are relatively competent 381 00:15:48,320 --> 00:15:51,440 Speaker 1: but not expert level. OK, but the challenge is this, right, 382 00:15:51,450 --> 00:15:54,880 Speaker 1: if the kids, the students today never even reach that 383 00:15:54,880 --> 00:15:58,419 Speaker 1: level of minimal competency, they will never cross at the chasm. OK. 384 00:15:58,599 --> 00:16:00,479 Speaker 1: So you can think of this curve, right? There's this 385 00:16:00,479 --> 00:16:02,080 Speaker 1: like big part of people that then there's. 386 00:16:02,174 --> 00:16:05,924 Speaker 1: AI and there's this experts. I call this AI because 387 00:16:05,924 --> 00:16:08,114 Speaker 1: of that. So if they're not able to bring the 388 00:16:08,114 --> 00:16:10,424 Speaker 1: skill level to this level, they will never cross over 389 00:16:10,424 --> 00:16:12,914 Speaker 1: to become experts, and that's where the money is, that's 390 00:16:12,914 --> 00:16:14,315 Speaker 1: where the job. So in other words, you're saying they 391 00:16:14,315 --> 00:16:17,405 Speaker 1: need to have a good foundation of that learning first 392 00:16:17,405 --> 00:16:19,234 Speaker 1: before so that they can evolve later. But then I 393 00:16:19,234 --> 00:16:22,515 Speaker 1: also argue like mathematics, we learn about, I know, simultaneous 394 00:16:22,515 --> 00:16:24,755 Speaker 1: equations and all that. I used to tease my uncle 395 00:16:24,755 --> 00:16:26,354 Speaker 1: as a math professor. I said what for at the 396 00:16:26,354 --> 00:16:27,325 Speaker 1: end of the day, all I need to know is 397 00:16:27,325 --> 00:16:29,195 Speaker 1: if I give $10 at the hawker center, I get 398 00:16:29,195 --> 00:16:31,114 Speaker 1: back $5 change when I buy my Nasi Lemma. 399 00:16:31,640 --> 00:16:33,840 Speaker 1: The rest, I mean, and I have a calculator, I 400 00:16:33,840 --> 00:16:37,520 Speaker 1: have a computer, so I don't need those steps involved, right? So, 401 00:16:37,599 --> 00:16:38,469 Speaker 1: so in a way. 402 00:16:38,900 --> 00:16:42,659 Speaker 1: That foundation, yes, if it, some of it is lost, 403 00:16:42,869 --> 00:16:44,590 Speaker 1: but some would argue that some of it is not 404 00:16:44,590 --> 00:16:46,669 Speaker 1: needed in this day and age, and I used to 405 00:16:46,669 --> 00:16:50,190 Speaker 1: learn chemistry. I could do all the chemistry equation isn't 406 00:16:50,190 --> 00:16:52,510 Speaker 1: that what is doing? No, no, no, but you must 407 00:16:52,510 --> 00:16:54,630 Speaker 1: understand what we're doing in school, right? OK, so I 408 00:16:54,630 --> 00:16:57,070 Speaker 1: would tell, OK, I'm actually a teacher of teachers, so 409 00:16:57,070 --> 00:16:59,270 Speaker 1: my students are actually MOE teachers. So I tell them 410 00:16:59,270 --> 00:17:01,309 Speaker 1: actually you can teach anything you want, right, because the 411 00:17:01,309 --> 00:17:04,150 Speaker 1: practice is that they are not in in K-12 to 412 00:17:04,150 --> 00:17:06,430 Speaker 1: learn other equations. They're not there to learn, uh, you know, 413 00:17:06,589 --> 00:17:08,189 Speaker 1: benzene rings, how to react with each other. 414 00:17:08,680 --> 00:17:10,719 Speaker 1: Right, they are there to learn how to learn. That's 415 00:17:10,719 --> 00:17:13,040 Speaker 1: the key, and that's what we need in life, right? 416 00:17:13,160 --> 00:17:14,560 Speaker 1: And let me address this this question which I try 417 00:17:14,560 --> 00:17:17,280 Speaker 1: to address, which is that why, why don't we let 418 00:17:17,280 --> 00:17:19,520 Speaker 1: the kids do what they have done in real life? 419 00:17:19,680 --> 00:17:22,430 Speaker 1: The reason is very simple. It's because we are trying 420 00:17:22,430 --> 00:17:24,119 Speaker 1: to get them to reach a certain level of mastery 421 00:17:24,119 --> 00:17:27,839 Speaker 1: so that hope, I must admit, I mean, every math 422 00:17:27,839 --> 00:17:30,040 Speaker 1: prof hopes that the kids will become math profs, and 423 00:17:30,040 --> 00:17:30,760 Speaker 1: then the computer scientists. 424 00:17:30,930 --> 00:17:34,560 Speaker 1: to be good programmers, right? So we are trying in 425 00:17:34,560 --> 00:17:37,719 Speaker 1: good faith to help our students acquire skills to become 426 00:17:37,719 --> 00:17:40,750 Speaker 1: masters in this. Some may want to. I agree with you, 427 00:17:40,760 --> 00:17:42,959 Speaker 1: we should be doing that, but maybe we need to 428 00:17:42,959 --> 00:17:45,400 Speaker 1: be doing it a different way now that the tools 429 00:17:45,400 --> 00:17:49,319 Speaker 1: have changed. No, no, no, there's no difference because the 430 00:17:49,319 --> 00:17:51,198 Speaker 1: learning is in your head. OK, no, no, no, let 431 00:17:51,199 --> 00:17:52,560 Speaker 1: me explain to you what the difference is. OK, so 432 00:17:52,560 --> 00:17:53,040 Speaker 1: first of all. 433 00:17:53,810 --> 00:17:55,900 Speaker 1: I want to make a very bold claim. There are 434 00:17:55,900 --> 00:17:59,859 Speaker 1: no revolutions in education, you know, 1015 years ago that 435 00:17:59,859 --> 00:18:01,349 Speaker 1: the MOOCs, right, came out and said, oh, we're going 436 00:18:01,349 --> 00:18:03,300 Speaker 1: to change the world, and I'll be out of a 437 00:18:03,300 --> 00:18:07,250 Speaker 1: job I last checked, I just had 800 students last semester. 438 00:18:07,660 --> 00:18:10,089 Speaker 1: It's not gonna happen, and I tell you why, because actually, 439 00:18:10,650 --> 00:18:14,810 Speaker 1: The business of education and teaching is not about communicating knowledge. 440 00:18:15,540 --> 00:18:18,020 Speaker 1: We are in the business of managing motivation. OK, 441 00:18:18,099 --> 00:18:18,699 Speaker 2: so can I ask 442 00:18:18,699 --> 00:18:21,939 Speaker 2: you, with large language models, you said your job is 443 00:18:21,939 --> 00:18:26,040 Speaker 2: to motivate the kids, right? In general, with the with 444 00:18:26,119 --> 00:18:28,139 Speaker 2: with LLM, would you say the motivation has gone up 445 00:18:28,140 --> 00:18:28,699 Speaker 2: or gone down? 446 00:18:28,900 --> 00:18:31,579 Speaker 1: OK, so that's the problem. So that's why I was 447 00:18:31,579 --> 00:18:34,300 Speaker 1: telling my students, my students who are not teachers that 448 00:18:34,300 --> 00:18:36,659 Speaker 1: actually the trouble is that we may think that we 449 00:18:36,660 --> 00:18:39,300 Speaker 1: are offer 20 years for good teachers. Yeah, maybe we 450 00:18:39,300 --> 00:18:39,899 Speaker 1: have certain skills. 451 00:18:40,219 --> 00:18:43,130 Speaker 1: The trouble is that the clientele has changed. The kids 452 00:18:43,130 --> 00:18:44,889 Speaker 1: have evolved, you know, and what has happened over the 453 00:18:44,890 --> 00:18:50,119 Speaker 1: last 20 years? Mobile phones, social media, COVID actually also 454 00:18:50,119 --> 00:18:52,250 Speaker 1: impacted the kids a lot. And, and now today we 455 00:18:52,250 --> 00:18:55,040 Speaker 1: have LMs. So and then the internet also also affected everything, 456 00:18:55,209 --> 00:18:58,400 Speaker 1: and that will keep changing. So you as a student 457 00:18:58,400 --> 00:18:59,649 Speaker 1: just graduated, I mean. 458 00:19:00,280 --> 00:19:02,790 Speaker 1: Do you agree with what Ben saying? 459 00:19:02,920 --> 00:19:03,079 Speaker 2: I 460 00:19:03,079 --> 00:19:04,040 Speaker 2: think for the most part it's 461 00:19:04,040 --> 00:19:07,399 Speaker 1: probably 462 00:19:07,400 --> 00:19:09,879 Speaker 2: correct, but, but, but the implications of that, you know, if, 463 00:19:09,939 --> 00:19:11,899 Speaker 2: if we really go with the implications of that and 464 00:19:11,910 --> 00:19:13,920 Speaker 2: and I agree you're you're in the business of motivation, 465 00:19:14,000 --> 00:19:17,079 Speaker 2: I think it's quite telling because what it means if 466 00:19:17,079 --> 00:19:19,500 Speaker 2: you if you extrapolate it out is that students who 467 00:19:19,500 --> 00:19:19,920 Speaker 2: are not motivated. 468 00:19:20,185 --> 00:19:21,765 Speaker 2: in the first place, they're going to go to school, 469 00:19:21,805 --> 00:19:24,204 Speaker 2: they're not going to be motivated and it's going to 470 00:19:24,204 --> 00:19:26,645 Speaker 2: be the same thing. They're gonna you're just keeping the 471 00:19:26,645 --> 00:19:29,125 Speaker 2: same student, not motivated and then giving them a bad score. 472 00:19:29,405 --> 00:19:31,004 Speaker 2: So that's the value for those people, which is the 473 00:19:31,005 --> 00:19:32,464 Speaker 2: majority of the people who we are saying are not 474 00:19:32,464 --> 00:19:34,165 Speaker 2: at that mastery, they're not at that level where they've 475 00:19:34,165 --> 00:19:36,045 Speaker 2: passed the curve, and then for the other people that 476 00:19:36,045 --> 00:19:38,295 Speaker 2: are going in and getting this validation. So the value, 477 00:19:38,444 --> 00:19:41,165 Speaker 2: I don't see it basically in this case. So Jeremy, 478 00:19:41,244 --> 00:19:42,244 Speaker 2: if you have to teach. 479 00:19:42,930 --> 00:19:45,530 Speaker 2: Aspiring students who want to ace their exams, how to 480 00:19:45,530 --> 00:19:48,089 Speaker 2: use AI, right? What would you say to them? I 481 00:19:48,089 --> 00:19:50,180 Speaker 2: think one of the things is I believe people are 482 00:19:50,180 --> 00:19:53,290 Speaker 2: quite output focused, especially in today's world. So you can 483 00:19:53,290 --> 00:19:55,409 Speaker 2: see that people want instant results and then after that 484 00:19:55,410 --> 00:19:57,649 Speaker 2: they want to know how to replicate those instant results. 485 00:19:57,689 --> 00:19:59,430 Speaker 2: It's very easy to hook people in if you show 486 00:19:59,430 --> 00:20:01,909 Speaker 2: them that this is the output you could get and 487 00:20:01,910 --> 00:20:03,329 Speaker 2: you just kind of need to figure out what's the 488 00:20:03,329 --> 00:20:05,439 Speaker 2: secret sauce, what are the inputs that leads to this output. 489 00:20:05,569 --> 00:20:07,369 Speaker 2: So I think actually one of the interesting things is 490 00:20:07,369 --> 00:20:10,530 Speaker 2: with AI I have started touching on a lot of 491 00:20:10,530 --> 00:20:12,239 Speaker 2: topics which previously I wouldn't have. 492 00:20:12,930 --> 00:20:15,310 Speaker 2: Because AI, when I ask, for example, Chat GBT currently 493 00:20:15,310 --> 00:20:17,479 Speaker 2: I'm using Cloud, I'm using Gemini, some other tools as well, 494 00:20:17,729 --> 00:20:19,609 Speaker 2: it starts telling me at a very high level, this 495 00:20:19,609 --> 00:20:21,050 Speaker 2: is how it works, this is how it works. It 496 00:20:21,050 --> 00:20:23,139 Speaker 2: condenses the information down to a 5 year old level, 497 00:20:23,329 --> 00:20:26,010 Speaker 2: then I say, OK, cool. Now let's elaborate, let's go 498 00:20:26,010 --> 00:20:29,280 Speaker 2: into this particular topic, let's dive even deeper. And so, 499 00:20:30,050 --> 00:20:32,839 Speaker 2: At all of these levels, I think my motivation constantly 500 00:20:32,839 --> 00:20:34,280 Speaker 2: increases a little bit, a little bit, a little bit, 501 00:20:34,290 --> 00:20:37,250 Speaker 2: and there's this compounding effect rather than I think what's 502 00:20:37,250 --> 00:20:40,489 Speaker 2: more common in a lot of educational institutions is they 503 00:20:40,489 --> 00:20:42,199 Speaker 2: start off kind of giving you the A, B's and C's, 504 00:20:42,209 --> 00:20:43,829 Speaker 2: but you don't see that you're going to write an essay. 505 00:20:44,050 --> 00:20:45,609 Speaker 2: So you don't see the story, you don't see where 506 00:20:45,609 --> 00:20:46,369 Speaker 2: the final outcome really. 507 00:20:46,489 --> 00:20:48,719 Speaker 2: It's going to be and I think all throughout, it's 508 00:20:48,719 --> 00:20:51,000 Speaker 2: just kind of this hazy black box. Students are not 509 00:20:51,000 --> 00:20:53,839 Speaker 2: that motivated partly because the methodology is not there. But 510 00:20:53,839 --> 00:20:56,958 Speaker 2: you also see students like obviously um abusing it, right? 511 00:20:57,000 --> 00:21:00,040 Speaker 2: Was it the UCLA graduate we saw, you know, it 512 00:21:00,040 --> 00:21:02,209 Speaker 2: went viral because he posted like he showed he was, 513 00:21:02,239 --> 00:21:03,000 Speaker 2: you know, using 514 00:21:03,339 --> 00:21:05,448 Speaker 2: Yeah, GPT and he's graduated, 515 00:21:06,640 --> 00:21:09,410 Speaker 1: so should we stop and eliminate and create all these 516 00:21:09,410 --> 00:21:12,010 Speaker 1: barriers to entry just because a small handful of people abuse. 517 00:21:12,290 --> 00:21:15,688 Speaker 2: I think all systems are capable. So is that where 518 00:21:15,689 --> 00:21:21,208 Speaker 2: again the institutions need to again set clearer defined lines 519 00:21:21,209 --> 00:21:23,719 Speaker 2: like how to use what and what to use. 520 00:21:24,050 --> 00:21:24,290 Speaker 1: I think 521 00:21:24,290 --> 00:21:27,920 Speaker 1: it be fair to universities, right? It's only been 2 years. 522 00:21:28,369 --> 00:21:30,319 Speaker 1: You know, it's not been like that. OK, it may 523 00:21:30,319 --> 00:21:34,550 Speaker 1: seem like lifetime like around, uh, but, uh, I mean 524 00:21:35,199 --> 00:21:38,599 Speaker 1: things evolve move not so quickly, right? And, and to 525 00:21:38,599 --> 00:21:41,020 Speaker 1: be honest, we, we are also not completely, like I said, 526 00:21:41,199 --> 00:21:43,479 Speaker 1: I admitted, right, we're not completely sure what the states 527 00:21:43,479 --> 00:21:45,639 Speaker 1: are doing today, how to use it, and that's part 528 00:21:45,640 --> 00:21:48,319 Speaker 1: of the learning process. So I, I, I think that 529 00:21:48,319 --> 00:21:49,770 Speaker 1: the thing about fraud, like I said, the whole this 530 00:21:49,770 --> 00:21:52,399 Speaker 1: whole about fraud is really about rules. I mean, the prof, OK, 531 00:21:52,479 --> 00:21:55,680 Speaker 1: whoever is instructor has the right to set the rules, right. 532 00:21:56,099 --> 00:21:58,949 Speaker 1: And the real question here is not whether you agree 533 00:21:58,949 --> 00:22:01,310 Speaker 1: the rules or not, right, is that if the rule 534 00:22:01,310 --> 00:22:01,910 Speaker 1: is broken. 535 00:22:02,699 --> 00:22:05,050 Speaker 1: What, what should be our response, right? I mean, OK, 536 00:22:05,329 --> 00:22:08,140 Speaker 1: you sound like anarchist. I like, OK, we just 537 00:22:08,140 --> 00:22:09,429 Speaker 1: training to be slaves? 538 00:22:09,439 --> 00:22:12,449 Speaker 1: No, no, no, no, fair enough. So you say this 539 00:22:12,449 --> 00:22:14,010 Speaker 1: is my class. This is what we're gonna learn. You 540 00:22:14,010 --> 00:22:15,849 Speaker 1: are not allowed to use, for example, you're not allowed 541 00:22:15,849 --> 00:22:18,849 Speaker 1: to use a calculator for this math equation, right? But 542 00:22:18,849 --> 00:22:21,569 Speaker 1: you make it clear, right? If you agree to sign 543 00:22:21,569 --> 00:22:23,489 Speaker 1: up for the course, you play by these rules. I 544 00:22:23,489 --> 00:22:25,969 Speaker 1: think they did make it clear, right then on that 545 00:22:25,969 --> 00:22:26,540 Speaker 1: basis that it. 546 00:22:26,800 --> 00:22:29,640 Speaker 1: Prosecute and and and you know, and it didn't seem 547 00:22:29,640 --> 00:22:29,800 Speaker 1: like 548 00:22:29,800 --> 00:22:30,229 Speaker 1: it's clear 549 00:22:30,229 --> 00:22:32,909 Speaker 2: because it's still in limbo. I mean it it it's 550 00:22:32,910 --> 00:22:33,479 Speaker 2: it's pending, 551 00:22:33,510 --> 00:22:33,640 Speaker 1: you 552 00:22:33,640 --> 00:22:37,560 Speaker 1: see. Why don't you just let NTU and the appeals 553 00:22:37,560 --> 00:22:39,639 Speaker 1: panel do their work, right? So, so that would be 554 00:22:39,640 --> 00:22:43,229 Speaker 1: the question just what is clear and what isn't clear. 555 00:22:43,359 --> 00:22:45,000 Speaker 1: As they go through it, they will then have to 556 00:22:45,000 --> 00:22:48,119 Speaker 1: further elaborate and then seek clarity. And even if it's 557 00:22:48,119 --> 00:22:50,920 Speaker 1: not clear, I think it to be fair to the instructors. 558 00:22:51,119 --> 00:22:53,170 Speaker 1: This is a brave new world we are entering and, 559 00:22:53,189 --> 00:22:55,510 Speaker 1: and I, you know, so there are also many of 560 00:22:55,510 --> 00:22:58,489 Speaker 1: them are also struggling to cope, uh, many of them are, 561 00:22:58,709 --> 00:23:01,060 Speaker 1: I mean, many of us I think are ill equipped to, to, 562 00:23:01,140 --> 00:23:02,949 Speaker 2: to know, but the tuition centers can do it really 563 00:23:02,949 --> 00:23:05,909 Speaker 2: well because we, we service some other tuition centers. I 564 00:23:05,910 --> 00:23:08,670 Speaker 2: have friends doing personal tuition. I know other business owners 565 00:23:08,670 --> 00:23:11,310 Speaker 2: who own tuition centers. The teachers are kind of forcing 566 00:23:11,310 --> 00:23:13,319 Speaker 2: it down on them and saying, OK, this is our business. 567 00:23:13,469 --> 00:23:15,500 Speaker 2: We're at stake if we don't adapt, we're gonna die. 568 00:23:15,790 --> 00:23:18,030 Speaker 2: So I'm gonna give you guys a suite of new tools. 569 00:23:18,349 --> 00:23:19,939 Speaker 2: Learn how to use and teach your students with it. 570 00:23:21,010 --> 00:23:22,290 Speaker 2: They they seem to be adapting fine. 571 00:23:22,329 --> 00:23:24,290 Speaker 1: So, so you're saying that the tuition centers are teaching 572 00:23:24,290 --> 00:23:25,170 Speaker 1: the students 573 00:23:25,170 --> 00:23:25,500 Speaker 1: how to, 574 00:23:25,650 --> 00:23:27,410 Speaker 2: I mean they are using AI and then they understand 575 00:23:27,410 --> 00:23:28,849 Speaker 2: that the students will use it as well. So the 576 00:23:28,849 --> 00:23:31,050 Speaker 2: way they come up, that kind of ideas and the 577 00:23:31,050 --> 00:23:33,329 Speaker 2: methods of verification, whether the work is good and whether 578 00:23:33,329 --> 00:23:35,339 Speaker 2: they thinking is solid, they're coming up with brand new 579 00:23:35,339 --> 00:23:37,290 Speaker 2: ways to. It's 580 00:23:37,290 --> 00:23:38,810 Speaker 1: like here at the company they say we have this 581 00:23:38,810 --> 00:23:41,770 Speaker 1: new system for operating your leave and all that. 582 00:23:42,209 --> 00:23:43,929 Speaker 1: Use it or you can't take leave. So I learned 583 00:23:43,930 --> 00:23:44,879 Speaker 1: how to use it real quick, right, same thing with AI, 584 00:23:44,890 --> 00:23:45,020 Speaker 1: but this is new to me. I'm not aware but 585 00:23:45,020 --> 00:23:53,449 Speaker 1: his argument being that we can, we can force this 586 00:23:53,449 --> 00:23:56,400 Speaker 1: change to be faster and it feels to a certain 587 00:23:56,400 --> 00:23:59,729 Speaker 1: extent that some of our educational institutions may be struggling 588 00:23:59,729 --> 00:24:03,968 Speaker 1: to keep up with the rapid pace of change. Would 589 00:24:03,969 --> 00:24:05,170 Speaker 1: you agree with that? I think we need to go 590 00:24:05,170 --> 00:24:08,969 Speaker 1: back to what is our role in society, right? 591 00:24:09,540 --> 00:24:11,859 Speaker 1: And I'm a little bit idealistic. I, I think our 592 00:24:11,859 --> 00:24:15,349 Speaker 1: role is to help our students learn better, OK? And, 593 00:24:15,500 --> 00:24:18,099 Speaker 1: and this thing about learning stuff, right? So, so the 594 00:24:18,099 --> 00:24:20,579 Speaker 1: fact is this, you're right, you are complaining about learning. 595 00:24:21,089 --> 00:24:23,540 Speaker 1: See these equations. OK. The fact is that many of 596 00:24:23,540 --> 00:24:25,280 Speaker 1: the things they learn in college actually don't don't need 597 00:24:25,280 --> 00:24:27,069 Speaker 1: to care about, OK, but it's part of the process. 598 00:24:27,150 --> 00:24:28,770 Speaker 1: It's good to know more things. But if they're going 599 00:24:28,770 --> 00:24:31,449 Speaker 1: to be a lawyer, a doctor, you know, you better 600 00:24:31,449 --> 00:24:33,719 Speaker 1: learn it well, right? They're going to die, right, you know, 601 00:24:33,849 --> 00:24:36,739 Speaker 1: and as a software engineer, they learn the craft very well. So, 602 00:24:36,760 --> 00:24:39,689 Speaker 1: so for that, in terms of learning for the things 603 00:24:39,689 --> 00:24:42,680 Speaker 1: that matter, then I, I think we need to do 604 00:24:42,680 --> 00:24:44,770 Speaker 1: what is right, OK, and, and let me just tell 605 00:24:44,770 --> 00:24:46,369 Speaker 1: you what I tell my students and I tell my 606 00:24:46,369 --> 00:24:47,410 Speaker 1: teachers to tell their students, right? 607 00:24:47,699 --> 00:24:50,030 Speaker 1: Which is a very simple argument, which is the following. 608 00:24:50,319 --> 00:24:53,629 Speaker 1: If it is quite clear that if they use AI 609 00:24:53,630 --> 00:24:54,829 Speaker 1: to do something they cannot do. 610 00:24:56,739 --> 00:24:59,149 Speaker 1: They will never be able to do it. OK. They 611 00:24:59,150 --> 00:25:00,229 Speaker 1: will never be able to do it. On the other hand, 612 00:25:00,349 --> 00:25:02,260 Speaker 1: if they can do it already, and then, you know, 613 00:25:02,469 --> 00:25:05,109 Speaker 1: if they use AI to do it, they they can 614 00:25:05,109 --> 00:25:08,270 Speaker 1: go home and get a cute girl, but if I 615 00:25:08,270 --> 00:25:08,750 Speaker 1: if I see 616 00:25:08,750 --> 00:25:10,909 Speaker 2: my other friends, they're using this chat GPT and they're 617 00:25:10,910 --> 00:25:13,160 Speaker 2: scoring higher marks than me and then I hear the 618 00:25:13,160 --> 00:25:13,209 Speaker 2: incentive structures. 619 00:25:16,994 --> 00:25:17,944 Speaker 1: But 620 00:25:17,944 --> 00:25:19,675 Speaker 2: then you're saying good luck to this person because if 621 00:25:19,675 --> 00:25:21,785 Speaker 2: he gets into the real world, maybe they won't be 622 00:25:21,785 --> 00:25:23,074 Speaker 2: able to think. 623 00:25:23,344 --> 00:25:25,714 Speaker 1: I think what you're going is that along the way 624 00:25:25,714 --> 00:25:28,823 Speaker 1: we may be losing some of that learning experience. They 625 00:25:28,824 --> 00:25:30,984 Speaker 1: may not be learning how to think, how to analyze. 626 00:25:32,755 --> 00:25:34,704 Speaker 1: You now have a device or AI to do it 627 00:25:34,704 --> 00:25:35,104 Speaker 1: for you. 628 00:25:35,489 --> 00:25:37,430 Speaker 1: So at the end of the day, I I I 629 00:25:37,430 --> 00:25:41,139 Speaker 1: do think that the people who actually put the effort, right, 630 00:25:41,290 --> 00:25:44,069 Speaker 1: will have much better chance of surpassing that that barrier 631 00:25:44,069 --> 00:25:46,550 Speaker 1: with the AI barrier and become experts. So you are 632 00:25:46,550 --> 00:25:49,229 Speaker 1: learning from the best, learning how he has learned how 633 00:25:49,229 --> 00:25:51,819 Speaker 1: to use technology and teaching that to the next generation 634 00:25:51,819 --> 00:25:54,910 Speaker 1: of students, which also means being a bit more open 635 00:25:54,910 --> 00:25:57,270 Speaker 1: to allowing the technology to come in. OK, I don't 636 00:25:57,270 --> 00:25:58,510 Speaker 1: think we're going to come to a clear black and 637 00:25:58,510 --> 00:26:01,910 Speaker 1: white answer today, but I guess my view is that 638 00:26:01,910 --> 00:26:03,849 Speaker 1: I think we need to evolve a bit more and 639 00:26:03,849 --> 00:26:05,219 Speaker 1: to be a bit more open. 640 00:26:05,650 --> 00:26:07,660 Speaker 1: To the use of technology and that we need to 641 00:26:07,660 --> 00:26:11,020 Speaker 1: change our way of assessment to find what is the 642 00:26:11,020 --> 00:26:13,500 Speaker 1: best in that person and to see whether they've actually 643 00:26:13,500 --> 00:26:15,699 Speaker 1: learned how to figure it out. I think that the 644 00:26:15,699 --> 00:26:18,469 Speaker 1: truth is that the, you know, profs actually know their business, 645 00:26:18,540 --> 00:26:19,729 Speaker 1: they know what they're trying to do, what they trying 646 00:26:19,729 --> 00:26:22,660 Speaker 1: to achieve for the students, and there is no one 647 00:26:22,660 --> 00:26:26,938 Speaker 1: size fits all kind of solution, and I hope you 648 00:26:26,939 --> 00:26:27,459 Speaker 1: believe that. 649 00:26:28,270 --> 00:26:30,489 Speaker 1: NTU NUS, all of us are trying to do our best, 650 00:26:30,630 --> 00:26:32,709 Speaker 1: the right things for students, right? So they don't punish 651 00:26:32,709 --> 00:26:34,709 Speaker 1: students because it's fun to do because they like the 652 00:26:34,709 --> 00:26:37,520 Speaker 1: publicity like this, it's not something that they do because 653 00:26:37,520 --> 00:26:38,069 Speaker 1: I think that's why 654 00:26:38,069 --> 00:26:38,869 Speaker 2: they're taking it seriously. 655 00:26:39,150 --> 00:26:40,260 Speaker 1: They're still deliberating 656 00:26:40,260 --> 00:26:40,530 Speaker 2: over 657 00:26:40,530 --> 00:26:41,750 Speaker 1: the other 658 00:26:41,750 --> 00:26:44,589 Speaker 2: students, yeah, and then this is this is a learning 659 00:26:44,589 --> 00:26:47,790 Speaker 2: platform for all of us, right, the students, the institutions, 660 00:26:47,880 --> 00:26:51,150 Speaker 2: the lecturers, and hopefully from here on, you know, we'd 661 00:26:51,150 --> 00:26:51,989 Speaker 2: be able to. 662 00:26:52,375 --> 00:26:55,115 Speaker 2: At least the students would have just a clearer idea 663 00:26:55,115 --> 00:26:57,326 Speaker 2: of like how, how, how do you make sure that 664 00:26:57,326 --> 00:27:00,806 Speaker 2: you are learning it properly and and not, I won't 665 00:27:00,806 --> 00:27:03,406 Speaker 2: say not getting caught, but you know you want to use, 666 00:27:03,686 --> 00:27:05,605 Speaker 1: you made a good point. I think what I what 667 00:27:05,605 --> 00:27:08,186 Speaker 1: I'm gonna ask you is, so if I'm a student now, 668 00:27:08,446 --> 00:27:11,326 Speaker 1: should I go and ask my professor A prof. So 669 00:27:11,326 --> 00:27:13,505 Speaker 1: for this next project, can I use AI? What can 670 00:27:13,505 --> 00:27:15,806 Speaker 1: I do? What can I not do and get those 671 00:27:15,806 --> 00:27:16,446 Speaker 1: boundaries set up. 672 00:27:16,631 --> 00:27:20,171 Speaker 1: As you mentioned, it's different for every subject, every different area. 673 00:27:20,271 --> 00:27:22,631 Speaker 1: I think I need to tell you what is available, 674 00:27:22,702 --> 00:27:26,421 Speaker 1: what is allowable, not, not, I come to my prof 675 00:27:26,421 --> 00:27:28,631 Speaker 1: and I ask you what can I use it for? 676 00:27:28,781 --> 00:27:30,181 Speaker 1: What are you allowing me to use but I should 677 00:27:30,182 --> 00:27:33,592 Speaker 1: assignment to you. I will tell you explicitly. That's right, right. 678 00:27:33,722 --> 00:27:35,781 Speaker 1: And then that's the approach that actually the school is 679 00:27:35,781 --> 00:27:38,411 Speaker 1: pro proposing, which is that the profs figure it out 680 00:27:38,411 --> 00:27:40,661 Speaker 1: and communicate this clearly, but like I said. 681 00:27:41,310 --> 00:27:43,510 Speaker 1: Be be kind to the props. I mean, we have 682 00:27:43,510 --> 00:27:45,319 Speaker 1: enough students to do with, you know, this is a 683 00:27:45,319 --> 00:27:48,260 Speaker 1: new thing. It will take some time, I think, for, for, 684 00:27:48,290 --> 00:27:51,109 Speaker 1: you know, the vast majority to actually, you know, figure 685 00:27:51,109 --> 00:27:54,310 Speaker 1: this out, and, and, but generally I think the, the 686 00:27:54,310 --> 00:27:57,510 Speaker 1: guidelines is that the school does, uh, ask that the 687 00:27:57,510 --> 00:28:01,739 Speaker 1: instructors be explicit what is the AI policy, right? And 688 00:28:01,739 --> 00:28:03,389 Speaker 1: generally like I said, yeah you can use it, you 689 00:28:03,390 --> 00:28:04,909 Speaker 1: cannot use it, you can use it normally you need 690 00:28:04,910 --> 00:28:06,750 Speaker 1: to declare. So at the end of the day, if 691 00:28:06,750 --> 00:28:07,510 Speaker 1: it's clear. 692 00:28:08,380 --> 00:28:10,410 Speaker 1: And everybody knows what is allowed, what is not allowed, 693 00:28:10,449 --> 00:28:13,329 Speaker 1: then there's no argument. No no no not so simple. 694 00:28:14,760 --> 00:28:14,849 Speaker 1: Yeah 695 00:28:17,310 --> 00:28:17,429 Speaker 1: We 696 00:28:17,430 --> 00:28:18,459 Speaker 2: have run out of time 697 00:28:18,459 --> 00:28:22,909 Speaker 1: that it's clear. Unfortunately, the students these days, they will 698 00:28:22,910 --> 00:28:24,469 Speaker 1: try to wriggle. 699 00:28:25,109 --> 00:28:28,270 Speaker 1: The way around that. I have no comment on that 700 00:28:28,270 --> 00:28:30,510 Speaker 1: because we don't know the case well. I can't agree 701 00:28:30,510 --> 00:28:32,550 Speaker 1: or disagree whether it's clear or not, so we will 702 00:28:32,550 --> 00:28:34,109 Speaker 1: have to let it play out on its own and 703 00:28:34,109 --> 00:28:36,589 Speaker 1: we will hear what the final outcome is. I'm sure 704 00:28:36,589 --> 00:28:38,349 Speaker 1: we'll get there. A, it's here to stay and it's 705 00:28:38,349 --> 00:28:39,869 Speaker 1: a part of our lives, so there's no running away 706 00:28:39,869 --> 00:28:40,359 Speaker 1: from it. OK. Alright. 707 00:28:42,390 --> 00:28:42,709 Speaker 1: And 708 00:28:42,709 --> 00:28:45,709 Speaker 2: uh that wraps up this week's edition of Deep Dive. 709 00:28:45,910 --> 00:28:51,510 Speaker 2: Big thanks to Tiffany Ag, Junai Juhari, Joann Chan, Sae, Hoeing. 710 00:28:52,069 --> 00:28:56,910 Speaker 2: Shaza Dalila, Ellison Jenner and video by Marcus Ramos, Faith 711 00:28:56,910 --> 00:28:58,650 Speaker 2: Ho and Hanida Armin. 712 00:28:58,810 --> 00:29:00,670 Speaker 1: Thank you. You know that's one day we're gonna take 713 00:29:00,670 --> 00:29:03,229 Speaker 1: a video and show you who's in the team because 714 00:29:03,229 --> 00:29:06,000 Speaker 1: you're probably thinking, who are these people, you know, behind us, 715 00:29:06,010 --> 00:29:07,630 Speaker 1: of course, with everything we do here is a big 716 00:29:07,630 --> 00:29:09,869 Speaker 1: team effort. So thanks so much for joining us. Any comments, 717 00:29:09,989 --> 00:29:11,910 Speaker 1: drop us a note. Love to hear from you. See 718 00:29:11,910 --> 00:29:12,270 Speaker 1: you next week.