1
00:00:03,160 --> 00:00:07,480
Speaker 1: Bloomberg Audio Studios, podcasts, radio news.

2
00:00:08,400 --> 00:00:12,160
Speaker 2: Moira Olmsted dreams of becoming an elementary school teacher, so

3
00:00:12,280 --> 00:00:15,520
Speaker 2: last year she enrolled in an online program at Central

4
00:00:15,560 --> 00:00:19,400
Speaker 2: Methodist University, working toward a degree while taking care of

5
00:00:19,440 --> 00:00:22,720
Speaker 2: for toddler. For one of her classes, Moira had to

6
00:00:22,760 --> 00:00:27,400
Speaker 2: turn in weekly writing assignments summarizing news articles, which was

7
00:00:27,680 --> 00:00:32,360
Speaker 2: easy enough, but just weeks into the fall semester, she

8
00:00:32,440 --> 00:00:34,120
Speaker 2: got an unexpected grade back.

9
00:00:34,280 --> 00:00:36,600
Speaker 3: Early on in the semester, in one of her courses,

10
00:00:37,040 --> 00:00:39,040
Speaker 3: she had received a zero.

11
00:00:39,640 --> 00:00:43,160
Speaker 2: Jackie Devallos is a tech reporter for Bloomberg. She spoke

12
00:00:43,200 --> 00:00:44,720
Speaker 2: to Moira over the phone.

13
00:00:44,680 --> 00:00:49,760
Speaker 4: And I've might completely freaked out there those summary articles. Mainly,

14
00:00:49,800 --> 00:00:52,879
Speaker 4: what we're doing is taking a bunch of information and

15
00:00:53,000 --> 00:00:57,160
Speaker 4: just summarizing in maybe two three paragraph smacks.

16
00:00:58,040 --> 00:01:00,320
Speaker 3: She didn't really know what had happened, and she just

17
00:01:00,400 --> 00:01:03,080
Speaker 3: kind of saw it pop up on her student portal.

18
00:01:03,520 --> 00:01:06,360
Speaker 3: She raised this with the professor, and the professor I

19
00:01:06,480 --> 00:01:11,119
Speaker 3: told her that she had been flagged for using AI.

20
00:01:11,760 --> 00:01:15,160
Speaker 1: She was like, hey, I run everybody's work through an

21
00:01:15,200 --> 00:01:20,360
Speaker 1: AI detector, and yours has been flagged many times.

22
00:01:20,800 --> 00:01:22,200
Speaker 4: It's just getting out of hand.

23
00:01:22,640 --> 00:01:26,800
Speaker 2: But Moira says she never used generative AI tools like

24
00:01:26,880 --> 00:01:29,320
Speaker 2: chat GPT to write her assignments.

25
00:01:29,400 --> 00:01:34,440
Speaker 3: To her, it was being blindsided by this technology that

26
00:01:34,800 --> 00:01:39,160
Speaker 3: she had never really seen before, and Moira immediately followed

27
00:01:39,240 --> 00:01:41,880
Speaker 3: up asking for additional details as to how this could

28
00:01:41,920 --> 00:01:42,480
Speaker 3: have happened.

29
00:01:42,680 --> 00:01:44,720
Speaker 1: I was just like, okay, thanks for bringing that to

30
00:01:44,760 --> 00:01:48,680
Speaker 1: my attention, Like I'm actually you know, I'm a future educator.

31
00:01:48,800 --> 00:01:54,200
Speaker 1: I really am super against the use of AI, specifically

32
00:01:54,360 --> 00:01:56,880
Speaker 1: in opinion and like thought pieces.

33
00:01:57,360 --> 00:02:01,600
Speaker 3: She had to raise it to several administrators at her school.

34
00:02:01,680 --> 00:02:05,760
Speaker 3: She had multiple meetings emails that showed you this back

35
00:02:05,800 --> 00:02:11,240
Speaker 3: and forth between her expressing this confidence in her work

36
00:02:11,800 --> 00:02:14,840
Speaker 3: that just was being put into question.

37
00:02:15,000 --> 00:02:19,680
Speaker 2: Basically, her grade was ultimately changed, but Moira started to

38
00:02:19,720 --> 00:02:24,040
Speaker 2: take extra precautions with her work, putting it through AI checkers, herself,

39
00:02:24,280 --> 00:02:28,000
Speaker 2: screen recording her progress, and attaching the recordings to her assignments,

40
00:02:28,680 --> 00:02:32,840
Speaker 2: anything to prove her work was original. But Jackie's reporting

41
00:02:32,919 --> 00:02:36,760
Speaker 2: found there was another reason Moira's work may have been

42
00:02:36,800 --> 00:02:39,240
Speaker 2: mistakenly flagged as AI generated.

43
00:02:40,680 --> 00:02:44,280
Speaker 3: She's on the autism spectrum and she's always written with

44
00:02:44,360 --> 00:02:48,360
Speaker 3: somewhat of a formulaic kind of style, and so Moira,

45
00:02:48,840 --> 00:02:52,880
Speaker 3: understanding that this might be perhaps one of the gaps

46
00:02:53,160 --> 00:02:57,240
Speaker 3: that AI detectors have, she knew she wanted to be

47
00:02:57,520 --> 00:03:00,200
Speaker 3: armed with proof that she had completed her work in

48
00:03:00,240 --> 00:03:04,239
Speaker 3: case this came up again. Students who fall into this category,

49
00:03:04,440 --> 00:03:08,440
Speaker 3: either their neuro divergent or English as their second language,

50
00:03:08,480 --> 00:03:12,400
Speaker 3: they tend to get picked up more than their peers

51
00:03:12,560 --> 00:03:14,840
Speaker 3: who might not fall into these categories.

52
00:03:15,360 --> 00:03:18,720
Speaker 2: Moira is just one student grappling with the challenges presented

53
00:03:18,760 --> 00:03:22,400
Speaker 2: by this new frontier in education, and those challenges are

54
00:03:22,440 --> 00:03:26,440
Speaker 2: playing out at schools and universities all over the country.

55
00:03:28,480 --> 00:03:30,800
Speaker 2: I'm Sarah Holder, and this is the big take from

56
00:03:30,840 --> 00:03:34,920
Speaker 2: Bloomberg News today. On the show, how universities and students

57
00:03:34,960 --> 00:03:38,400
Speaker 2: are adapting to the emergence of generative AI and what

58
00:03:38,480 --> 00:03:44,720
Speaker 2: happens when efforts to crack down on its use backfire.

59
00:03:46,800 --> 00:03:50,440
Speaker 2: Jackie Moira wasn't using generative AI to do her homework.

60
00:03:50,480 --> 00:03:54,240
Speaker 2: She's adamant that she wasn't, But other students are using

61
00:03:54,280 --> 00:03:58,120
Speaker 2: tools like chat GPT to help write their papers. Can

62
00:03:58,160 --> 00:04:00,360
Speaker 2: you just give us a sense of how big of

63
00:04:00,400 --> 00:04:03,120
Speaker 2: a thing is this really? Right now?

64
00:04:03,480 --> 00:04:07,200
Speaker 3: It's huge. Some students like to use tools just for

65
00:04:07,280 --> 00:04:09,680
Speaker 3: the spell check just for the syntax, and then a

66
00:04:09,760 --> 00:04:13,560
Speaker 3: level up to help me rewrite this one section to

67
00:04:13,680 --> 00:04:17,320
Speaker 3: all the way to just help me write my entire essay.

68
00:04:17,920 --> 00:04:22,719
Speaker 3: And this is where you've seen this other cottage industry

69
00:04:22,720 --> 00:04:27,200
Speaker 3: of startups and tools crop up to help basically detect

70
00:04:27,200 --> 00:04:27,800
Speaker 3: against that.

71
00:04:28,240 --> 00:04:31,320
Speaker 2: I want to know more about how these AI detectors

72
00:04:31,440 --> 00:04:35,159
Speaker 2: come to determine whether and how much a student has

73
00:04:35,279 --> 00:04:38,080
Speaker 2: used AI and their writing or in their homework assignments.

74
00:04:38,400 --> 00:04:40,520
Speaker 2: How do these tools work at a basic level?

75
00:04:41,320 --> 00:04:46,359
Speaker 3: AI detection software like turn it in, copy Leaks, and

76
00:04:46,480 --> 00:04:49,359
Speaker 3: GPT zero, which were some of the other startups that

77
00:04:49,400 --> 00:04:54,320
Speaker 3: we had looked at, basically use technology not so dissimilar

78
00:04:54,600 --> 00:04:58,920
Speaker 3: from that of a chat GBT. They train their systems

79
00:04:59,279 --> 00:05:01,960
Speaker 3: off of just a lot of text in the same

80
00:05:02,000 --> 00:05:06,920
Speaker 3: way that chat gipd does. However, AI writing detectors look

81
00:05:06,960 --> 00:05:10,680
Speaker 3: at what's called perplexity, and this is just a fancy

82
00:05:10,880 --> 00:05:15,400
Speaker 3: term for a measure of how complex the words are

83
00:05:15,800 --> 00:05:19,200
Speaker 3: in any given submission or a sentence or a paragraph.

84
00:05:19,720 --> 00:05:22,960
Speaker 3: We speak with a lot of variety. We vary our

85
00:05:23,040 --> 00:05:28,040
Speaker 3: sentence structure and diction throughout a particular sentence or passage.

86
00:05:28,520 --> 00:05:31,680
Speaker 3: If word choices are a little bit more generic and

87
00:05:31,720 --> 00:05:34,960
Speaker 3: formulaic that's going to have a higher chance of being

88
00:05:35,000 --> 00:05:39,719
Speaker 3: flagged by an AI detector. And it basically spits out

89
00:05:40,680 --> 00:05:45,960
Speaker 3: percentage of how much it believes the assignment is AI generated,

90
00:05:46,160 --> 00:05:49,240
Speaker 3: and so in Moiris's case, it was a majority of

91
00:05:49,279 --> 00:05:53,280
Speaker 3: It doesn't highlight which passages. It also doesn't give you

92
00:05:53,320 --> 00:05:55,560
Speaker 3: an answer for how it got there. It's kind of

93
00:05:55,560 --> 00:05:56,600
Speaker 3: this black box.

94
00:05:58,279 --> 00:06:01,880
Speaker 2: So educators are using these AI powered detectors to root

95
00:06:01,920 --> 00:06:06,320
Speaker 2: out AI powered papers, but how well do these tools

96
00:06:06,839 --> 00:06:07,640
Speaker 2: actually work?

97
00:06:08,360 --> 00:06:13,240
Speaker 3: We found that they're actually highly accurate. So we tested

98
00:06:13,640 --> 00:06:18,919
Speaker 3: GPT zero and copy leaks on a random sample of

99
00:06:19,120 --> 00:06:23,480
Speaker 3: five hundred college application essays that were submitted to Texas

100
00:06:23,520 --> 00:06:25,839
Speaker 3: A and M in the summer of twenty twenty two.

101
00:06:26,520 --> 00:06:30,039
Speaker 3: This is important because, as we know, chat GPT was

102
00:06:30,080 --> 00:06:34,880
Speaker 3: released in the fall of twenty twenty two, so we

103
00:06:34,960 --> 00:06:39,400
Speaker 3: know that these essays were not AI generated because chat

104
00:06:39,440 --> 00:06:43,360
Speaker 3: GPT hadn't even been released yet. After running the analysis,

105
00:06:43,440 --> 00:06:47,960
Speaker 3: we found that these startups falsely flagged about one to

106
00:06:48,080 --> 00:06:52,000
Speaker 3: two percent of the essays as likely written by AI,

107
00:06:52,760 --> 00:06:55,520
Speaker 3: and in some cases they claim to have near one

108
00:06:55,640 --> 00:06:59,159
Speaker 3: hundred percent certainty. But the problem is one to two

109
00:06:59,200 --> 00:07:04,320
Speaker 3: percent of essays is still high in some ways, and

110
00:07:04,400 --> 00:07:07,920
Speaker 3: that small error rate can add up just given how

111
00:07:07,960 --> 00:07:11,760
Speaker 3: many student assignments are submitted throughout the year across the country.

112
00:07:12,240 --> 00:07:15,960
Speaker 2: Yet two out of every one hundred students running the

113
00:07:16,040 --> 00:07:19,440
Speaker 2: risk of being mistakenly accused of plagiarism, maybe getting expelled

114
00:07:20,200 --> 00:07:23,720
Speaker 2: still feels pretty bad. Who is this affecting most?

115
00:07:24,120 --> 00:07:28,480
Speaker 3: We found two groups and that can be particularly vulnerable

116
00:07:28,760 --> 00:07:33,360
Speaker 3: to some of the flaws in AI detection software. One

117
00:07:33,520 --> 00:07:37,520
Speaker 3: is if you're neurodivergent like Moira, if you're on the spectrum,

118
00:07:37,600 --> 00:07:42,000
Speaker 3: for example. Another is if English is your second language.

119
00:07:42,640 --> 00:07:48,080
Speaker 2: How disproportionately are these kinds of students being impacted by

120
00:07:48,120 --> 00:07:49,480
Speaker 2: these false flags?

121
00:07:50,200 --> 00:07:55,200
Speaker 3: Stanford researchers found that AI detectors were almost perfect when

122
00:07:55,280 --> 00:07:59,720
Speaker 3: checking essays written by US born eighth grade students, but

123
00:07:59,800 --> 00:08:03,360
Speaker 3: they flagged over half of those essays written by non

124
00:08:03,480 --> 00:08:07,680
Speaker 3: native English speakers as AI generated. So the false flag

125
00:08:07,760 --> 00:08:09,760
Speaker 3: there is extremely high.

126
00:08:09,880 --> 00:08:12,760
Speaker 2: What about the impact on professors themselves? Is it making

127
00:08:12,800 --> 00:08:15,840
Speaker 2: them more skeptical and more paranoid of the work that

128
00:08:15,920 --> 00:08:17,000
Speaker 2: students are turning in.

129
00:08:17,640 --> 00:08:22,600
Speaker 3: On the whole, professors are still a little bit on

130
00:08:22,640 --> 00:08:26,000
Speaker 3: the fence about how exactly AI should be used in

131
00:08:26,040 --> 00:08:30,280
Speaker 3: the classroom. You have some who are wanting to incorporate

132
00:08:30,320 --> 00:08:34,319
Speaker 3: it into aspects of the curriculum, like using it to

133
00:08:34,720 --> 00:08:38,200
Speaker 3: help you brainstorm or do some of the initial research.

134
00:08:38,760 --> 00:08:42,000
Speaker 3: Other professors are telling me, we don't mind if you

135
00:08:42,080 --> 00:08:47,640
Speaker 3: want to have chadgbt write this aspect of your essay,

136
00:08:47,960 --> 00:08:51,720
Speaker 3: just cite it appropriately. Professors are trying to figure out

137
00:08:52,520 --> 00:08:55,720
Speaker 3: at what point does AI kind of erode the experience

138
00:08:55,760 --> 00:08:59,440
Speaker 3: of learning and at what point does it actually help it.

139
00:08:59,559 --> 00:09:02,080
Speaker 3: But if they're there's one thing that professors agree on,

140
00:09:02,160 --> 00:09:04,760
Speaker 3: it's that it's not going anywhere.

141
00:09:06,120 --> 00:09:10,480
Speaker 2: AI isn't going anywhere, But how can students and educators

142
00:09:10,840 --> 00:09:20,800
Speaker 2: use the technology responsibly? That's after the break, We're back.

143
00:09:21,120 --> 00:09:24,360
Speaker 2: I've been speaking with Bloomberg reporter Jackie Devalis about the

144
00:09:24,400 --> 00:09:28,240
Speaker 2: shortcomings of software that colleges and universities are using to

145
00:09:28,320 --> 00:09:32,520
Speaker 2: detect and root out AI generated work. Are they trying

146
00:09:32,559 --> 00:09:37,000
Speaker 2: to set new policies to incorporate the understanding that these

147
00:09:37,120 --> 00:09:39,920
Speaker 2: AI detection tools have some blind spots.

148
00:09:40,440 --> 00:09:44,880
Speaker 3: Definitely. You know, you're seeing some schools put down firmer

149
00:09:44,960 --> 00:09:49,280
Speaker 3: policies around what's considered plagiarism if you use chatchipt for

150
00:09:49,320 --> 00:09:51,520
Speaker 3: a part of your essay and don't cite it, that

151
00:09:51,559 --> 00:09:54,360
Speaker 3: can be considered plagiarism. But if you do cite it,

152
00:09:54,679 --> 00:10:00,280
Speaker 3: then it's okay. And others are basically allowing professor to

153
00:10:00,400 --> 00:10:04,280
Speaker 3: use these AI detection tools however they please without actually

154
00:10:04,320 --> 00:10:09,240
Speaker 3: saying if your essay is fifty percent AI generated or

155
00:10:09,520 --> 00:10:14,080
Speaker 3: ninety eight percent AI generated, then you will face a consequence.

156
00:10:14,120 --> 00:10:17,839
Speaker 3: So it's left up to the professor to what's acceptable.

157
00:10:18,240 --> 00:10:21,560
Speaker 3: But some universities are really mindful of the fact that

158
00:10:21,640 --> 00:10:24,400
Speaker 3: these AI detectors aren't completely accurate.

159
00:10:25,160 --> 00:10:27,680
Speaker 2: What are students doing to make sure that their original

160
00:10:27,720 --> 00:10:29,680
Speaker 2: work isn't mistaken for AI?

161
00:10:30,200 --> 00:10:34,320
Speaker 3: Students are really starting to get creative with how they

162
00:10:34,360 --> 00:10:38,160
Speaker 3: can protect themselves. Many of them told me that, like Moira,

163
00:10:38,720 --> 00:10:41,520
Speaker 3: they're starting to do their work in Google docs and

164
00:10:41,960 --> 00:10:46,760
Speaker 3: tracking everything to create this digital paper trail. Others tell

165
00:10:46,760 --> 00:10:50,200
Speaker 3: me that they're using other tech tools out there that

166
00:10:50,280 --> 00:10:54,680
Speaker 3: are almost created as a way to humanize your text.

167
00:10:54,880 --> 00:10:57,840
Speaker 3: I had a conversation with a student who went to

168
00:10:57,840 --> 00:11:01,520
Speaker 3: school in California and telling me how he tweaks his

169
00:11:02,120 --> 00:11:05,880
Speaker 3: wording in some parts of an essay to actually sound

170
00:11:06,000 --> 00:11:09,199
Speaker 3: worse because he's afraid that if it sounds too good

171
00:11:09,880 --> 00:11:12,360
Speaker 3: then that it might get caught by an AI detector.

172
00:11:12,960 --> 00:11:17,120
Speaker 2: This all sounds like so much work for students and

173
00:11:17,160 --> 00:11:20,320
Speaker 2: for educators to kind of work around the blind spots

174
00:11:20,360 --> 00:11:23,559
Speaker 2: that this technology has. What are companies trying to do

175
00:11:23,880 --> 00:11:25,320
Speaker 2: to improve their models.

176
00:11:25,679 --> 00:11:28,640
Speaker 3: We spoke to almost all of the companies that we

177
00:11:28,720 --> 00:11:31,559
Speaker 3: looked at, and what they told us is that they

178
00:11:31,600 --> 00:11:38,640
Speaker 3: actually intentionally oversample underrepresented groups like students who might not

179
00:11:38,720 --> 00:11:42,000
Speaker 3: be native English speakers, and that because of that, it's

180
00:11:42,160 --> 00:11:46,480
Speaker 3: kind of this ever evolving process of iterating and making

181
00:11:46,520 --> 00:11:49,440
Speaker 3: it more accurate. We also spoke to the copy leaks

182
00:11:50,000 --> 00:11:52,800
Speaker 3: co founder and CEO who told us that they're ninety

183
00:11:52,880 --> 00:11:55,440
Speaker 3: nine percent accurate, but still a small number of errors

184
00:11:55,480 --> 00:11:59,000
Speaker 3: can occur from time to time. GPT zero was another

185
00:11:59,040 --> 00:12:01,880
Speaker 3: company who told us that they're actually coming out with

186
00:12:01,920 --> 00:12:06,120
Speaker 3: another tool that is almost like a tool that students

187
00:12:06,160 --> 00:12:09,040
Speaker 3: will be able to write into, and it not only

188
00:12:09,280 --> 00:12:13,800
Speaker 3: tracks your work, it has time stamps around when you

189
00:12:14,000 --> 00:12:15,840
Speaker 3: entered the document, when you exited.

190
00:12:16,200 --> 00:12:19,440
Speaker 2: So these companies are creating the problem and then offering

191
00:12:19,480 --> 00:12:20,640
Speaker 2: solutions for the problem.

192
00:12:20,880 --> 00:12:24,520
Speaker 3: In some ways, yes, it's funny because it also shows

193
00:12:24,520 --> 00:12:29,080
Speaker 3: that they acknowledge that the detection software itself is imperfect.

194
00:12:29,480 --> 00:12:33,600
Speaker 3: The thing that these companies emphasize is that they're now

195
00:12:33,640 --> 00:12:38,400
Speaker 3: trying to get that professor feedback and relate to them too,

196
00:12:38,760 --> 00:12:42,600
Speaker 3: that this isn't the end all, be all tool that

197
00:12:42,640 --> 00:12:45,199
Speaker 3: you should use to grade your student's work.

198
00:12:46,800 --> 00:12:49,520
Speaker 2: Jackie, My last question for you is just about Moira.

199
00:12:50,160 --> 00:12:52,640
Speaker 2: How is she doing now? Has she finished her studies

200
00:12:52,679 --> 00:12:54,880
Speaker 2: and is she becoming a teacher herself.

201
00:12:55,400 --> 00:13:00,120
Speaker 3: She's on track to continue her coursework this semester, a

202
00:13:00,160 --> 00:13:05,600
Speaker 3: mom of two now, and she's really excited about what's ahead.

203
00:13:06,160 --> 00:13:08,840
Speaker 3: It's an ever evolving world, and she tells us that

204
00:13:09,360 --> 00:13:14,560
Speaker 3: despite this overwhelming incident, which was unfortunate, she's still looking

205
00:13:14,600 --> 00:13:16,360
Speaker 3: forward to being an educator in the future.

206
00:13:19,920 --> 00:13:23,640
Speaker 2: Thank you so much, Jackie, Thank you. This is the

207
00:13:23,640 --> 00:13:27,200
Speaker 2: Big Take from Bloomberg News. I'm Sarah Holder. This episode

208
00:13:27,240 --> 00:13:29,800
Speaker 2: was produced by Thomas lou and Jessica Beck. It was

209
00:13:29,920 --> 00:13:33,160
Speaker 2: edited by Aaron Edwards and Seth Fiegerman. It was mixed

210
00:13:33,200 --> 00:13:36,360
Speaker 2: by Alex Suguia. It was fact checked by Adrianna Tapia.

211
00:13:36,920 --> 00:13:40,520
Speaker 2: Our senior producer is Naomi Shavin, who also edited this episode.

212
00:13:40,679 --> 00:13:44,319
Speaker 2: Our senior editor is Elizabeth Ponso, Our executive producer is

213
00:13:44,400 --> 00:13:48,280
Speaker 2: Nicole Beamster Boor Sage Bouman is Bloomberg's head of podcasts.

214
00:13:48,840 --> 00:13:51,360
Speaker 2: If you liked this episode, make sure to subscribe and

215
00:13:51,440 --> 00:13:54,400
Speaker 2: review The Big Take wherever you get your podcasts. It

216
00:13:54,440 --> 00:13:57,680
Speaker 2: helps people find the show. Thanks for listening, we'll be

217
00:13:57,760 --> 00:13:59,000
Speaker 2: back next week.

218
00:14:00,040 --> 00:14:00,559
Speaker 3: Hello,