1 00:00:15,396 --> 00:00:33,356 Speaker 1: Pushkin. I'm Jacob Goldstein, and this is What's Your Problem? 2 00:00:33,616 --> 00:00:36,976 Speaker 1: My guest today is Max Spiro. He's the co-founder and 3 00:00:37,036 --> 00:00:40,936 Speaker 1: CEO of a company called Pangram. And Max's problem is this. 4 00:00:41,656 --> 00:00:44,566 Speaker 1: How do you build a tool that that reliably identifies 5 00:00:44,686 --> 00:00:47,826 Speaker 1: whether a piece of writing or an image was created 6 00:00:47,926 --> 00:00:51,706 Speaker 1: by AI. If you've heard about any of the recent 7 00:00:51,806 --> 00:00:56,376 Speaker 1: controversies over whether something was written by AI, whether it 8 00:00:56,436 --> 00:01:01,696 Speaker 1: was a prize-winning novel or an op-ed in a major newspaper, 9 00:01:02,296 --> 00:01:05,076 Speaker 1: there's a good chance that you've actually heard about Pangram, 10 00:01:05,216 --> 00:01:07,996 Speaker 1: whether you know it or not, because Pangram is the 11 00:01:08,036 --> 00:01:12,226 Speaker 1: tool people almost always cite in those cases. It is 12 00:01:12,306 --> 00:01:18,306 Speaker 1: generally viewed as the most reliable, widely available AI identification tool. 13 00:01:18,916 --> 00:01:21,316 Speaker 1: Max and I talked about a lot of different things 14 00:01:21,436 --> 00:01:25,376 Speaker 1: in our conversation. We talked about changing norms around when 15 00:01:25,416 --> 00:01:28,076 Speaker 1: it's OK to use AI and when it's not OK. 16 00:01:28,816 --> 00:01:31,996 Speaker 1: We talked about the risk of false positives, of PanGram 17 00:01:32,216 --> 00:01:35,976 Speaker 1: wrongly accusing people of using AI to write things. And 18 00:01:36,036 --> 00:01:39,056 Speaker 1: we talked about the new PanGram tool to detect whether 19 00:01:39,136 --> 00:01:42,656 Speaker 1: an image was AI generated. But to start, we talked 20 00:01:42,676 --> 00:01:45,996 Speaker 1: about two domains where PanGram is already in wide use, 21 00:01:46,676 --> 00:01:51,556 Speaker 1: education and publishing. Tell me about PanGram and sort of 22 00:01:51,696 --> 00:01:54,266 Speaker 1: where you are now. Who are your customers? Who's using it? 23 00:01:54,676 --> 00:01:58,536 Speaker 2: Our customers, a bunch of customers in education, higher ed, 24 00:01:58,576 --> 00:02:01,916 Speaker 2: and also like high schools, K-12, they want to know 25 00:02:02,096 --> 00:02:05,946 Speaker 2: whether their students are submitting AI-generated essays. They want a 26 00:02:05,966 --> 00:02:12,596 Speaker 2: way to teach writing or, like, have any writing-based assignments 27 00:02:13,136 --> 00:02:16,596 Speaker 2: without having it sort of just devolve into students using AI. 28 00:02:16,676 --> 00:02:20,346 Speaker 1: It seems like an extremely strong case. Like, you know, 29 00:02:20,466 --> 00:02:23,146 Speaker 1: writing is a way of thinking hard about things. I 30 00:02:23,226 --> 00:02:26,726 Speaker 1: buy that it is useful still for students to write. 31 00:02:27,906 --> 00:02:31,806 Speaker 1: My sense is that students are using AI a lot and, 32 00:02:32,046 --> 00:02:34,306 Speaker 1: you know, going to great lengths to sort of humanize 33 00:02:34,356 --> 00:02:36,536 Speaker 1: what the AI says, to sort of write through AI 34 00:02:37,096 --> 00:02:38,616 Speaker 1: or do what they have to do. Like, what is 35 00:02:38,656 --> 00:02:39,956 Speaker 1: your sense of how it is going? 36 00:02:40,316 --> 00:02:42,956 Speaker 2: Yeah, something like pangram here would be more of like, 37 00:02:43,366 --> 00:02:46,986 Speaker 2: we're adding friction to the process of cheating. And, you know, 38 00:02:47,466 --> 00:02:49,546 Speaker 2: it's going to make it less enticing if you have 39 00:02:49,606 --> 00:02:52,256 Speaker 2: to go start with AI bones and rewrite it. 40 00:02:52,426 --> 00:02:54,366 Speaker 1: Yes, you can't one-shot your assignment. 41 00:02:54,386 --> 00:02:54,886 Speaker 2: Exactly. 42 00:02:55,086 --> 00:02:55,846 Speaker 1: Yeah, yeah. 43 00:02:55,966 --> 00:02:59,426 Speaker 2: And that's helpful in some way. And, you know, you 44 00:02:59,466 --> 00:03:02,466 Speaker 2: can encourage students to, you know, start from scratch and 45 00:03:02,486 --> 00:03:03,346 Speaker 2: not use AI at all. 46 00:03:03,386 --> 00:03:06,436 Speaker 1: But I mean, certainly there are teachers who are requiring 47 00:03:06,496 --> 00:03:10,086 Speaker 1: students to do their writing in the room. Right? Like, 48 00:03:10,206 --> 00:03:13,066 Speaker 1: they're just going hard at, you have to sit here 49 00:03:13,106 --> 00:03:14,896 Speaker 1: in front of me. So what did somebody call it? 50 00:03:14,986 --> 00:03:16,906 Speaker 1: AI jail or something? You have to sit here in 51 00:03:16,956 --> 00:03:18,016 Speaker 1: front of me and write an essay. 52 00:03:18,076 --> 00:03:20,796 Speaker 2: It's kind of a bummer. Like, I feel like we 53 00:03:20,836 --> 00:03:24,336 Speaker 2: lose a lot here. We lose the in-class instruction. We 54 00:03:24,456 --> 00:03:29,496 Speaker 2: lose the ability to assign longer written pieces. It's true. 55 00:03:29,516 --> 00:03:31,136 Speaker 2: When I was in college, I wrote a lot of 56 00:03:31,156 --> 00:03:33,526 Speaker 2: like eight and 10, 12 page essays. 57 00:03:34,406 --> 00:03:34,706 Speaker 1: Yeah. 58 00:03:34,746 --> 00:03:37,146 Speaker 2: Like those, you can't do that in class. That takes 59 00:03:37,186 --> 00:03:37,706 Speaker 2: a lot of hours. 60 00:03:38,266 --> 00:03:40,546 Speaker 1: I mean, but that's just the nature of the world now, right? Like, 61 00:03:40,626 --> 00:03:41,766 Speaker 1: it sort of is what it is. 62 00:03:42,406 --> 00:03:45,876 Speaker 2: Yeah, and it's a little bit adversarial, as many of 63 00:03:46,116 --> 00:03:49,316 Speaker 2: our use cases are, where, like, the way that Pangram 64 00:03:49,336 --> 00:03:51,976 Speaker 2: is used is somebody is trying to get away with 65 00:03:52,016 --> 00:03:55,076 Speaker 2: AI writing in a way. And so there is this 66 00:03:55,196 --> 00:03:56,256 Speaker 2: adversarial component. 67 00:03:56,396 --> 00:03:58,936 Speaker 1: Yes. I mean, it's got to be mostly adversarial, right? 68 00:03:58,956 --> 00:04:02,206 Speaker 1: The weird non-adversarial one would be, I actually wrote this thing, 69 00:04:02,326 --> 00:04:04,266 Speaker 1: and I don't want people to think it's AI, so 70 00:04:04,286 --> 00:04:06,006 Speaker 1: I'm going to put it in Pangram, right? Like, that's 71 00:04:06,486 --> 00:04:10,686 Speaker 1: an edge case. Like, I've started using fewer em dashes, actually. 72 00:04:10,826 --> 00:04:12,806 Speaker 2: Oh, man. You got to use the em dashes. 73 00:04:13,306 --> 00:04:17,526 Speaker 1: Journalists love em dashes. And actually, maybe because I love them, 74 00:04:17,606 --> 00:04:21,076 Speaker 1: I didn't even notice it in AI pros. But once 75 00:04:21,116 --> 00:04:23,396 Speaker 1: everybody started talking about the em dash as a giveaway, 76 00:04:23,496 --> 00:04:27,056 Speaker 1: I just started doing, like, shorter independent clauses, basically. 77 00:04:27,176 --> 00:04:30,096 Speaker 2: Em dashes are, like, they're no longer an AI tell. 78 00:04:30,416 --> 00:04:34,426 Speaker 2: This was, like, a GPT-4 era thing. maybe like 4.0, 79 00:04:34,426 --> 00:04:37,286 Speaker 2: really overuse the em dash. We're over it now. Em 80 00:04:37,306 --> 00:04:38,626 Speaker 2: dashes are used a normal amount. 81 00:04:38,746 --> 00:04:40,616 Speaker 1: I can bring the em dash back? I can bring 82 00:04:40,676 --> 00:04:40,946 Speaker 1: it back? 83 00:04:40,976 --> 00:04:42,056 Speaker 2: Yes, you should bring it back. 84 00:04:42,496 --> 00:04:46,936 Speaker 1: Okay, great. So education is obviously a big use case. 85 00:04:47,556 --> 00:04:48,996 Speaker 2: I'll tell you about publishing as well. 86 00:04:49,376 --> 00:04:51,156 Speaker 1: Oh yeah, publishing. Well, so that one has been in 87 00:04:51,196 --> 00:04:53,616 Speaker 1: the news, right? There have been a couple of people, 88 00:04:53,776 --> 00:04:56,916 Speaker 1: one person lost a seven-figure book deal, right? Because, I 89 00:04:56,916 --> 00:04:59,536 Speaker 1: don't want to say because of you, but because they 90 00:04:59,576 --> 00:05:01,996 Speaker 1: were accused of using AI to write the book and 91 00:05:03,056 --> 00:05:04,176 Speaker 1: you were definitely involved in that. 92 00:05:04,636 --> 00:05:06,816 Speaker 2: Yeah, there have been a series of scandals this year 93 00:05:07,036 --> 00:05:10,696 Speaker 2: on AI use, either in books that are already published 94 00:05:10,836 --> 00:05:14,146 Speaker 2: or like big book deals. But kind of what it 95 00:05:14,436 --> 00:05:16,886 Speaker 2: comes down to is that a lot of people in 96 00:05:16,906 --> 00:05:19,466 Speaker 2: the industry are on the left side of the bell 97 00:05:19,506 --> 00:05:23,586 Speaker 2: curve where they're not heavy AI users. They're like, they're 98 00:05:23,786 --> 00:05:30,696 Speaker 2: book editors and publishers. They read like real books all day. 99 00:05:31,076 --> 00:05:34,636 Speaker 2: So they're not like, you know, talking to chat. So 100 00:05:34,896 --> 00:05:36,796 Speaker 2: I think like a lot of them, they like never 101 00:05:36,836 --> 00:05:39,476 Speaker 2: really built up this intuition in a way that like 102 00:05:39,556 --> 00:05:42,816 Speaker 2: all of a sudden, like AI getting good enough to 103 00:05:42,916 --> 00:05:46,616 Speaker 2: like produce publishable prose was like very surprising and kind 104 00:05:46,636 --> 00:05:47,636 Speaker 2: of caught them by surprise. 105 00:05:48,316 --> 00:05:53,566 Speaker 1: So recently, famous investor Stanley Druckenmiller wrote an op-ed in 106 00:05:53,586 --> 00:05:57,126 Speaker 1: the Wall Street Journal about the bond market. And everybody's like, wait, 107 00:05:57,306 --> 00:06:00,836 Speaker 1: AI wrote that. And then Druckenmiller was like, you're damn 108 00:06:00,876 --> 00:06:02,776 Speaker 1: right AI wrote that. I'm 80 years old and I 109 00:06:02,816 --> 00:06:05,696 Speaker 1: know how to use AI. And then maybe most interestingly, 110 00:06:06,356 --> 00:06:09,336 Speaker 1: the editor of the journal op-ed page was like, it's 111 00:06:09,376 --> 00:06:12,476 Speaker 1: fine that AI wrote it. Like Druckenmiller is a famous investor. 112 00:06:12,556 --> 00:06:15,966 Speaker 1: He really believes this. He stands behind it. Any famous 113 00:06:16,006 --> 00:06:19,046 Speaker 1: person writing an op-ed in the paper usually has somebody 114 00:06:19,066 --> 00:06:21,186 Speaker 1: else write it for him anyway. There's no problem here. 115 00:06:21,986 --> 00:06:23,846 Speaker 1: Sort of surprisingly, I might think I would have a 116 00:06:23,846 --> 00:06:26,386 Speaker 1: problem with it, but upon reflection, I don't. Do you 117 00:06:26,406 --> 00:06:27,026 Speaker 1: think it's bad? 118 00:06:27,806 --> 00:06:30,486 Speaker 2: I think the argument against is that it still does 119 00:06:30,926 --> 00:06:34,026 Speaker 2: a disservice to the readers. Like, if you read it, it's, like, 120 00:06:34,146 --> 00:06:36,066 Speaker 2: full of AI prose. 121 00:06:36,596 --> 00:06:39,056 Speaker 1: So your critique is as a piece of writing. It's 122 00:06:39,136 --> 00:06:41,256 Speaker 1: bad writing, and that is the criticism. 123 00:06:41,656 --> 00:06:44,976 Speaker 2: Yeah, and, like, the Wall Street Journal has great editors. 124 00:06:45,076 --> 00:06:47,236 Speaker 2: I'm sure, like, he could have just, like, written up 125 00:06:47,736 --> 00:06:51,816 Speaker 2: a draft himself that was, like, not particularly flowery or 126 00:06:51,836 --> 00:06:53,916 Speaker 2: whatever and then, you know, sent it to his editors and, like, 127 00:06:54,336 --> 00:06:58,476 Speaker 2: they help him work it into something that's, like, very readable. 128 00:06:58,916 --> 00:07:01,996 Speaker 1: You could also argue, like, it certainly would be less 129 00:07:02,036 --> 00:07:05,656 Speaker 1: objectionable if it just said at the bottom, this was 130 00:07:05,716 --> 00:07:08,696 Speaker 1: written with the assistance of AI, right? Like, that would 131 00:07:08,856 --> 00:07:11,516 Speaker 1: seem to, like, if you have any criticism, it's like, well, 132 00:07:11,536 --> 00:07:14,576 Speaker 1: you should tell me because it's weird. Yeah. Although you 133 00:07:14,616 --> 00:07:16,846 Speaker 1: could make the same argument for ghostwriters and, like, pretty 134 00:07:16,876 --> 00:07:21,266 Speaker 1: much anything you see written by a CEO, in most cases, 135 00:07:21,526 --> 00:07:23,866 Speaker 1: of a big company is going to be ghostwritten, right? Like, 136 00:07:23,886 --> 00:07:26,366 Speaker 1: there are people whose job is writing op-eds for CEOs. 137 00:07:26,666 --> 00:07:28,566 Speaker 2: I'm going to make this other argument now, which I 138 00:07:28,606 --> 00:07:31,226 Speaker 2: think is maybe more relevant. So I think there's like 139 00:07:31,286 --> 00:07:36,006 Speaker 2: two types of AI use and AI writing. There's basically 140 00:07:36,066 --> 00:07:39,466 Speaker 2: making something more concise or distilling. So for example, if 141 00:07:39,846 --> 00:07:42,726 Speaker 2: Druckenmiller's like, hey, I wrote like, you know, five pages, 142 00:07:42,786 --> 00:07:44,926 Speaker 2: here's all my research, here's all my thoughts. I need 143 00:07:45,026 --> 00:07:47,386 Speaker 2: AI to help me like make this concise and get 144 00:07:47,426 --> 00:07:49,946 Speaker 2: it into like op-ed size. And so I think typically 145 00:07:49,966 --> 00:07:53,786 Speaker 2: AI writing in this case ends up being quite helpful 146 00:07:53,826 --> 00:07:56,656 Speaker 2: because there's a lot to work with and it's distilling 147 00:07:56,696 --> 00:07:59,976 Speaker 2: it down to its core components. The other side of 148 00:07:59,996 --> 00:08:03,096 Speaker 2: AI writing that I see is expanding or padding, where 149 00:08:03,136 --> 00:08:07,156 Speaker 2: somebody's like, here's a few bullet points. Here's a few 150 00:08:07,196 --> 00:08:10,426 Speaker 2: things that I believe. Turn this into an email or 151 00:08:10,466 --> 00:08:14,006 Speaker 2: an op-ed. I think that's where it's really doing a 152 00:08:14,046 --> 00:08:17,416 Speaker 2: disservice to the readers because you're just putting more words 153 00:08:17,456 --> 00:08:19,396 Speaker 2: that are basically padding and don't mean that much. 154 00:08:19,416 --> 00:08:21,336 Speaker 1: Just publish the bullet points, bro. 155 00:08:21,376 --> 00:08:22,416 Speaker 2: Exactly, exactly. 156 00:08:22,436 --> 00:08:25,676 Speaker 1: Yeah, publish it in Axios. They love bullet points over there. 157 00:08:26,596 --> 00:08:29,416 Speaker 1: There's a broader thing that's interesting to me about the 158 00:08:30,016 --> 00:08:33,876 Speaker 1: Druckenmiller Wall Street Journal thing, and that is, like, it 159 00:08:33,956 --> 00:08:37,946 Speaker 1: does make me wonder to what extent we're right now 160 00:08:38,106 --> 00:08:42,566 Speaker 1: in a transitional moment where, you know, I could imagine 161 00:08:42,586 --> 00:08:44,006 Speaker 1: in a year, a few years. 162 00:08:45,456 --> 00:08:48,336 Speaker 1: In many settings, saying, did a person use AI to 163 00:08:48,356 --> 00:08:50,656 Speaker 1: write this, would be like today saying, did they use 164 00:08:50,736 --> 00:08:54,616 Speaker 1: the internet to do the research? Like, probably, and so 165 00:08:54,656 --> 00:08:59,216 Speaker 1: what are the answers? Does that seem plausible to you? 166 00:08:59,996 --> 00:09:03,226 Speaker 2: I think we're absolutely in this space or this time 167 00:09:03,236 --> 00:09:06,066 Speaker 2: period where the norms are completely up in the air 168 00:09:06,226 --> 00:09:09,586 Speaker 2: and people are so polarized on one side or the other, 169 00:09:09,606 --> 00:09:12,406 Speaker 2: or some people are like, I'm going to let AI 170 00:09:12,666 --> 00:09:15,666 Speaker 2: automate my entire online presence. And other people are swearing 171 00:09:15,986 --> 00:09:20,106 Speaker 2: they'll never use AI ever. And so we have this 172 00:09:20,146 --> 00:09:25,216 Speaker 2: really wide range of norms. Nobody's really settled on what 173 00:09:25,276 --> 00:09:29,896 Speaker 2: is acceptable versus unacceptable. I think AI and LLMs, they're 174 00:09:29,916 --> 00:09:33,216 Speaker 2: very powerful technology. They could do a lot of work 175 00:09:33,316 --> 00:09:36,096 Speaker 2: for me. They can go do research for me. They 176 00:09:36,116 --> 00:09:39,696 Speaker 2: can compile this into a spreadsheet for me to read 177 00:09:39,776 --> 00:09:40,746 Speaker 2: and peruse. 178 00:09:40,886 --> 00:09:43,676 Speaker 1: They can determine whether people are using AI to write. 179 00:09:44,696 --> 00:09:48,516 Speaker 2: Yeah. But yeah, I think the answer is we really 180 00:09:48,596 --> 00:09:53,276 Speaker 2: haven't settled on any norms societally. And the norms that 181 00:09:53,316 --> 00:09:56,196 Speaker 2: we will settle to are probably that some degree of 182 00:09:56,216 --> 00:09:59,806 Speaker 2: AI use is okay, but using AI to replace your 183 00:09:59,866 --> 00:10:03,966 Speaker 2: voice completely, especially in settings where it's going to be 184 00:10:04,046 --> 00:10:08,846 Speaker 2: read by thousands or millions, is probably less acceptable. Yeah. 185 00:10:08,986 --> 00:10:11,736 Speaker 1: Well, and disclosure. I do feel like disclosure can solve 186 00:10:11,776 --> 00:10:14,836 Speaker 1: a lot. It's also interesting to me. You see this 187 00:10:14,916 --> 00:10:17,986 Speaker 1: certainly on LinkedIn and on Twitter. There are a lot 188 00:10:18,016 --> 00:10:21,826 Speaker 1: of things that are so obviously AI. And initially, I 189 00:10:22,046 --> 00:10:24,526 Speaker 1: was like, don't people know? But now I think maybe 190 00:10:24,566 --> 00:10:27,446 Speaker 1: people know and don't care. Like, I don't know. It 191 00:10:27,486 --> 00:10:30,216 Speaker 1: does feel... I want to be clear, I don't like 192 00:10:30,376 --> 00:10:32,696 Speaker 1: AI prose. I'm not speaking of a moral objection, but 193 00:10:32,736 --> 00:10:36,756 Speaker 1: just as a matter of taste. And then there is 194 00:10:36,796 --> 00:10:38,916 Speaker 1: a little bit, like, it feels a little rude. It 195 00:10:38,956 --> 00:10:41,646 Speaker 1: feels a little spammy. Like, I'm reading this thing, and 196 00:10:41,666 --> 00:10:44,566 Speaker 1: this person didn't even take the time to lightly edit 197 00:10:44,626 --> 00:10:47,866 Speaker 1: it to get rid of the obvious AI-isms. 198 00:10:47,886 --> 00:10:50,576 Speaker 2: Yeah, or if you aren't going to take the time 199 00:10:50,616 --> 00:10:53,276 Speaker 2: to write it yourself, then I'm so much less inclined 200 00:10:53,346 --> 00:10:53,786 Speaker 2: to read it. 201 00:10:54,156 --> 00:10:56,646 Speaker 1: So the question I have is, do the people who 202 00:10:56,706 --> 00:10:59,526 Speaker 1: are posting that know that it is obviously AI and 203 00:10:59,546 --> 00:11:01,506 Speaker 1: they just don't care? I mean, I guess it varies, 204 00:11:01,586 --> 00:11:03,686 Speaker 1: but that's a weird one to me still. 205 00:11:03,766 --> 00:11:05,326 Speaker 2: I think there's a little bit of a bell curve. 206 00:11:05,716 --> 00:11:07,616 Speaker 2: There's a lot of people who are either new to 207 00:11:07,696 --> 00:11:11,076 Speaker 2: using AI or they haven't really used that much AI before, 208 00:11:11,116 --> 00:11:13,956 Speaker 2: and they can't really look at text and immediately grok 209 00:11:13,976 --> 00:11:17,676 Speaker 2: that it's AI-generated. They can't immediately tell. And so, yeah, 210 00:11:17,816 --> 00:11:19,906 Speaker 2: obviously you have the people who are just like, don't 211 00:11:19,986 --> 00:11:21,736 Speaker 2: use AI much. And then you also have the people 212 00:11:21,746 --> 00:11:25,016 Speaker 2: who like just discovered this technology and they're like, wow, 213 00:11:25,156 --> 00:11:27,236 Speaker 2: it like writes so well, this is amazing. And I 214 00:11:27,236 --> 00:11:31,396 Speaker 2: don't think they fully realized that like regular AI users 215 00:11:31,876 --> 00:11:35,156 Speaker 2: can read this and immediately smell that it's AI generated. Yeah. Then, 216 00:11:35,236 --> 00:11:36,566 Speaker 2: then you get to the middle of the bell curve 217 00:11:36,586 --> 00:11:39,846 Speaker 2: where it's like, yeah, like, like people who use AI 218 00:11:40,566 --> 00:11:42,846 Speaker 2: on a somewhat regular basis, they, they can kind of 219 00:11:42,866 --> 00:11:44,846 Speaker 2: just tell. And I think you get to the other 220 00:11:44,926 --> 00:11:46,926 Speaker 2: end where there's people who are like so deep in 221 00:11:47,026 --> 00:11:51,446 Speaker 2: AI psychosis that, that they also can't tell because like 90% 222 00:11:51,446 --> 00:11:54,466 Speaker 2: of what they're reading is like AI already. They talk 223 00:11:54,506 --> 00:11:56,686 Speaker 2: to chatbots more than other people and they just like 224 00:11:57,076 --> 00:12:00,376 Speaker 2: have lost this ability to differentiate. Uh-huh. 225 00:12:00,916 --> 00:12:02,936 Speaker 1: So maybe we'll all be at that far end of 226 00:12:02,976 --> 00:12:03,956 Speaker 1: the curve in a year. 227 00:12:04,196 --> 00:12:05,016 Speaker 2: I really hope not. 228 00:12:05,036 --> 00:12:10,446 Speaker 1: Yeah, I suppose I also hope not. So the companies 229 00:12:10,486 --> 00:12:17,146 Speaker 1: that are making frontier models, Anthropic, OpenAI, arguably Alphabet, Google, 230 00:12:17,886 --> 00:12:23,426 Speaker 1: are giant companies with wild amounts of money, full of geniuses. 231 00:12:23,946 --> 00:12:27,026 Speaker 1: You'd think those companies could just do what you're doing. 232 00:12:27,186 --> 00:12:31,246 Speaker 1: They could just build their own AI detectors. Why don't they? 233 00:12:31,726 --> 00:12:35,046 Speaker 2: I agree. I don't disagree here. But I also think 234 00:12:35,116 --> 00:12:39,296 Speaker 2: that this is just not in their top 10 priority list. 235 00:12:40,256 --> 00:12:43,736 Speaker 2: They're building AGI. They're automating white-collar work. 236 00:12:44,226 --> 00:12:47,326 Speaker 1: Probably the right answer. I mean, there's an answer, which 237 00:12:47,386 --> 00:12:52,006 Speaker 1: is like, is it contrary to their incentives to do 238 00:12:52,046 --> 00:12:56,036 Speaker 1: what you're doing, right? Is it in their interest for 239 00:12:56,096 --> 00:12:59,196 Speaker 1: people to be able to distinguish AI pros from human pros, 240 00:12:59,236 --> 00:13:00,756 Speaker 1: or is it contrary to their interest? 241 00:13:00,836 --> 00:13:02,956 Speaker 2: I think it is contrary to their interest. There was 242 00:13:02,976 --> 00:13:07,136 Speaker 2: an interesting article that I was exploring a couple of 243 00:13:07,156 --> 00:13:11,936 Speaker 2: years ago, actually, when OpenAI had watermarking technology, they'd built it, 244 00:13:12,336 --> 00:13:14,376 Speaker 2: but then they didn't want to release it. There was 245 00:13:14,396 --> 00:13:16,336 Speaker 2: a lot of internal politics on this, and this was 246 00:13:16,436 --> 00:13:20,316 Speaker 2: largely because they were afraid it would hurt their core 247 00:13:20,376 --> 00:13:23,436 Speaker 2: user base, which was at least a plurality of students 248 00:13:23,476 --> 00:13:23,956 Speaker 2: at this time. 249 00:13:24,416 --> 00:13:26,776 Speaker 1: They would get caught cheating if there was a watermark. 250 00:13:26,836 --> 00:13:27,796 Speaker 1: It's basically what you're saying. 251 00:13:27,836 --> 00:13:30,216 Speaker 2: They would get caught cheating, or they would churn to 252 00:13:30,716 --> 00:13:32,686 Speaker 2: a different AI service that didn't have a watermark. 253 00:13:33,036 --> 00:13:37,206 Speaker 1: So you brought up watermarks, so let's talk about watermarks. 254 00:13:37,806 --> 00:13:41,526 Speaker 1: The EU, as I understand, just passed a law requiring... 255 00:13:41,566 --> 00:13:45,066 Speaker 1: watermarks on text from language models. Is that true? 256 00:13:45,146 --> 00:13:49,006 Speaker 2: Yes, that's true. And OpenAI and Anthropic are both working 257 00:13:49,046 --> 00:13:51,526 Speaker 2: on this. Google already watermarks text outputs. 258 00:13:52,186 --> 00:13:53,426 Speaker 1: Is that bad for your business? 259 00:13:54,686 --> 00:13:56,826 Speaker 2: I think it's really good that people are taking this 260 00:13:56,866 --> 00:14:00,006 Speaker 2: problem seriously. It is competitive with us, but I think 261 00:14:00,106 --> 00:14:03,266 Speaker 2: Pangram has a lot of advantages that watermarks don't have, 262 00:14:03,486 --> 00:14:05,386 Speaker 2: which is that a watermark is just going to tell 263 00:14:05,406 --> 00:14:07,756 Speaker 2: you either Claude was here or not, and Pangram will 264 00:14:07,796 --> 00:14:10,196 Speaker 2: be able to tell you with higher granularity of like, 265 00:14:10,736 --> 00:14:12,716 Speaker 2: What was the degree of AI assistance? Was it just 266 00:14:12,896 --> 00:14:14,586 Speaker 2: helping or did it write the whole thing? 267 00:14:14,606 --> 00:14:20,946 Speaker 1: I mean, presumably, there's some universe where Claude outputs the 268 00:14:20,966 --> 00:14:24,326 Speaker 1: copy that's watermarked, and then you literally just retype it 269 00:14:24,406 --> 00:14:28,866 Speaker 1: into a Google Doc. I mean, naively, I would think 270 00:14:28,906 --> 00:14:31,386 Speaker 1: the watermark would not follow. Or is the watermark somehow 271 00:14:31,526 --> 00:14:34,096 Speaker 1: encoded in the prose so that if you retype it, 272 00:14:34,126 --> 00:14:34,706 Speaker 1: it's still there? 273 00:14:34,806 --> 00:14:40,026 Speaker 2: Yes, the watermark is statistically encoded in the word choices themselves. 274 00:14:40,046 --> 00:14:43,706 Speaker 1: Oh, that is rad. So even retyping it, the watermark 275 00:14:43,766 --> 00:14:44,326 Speaker 1: is still there. 276 00:14:44,806 --> 00:14:50,326 Speaker 2: Yes. With that said, you could run some automated paraphraser 277 00:14:50,406 --> 00:14:53,386 Speaker 2: to just paraphrase all the words, and that would erase 278 00:14:53,426 --> 00:14:53,966 Speaker 2: the watermark. 279 00:14:54,286 --> 00:14:57,786 Speaker 1: Yes. Well, so there are humanizers now, right? There are 280 00:14:59,066 --> 00:15:03,306 Speaker 1: things you can upload your language model pros into that 281 00:15:03,386 --> 00:15:07,086 Speaker 1: claim to evade detection. Although I will say I tried one, 282 00:15:07,106 --> 00:15:09,716 Speaker 1: I won't name it, but I tried one in preparing 283 00:15:09,736 --> 00:15:11,836 Speaker 1: for this interview, and you'll be glad to know it 284 00:15:11,856 --> 00:15:14,396 Speaker 1: didn't work. I had the language model output pros, I 285 00:15:14,436 --> 00:15:16,656 Speaker 1: ran it through a humanizer, put it into PanGram, and 286 00:15:16,676 --> 00:15:18,076 Speaker 1: PanGram said 100% AI. 287 00:15:18,236 --> 00:15:21,816 Speaker 2: Amazing. Yeah, I mean, this is part of the adversarial 288 00:15:21,876 --> 00:15:24,766 Speaker 2: nature of the business is We are also training on 289 00:15:24,866 --> 00:15:28,546 Speaker 2: humanizer outputs because we need to make sure that pangram 290 00:15:28,906 --> 00:15:33,246 Speaker 2: can detect anything that's been run through a humanizer. Otherwise, 291 00:15:33,306 --> 00:15:35,946 Speaker 2: it would kind of be trivial to evade pangram. 292 00:15:36,586 --> 00:15:39,546 Speaker 1: Why can't they beat you? I would think you could 293 00:15:39,586 --> 00:15:43,886 Speaker 1: be at a degree of randomness at the level of 294 00:15:44,486 --> 00:15:50,216 Speaker 1: word choice and syntax that would defeat you. Why can't they? 295 00:15:50,516 --> 00:15:52,836 Speaker 2: I think these are kind of like two, there's two 296 00:15:52,936 --> 00:15:57,136 Speaker 2: competing poles for text when you're humanizing. You want it 297 00:15:57,176 --> 00:16:02,776 Speaker 2: to be coherent and, you know, retain its logical flow. 298 00:16:02,996 --> 00:16:05,356 Speaker 2: And then on the other side, you want to just 299 00:16:05,436 --> 00:16:07,656 Speaker 2: degrade it or perturb it a little bit enough that 300 00:16:07,696 --> 00:16:11,266 Speaker 2: it is outside of the range of AI detectors. There 301 00:16:11,426 --> 00:16:15,226 Speaker 2: are humanizers that can beat pangram and you could do 302 00:16:15,266 --> 00:16:19,066 Speaker 2: this by just like perturbing the text way more just 303 00:16:19,186 --> 00:16:19,746 Speaker 2: a lot more. 304 00:16:19,966 --> 00:16:24,566 Speaker 1: Well, sure. I mean, there's a difference between degrading and perturbing, right? Like, 305 00:16:25,086 --> 00:16:28,266 Speaker 1: you can make it really bad, but that's obviously not useful. 306 00:16:28,426 --> 00:16:31,726 Speaker 2: Well, I think degrading is just a more extreme version 307 00:16:31,786 --> 00:16:32,386 Speaker 2: of perturbing. 308 00:16:32,966 --> 00:16:35,446 Speaker 1: Well, you could improve. You could perturb it a lot 309 00:16:35,486 --> 00:16:37,106 Speaker 1: and improve it. Oh, yeah, yeah. 310 00:16:37,126 --> 00:16:39,406 Speaker 2: But these humanizers, they don't have the resources to actually 311 00:16:39,476 --> 00:16:40,146 Speaker 2: do that, right? 312 00:16:40,216 --> 00:16:40,696 Speaker 1: Yeah. 313 00:16:41,016 --> 00:16:45,156 Speaker 2: Most of them are, like, one- to ten-person companies. 314 00:16:45,716 --> 00:16:48,216 Speaker 1: There's a more fundamental question here that is interesting to me, 315 00:16:48,236 --> 00:16:52,326 Speaker 1: and I don't really understand the answer, which is, like, 316 00:16:52,566 --> 00:16:55,526 Speaker 1: why does AI slop all sound the same? You know 317 00:16:55,546 --> 00:16:57,366 Speaker 1: what I mean? Like, that's the thing that's so annoying. 318 00:16:57,386 --> 00:17:00,706 Speaker 1: Why can't you just turn up the volume, not on, like, 319 00:17:00,926 --> 00:17:04,426 Speaker 1: make it bad, but just, like, don't use that same rhythm. 320 00:17:04,466 --> 00:17:06,886 Speaker 1: Don't do, like, the one sentence. Don't use these annoying 321 00:17:06,966 --> 00:17:10,306 Speaker 1: words that are AI words. Like, why is that at 322 00:17:10,386 --> 00:17:11,666 Speaker 1: a technical level? 323 00:17:12,046 --> 00:17:15,226 Speaker 2: It's really hard to say. I think... So there's a 324 00:17:15,246 --> 00:17:18,526 Speaker 2: concept in machine learning called mode collapse. And that means like, 325 00:17:18,906 --> 00:17:23,326 Speaker 2: you know, once an AI model learns what is best 326 00:17:23,626 --> 00:17:26,846 Speaker 2: or most common, which is the mode, then it will 327 00:17:27,126 --> 00:17:30,506 Speaker 2: kind of like collapse the edges of its distribution and 328 00:17:30,546 --> 00:17:32,666 Speaker 2: just output the mode much more often. 329 00:17:33,126 --> 00:17:36,526 Speaker 1: Yes. The mode being the modal word, the most common word. 330 00:17:36,546 --> 00:17:39,386 Speaker 2: Yeah, the most common word or phrase or sentence construction. 331 00:17:39,406 --> 00:17:42,266 Speaker 2: I think that's part of it. And I think also... 332 00:17:42,536 --> 00:17:45,706 Speaker 2: a lot of these labs, they do post-training. They'll create 333 00:17:45,806 --> 00:17:49,206 Speaker 2: a reward model and they'll give the language model a reward. 334 00:17:49,406 --> 00:17:52,846 Speaker 2: And so I can only suspect that there's somebody in 335 00:17:52,926 --> 00:17:57,096 Speaker 2: here who's saying there's some reward for, quote, like good writing. 336 00:17:57,956 --> 00:18:01,976 Speaker 2: And that might include, you know, writing with em dashes 337 00:18:02,136 --> 00:18:05,176 Speaker 2: is probably correlated with good writing in the pre-AI world. 338 00:18:05,736 --> 00:18:08,206 Speaker 2: Same thing with the like Like, it's not just X 339 00:18:08,246 --> 00:18:09,886 Speaker 2: but Y sort of metaphors. 340 00:18:10,006 --> 00:18:12,026 Speaker 1: Yes. That's another one that I have taken out of 341 00:18:12,086 --> 00:18:13,696 Speaker 1: my own writing, actually. Yeah, yeah. 342 00:18:13,756 --> 00:18:16,956 Speaker 2: Like, it's a very reasonable construction, but then you feed 343 00:18:16,976 --> 00:18:20,116 Speaker 2: it into the reward model, and then the AI learns 344 00:18:20,136 --> 00:18:21,656 Speaker 2: in its prose to turn that up to 11. 345 00:18:21,656 --> 00:18:23,176 Speaker 1: It likes it too much, yeah. 346 00:18:23,416 --> 00:18:26,436 Speaker 2: Yeah. So it just overuses what would have normally been 347 00:18:26,456 --> 00:18:27,876 Speaker 2: considered good writing. 348 00:18:31,476 --> 00:18:43,016 Speaker 1: We'll be back in just a minute. Okay, let's talk 349 00:18:43,056 --> 00:18:45,436 Speaker 1: about false positives, right? This is a thing people care 350 00:18:45,476 --> 00:18:48,176 Speaker 1: about and have written about. So obviously in any test, 351 00:18:48,196 --> 00:18:50,056 Speaker 1: you can have a false positive or a false negative. 352 00:18:50,096 --> 00:18:52,416 Speaker 1: A false negative, you're saying this is human generated and 353 00:18:52,456 --> 00:18:55,476 Speaker 1: actually it's AI. That one people don't get so upset about. 354 00:18:55,496 --> 00:18:59,156 Speaker 1: The one people are really afraid of is wrongly accused, right? 355 00:18:59,196 --> 00:19:02,776 Speaker 1: Pangram says it was AI. Actually, it was written by 356 00:19:02,816 --> 00:19:06,896 Speaker 1: a person. I've seen you say one in 10,000 is 357 00:19:07,196 --> 00:19:11,516 Speaker 1: the false positive rate. And I've also seen outside studies that, 358 00:19:12,066 --> 00:19:14,806 Speaker 1: find a very low false positive rate, a similar order 359 00:19:14,826 --> 00:19:16,966 Speaker 1: of magnitude, one from the University of Chicago I found 360 00:19:17,006 --> 00:19:20,876 Speaker 1: particularly compelling. But like, I don't know how many writing 361 00:19:20,916 --> 00:19:23,286 Speaker 1: assignments students have in high school and college, but it's 362 00:19:23,496 --> 00:19:27,716 Speaker 1: probably order of magnitude of hundreds, right? And so that 363 00:19:27,756 --> 00:19:29,876 Speaker 1: means that some share of students are going to be 364 00:19:29,936 --> 00:19:35,476 Speaker 1: wrongly accused, right? If it's 500 writing assignments, then it 365 00:19:35,496 --> 00:19:38,676 Speaker 1: would be 5% of students will be wrongly accused by 366 00:19:38,716 --> 00:19:41,316 Speaker 1: Pan Graham of using AI to write a thing that 367 00:19:41,336 --> 00:19:42,056 Speaker 1: they actually wrote? 368 00:19:42,416 --> 00:19:45,476 Speaker 2: Yeah, I do think like 1 in 10,000 is beyond 369 00:19:45,516 --> 00:19:48,996 Speaker 2: the threshold where like you can use this as a 370 00:19:49,016 --> 00:19:52,686 Speaker 2: tool and generally typically be confident that when it says 371 00:19:52,746 --> 00:19:53,866 Speaker 2: it's AI, it's correct. 372 00:19:54,186 --> 00:19:56,726 Speaker 1: It's true. That's definitely true. Like, I mean, I will 373 00:19:56,746 --> 00:19:59,656 Speaker 1: say just I was actually accused by a teacher in 374 00:19:59,696 --> 00:20:02,236 Speaker 1: high school of plagiarizing. That's the old school. And I 375 00:20:02,276 --> 00:20:05,676 Speaker 1: had not plagiarized, for the record. And so that is 376 00:20:05,736 --> 00:20:08,666 Speaker 1: the old school human version of this. And I'm sure 377 00:20:08,726 --> 00:20:12,846 Speaker 1: his error rate was higher than 1 in 10,000. But 378 00:20:12,906 --> 00:20:14,886 Speaker 1: I am sensitive to it. Like, that feeling of being 379 00:20:14,926 --> 00:20:17,046 Speaker 1: wrongly accused is a terrible feeling. Mm-hmm. 380 00:20:17,586 --> 00:20:19,896 Speaker 2: Yeah, so I think one thing that pangram helps with 381 00:20:19,996 --> 00:20:24,356 Speaker 2: actually is it's much better than like the average human intuition. 382 00:20:24,736 --> 00:20:27,666 Speaker 2: And it's also better than any other AI detector. So 383 00:20:27,746 --> 00:20:32,026 Speaker 2: because of that, the existence of pangram is, I truly 384 00:20:32,046 --> 00:20:35,826 Speaker 2: believe this, reducing the number of false accusations for AI 385 00:20:35,956 --> 00:20:38,156 Speaker 2: use in a scenario like education. 386 00:20:38,956 --> 00:20:42,036 Speaker 1: Yeah, that's fair. So like notionally, like, of course, teachers 387 00:20:42,076 --> 00:20:43,796 Speaker 1: are on the lookout for AI one way or the 388 00:20:43,876 --> 00:20:45,476 Speaker 1: other if they want students to write it. 389 00:20:45,496 --> 00:20:47,776 Speaker 2: Exactly. And then the other side, I think, which is 390 00:20:47,816 --> 00:20:53,036 Speaker 2: really important, is that it's not the only source of information. 391 00:20:53,556 --> 00:20:56,806 Speaker 2: Once Pangram says that something's AI, we can always go deeper. 392 00:20:56,886 --> 00:21:01,146 Speaker 2: We can either ask the student for drafts. If you 393 00:21:01,266 --> 00:21:04,666 Speaker 2: use Google Docs, Pangram actually has this Chrome extension where 394 00:21:04,686 --> 00:21:07,586 Speaker 2: you could see the full revision history, you could see 395 00:21:07,626 --> 00:21:09,006 Speaker 2: that the document being typed out. 396 00:21:09,186 --> 00:21:11,946 Speaker 1: And that history is available even without PanGram, right? 397 00:21:11,966 --> 00:21:12,366 Speaker 2: Exactly. 398 00:21:12,466 --> 00:21:14,476 Speaker 1: And if you're using Chrome, you can see the version 399 00:21:14,496 --> 00:21:15,316 Speaker 1: history and go back. 400 00:21:15,676 --> 00:21:19,216 Speaker 2: Exactly. Yes. Yeah. So that version history is there. But 401 00:21:19,596 --> 00:21:22,176 Speaker 2: I think PanGram is very important because you don't want 402 00:21:22,196 --> 00:21:24,516 Speaker 2: to go look at the version history for every single doc. 403 00:21:24,836 --> 00:21:27,506 Speaker 2: You only want to look into it once you suspect 404 00:21:27,586 --> 00:21:30,286 Speaker 2: that something was wrong or there are some AIUs there. 405 00:21:30,906 --> 00:21:33,066 Speaker 1: That's funny. I thought, well, you could automate looking at 406 00:21:33,106 --> 00:21:34,586 Speaker 1: the version history. And then I was like, oh, but 407 00:21:34,606 --> 00:21:37,306 Speaker 1: you could also automate a little script that would make 408 00:21:37,326 --> 00:21:40,066 Speaker 1: it look like you wrote it. It's the adversarial thing. 409 00:21:41,046 --> 00:21:42,746 Speaker 1: It is amazing. I was reading an article in New 410 00:21:42,786 --> 00:21:44,846 Speaker 1: York Magazine about students using AI to cheat. And one 411 00:21:44,866 --> 00:21:46,506 Speaker 1: of the things they pointed out is like how much 412 00:21:46,666 --> 00:21:50,966 Speaker 1: time students spend cheating, like trying to use AI and 413 00:21:51,006 --> 00:21:53,586 Speaker 1: not get caught. And like that amount of time you 414 00:21:53,626 --> 00:21:56,086 Speaker 1: could have written the paper, right? Which is a classic thing, 415 00:21:56,126 --> 00:21:57,666 Speaker 1: but maybe it's more fun to cheat. Well, yeah. 416 00:21:57,726 --> 00:22:00,196 Speaker 2: I mean, I think that's part of the like, friction, 417 00:22:00,316 --> 00:22:04,116 Speaker 2: like adding friction to this cheating process where fewer students 418 00:22:04,156 --> 00:22:06,836 Speaker 2: are going to do it if it is time consuming 419 00:22:06,876 --> 00:22:10,096 Speaker 2: to cheat. If it's just one prompt, I think that's 420 00:22:10,156 --> 00:22:14,216 Speaker 2: where like a lot of students and faculty, like everyone's 421 00:22:14,236 --> 00:22:16,776 Speaker 2: kind of been frustrated with this problem. It also sucks 422 00:22:16,796 --> 00:22:19,856 Speaker 2: when your peers are doing one sentence prompts and getting 423 00:22:19,896 --> 00:22:20,416 Speaker 2: away with it. 424 00:22:21,116 --> 00:22:24,946 Speaker 1: Right. That's a bad equilibrium. That's a really bad equilibrium. So, okay, 425 00:22:24,966 --> 00:22:27,846 Speaker 1: so out of every 10,000 times you say AI had 426 00:22:28,046 --> 00:22:30,266 Speaker 1: anything to do with it, you only get it wrong once. 427 00:22:30,506 --> 00:22:32,946 Speaker 2: Correct. It's a little bit more of a spectrum in 428 00:22:32,986 --> 00:22:38,876 Speaker 2: the middle when somebody believes that they had 50% input 429 00:22:38,996 --> 00:22:42,416 Speaker 2: into the document and AI had 50% input and then 430 00:22:42,456 --> 00:22:46,456 Speaker 2: Paragram said it's 100% AI. Like, this is sort of 431 00:22:46,496 --> 00:22:49,816 Speaker 2: the gray area where it's much harder to measure if 432 00:22:49,836 --> 00:22:50,516 Speaker 2: we're right or not. 433 00:22:50,616 --> 00:22:54,126 Speaker 1: Uh-huh. And so that's the area where you get it 434 00:22:54,166 --> 00:22:57,066 Speaker 1: wrong more is kind of at that margin. So it's like, no, no, 435 00:22:57,226 --> 00:22:59,696 Speaker 1: I wrote it and I got some edits from AI, 436 00:23:00,216 --> 00:23:04,256 Speaker 1: but PanGram returns 100% AI. Like that happens more often 437 00:23:04,296 --> 00:23:05,136 Speaker 1: than 1 in 10,000. 438 00:23:05,136 --> 00:23:09,276 Speaker 2: Yeah, that happens at least for like light AI assistants 439 00:23:09,336 --> 00:23:13,686 Speaker 2: or medium AI assistants. It's like a few percent of 440 00:23:13,726 --> 00:23:15,666 Speaker 2: the time where we say it's still like 100% AI. 441 00:23:17,336 --> 00:23:21,096 Speaker 1: So a few percent is orders of magnitude higher than 442 00:23:21,176 --> 00:23:23,996 Speaker 1: one in 10,000. It happens kind of a lot. 443 00:23:24,636 --> 00:23:27,196 Speaker 2: We try to be very transparent about all of our benchmarks. 444 00:23:27,836 --> 00:23:30,876 Speaker 2: I think this is also the area of greatest debate 445 00:23:31,016 --> 00:23:35,416 Speaker 2: where it's not actually so black and white. What the 446 00:23:35,456 --> 00:23:40,256 Speaker 2: degree of AI assistance was, people on average tend to under-disclose. 447 00:23:40,276 --> 00:23:42,776 Speaker 2: They're like, oh, I use AI to help me. I 448 00:23:42,816 --> 00:23:45,516 Speaker 2: talked to someone who's like, I used AI for copy edits. 449 00:23:45,676 --> 00:23:47,336 Speaker 2: And then you go through what that means. It's like 450 00:23:47,616 --> 00:23:51,596 Speaker 2: they actually, like, every word was produced by AI. And 451 00:23:52,696 --> 00:23:56,396 Speaker 2: it had gone through so many revisions with AI. And 452 00:23:56,436 --> 00:23:58,436 Speaker 2: so I think this is the biggest gray area where 453 00:23:59,016 --> 00:24:02,816 Speaker 2: Pangram is not going to be, like, a perfect oracle 454 00:24:02,936 --> 00:24:05,336 Speaker 2: and tell you this was the exact prompt and these 455 00:24:05,356 --> 00:24:08,076 Speaker 2: were the exact human edits. But I think it can, like, 456 00:24:08,096 --> 00:24:09,176 Speaker 2: get approximately there. 457 00:24:10,376 --> 00:24:14,346 Speaker 1: Are there... groups of people or kinds of speakers where 458 00:24:14,386 --> 00:24:18,426 Speaker 1: pangram is worse at detecting whether their writing is their 459 00:24:18,506 --> 00:24:20,086 Speaker 1: own or generated by AI? 460 00:24:20,776 --> 00:24:25,576 Speaker 2: So we've run this on three different groups. I think 461 00:24:25,596 --> 00:24:29,376 Speaker 2: there's been different studies in the past that have found 462 00:24:29,396 --> 00:24:35,706 Speaker 2: that AI detectors are biased against English as second language speakers. 463 00:24:36,406 --> 00:24:41,186 Speaker 2: There's also been questions of if AI detectors are biased 464 00:24:41,226 --> 00:24:47,366 Speaker 2: against autistic or neurodivergent speakers. And so what we've done 465 00:24:47,526 --> 00:24:51,776 Speaker 2: is we ran studies where we looked at 25,000 texts 466 00:24:52,936 --> 00:24:56,176 Speaker 2: by English as second language speakers and found that there's 467 00:24:56,356 --> 00:25:00,376 Speaker 2: no elevated false positive rate. We have one false positive 468 00:25:00,496 --> 00:25:05,316 Speaker 2: in the 25,000. And then so similarly, somebody else who 469 00:25:05,396 --> 00:25:09,666 Speaker 2: is not us asked for credits to run a similar 470 00:25:09,726 --> 00:25:14,306 Speaker 2: study for autistic speakers and they did a really great job. 471 00:25:14,366 --> 00:25:17,786 Speaker 2: They had like a 20 page report on their findings 472 00:25:17,846 --> 00:25:21,196 Speaker 2: and they also found no bias against autistic speakers, which 473 00:25:21,216 --> 00:25:25,296 Speaker 2: is really great. And then the third one that we've 474 00:25:25,336 --> 00:25:28,236 Speaker 2: looked at was just like race. Cause I think this 475 00:25:28,276 --> 00:25:31,906 Speaker 2: has also been a question and we again found no 476 00:25:32,336 --> 00:25:37,716 Speaker 2: statistically significant difference, which it was really cool to see. What? 477 00:25:37,856 --> 00:25:38,176 Speaker 1: Yes. 478 00:25:38,396 --> 00:25:38,616 Speaker 2: Yes. 479 00:25:38,736 --> 00:25:40,916 Speaker 1: One hopes that would not be a problem. You could 480 00:25:40,956 --> 00:25:43,636 Speaker 1: imagine how for someone who was not a native speaker, 481 00:25:43,696 --> 00:25:45,936 Speaker 1: for example, you might have a higher rate of false positives. 482 00:25:46,356 --> 00:25:48,826 Speaker 1: Was it a problem in the past? Like were earlier 483 00:25:48,886 --> 00:25:52,106 Speaker 1: versions of PanGram less reliable in that way? 484 00:25:52,526 --> 00:25:54,766 Speaker 2: I would say like PanGram 2, like definitely had a 485 00:25:54,766 --> 00:25:58,046 Speaker 2: bit of this issue. And then starting like PanGram 3, 486 00:25:58,046 --> 00:26:01,006 Speaker 2: we were taking it very seriously. But I think every, 487 00:26:02,076 --> 00:26:04,596 Speaker 2: we've had a lot of candidate models that were biased 488 00:26:04,696 --> 00:26:08,066 Speaker 2: against English second language speakers and And so because of that, 489 00:26:08,146 --> 00:26:10,416 Speaker 2: we kind of threw away the model and never released it. 490 00:26:11,226 --> 00:26:12,206 Speaker 1: And when was Pangram 2? 491 00:26:12,186 --> 00:26:16,506 Speaker 2: How long ago was that? Like a year and a 492 00:26:16,546 --> 00:26:20,896 Speaker 2: half ago. Even Pangram 2, like our false positive rate 493 00:26:20,916 --> 00:26:24,896 Speaker 2: was like much lower than the other AI detectors. But 494 00:26:25,236 --> 00:26:28,416 Speaker 2: I think it was still like compared to our baseline 495 00:26:28,436 --> 00:26:32,186 Speaker 2: false positive rate, it was elevated on ESL. And I 496 00:26:32,226 --> 00:26:35,646 Speaker 2: think like part of the difficulty here is going back 497 00:26:35,666 --> 00:26:38,236 Speaker 2: to the humanizers. a lot of them are trying to 498 00:26:38,356 --> 00:26:42,896 Speaker 2: inject spelling mistakes, grammar mistakes as a way to evade 499 00:26:42,996 --> 00:26:48,936 Speaker 2: AI detection. And so training on these naively will make 500 00:26:48,996 --> 00:26:53,696 Speaker 2: your model correlate these spelling and grammar mistakes with AI, actually, 501 00:26:53,776 --> 00:26:54,876 Speaker 2: which is not correct. 502 00:26:54,896 --> 00:26:57,736 Speaker 1: Well, only if those are occurring at a higher rate 503 00:26:57,776 --> 00:27:00,026 Speaker 1: in the humanizers than they are in the human texts 504 00:27:00,046 --> 00:27:01,626 Speaker 1: you're putting in, right? Yeah. 505 00:27:02,126 --> 00:27:05,926 Speaker 2: I think the humanizers, some of them are perturbing and 506 00:27:05,966 --> 00:27:09,416 Speaker 2: degrading the text more than your average human. So it's 507 00:27:09,436 --> 00:27:11,956 Speaker 2: taking it like all the way down to the other side. Yeah. 508 00:27:12,836 --> 00:27:15,356 Speaker 1: There have been a number of cases where, you know, 509 00:27:15,416 --> 00:27:17,856 Speaker 1: Pangram found a piece of writing to be AI generated. 510 00:27:17,896 --> 00:27:20,836 Speaker 1: The person said, no, it wasn't. I wrote it. Has 511 00:27:20,856 --> 00:27:22,876 Speaker 1: there been a case where the person was validated? A 512 00:27:23,096 --> 00:27:24,056 Speaker 1: high profile case? 513 00:27:24,676 --> 00:27:28,456 Speaker 2: Yeah. So there was like one case, Taylor Lorenz had 514 00:27:28,516 --> 00:27:31,706 Speaker 2: this like article in Vanity Fair. So Pangram has a 515 00:27:31,686 --> 00:27:37,826 Speaker 2: 50 word minimum. Somebody selected a 54 word segment and 516 00:27:37,846 --> 00:27:40,846 Speaker 2: Pangram said this was AI generated. But what they did 517 00:27:40,866 --> 00:27:43,966 Speaker 2: was they just posted the like a hundred percent AI 518 00:27:44,386 --> 00:27:46,606 Speaker 2: for the 54 word segment. And so then I looked 519 00:27:46,766 --> 00:27:50,226 Speaker 2: into it. If I checked like the full Vandy fair article, 520 00:27:50,266 --> 00:27:53,306 Speaker 2: Pangram actually said a hundred percent human. And then, you know, 521 00:27:53,346 --> 00:27:56,606 Speaker 2: Taylor's like, Hey, like here's my edit history. Like we 522 00:27:56,636 --> 00:27:58,516 Speaker 2: went through a lot of provisions with, with an editor, 523 00:27:58,576 --> 00:28:01,096 Speaker 2: like this is all my writing. And so, yeah, like it's, 524 00:28:01,576 --> 00:28:05,316 Speaker 2: it was absolutely a false positive and, Yeah, partially because 525 00:28:05,356 --> 00:28:07,496 Speaker 2: it was a small piece of text and partially, I think, 526 00:28:07,536 --> 00:28:10,116 Speaker 2: because this person was kind of using pangram in an 527 00:28:10,256 --> 00:28:12,636 Speaker 2: inappropriate way of, like, I don't know, trying to go 528 00:28:12,676 --> 00:28:15,566 Speaker 2: for the gotcha rather than, like, going for the full story. 529 00:28:15,606 --> 00:28:19,146 Speaker 1: Well, that 1 in 10,000 false positive rate, is that 530 00:28:19,186 --> 00:28:21,466 Speaker 1: contingent on some minimum number of words? And if so, 531 00:28:21,486 --> 00:28:22,146 Speaker 1: what number of words? 532 00:28:23,166 --> 00:28:26,556 Speaker 2: This is aggregate over positive. all of our texts, but 533 00:28:26,936 --> 00:28:28,816 Speaker 2: it's a little bit higher for 50 words. 534 00:28:28,896 --> 00:28:31,076 Speaker 1: If you put in 50 words, is the false positive 535 00:28:31,116 --> 00:28:31,986 Speaker 1: rate notably higher? 536 00:28:32,126 --> 00:28:35,366 Speaker 2: Yes, it's closer to 1 in 5,000. Uh-huh. 537 00:28:36,586 --> 00:28:39,986 Speaker 1: I mean, maybe the person looked for the most AI-sounding 50. 538 00:28:39,586 --> 00:28:41,366 Speaker 2: Words, right? I mean, that's what they said, is they 539 00:28:41,386 --> 00:28:43,786 Speaker 2: said this sounded like AI to them. So it's a 540 00:28:43,826 --> 00:28:46,246 Speaker 2: false positive to them and also to PanGram. 541 00:28:46,766 --> 00:28:50,036 Speaker 1: Okay, let's talk about images. Images are new. I think 542 00:28:50,076 --> 00:28:51,996 Speaker 1: when we booked you, maybe, I don't even know if 543 00:28:52,036 --> 00:28:54,136 Speaker 1: the images thing was out, but tell me about your 544 00:28:54,156 --> 00:28:55,366 Speaker 1: image business. Yeah. 545 00:28:55,446 --> 00:28:59,526 Speaker 2: So it's a new model. We released it in July, 546 00:28:59,546 --> 00:29:03,986 Speaker 2: end of July, but it is state-of-the-art for AI image detection. 547 00:29:04,066 --> 00:29:06,826 Speaker 2: So what that means is you can put in an 548 00:29:06,926 --> 00:29:10,006 Speaker 2: image and we'll tell you, yes, this is AI generated 549 00:29:10,586 --> 00:29:14,916 Speaker 2: and it works on images from nano banana, like Gemini, 550 00:29:15,476 --> 00:29:20,276 Speaker 2: GPT image, mid journey, all of the major AI image models. 551 00:29:20,436 --> 00:29:22,516 Speaker 2: And so I think this is pretty cool because it 552 00:29:23,126 --> 00:29:24,766 Speaker 2: A couple of years ago, it was like really easy 553 00:29:24,806 --> 00:29:27,206 Speaker 2: to tell whether an image was AI generated. You just 554 00:29:27,226 --> 00:29:28,926 Speaker 2: like count the fingers and you look at the text. 555 00:29:28,946 --> 00:29:30,926 Speaker 1: Count the fingers, right. Or see if there's something that 556 00:29:30,966 --> 00:29:32,406 Speaker 1: defies the laws of physics, right? 557 00:29:32,426 --> 00:29:35,246 Speaker 2: Yeah. And today that happens a little bit, but like 558 00:29:35,526 --> 00:29:39,176 Speaker 2: you can get some really hyper-realistic AI images. It's scary. 559 00:29:39,356 --> 00:29:41,776 Speaker 2: And I think it's really hard to tell. 560 00:29:42,356 --> 00:29:44,836 Speaker 1: Certainly hard for people to tell. I mean, I certainly 561 00:29:44,916 --> 00:29:48,816 Speaker 1: buy that you are at the forefront of language detection. 562 00:29:49,266 --> 00:29:52,306 Speaker 1: What is the competitive landscape like for image detection? 563 00:29:52,916 --> 00:29:57,316 Speaker 2: There's a number of AI image detectors. I think many 564 00:29:57,376 --> 00:30:01,936 Speaker 2: of them are more focused on deepfakes and less focused 565 00:30:02,076 --> 00:30:03,636 Speaker 2: on AI images in general. 566 00:30:04,076 --> 00:30:07,386 Speaker 1: Yes, I talked to the head of Lodi. And they're 567 00:30:07,426 --> 00:30:10,026 Speaker 1: focused in particular on videos. Like his claim, at least, 568 00:30:10,066 --> 00:30:13,746 Speaker 1: was that they are good at understanding video, ingesting video 569 00:30:13,886 --> 00:30:15,026 Speaker 1: in a way that other people are not. 570 00:30:15,126 --> 00:30:15,646 Speaker 2: Oh, that's pretty cool. 571 00:30:15,666 --> 00:30:17,216 Speaker 1: Because video is hard. 572 00:30:17,246 --> 00:30:19,236 Speaker 2: Yeah. Yeah, we've been working a bit with video. So 573 00:30:19,276 --> 00:30:20,996 Speaker 2: if you put in a single frame from a video 574 00:30:21,016 --> 00:30:24,316 Speaker 2: into the AI image detector, it will do pretty well. 575 00:30:24,616 --> 00:30:28,296 Speaker 1: What kinds of businesses are natural customers for your image product? 576 00:30:28,316 --> 00:30:33,286 Speaker 2: Yeah, I think naturally it would be someone like Pinterest 577 00:30:33,726 --> 00:30:38,246 Speaker 2: or like any like stock photo company. image website, things 578 00:30:38,266 --> 00:30:41,666 Speaker 2: like that, where you want your users to have some 579 00:30:41,706 --> 00:30:43,936 Speaker 2: degree of trust that what they're seeing is real. 580 00:30:44,476 --> 00:30:47,256 Speaker 1: Dating apps or websites was the first one I thought of. 581 00:30:47,336 --> 00:30:51,356 Speaker 2: Yeah, I think dating apps, absolutely. You don't want AI-generated 582 00:30:51,876 --> 00:30:53,726 Speaker 2: people or AI catfishing. 583 00:30:54,386 --> 00:30:56,866 Speaker 1: Real estate apps was another one I thought. I mean, 584 00:30:56,926 --> 00:30:59,326 Speaker 1: I know they will explicitly sometimes be like, and here's 585 00:30:59,376 --> 00:31:01,716 Speaker 1: what it could look like. But like, they don't always 586 00:31:01,796 --> 00:31:03,716 Speaker 1: say that. I feel like a lot of the time, 587 00:31:04,216 --> 00:31:06,576 Speaker 1: like my neighborhood has all these old houses that people 588 00:31:06,596 --> 00:31:08,116 Speaker 1: have lived in for 50 years, but they sell for 589 00:31:08,136 --> 00:31:09,756 Speaker 1: a lot of money now, right? Which is the classic 590 00:31:09,796 --> 00:31:11,556 Speaker 1: thing where the realtor wants to be like, look, you 591 00:31:11,576 --> 00:31:13,116 Speaker 1: can pay a lot of money for this house because 592 00:31:13,176 --> 00:31:15,626 Speaker 1: it could look like this. But they don't show you 593 00:31:15,666 --> 00:31:17,366 Speaker 1: what it looks like now for a reason. 594 00:31:17,386 --> 00:31:18,706 Speaker 2: Oh, that's horrendous. 595 00:31:18,846 --> 00:31:19,286 Speaker 1: Yeah, yeah. 596 00:31:19,326 --> 00:31:21,486 Speaker 2: I mean, I think that sort of stuff, like it 597 00:31:21,506 --> 00:31:25,516 Speaker 2: should be illegal to... to put an AI photo instead 598 00:31:25,536 --> 00:31:26,996 Speaker 2: of the actual photo in your listing? 599 00:31:27,256 --> 00:31:28,876 Speaker 1: They may well say it. I don't want to be 600 00:31:28,916 --> 00:31:32,256 Speaker 1: accusing people of malice, but, like... No, I've. 601 00:31:32,136 --> 00:31:34,756 Speaker 2: Definitely seen it for apartment listings in New York. Yeah. Like, 602 00:31:34,896 --> 00:31:37,896 Speaker 2: where I show up, and it's like, huh, it looks 603 00:31:37,936 --> 00:31:39,436 Speaker 2: a little bit different than the photos. And I go 604 00:31:39,476 --> 00:31:41,476 Speaker 2: back to the photos, and, like, some of the details 605 00:31:41,516 --> 00:31:43,996 Speaker 2: are different. And I'm like, okay, I think... I'm, like, 606 00:31:44,216 --> 00:31:45,336 Speaker 2: pretty sure this was AI. 607 00:31:45,496 --> 00:31:48,636 Speaker 1: Wait, it's a shady apartment listing in New York? No, 608 00:31:48,796 --> 00:31:50,806 Speaker 1: I don't believe you. I. 609 00:31:51,416 --> 00:31:54,636 Speaker 2: Whenever it's too good to be true, half the time 610 00:31:54,656 --> 00:31:55,096 Speaker 2: it's AI. 611 00:31:55,116 --> 00:31:57,836 Speaker 1: One time I called for an apartment in New York 612 00:31:58,216 --> 00:32:00,176 Speaker 1: to go look at it, and the guy's first question was, 613 00:32:00,576 --> 00:32:03,496 Speaker 1: how tall are you? I was like, I'm out. I'm 614 00:32:03,516 --> 00:32:10,426 Speaker 1: not going to go look. We'll be back in a 615 00:32:10,446 --> 00:32:23,186 Speaker 1: minute with The Lightning Round. Okay, let's finish with the 616 00:32:23,206 --> 00:32:28,446 Speaker 1: lightning round. Cool. How has building PanGram changed the way 617 00:32:28,486 --> 00:32:29,746 Speaker 1: you think about language? 618 00:32:31,986 --> 00:32:33,426 Speaker 2: I don't know if I have an answer for this. 619 00:32:34,246 --> 00:32:35,626 Speaker 1: Maybe it hasn't. Maybe it hasn't. 620 00:32:35,646 --> 00:32:39,266 Speaker 2: No, I mean, I think some of the biggest learnings 621 00:32:39,306 --> 00:32:43,206 Speaker 2: for me were that, oh, like, these language models are 622 00:32:43,366 --> 00:32:47,466 Speaker 2: not at all as diverse as I thought. Yeah. When 623 00:32:47,506 --> 00:32:50,686 Speaker 2: I started, like, ChatGPT had come out, and we were 624 00:32:50,706 --> 00:32:54,406 Speaker 2: all like, this is amazing. incredible technology. And then spending 625 00:32:54,446 --> 00:32:56,546 Speaker 2: a little more time with it made me realize, oh, actually, 626 00:32:56,586 --> 00:32:58,826 Speaker 2: it's kind of just like saying the same thing every time. 627 00:32:59,106 --> 00:33:02,706 Speaker 2: It's quite mode collapse. And humans are much more diverse. 628 00:33:02,766 --> 00:33:04,726 Speaker 2: And I think that's valuable. 629 00:33:05,306 --> 00:33:06,966 Speaker 1: Well, what's the last thing you read that you loved? 630 00:33:08,046 --> 00:33:11,976 Speaker 2: I read this book called Sublimation by Isabel J. Kim. 631 00:33:11,996 --> 00:33:15,616 Speaker 2: It was about sort of this concept where if somebody immigrates, 632 00:33:15,716 --> 00:33:20,116 Speaker 2: they split into two people. And one person stays, one 633 00:33:20,136 --> 00:33:22,216 Speaker 2: person goes. It was very interesting. It's a good book. 634 00:33:22,376 --> 00:33:23,016 Speaker 1: It's a novel? 635 00:33:23,216 --> 00:33:23,696 Speaker 2: Novel, yeah. 636 00:33:24,256 --> 00:33:26,636 Speaker 1: You ever read anything by AI that you loved? 637 00:33:27,596 --> 00:33:30,916 Speaker 2: There's this kind of funny green text from an old 638 00:33:31,016 --> 00:33:36,536 Speaker 2: AI model about being a bottomless pit supervisor. 639 00:33:37,476 --> 00:33:39,796 Speaker 1: Wait, I don't know what green text is. What's green text? 640 00:33:40,376 --> 00:33:46,156 Speaker 2: Green text, it's this format of storytelling from 4chan where 641 00:33:46,836 --> 00:33:52,366 Speaker 2: it's basically describing a story in first person. Every line 642 00:33:52,426 --> 00:33:54,266 Speaker 2: starts with this carrot or this like. 643 00:33:54,306 --> 00:33:56,646 Speaker 1: I see. Kind of like looking at a command line 644 00:33:56,706 --> 00:33:57,146 Speaker 1: or something. 645 00:33:57,286 --> 00:33:59,266 Speaker 2: Yeah, like the command line thing. And then I think 646 00:33:59,766 --> 00:34:02,166 Speaker 2: because of that, like the formatting, it was green. 647 00:34:02,726 --> 00:34:04,846 Speaker 1: Okay, so go on. So you saw something that was 648 00:34:04,886 --> 00:34:05,946 Speaker 1: like this and it was about? 649 00:34:06,846 --> 00:34:08,846 Speaker 2: Yeah, it was kind of just like a joke story 650 00:34:08,906 --> 00:34:12,546 Speaker 2: about being a bottomless pit supervisor. And I was like, okay, 651 00:34:12,566 --> 00:34:14,046 Speaker 2: this is like, this is kind of funny. This is 652 00:34:14,066 --> 00:34:16,366 Speaker 2: the first time I've like laughed at something that was 653 00:34:16,846 --> 00:34:17,506 Speaker 2: AI generated. 654 00:34:18,026 --> 00:34:19,006 Speaker 1: How did you come across it? 655 00:34:19,696 --> 00:34:21,996 Speaker 2: It was posted on Twitter. I can read it for 656 00:34:22,016 --> 00:34:22,576 Speaker 2: you if you like. 657 00:34:23,136 --> 00:34:23,916 Speaker 1: Yeah, that'd be great. 658 00:34:24,016 --> 00:34:28,576 Speaker 2: Okay. So it starts with the human prompt is write 659 00:34:28,616 --> 00:34:32,726 Speaker 2: me a 4chan green text, be me bottomless pit supervisor. 660 00:34:33,396 --> 00:34:37,186 Speaker 2: And then AI continues this with in charge of making 661 00:34:37,226 --> 00:34:40,666 Speaker 2: sure the bottomless pit is in fact bottomless. Occasionally have 662 00:34:40,706 --> 00:34:42,546 Speaker 2: to go down there and check if the bottomless pit 663 00:34:42,586 --> 00:34:45,446 Speaker 2: is still bottomless. One day I go down there and 664 00:34:45,466 --> 00:34:48,376 Speaker 2: the bottomless pit is no longer bottomless and The bottom 665 00:34:48,396 --> 00:34:52,396 Speaker 2: of the bottomless pit is now just a regular pit. Distress.jpg. 666 00:34:53,056 --> 00:34:56,026 Speaker 2: Ask my boss what to do. He says, just make 667 00:34:56,066 --> 00:34:59,976 Speaker 2: it bottomless again. I say, how? He says, I don't know. 668 00:35:00,066 --> 00:35:05,026 Speaker 2: You're the supervisor. Rage.jpg. Quit my job. Become a regular 669 00:35:05,066 --> 00:35:08,206 Speaker 2: pit supervisor. First day on the job, go down to 670 00:35:08,246 --> 00:35:09,786 Speaker 2: the new hole. It's bottomless. 671 00:35:11,546 --> 00:35:17,866 Speaker 1: That's great. It also doesn't sound like AI. Yeah. To me, 672 00:35:17,986 --> 00:35:18,926 Speaker 1: to my ear, anyway. 673 00:35:19,446 --> 00:35:23,046 Speaker 2: Yeah, I think that's because this was, like, old AI. 674 00:35:23,146 --> 00:35:26,086 Speaker 2: I want to say it's, like, GPT-2 or 3. 675 00:35:26,086 --> 00:35:28,266 Speaker 1: I also wonder, like, it sounds like it's not a 676 00:35:28,336 --> 00:35:30,156 Speaker 1: form that I'm familiar with, but it sounds like it 677 00:35:30,196 --> 00:35:33,056 Speaker 1: is this very particular form of writing that exists in 678 00:35:33,096 --> 00:35:35,776 Speaker 1: the world. And so, just like if you say, write 679 00:35:35,816 --> 00:35:38,636 Speaker 1: a sonnet or whatever, I don't know, maybe Pangram can 680 00:35:38,676 --> 00:35:40,956 Speaker 1: tell an AI sonnet from a human-generated sonnet, or maybe 681 00:35:40,996 --> 00:35:45,406 Speaker 1: it's not enough words. But, like... When there is a form, 682 00:35:45,526 --> 00:35:48,486 Speaker 1: maybe we all sort of collapse. Humans collapse to the 683 00:35:48,546 --> 00:35:50,646 Speaker 1: form as well. And so the collapse of AI is 684 00:35:50,686 --> 00:35:51,886 Speaker 1: maybe less obvious. I don't know. 685 00:35:52,086 --> 00:35:54,356 Speaker 2: I think so. Yeah. The collapse of the form and 686 00:35:54,456 --> 00:35:59,816 Speaker 2: adherence to this structure is, I think, makes it interesting. 687 00:35:59,976 --> 00:36:01,936 Speaker 2: And it's very interesting that AI is able to adhere 688 00:36:01,976 --> 00:36:02,956 Speaker 2: to the structure so well. 689 00:36:03,456 --> 00:36:06,966 Speaker 1: And it gets the joke, right? Like bottomless pit supervisor 690 00:36:07,046 --> 00:36:11,296 Speaker 1: is an absurd premise, right? And it treats it just right. 691 00:36:11,356 --> 00:36:13,436 Speaker 1: Like it's a deaf comic touch, right? 692 00:36:14,236 --> 00:36:14,856 Speaker 2: Yeah, it was good. 693 00:36:15,796 --> 00:36:18,036 Speaker 1: Tell me about your Twitter game, your X game. You're 694 00:36:18,116 --> 00:36:20,196 Speaker 1: on a lot and you're kind of a poster. Like 695 00:36:20,796 --> 00:36:23,686 Speaker 1: your persona on Twitter is a little bit more, I 696 00:36:23,686 --> 00:36:25,866 Speaker 1: don't know if combative is too strong of a word, 697 00:36:26,246 --> 00:36:27,666 Speaker 1: than your persona in this interview. 698 00:36:28,926 --> 00:36:30,886 Speaker 2: I like to be on Twitter a lot. I post 699 00:36:31,086 --> 00:36:33,446 Speaker 2: all the time. I kind of just like post what 700 00:36:33,466 --> 00:36:36,146 Speaker 2: I think. I don't think too hard about it. But 701 00:36:36,266 --> 00:36:40,046 Speaker 2: I think Twitter is a really great community because... First off, 702 00:36:40,086 --> 00:36:43,446 Speaker 2: I like text. I'm not on TikTok. I'm not filming 703 00:36:43,486 --> 00:36:46,766 Speaker 2: videos of myself. I like just typing things out. So 704 00:36:46,846 --> 00:36:49,806 Speaker 2: I think it's a great place for me. And it's 705 00:36:49,866 --> 00:36:52,596 Speaker 2: also kind of like where all the AI happenings happen. 706 00:36:53,176 --> 00:36:55,676 Speaker 2: So a lot of the people I follow are people 707 00:36:55,696 --> 00:36:59,196 Speaker 2: who work in AI in various capacities, either AI researchers 708 00:36:59,556 --> 00:37:04,336 Speaker 2: or independent people doing cool things with AI. So I 709 00:37:04,376 --> 00:37:04,976 Speaker 2: post a lot. 710 00:37:05,696 --> 00:37:08,516 Speaker 1: Are you not more combative on Twitter? Am I misreading that? 711 00:37:09,666 --> 00:37:13,206 Speaker 2: Sometimes I think, so I often will get like, somebody 712 00:37:13,246 --> 00:37:15,766 Speaker 2: will say, you know, pangram was wrong on like so-and-so 713 00:37:15,826 --> 00:37:18,476 Speaker 2: and then I'll have to jump in and be like, Hey, 714 00:37:18,506 --> 00:37:20,256 Speaker 2: you know, can you send me the text? Like we 715 00:37:20,276 --> 00:37:22,596 Speaker 2: want to go like triage it and figure that out. 716 00:37:22,636 --> 00:37:22,836 Speaker 1: Yeah. 717 00:37:23,796 --> 00:37:28,116 Speaker 2: Uh, in terms of combativeness, I, I'm not like picking 718 00:37:28,176 --> 00:37:30,916 Speaker 2: fights all the time, but I, I mean, I think 719 00:37:30,956 --> 00:37:33,946 Speaker 2: maybe it's just a function of the platform and the 720 00:37:33,986 --> 00:37:36,446 Speaker 2: format where like, you know, here we're having such a 721 00:37:36,506 --> 00:37:40,366 Speaker 2: like nice, pleasant conversation. But, like, there's no pleasant conversations 722 00:37:40,446 --> 00:37:40,846 Speaker 2: on Twitter. 723 00:37:41,366 --> 00:37:45,206 Speaker 1: Yeah, it's interesting. I will occasionally get angry emails, as 724 00:37:45,306 --> 00:37:48,786 Speaker 1: one does. And almost every time, if I write back 725 00:37:48,826 --> 00:37:51,326 Speaker 1: in a polite way, the person apologizes. I don't even 726 00:37:51,466 --> 00:37:53,306 Speaker 1: ask them for an apology. I just say, yes, thank 727 00:37:53,326 --> 00:37:55,946 Speaker 1: you for pointing that out. Here is my thinking. And 728 00:37:57,006 --> 00:37:58,986 Speaker 1: more often than not, if I do that, the person 729 00:37:59,826 --> 00:38:06,336 Speaker 1: will write back apologetically, basically, or certainly gratefully. What's your 730 00:38:06,396 --> 00:38:07,496 Speaker 1: least favorite AI thing? 731 00:38:10,376 --> 00:38:14,756 Speaker 2: My least favorite is when it's like, and honestly. I'm 732 00:38:14,776 --> 00:38:15,756 Speaker 2: just like, that's horrible. 733 00:38:18,946 --> 00:38:21,546 Speaker 1: Load-bearing and doing the work are the two. 734 00:38:21,406 --> 00:38:24,946 Speaker 2: That, yeah, those make me cringe. 735 00:38:32,196 --> 00:38:36,696 Speaker 1: Max Biro is the co-founder and CEO of Pangram. Today's 736 00:38:36,736 --> 00:38:40,106 Speaker 1: show was produced by Gabriel Hunter Chang. It was engineered 737 00:38:40,146 --> 00:38:44,086 Speaker 1: by Sarah Bruguere. And our showrunner and editor is Jake Harper. 738 00:38:44,886 --> 00:38:47,136 Speaker 1: I'm Jacob Goldstein, and we'll be back next week with 739 00:38:47,196 --> 00:38:48,576 Speaker 1: another episode of What's Your Problem?