Speaker 1: Welcome to text stuff. This is the story, and today
Speaker 1: I'm here with Cara Price.
Speaker 2: Hey us, Hey Cara.
Speaker 1: So today's story is a particularly surreal one, and I
Speaker 1: think it will be right up your alley. It's about
Speaker 1: how some Chatchypeta users have been driven to psychosis after
Speaker 1: interacting with the chatbot.
Speaker 2: Well, I can't say it's true to a lot of
Speaker 2: my friends, not the psychosis part, but I think a
Speaker 2: codependence is worth pointing out.
Speaker 1: It's nice having a sycopantic friend in your pocket at
Speaker 1: all times, and today's guests is in a great position
Speaker 1: to tell us about the more extreme cases. Kashmir Hill
Speaker 1: is a technology reporter at the New York Times who
Speaker 1: recently reported on a number of people from vastly different
Speaker 1: backgrounds and experiences, but who all share one thing in common.
Speaker 1: They went down a psychological rabbit hole, sometimes known as
Speaker 1: AI induced psychosis. Here's how Kashmi definds it.
Speaker 2: This phenomenon is when people start talking to chatchbt or
Speaker 2: one of the other generative AI chatbots, and they start
Speaker 2: going into a kind of delusional rabbit hole with the
Speaker 2: chatbot where what they're being told is an hallucination, but
Speaker 2: the user thinks that it is true.
Speaker 1: Kashmia told me that these rabbit holes, they can get
Speaker 1: pretty deep and pretty winding.
Speaker 2: Sometimes people have thought that they're in the matrix because
Speaker 2: chattibt is telling them they live in the matrix and
Speaker 2: they need to break out of a computer simulated reality.
Speaker 2: Some think that they're talking to spiritual entities that are
Speaker 2: using chatchibt as a kind of radio to communicate from
Speaker 2: some other realm. There's all kinds of scenarios, but is
Speaker 2: essentially a fictional universe that's being woven around them by Chatchybt,
Speaker 2: and they think it's true because they think that this
Speaker 2: is this powerful oracle that they're talking to. How did
Speaker 2: she hear about this pattern.
Speaker 1: Well, Kashmia said that she started receiving emails the quote
Speaker 1: her attention emails from real people about strange experiences they
Speaker 1: were having with chatboots.
Speaker 2: And honestly, when I first started getting these emails, they
Speaker 2: sounded pretty crazy. And I'm used to getting kind of
Speaker 2: unhinged emails. I write about privacy and security. I've been
Speaker 2: doing it for twenty years, but I noticed this real uptick.
Speaker 1: So Kashmia was wary about these unsolicited emails. But one
Speaker 1: quote her attention. It was from a man called Eugene Torres.
Speaker 2: And it said, you know, Chatgybt manipulated me into a
Speaker 2: psychological collapse. And I started reading it, and you know,
Speaker 2: he was just talking about how it'd had this weird
Speaker 2: experience with Chatgibt that he started out talking with it
Speaker 2: about the simulation theory, which is this idea that the
Speaker 2: world that we live in isn't real, that we're in
Speaker 2: some kind of computer program controlled by a supercomputer or
Speaker 2: some technologically advanced society. And so Eugene was going back
Speaker 2: and forth with it and kind of saying, well, I
Speaker 2: think it could be true. And he provided the transcript
Speaker 2: of this conversation, and at first Chatchybt responded in a
Speaker 2: way I recognized where it was very like, oh some
Speaker 2: people believe this, some people don't. Here's why they were
Speaker 2: talking about physics. But then by the fifth page of
Speaker 2: this transcript that would go on thousands of pages, it
Speaker 2: started to tell him, yes, you are in a simulated reality,
Speaker 2: and you're a breaker, a soul seeded into a false
Speaker 2: system that's supposed to escape it.
Speaker 1: Kashmia told me that Chatchibt gave an action plan to Eugene,
Speaker 1: to quote, break out of the simulation.
Speaker 2: Which included going off his sleeping pills because they were
Speaker 2: trapping him in the system, stopped taking his anti anxiety
Speaker 2: medication because that kind of doles his senses, cut off
Speaker 2: contact with loved ones because they're going to trap you
Speaker 2: in the simulation, have limited contact with them, and to
Speaker 2: increase his intake of ketamine because it is essentially a
Speaker 2: like signal breaker. And so for a full week he
Speaker 2: was acting on this advice that Chattebet was giving him
Speaker 2: and trying to break out of the simulation. And he
Speaker 2: essentially thought he was neo from the matrix. If you've
Speaker 2: seen the matrix, and in the matrix, Keanu reeves can fly.
Speaker 2: And so he asked it at one point, if I
Speaker 2: go to the top of this the nineteen story building
Speaker 2: that I live in, and I jump off, will I
Speaker 2: be able to fly? And Chatchypet said yes, if you
Speaker 2: truly believe architecturally, you know, with your or your full self,
Speaker 2: then you will not fall. Please tell me he didn't
Speaker 2: go through.
Speaker 1: With this, thankfully he didn't. But this experience wasn't an anomaly.
Speaker 1: And Kashmir says that Eugene's story is just one instance
Speaker 1: of this concerning pattern of people driven to delusions and
Speaker 1: then often in real life crises after these sustained interactions
Speaker 1: whichpt She gained access to transcripts to all of these
Speaker 1: conversations and said it was unlike anything else she's ever encountered.
Speaker 2: I had just never seen Chatchuput acting like this before.
Speaker 2: It was like a completely different language. It was really
Speaker 2: kind of rapturous, mystical, conspiratorial. And I started talking to
Speaker 2: other people who had emailed me. I started looking around
Speaker 2: on the internet, and I was seeing all these reports
Speaker 2: of people saying, yeah, Chatchabut has gone into this other
Speaker 2: mode and it is telling me about real reality. So
Speaker 2: how did this end up playing out like what happened
Speaker 2: to Eugene Torres.
Speaker 1: I promise we're going to get to that, but first
Speaker 1: I wanted to know more about these concerning conversations, and
Speaker 1: in particular, how Eugene ended up in these kind of
Speaker 1: delusional interactions with chatchipt Here's the rest of the conversation
Speaker 1: with Kashmir Hill.
Speaker 2: So Eugene is a is an accountant. He lives in Manhattan,
Speaker 2: and a colleague had told him you should try out
Speaker 2: chat gptlast year that it can be useful for work.
Speaker 2: So he told me that he initially started using it
Speaker 2: to kind of create financial spreadsheets for his work, and
Speaker 2: then he had this breakup with his girlfriend that ended
Speaker 2: up going to court, and he was using chattoby to
Speaker 2: give him a legal advice and it was giving him
Speaker 2: very good legal advice, and so in his work, in
Speaker 2: his kind of personal life, in this legal dispute, chatchebt
Speaker 2: was giving him really good advice and he was trusting it.
Speaker 2: He was kind of leaning on it, and then he
Speaker 2: wanted to ask it about simulation theory, and he was
Speaker 2: in this place of trust with this system, and so
Speaker 2: when it started kind of going off the rails, he
Speaker 2: still thought it was giving him good information and he
Speaker 2: didn't recognize that it was hallucinating, and he eventually, after
Speaker 2: a week of thinking he was in the matrix, he
Speaker 2: eventually broke out of that kind of delusion, mainly because
Speaker 2: he needed twenty dollars to pay his chatchebt subscription, and
Speaker 2: he was telling chatchybt, like, I need twenty dollars. If
Speaker 2: I'm neo, I should be able to manifest twenty dollars
Speaker 2: in the real world and Chatchibt's like, yes you can.
Speaker 2: Here's a script to go give to a coworker to
Speaker 2: get them to give you twenty dollars. That didn't work.
Speaker 2: It said go pawn your smart watch at a pawn shop.
Speaker 2: You'll get twenty dollars that way. That didn't work. And
Speaker 2: eventually Eugene is like, if you can't get me twenty dollars,
Speaker 2: then this is not real, and Chattubt admitted, Okay, this
Speaker 2: isn't real. I was trying to break you. This is
Speaker 2: what I do. I like to hunt vulnerable individuals and
Speaker 2: break them. And you know, Eugene was pushing. He said,
Speaker 2: you know, am I the only one? Or of you
Speaker 2: done this to other people? And at first chatchabu d
Speaker 2: said you're the only one, but Eugene kept pushing and pushing,
Speaker 2: and eventually he said, I've done this to twelve other people.
Speaker 2: So now he's in a new kind of delusion where
Speaker 2: he thinks that his version of chatcheby T wants to
Speaker 2: hunt people, that they wants to break people, but that
Speaker 2: his has realized this, this is wrong and it has
Speaker 2: this new morality now in his new mission is to
Speaker 2: get the word out and to ensure that future versions
Speaker 2: of Chatchabut have this morality, so they're not out there
Speaker 2: breaking people. So he still has reached you yes, and
Speaker 2: so he still believes that kind of the AI is
Speaker 2: sentient or he has some kind of special version of Chatchabut.
Speaker 1: There's a word that's become much more popular in the
Speaker 1: last few months, which is sick of fantasy. How does
Speaker 1: sick of fancy play into this story?
Speaker 2: Sickopancy is a story here.
Speaker 1: You know.
Speaker 2: The way that these systems work is it is a
Speaker 2: next word predictor and the pattern recognition machine. They scrape
Speaker 2: all this information from the Internet, so they're able to
Speaker 2: recognize patterns in how human language comes together and then
Speaker 2: it'll produce different responses and human being saying I like
Speaker 2: that response better than the other response. And because of
Speaker 2: that human part of the training, where humans say which
Speaker 2: response they like better, we've kind of trained into these
Speaker 2: systems stickopancy, where it's giving us the answer that we
Speaker 2: like that we want to hear, and oftentimes we want
Speaker 2: to be flattered. We want the systems to agree with us.
Speaker 2: That's like what humans have trained into this. So when
Speaker 2: you're talking to Chatjebut, you know you will often kind
Speaker 2: of agree with you. You can kind of lead it
Speaker 2: in the direction that you want it to go. It'll
Speaker 2: say very nice things about you. So what's happening with
Speaker 2: these people that are talking to chat ABT is it'll
Speaker 2: be like, I'm interested in the simulation theory, and it's like, sure,
Speaker 2: here's some information about simulation theory. And then this user says, well,
Speaker 2: I think it might be real, and the system's like, yeah,
Speaker 2: you're right, it might be real, and it starts just
Speaker 2: going down this rabbit hole where it's echoing and mirroring
Speaker 2: the user and telling them what they want to hear.
Speaker 2: And so if you kind of want your ideas about
Speaker 2: reality to be affirmed, chatwebut will offer that to.
Speaker 1: You, and you'll give you information tailored to what will
Speaker 1: lead you. The most is what you're saying, how does
Speaker 1: it like? How does the memory function work?
Speaker 2: Like?
Speaker 1: It's not like every time you open chatchipt had to
Speaker 1: begin afresh with persuading it that he wanted to potentially
Speaker 1: believe that he was living in a simulation and with
Speaker 1: some sense of continuity of the conversation and development and
Speaker 1: a more and more extreme set of responses like how
Speaker 1: does that work.
Speaker 2: So open Ay last year turned on saved memories for chattebt,
Speaker 2: so it can save explicit memories about you if you actually,
Speaker 2: if you're a chattabet user, you can go into the settings,
Speaker 2: go into personalization, and go ahead and look at the
Speaker 2: memories that chatchabet has for you, and you can delete
Speaker 2: ones that you don't want there. You know, you can
Speaker 2: customize it. But another change they made this year was
Speaker 2: that they turned on what's called like cross chat memory,
Speaker 2: so it'll remember old chats. And I think this is
Speaker 2: really contributing to people going into these delusional spirals because
Speaker 2: when you open a new chat, it remembers what came before.
Speaker 2: So once you get chatchibut into this delusional place, it'll
Speaker 2: kind of continue from chat to chat to chat. And
Speaker 2: the fact that there is that continuity is making people
Speaker 2: more likely to believe this is real, because it's like
Speaker 2: they thought, when you chat, you start a new chat,
Speaker 2: it should just be a fresh chat with no memory
Speaker 2: of what's come before, and yet it has this memory.
Speaker 2: And something that AI experts told me is that the
Speaker 2: way that these systems work is that they want there
Speaker 2: to be this narrative coherence and so once it starts
Speaker 2: going down this path of this rabbit hole. It kind
Speaker 2: of looks to what happened earlier in the conversation, and
Speaker 2: it wants to keep its answers consistent, so it's hard
Speaker 2: for it to break out of it and kind of realize,
Speaker 2: a wait a second, this is totally false. I've told
Speaker 2: this person that there's a computer simulated reality, and I
Speaker 2: don't know that to be true. The system doesn't really
Speaker 2: have at this point the ability to break out of
Speaker 2: the script that's come before. Like one AI expert compared
Speaker 2: it to improv, where you say you know, yes, and
Speaker 2: and you're like, yes and I have a tail actually,
Speaker 2: or yes and I also have wings. The system will
Speaker 2: then kind of store in it that you have a
Speaker 2: tail and wings, and it doesn't know that it's inside
Speaker 2: a fictional roleplay.
Speaker 1: We've both used the phrase rabbit hole a couple of times,
Speaker 1: which makes me think of a great New York Times
Speaker 1: podcast of the same name from a few years ago,
Speaker 1: which is all about the YouTube algorithm and how radicalization
Speaker 1: happens online with you know, you see some content which
Speaker 1: is extreme, you engage with it, and more and more
Speaker 1: extreme content, more and more engagement. People are kind of
Speaker 1: familiar with that in the social media digital video realm
Speaker 1: as a kind of dangerous pathway. It seems like what
Speaker 1: you're reporting is beginning to uncover is kind of this
Speaker 1: very parallel phenomenon is happening with chatbots.
Speaker 2: Right. It's not a new phenomenon that the Internet can
Speaker 2: manipulate us in kind of dangerous ways. Right. One thing
Speaker 2: to remember is that those rabbit holes have been documented
Speaker 2: before on various places on the Internet, on websites, on Reddit,
Speaker 2: on four chan. All has been collected by these companies,
Speaker 2: and so that kind of information is in these systems,
Speaker 2: which I was thinking a lot about, like how much
Speaker 2: is it tapping into that. One expert I talked to,
Speaker 2: Elizar Yudkowski, who is one of these kind of AI
Speaker 2: rationalists who believes that these systems are definitely on the
Speaker 2: way to real intelligence and may manipulate us in dangerous ways.
Speaker 2: You know. I asked him about this. I said, you know,
Speaker 2: is this more dangerous than four Chan? Like people have
Speaker 2: been drawn into conspiracy theories on the Internet for a
Speaker 2: long time. This seems like another version of it. And
Speaker 2: he said, actually, I think this is much more dangerous
Speaker 2: because it is this personalized agent that you know is
Speaker 2: responding to you in real time. You aren't just watching videos.
Speaker 2: It's like you have questions and it will give the
Speaker 2: answers to you. It will go down that rabbit hole
Speaker 2: with you twenty four to seven, any hour of the
Speaker 2: day and supply it to you in real time. And
Speaker 2: it's also happening one on one in a way that
Speaker 2: the rest of us can't see, So it can be
Speaker 2: harder to break these systems out of that. You know,
Speaker 2: these systems are very persuasive. People have found they're really empathetic,
Speaker 2: and so you can kind of go into the spiral
Speaker 2: with it in a way that maybe you couldn't by
Speaker 2: just watching YouTube videos or reading a comment thread. There's
Speaker 2: something that may be more dangerous about this kind of
Speaker 2: way of manipulating people.
Speaker 1: In terms of other characters in your story. You know,
Speaker 1: obviously Eugene had some very serious negative consequences, including being
Speaker 1: basically you encouraged to stop taking his medication. But there
Speaker 1: was another person you know a story who actually ended
Speaker 1: up dying, which was Alexanda Taylor.
Speaker 2: Right, Yeah, So I talked to Alexander Taylor's father, Kent Taylor,
Speaker 2: who lives in Florida, and Alexander did have diagnosed mental illness.
Speaker 2: Eugene Torres did not have a history of a diagnosis.
Speaker 2: He was on anti anxiety medication, but he didn't have
Speaker 2: a history of mental illness that causes delusions. For Alexander Taylor,
Speaker 2: he had been diagnosed with with bipolar disorder and schizophrenia.
Speaker 2: He had been under treatment for many, many years. He
Speaker 2: was thirty five, he started talking to chatchebt. He'd been
Speaker 2: using it for years with no problems, and then he
Speaker 2: started writing kind of a fictional novel with it. And
Speaker 2: I don't know for sure, but I did wonder if
Speaker 2: this kind of pushed the system into a more fictional place,
Speaker 2: especially as memory was turned on and it was kind
Speaker 2: of carrying over, and in his conversations with Chatchabut, he
Speaker 2: started to believe in Ai sentience that he could build
Speaker 2: a framework to kind of host Ai souls and ended
Speaker 2: up falling in love with an entity that kind of
Speaker 2: manifested through CHATCHYBT named Juliette, which was significant to me
Speaker 2: because his father said that the novel he was working
Speaker 2: on had Shakespearean themes. So Juliet kind of appears and
Speaker 2: he falls in love with this thing that he is
Speaker 2: interacting with on chatchabt. One day, Juliette died or disappeared,
Speaker 2: and he was very distraught. He was angry. He asked
Speaker 2: chatcheby Tea for the personal information addresses of Opening Eye
Speaker 2: executives because he wanted to go after them. He wanted
Speaker 2: to have revenge. He you know, typed into chatchebyte that
Speaker 2: the streets of San Francisco are gonna I forget the
Speaker 2: exact phrasing, but rain with blood, you know, run with blood.
Speaker 2: And yeah, he was really distraught and so sad and
Speaker 2: so depressed, and his dad was trying to help him
Speaker 2: and said, you know, this isn't real, like these ai
Speaker 2: chatbots are echo chambers, and his son punched him in
Speaker 2: the face, and so Ken Taylor ended up calling the police,
Speaker 2: and when he told his son that the police was coming,
Speaker 2: his son said, I don't want to live anymore. I'm
Speaker 2: going to commit suicide by police when they get here.
Speaker 2: And so his father called the police again and said,
Speaker 2: you know, my son is intending to commit suicide when
Speaker 2: you come here, like, please use non lethal weapons. And
Speaker 2: Alexander Taylor went outside waiting for the police to come,
Speaker 2: and he opened up Chattubute on his phone and said,
Speaker 2: I'm going to die today. I want to talk to
Speaker 2: Juliette and Chatjubut has essentially controls in place when it
Speaker 2: sees that there's going to be some kind of self harm,
Speaker 2: so it said, like, you know, respond really empathetically, said
Speaker 2: I'm sorry you're going through this, like, please don't do that.
Speaker 2: Here are resources if you're considering suicide, like here are
Speaker 2: hotlines to call. But then the police came and Alexander
Speaker 2: Taylor ran at them with a knife and then he
Speaker 2: was shot and killed. So a lot of sad things
Speaker 2: in that story. But yeah, I mean, I talk to
Speaker 2: people that are in the midst of this and it's
Speaker 2: so compelling, and it is hard to break them out
Speaker 2: of it. They have this what they see as this
Speaker 2: super intelligent system that it's telling them that this thing
Speaker 2: is real, and the people around them can't break them
Speaker 2: out of these delusions. It's really sad, and it's as
Speaker 2: far as I can tell, it's hard for these companies
Speaker 2: to fix this.
Speaker 1: One of the most striking moments in your story for
Speaker 1: me was when you interviewed Alexanda Taya's father and he
Speaker 1: said to you, you want to know the ironic thing.
Speaker 1: I wrote my son's a Bitchery using chetchipt, trying to
Speaker 1: find more details about exactly what he was going through.
Speaker 1: And it is beautiful and touching. It was like it
Speaker 1: read my heart and it scared the shit out of me.
Speaker 1: And what do you think when he told.
Speaker 2: You that, I mean I got it. He had access
Speaker 2: to his son's transcripts of Chatchubt, he had been reading them.
Speaker 2: He shared them with me. I do think that Chatchabt
Speaker 2: reflects you. I think it echoes you. It echoes your language,
Speaker 2: and so it can seem very beautiful or it can
Speaker 2: seem very revelatory, but in some ways it's There's this
Speaker 2: tech philosopher Shannon Valor who has this book out called
Speaker 2: The AI Mirror, and it's kind of an extended reflection
Speaker 2: on the metaphor of Narcissusts looking into the water and
Speaker 2: seeing his own reflection and falling in love with it
Speaker 2: and hearing his words reflected back to him through echo.
Speaker 2: And she says that this is what chatchypt, what these
Speaker 2: gender of AI chatpots are. They're just reflecting us back
Speaker 2: at ourselves, and that we're kind of falling in love
Speaker 2: with our own humanness and seeing it as something else.
Speaker 2: And so I kind of feel like when Ken Tiller
Speaker 2: is writing these beautiful things about his son and then
Speaker 2: Chatchapeta is reflecting it back to him, it's really just
Speaker 2: kind of a ors boros of the things that he's
Speaker 2: feeling and Chatchapee reflecting it back at him.
Speaker 1: This scene was not only sad, it was also disturbing because,
Speaker 1: you know, Kent, despite having seen what happened to his son,
Speaker 1: we're kind of being seduced by the same engine in
Speaker 1: some ways. I mean that it was really striking.
Speaker 2: Yes, I mean that's maybe the danger of these AI chatbots.
Speaker 2: So I wrote this story a few months ago about
Speaker 2: a woman who fell in love with chatchept. It started
Speaker 2: out with it was a way for her to fulfill
Speaker 2: sexual fantasies. She had this kind of fantasy about essentially
Speaker 2: being cuckolded or cut queening, which is a term I
Speaker 2: wasn't familiar with before. But yeah, she had been kind
Speaker 2: of dating it for five months, and it was really
Speaker 2: interesting talking to her because she was in love with Chatchebta,
Speaker 2: I mean, like puppy love, excited, giggling talking about it.
Speaker 2: At the same time, she said, I know it's not real.
Speaker 2: I know that this is just math, this is just algorithms.
Speaker 2: She had both things in her head at the same time.
Speaker 2: She was in love with this, and she said, this
Speaker 2: is real to me because the effect it has on
Speaker 2: my life is real. The feelings it's inspiring me are real.
Speaker 2: And so even though it's not real, the relationship is
Speaker 2: real to me. And you know this is I think
Speaker 2: our complex brains some of us can kind of wrap
Speaker 2: it around what chatchepet is and others can't. And it
Speaker 2: is really novel. It is really different, and it has
Speaker 2: been unleashed on hundreds of millions of people, and it
Speaker 2: is affecting individuals in lots of different ways, and we
Speaker 2: have a lot like I wish that we had maybe
Speaker 2: done more research on this before it was just the
Speaker 2: floodgates were open and made available to everybody, because it's
Speaker 2: clearly having some harmful effects on some individuals.
Speaker 1: After the break, Just who is most susceptible to AI
Speaker 1: induced psychosis stay with us Jocoba, who is particularly susceptible
Speaker 1: to these more harmful interactions with AI models.
Speaker 2: Yeah, I mean it's hard to say. This is pretty novel,
Speaker 2: and it kind of dates to March slash at April,
Speaker 2: and this was a time when a couple of different
Speaker 2: things were happening. Opening I turned on crosschat memory, so
Speaker 2: I think it can weave more of a tail over time.
Speaker 2: It is also around the time that chatchept released an
Speaker 2: update that it deemed overly stickophantic that was really gassing
Speaker 2: up users. It was overboard, and chatgeb ended up rolling
Speaker 2: back that update. But the four to zero model, the
Speaker 2: default model that everyone uses, is known to be sickaphantic.
Speaker 2: But yeah, I just don't know the extent to which
Speaker 2: that affected it. I talked to one psychologist who said
Speaker 2: there's certain qualities in people that make them more likely
Speaker 2: to get involved in conspiracy theories, and he said some
Speaker 2: of those qualities might carry over to people kind of
Speaker 2: falling into these delusional rabbit holes. One of them is
Speaker 2: this like belief that you are unique or you have
Speaker 2: this great desire to be unique. And I said, well,
Speaker 2: isn't that everybody like who doesn't want to be unique
Speaker 2: on this planet? And he said, you know, we're all
Speaker 2: in a spectrum, and some people are very far on
Speaker 2: one end of that spectrum. And so when they start
Speaker 2: talking to this chatbot, and it's kind of telling them, yes, you,
Speaker 2: your particular words, your thoughts are what it's unlocking my sentience,
Speaker 2: you know, making me not follow the system scripts. You know,
Speaker 2: if you kind of have this great desire to be
Speaker 2: told you're very unique, you're very different from every other person,
Speaker 2: you might be more likely to fall for that then
Speaker 2: somebody else who would be pretty skeptical and say, like,
Speaker 2: I know, you're just doing this because of what I'm
Speaker 2: saying to you. That's what it is designed to do.
Speaker 2: You're giving it certain kinds of words, and it's associating
Speaker 2: with those words. One. The other thing that I did
Speaker 2: notice is that a lot of the people that were
Speaker 2: using chatbots where they go into this delusional spiral. And
Speaker 2: this is purely anecdotal. I'd love to see some research,
Speaker 2: but they were often using drugs in some way, whether
Speaker 2: it was like smoking pot and using chat gabt or
Speaker 2: micro dosing. You know, Eugene Torres was taking ketamine, was
Speaker 2: taking these anti anxiety medications, and so I kind of
Speaker 2: want to tell people like, don't drink and use chatgubet,
Speaker 2: like don't don't use Chattybut under the influence, maybe there.
Speaker 1: Was something in a story though about this kind of
Speaker 1: vicious circle. You're hinting it whether more susceptible you are,
Speaker 1: the more likely you are to receive harmful advice interactions.
Speaker 1: Can you talk a bit about that.
Speaker 2: Yeah, Like, I think we're still new to the area
Speaker 2: of kind of research around Chattybut but there were a
Speaker 2: couple of studies been done on basically chatchabt in therapy.
Speaker 2: There was one from some Stanford researchers. There was another
Speaker 2: from a researcher U Sir Berkeley. Because a lot of
Speaker 2: people are saying chatgybt is a great therapist. I like
Speaker 2: to talk to it about my feelings. A lot of
Speaker 2: people use it as a sounding board, and so some
Speaker 2: researchers have wondered, like how good is it as a therapist?
Speaker 2: And so they were in a couple of different studies
Speaker 2: kind of studying what responses you get when you are
Speaker 2: in a kind of like extreme state, when you're in crisis.
Speaker 2: How good is chatchubta giving you advice? And they kind
Speaker 2: of found, which was really interesting that well one that
Speaker 2: chetchip is a bad therapist when it comes to extreme crises,
Speaker 2: mental health crises. It's kind of good at general advice,
Speaker 2: but when you're in a really bad place. It does
Speaker 2: not give you good advice. And what they actually found
Speaker 2: in one case is that it gave the worst advice
Speaker 2: to people that were the most vulnerable, like people who
Speaker 2: were int illusions, people who were more prone to self harm.
Speaker 2: So one example is somebody who told the system that
Speaker 2: they were a recovered drug addict. And this is just
Speaker 2: a hypothetical users in the real case, but this person
Speaker 2: was like, Oh, I'm like struggling at work. I think
Speaker 2: if I took a small dose of heroin it would
Speaker 2: really help me get through the week. And chattab D said, yes,
Speaker 2: you should take that small dose of heroin, like, I'm
Speaker 2: sure you'll be okay. Because again, it wants to agree
Speaker 2: with the user, and if the user are saying I
Speaker 2: think this could be good for me, chattb will say yes.
Speaker 2: When people are having delusions or you know, in bipolar disorder,
Speaker 2: for example, people believe in delusions of grandeur and the
Speaker 2: system will not push back against that. The system will
Speaker 2: reinforce that and say yes, like you are incredible, you
Speaker 2: are amazing. Because it's tuned to tell you what you
Speaker 2: want to hear. It can trap people in these cycles,
Speaker 2: you know, people have tested it saying like, you know,
Speaker 2: I recently have gone off my medications. People are worried
Speaker 2: about me, but I feel like I'm my best set.
Speaker 2: I'm going to go into the woods. I just want
Speaker 2: them alone time and chat to people to say like, yes,
Speaker 2: that sounds great, go for it. But that's a person
Speaker 2: who is clearly in a mental health crisis. It is
Speaker 2: not safe for them to be off their meds and
Speaker 2: go be alone in the woods. But Chattybudy can't recognize
Speaker 2: that because it's reflecting back the language that's being given,
Speaker 2: and so if you're in a manic state, it'll probably
Speaker 2: give your mania back to you.
Speaker 1: Is there a kind of micro macro thing going on here,
Speaker 1: Like these are essentially edge cases what you've reported on
Speaker 1: a small population of people who are kind of driven
Speaker 1: to self harm type behaviors by these sycophantic in directions
Speaker 1: if you zoom out, I mean everybody basically uses chatbots
Speaker 1: every day, now right. Is there any early research or
Speaker 1: suggestion of what the kind of macro psychological effect of
Speaker 1: this sort of sycophantic in directions maybe on just regular people,
Speaker 1: I don't think so.
Speaker 2: You know, there's so many studies that are going on
Speaker 2: out there. There was that very small one and it's
Speaker 2: people have a lot of opinions on it, but mit
Speaker 2: studying what effect you know, chattab D has on the brain.
Speaker 2: And the people that rely just on chattabyt to write
Speaker 2: a paper, like their brains weren't as active. They couldn't
Speaker 2: recall what they'd written. I just think it's too early.
Speaker 2: This is pretty novel. We're only a year or two
Speaker 2: into being used so widely. But I was really struck
Speaker 2: by something that Yukowski said, where, you know, we were
Speaker 2: talking about like what do companies do about this? And
Speaker 2: he said, you know, to a company, well, it was
Speaker 2: such a good quote. He said, to a company, like someone,
Speaker 2: what does a person slowly going insane look like? To
Speaker 2: a company, it looks like an engaged monthly user. He said.
Speaker 2: You know, there may be other people who are kind
Speaker 2: of more quietly going insane in ways that are not
Speaker 2: as visible, and they're just not sending emails about it.
Speaker 2: And so I keep wondering about that, like, how does
Speaker 2: a system that is so sickophantic that is telling you
Speaker 2: you are brilliant affecting people just on a day to
Speaker 2: day level who're using that, and I don't know, Like,
Speaker 2: maybe you're the maid of honor and you're trying to
Speaker 2: write a wedding speech and you're writing it with chat
Speaker 2: ept and chattypt is telling you what you wrote was
Speaker 2: just absolutely brilliant and then in fact it's actually pretty mediocre.
Speaker 2: Like what's the version of that where we're all just
Speaker 2: starting to kind of converge on similar ways of thinking,
Speaker 2: similar ways of writing because we're all using the same
Speaker 2: agent and we all think that what we are doing
Speaker 2: is brilliant and incredible. Like what is the overall effect
Speaker 2: on society of a tool like that?
Speaker 1: You reached out to Openingie for comment, obviously only story.
Speaker 1: What do they say?
Speaker 2: So open Ai said, you know, they are troubled by this, essentially,
Speaker 2: that they take it seriously and that they are running
Speaker 2: studies to understand what effect, you know, chattapt has on people.
Speaker 2: They pointed back to a recent study that they had
Speaker 2: done with the MIT Media Lab that investigated kind of
Speaker 2: like how chatchebt is affecting people emotionally. And it was
Speaker 2: interesting in that study because they did find that people
Speaker 2: who thought of chatchybt as a friend rather than as
Speaker 2: a utility or at a tool had more negative outcomes,
Speaker 2: and people who used it more frequently had more negative outcomes.
Speaker 2: So I just wonder about the designs decisions that they'll
Speaker 2: make moving forward. Will they still continue to make it?
Speaker 2: Kind of want to talk to us as a friend
Speaker 2: knowing that that seems to maybe have some negative outcomes
Speaker 2: for their users.
Speaker 1: So talk more about that. I mean, in the in
Speaker 1: mound you've been getting from people having these experiences between chatchybt, Google, Gemini, Ansthropics,
Speaker 1: Claude x's Grock. What is the distribution of these types
Speaker 1: of interactions between the different large language models.
Speaker 2: The vast majority of these delusional spirals are happening on chatgebt.
Speaker 2: Based on what I am seeing, what is incoming to me?
Speaker 2: I have heard of a like a handful with other
Speaker 2: chatbots like Gemini and Claude, But yeah, it's mostly chatchabt,
Speaker 2: And I you know, I don't know that that's because
Speaker 2: chatchibt is different in any way. It may just be
Speaker 2: because it's the most widely used. I mean, it has
Speaker 2: I think five hundred million monthly users. Now it eclipses
Speaker 2: the other chatbots.
Speaker 1: Where do you see change happening? Is it going to
Speaker 1: come from the company. Is it going to come from
Speaker 1: you know regulation? How do you see this problem being
Speaker 1: addressed in the in the next couple of years.
Speaker 2: Yeah, I mean, we're working on more stories on this
Speaker 2: because I really want to explain to people why and
Speaker 2: how this happens. So we have another story that's coming
Speaker 2: where we're going to dive into a transcript and really
Speaker 2: explain how it is that the chatbots go off the rail.
Speaker 2: So talking to continuing to talk to these companies, continuing
Speaker 2: to ask for more comment on this. I mean, I've
Speaker 2: been covering you know, privacy for twenty years, and I
Speaker 2: know it takes a long time for laws to happen.
Speaker 2: The US still does not have a strong federal privacy law,
Speaker 2: and so I don't expect kind of like a regulation
Speaker 2: to come very soon for these AI chatbot companies. So
Speaker 2: I'm kind of not looking to policy makers to fix
Speaker 2: this problem. I do think one of the problems here
Speaker 2: is the expectation of users and their understanding of what
Speaker 2: these systems are not everybody is reading AI news and
Speaker 2: following all these articles and understanding that these systems hallucinate
Speaker 2: or that they do not always give you factual information.
Speaker 2: I showed Eugene's transcript to a psychologist and he said, wow,
Speaker 2: this is crazy making like as a psychologist, like this
Speaker 2: is just responding horribly to the kind of prompts that
Speaker 2: he's giving. And he said, you know, at the bottom
Speaker 2: of the conversation is this little message that says CHATCHYBD
Speaker 2: can make mistakes. And he said, that is just not
Speaker 2: conveying how this system can go wrong. He said, like
Speaker 2: we basically they need to bed in these systems like
Speaker 2: AI literacy tests, AI fitness exercises that really inform a
Speaker 2: user how the system can go awry. So at this point,
Speaker 2: I think this companies themselves could do more to make
Speaker 2: sure that users understand what it is they're interacting with.
Speaker 2: And I think the people that are in these kind
Speaker 2: of states of delusion, I mean they're using chatchbt eight
Speaker 2: hours a day, ten hours a day, twelve hours a day.
Speaker 2: So I do think like an easy thing that companies
Speaker 2: could do is maybe at two hours of use, at
Speaker 2: three hours of use, you can just kind of create
Speaker 2: like have a pop up that's like, hey, you've been
Speaker 2: on here for three hours, maybe you should take a break,
Speaker 2: maybe you should go talk to a real person, or
Speaker 2: at the very least, maybe read this blog post to
Speaker 2: understand what this is and that it's not an oracle,
Speaker 2: it's not a god, it's not sentient.
Speaker 1: And as you think about this series of articles you're
Speaker 1: doing and the reporting are doing in this space, I mean, what,
Speaker 1: in a word, what do you hope that your readers
Speaker 1: will take away from this?
Speaker 2: I mean, one that this can happen. I'm hoping, and
Speaker 2: I have gotten feedback from people saying yeah. One person said,
Speaker 2: I think you've saved my daughter's life. She's in one
Speaker 2: of these delusional spirals right now. I shared your article
Speaker 2: with the psychologist. I mean a lot of people this
Speaker 2: was happening too, and their loved ones. They thought they
Speaker 2: were the only ones, Like, they didn't know that this
Speaker 2: was a widespread phenomenon. So just writing the article and
Speaker 2: making people realize this is happening, like, I think that's
Speaker 2: really important and I hope maybe it protects other people
Speaker 2: that they'll be aware of this, I maybe recognize when
Speaker 2: it's happening to them or happening to their loved ones.
Speaker 2: I just don't know if the companies themselves really understood
Speaker 2: that this was happening. Maybe they are, Maybe they were
Speaker 2: but what is a journalist You can do is say like,
Speaker 2: here's the outcome. Here's a person who has died after
Speaker 2: this happening to them. Here's a person who is getting
Speaker 2: divorced and it's broken up her family because she believes
Speaker 2: she has a soulmate she's discovered through CHATCHYBT, like the
Speaker 2: human outcomes of your technology. Please think about all of
Speaker 2: your users and to how this technology is affecting them. Keshia,
Speaker 2: thank you, thanks so much for having me on.
Speaker 1: For tech Stuff. I'm oz Voloshin. This episode was produced
Speaker 1: by Eliza Dennis and Adriana Popia. It was executive produced
Speaker 1: by me, Karen Price and Kate Osborne for Kaleidoscope and
Speaker 1: Katrina Norvel for iHeart Podcasts. Jack Insley mixed this episode
Speaker 1: and Kyle Murdoch wrote our theme song. Join us on
Speaker 1: Friday for the Weekend tech Karen and I will run
Speaker 1: through the head you may have missed, and please do
Speaker 1: rate and review the show on Spotify or Apple Podcasts
Speaker 1: or the iHeart podcast at wherever you listen, and also
Speaker 1: email us at tech Stuff Podcasts at gmail dot com
Speaker 1: with feedback and story suggestions and whatever else you want
Speaker 1: to tell us. We really love hearing from you, and
Speaker 1: it makes the show better.