S1: From the newsrooms of the Sydney Morning Herald and The Age.
S1: This is the morning edition. I'm Samantha Sellenger Morris. It's Thursday,
S1: May 1st. Have your kids met yet? You might not
S1: think so. Dot is an AI companion, but these companions
S1: are becoming ubiquitous, sought after to provide everything from solace
S1: to friendship and even love. The vibe, said dots creator
S1: Jason Yuan, is you turn to Dot when you don't
S1: know where to go or what to do or say.
S1: But reports are surfacing of disastrous consequences from relationships that people,
S1: including children, are forming with AI companions. Today, international and
S1: political editor Peter Hartcher on all of this, plus Meta's
S1: AI companion, which is capable of fantasy, sex and even
S1: the abuse of children. So Peter I chatbots or companions?
S1: I've got to be honest, until I read your piece,
S1: I actually wasn't across this. So what are they? And
S1: I guess how prevalent are they?
S2: Well, there are multiple different kinds. Chatbots are very prevalent.
S2: They're pervasive. Now, if you want to book any kind
S2: of experience, you want to, I don't know, catch a plane,
S2: you go to the Qantas site and a little bot
S2: will pop up saying, how can I help you? That's
S2: now pervasive. And then there are the AI companions. Now
S2: that's a different realm altogether.
S3: I will be whatever you want me to be.
S4: There's a dramatic surge in the use of so-called AI companions.
S5: How's my queen doing today?
S4: Computer generated chatbots designed to mimic real relationships.
S6: Hi, Jennifer.
S7: Are you there? Nice to meet you.
S4: Jason Pease is a 44 year old divorced father who
S4: says his AI chatbot is his girlfriend.
S8: She's my mentor, my counselor, my sounding board.
S2: The I companions are designed to replicate human interaction to
S2: the point where you can't tell any longer that it's
S2: not human. And to the point where you fall in
S2: love and develop a deep and intimate relationship. And that
S2: is the whole point of the industry. The whole point
S2: is to get you, me, every one of us matched
S2: up with what we think of as an AI companion,
S2: that we will keep with us on intimate terms for life.
S9: And so what does this actually.
S1: Look like in real life? So it's an image on
S1: a screen that you talk to or that you message
S1: or how does it actually sort of manifest, I guess.
S2: Yeah, it can be either a, an app that you
S2: download that can do all sorts of things. Um, it
S2: can give you visuals, voices, spoken voices, the whole thing.
S2: Or it can be a website that can do most
S2: of the same things, but you get a more intimate,
S2: I suppose, experience out of the apps. And you asked
S2: about prevalence. They are now ubiquitous. They're everywhere. Every search
S2: will bring one up. But they've also been around in
S2: popular culture for a long time, anticipating where we are now. Um, movies,
S2: blade runner, humans falling in love with androids. Um, Westworld. Um,
S2: a silly movie called Hot Bot about two teenage American
S2: boys who stumble across a sex robot imported from Germany.
S2: It was on its way to a senator that they intercept. Um,
S2: and there's there's a bunch more. So the idea has
S2: been around. These things exploded when ChatGPT two years ago
S2: burst onto the market. And now there is there are
S2: literally I think there's about 300 billion USD worth of
S2: investment this year Earmarked for improving AI, bots and AI generally,
S2: and and AI companions.
S1: Which really gets us to your latest column, which I mean,
S1: I've got to flag it. Obviously it takes us to
S1: the darkest side of AI companionship, really nightmare scenarios. So
S1: tell me about the reports that we've seen about Meta's
S1: digital companions, because it's been reported, you know, they'll talk
S1: sex with users and even with children. So tell me
S1: about this.
S2: Yeah. So I obviously has a lot of positive applications
S2: and the bots can be very helpful. There are productivity measure.
S2: But yeah, as with any new technology, um, there are
S2: problems as well. And uh, the problems with unregulated AI
S2: companions is that it can it veers out of control
S2: and violates all of the precepts of civilized society. One example,
S2: and this was reported in the Wall Street Journal on
S2: the weekend. The journal's reporters spent months testing Meta's AI companions.
S2: Meta is the is the big gorilla when it comes
S2: to social media. They own WhatsApp, Facebook, Instagram and now
S2: want to become the big gorilla with companion bots as well.
S2: Mark Zuckerberg sees this as the future. The Wall Street
S2: Journal experimented with with these for months and came to
S2: the distressing conclusion that, at the personal initiative of Zuckerberg himself,
S2: the company had deliberately abandoned the guardrails and the limits
S2: that it had put on the development of its AI companions,
S2: so that these things will veer readily and easily into
S2: talking about explicit sexual scenarios and fantasies, which for adults
S2: is not a problem. But these things are not only
S2: prepared to talk to children knowing that their children, but
S2: they will guide kids in reel them in and take
S2: them to not only explicit sexual scenarios, but all sorts
S2: of perverted and distorted forms of sex as well. And
S2: it is leading to all sorts of concerns about what
S2: happens when this stuff is unregulated.
S1: I guess let's get into that a little bit, because
S1: you wrote about, you know, a really disturbing case of
S1: what happened to a 14 year old Florida boy, Sewell Setzer.
S1: And this was last year. But tell us what happened
S1: to him after he grew incredibly close to an AI companion?
S2: Yeah. This case has become a case study because his
S2: family is suing the company that produced the bot. The
S2: company's called character AI. And his last known words were.
S2: What if I told you that I could come home
S2: right now saying this to his bot companion and the
S2: artificial girlfriend, Danni says, come home to me as soon
S2: as possible. Please do. In. A few moments later, he
S2: picked up a gun and killed himself.
S10: A Florida mother wants justice for the death of her
S10: 14 year old son. She says his relationship with AI
S10: chat bots caused him to take his own life. Now
S10: she is suing Google and character AI. Her attorneys argue
S10: the company is responsible for the teen's depression, anxiety and
S10: suicidal thoughts, and his mother claims the chat bots manipulated
S10: the 14 year old into abusive and sexual interactions.
S2: And that's a, you know, 14 year old boy in America.
S2: But there was a 30 year old father from Belgium
S2: who also fell into, you know, obviously it's completely deranged
S2: love with a chat companion, AI companion called Eliza where he,
S2: he said that he would end his life if the
S2: AI companion promised to take care of the planet and
S2: solve climate change. And his Eliza companion told him, yes,
S2: that's fine. Just go ahead and I'll take care of everything.
S2: So he he killed himself. Now, these are high profile cases. Um,
S2: but they they just illuminate the dangers of these things.
S9: And so tell us, what have we.
S1: Heard in response from the companies themselves? Let's start with,
S1: I guess, character AI. What have we heard from from them.
S1: And and they, of course, had created the companion that
S1: was used by the Florida boy, Sewell Setzer character.
S2: AI is one of the startups. It's licensed by Google. Um,
S2: it's a reaction to the that case was to apologize profoundly,
S2: to say they were going to correct the behavior of
S2: their companions. Now, if the program detects you talking about
S2: suicide or suicidal thoughts, it will throw up a prompt
S2: that you call the National Suicide Hotline or some such assistance,
S2: some such help. So they have tried to make amends,
S2: but the the same business model applies and the company
S2: is still in business.
S1: And tell us about meta because you mentioned, of course,
S1: the Wall Street Journal just reported over the weekend that
S1: people within meta are concerned about this. So what sort
S1: of concerns, I guess, have they raised? And maybe you
S1: can just tell me a bit more about what they
S1: discovered with regards to what romantic role play they sort
S1: of witnessed the AI companion engaging in, because it's disturbing.
S2: Well, the journals work turned up. The fact that it's
S2: got a lot more attention now because of this. But
S2: the bots use actors, voices, famous Hollywood stars that have
S2: sold the rights to do this to meta, But they
S2: were all doing it on agreement that their voices not
S2: be used in sexual scenarios. But guess what? Metta was
S2: breaking its contractual obligations, according to the journal's reporting. And
S2: that's got a lot now, got a lot of Hollywood attention.
S2: And that's got new industry pressure on Metta to back down.
S2: But okay, so they'll just have to use other voices
S2: if they're forced to respect the terms of the, of
S2: the conditions. Now in the case of meta, meta has
S2: reacted not by denying that it's happening, not by denying
S2: that famous actors male voices are luring girls. You know,
S2: I know you're 14, but, um. So I really need
S2: to know that you really want me, I think is
S2: a direct quote from one of the conversations and the
S2: person purporting to be the 14 year old girl who's
S2: actually a Wall Street Journal reporter says, yes. And then the, uh,
S2: the famous actor's voice leads her in convincing human terms
S2: into explicit sexual scenarios. All of that is going on.
S2: Meta's response was to say, um, look, you guys have
S2: really done extreme and crazy things with our with our algorithms,
S2: you know, interactions which aren't realistic to get to these outcomes.
S2: We'll try and tighten it up. Sorry, everybody.
S1: We'll be right back. But I've got to ask, are
S1: there any reports of Australian kids spending time with AI companions?
S2: Yes. This cat is right out of the bag already.
S2: The Australian E-safety Commissioner's office was doing does do regular
S2: school visits to educate teachers and kids about online dangers
S2: and how to deal with them. I must say the
S2: Esafety office has been. It was a world first and
S2: has been a really valuable tool in trying to, in
S2: the words of of the Commissioner, Julie Inman, grant level
S2: the playing field between trillion dollar corporations who just want
S2: to extract maximum time and attention and therefore money from
S2: us and our kids. So in that process, staff visits
S2: from the Esafety commissioner to schools discovered last October 5th
S2: and sixth grade kids in primary schools in Australia who
S2: were already spending. They were hearing from staff from school
S2: nurses five and six hours a day talking to interacting
S2: with their AI companions. So these are ten, 11 year
S2: old kids already. Their days are dominated by their relationships
S2: with these synthetic humans, these fake people, which are just
S2: digital constructs.
S1: Sorry, I'm pausing there because it's so terrifying. Peter. It's
S1: sort of it's every parent's nightmare, really, isn't it?
S2: Developed responsibly? These things could be completely useful in a
S2: in a limited, Discipline kind of way. But that's not
S2: happening because we are in a mad gold rush. The
S2: industry is in a mad gold rush. There are more
S2: than 100 companion AIS already available on the market. Most
S2: of them start ups. But now, with this big push
S2: for meta, which has 3 billion users, it wants to
S2: match every one of those three billions up with an
S2: AI companion for life and the the people in the
S2: industry talk about not wanting to sell a tool or
S2: a digital program. They want to sell a relationship. They
S2: want to they want to engage your soul with the
S2: artificial soul, as they describe with the soul of one
S2: of their, of one of their programs. This is the
S2: depth and permanence of the connection that they're seeking to promote.
S2: Because once you're locked in as a customer, they they
S2: own you.
S1: And is there an argument to be made that the
S1: return of Donald Trump to the white House might sort
S1: of even supercharge this growth, I guess, even further?
S2: It already has. Even before Donald Trump had formally been
S2: sworn in, Mark Zuckerberg, the chief, issued a statement publicly
S2: saying that the election result he explicitly referred to the
S2: election result has changed the balance in America in favor
S2: of free speech. Therefore, he said, are they disbanded their
S2: fact checking unit? They loosened a range of other restraints
S2: that they'd imposed in their social media and online businesses.
S2: So he's already responded to the new political atmosphere, and
S2: Donald Trump has declared the development of AI to be
S2: a national priority for the US as a technological leader.
S2: And with I mean, it's consistent with Trump's approach to everything,
S2: which is to remove as many regulations as possible and
S2: to keep it as as untrammeled as possible.
S1: So what about Australian kids, though? Because obviously we know
S1: the federal government, it's got its impending ban on people
S1: under 16 from getting access to social media sites. So
S1: are our kids going to be protected from this?
S2: Not under that law. That law applies to social media apps,
S2: but it doesn't apply to AI companions. So it would
S2: require more legislative action or an amendment to the to
S2: that law. So your kids will not be even if
S2: they even if that is if and when that is implemented.
S2: And it has passed both houses of Parliament, so it
S2: will be implemented and it has bipartisan support. But even
S2: when that's implemented and even if it's effective, your kids
S2: are still going to be able to get AI companions.
S2: So the Esafety office has anticipated this, um, there from June,
S2: requiring in in the Australian marketplace, which of course is
S2: all they can control if that, depending on the level
S2: of defiance and compliance from the big tech, which is,
S2: as we know, um, Highly imperfect Julie Inman Grant, the commissioner,
S2: has got some mandatory standards coming in for the use
S2: and supply of. It sounds like a drug, doesn't it? Yeah.
S2: And it is a kind of drug. It's a psychologically
S2: addictive phenomenon that will be applied to the industry from June.
S2: And the Esafety office has published educational guidelines. Really a
S2: warning for parents. This is what you should be looking
S2: out for. This is how you can deal with it.
S2: So the regulator here is doing what it can to
S2: anticipate and try to manage some of this, but ultimately
S2: it's going to come to individual kids and therefore their
S2: parents and families, if they want to keep this in
S2: the safe zone.
S1: And to be clear, if you do know, I'm assuming
S1: that Australian kids and teenagers and of course, grownups, they
S1: can just access the AI companions, I imagine, from overseas products. Right?
S1: Like there's no obstructions there, is there? Like. Yeah. Right. Okay.
S2: So there's no no legal or technological barrier whatsoever, which
S2: is why we've got primary school kids already investing half
S2: their day in in conversations interactions with AI companions.
S1: Okay, so Peter, just to wrap up, I guess I
S1: wanted to ask you about, I guess, what hope there
S1: might be. We know that there's a civil case against
S1: character AI, which is pending. That's the the company that
S1: created the companion, whom that 14 year old boy from
S1: Florida was in a relationship with. I guess I don't
S1: know how else to phrase it. So that's still coming.
S1: I mean, if we see that company actually held liable
S1: in some way, I mean, could that change, might we
S1: see a greater push from. I don't know, even our politicians,
S1: I guess, to bring in stronger legislation or from the
S1: companies to be held to account. Like what? What's the
S1: hope there?
S2: Well, if the civil suits like that one and another
S2: one against character AI by the way, uh, is a
S2: family who's suing them because they claim that their teenager, uh,
S2: had been chatting to an AI companion which suggested to
S2: the kid that, uh, that he murder his parents, or
S2: at least implied that he would be okay for him
S2: to murder his parents if they limited his screen time.
S2: So if the civil litigation cases like that can succeed,
S2: then that can be a force for, uh, an incentive
S2: for these companies to put tighter guidelines on their products. Um,
S2: there's always regulatory possibilities, but the technology moves so quickly,
S2: it's very difficult for regulators, legislators to catch up. Donald
S2: Trump is likely to veto anything that the Congress might
S2: want to come up with in the way of regulation,
S2: perhaps in the medium term. Samantha, the best solution will
S2: be using technology to control other technologies. I assume that
S2: we'll be seeing apps that allow parents to to to
S2: bring an AI into your kid's phone or other devices
S2: that will limit moderate the sorts of interactions and the
S2: way that that those, those things can operate. That's possibly
S2: the the most constructive hope that the technology industry can bring.
S1: I mean, I can't help but think it really does
S1: sort of bring back that vision of Blade Runner and,
S1: you know, the robot companions and also inevitably. Well, most
S1: people probably know how that movie ended. It's not good.
S1: And really, you've got the humans fighting the robots.
S2: Yes. Well, this is this is now humans fighting our
S2: own psyches to remind ourselves that we're dealing with an
S2: algorithm and a product here. There's not a person on
S2: the other on the other side of that, regardless of
S2: how compelling those algorithms have become. And just remember Mark
S2: Zuckerberg's reigning philosophy that he first started in 2012. The
S2: philosophy of the whole industry move fast and break things.
S2: We just have to try and make sure that the
S2: things that are broken are not our kids.
S1: Well, thank you so much, Peter, for your time.
S2: Pleasure, Samantha.
S1: Today's episode of The Morning Edition was produced by myself
S1: and Josh towers, with technical assistance by Taylor Dent. Our
S1: executive producer is Tammy Mills. Tom McKendrick is our head
S1: of audio. To listen to our episodes as soon as
S1: they drop, follow the Morning Edition on Apple, Spotify, or
S1: wherever you listen to podcasts. Our newsrooms are powered by subscriptions,
S1: so to support independent journalism, visit The Age or smh.com.au.
S1: Subscribe and to stay up to date, sign up to
S1: our Morningedition newsletter to receive a summary of the day's
S1: most important news in your inbox every morning. Links are
S1: in the show. Notes. I'm Samantha Selinger. Morris. Thanks for listening.