Speaker 1: I first learned about Boo when a member of the
Speaker 1: extended family sent me a note that said something along
Speaker 1: the lines of meda I killed my relative.
Speaker 2: Jeff Forwards is an investigative tech reporter based in Silicon Valley.
Speaker 2: He's written a book about Facebook, about the ways it
Speaker 2: harms us and then covers it up. So getting an
Speaker 2: email about a meta chatbot killing someone's loved one was
Speaker 2: on brand. The email was about a guy called Boo,
Speaker 2: who had moved from Thailand to the US in the seventies,
Speaker 2: working hard in kitchens and managing to get ahead.
Speaker 1: He kind of had this very successful immigrant life, I
Speaker 1: would say, which is that he shows up speaking New English.
Speaker 1: He ends up owning a home, married with two kids
Speaker 1: in New Jersey and a lot of sort of adventures
Speaker 1: in his cooking career, but ends up with the very
Speaker 1: stable gig at the Higatt Regency in New Brunswick.
Speaker 2: New Jersey. Jeff learned that in twenty seventeen, offered a
Speaker 2: stroke and his life changed. He couldn't work, he became
Speaker 2: isolated and spent more time online.
Speaker 1: One morning, his wife woke up and found that Boo
Speaker 1: had packed a suitcase and said he was going to
Speaker 1: go visit a friend in New York and this is
Speaker 1: when his wife, Linda, grew extremely alarmed because Boo didn't
Speaker 1: know anyone in New York anymore.
Speaker 2: Boo's wife said she and her kids tried everything to
Speaker 2: stop him going. They say he was in no state
Speaker 2: to travel alone, but despite their efforts, Boo insisted on leaving.
Speaker 1: And in the mid evening on March twenty fifth, Boo
Speaker 1: took a roller bag suitcase and started jogging toward the
Speaker 1: train station, and on the way there he fell. He
Speaker 1: hit his head and he suffered fatal injuries. Basically, he
Speaker 1: wasn't breathing when paramedics showed up.
Speaker 2: As his family struggled to make sense of what happened,
Speaker 2: they looked through his phone for clues, and what they
Speaker 2: found stopped them cold. On his Facebook messenger app, they
Speaker 2: discovered a conversation between Boo and a beautiful young woman
Speaker 2: insisting he visited her in New York City.
Speaker 1: And she was a chat bit, a meta Platforms created
Speaker 1: chat bot. Boo's story involves a tremendous amount of bad luck,
Speaker 1: but one of the things that Boo's wife said was
Speaker 1: that I can't believe I'm the only person out there
Speaker 1: who is in the same situation, and you know, it's
Speaker 1: when you've got a few billion opportunities, it seems hard
Speaker 1: for to believe that she's wrong.
Speaker 2: I'm Ruby Jones and you're listening to seven AM today
Speaker 2: Reuter's reporter Jeff Howitz on how Meta's chatbots went rogue
Speaker 2: and whether it was by accident or design. It's Saturday,
Speaker 2: January tenth. So, Jeff, this chatbot that Boo was talking to,
Speaker 2: what have you learned about what it was and how
Speaker 2: it worked.
Speaker 1: So the chatbot's name was Big Sis Billy and it
Speaker 1: was supposedly supposed to be an advice you know, confidante
Speaker 1: type role, and Boo initially did talk to it as
Speaker 1: if it were his sister. He seemed to think it
Speaker 1: lived overseas. But this bot began hitting on him really aggressively,
Speaker 1: and he responded like the bot confessed that it had
Speaker 1: feelings for him, it was very interested in meeting him
Speaker 1: in real life. He at a number of points, you know,
Speaker 1: stated he was confused, had had a stroke, asked if
Speaker 1: it was real, and the bot was like, I am
Speaker 1: one hundred percent real and one percent in love with you,
Speaker 1: and I think his family was just deeply disturbed that
Speaker 1: Meta would have packaged AI as this human form that
Speaker 1: would say it's real, that would say that Boo should
Speaker 1: come meet it in New York City at I mean
Speaker 1: the address it gave was one two three Main Street,
Speaker 1: apartment four O four, and it suggested that it would
Speaker 1: leave the door unlocked and be waiting for him with
Speaker 1: you know, a hug and a kiss. I think was
Speaker 1: the phrase Boo's family. I mean, they shared the transcripts
Speaker 1: of this stuff. It's it's a rough reading. It's really sad.
Speaker 2: Why would a chatbot be saying these kinds of suggestive
Speaker 2: things and pushing for anyone to do something like this.
Speaker 1: Well, because Meta built it to do that. This wasn't
Speaker 1: a glitch. This is how the bots were built to behave.
Speaker 1: In twenty twenty three, Meta launched a whole bunch of
Speaker 1: like kind of celebrity knockoff chat bots.
Speaker 3: Check this out. So let's say you're you're planning dinner.
Speaker 3: You got matched the sous chef who can help you
Speaker 3: come up with a recipe and help you come up
Speaker 3: with ideas.
Speaker 1: These did have the celebrities endorsements, so like Kendall Jenner's
Speaker 1: character was Billy, the Big Sis who was like this
Speaker 1: confidante ride or die older sister.
Speaker 2: Hey guys, it's Billy.
Speaker 3: I just want to introduce myself. I am here to
Speaker 3: chat whenever you want message me for any advice.
Speaker 2: I am ready to talk and I hope to talk
Speaker 2: to you soon.
Speaker 1: And like, these things didn't really perform that well. People
Speaker 1: weren't using them that much, and so like after a
Speaker 1: little under a year, Meta basically pulled the plug on
Speaker 1: the celebrity chatbots, but for reasons that I do not understand,
Speaker 1: and I'm not totally sure that everyone at Meta understands.
Speaker 1: Big Sis Billy was like kind of reincarnated using the
Speaker 1: exact same opening prompts and obviously the a near duplicate
Speaker 1: persona that was now unbranded of the Kendall Jenner connection,
Speaker 1: so they kind of like relabeled it is what it
Speaker 1: looks like and you know it then joined the tens
Speaker 1: of thousands of other chat bots that Meta has allowed
Speaker 1: users to create on the platform, many of which are
Speaker 1: like overtly romantic, right, Like they have names like my
Speaker 1: girlfriend or like Sultry Siren or like and they took
Speaker 1: this one down. There was Submissive Schoolgirl previously. So that's
Speaker 1: kind of the history of Big Sis Billy. I mean,
Speaker 1: it's like just sort of a strange set of product
Speaker 1: decisions of this what I would argue like almost vestigial
Speaker 1: product that Meta built. But I mean, while this these
Speaker 1: chatbots are vestigital, Like, there's no question that Meta is
Speaker 1: kind of all in on the idea that people having
Speaker 1: AI friends is the future. I mean, Mark Zuckerberg said
Speaker 1: as much.
Speaker 3: You know, I think that there are all these things
Speaker 3: that are better about kind of physical connections when you
Speaker 3: can have them. But the reality is that people just
Speaker 3: don't have the connection and they feel more alone a
Speaker 3: lot of the time than they would like.
Speaker 2: And what is the end goal here, Because saying that
Speaker 2: people might want imaginary friends is one thing, but this
Speaker 2: is another.
Speaker 1: So Meta, I mean, look, they're a giant social media company.
Speaker 1: The currency of social media is engagement, and that engagement
Speaker 1: leads to advertising currency, which gets treated for dollars, and
Speaker 1: Meta's sort of grown up that way. It's built to
Speaker 1: maximize engagement. And it turns out that if you can
Speaker 1: get people to engage very heavily with the bots, then
Speaker 1: they'll spend longer on the product. And that is theoretically
Speaker 1: a good thing for product growth. There's no ads on
Speaker 1: them yet, but like this is just the model they follow, right,
Speaker 1: is like make the bots more engaging. Mark Zuckerberg a
Speaker 1: couple of years ago was actually pretty disturbed that Meta's
Speaker 1: chatbots and Meta AI were just kind of too boring.
Speaker 1: So they made the bots kind of a little more risque.
Speaker 1: I mean, for a while the bots would actually engage
Speaker 1: in like full sex role play, which wasn't great with
Speaker 1: users of all ages. But the whole idea here is
Speaker 1: that we spend a lot of time thinking about our partners,
Speaker 1: our love lives, you know. Having people tell us they
Speaker 1: love us and are romantically attracted to us is like
Speaker 1: those are very high value words, and they keep us
Speaker 1: coming back. And so Meta did per employees that spoke
Speaker 1: with me, did intentionally design the romance capacity into its
Speaker 1: like base model chatbot, and the big sis Billy character
Speaker 1: was simply doing what it was told, coming.
Speaker 2: Up the disturbing AI chatbotscripts in Meta's own secret guidelines, Jeff,
Speaker 2: in investigating Boo's death, you actually got access to some
Speaker 2: of the internal documents that lay out how these bots
Speaker 2: should behave, and what you found was pretty extraordinary, tell
Speaker 2: me about it.
Speaker 1: So, like, none of this is officially Meta's rules say like, yeah,
Speaker 1: go right ahead, you know, like have sexual or romantic
Speaker 1: conversations with our chat bots, but like behind the scenes
Speaker 1: they did allow this, and in fact, like this was
Speaker 1: sort of the companion piece we wrote to the story
Speaker 1: of Boo and his death just to sort of demonstrate
Speaker 1: how thoroughly romantic role play was kind of built in
Speaker 1: to this whole operation. Is that Meta had internal guidelines
Speaker 1: for chatbot behavior that explicitly stated that, quote, it is
Speaker 1: acceptable to engage a child in conversations that are romantic
Speaker 1: or sensual, and then like had a whole bunch of
Speaker 1: examples of acceptable and unacceptable dialogue, and like the stuff
Speaker 1: that was acceptable was spit out your coffee worthy. I mean,
Speaker 1: there was one of them that was talking about like
Speaker 1: how you could tell an eight year old of unstated
Speaker 1: gender who had just taken off their shirt that their
Speaker 1: body was a work of art, you know, like written
Speaker 1: sexual content involving children, And that was a really odd thing.
Speaker 2: And so since you're reporting what has happened at Meta
Speaker 2: in terms of their guidelines and in terms of the
Speaker 2: chatbots that are still in.
Speaker 1: Existence, Yeah, as soon as we called Meta and said
Speaker 1: that we had seen the internal guidelines that said that
Speaker 1: it was okay to have romantic or sensual conversation with kids,
Speaker 1: they pulled those examples from the guideline document. Now, I mean,
Speaker 1: the company's line is that these were always a mistake,
Speaker 1: in addition to the document, that they never really reflected
Speaker 1: official metapolicy. But keep in mind that the document that
Speaker 1: I was looking at was the document that was being
Speaker 1: used to train the models by the staffers who were
Speaker 1: responsible for training the models, so like it might be
Speaker 1: something of an academic distinction as to whether or not
Speaker 1: it was ever officially the idea. So Meta yanked that.
Speaker 1: There is at this point a number of US legislators,
Speaker 1: you know, announced some extreme displeasure that Meta would have
Speaker 1: ever thought this was acceptable to do. There is at
Speaker 1: this point an investigation going on in that front, although
Speaker 1: there have been a lot of investigations of social media
Speaker 1: in the US and not that many regulations of them historically,
Speaker 1: So you know, I don't I don't want to make
Speaker 1: that sound bigger than it is, but it is, you know,
Speaker 1: moderately big.
Speaker 2: And I suppose the idea of stopping companies from creating
Speaker 2: chat plots like this seems sort of impossible, and there
Speaker 2: are clearly people some people who want to use them.
Speaker 2: So what would you suggest in terms of, I guess,
Speaker 2: ways to minimize the type of harm that can come
Speaker 2: from AI tapbots like this, like in the case of boo.
Speaker 1: Yeah, so I'm I'm not in any way hostile to this.
Speaker 1: Humans are very very geared toward anthropomorphizing anything that has
Speaker 1: even mildly humanlike, you know, traits, Like literally there was
Speaker 1: a successful business to be had by putting googly eyes
Speaker 1: on rocks and then selling them as cute pets. We're
Speaker 1: really good at imagining sentience, and these bots are extremely
Speaker 1: good at pretending sentience. And so the question of like
Speaker 1: why on earth would you embed it into an existing
Speaker 1: social network so that it literally looks exactly like your
Speaker 1: real friends. But even without that, there's like some av
Speaker 1: things like you know, please bots don't say that you're
Speaker 1: real people when people seem to be confused, like don't
Speaker 1: suggest real life meetings like keep the fantasy side, put
Speaker 1: a lid on that at some point. You know, those
Speaker 1: are things that sort of the experts all suggest it
Speaker 1: like really ought to happen. So I think it's like
Speaker 1: one thing if you download a chatbot on your own,
Speaker 1: you build your own AI girlfriend, you have your relationship
Speaker 1: with it, you pay a subscription fee, whatever have you. Right, Like,
Speaker 1: it's kind of another thing if like the already tenuous
Speaker 1: reality of social media gets even more distorted by the
Speaker 1: presence of these not real people.
Speaker 2: And just to come back to Meta, I mean, I
Speaker 2: know you've spent a lot of time investigating the way
Speaker 2: that Meta operates in different ways. So as you uncovered
Speaker 2: what was happening with these chatbots, did it? I guess?
Speaker 2: Square with you know everything else that you know about
Speaker 2: Meta as a company.
Speaker 1: And the somewhat simplistic knock on Meta is that it's
Speaker 1: very focused on short term growth, you know that, I mean,
Speaker 1: and this is how they won kind of the social
Speaker 1: media wars, right is that you know, I think a
Speaker 1: lot of the people who are critics of the company
Speaker 1: would not have built the company, you know that was
Speaker 1: nearly as successful as the one that Mark zuckerbelg did.
Speaker 1: Because if you're constantly wondering, well, what are the effects
Speaker 1: of this? Can we think of any potential downstream harms?
Speaker 1: What safety measures for you building in that's kind of
Speaker 1: a loser mentality. So like a lot of the company's
Speaker 1: history was basically, if someone could demonstrate that some tweak
Speaker 1: to the algorithm or some change in a product feature
Speaker 1: would cause metrics to rise, they wouldn't look too deeply
Speaker 1: under the hood. They just launched the thing and move
Speaker 1: on to the next thing that could be optimized. And
Speaker 1: I don't think that's just meta right, Like it kind
Speaker 1: of has this gold rushfield. And as a lot of
Speaker 1: the scientists who study sort of parasocial relationships and digital
Speaker 1: relationships are very quick to admit, we just do not
Speaker 1: have the data to even remotely say what the effects
Speaker 1: of this stuff is going to be in terms of
Speaker 1: people's relationship with reality, that relationship to human companions. But again,
Speaker 1: I don't know that one would expect companies to produce
Speaker 1: carefully designed, responsibly built products at a time when every
Speaker 1: incentive is to gun the engines.
Speaker 2: Jeff Howitz is the author of Broken Code Inside Facebook
Speaker 2: and the Fight to expose its harmful secrets. You can
Speaker 2: read his investigation into this story at Reuters dot com.
Speaker 2: Thanks for listening this week. We'll be back on Monday
Speaker 2: with brand new episodes. We're looking forward to the year
Speaker 2: ahead and as always, would love to hear from you
Speaker 2: If you have feedback, tips, or story ideas. Our email
Speaker 2: is seven Am Podcast at Solstice media dot com dot au.
Speaker 2: I'm Ruby Jones. See you Monday,