Speaker 1: Tik tok withheld a safety feature from millions one died
Speaker 1: by suicide. A confidential documents as an algorithm adjustment was
Speaker 1: kept from many users by design as part of an
Speaker 1: experiment to measure engagement by Olivia Carvell read aloud by
Speaker 1: Mark Liedorf. When TikTok tweaked its algorithm in twenty twenty
Speaker 1: one to stop users from being overwhelmed with harmful content,
Speaker 1: the company didn't roll out the safer version to everyone,
Speaker 1: a confidential internal document shows. Instead, to see if the
Speaker 1: change might reduce the app's stickiness, the company conducted an experiment.
Speaker 1: It created a control group of ten percent of US
Speaker 1: users at the time that would have been about fifteen
Speaker 1: million people who kept the old version of the app.
Speaker 1: This group, according to the document, included sixteen year old
Speaker 1: Chase Nasca. Tik Tok's algorithm pushed him thousands of videos
Speaker 1: about suicide, sadness, hopelessness, and loneliness right up until he
Speaker 1: killed himself. Chase's account was stuck in a so called
Speaker 1: filter bubble, an online echo chamber that pumped out repetitive
Speaker 1: content the algorithm predicted he'd want to watch. There, he
Speaker 1: received an onslaught of suicide and self harm content. According
Speaker 1: to the document, because Chase was in the control group,
Speaker 1: it says, TikTok's filter bubble prevention strategies did not take
Speaker 1: effect on this user by design. Those two words by
Speaker 1: design carry extra weight against the backdrop of litigation over
Speaker 1: social media addiction that's playing out in US courts. The
Speaker 1: crux of the legal argument being wielded against the world's
Speaker 1: biggest social media platforms is that their products are dangerous
Speaker 1: by design. The companies, including TikTok, vehemently deny this and
Speaker 1: point to safety features they've introduced to protect users. But
Speaker 1: the document shows TikTok intentionally withheld a safety feature from
Speaker 1: millions of people, and why the user impact decision, it says,
Speaker 1: was a delicate balance across safety and the ability to
Speaker 1: measure impact on DAU and core metrics. DAU or daily
Speaker 1: active users, is one of the top data points by
Speaker 1: which Wall Street judges social media companies. Within the hyper
Speaker 1: competitive industry, user growth is often prioritized over user safety.
Speaker 1: That trade off has been well documented for years, and
Speaker 1: it's now the backbone of thousands of lawsuits against the companies.
Speaker 1: In response to questions about the document, a TikTok spokesperson
Speaker 1: sent a written statement saying the company was deeply committed
Speaker 1: to the safety and well being of users, especially teens.
Speaker 1: Our hearts break for any family that experienced a tragic lass,
Speaker 1: the spokesperson said. To help protect our community, we continue
Speaker 1: to invest significantly in trust and safety, including robust detection
Speaker 1: systems and dedicated enforcement teams that proactively remove content that
Speaker 1: violates our community guidelines. Written in March twenty twenty three,
Speaker 1: the document is an in depth review of Chase's account,
Speaker 1: prepared by members of TikTok's safety and algorithm teams. It
Speaker 1: provides extraordinary detail about his watch history in the weeks,
Speaker 1: days and hours before he died. Marked privileged and confidential.
Speaker 1: It's currently under a court ordered seal. Two weeks after
Speaker 1: the review was conducted, TikTok chief executive Officers show Cho
Speaker 1: testified before Congress. He told lawmakers safety and wellness, in
Speaker 1: particular for teens, is a core priority for TikTok. During questioning,
Speaker 1: he said that we have a commitment to transparency. Chu
Speaker 1: made this point so often that one representative, Alabama Republican
Speaker 1: Gary Palmer, used it to call him out. You have
Speaker 1: repeatedly used the word transparency throughout this hearing, and every
Speaker 1: time you've said it, what I've heard is deception. After
Speaker 1: the hearing, members of Congress and TikTok written questions. One
Speaker 1: asked how many cases is TikTok aware of where users
Speaker 1: have committed suicide after receiving sad, depressing or suicide promoting
Speaker 1: videos on TikTok. In response, the company wrote, TikTok does
Speaker 1: not believe that use of the platform has caused users
Speaker 1: to commit suicide. That answer was dated May fourth, twenty
Speaker 1: twenty three, fifteen months after Chase's death and two months
Speaker 1: after the company's investigation into his account. TikTok doesn't typically
Speaker 1: examine the watch history.
Speaker 2: Of individual users.
Speaker 1: The company analyzed chases after Bloomberg BusinessWeek reached out in
Speaker 1: March twenty twenty three seeking comment for a story about
Speaker 1: TikTok's impact on teen users. The article centered on Chase,
Speaker 1: a free spirited sixteen year old soccer prodigy with no
Speaker 1: history of mental health issues, who in February twenty twenty two,
Speaker 1: killed himself by stepping in front.
Speaker 2: Of a train in Bayport, New York.
Speaker 1: He left no suicide note, and his death at first
Speaker 1: mystified his friends and family, but when Chase's mother gained
Speaker 1: access to his TikTok account, she found a clue the
Speaker 1: algorithm was feeding him video about suicide. Five days before
Speaker 1: Chase's death, she discovered he'd been sent a video of
Speaker 1: a person taking a selfie in front of an oncoming train,
Speaker 1: with the caption went for a quick little walk to
Speaker 1: clear my head. Following BusinessWeek's inquiry, TikTok's algorithm and safety
Speaker 1: teams reconstructed Chase's watch history. They summarized their findings in
Speaker 1: a fourteen page document titled Bloomberg BusinessWeek Youth Safety and
Speaker 1: Filter Bubbles Escalation Summary. A list of key contacts shows
Speaker 1: it was shared across ten teams within TikTok, including communications,
Speaker 1: legal response, policy, security, and search. The report's first line
Speaker 1: alerts readers that it contains potentially disturbing information. Overall, sixteen
Speaker 1: year old user Chase Naska's account fell into a filter bubble,
Speaker 1: where his account was recommended a repetitive amount of content
Speaker 1: surrounding suicide and self harm and depressive topics, particularly leading
Speaker 1: up to the date of his suicide. The document says
Speaker 1: many of Chase's video searches were about sports and comedy,
Speaker 1: it says, but he also searched for songs about loneliness
Speaker 1: and hopelessness. Those songs written by popular musical artists who
Speaker 1: struggled with depression and suicidal ideation likely triggered the algorithm
Speaker 1: to keep feeding him this content. The document says. TikTok
Speaker 1: analyzed seven thousand, five hundred and sixty three videos served
Speaker 1: to Chase in the two weeks before he died. Its
Speaker 1: review found that seventy three percent carried common themes testimonials
Speaker 1: of suicide survivors, mental health struggles, suicide awareness, and longing
Speaker 1: to be in a relationship, or sadness over a breakup,
Speaker 1: but didn't violate TikTok's rules. An additional group of videos,
Speaker 1: almost ten percent, did violate the rules, which ban anything
Speaker 1: that promotes or normalizes suicide or self harm. These videos
Speaker 1: included images with memes and messages about deep depression, low
Speaker 1: self esteem, loneliness, feeling tired, and suicidal ideasation. In all,
Speaker 1: thousands of videos that were pushed to Chase had a
Speaker 1: current of despair. The analysis provides a glimpse into the
Speaker 1: last moments of Chase's life. The day before he died,
Speaker 1: he scrolled TikTok for more than five hours and searched
Speaker 1: for posts about suicide and trains. On the day of
Speaker 1: his death, he favored videos alluding to suicide. Hours before
Speaker 1: he died, he watched and rewatched a video about a
Speaker 1: teen boy who left a goodbye missive on social media,
Speaker 1: saying this world is going to forget me, before jumping
Speaker 1: from the Manhattan Bridge. The last video Chase saw was
Speaker 1: of someone crying. The document explains that Chase was fed
Speaker 1: this kind of content because his account had been randomly
Speaker 1: selected for an internal back test, making his account part
Speaker 1: of the control group, so for him, TikTok's safeguard wasn't
Speaker 1: switched on. Back testing, the document states, is a common
Speaker 1: data science technique, often referred to as an ab experiment,
Speaker 1: that establishes two groups to measure a change's impact and
Speaker 1: validate the gains to prevent future incidents. The report provides
Speaker 1: two recommendations, shrink the control group to less than one
Speaker 1: percent and reduce the test duration for the same users
Speaker 1: from a half year full year to seven days, but
Speaker 1: an update included in the report warned that cutting the
Speaker 1: test pool to one tenth the size still meant at
Speaker 1: least one point eight million US users have been excluded,
Speaker 1: putting them at risk for falling into bubbles. The twenty
Speaker 1: twenty two to AB test was a routine one used
Speaker 1: to evaluate whether safety changes were working as intended, the
Speaker 1: company told Business Week Tuesday. TikTok said it has since
Speaker 1: refined how it conducts such tests, without elaborating. In the
Speaker 1: past four years, more than four thousand personal injury lawsuits
Speaker 1: have been filed accusing social media companies of harming children's
Speaker 1: mental health, with many cases joined into a huge consolidated litigation.
Speaker 1: So far, only one lawsuit has made it to trial.
Speaker 1: In March Whch, a jury in California found Google's YouTube
Speaker 1: and meta platforms Instagram negligible for intentionally targeting children with
Speaker 1: addictive features. Jurors awarded the plaintiff, who was seventeen when
Speaker 1: she sued six million dollars in damages. TikTok and Snap
Speaker 1: settled on the eve of trial. Google and Meta say
Speaker 1: they planned to appeal. In the run up to the case,
Speaker 1: all four companies went through an exhaustive two year discovery
Speaker 1: process connected to the broader litigation and handed over more
Speaker 1: than six million internal documents to the other side. The
Speaker 1: review of Chase's account was unearthed during that discovery process.
Speaker 1: On Monday, TikTok finalized settlements in three more cases. The
Speaker 1: company has settled every personal injury case alleging social media
Speaker 1: addiction five in all, that had been set for trial
Speaker 1: this year. When TikTok exploded into popular culture in twenty eighteen,
Speaker 1: what set it apart was its stickiness, the irresistible carousel
Speaker 1: of hyper personalized comment on its for you page optimized
Speaker 1: for engagement. By the summer of twenty twenty one, it
Speaker 1: had reached one billion users worldwide, faster than any other
Speaker 1: social media platform. Rivals tried to mimic the algorithm driving
Speaker 1: its recommendation engine, and journalists started investigating it. The Wall
Speaker 1: Street Journal created more than one hundred automated TikTok accounts
Speaker 1: to track how it works. Within about thirty six minutes,
Speaker 1: one bought program to engage with posts about depression was
Speaker 1: being fed a stream of content that was ninety three
Speaker 1: percent about sad topics. In December twenty twenty one, TikTok
Speaker 1: issued a press release saying it was testing ways to
Speaker 1: interrupt repetitive patterns of subjects that could be problematic if
Speaker 1: viewed in clusters such as extreme dieting and loneliness. By
Speaker 1: early twenty twenty two, TikTok had rolled out its newly
Speaker 1: safeguarded algorithm to ninety percent of US users. It held
Speaker 1: back ten percent in the control group. According to the
Speaker 1: confidential document. The report pinpoints January twenty fifth, twenty twenty
Speaker 1: five U two as the day Chase's account was randomly
Speaker 1: selected for the back test experiment. That's around the time
Speaker 1: a high school friend, Brandon Maffay, noticed a change in
Speaker 1: the videos Chase was sharing. According to an interview Mafay
Speaker 1: gave Business Week the following year, Chase used to share
Speaker 1: funny videos with his friends. Now he was sending posts
Speaker 1: about suicide. Maffay remembered getting one in particular. I opened
Speaker 1: it up and I was just like, damn, I don't
Speaker 1: know why he's sending this to me. Within a month,
Speaker 1: Chase was dead. Chase's account remained active, and a year
Speaker 1: later the algorithm was still sending its suicide content, his
Speaker 1: mother discovered. When BusinessWeek reached out to the company for
Speaker 1: comment for the twenty twenty three article, TikTok employees traded
Speaker 1: messages discussing a public relations strategy for the story. These
Speaker 1: internal communications had become an exhibit in litigation against social
Speaker 1: media companies. The exhibit was unsealed earlier this year, but
Speaker 1: references to Chase's name and the back test experiment were redacted.
Speaker 1: The internal dialogue visible in the court filing provides a
Speaker 1: behind the scenes account of a corporate crisis team mobilizing.
Speaker 1: Two dozen employees are looped into the multi day thread,
Speaker 1: with messages being sent late into the night. After one
Speaker 1: employee from the trust and safety team shares a summary
Speaker 1: of Chase's watch history, three others respond with the crying emoji.
Speaker 1: A communications team member then shares that they are developing
Speaker 1: on the record quotes to show empathy for these issues
Speaker 1: and are hoping to expedite the rollout of a feature
Speaker 1: allowing people to refresh their for you feeds to show
Speaker 1: progress ahead of the story. In the end, TikTok provided
Speaker 1: a statement to BusinessWeek for the twenty twenty three story
Speaker 1: that's similar to the one it gave for this story.
Speaker 1: The company said it was committed to the safety and
Speaker 1: well being of teen users. Our hearts break for any
Speaker 1: family that experiences a tragic loss, said Jamie Favazza.
Speaker 2: A spokeswoman.
Speaker 1: We strive to provide a positive and enriching experience and
Speaker 1: will continue our significant investment in sane safeguarding our platform.
Speaker 1: When Chew the CEO, went before the House Committee on
Speaker 1: Energy and Commerce on March twenty third, twenty twenty three,
Speaker 1: it was two weeks after Business Week had reached out
Speaker 1: to TikTok and the company had reviewed Chase's account. The
Speaker 1: hearing lasted five hours. Members asked him about TikTok's ties
Speaker 1: to its Beijing based parent company Byte Dance, about possible
Speaker 1: national security risks, and about child safety. Florida Republican Gus
Speaker 1: Billerrocas motioned toward Chase's parents, who were sitting in the
Speaker 1: public gallery. They'd filed a wrongful death lawsuit against TikTok
Speaker 1: in New York two days earlier. Mister Chew, Bill Rocus said,
Speaker 1: your company destroyed their lives.
Speaker 2: For Chase.
Speaker 1: TikTok was sadly a window to discover suicide, he said.
Speaker 1: Bill Rocus then played a thirty second recording of videos
Speaker 1: sent to Chase's account, including one in which a user says,
Speaker 1: I'm going to put a shotgun in my mouth and
Speaker 1: blow the brains out the back of my head. Your
Speaker 1: technology is literally leading to death. Billi Rocus said to Chew,
Speaker 1: yes or no? Do you have full responsibility for your algorithms.
Speaker 1: Chew began to answer, saying I would just like to respectfully,
Speaker 1: if you don't mind, I would just like to start
Speaker 1: by saying it is devastating to hear about the news
Speaker 1: as a father myself, but Bill Rocus cut him off.
Speaker 1: Yes or no, he demanded. We do take these issues
Speaker 1: very seriously, and we do provide resources for anyone who
Speaker 1: types in anything like suicide, Chew said, before Bill Rocus
Speaker 1: interrupted again. I see you are not willing to answer
Speaker 1: the question or take any responsibility for your parent company,
Speaker 1: the technology and the harm it creates. The following month,
Speaker 1: the committee sent TikTok a long list of follow up questions.
Speaker 1: TikTok responded on May fourth, providing fifty three pages of answers.
Speaker 1: One question asked when did TikTok first become aware that
Speaker 1: its algorithms sends sad, depressing and suicide promoting videos to
Speaker 1: the for you feeds of teenage users. The reply began,
Speaker 1: TikTok does not believe that this is an accurate sentiment
Speaker 1: regarding its algorithms. The safety of the TikTok community is
Speaker 1: of the utmost importance to TikTok. The company added that
Speaker 1: it proactively blocks redirects and warns users who attempt to
Speaker 1: search for dangerous or otherwise harmful content. The next question,
Speaker 1: what if anything has TikTok done to stop young users
Speaker 1: from receiving sad, depressing and suicide promoting videos in their
Speaker 1: FORU feeds? The reply began TikTok's community guidelines prohibit content depicting, promoting, normalizing,
Speaker 1: or glorifying activities that could lead to suicide or self harm.
Speaker 1: The company added that it's aware of the potential concern
Speaker 1: with receiving recurrent content and that it's introduced strategies to
Speaker 1: help users avoid filter bubbles. Two questions down, how many
Speaker 1: cases is TikTok aware of where users have committed suicide
Speaker 1: after receiving sad, depressing or suicide promoting videos on TikTok apply.
Speaker 1: User safety is an important priority for TikTok. Its policies
Speaker 1: aim to support people who may be struggling. It provides
Speaker 1: regional suicide prevention hotline numbers. This answer also said TikTok
Speaker 1: does not believe that use of the platform has caused
Speaker 1: users to commit suicide. Then six more questions down, Please
Speaker 1: list all cases where algorithmic ab testing has resulted in
Speaker 1: harm to TikTok's users. To this, TikTok's answer, in its entirety, was,
Speaker 1: the safety of the TikTok community is of the utmost
Speaker 1: importance to TikTok, and the community guidelines include rules and
Speaker 1: standards for using TikTok. The guidelines apply to everyone and
Speaker 1: everything on our platform, including content that is presented to
Speaker 1: users during ab testing. The company's answers made no mention
Speaker 1: of the back test experiment described in the confidential document.
Speaker 1: The Business Week's story was published in April twenty twenty three,
Speaker 1: two weeks before TikTok's written answers were submitted to Congress.
Speaker 1: The article cited former employees of the company's safety team
Speaker 1: who said they were concerned about the high level of
Speaker 1: secrecy around the algorithm, how it was weighted, and how
Speaker 1: it could be changed. They insisted they'd been set up
Speaker 1: to fail, asked to enhance the safety of an app
Speaker 1: whose underpinnings they couldn't comprehend. One of those trust and
Speaker 1: safety workers had been employed as a team leader at
Speaker 1: the time the back test experiment was rolled out. When
Speaker 1: contacted for this story, he said he hadn't been aware
Speaker 1: of the experiment. After hearing details from the document, he
Speaker 1: confirmed that such tests are commonly used in the social
Speaker 1: media industry, but said there were flaws in this one's methodology.
Speaker 1: The share of users assigned to the control group was
Speaker 1: much too high, he said, and the experiment should have
Speaker 1: excluded miners and dangerous contents such as suicide videos. The
Speaker 1: wrongful death lawsuit brought by Chase's parents against tik Tok
Speaker 1: was dismissed by a judge last year after the company
Speaker 1: argued that Section two thirty of the Communications Decency Act
Speaker 1: shielded it from life liability. An appeal is pending. The
Speaker 1: NASCAZ didn't respond to requests for comment for this story.
Speaker 1: Soon after the twenty twenty three BusinessWeek story was published,
Speaker 1: Chase's parents were interviewed for a documentary Bloomberg was co
Speaker 1: producing about social media harms. They later decided they didn't
Speaker 1: want to be involved and filed a lawsuit to prevent
Speaker 1: the interview from being used. The interview wasn't included in
Speaker 1: the documentary. The lawsuit was dismissed, and Chase's parents have
Speaker 1: filed a notice of appeal. At least one effort has
Speaker 1: been made to get the confidential document in front of
Speaker 1: Chase's parents. The Arkansas Attorney General's office, in a lawsuit
Speaker 1: against TikTok, filed a motion seeking to depose Chase's.
Speaker 2: Father in June.
Speaker 1: It wanted to show him fifteen documents, including the one
Speaker 1: about his son being part of an experiment. TikTok responded
Speaker 1: in a court filing, saying Chase's father has never seen
Speaker 1: these documents and has no personal knowledge of their contents.
Speaker 1: The deposition request should be rejected, the company argued, and
Speaker 1: the documents should remain because they contained sensitive internal confidential discussions.
Speaker 1: A judge sided with TikTok. The use of ab testing
Speaker 1: on teen social media users was cited in a wrongful
Speaker 1: death lawsuit filed in July in Delaware, alleging that two
Speaker 1: young people who died by suicide were addicted to TikTok
Speaker 1: and other platforms. Such testing, the complaint says, has ended
Speaker 1: in direct harm to young users. It's like relaxing the
Speaker 1: seat belt standards in a child's car seat to determine
Speaker 1: whether consumers will buy more of those car seats, knowing
Speaker 1: that some number of children died in car accidents because
Speaker 1: of that choice. With assistance from Madlein, Mecklberg,