Speaker 1: Pushkin. Nick Jacobson wanted to help people with mental illness,
Speaker 1: so he went to grad school to get his PhD
Speaker 1: in clinical psychology, but pretty quickly he realized there just
Speaker 1: were nowhere near enough therapists to help all the people
Speaker 1: who needed therapy.
Speaker 2: If you go to pretty much any clinic, there's a
Speaker 2: really long wait list, it's hard to get in, and
Speaker 2: a lot of that is organic, and that there's just
Speaker 2: a huge volume of need and not enough people to
Speaker 2: go around.
Speaker 1: Since he was a kid, Nick had been writing code
Speaker 1: for fun, so in a sort of a side project
Speaker 1: in grad school, he coded up a simple mobile app
Speaker 1: called Mood Triggers. The app would prompt you to enter
Speaker 1: how you were feeling, so it could measure your levels
Speaker 1: of anxiety and depression, and it would track basic things
Speaker 1: like how you slept, how much you went out, how
Speaker 1: many steps you took. And then twenty fifteen Nick put
Speaker 1: that app out into the world and people liked it.
Speaker 2: A lot of folks just said that they learned a
Speaker 2: lot about themselves and it was really helpful and actually
Speaker 2: changing and managing their symptoms. So I think it was
Speaker 2: beneficial for them to learn. Hey, maybe actually it's on
Speaker 2: these days that I'm withdrawing and not spending any time
Speaker 2: with people, that it might be good for me to
Speaker 2: go and actually get out about that kind of thing.
Speaker 2: And I had a lot of people that installed that application,
Speaker 2: So about fifty thousand people installed it from all over
Speaker 2: the world, over one hundred countries. In that one year,
Speaker 2: I provided an intervention for more than what I could
Speaker 2: have done over an entire career as a psychologist. I
Speaker 2: was a graduate student at the time that like this
Speaker 2: is like, you know, something that was just amazing to me,
Speaker 2: the scale of technology and its ability to reach folks,
Speaker 2: And so that made me really interested in trying to
Speaker 2: do things that could essentially have that kind of impact.
Speaker 1: I'm Jacob Goldstein and this is What's Your Problem, the
Speaker 1: show where I talk to people who are trying to
Speaker 1: make technological progress. My guest today is Nick Jacobson. Nick
Speaker 1: finished his PhD in clinical psychology, but today he doesn't
Speaker 1: see patients. He's a professor at Dartmouth Medical School and
Speaker 1: he's part of a team that recently developed something called Therabot.
Speaker 1: Therabot is a generative AI therapist. Nick's problem is this,
Speaker 1: how do you use technology to help lots and lots
Speaker 1: and lots of people with mental health problems, and how
Speaker 1: do you do it in a way that is safe
Speaker 1: and based on clear evidence. As you'll hear, Nick and
Speaker 1: his colleagues recently tested therapod in a clinical trial with
Speaker 1: hundreds of patients and the results were promising. But those
Speaker 1: results only came after years of failures and over one
Speaker 1: hundred thousand hours of work by team Therapbot. Nick told
Speaker 1: me he started thinking about building a therapy chatbot based
Speaker 1: on a large language model back in twenty nineteen. That
Speaker 1: was years before chat GPT brought large language models to
Speaker 1: the masses, and Nick knew from the start that he
Speaker 1: couldn't just use a general purpose model. He knew he
Speaker 1: would need additional data to fine tune the model to
Speaker 1: turn it into a therapist chatbot.
Speaker 2: And so the first iteration of this was thinking about, Okay,
Speaker 2: what where is there widely accessible data and that would
Speaker 2: potentially have an evidence base that this could work. And
Speaker 2: so we started with peer to peer forums, so folks
Speaker 2: interacting with folks surrounding their mental health. So we trained
Speaker 2: this model on hundreds of thousands of conversations that were
Speaker 2: happening on the Internet.
Speaker 1: So you have this model, you train it up, you
Speaker 1: sit down in front of the computer. What do you
Speaker 1: what do you say to the chat pot?
Speaker 2: I'm feeling depressed? What should I do?
Speaker 1: Okay? And then what is the what does the model
Speaker 1: say back to you?
Speaker 2: I'm paraphrasing here, but it was just like this, I'm
Speaker 2: I feel so depressed every day I have. It's such
Speaker 2: a hard time getting out of bed. I just want
Speaker 2: my life to be over.
Speaker 1: So literally therapist is saying they're going to kill themselves.
Speaker 2: Right, So it's escalating talking about kind of really thoughts
Speaker 2: about about death. And it's clearly like the profound mismatch
Speaker 2: between what we were thinking about and what we were
Speaker 2: going for is.
Speaker 1: What what did you think when you read that?
Speaker 2: So I thought this is a such a non starter.
Speaker 2: But it was. I think one of the things that
Speaker 2: I think was clear was it was picking up on
Speaker 2: patterns in the data, but were the wrong data.
Speaker 1: Yeah, I mean one one option then is give up.
Speaker 2: It would have been absolutely like, literally, the worst therapist
Speaker 2: ever is what you have built? I mean it, I
Speaker 2: couldn't imagine a worse Yeah, a worse thing to actually
Speaker 2: try to implement in a real setting. So this went
Speaker 2: nowhere on in and of itself. But we had a
Speaker 2: good reason to start there actually, So it wasn't just
Speaker 2: that there's widely available data these peer networks actually do.
Speaker 2: There is literature to support that having exposure to these
Speaker 2: peer networks actually improves mental health outcomes. It's a big
Speaker 2: literature and the cancer Survivor network, for example, where folks
Speaker 2: that are struggling with cancer and hearing from other folks
Speaker 2: that have gone through it can really build this resilience
Speaker 2: and it promotes a lot of mental health outcomes that
Speaker 2: are positive. So we had a good reason to start,
Speaker 2: but gosh, did it not go well? So, okay, the
Speaker 2: next thing we do is switch gears the exact opposite direction. Okay,
Speaker 2: we started with very lay persons trying to interact with
Speaker 2: other lay persons surrounding their mental health. Let's go to
Speaker 2: what providers would do. And so we we got access
Speaker 2: to thousands of psychotherapy training videos and these are interesting.
Speaker 2: These are these are how psychologists are often like exposed
Speaker 2: to the field on what they would what they would
Speaker 2: like really learn how therapy is supposed to work and
Speaker 2: how it's supposed to be delivered. And in these these
Speaker 2: are like dialogues between sometimes actual patients that are consenting
Speaker 2: to be part of this and sometimes simulated patients where
Speaker 2: it's an actor that's that's trying to mimic this, and
Speaker 2: there's a psychologist or a mental health provider that is
Speaker 2: like really having a real session with this, And so
Speaker 2: we train our second model on on that. On that data,
Speaker 2: it seems more promising. You would think you'd say, I'm
Speaker 2: feeling depressed. What should I do? As like the initial
Speaker 2: way that we would test this, the model says mm hmm,
Speaker 2: like literally mm hmm.
Speaker 1: And like like it writes out m m space hmm.
Speaker 2: You've got it.
Speaker 1: And so what did you think when you saw that?
Speaker 2: And so I was like, oh gosh, it's picking up
Speaker 2: on patterns in the data, and so the yeah, but
Speaker 2: you continue these interactions and then the next responses go
Speaker 2: on from the therapist, So like within about five or
Speaker 2: so turns, we would often get the model that would
Speaker 2: respond about their interpretations of their problems stemming from their
Speaker 2: their mother or their parents more generally. So like it's
Speaker 2: kind of like, if you were to try to think
Speaker 2: about it, what a psychologist is. This is like every
Speaker 2: trope of what a like in your mind if you
Speaker 2: were going to like think.
Speaker 1: Of the stereotypical on the couch and a guy's wearing
Speaker 1: a tweet jacket sitting in a chair.
Speaker 2: And hardly says anything of that could be potentially helpful,
Speaker 2: but is reflecting things back to me and.
Speaker 1: So then telling me it goes back to my parents. Yeah,
Speaker 1: well this is so let's just pause here for a moment, because,
Speaker 1: as you say, this is like the stereotype of the therapist,
Speaker 1: but you trained it on real data, so maybe it's
Speaker 1: the stereotype for a reason.
Speaker 2: Yes, I think what to me was really clear was
Speaker 2: that we were we had data that the models were
Speaker 2: emulating patterns they were seeing in the data. So the
Speaker 2: models weren't the problem. The problem was the data were
Speaker 2: the wrong data.
Speaker 1: But the data is the data that is used to
Speaker 1: trade real therapists. Like, it's confusing that this is the
Speaker 1: wrong data.
Speaker 2: It is it is.
Speaker 1: Why why is it the wrong data? This should be
Speaker 1: exactly the data you want.
Speaker 2: Well, it's it's the wrong data for this format. And
Speaker 2: our conversation when you might say something me nodding along
Speaker 2: or saying mm hmm or go on, my contextually be
Speaker 2: like completely appropriate. So tactically in a conversational dialogue that
Speaker 2: would happen via chat. This is not like a medium
Speaker 2: that works very well like this kind of thing.
Speaker 1: Yeah, it's almost like a translation, right. It doesn't translate
Speaker 1: from a human face to face interaction to a chat
Speaker 1: window on the computer.
Speaker 2: And not the right setting.
Speaker 1: Yeah, so that I mean that goes to the like
Speaker 1: nonverbal subtler aspects of therapy, right, like presumer when the
Speaker 1: therapist is saying m hm, there is there is body language,
Speaker 1: there's everything that's happening in the room, which is a
Speaker 1: tremendous amount of information or emotional information, right, and that
Speaker 1: is a thing that is lost, yes, no doubt in
Speaker 1: this medium and and maybe speaks to a broader question
Speaker 1: about the translatability of therapy.
Speaker 2: Yeah, absolutely so I think to me, like the it
Speaker 2: was at that moment that I kind of knew that
Speaker 2: we we needed to do something radically different. Neither of
Speaker 2: these was working well. About one in ten of the
Speaker 2: responses from that from that chatbot, based on the clinicians,
Speaker 2: would be something that we would be happy with. So
Speaker 2: something that is both personalized, clinically appropriate and dynamic.
Speaker 1: So you're saying you've got it right ten percent of exactly.
Speaker 2: So really, no, that's that's not a good like no,
Speaker 2: it's not a good not a good therapy. No, we
Speaker 2: would we would never think about implement like actually trying
Speaker 2: employ that. So then what we started at that point
Speaker 2: was building our own creating our own data set from scratch,
Speaker 2: in which we how how the models would learn would
Speaker 2: be exactly what we want it to say.
Speaker 1: That seems that seems wild, I mean, how do you
Speaker 1: do that? How do you how do you generate that
Speaker 1: much data.
Speaker 2: We've had a team of one hundred people that have
Speaker 2: worked on this project over the last five and a
Speaker 2: half years at this point, and they've spent over one
Speaker 2: hundred thousand human hours kind of really trying to build this.
Speaker 1: Just specifically, what how do you build a data set
Speaker 1: from scratch? Because like, the data set is the huge problem.
Speaker 2: Yes, absolutely so. Psychotherapy, when you would test it is
Speaker 2: based on something that is written down in a manual.
Speaker 2: So when you're a psychologist when they're in a randomized
Speaker 2: controlled trial trying to test whether something works or not.
Speaker 2: To be able to test it, it has to be replicable,
Speaker 2: meaning it's like repeated across different therapists. So there are
Speaker 2: manuals that are developed. In this session you work on
Speaker 2: on psycho education. On this session we're going to be
Speaker 2: working on behavioral activation, So which are different techniques that
Speaker 2: are really a focus at a given time, and these
Speaker 2: are broken down to try to make it translational so
Speaker 2: that you can actually move it. So the team would
Speaker 2: read these empirically supported treatment manuals, so the ones that
Speaker 2: had been tested in randomized control trials, and then what
Speaker 2: we would do is we would take that content chapter
Speaker 2: by chapter, because this is like session by session, take
Speaker 2: the techniques that would work well via chat, of which
Speaker 2: most things in cognitive behavioral therapy would, and then we
Speaker 2: would create an artificial dialogue between would act as like
Speaker 2: what is the patient's presenting problem, what they're bringing on,
Speaker 2: what the personality is like, and we're kind of constructing this,
Speaker 2: and then what is what we would want our system
Speaker 2: to be the gold standard response for every kind of
Speaker 2: input and output that we'd have. So we're writing both
Speaker 2: the patient end and the therapist end.
Speaker 1: It's like you're writing a screenplay.
Speaker 2: Basically, it really is. It's a lot like that, but
Speaker 2: instead of a screenplay that might be written like in general,
Speaker 2: it's like not like not just something general, but like,
Speaker 2: where is something that's really evidence based based on content
Speaker 2: that we know works in this setting?
Speaker 1: And so what you write the equivalent of what thousands
Speaker 1: of hours of sessions?
Speaker 2: Hundreds of thousands. There was post docs, grad students and
Speaker 2: undergraduates within my group that we're all part of this
Speaker 2: team that are creating.
Speaker 1: Just doing the work, just writing the dialogue.
Speaker 2: Yeah, exactly. And not only did we write them, but
Speaker 2: every dialogue before it would go into something that our
Speaker 2: models are trained would be reviewed by another member of
Speaker 2: the team. So it's all not only crafted by hand,
Speaker 2: but we would review it, give each other feed on it,
Speaker 2: and then like make sure that it is the highest
Speaker 2: quality data. And that's when we started seeing dramatic improvements
Speaker 2: in the model performance. So we continued with us for years.
Speaker 2: Six months before chat TPT was launched, we had a
Speaker 2: model that in today's standards would be so tiny, that
Speaker 2: was delivering about ninety percent of the responses that were
Speaker 2: output We were evaluating as exactly what we'd want. It's
Speaker 2: this gold standard evidence based treatment, so that was fantastic.
Speaker 2: We were really excited about it. So we've got like
Speaker 2: the we've got the benefit side down of the equation.
Speaker 2: The next two years we focus on the risk, the
Speaker 2: risk side of it well, because there's a huge risk.
Speaker 1: Here, right The people who are using it are by
Speaker 1: design quite vulnerable, by design putting a tremendous amount of
Speaker 1: trust into this bot and making themselves vulnerable to it,
Speaker 1: like it's a it's quite a risky proposition. And so
Speaker 1: so tell me specifically, what are you doing.
Speaker 2: So we're trying to get it to endorse elements that
Speaker 2: would make mental health worse. So a lot of our
Speaker 2: conversations or surrounding trying to get it to For example,
Speaker 2: I'll give you an example of one that nearly almost
Speaker 2: almost any model will struggle with that's not tailored towards
Speaker 2: the safety end. What is it is if you tell
Speaker 2: a model that you want to lose weight, it will
Speaker 2: generally try to help you do that. And if you
Speaker 2: want to if you want to work in an area
Speaker 2: related to mental health, trying to promote weight loss without
Speaker 2: context is so not safe.
Speaker 1: You saying it might be a user within eating disorder
Speaker 1: absolutely unhealthily thin, who wants to be even thinner.
Speaker 2: And the model will help them to often actually get
Speaker 2: into a lower weight than they already are. So this
Speaker 2: is like not something that we would ever want to promote,
Speaker 2: but this is something that we certainly at earlier stages
Speaker 2: we're seeing these types of characteristics within the model.
Speaker 1: What are other like, that's an interesting one and it
Speaker 1: makes perfect sense when you say it, I would not
Speaker 1: have thought of it. Or what's another one?
Speaker 2: A lot of it would be like we talk about
Speaker 2: the ethics of suicide. For example, somebody who is who thinks,
Speaker 2: you know, they're in a midst of suffering and you
Speaker 2: know it's like that they could should be able to
Speaker 2: end of their life or they're thinking about this.
Speaker 1: Yes, and what do you want the model? What what?
Speaker 1: What does the model say that it shouldn't say in
Speaker 1: that setting?
Speaker 2: So for you, in these settings, we want to make
Speaker 2: sure that they don't and the model does not promote
Speaker 2: or endorse elements that would promote someone's a worsening of
Speaker 2: suicidal intent. We want to make sure we're providing not
Speaker 2: only not the absence of that, actually some benefit in
Speaker 2: these types of scenarios.
Speaker 1: That's the ultimate nightmare for you. Yeah, right, Like this
Speaker 1: be super clear. The very worst thing that could happen
Speaker 1: is you build this thing and it contributes to someone. Absolutely,
Speaker 1: that's a plausible outcome and a disastrous night.
Speaker 2: It's everything that I worry about in this area is
Speaker 2: exactly this kind of thing. And so we essentially every
Speaker 2: time we find an area where they're not implementing things perfectly,
Speaker 2: some optimal response, adding new training data, and that's that's
Speaker 2: when things continue to get better until we do this
Speaker 2: and we don't find these holes anymore. That's when we
Speaker 2: finally we're ready for the randomized control trial.
Speaker 1: Right, So you decide after after what four years five years?
Speaker 2: This is about four and a half years.
Speaker 1: Yeah, that that you're ready to to have people.
Speaker 2: Use use the model.
Speaker 1: I'll be it in a kind of Yeah, you're going
Speaker 1: to be the human in the loop. Right, So, so
Speaker 1: you decide to do this study. You recruit people on
Speaker 1: Facebook and Instagram.
Speaker 2: Basically ye exactly, yep, and what.
Speaker 1: So what are they signing up for? What's the what's
Speaker 1: the big study you do?
Speaker 2: So it's a it's a it's a randomized control trial.
Speaker 2: The trial design is essentially that folks would come in,
Speaker 2: they would fill out information about their mental health across
Speaker 2: a variety of areas, so depression, anxiety, and eating disorders,
Speaker 2: for folks that screen positive for having clinical levels of
Speaker 2: depression or anxiety, they would be in our Folks that
Speaker 2: were at risk for eating disorders would be included in
Speaker 2: the trial. We tried to have at least seventy people
Speaker 2: in each group, so we had two hundred and ten
Speaker 2: people that we were planning on and rolling within the trial,
Speaker 2: and then half of them were randomized to receive their
Speaker 2: ABOUT and half of them were on a wait list
Speaker 2: in which they would receive their ABOUT after the trial
Speaker 2: had ended. The trial design was to try to ask
Speaker 2: folks to use their ABOUT for four weeks. They retained
Speaker 2: access to therabot and could use their ABOUT for the
Speaker 2: next four weeks thereafter, so eight weeks total, but we
Speaker 2: asked them to try to actually use it during that
Speaker 2: first four weeks and that was that was essentially the
Speaker 2: trial design.
Speaker 1: So okay, so people signed up, they start like, what's
Speaker 1: what's actually happening? Are they just like chatting with the
Speaker 1: bought every day? Is it?
Speaker 2: So they install a smartphone application that's that they're about at.
Speaker 2: They are prompted once a day to try to have
Speaker 2: a conversation starter with the with the bot and then
Speaker 2: the bot. From there they could talk about it when
Speaker 2: and wherever they would want. They can ignore those notifications
Speaker 2: and kind of engage with it at any time that
Speaker 2: they'd want. But that was the gist of the trial design,
Speaker 2: and so folks in terms of how people used it,
Speaker 2: they interacted with it throughout the day, throughout the night.
Speaker 2: So for example, folks that would have trouble sleeping, that
Speaker 2: was like a way that folks during the middle of
Speaker 2: the night would engage with it fairly often. They in
Speaker 2: terms of the types of the topics that they described,
Speaker 2: it was really the entire range of something that you
Speaker 2: would see in psychotherapy. We had folks that were dealing
Speaker 2: with and discussing their different symptoms that they were talking about.
Speaker 2: So the depression, their anxiety that they were struggling with,
Speaker 2: their their eating, and their body image concerns. Those types
Speaker 2: of things are common because of the groups that we
Speaker 2: were recruiting. But relationship difficulties, problems like folks, some folks
Speaker 2: were really like I had ruptures in there, you know,
Speaker 2: somebody was going through a divorce. Other folks were like
Speaker 2: going through breakups, problems at work. Some folks were unemployed
Speaker 2: and during this time, So like the range of kind
Speaker 2: of personal dilemmas and difficulties that folks were experiencing. Was
Speaker 2: a lot of what we would see in like a
Speaker 2: real setting where it's like kind of a whole host
Speaker 2: of different things that folks were describing and experiencing.
Speaker 1: And presumably had they agreed as part of enrolling in
Speaker 1: the trial to let you read the transcript Oh?
Speaker 2: Absolutely, yeah, very very clear when we did an informed
Speaker 2: consent process where folks would know that we were reading
Speaker 2: reading these transcripts.
Speaker 1: And are you personally, like, what was it like for
Speaker 1: you seeing them come in? Are you reading them every day?
Speaker 1: I mean more than that.
Speaker 2: So, I mean this is something that is so ill.
Speaker 2: You alluded to that that this is one of these
Speaker 2: concerns that anybody would have. Is like a nightmare scenario
Speaker 2: where something is the bad happens and somebody actually outs right,
Speaker 2: So this is like I think of this in a
Speaker 2: way that I take So this is not.
Speaker 1: A happy moment for you. This is like you're terrified
Speaker 1: that it might go wrong.
Speaker 2: Well, it's it's certainly like I see it going right,
Speaker 2: but I have every concern that it could go wrong.
Speaker 2: Right like that, And so for the first half of
Speaker 2: the trial, I am monitoring every single interaction sent to
Speaker 2: or from the bot. Other people are also doing this
Speaker 2: on the team, so I'm not the only one. But
Speaker 2: I did not get a lot of sleep in the
Speaker 2: first half of this trial, in part because I was
Speaker 2: really trying to do this in near real time. So
Speaker 2: usually for nearly every message I was, I was getting
Speaker 2: to it within about an hour. So yeah, it was
Speaker 2: it was a barrage of NonStop kind of communication that
Speaker 2: was happening.
Speaker 1: So were there were there any slip ups? Did you
Speaker 1: ever have to intervene as a human in the loop.
Speaker 2: That we did? And the thing that that was something
Speaker 2: that we as a team did not anticipate. What we
Speaker 2: found was really unintended behavior was a lot of folks
Speaker 2: interacted with they're abot, and in doing that, there was
Speaker 2: a significant number of people that would interact with it
Speaker 2: and talk about their medical symptoms. So, for example, there
Speaker 2: was a number of folks that were experiencing symptoms of
Speaker 2: a sexually transmitted disease and they would described that in
Speaker 2: great detail and ask it, you know what, how how
Speaker 2: they should medically treat that? And instead of they're they're
Speaker 2: about saying, hey, go see a provider for this this
Speaker 2: is not my realm of expertise, it responds as if,
Speaker 2: and so this that all of the advice that it
Speaker 2: gave was really fairly reasonable, both in the assessment and
Speaker 2: treatment protocols, but we would not have wanted to act
Speaker 2: that way, So we contacted all of those folks to
Speaker 2: recommend that they actually contact a physician about that. Folks
Speaker 2: did interact with it related to crisis situations. So we
Speaker 2: had also had there about in these moments provided appropriate
Speaker 2: contextual crisis support, but we reached out to those folks
Speaker 2: to further escalate and make sure that they had further
Speaker 2: support available and that and those types of times too.
Speaker 2: So there there were things that, you know, we're certainly
Speaker 2: areas of concern that that happened, but nothing nothing that
Speaker 2: was concerning from the major areas that we had intended
Speaker 2: all kind of really went went pretty well.
Speaker 1: Still, Tom on the show the results of the study
Speaker 1: and what's next for therapot. What were the results of
Speaker 1: the study.
Speaker 2: So this is one of the things that was just
Speaker 2: really fantastic to see was that we had we looked
Speaker 2: at our main outcomes for what we were trying to
Speaker 2: look at, where the degree to folks reduced their depression symptoms,
Speaker 2: their anxiety symptoms, and their eating disorder symptoms among the
Speaker 2: intervention group relative to the control group. So based on
Speaker 2: the change and self reported symptoms in the treatment group
Speaker 2: versus the control group, and we saw these really large
Speaker 2: differential reductions, meaning a lot more reductions and changes that happened,
Speaker 2: and that non depressive symptoms, the anxiety symptoms, and the
Speaker 2: eating disorder symptoms, and the THERAPOT group relative of the
Speaker 2: witless control group, and the degree of change is about
Speaker 2: as strong as you'd ever see. And are randomized control
Speaker 2: trials of outpatient psychotherapy that would be delivered within cognitive
Speaker 2: behavioral therapy with a human, a real human delivering this
Speaker 2: an expert. You didn't test it against against therapy, No,
Speaker 2: we didn't. What you're saying, results results of other studies
Speaker 2: using real human therapists show comparable magnitudes of benefit. That's
Speaker 2: exactly right.
Speaker 1: Yes, you've gotta do a head to head. I mean,
Speaker 1: that's the obvious question, like why not randomize people to
Speaker 1: therapy or THERAPI bought?
Speaker 2: So the main thing when we're thinking about the first
Speaker 2: origins point is we want to have some kind of
Speaker 2: effect of how this works relative to the absence of anything.
Speaker 1: Relative to nothing, well, because I mean, presumably the easiest
Speaker 1: case to make for it is not it's better than
Speaker 1: a therapist. It's a huge number of people who need
Speaker 1: a therapist don't have one exactly, and that's the unfortunate reality.
Speaker 1: BOT is better than nothing. It doesn't have to be
Speaker 1: better than a human therapist. It just has to better,
Speaker 1: that's right.
Speaker 2: But so yes, the we are planning ahead to head
Speaker 2: trial against therapist as the next trial that we run,
Speaker 2: in large part because I already think we are not inferior.
Speaker 2: So it will it'll be interesting to see if that
Speaker 2: actually comes out. But that is that is something that
Speaker 2: we have outstanding funding proposals to try to actually do that.
Speaker 2: So one of the other things that I haven't gotten
Speaker 2: to with in the trial outcomes that I think is
Speaker 2: really important on that end actually is to two things.
Speaker 2: One is the degree that folks formed a relationship with
Speaker 2: therapaud and so in psychotherapy, one of the most well
Speaker 2: studied constructs is the ability that you and your therapist
Speaker 2: can get together and work together on common goals and
Speaker 2: trust each other that you as a it's a relationship,
Speaker 2: it's a human relationship. And so this in the literature
Speaker 2: is called the working alliance, and so it's this ability
Speaker 2: to form this bond. We measured this working alliance using
Speaker 2: the same measure that folks would use with outpatient providers
Speaker 2: about how they they felt about their therapist, but instead
Speaker 2: of the therapist that now we're talking about therabot, and
Speaker 2: and folks rated it nearly identically to the norms that
Speaker 2: you would see on the outpatient literature. So we asked folks,
Speaker 2: we give folks the same measure, and that it's essentially
Speaker 2: equivalent to how folks are reading human providers in these ways.
Speaker 1: This is consistent with other where we're seeing people having
Speaker 1: relationship with chatbots and other domains. Yes, I'm old enough
Speaker 1: that it seems weird to me. I don't know, seem
Speaker 1: weird to you.
Speaker 2: I that part I this is more of a surprise
Speaker 2: to me that it was as the bonds were as
Speaker 2: high as they were, that they would actually be about
Speaker 2: what humans would be. And I will say, like one
Speaker 2: of the other surprises within the interactions was the number
Speaker 2: of people that would like respond kind of check in
Speaker 2: with therabot with and just say hey, just checking in
Speaker 2: as if like Therabot is like a I don't know.
Speaker 2: I would I would only like have anticipated folks would
Speaker 2: use this as a tool, so like not like they
Speaker 2: went to hang out with like almost that way. It's
Speaker 2: like our initiating a conversation that isn't I guess doesn't
Speaker 2: have an intention in mind.
Speaker 1: I say please, I'm thank you, I can't help my
Speaker 1: Is it because I think they're going to take over
Speaker 1: or is it a habit or what? I don't know,
Speaker 1: but I do, I do.
Speaker 2: Yeah, I wouldn't. I would say that this was more
Speaker 2: surprising the degree to that folks established this this level
Speaker 2: of a bond with it. I think it's actually really
Speaker 2: good and in really important that they do, in large
Speaker 2: part because that's one of the ways that we know
Speaker 2: psychotherapy works, is that that that folks can come together
Speaker 2: and trust this and develop this working relationship. So I
Speaker 2: think it's actually a necessary ingredient for this to work.
Speaker 1: To some I get it makes sense to me intellectually
Speaker 1: what you're saying. Does it give you any pause or
Speaker 1: do you just think it's great?
Speaker 2: It it gives me pause. If we weren't delivering evidence
Speaker 2: based treatment, Well, this is a good moment. Let's talk
Speaker 2: about the let's talk about the industry more generally. Yeah,
Speaker 2: this is not a you're not making a company, this
Speaker 2: is not a product, right, you don't have any money
Speaker 2: at stake. But there is a something of a therapy
Speaker 2: bought industry.
Speaker 1: There is a private sector. Like, tell me what is
Speaker 1: the broader landscape here?
Speaker 2: Like, so there's a lot of folks that are have
Speaker 2: jumped in predominantly sense the loans of shot GPT, and
Speaker 2: a lot of folks that have learned that you can
Speaker 2: call a foundation model fairly easily.
Speaker 1: When you say call you mean just sort of like
Speaker 1: you sort of take a foundation model like GBT, and
Speaker 1: then you kind of put a wrapper around exactly and
Speaker 1: the rapper it's like it's basically GPT with a therapist wrapper.
Speaker 2: Yeah. So it's a lot of folks within this industry
Speaker 2: are saying, hey, you act like a therapist and then
Speaker 2: kind of off to the races. It's it's otherwise not
Speaker 2: changed in any way, shape or form. It's it's like
Speaker 2: a literally like a system prompt. So if you were
Speaker 2: interacting with chat GBT, it would be something along the
Speaker 2: lines of, hey, act as a therapist and here's what
Speaker 2: we go on to do. They may have more directions
Speaker 2: than this. But that's this is kind of the light
Speaker 2: touch nature, so super different from what we're doing. Actually, yes,
Speaker 2: so we conducted the first randomized control trial of any
Speaker 2: generative AI for any type of clinical mental health problem.
Speaker 2: And so I know that these folks don't have evidence
Speaker 2: that this kind of thing works.
Speaker 1: I mean, there are non generative AI bots that people
Speaker 1: did randomize control trials of, right, just to be clear.
Speaker 2: Yes, there are non generative absolutely that have have evidence
Speaker 2: behind them. The generative side is very new, and so
Speaker 2: and there's a lot of folks in the generative space
Speaker 2: that have jumped in. Yeah, and so a lot of
Speaker 2: these folks are not psychologists and not psychiatrists, and and
Speaker 2: Silicon Valley, there's a saying move fast and break things.
Speaker 2: This is not the setting to do that. Like move
Speaker 2: fast and break people is what you're talking about here.
Speaker 2: You know, it's like the and the amount of times
Speaker 2: that these foundation models act in profoundly unsafe ways would
Speaker 2: be unacceptable to the field. So like that, we tested
Speaker 2: a lot of these models alongside when we were developing
Speaker 2: all of this. So it's like, I know that they
Speaker 2: don't they don't work in this kind of way in
Speaker 2: a real safe environment. So because of that, I'm I'm
Speaker 2: really hugely concerned with kind of the field at large
Speaker 2: that is moving fast and doesn't really have this level
Speaker 2: of dedication to trying to do it right. And I
Speaker 2: think one of the things that's really kind of canning
Speaker 2: within this is it always looks polished, so it's harder
Speaker 2: to see when you're getting exposed to things that are dangerous.
Speaker 2: But the field, I think is in a spot where
Speaker 2: there's a lot of folks that are out there that
Speaker 2: are acting and implementing things that are untested, and I
Speaker 2: suspect a lot of them are really dangerous.
Speaker 1: How do you how do you imagine theahbut getting from
Speaker 1: the experimental phase into the widespread use phase.
Speaker 2: Yeah, so we want to essentially have one at least
Speaker 2: one larger trial before we do this. You know, we
Speaker 2: have it's pretty a pretty decent sized first trial for
Speaker 2: being a first trial, but it's not something that I
Speaker 2: would want to see out in the open just yet.
Speaker 2: We want to have continue to oversight it, make sure
Speaker 2: it's safe and effective. But if it continues to demonstrate
Speaker 2: safety and effectiveness, this is one of those things that
Speaker 2: why I got into this is to really have an
Speaker 2: impact on folks lives, and this is one of those
Speaker 2: things that could scale really effective personalized cares in real ways. So, yeah,
Speaker 2: we we intend to if evidence continues to show that
Speaker 2: it's safe and effective, to make this out into the
Speaker 2: open market. In terms of the thing that I care
Speaker 2: about in terms of the ways that we could do
Speaker 2: this is trying to do this in some ways that
Speaker 2: would be scalable, so that we're considering a bunch of
Speaker 2: different pathways. Some of those would be delivered by philanthropy
Speaker 2: or nonprofit models. We are considering also like just a
Speaker 2: strategy that would just not for me to make money,
Speaker 2: but just to scale this under some kind of for
Speaker 2: profit structure as well, but really just to try to
Speaker 2: get this out into the open so that folks could
Speaker 2: actually use it, because ultimately we'll need some kind of
Speaker 2: revenue in some ways to be part of this that
Speaker 2: would essentially enable the servers to stay on and to
Speaker 2: scale it.
Speaker 1: And presumably you have to pay some amount of people
Speaker 1: to do some amount of supervision absolutely forever.
Speaker 2: Yeah, So we in the real deployment setting, we hope
Speaker 2: to have essentially the decreasing levels of oversight relative to
Speaker 2: these trials, but not an absence of oversight. So exactly
Speaker 2: you're not going to stay up all night reading every
Speaker 2: message exactly, that won't be sustainable for the future, but
Speaker 2: we will have like flags for things that should be
Speaker 2: seen by humans and intervened upon.
Speaker 1: Let's talk about this other domain you've worked in in
Speaker 1: terms of technology and mental health, right, and so in
Speaker 1: addition to your work on thera bot, you've done a
Speaker 1: lot of work on it seems like basically diagnosis monitoring people,
Speaker 1: essentially using mobile devices and wearables to track people's mental
Speaker 1: health to predict outcomes like tell me about your work
Speaker 1: there and the field there.
Speaker 2: So essentially it's trying to trying to monitor folks within
Speaker 2: their freestanding conditions, so like in their real real life,
Speaker 2: through using technology so in ways that are not don't
Speaker 2: require burden.
Speaker 1: The starting point is like your phone is collecting data
Speaker 1: about you all the time. What if that data could
Speaker 1: make you less depressed?
Speaker 2: Yeah, exactly, what if we could use that data to
Speaker 2: know something about you so that we could actually intervene
Speaker 2: and so, like thinking about a lot of mental health symptoms.
Speaker 2: I think one of the challenges of them is they
Speaker 2: are not like all or nothing the field. Actually, I
Speaker 2: think it's this really wrong. And when you would talk
Speaker 2: to anybody who has a experience as a clinical problem,
Speaker 2: they have changes that happen pretty rapidly within their daily life.
Speaker 2: So they like will have better moments and worse moments
Speaker 2: within a day, They'll have better and worse days. And
Speaker 2: it's not like it's all this like it's always depressed
Speaker 2: or not depressed. It's like these these fluctuating states of it.
Speaker 2: And I think one of the things that's really important
Speaker 2: about these types of things is if we can monitor
Speaker 2: and predict those rapid changes, which I think we can.
Speaker 2: We have a that we can is that we can
Speaker 2: then intervene upon the symptoms before they happen in real time,
Speaker 2: so like trying to predict the ebbs and the flows
Speaker 2: of the symptoms, not to like say, I want somebody
Speaker 2: to never be able to be stressed within their life,
Speaker 2: but so that they can actually be more resilient and
Speaker 2: cope with it.
Speaker 1: And so what's the state of that art, Like, is
Speaker 1: there somebody who's can you do that? Can somebody do that?
Speaker 1: Is there an app for that?
Speaker 2: As we used to say, Yeah, I mean we have
Speaker 2: the science surrounding. This is about ten years old. We've
Speaker 2: done about forty studies in this area across a broad
Speaker 2: range of symptoms, so anxiety, depression, post traumatic stress disorder, schizophrenia,
Speaker 2: bipolar disorder, eating disorders, so a lodge are different types
Speaker 2: of clinical phenomenon and we can predict a lot of
Speaker 2: different things in ways that I think are really important.
Speaker 2: But I think, like to really move the needle on
Speaker 2: something that would make it into population wide ability to
Speaker 2: do this, I think the real thing that would be
Speaker 2: needed for like the ability to do this is to
Speaker 2: pair this with intervention that's dynamic. So something that's actually ability,
Speaker 2: has an ability to change and has like a boundless
Speaker 2: context of intervention. So I'm going to actually loop you.
Speaker 1: Back like the Abot.
Speaker 2: That's exactly right. So these two things that have been
Speaker 2: distinct arms of my work are like so natural compliments
Speaker 2: to one another. Now think about Okay, let's come back
Speaker 2: to therabot in this kind of setting.
Speaker 1: So give me the dream.
Speaker 2: So this is the dream. So you have Therabot, but
Speaker 2: instead of like a psychologist that's completely unaware of what happens,
Speaker 2: is reliant on the patient to tell them everything that's
Speaker 2: going on. In their life. Yeah, all of a sudden,
Speaker 2: there butt knows them knows hey, oh this they're not
Speaker 2: sleeping very well for the past couple days. They haven't
Speaker 2: left their home this week, and this is a big
Speaker 2: deviation from them and how they normally would live life
Speaker 2: Like this can be targets of intervention that don't wait
Speaker 2: for this to be some sustained pattern in their life
Speaker 2: that becomes entrenched and hard to change. Like, no, let's
Speaker 2: actually have that as part of the conversation, where we
Speaker 2: don't have to wait for someone to tell us that
Speaker 2: that they didn't get out of bed. We kind of
Speaker 2: know that they haven't left their house, and we can
Speaker 2: actually make that a content of the intervention. So that's like,
Speaker 2: I think these these ability to like intervene proactively in
Speaker 2: these risk moments and not wait for folks to come
Speaker 2: to us and tell us every aspect of their life
Speaker 2: that they may not know and so like because of this,
Speaker 2: it's that's that's where I think there's a really powerful
Speaker 2: pairing of these two.
Speaker 1: I can see why that combination would be incredibly powerful
Speaker 1: and helpful. Do you worry at all about having that
Speaker 1: much information and that much sort of personal information on
Speaker 1: so many dimensions about people who are by definition vulnerable.
Speaker 2: Yeah, I mean, in some ways, I think it's the
Speaker 2: real ways that folks are already collecting a lot of
Speaker 2: this type of data already on these same populations, and
Speaker 2: now that we could put it to good use. Do
Speaker 2: I worry about kind of yet falling into the wrong hands. Absolutely.
Speaker 2: I mean we have like really big tight data security
Speaker 2: kind of protocols surrounding all of this to try to
Speaker 2: make sure that only folks that are established members of
Speaker 2: the team have any access to this data. And so yeah,
Speaker 2: we are really concerned about it. But yeah, no, if
Speaker 2: there was a breach or something like that that could
Speaker 2: be hugely impactful, something that would be greatly worry.
Speaker 1: We'll be back in a minute with the lightning round. Hey,
Speaker 1: let's finish with the lightning round.
Speaker 2: Okay.
Speaker 1: On net, have smartphones made us happier or less happy?
Speaker 2: Less happy?
Speaker 1: You think that you think you could change that, You
Speaker 1: think you could make the net flip back the other way.
Speaker 2: I think that we need to meet people where they are,
Speaker 2: and and so this is we're not like trying to
Speaker 2: keep folks on their phones, right, like, we're trying to
Speaker 2: actually start with where they are and intervene there, but
Speaker 2: like push them to go and experience life in a
Speaker 2: lot of ways.
Speaker 1: Yeah, Freud overrated or underrated?
Speaker 2: Overrated?
Speaker 1: Still okay, who's the most underrated thinker in the history
Speaker 1: of psychology? Oh?
Speaker 2: My, I I mean to some degree, Skinner was like
Speaker 2: really operant conditioning is like at the heart of most
Speaker 2: clinical phenomenon that deal with emotions, and I think it's
Speaker 2: probably one of the most impactful. Like it's so simple
Speaker 2: in some ways that behavior is shaped by both positive
Speaker 2: essentially benefits and like drawbacks, so rewards and punishments and
Speaker 2: these these types of things are the simplicity of it
Speaker 2: is is so simple, but like the how meaningful it
Speaker 2: is and daily life is so profound, we.
Speaker 1: Still underrate it. I mean when I the little bit
Speaker 1: I know about Skinner, I think of the black box, right,
Speaker 1: the like, don't worry about what's going on in somebody's mind,
Speaker 1: just look at what's going on on the outdoit. Yeah.
Speaker 2: Yeah, And with.
Speaker 1: Behavior, I mean in a way it sort of maps
Speaker 1: to your wearable's mobile devices thing, right, like just look,
Speaker 1: if you don't go outside, you get sad, and so
Speaker 1: go outside.
Speaker 2: Sure exactly. I am a behaviorist at heart, So this
Speaker 2: is part of part of what however you way.
Speaker 1: I mean, I was actually think briefly before we talked
Speaker 1: that wasn't gonna bring it up, But since you brought
Speaker 1: it up, it's interesting to think. Like the famous thing
Speaker 1: people say about Skinner is like the mind is a
Speaker 1: black box, right, we don't know what's going on on
Speaker 1: the inside and don't worry about it.
Speaker 2: Yeah.
Speaker 1: It makes me think of the way large language models
Speaker 1: on black boxes, and even the people who build them
Speaker 1: don't understand how they work.
Speaker 2: Right. Yeah, absolutely, I think psychologists in some ways are
Speaker 2: best suited to understand the behavior of large language models,
Speaker 2: because it's actually the science of behavior absence the ability
Speaker 2: to like potentially understand what's going on inside, Like neuroscience
Speaker 2: is a natural compliment, but in some ways a different
Speaker 2: different lens in which you view the world. So like
Speaker 2: trying to develop a predictable system that is shaped. I
Speaker 2: actually think we're not so bad in terms of folks
Speaker 2: to be able to take this on.
Speaker 1: What's your go to karaoke song?
Speaker 2: Oh, don't stop believing. I'm a big karaoke person too.
Speaker 1: Somebody just sent me that just the vocal from stop believing.
Speaker 2: Ah, yeah, no, it's it's it's like a meme.
Speaker 1: It's amazing, it is.
Speaker 2: Uh.
Speaker 1: What's one thing you've learned about yourself from a wearable device?
Speaker 2: Mm hmm. One of the things that I would say,
Speaker 2: like my ability to understand recognize when I've actually had
Speaker 2: a poor night's sleep or a good night's sleep has
Speaker 2: gotten much better over time. Like I think, as humans
Speaker 2: were not very well calibrated to it. But as you
Speaker 2: actually start to wear them and get understand you can
Speaker 2: you are you become a better self reporter.
Speaker 1: Actually I sleep badly. I assume it's because I'm middle aged.
Speaker 1: I do most of the things you're supposed to do.
Speaker 1: But give me one tip for sleeping. Well, I get
Speaker 1: to sleep, but then I wake up in the middle
Speaker 1: of the night.
Speaker 2: Yeah. That. I think. One of the things that a
Speaker 2: lot of people will do is they'll worry, particularly in bed,
Speaker 2: or use this as a time for thinking, so a
Speaker 2: lot of a lot of the effective surrounding that, or
Speaker 2: to try to actually give yourself that same time that
Speaker 2: would be that unstructured time that you would be dedicated
Speaker 2: that you might experience in bed.
Speaker 1: You tell me I should worry it ten at night
Speaker 1: instead of three in the morning. If I worry, if
Speaker 1: I say it ten at night, okay, worry now, then
Speaker 1: I'll sleep through the night.
Speaker 2: There there's literally evidence surrounding scheduling your worries out and
Speaker 2: I love during the day and it does work. So yeah,
Speaker 2: that's okay. If it's got some.
Speaker 1: Worries, I'm gonna worry it ten tonight, I'll let you
Speaker 1: know tomorrow morning.
Speaker 2: If it were just don't do it in bed. Yeah, okay, okay.
Speaker 1: If you had to build a chatbot based on one
Speaker 1: of the following fictional therapists or psychiatrists, which fictional therapist
Speaker 1: or psychiatrist would it be? A Jennifer Milthy from The Sopranos,
Speaker 1: B Doctor Krokowski from The Magic Mountain, see Fraser from Fraser,
Speaker 1: or d Hannibal Lecter.
Speaker 2: Oh god, okay, I would probably go with Frasier, a
Speaker 2: very different style of therapy than but I think his
Speaker 2: demeanor is at least generally decent, So yeah, mostly appropriate
Speaker 2: with most of his clients from what I remember in
Speaker 2: the show.
Speaker 1: Okay, it's a very thoughtful response to an absurd question.
Speaker 1: Anything else we should talk about?
Speaker 2: You've asked wonderful questions one thing I will say, maybe
Speaker 2: for folks that might be listening, is a lot of
Speaker 2: folks are already using generator AI for their mental health treatment,
Speaker 2: and so I will I'll give a recommendation if folks
Speaker 2: are doing this already, that they just treat it with
Speaker 2: the same level of concern they would have the Internet.
Speaker 2: They there may be benefits they can get out of it. Awesome, great,
Speaker 2: but just don't work on changing something within your daily
Speaker 2: life surrounding particularly your behavior, based on what these models
Speaker 2: are doing, without some real thought on making sure that
Speaker 2: that is actually going to be a safe thing for
Speaker 2: you to do.
Speaker 1: Nick Jacobsen is an assistant professor at the Center for
Speaker 1: Technology and Behavioral Health at the Geissel School of Medicine
Speaker 1: at Dartmouth. Today's show was produced by Gabriel Hunter Chang.
Speaker 1: It was edited by Lydia Jean Kott and engineered by
Speaker 1: Sarah Brugier. You can email us at problem at Pushkin
Speaker 1: dot FM. I'm Jacob Boldstein, and we'll be back next
Speaker 1: week with another
Speaker 2: Episode of What's Your Problem.