Speaker 1: Bloomberg Audio Studios, Podcasts, Radio News. Hello and welcome to
Speaker 1: another episode of The Odd Laws podcast.
Speaker 2: I'm Joe Wisenthal and I'm Tracy Alloway.
Speaker 1: Tracy, have you played around with GPT five much?
Speaker 2: Not really, I've been perplexity pills. Oh that's what your
Speaker 2: main Yeah, that's my main one at the moment. But
Speaker 2: is it good? I hear mixed.
Speaker 1: I use it because I use GPT every day. It
Speaker 1: does not strike me as like obviously better yeah for
Speaker 1: my uses than like the three models, which I've been
Speaker 1: very impressed by because you know, I want to establish them.
Speaker 3: No hater or anything like that.
Speaker 1: But like, it did not strike me as like, oh,
Speaker 1: this is like an.
Speaker 2: Amazing Yeah, this is the thing.
Speaker 3: Step function or whatever.
Speaker 2: It feels like the sort of breakthroughs awe inspiring breakthroughs
Speaker 2: are kind of behind us, and a lot of the
Speaker 2: progress on the models feels very incremental at this point,
Speaker 2: even though people are spending a lot of time and
Speaker 2: resources on doing it.
Speaker 1: The one thing GPG five does prompt me and say, oh,
Speaker 1: that's a great question. Would you like to follow up
Speaker 1: more on that?
Speaker 3: But it's like does it.
Speaker 2: Say, o, Joe, you're so smart? That's such a smart question.
Speaker 3: Say you know what it did? Say?
Speaker 1: I asked to follow up, and it started an answer
Speaker 1: with love it and then love it? Do you want
Speaker 1: me to look into that?
Speaker 4: Yes?
Speaker 2: They are very flattering, aren't they. Actually, that's one thing
Speaker 2: I like about perplexity is it doesn't really flatter you.
Speaker 2: It just spits out an answer.
Speaker 1: So anyway, there's so many questions I have about AI,
Speaker 1: and we talk about the business old fair amount and
Speaker 1: video and all that stuff. We actually don't really talk
Speaker 1: that much about the pure research side as much. But
Speaker 1: it's pretty important, I think, because I think a lot
Speaker 1: of people would agree that if the skills are like
Speaker 1: slowing down, or if there were a wall or something
Speaker 1: like that, that might change some of these business model calculations,
Speaker 1: et cetera. So I think it's good we need to
Speaker 1: get an update on just sort of the state of
Speaker 1: the art the science of AI.
Speaker 4: Yeah.
Speaker 2: Also, it would be nice just to understand what's possible
Speaker 2: in terms of the AI models and what people are
Speaker 2: actually researching, what they're working towards, work like, is it
Speaker 2: mostly about price? Is it mostly about the output? Is
Speaker 2: it mostly about energy use? All those things?
Speaker 1: All those things, Well, I'm really excited to say we
Speaker 1: have the perfect guest, someone who is an AI researcher.
Speaker 1: We're gonna be speaking with Jack Morris. He's currently about
Speaker 1: to finish his PhD.
Speaker 3: At Cornell in AI.
Speaker 1: He's been affiliated with Meta professionally, so presumably he already
Speaker 1: has a one hundred million dollar pay package in the bank.
Speaker 1: Now he's shaking his head, he's not that's a joke.
Speaker 1: But Jack, thank you so much for coming on odd lots.
Speaker 4: Yeah, thanks for having me. This is gonna be fun.
Speaker 1: What do you explain to me, like what you're up to,
Speaker 1: because I don't really understand how.
Speaker 3: It works where people are.
Speaker 1: They're at a university and they're also at a company,
Speaker 1: and this isn't how it works. And much of the world, right,
Speaker 1: people get their degree and then they get a job.
Speaker 1: I get the impression that in the AI world it's
Speaker 1: a little fuzzier in terms of one's affiliations between industry
Speaker 1: and education and stuff like that.
Speaker 4: Yeah, that's definitely true. I think might be on the
Speaker 4: way out, but I can tell you about my situation.
Speaker 4: So there's kind of a public research world and like
Speaker 4: a private research world, and all the academic institutions do
Speaker 4: public research, and the AI labs like Open Ai, Anthropic, Google,
Speaker 4: deep Mind, they essentially do private research where they have
Speaker 4: these people in house that are running experiments and learning
Speaker 4: more about their systems, but they don't publish anything or
Speaker 4: share any of their knowledge. And so a cool thing
Speaker 4: about getting your PhD right now is you can do
Speaker 4: research right about it and then publicize it like put
Speaker 4: it online, I tweet about it. I kind of like
Speaker 4: can talk to you about it. And there's a few
Speaker 4: places left that will still kind of moment, we're never.
Speaker 3: Going to hear from you again.
Speaker 4: Yeah, I'll make sure they have a clause in my
Speaker 4: contract that I can still talk to Joe and Tracy.
Speaker 2: The all thoughts clause. Yes, that would be important. So
Speaker 2: when we say AI research or an AI researcher, what
Speaker 2: exactly does that entail? Can't the AI models just research themselves?
Speaker 2: Just let them do it?
Speaker 4: Yeah, that's actually a very smart idea, and like people
Speaker 4: are really worried about that. Actually, Like if we get
Speaker 4: to the point where the AI can improve itself into researching, yeah,
Speaker 4: then it sort of gets smarter and then it improves
Speaker 4: themself again and it ends up being this kind of
Speaker 4: exponential improvement that ends up with all of our demise.
Speaker 4: But I think right now it's not quite there yet.
Speaker 4: Like maybe you can talk to CHGBT what good Yeah,
Speaker 4: And good news for me too, because it means I
Speaker 4: can still get a degree and be gainfully employed. But
Speaker 4: I think it's it's still helpful, but we still need
Speaker 4: like humans to make these improvements. And in terms of
Speaker 4: what the actual day to day work looks like, I
Speaker 4: think it really varies. Like there's some people working on
Speaker 4: trying to make the models run faster, or trying to
Speaker 4: make the hardware that runs the models run faster more efficiently.
Speaker 4: There's people that try to work on the data, like
Speaker 4: what should we train on more coding problems or more
Speaker 4: textbooks or more Reddit posts, what works best to make
Speaker 4: the model? And then there's a lot more people working
Speaker 4: on different areas of the stack, like training algorithms. I
Speaker 4: kind of have my own little niche and niche. There's
Speaker 4: this old field of information theory from like the twentieth
Speaker 4: century where they talk about bits like a zero or
Speaker 4: a one is a bit and you can add them
Speaker 4: up and have kilobytes and megabytes. And so I've been
Speaker 4: trying to think about what that means in like the
Speaker 4: chat GBT world, if you train a model on a
Speaker 4: certain number of bits, how many bits does it actually learn?
Speaker 4: And like can you look at the model and figure
Speaker 4: out like if you have one slice of the model,
Speaker 4: how many bits that is and stuff like that. So
Speaker 4: maybe the easiest way to explain is if you had,
Speaker 4: for some god forsaken reason to use chat GBT as
Speaker 4: like a flash drive, like you had a certain set
Speaker 4: of data and it had to memorize all that data,
Speaker 4: Like how much data could it actually store? That's the
Speaker 4: kind of area I've been working in. And then you know,
Speaker 4: once you're there, you kind of realize we could do this,
Speaker 4: or maybe next semester, if we have time, we could
Speaker 4: try this other thing. And so there's it kind of
Speaker 4: branches out and there's a lot of little problems that
Speaker 4: you can try.
Speaker 1: I mentioned GPT five fine to me, It does not
Speaker 1: strike me as like you know, because actually so the
Speaker 1: first time I use cha GPT is genuinely blown away
Speaker 1: like most people. And then actually I was pretty blown
Speaker 1: away by the three models, in part because of how
Speaker 1: well they could do document search and superior to Google
Speaker 1: Search in many respects and also just the organization of
Speaker 1: a lot of unstructured data, et cetera. Like I didn't
Speaker 1: have like some oh my god wow moment with GPT five.
Speaker 1: It's like, this seems like, how do we measure whether
Speaker 1: AI is getting better all the time.
Speaker 4: Yeah, that's that's a huge question, right.
Speaker 1: Well, let me ask you, Okay, let me ask you
Speaker 1: actually a more specific question. How do the entities that
Speaker 1: test AI models as their job or as their function?
Speaker 1: What does the formal testing process look like to rank
Speaker 1: the quality of AI models?
Speaker 4: Okay, yeah, that's that's more tractable. We can we can
Speaker 4: start there, and then we can talk about three and
Speaker 4: GPT five. So there's essentially two ways people do this
Speaker 4: kind of model evaluation. The main one is just by
Speaker 4: testing them on different data sets. So, for example, there's
Speaker 4: this data set called swee bench that's a bunch of
Speaker 4: software engineering related coding problems and they all have a
Speaker 4: human written solution and tests, and so you can ask
Speaker 4: GPT five, can you write the code for this and
Speaker 4: then run the tests and see if it's right? And
Speaker 4: still the models are pretty bad at that. I think
Speaker 4: they can do about half of them. They're very hard.
Speaker 4: They're like entire days of work for professional software engineers.
Speaker 4: But when a new model comes out, they can say, oh, look,
Speaker 4: we actually got a higher score on sweet bench. And
Speaker 4: there's a ton of different data sets like that. So
Speaker 4: when GBT five comes out, they say, you know, it's
Speaker 4: better at these types of coding tests. And a big
Speaker 4: one that specifically open AI has been advocating for is math,
Speaker 4: like they did the International Math Olympiad, and they said
Speaker 4: essentially GBT five scored at the level of the best
Speaker 4: high school mathematicians, which is pretty cool. But you raise
Speaker 4: a good question of how is that actually map to
Speaker 4: real world usage? And I think this is like a
Speaker 4: really hard problem that people still haven't figured out.
Speaker 2: Does anyone try to capture that sort of like genes sequah?
Speaker 2: I guess when it comes to AI models, is one
Speaker 2: of the tests asking it to I don't know, come
Speaker 2: up with a stupid limerick or something.
Speaker 4: Yeah, there are a lot of tests like that. There's
Speaker 4: some creative writing benchmarks and some poetry related ones. But
Speaker 4: I think you point out something interesting that for example,
Speaker 4: I mostly use Claude from Anthropic and I think Claude
Speaker 4: does have this something to it that's like a little
Speaker 4: bit different, and it's very difficult to characterize. It's just
Speaker 4: sort of the way it speaks to you and the
Speaker 4: way it thinks of itself is I like it a
Speaker 4: lot better, but I don't know how you would design
Speaker 4: like a data set that can really capture that. The
Speaker 4: second way they do the evaluation is by they call
Speaker 4: it it's Elo scores, like in chess. So they, for example,
Speaker 4: ask the two models to write a limerick, and then
Speaker 4: they have humans rank which one is better, and they
Speaker 4: make this kind of lat of Elo rankings for models.
Speaker 4: So I think right now Claude or GPT five or
Speaker 4: maybe the Google model is top on this ladder.
Speaker 1: The algorithm made famous in the social network that Mark
Speaker 1: Zuckerberg used to rate the of his colleagues still the
Speaker 1: workhorse model for comp evaluation.
Speaker 2: That's some good trivia, Joe, very good and no comment. Well,
Speaker 2: I assume just on the hard number evaluation. People are
Speaker 2: also ranking these on data usage, energy, that sort of.
Speaker 4: Thing as well.
Speaker 2: Right speed, speed would be a definitely.
Speaker 4: The AI companies like to use price as a metric,
Speaker 4: which is kind of interesting because there's a lot that
Speaker 4: goes on behind the scenes, including just sort of like
Speaker 4: free money that drives the prices down, but they also
Speaker 4: do benchmark speed, and I think you make a good
Speaker 4: point that the benchmarks can be pretty misleading, Like, for example,
Speaker 4: there's a bunch of recent open source models that came
Speaker 4: from different Chinese AI labs that have really, really high
Speaker 4: scores on certain benchmarks, but people kind of think they're
Speaker 4: not as good for real world usage for whatever reason.
Speaker 1: I've seen people talk about this isn't part of the
Speaker 1: problem with testing AI or evaluating AI. That a lot
Speaker 1: of these problems exist in the real world already, right,
Speaker 1: You see this a lot, and people are always finding this,
Speaker 1: which is that here's an AI model that is amazing
Speaker 1: at math on the math Olympiad, and yet it gets
Speaker 1: tripped up by questions like which is heavier a pound
Speaker 1: of steel or two pounds of feathers, And they'll say
Speaker 1: that that's a trick question. A pound of steel weighs the
Speaker 1: same as two pounds of feathers, which is clearly like
Speaker 1: it was clearly then been trained in some sense to
Speaker 1: recognize these steel versus feathers thing or whatever it is.
Speaker 1: I forget if it's steel, But it also clearly can't
Speaker 1: measure whether one or.
Speaker 3: Two is bigger.
Speaker 4: Yeah, that's a really good example. I think they kind
Speaker 4: of successively include these kinds of things in more rounds
Speaker 4: of training data, and so every time a new model
Speaker 4: comes out, they kind of patch little holes that appeared
Speaker 4: in the previous models. So you're pointing to this, like
Speaker 4: they probably started with the classic riddle that's like a
Speaker 4: pound of bricks or a pound of feathers bricks and
Speaker 4: they're equal, but then like the models got that wrong,
Speaker 4: and so they added to.
Speaker 1: Something a very efficient way to achieve intelligence, like, oh yeah,
Speaker 1: we should have included that.
Speaker 3: Oh yeah, we got to include that trick. Oh yeah,
Speaker 3: we gotta have right.
Speaker 1: Like ever, like going that does not speak to me
Speaker 1: of a line towards something that we would call anything
Speaker 1: resembling human intelligence.
Speaker 4: I definitely agree. I think one counter example is people
Speaker 4: said this for a long time about self driving cars,
Speaker 4: Like everyone was really excited about them for a long time,
Speaker 4: and then they kind of didn't really work, like eight
Speaker 4: or so years ago, and there was this period where
Speaker 4: they were saying, oh, the models can't do green cones.
Speaker 4: We're going out there trying to take videos of green cones,
Speaker 4: and yeah, they can't do snow. I'm saying that it
Speaker 4: worked for them, and so it might be possible. But
Speaker 4: in the case of language models, there's something a little
Speaker 4: more interesting happening, because we now have two ways to learn.
Speaker 4: If you guys are ready, we could we could get
Speaker 4: into something a little technical, which I think gives you
Speaker 4: some insights. So there's essentially two ways you can teach
Speaker 4: machines to learn from data. One is called supervised learning,
Speaker 4: where the computer will copy what you did, which is
Speaker 4: like basically what we're talking about now, and the other
Speaker 4: is called reinforcement learning, where the computer just does something
Speaker 4: and then you give it a reward if it does
Speaker 4: something well. And so for a long time, like the
Speaker 4: original chat GBT was mostly just trained with supervised learning,
Speaker 4: like it would just copy the text from all of
Speaker 4: the Internet, and so the best it could ever do
Speaker 4: is emulate Reddit posts very well. And there was a
Speaker 4: tiny bit of reinforcement learning, but people didn't know how
Speaker 4: to do it right. And then you mentioned this three model,
Speaker 4: which is kind of in some ways like a big jump,
Speaker 4: like it made the models much better at math, much
Speaker 4: better at certain things. And the way they did that
Speaker 4: is actually through reinforcement learning. Found out a way to
Speaker 4: kind of like let the model think for a while
Speaker 4: and then give it a reward when it gets the
Speaker 4: answer at the end. It's kind of scary.
Speaker 2: Yeah, when you say give it a reward, is.
Speaker 3: It like take a cookie paying robots?
Speaker 1: Yeah?
Speaker 2: Well no, genuinely, like what is the reward? You just
Speaker 2: tell it it did a good job.
Speaker 4: You just give it like a higher number. Okay, and
Speaker 4: that makes you happy, all right.
Speaker 2: I'd get a little bit worried when we're like giving
Speaker 2: it cupcakes or something like here you go, good job.
Speaker 2: Just going back to the intro, you know, we were
Speaker 2: talking about how it feels like a lot of the
Speaker 2: progress on AI models is a little bit more incremental,
Speaker 2: and I guess it's hard to tell whether that's just
Speaker 2: personal bias because now we're used to them and the
Speaker 2: sort of wow moment has passed. But what does it
Speaker 2: feel like to you in terms of improvements? Are we
Speaker 2: seeing the improvement cycle accelerate or decelerate at this point?
Speaker 4: I think it's kind of like the market, where it's
Speaker 4: like always it gets faster for a little while, and
Speaker 4: then it feels like things have slowed down and the
Speaker 4: progress is never quite in the areas that you expect
Speaker 4: as one example, people really thought this year was the
Speaker 4: year when the assistance would start being able to act
Speaker 4: like actual assistants, like the Year of agents. People actually
Speaker 4: coined that term, I think, like the year of agents,
Speaker 4: and it really it didn't happen for whatever reason. Maybe
Speaker 4: it will in the next three months. But the agents
Speaker 4: are still pretty bad the ones that you can use.
Speaker 4: But they did get way better at competitive math, Like
Speaker 4: now they can do these like world class proofs that
Speaker 4: they couldn't do before. So it's almost unpredictable, like which
Speaker 4: areas the AI will kind of conquer next, But it
Speaker 4: does feel like progress is continuing.
Speaker 1: Actually, what happened with agents? I've never had a successful
Speaker 1: agent experience, even basic things like come up with a
Speaker 1: list of every past odd Lots guests, yeah and put
Speaker 1: it in a file or something like that, which just
Speaker 1: there's an RSS feed that exists for odd Lots. This
Speaker 1: should be ray stick for it all around, and then
Speaker 1: something will happen or it'll get lazy. Here's like here's
Speaker 1: fifteen and what is actually this is thought leaders love
Speaker 1: this stuff. They love to talking about the agents. So
Speaker 1: what actually happened with agents? Maybe they'll get there, but
Speaker 1: what do you use to what is the roadblock there.
Speaker 4: I don't think there's any conceptual roadblock, Like there's no
Speaker 4: reason why you couldn't collect data for that and train
Speaker 4: them either in a supervised way or using reinforcement learning.
Speaker 4: It just hasn't happened yet. So I think maybe behind
Speaker 4: the scenes it turned out that the problem was harder
Speaker 4: than people thought, Like getting data from all those scenarios
Speaker 4: is really hard. And there have been some stories from
Speaker 4: like people that I've heard of that found these little
Speaker 4: companies in San Francisco and they build these tiny environments
Speaker 4: for the AI labs to do reinforcement learning on for agents,
Speaker 4: like for example, doing a calendar. They'll build like a
Speaker 4: little calendar app, but make it have rewards so you
Speaker 4: can do reinforcement learning, and they can just sell that
Speaker 4: for like hundreds of thousands of dollars. So I think
Speaker 4: the progress is ongoing behind the scenes, Like there's a
Speaker 4: whole ecosystem built around it. It just hasn't really manifested
Speaker 4: in the products that we use.
Speaker 2: I was going to ask, how much of the difficulty is,
Speaker 2: you know, the actual development of the models, the thinking part,
Speaker 2: versus just getting them to plug in seamlessly with other applications.
Speaker 4: Yeah, I think the second thing is probably the biggest
Speaker 4: barrier in terms of time, Like it just takes a
Speaker 4: really long time to figure out what data you need
Speaker 4: and collect it properly and actually train the models on
Speaker 4: that data. But at the same time, there are people
Speaker 4: like me who are trying to work on better like
Speaker 4: conceptual frameworks for training the models. So to go back
Speaker 4: to the three example, doing reinforcement learning on CHATGBT, like
Speaker 4: that seems to me like a huge breakthrough, Like we
Speaker 4: didn't know how to do that before. It unlocks all
Speaker 4: sorts of doors and ways to train the models. So
Speaker 4: even if maybe you don't think that model was that
Speaker 4: much better than the previous one, it seems like it
Speaker 4: will give us huge improvements in the future.
Speaker 1: So you mentioned at the intro that it's possible, hopefully
Speaker 1: you'll get a close but you might end up in
Speaker 1: a situation which you go to work for some frontier
Speaker 1: AI lab and we never hear from you again, or
Speaker 1: you just post cryptic tweets like oh no idea, what's coming,
Speaker 1: Oh it's gonna.
Speaker 3: Be so over or whatever. Yeah, an the death Star, Yeah,
Speaker 3: it's very annoying.
Speaker 1: The way they all tweet, it's possible talk to us
Speaker 1: about like why not work on an open source project?
Speaker 1: And this is of course when people talk about deep
Speaker 1: seek and a lot of the Chinese models that the
Speaker 1: US competes with, a lot of those are open source.
Speaker 1: Presumably you could keep coming on odd lads over and
Speaker 1: over again, why like what is even the case for
Speaker 1: the best and the brightest to work on a closed
Speaker 1: source frontier models.
Speaker 4: Yeah, it's a really hard question, Like I've I've struggled
Speaker 4: with this in my own personal decision making. I was
Speaker 4: originally thinking, Oh, I'd love to become a professor and
Speaker 4: mentor younger students and get a whole like group of
Speaker 4: these ideas going and start working on similar related problems
Speaker 4: to the stuff I was talking about. And I still
Speaker 4: think that would be fun. But there's a big gap
Speaker 4: in terms of the things we can do at Cornell
Speaker 4: and the things that you can do at open AI.
Speaker 4: Like they just have like crazy infrastructure for training models
Speaker 4: really easily and data and a ton of really good data.
Speaker 4: And so I think as that gap has widened, I've
Speaker 4: felt like a lot of what we're doing is like
Speaker 4: kind of devising these toy scenarios where we can study
Speaker 4: interesting things, but I feel a bit disconnected from the
Speaker 4: real like progress of humanity. You know, like if you
Speaker 4: really agree that this is like the biggest problem of
Speaker 4: our time. I don't want to say it's like the
Speaker 4: Manhattan Project, but like, what's more like trying to go
Speaker 4: to the Moon in the sixties? The space race. It's
Speaker 4: kind of like a space race going on in these
Speaker 4: different private labs. You want to be a part of it.
Speaker 4: Like there's crazy energy that it has huge implications for
Speaker 4: the future of society. So I think I am interested
Speaker 4: in participating in that. My big question is like, if
Speaker 4: you think that the reinforcement learning thing was the most
Speaker 4: recent big scientific breakthrough, like oh one, and then three,
Speaker 4: what's next? And then like where will that actually be happening.
Speaker 4: That's kind of what I'm thinking about right now.
Speaker 2: Just on the data point. I was reading your excellent
Speaker 2: substack and you argue that there's probably an upper bound
Speaker 2: to what you can get out of a given data set,
Speaker 2: and at some point, like the training starts to look
Speaker 2: pretty similar, right, and the data becomes the differentiating factor.
Speaker 2: How important are data sets to AI research? And I guess, like,
Speaker 2: how do you go about finding really cool ones and
Speaker 2: what's left. Because I feel like, you know, using the
Speaker 2: space race analogy, everyone has been running so fast on this.
Speaker 2: It feels like all the data sets must have been
Speaker 2: a explored by now, but I guess they haven't.
Speaker 4: Yeah, yeah, I think this is really getting to the
Speaker 4: heart of what people are trying to figure out right
Speaker 4: now in all these different labs. So I think you're
Speaker 4: pretty much right that all of the public data sets
Speaker 4: we have are pretty much used to TRAIN three or
Speaker 4: GPT five or whatever. If there is a really good
Speaker 4: website that should have been scraped and downloaded into the model,
Speaker 4: it should probably be used. But there apparently is a
Speaker 4: much larger amount of private data than public data. I mean,
Speaker 4: you all work for Bloomberg, so you're probably intimately familiar
Speaker 4: with this. But if you think about the different AI
Speaker 4: labs that exist, they actually now do kind of have
Speaker 4: different data related modes. Like XAI, they have all of
Speaker 4: the Twitter data that's basically impossible to get elsewhere. CHADGBT
Speaker 4: now has all of the user conversations with CHATGBT, which
Speaker 4: are really useful. Claude has a ton of coding data
Speaker 4: that other people don't have. Google has YouTube, which some
Speaker 4: people think might be like the next source of making
Speaker 4: really good models, and none of those things are really included,
Speaker 4: at least not much in today's models.
Speaker 3: This is really important.
Speaker 1: Like once a lab builds some sort of base, whether
Speaker 1: it's anthropic encoding or maybe cursor encoding, even though they're
Speaker 1: not like a core lab, et cetera, like they become
Speaker 1: a source of their own data that literally nobody else has.
Speaker 4: Yeah, actually Cursor is a great example. So they are
Speaker 4: very technical, they have really smart people. They're very small,
Speaker 4: so they haven't quite scaled to at least in terms
Speaker 4: of the number of people.
Speaker 1: But I think about this like every time I was like,
Speaker 1: when I've played with this is like this is good,
Speaker 1: this is bad. I'm constantly teaching their model to get better,
Speaker 1: right right, right right.
Speaker 4: They're in a problem where they have the data. They
Speaker 4: just have to take the right algorithms and scale it
Speaker 4: up to train the model to be as good as
Speaker 4: Claude is. But that actually seems a lot more feasible
Speaker 4: than other companies that have no data and want to
Speaker 4: train good models, even if they know how, it seems
Speaker 4: very difficult.
Speaker 2: How closely are AI researchers working or talking to I
Speaker 2: guess other parts of the AI ecosystem, so you know,
Speaker 2: chip makers, maybe cloud providers, that sort of thing. Is
Speaker 2: there a lot of dialogue or not really.
Speaker 4: I think certain people talk all the time to the
Speaker 4: chip makers, Like there's a big community of people. You know,
Speaker 4: the AI models all run on GPUs, and there are
Speaker 4: a lot of people that are getting really good at
Speaker 4: writing fast GPU code. It's called kernels, and all those
Speaker 4: people who work on kernels talk to the chip makers
Speaker 4: all the time. Like Amazon's making their own chips, Google
Speaker 4: has their own chip. Now all the hyperscalers are making chips,
Speaker 4: and I think they're all trying to talk to the
Speaker 4: people that actually write the fast code that runs on
Speaker 4: chips to figure out I think they call it hardware
Speaker 4: software code design, Like everyone's kind of getting together and
Speaker 4: trying to figure out what the best way is to
Speaker 4: design the next round of GPUs.
Speaker 1: So you mentioned, okay, Google might have an advantage because
Speaker 1: it owns YouTube and there's just tons of obviously just
Speaker 1: tons of.
Speaker 3: Data in there.
Speaker 1: So one way you could get access to the YouTube
Speaker 1: data is to literally be Google and own it. But
Speaker 1: another way that maybe you could get access to YouTube
Speaker 1: data is operate in China where there are no laws
Speaker 1: about this type of thing, or no, they're not beholding
Speaker 1: the US copyright and just sort of scrape at all. Again,
Speaker 1: since most of the Chinese AI labs are open to source,
Speaker 1: why isn't this just a huge advantage for the Chinese
Speaker 1: labs that they're really not going to be Hey, open
Speaker 1: AI they get super at the New York Times all
Speaker 1: these deepseek isn't having to deal with all these headaches?
Speaker 4: Yeah, I think the American AI labs will probably do
Speaker 4: things behind the scenes that they wouldn't tell you about
Speaker 4: to get good data solution. Just don't so Yeah, Like
Speaker 4: I think they wouldn't release the models that are potentially
Speaker 4: trained on scraped or copyrighted data. But if that's the
Speaker 4: way to get better math Olympiad scores, then people will
Speaker 4: I think I would guess do that. But you're right
Speaker 4: that like the Chinese, the Chinese model makers can to
Speaker 4: sort of take all the books that they can pirate
Speaker 4: from the Internet and train on them and they're not
Speaker 4: violating any laws and they can release the model to
Speaker 4: the public and it's all fine, which is honestly great
Speaker 4: for us because then people like me could probably download
Speaker 4: a model that's better than we would get otherwise.
Speaker 2: What was your impression of deep Seek when it came out?
Speaker 2: And now?
Speaker 4: I was pretty surprised at how much of a splash
Speaker 4: they made. The model is really good, and I think
Speaker 4: a lot of people are building on it, including me,
Speaker 4: and like most people that are at AI companies that
Speaker 4: aren't super super big are building on deep Seek. But
Speaker 4: it was surprising, like what a huge deal it was
Speaker 4: to people, like my mom's asking me about deep Seek.
Speaker 4: I think my grandma knew about deep Seek, and she
Speaker 4: barely knew about chat GBT.
Speaker 2: So that's when you know it's gone mainstream when starts.
Speaker 4: Asking you and there was nothing else so far. I
Speaker 4: think in the AI space that's made quite that much news.
Speaker 1: But it sounds like what you're saying is that it's
Speaker 1: a very good model, but that on the actual specs
Speaker 1: from your perspective, it didn't quite deserve is much attention,
Speaker 1: Like it was good, but like in your view, it's
Speaker 1: not so good that everyone needed to be talking about it.
Speaker 4: Yeah, I think it's really useful because they released all
Speaker 4: their model weights and they said exactly what they did
Speaker 4: to train it. Although they didn't say what the data was,
Speaker 4: but it gave me the impression of there maybe six
Speaker 4: to twelve months behind the American AI labs in terms
Speaker 4: of how well they can do the training and stuff.
Speaker 4: But it still was a pretty big update for me
Speaker 4: to know that, Wow, there are one hundred people that
Speaker 4: don't have PhDs working at a Chinese hedge fund that
Speaker 4: are training these like cutting edge models. Like it is
Speaker 4: incredible and they work very hard, they're very good.
Speaker 2: Do you have pressure or do you feel pressure or
Speaker 2: do AI researchers in general fuel pressure to consider monetization
Speaker 2: when they're researching things or is it you know, mostly
Speaker 2: still curiosity driven, that sort of old school Silicon Valley
Speaker 2: we're improving the world kind of thing. Or is it
Speaker 2: much more mercenary given that all of these big companies
Speaker 2: seem to be competing in the same space.
Speaker 4: Yeah. I think that over time it's gotten harder and
Speaker 4: harder to do things that are just like cool ideas
Speaker 4: or seem cute but don't have any necessary application, and
Speaker 4: things are getting closer and closer to products, you know,
Speaker 4: even like the language models that power CHAGBT. I was
Speaker 4: working in those before CHAGBT, and they had some uses,
Speaker 4: but also they're intellectually interesting and like fun to build.
Speaker 4: But now if I came up with a better way
Speaker 4: to train CHAGBT, that's like a multi billion dollar innovation.
Speaker 2: The stakes are higher.
Speaker 4: Yeah, I'd be like an asset to like the United
Speaker 4: States government or something if I knew how to do that.
Speaker 4: So I guess it depends on what kind of problems
Speaker 4: you work on. Like, I'm more interested in understanding how
Speaker 4: things work, so it becomes a bit less financially dire.
Speaker 1: I think that six to twelve month gap between what
Speaker 1: was that that was a January deep segment. Yeah, everyone
Speaker 1: was in December that they first got at attention, then
Speaker 1: for some reason really hit in January. Is that a
Speaker 1: sustainable gap? Is there something either in access to data,
Speaker 1: access to talent, access to compute, access to chips, whatever,
Speaker 1: access to energy that in your view will allow us
Speaker 1: frontier lebs to maintain some sort of six to twelve
Speaker 1: month gap for a while.
Speaker 4: It's pretty unclear to me. I think there are different
Speaker 4: beliefs you can have. You can believe that the ideas
Speaker 4: and the people are really the thing that differentiates the models,
Speaker 4: and in that case, I think we haven't so far
Speaker 4: seen a lot of like the top USAI researchers going
Speaker 4: to work at Chinese labs, so that seems stable. You
Speaker 4: could think that chips really matter, and in that case
Speaker 4: the chip race is really happening between big American companies.
Speaker 4: Like I think, actually China has a pretty big deficit
Speaker 4: coming up in terms of like the GPUs we're exporting,
Speaker 4: or you can think that the data matters, and I
Speaker 4: guess actually any of these point in the favor of
Speaker 4: the US. I think if you think the data really matters,
Speaker 4: maybe the data they gather through like deepseek dot com
Speaker 4: usage is really good and they can use it to
Speaker 4: like bootstrap a better model. But I think the American
Speaker 4: companies really do have an advantage. Like you all might
Speaker 4: have heard this story just as an anecdote. Apparently at
Speaker 4: Anthropic they've been buying and scanning thousands of old books
Speaker 4: for several years, so they have this division. I think
Speaker 4: they're based in New York that buys like shipping containers
Speaker 4: full of old manuscripts, cuts off the spines and puts
Speaker 4: them in these scanning machines and then they turn them
Speaker 4: into like really high quality text. And so I'm noting
Speaker 4: Claude has this like weird aspect to it. Maybe part
Speaker 4: of the reason is they've gathered like trillions of words
Speaker 4: worth of like old book data over many years, and
Speaker 4: that's pretty hard to replicate elsewhere. So I think that
Speaker 4: head start really does mean a lot.
Speaker 2: What are you most excited about at the moment? The
Speaker 2: book thing sounds very cool, but what is getting all
Speaker 2: your attention right now?
Speaker 4: Thanks for asking. I think I mentioned before I'm really
Speaker 4: trying to figure out what's coming next. There are some
Speaker 4: obvious things like we can get computer usage data and
Speaker 4: train better agents, or we can get more coding data
Speaker 4: and make them better coding or writing gp code or whatever,
Speaker 4: But like, what are the non obvious advancements? And my
Speaker 4: personal opinion is that the next round of improvements and
Speaker 4: AI models will come from some type of personalization and
Speaker 4: online learning, which means like models that one are trained
Speaker 4: like per person or per company. So like you could
Speaker 4: think of like CHADGBT is the same model that gets
Speaker 4: served to everyone, so it has to store information about
Speaker 4: random restaurants and like countries you never go to. But
Speaker 4: instead if you had a CHAGBT that's specific to Bloomberg
Speaker 4: or specific to your work, it might be able to
Speaker 4: like use more of its brain to do work for you.
Speaker 4: And then the second thing is if it was updated
Speaker 4: every day, so like if you ask it to make
Speaker 4: your odd lots calendar, yeah, or RSS feed and you're like, no,
Speaker 4: that was wrong, Like you did it wrong for this
Speaker 4: reason this reason, and you try again tomorrow, it'll still
Speaker 4: break tomorrow because it doesn't like continuously improve its capabilities.
Speaker 4: So oh yeah, I think that's the direction things are going.
Speaker 3: I've heard people talk about this now.
Speaker 1: Granted, models are getting better over time, but you know,
Speaker 1: people might compare a coding model to a beginning software
Speaker 1: engineer and say, the coding model is better, but that
Speaker 1: software engineer is going to start getting better the next
Speaker 1: day they're on the job, and every day for the
Speaker 1: rest of their career, they're probably going to be a
Speaker 1: better software engineer than they were the day before, whereas
Speaker 1: at least that version of the model will not be better.
Speaker 3: That is that right? Yeah?
Speaker 1: Yeah, that seems like an issue that people talk about
Speaker 1: in your world.
Speaker 4: Yeah, yeah, I think this is a big problem. It's
Speaker 4: like we have to wait six months for the chat
Speaker 4: GPT five point one to come out, and then maybe
Speaker 4: they'll include your problems as the training data, and so
Speaker 4: maybe it'll get better, but it might not. And instead,
Speaker 4: I think people need to think about ways to do
Speaker 4: that update more dynamically, like every time you talk to it,
Speaker 4: or maybe every night when you go to sleep, the
Speaker 4: model kind of like gets to work and studies what
Speaker 4: it was talking to you about and crafts better tests
Speaker 4: for itself and then learns and then when you wake up,
Speaker 4: the model's actually better.
Speaker 1: The other big question that I have and is kind
Speaker 1: of related to this, especially when we're talking about AI
Speaker 1: replacing the humans in certain forms of labor, is that
Speaker 1: like do we need really really advanced aid like in
Speaker 1: other words, like there is a lot of again, the
Speaker 1: existing models are extremely impressive, Like in your view, do
Speaker 1: we need to get a lot better technically for them
Speaker 1: to have economic impact? And since these are in many
Speaker 1: cases businesses at the end of the day, is it
Speaker 1: necessary that there's so much work being done towards advancing
Speaker 1: the cutting edge?
Speaker 4: Yeah, yeah, that's a great question, Like we could have
Speaker 4: really good interns without ever getting better scores on the
Speaker 4: Math Olympiad, Like that's not necessarily something that we ever
Speaker 4: had to go after. I think part of the reason
Speaker 4: for that is that AI labs are engaged in this
Speaker 4: kind of neck and neck race to have the smartest model.
Speaker 4: But I totally agree that AI could be economically transformative
Speaker 4: without having a higher ceiling in terms of what it
Speaker 4: can do. It's more like it needs to be more
Speaker 4: consistent or like dependable than actually smarter.
Speaker 2: This might be a weird question, but once you've made
Speaker 2: a sort of foundational improvement to a particular model, how
Speaker 2: easy or difficult is it to rewind if you need to.
Speaker 2: And one of the reasons I ask is because you know,
Speaker 2: some people have been complaining that they've been training chat
Speaker 2: GPT to I don't know, be their boyfriend or whatever,
Speaker 2: be their therapist topic. Yeah, and then it gets upgraded
Speaker 2: and all of that training suddenly disappears and the personality
Speaker 2: of the model changes.
Speaker 4: Yeah, that was a really interesting story. So I think
Speaker 4: the model before GPT five was four to zero. And
Speaker 4: they said that they thought internally, like all the scientists
Speaker 4: encoder people, that the new model was superior in every way.
Speaker 4: It gives you shorter responses, it's a bit nicer, it's
Speaker 4: much smarter. And then people got really upset because they
Speaker 4: had spent so much time talking to the old model
Speaker 4: that they felt like they'd experience like a serious loss
Speaker 4: in their life.
Speaker 2: Joe would miss the love it love it No.
Speaker 1: But for real, this is un Ironically this strikes me
Speaker 1: as another example for open source, which is that if
Speaker 1: I'm going to form a I don't see it. I'm
Speaker 1: forty five, I'm too old for that. But if someone
Speaker 1: is going to form like some sort of friendship with
Speaker 1: an AI model, I don't want it to be at
Speaker 1: the whim of Sam Altman deciding it was like, oh
Speaker 1: there's an upgrade. I would like to be friends, so
Speaker 1: weird to be friends with the model that I know
Speaker 1: that I can run in perpetuity and it will never change.
Speaker 4: Yeah. I think that's definitely a good argument for why
Speaker 4: open source is important, And if you ever fall in
Speaker 4: love with a model, you should fall in love with
Speaker 4: an openness.
Speaker 2: That's good life advice, practical life.
Speaker 3: Advice, really good life advice.
Speaker 2: Well, speaking of open source, you know, I know programmers
Speaker 2: tend to like open source for obvious reasons, but are
Speaker 2: there any downsides to open source for AI specifically?
Speaker 4: I think if you're running a company, there are a
Speaker 4: lot of downsides potentially to open source. If you have
Speaker 4: some brand new, fancy way of doing computation inside the
Speaker 4: model that's actually better, you might want to keep that
Speaker 4: information to yourself. And when you release the model, to
Speaker 4: make it runnable, you have to release all the code
Speaker 4: to run the model, which might contain like your secrets,
Speaker 4: and so I think that's why people are hesitant to
Speaker 4: do it. The other reason is because when you release
Speaker 4: the model, it actually contains quite a lot of residual
Speaker 4: information about how you actually trained it, Like you might
Speaker 4: be able to infer what the data set was and
Speaker 4: what the training process was, or even reconstruct the entire
Speaker 4: training data set given just the weights of the model.
Speaker 4: And so if you're worried about people finding out that
Speaker 4: a certain thing was in your training data, you probably
Speaker 4: can't release that model open source.
Speaker 2: That reminds me how much of an AI researcher's day
Speaker 2: to day life is just like looking at other model,
Speaker 2: other people's models, and trying to, like I guess, pull
Speaker 2: them apart and figure out how they were made and
Speaker 2: sort of work backwards.
Speaker 4: That definitely happens from time to time. I think usually
Speaker 4: the scientific process is something like you start with other
Speaker 4: people's models, and you run them and you see what happens,
Speaker 4: and then you decide on some part of that process
Speaker 4: that you think could be improved or could be explored further,
Speaker 4: and you make some tiny changes to it, and then
Speaker 4: you run it again and you compare like numbers, or
Speaker 4: you make graphs of what happened before and what happens after.
Speaker 4: So actually quite a bit of it, like, for example,
Speaker 4: pull the GPT two model from open Ai, which was
Speaker 4: twenty nineteen or something, their first kind of really larger
Speaker 4: scale chatbot. Like I've spent hundreds of hours kind of
Speaker 4: like playing with that code and talking to the model
Speaker 4: and stuff like that. So thank goodness for open source.
Speaker 1: For that reason, I joked in the beginning about you
Speaker 1: having one hundred million dollar salary, but for real, as
Speaker 1: you think about your career, and I hope you do
Speaker 1: get a hundred million dollar salary, but as you think
Speaker 1: about your career, what excites you?
Speaker 3: And how much is it money?
Speaker 1: But the reason I think about this is like they're
Speaker 1: huge checks out there, but maybe some things are more.
Speaker 1: Maybe achieving AGI is more excited than making an ad
Speaker 1: network more efficient. Maybe something there's something more exciting than
Speaker 1: shaving off a billionth of a second in terms of
Speaker 1: a trade execution, all these things like how much is
Speaker 1: it about exploring the frontiers of science, the new space race,
Speaker 1: landing on the Moon versus the paycheck?
Speaker 4: It's all about the paycheck. I'm just kidding. No, no,
Speaker 4: not at all. Yeah, it's funny you ask. So this
Speaker 4: hasn't happened to me, But just in the past two
Speaker 4: weeks or so, a good friend of mine has been
Speaker 4: dealing with this problem because she got an offer on
Speaker 4: the order of like tens of millions of dollars per
Speaker 4: year from a big AI company and she wasn't sure
Speaker 4: if she wanted to work there, and I think originally
Speaker 4: she said no, and then they doubled her offer, and
Speaker 4: then like it's the exact same amount of cash, but
Speaker 4: twice as much per year for certain number of years.
Speaker 4: And you know, we were talking amongst ourselves like what
Speaker 4: does this even mean at this point, Like you're, you know,
Speaker 4: a twenty eight year old computer scientist that's been coming
Speaker 4: from a PhD. So you make more on the order
Speaker 4: of tens of thousands of dollars per year. I honestly
Speaker 4: think personally, the marginal difference between having like ten and
Speaker 4: twenty million dollars is like very low, Like I don't
Speaker 4: even know what I would do with this.
Speaker 1: Is this is my experience for me making ten million
Speaker 1: twenty mine has basically.
Speaker 3: Been the same to me.
Speaker 4: Yeah, congratulations, but so yeah, I think there's more of
Speaker 4: a desire to like be there the next time something
Speaker 4: really interesting happens, and that kind of supersedes the money.
Speaker 4: Like any of these places will pay you what's like
Speaker 4: a really good salary to live on, and so it's
Speaker 4: actually not a big consideration. It only becomes complicated when
Speaker 4: you have like one option that's going to pay you
Speaker 4: like forty times more than the other option, and then
Speaker 4: things get confusing.
Speaker 2: No, this isn't this should actually I was just thinking
Speaker 2: about making twenty million.
Speaker 3: No, I think.
Speaker 1: Because I think about, Okay, what if you have this
Speaker 1: great salary and you're like can live very easily in
Speaker 1: New York City and have a really great life, or
Speaker 1: you could make ten times that, which is a stupid
Speaker 1: insane salary, right, but you don't write like your job.
Speaker 3: But it's so.
Speaker 1: Much money that strikes me is like not a trivial life.
Speaker 1: You only live one time. There's like a different so
Speaker 1: it could be a difficult question.
Speaker 4: Yeah, yeah, but you can remind yourself that, like the
Speaker 4: job you take once isn't the job that defines you forever.
Speaker 4: Maybe maybe the right thing to do is to take
Speaker 4: it for a few years but not the whole time,
Speaker 4: and then go do something.
Speaker 1: Everyone says they're going to do that and then.
Speaker 2: They get locked in. Speaking of insanely large salaries, we
Speaker 2: know that people are earning these salaries because they're like
Speaker 2: star AI researchers. How much does personality play into where
Speaker 2: you want to go work? Would you want to go
Speaker 2: work somewhere specifically because there's an absolutely amazing researcher, or
Speaker 2: does it tend to be again more about the paycheck,
Speaker 2: maybe more about the data that's available to you, or
Speaker 2: maybe more about the specific project that you're going to
Speaker 2: be working on.
Speaker 4: Yeah, I think different people assign different amounts of weight
Speaker 4: to each of those things. In my experience, like most
Speaker 4: of the people I know come from academia, which means
Speaker 4: they already kind of gave up more of a salary
Speaker 4: to do study things more deeply for several years. So
Speaker 4: I think people that I know are more biased against money.
Speaker 4: But like people do care about that. But I think
Speaker 4: that the ego thing really matters. Some people want to
Speaker 4: feel like they're very important and they're working on a
Speaker 4: problem that matters. One way some companies are able to
Speaker 4: pull researchers away from other companies is by saying, we'll
Speaker 4: sign you more importance in your role and we'll give.
Speaker 2: You we'll give you a really big title.
Speaker 4: Yeah, exactly. Seriously, the title is like, Okay, maybe before
Speaker 4: you were like a researcher or not. You get to
Speaker 4: be like a head researcher. You get to have people
Speaker 4: under you, or you're a chief scientist, and all these
Speaker 4: things do matter to people.
Speaker 3: It's a very good book about it.
Speaker 1: Pursuing a mission in the realm of like a driven
Speaker 1: visionary even when it's commercially.
Speaker 2: Just say it, just say yeah, that's right. No.
Speaker 1: I think about this all the time. Do you want
Speaker 1: to work for Ilia or do you want to work
Speaker 1: for Sam? And which one is the ahab and which
Speaker 1: one is just trying to make an honest living selling ads.
Speaker 1: I find this to be like a genuinely interesting, interesting
Speaker 1: question for any individual to have to reckon with in
Speaker 1: this career.
Speaker 4: Oh. Absolutely, And sometimes it can be.
Speaker 1: Very difficult to tell Jack Morris, thank you so much
Speaker 1: for coming on. Please pursue a career that will allow
Speaker 1: you to come back on a log.
Speaker 2: Or insert the odd lots close when you're negotiating your
Speaker 2: one hundred million dollar salary, or.
Speaker 1: Take the fifty so you know what, fifty million, but
Speaker 1: let me I don't need one hundred million, fifty million.
Speaker 3: But keep the album.
Speaker 4: Yeah, that would be fine with me.
Speaker 3: All right, great, Well, thank you so much.
Speaker 2: Yeah, thanks, thank you so much.
Speaker 4: That was great.
Speaker 1: Appreciate I think about that sometimes, like what if you
Speaker 1: got like an insane salary like that, you just could
Speaker 1: you would be insane to say no to But like
Speaker 1: I don't know, that's I mean.
Speaker 3: It's not our problem, but like, wouldn't it be fun?
Speaker 1: You know? It's like, oh, but you're gonna be working
Speaker 1: on ad optimization or whatever and you're not going to
Speaker 1: be there when they land.
Speaker 3: On the moon. But you got paid ten times.
Speaker 1: More than the people at the Bay station working on
Speaker 1: landing on the moon. That strins me as a kind
Speaker 1: of a tough life choice.
Speaker 2: I think you're using up a lot of brain power
Speaker 2: and energy on a problem which will Jem said is
Speaker 2: not you.
Speaker 3: That's exactly right.
Speaker 2: No, that conversation was really fun. Nice to talk to
Speaker 2: an actual researcher just doing stuff in the space. One
Speaker 2: thing I thought was very interesting was this idea that
Speaker 2: everyone gets excited about a specific improvement in AI, and
Speaker 2: then it seems like that particular one doesn't materialize and
Speaker 2: instead something else emerges, as like the big breakthrough. So
Speaker 2: instead of agents, we have math.
Speaker 1: And math which none of us will ever. I would
Speaker 1: really like for an agent to do something simple. I'm
Speaker 1: going to a city book on the trip or whatever.
Speaker 1: Or change my flight. Oh my god, I tried to.
Speaker 2: That would be amazing.
Speaker 1: Recently change my flight. Here's my information. I don't I
Speaker 1: would like that. I do not need the math olympiad.
Speaker 1: I am very impressed.
Speaker 3: I don't need it.
Speaker 2: Also, I am now very very intrigued by reinforced learning
Speaker 2: and how you actually reward the computers for doing good stuff.
Speaker 2: I feel like, actually that would be a really interesting
Speaker 2: area to mine. Which is motivating motivating the models to
Speaker 2: do better?
Speaker 1: Yeah, I've thought about that, like in chess, like how
Speaker 1: do how do the computers know.
Speaker 3: They want to win?
Speaker 4: Yeah?
Speaker 3: You know, like why do they care?
Speaker 2: You know, all they're saying anyway, why are they here?
Speaker 2: Why are we here?
Speaker 3: That's the thing with AI conversations.
Speaker 2: That's existential fact, something.
Speaker 1: We didn't talk about, which I am interested. No one
Speaker 1: really talks about AI safety anymore. If you notice, like
Speaker 1: they like very little, like for better or worse. You
Speaker 1: don't hear people just all money and they don't really
Speaker 1: talk about what the AI kill us all one day.
Speaker 3: But one thing I did wonder about.
Speaker 1: So when Deep Seat came out, one of its breakthroughs
Speaker 1: was it showed the whole chain of thought, right, you
Speaker 1: could see that, which prior to that open AI or
Speaker 1: chatchybt's chain of thought model didn't show you.
Speaker 4: That, right.
Speaker 1: And it does strike me that if there are certain
Speaker 1: things that are for safety reasons or whatever held back
Speaker 1: or they don't want to do this, the nature of
Speaker 1: competition means all the guardrails are coming off of Actually,
Speaker 1: like that's if there's some guardrail you you have on
Speaker 1: someone's going to open source whatever it is and they're
Speaker 1: going to all give it up.
Speaker 2: Yeah, both on the guardrails and on the data use ys.
Speaker 2: All right, well shall we leave it there.
Speaker 3: Let's leave it there.
Speaker 2: This has been another episode of the aud Loots podcast.
Speaker 2: I'm Tracy Alloway. You can follow me at Tracy Alloway.
Speaker 1: And I'm Jill Wisenthal. You can follow me at the Stalwart.
Speaker 1: Follow our guest Jack Morris, He's at j xmnop. Follow
Speaker 1: our producers Kerman Rodriguez at Kerman armand dash O Bennett
Speaker 1: at Dashbod and kil Brooks at Kilbrooks. More odd Loss content,
Speaker 1: go to Bloomberg dot com slash od Lots with the
Speaker 1: daily newsletter and all of our episodes, and you can
Speaker 1: chat about all of these topics twenty four to seven
Speaker 1: in our discord Discord dot gg slash.
Speaker 2: Odd Lots And if you enjoy odd Lots, if you
Speaker 2: like it when we talk about twenty million dollars salaries
Speaker 2: that will never be ours, then please leave us a
Speaker 2: positive review on your favorite podcast platform. And remember, if
Speaker 2: you are a Bloomberg subscriber, you can listen to all
Speaker 2: of our episodes absolutely ad free. All you need to
Speaker 2: do is find the Bloomberg channel on Apple Podcasts and
Speaker 2: follow the instructions there. Thanks for listening it