Speaker 1: Already and this is the Daily This is the Daily
Speaker 1: ohs oh, now it makes sense.
Speaker 2: Good morning, and welcome to the Daily OS. It's Thursday,
Speaker 2: the fourteenth of August. I'm Sam Kazlowski.
Speaker 1: I'm Emma Gillespie.
Speaker 2: This month, the tech company behind chat GBT, released what
Speaker 2: they claim is their smartest AI model yet now. According
Speaker 2: to Open Ai, GPT five operates at the level of
Speaker 2: a PhD student. But experts are warning that the AI
Speaker 2: race has become a bit of a marketing battle, as
Speaker 2: companies manipulate test results to claim their product is the best.
Speaker 2: On today's podcast, we're going to unpack how AI companies
Speaker 2: measure intelligence and why that's become a problem.
Speaker 3: Sam.
Speaker 1: I was originally skeptical about having this conversation with you
Speaker 1: because I, like maybe some listeners here AI and I
Speaker 1: kind of roll my eyes a little.
Speaker 3: Bit and switch off.
Speaker 1: But if you are that person hearing this right now,
Speaker 1: hang in there, because this is actually a fascinating conversation,
Speaker 1: this idea that we're sort of being marketed to about
Speaker 1: this arms race of who is the smartest, which AI model.
Speaker 3: Is the best.
Speaker 1: Let's start with the basics here, though, when we're talking
Speaker 1: about AI models.
Speaker 3: What exactly does that mean?
Speaker 2: There is a certain brand of satisfaction that is reserved
Speaker 2: for when I can change your mind and whether a
Speaker 2: story is going to be interesting or.
Speaker 3: Not, especially if it's a tech story.
Speaker 2: This is this is going to be awesome. So AI
Speaker 2: models are computer programs that can understand and generate language
Speaker 2: human language. Just think of them as very advanced AU
Speaker 2: though complete systems like the ones that could fill in
Speaker 2: a form for you or you know, password, remembering little
Speaker 2: widgets in your browser, anything that presumes that what you're
Speaker 2: about to do or want it can kind of fill
Speaker 2: in those gaps for you.
Speaker 3: That's actually a really good way to think of it.
Speaker 2: See we're off to a flyer. You type in a
Speaker 2: question or a quest, those responses are generated. The most
Speaker 2: famous ones you might have heard of include chat GBT
Speaker 2: from open Ai, You've Got Clawed from Anthropic, and Gemini
Speaker 2: from Google.
Speaker 1: Okay, now, it does seem like every AI company out
Speaker 1: there claims that its model is the smartest or the
Speaker 1: most capable or better than the best one that we've
Speaker 1: ever seen. And one of the biggest players in this space,
Speaker 1: open ai has just released GPT five this week. What
Speaker 1: are their claims about this new model.
Speaker 2: So they're making some big statements here. They're saying that
Speaker 2: GBT five scored ninety four point six percent on a
Speaker 2: test that measures its ability to solve advanced maths problems,
Speaker 2: seventy four point nine percent on real world coding tasks,
Speaker 2: and produces forty five percent fewer factual errors than their
Speaker 2: previous models. To the CEO of the company, Sam Oltman,
Speaker 2: he called it the model in the world, which kind
Speaker 2: of sounds like those places you were saying before, and
Speaker 2: said it represents a significant step towards what's called artificial
Speaker 2: general intelligence AGI, which is basically the idea that AI
Speaker 2: can actually perform an intellectual task better than humans can.
Speaker 1: Okay, so that's when we start to imagine like the
Speaker 1: I robot future.
Speaker 2: Yeah, and it's when we get into those examples of
Speaker 2: things like AI blackmailing you if you decide to stop
Speaker 2: using it and kind of taking on a life of
Speaker 2: its own.
Speaker 1: So those numbers from Open AI about this new model
Speaker 1: sound pretty impressive, like ninety five percent on advanced maths.
Speaker 1: Particularly interesting this kind of idea of producing fewer factual errors,
Speaker 1: because that's always kind of in the spotlight around the
Speaker 1: skepticism towards AI, But I'm interested in how these companies
Speaker 1: are actually measuring the intelligence of these products. You mentioned
Speaker 1: in the intro SAM that this is becoming a bit
Speaker 1: of an issue. Yeah, So what exactly is the concern?
Speaker 2: Well, ultimately it's the idea that AI come companies are
Speaker 2: all using different tests to prove that their model is
Speaker 2: the best. It's like if all car companies all claims
Speaker 2: to make the fastest car ever or the safest car ever,
Speaker 2: but one tested on a highway, the other tested on
Speaker 2: a racetrack, and the other one went downhill on a
Speaker 2: windy day. A major study published earlier this year into
Speaker 2: AI models actually compared the situation to Volkswagen, who were
Speaker 2: found guilty of lying about the emissions or the lack
Speaker 2: of emissions that their cars were producing when it basically
Speaker 2: cheated on pollution tests. The researchers noted that when companies
Speaker 2: manipulated car testing, people were going to jail, but similar
Speaker 2: manipulation in AI isn't really coming into our attention.
Speaker 3: Wow, it's fascinating.
Speaker 1: I remember that Volkswagen emission scandal, So a good comparison,
Speaker 1: and how the tick for SAM? So, how can these
Speaker 1: AI models then be tested in a fair way.
Speaker 3: What does testing.
Speaker 1: Out official intelligence kind of transparently and consistently look like.
Speaker 2: Well, naturally, the first thing to do would be the
Speaker 2: standardize the same test across every model, and that would
Speaker 2: be described as a benchmark, and you global benchmark for
Speaker 2: how these models are performing. And that could be to
Speaker 2: measure a specific ability, say in maths, you could give
Speaker 2: all of them the same advanced maths problem and then
Speaker 2: measure not only the output, but how long it takes
Speaker 2: for them to get there, what processes it undertook to
Speaker 2: reach that final destination of the answer. You could give
Speaker 2: that a score and then actually compare like for like
Speaker 2: these models.
Speaker 3: It kind of sounds pretty straightforward.
Speaker 1: That to me seems like the obvious path towards getting
Speaker 1: consistent testing. So where does the manipulation come from?
Speaker 2: Well, I think the first thing to acknowledge is that
Speaker 2: there is no centralized global body that has the respect
Speaker 2: or the ability to actually execute that sort of standardized testing.
Speaker 2: There is no say, TGA for drugs, there's no government
Speaker 2: sponsored hub that can execute that kind of stuff. So
Speaker 2: reason A, there's nobody to do it. But reason B
Speaker 2: would be that these models are still in this accelerating
Speaker 2: period of marketing where they're cherry picking tests that would
Speaker 2: favor their models' strengths while hiding poor performance in other areas.
Speaker 2: And one other problem that has come up is that
Speaker 2: if AI knows the problem is coming, because it's AI
Speaker 2: and it knows how tests are done, then it can
Speaker 2: actually almost train itself for the test, and so there's
Speaker 2: a bit of a data contamination problem. You'd have to
Speaker 2: keep these tests almost offline entirely for the models to
Speaker 2: see them for the first time. One study found, for example,
Speaker 2: that GPT four, which is the one older model from
Speaker 2: open AI, it could solve coding problems from before twenty
Speaker 2: twenty one that were published online, but it couldn't solve
Speaker 2: new problems. And so then you get a sense of
Speaker 2: kind of in the great big world of its brain,
Speaker 2: which is the Internet, if those answers are somewhere out there,
Speaker 2: it could just regurgitate them.
Speaker 1: So it's like if you've got an advanced copy of
Speaker 1: an exam or a test at unior in school, you
Speaker 1: can train for the test. That doesn't necessarily mean that
Speaker 1: you have the comprehension levels to speak to a certain
Speaker 1: topic or question. In the same subject outside of the
Speaker 1: confines of that context.
Speaker 2: And if we think about what all of this is for,
Speaker 2: it's about trying to work out if these models are
Speaker 2: going to be good in practice for us to spend
Speaker 2: twenty bucks a month on them. I mean, let's get
Speaker 2: back to the real core problem here. We're trying to
Speaker 2: work out if it's worth our money. And there was
Speaker 2: a great quote from the British Prime Minister, former British
Speaker 2: Prime Minister Richie Sunak. He said AI models shouldn't be
Speaker 2: trusted to mark their own homework. And I think that
Speaker 2: we can all relate to that. Yeah, and it kind
Speaker 2: of encapsulates what's the problem with this independent benchmarking framework.
Speaker 1: You also mentioned that companies are testing multiple versions, or
Speaker 1: that they're cherry picking their data and choosing the kind
Speaker 1: of findings that favor their models the most.
Speaker 3: What's happening there.
Speaker 2: Tell us a bit more well. Some research found that
Speaker 2: major companies were talking mesha, Open Ai and Google have
Speaker 2: been privately testing dozens of different model versions on popular tests.
Speaker 2: They're only revealing the scores from their best performing versions.
Speaker 2: So and it's like, you know, you're on a night out,
Speaker 2: you take twenty selfies, you put up the best one. Yeah,
Speaker 2: of course, and I think at some stage you have
Speaker 2: to admit that all businesses would do that. Yeah, you know, TDA,
Speaker 2: if we had to report results to the stock market,
Speaker 2: you know, we would probably highlight more the pieces that
Speaker 2: did really, really well. Not that there's ever any pieces
Speaker 2: that don't, but you know.
Speaker 3: A flawless company that never makes mistakes.
Speaker 2: Obviously, but we have to. I think it's good to
Speaker 2: acknowledge this bit of kind of business reality there. But
Speaker 2: I do think that in this case it's different because
Speaker 2: there's no transparency at all in terms of the testing process.
Speaker 2: It's to continue with our university kind of example. It's
Speaker 2: like a student taking the same exam twenty seven times
Speaker 2: and then only reporting the best score. Yep.
Speaker 1: So without that transparency, there's that issue around trust, and
Speaker 1: I think we see that really playing out in real
Speaker 1: time right now, that there is a lack of trust
Speaker 1: in the broader community about AI models because we don't
Speaker 1: know how they come to these answers. What are some
Speaker 1: of the other consequences of this manipulation. How does this
Speaker 1: play out in the real world every day?
Speaker 2: Well, there's definitely that marketing angle of misleading consumers and
Speaker 2: you and I signing up to an AI platform because
Speaker 2: we think it's ninety six percent going to be great,
Speaker 2: and in fact it might be eighty one percent great,
Speaker 2: which is still an incredible feat of technology. But then
Speaker 2: from a government perspective, governments are looking at these benchmarks
Speaker 2: for the way that they're thinking about regulation or policy decisions.
Speaker 2: So the European Union's AI Act it uses benchmarks to
Speaker 2: determine whether new AI models pose systemic risk. Can they
Speaker 2: be used by extremists? Can they be used to spread
Speaker 2: race online? Can they be used to mislead and deliberately
Speaker 2: spread misinformation? And if companies are manipulating those scores, it
Speaker 2: could affect how these powerful technologies are indeed regulated.
Speaker 1: Okay, because if these scores say that eighty percent of
Speaker 1: the content is factual, or that there are these really
Speaker 1: great systems in place to catch miss and disinformation or
Speaker 1: hate speech, then that might not concern leaders to the
Speaker 1: point where they think there needs to be certain levels
Speaker 1: of regulation.
Speaker 2: One hundred percent.
Speaker 1: You mentioned this idea of artificial general intelligence earlier. We
Speaker 1: used the I robot example. One of Will Smith's best
Speaker 1: open AI is claiming that GPT five is a step
Speaker 1: forward in AGI, But what does that actually mean in
Speaker 1: a not Hollywood kind of fantasy world.
Speaker 2: Well, I gave the example before of outperforming humans. That's
Speaker 2: a very broad definition, and the problem is that I
Speaker 2: can't really give you a more specific definition because even
Speaker 2: open ai can't really do that right. One open Ai
Speaker 2: statement said, AGI is still a weekly defined term and
Speaker 2: means different things to different people. We don't really know
Speaker 2: what we don't know.
Speaker 3: So how can GPT five verse step forward?
Speaker 2: Then?
Speaker 3: If the company itself isn't sure?
Speaker 2: Interesting question very much raises some questions about how do
Speaker 2: we know when we got there? Even? Yeah, I mean
Speaker 2: this is the exciting and terrifying part of living through
Speaker 2: rapidly emerging technology is that we're learning as we go
Speaker 2: as a society, and that is not always pretty.
Speaker 1: So for people listening who might be using AI kind
Speaker 1: of casually or infrequently in their maybe work or UNI life,
Speaker 1: maybe they're building up their understanding of the different platforms
Speaker 1: out there. What should we make of all these competing claims?
Speaker 1: You know, how do we make better decisions about which
Speaker 1: AI model is actually the good one, or the right one,
Speaker 1: or or the best one for us?
Speaker 2: I'm constantly asked as somebody who is known now in
Speaker 2: my friend group and in the workplace as somebody who's
Speaker 2: really interested in AI. I'm constantly asked which one should
Speaker 2: I use, what's the best one, And the answer is,
Speaker 2: it's about what you're trying to do, essentially, So one
Speaker 2: model might be better for creative writing, but another might
Speaker 2: excel more a data analysis and crunching some numbers. Studies
Speaker 2: are showing though, that AI models often fail when you
Speaker 2: move from those controlled test conditions or those use cases
Speaker 2: or features that are rolled out by these platforms as
Speaker 2: part of marketing campaigns to the messy real world use
Speaker 2: that humans actually use these tools for.
Speaker 1: It actually reminds me of and I'm not even sure
Speaker 1: if this is the same thing, but when Siri was
Speaker 1: first rolled out and Apple kind of in their big
Speaker 1: announcements it's like, you can ask her this, or you
Speaker 1: can ask her that, or if you want to know what.
Speaker 3: The weather's like, should you take an umbrella?
Speaker 1: And I found when I first started using Siri, like, yeah,
Speaker 1: you could definitely answer those sorts of questions, but not
Speaker 1: a whole lot else outside of the almost like a
Speaker 1: prescribed text from Apple about how to use Siri.
Speaker 2: When you get into the world of trying to engage
Speaker 2: with the user no matter what they're about to say.
Speaker 2: It can take a little bit of time for the
Speaker 2: technology to be refined and to keep learning from what
Speaker 2: users actually want.
Speaker 3: So, Sam, what is the way forward in all of this?
Speaker 1: Is there a conversation happening at a more global scale
Speaker 1: about this regulation?
Speaker 2: Definitely, and there's no clear leader here. I mentioned the
Speaker 2: work being done by the European Union before. There's a
Speaker 2: coalition of countries including Australia that signed on to kind
Speaker 2: of key principles of how to keep AI safe. That
Speaker 2: was in mid twenty twenty three, so there's a bit
Speaker 2: of a global movement there. From a government perspective, there's
Speaker 2: some really interesting work being done out of universities, particularly
Speaker 2: Stanford University. They developed an AI Index report which does
Speaker 2: try to compare the models like for like. But I
Speaker 2: think we first need to determine who the authority is
Speaker 2: going to be in this space before we can kind
Speaker 2: of put the burden on them to roll out this
Speaker 2: standardized testing. And I do think in a few decades
Speaker 2: it will take a while. I do think we'll get there.
Speaker 2: I mean, we have the TGA to regulate medicine. We
Speaker 2: have a central aviation authority to regulate what a plane
Speaker 2: that's airworthy looks like. Yep. I do think that we're
Speaker 2: going to see a central AI authority in Australia and
Speaker 2: maybe around the world someday. But we are very early
Speaker 2: in this story. We're like one percent through in the
Speaker 2: AI story if that, and that's really exciting. But it's
Speaker 2: also really important to continuously discuss the potential flaws and
Speaker 2: the gaps that exist in this big, new scary Well.
Speaker 1: Yeah, I think that healthy dose of skepticism is what
Speaker 1: we will be carrying forward. But I look forward to
Speaker 1: many more conversations like this with you, Sam.
Speaker 2: Well, we don't have a choice.
Speaker 3: Help me understand it all.
Speaker 1: Thank you so much for breaking that down for us, Sam,
Speaker 1: and thank you for listening to today's deep Dive. We'll
Speaker 1: be back a little later on with your news headlines,
Speaker 1: but until then, have a great day.
Speaker 2: My name is Lily Maddon and I'm a proud Arunda
Speaker 2: Bunjelung Calkatin woman from Gadighl Country. The Daily oz acknowledges
Speaker 2: that this podcast is recorded on the lands of the
Speaker 2: Gadighl people and pays respect to all Aboriginal and torrest
Speaker 2: rate island and nations. We pay our respects to the
Speaker 2: first peoples of these countries, both past and present.