WEBVTT - Monologue: LLM Code Is Already Breaking Big Tech

0:00:02.800 --> 0:00:07.880
<v Speaker 1>Zone Media. Hello and welcome to this week's Better Offline Monologue.

0:00:07.920 --> 0:00:18.800
<v Speaker 1>I'm your host ed zetron. I heard something really worrying

0:00:18.880 --> 0:00:22.160
<v Speaker 1>the other day about a major hyperscaler. According to the source,

0:00:22.160 --> 0:00:26.040
<v Speaker 1>said hyperscaler was allowing and even encouraging non technical workers

0:00:26.040 --> 0:00:30.120
<v Speaker 1>to deploy code to consumer facing products, specifically those who

0:00:30.360 --> 0:00:33.520
<v Speaker 1>cannot read or write code vibe coding their own projects

0:00:33.800 --> 0:00:37.480
<v Speaker 1>using generative AI, with their code at some point theoretically

0:00:37.560 --> 0:00:40.040
<v Speaker 1>reviewed by an actual software engineer before it gets pushed

0:00:40.040 --> 0:00:43.199
<v Speaker 1>to production. The cold comfort of that review is that

0:00:43.280 --> 0:00:45.400
<v Speaker 1>it assumes that software engineers are at least the ones

0:00:45.440 --> 0:00:48.760
<v Speaker 1>reviewing that code are actually adept at code review, or

0:00:48.920 --> 0:00:50.880
<v Speaker 1>even if they are, that they have sufficient time to

0:00:50.920 --> 0:00:54.680
<v Speaker 1>look over the overlea verbose code that lllms spew. In

0:00:54.720 --> 0:00:58.040
<v Speaker 1>some cases, I've heard management is actively encouraging and even

0:00:58.120 --> 0:01:01.319
<v Speaker 1>mandating these non technical workers to use lms to make

0:01:01.400 --> 0:01:04.720
<v Speaker 1>these features, creating a mutation of tech debt where somebody

0:01:04.720 --> 0:01:07.560
<v Speaker 1>who cannot code uses a machine that doesn't think to

0:01:07.600 --> 0:01:11.280
<v Speaker 1>create code with no intention that nobody really understands, and

0:01:11.319 --> 0:01:13.320
<v Speaker 1>does so at such a velocity that it burdens the

0:01:13.360 --> 0:01:16.840
<v Speaker 1>actual technical workers with constantly having to monitor and fix it.

0:01:18.080 --> 0:01:21.640
<v Speaker 1>LLMS do not understand anything, nor do they think, which

0:01:21.680 --> 0:01:24.760
<v Speaker 1>means any solutions they build or theoretical bug reports that

0:01:24.800 --> 0:01:28.560
<v Speaker 1>they may make are immediately questionable. Their hallucinations are such

0:01:28.560 --> 0:01:30.559
<v Speaker 1>that even features you believe a part of your code,

0:01:30.640 --> 0:01:33.000
<v Speaker 1>after all you can't read it might not be there,

0:01:33.200 --> 0:01:35.360
<v Speaker 1>or might be poorly designed, or might have some sort

0:01:35.360 --> 0:01:37.679
<v Speaker 1>of unforeseen problem that neither you nor the LM are

0:01:37.720 --> 0:01:40.200
<v Speaker 1>aware of, because its training data is based on code

0:01:40.240 --> 0:01:44.800
<v Speaker 1>that already exists, versus any ability to solve novel problems. Also,

0:01:44.920 --> 0:01:47.720
<v Speaker 1>the idea of having any number of non technical peopleship

0:01:47.760 --> 0:01:51.320
<v Speaker 1>code is fucking insane, an indicative of an overwhelming ignorance

0:01:51.360 --> 0:01:55.080
<v Speaker 1>on the part of management. Of even a few years

0:01:55.120 --> 0:01:57.800
<v Speaker 1>of having overwhelming amounts of code written by LMS, even

0:01:57.800 --> 0:02:00.160
<v Speaker 1>by engineers who know what it says and have some

0:02:00.840 --> 0:02:03.360
<v Speaker 1>intention in the prompts they do, is going to create

0:02:03.360 --> 0:02:05.800
<v Speaker 1>a situation where most of the code is written without

0:02:05.840 --> 0:02:09.120
<v Speaker 1>any intention, making it much harder to the debug because

0:02:09.160 --> 0:02:11.200
<v Speaker 1>nobody really knows why it was written that way, because

0:02:11.680 --> 0:02:14.960
<v Speaker 1>LMS doesn't think. I know. I'm repeating myself already, but

0:02:15.040 --> 0:02:18.760
<v Speaker 1>this situation is chilling me, even outside of the vibe coding.

0:02:18.800 --> 0:02:21.480
<v Speaker 1>There's a larger problem of developers writing code with llms

0:02:21.480 --> 0:02:24.200
<v Speaker 1>that they barely review in some cases because they don't

0:02:24.200 --> 0:02:26.200
<v Speaker 1>feel they need to and just skim read it. In

0:02:26.280 --> 0:02:28.600
<v Speaker 1>many many more because their bosses are demanding their ship

0:02:28.639 --> 0:02:33.600
<v Speaker 1>features faster than is responsible or safe. Remember many many

0:02:33.680 --> 0:02:38.680
<v Speaker 1>tech companies are mandating llmuse harassing their workers checking how

0:02:38.760 --> 0:02:42.160
<v Speaker 1>much they use lllms. I've heard this from multiple companies,

0:02:42.520 --> 0:02:45.560
<v Speaker 1>and really it's not just for them, but it's especially

0:02:45.560 --> 0:02:49.040
<v Speaker 1>hard on software engineers. Adding a layer of code written

0:02:49.040 --> 0:02:51.600
<v Speaker 1>by people who quite literally do not understand what it

0:02:51.639 --> 0:02:56.239
<v Speaker 1>says or does guarantees future situations where major services simply break.

0:02:56.680 --> 0:02:59.520
<v Speaker 1>And the more of this nonsensical code that's allowed to

0:02:59.560 --> 0:03:01.919
<v Speaker 1>be stood up on these services, the harder it will

0:03:01.919 --> 0:03:05.360
<v Speaker 1>be to fix. Code isn't just something you write once

0:03:05.400 --> 0:03:08.440
<v Speaker 1>and leave forever. It needs to be maintained by other people.

0:03:08.720 --> 0:03:11.000
<v Speaker 1>Sometimes he is in the future, especially when people keep

0:03:11.040 --> 0:03:13.959
<v Speaker 1>being laid off, which becomes much harder when there's lots

0:03:14.000 --> 0:03:16.639
<v Speaker 1>of code to go through written by an LLM piloted

0:03:16.680 --> 0:03:19.280
<v Speaker 1>by somebody who doesn't know what they're actually doing and

0:03:19.320 --> 0:03:22.400
<v Speaker 1>relies on the LM, which doesn't know anything to tell

0:03:22.440 --> 0:03:27.639
<v Speaker 1>them what's going on. Again, I realize I'm repeating myself,

0:03:27.639 --> 0:03:29.919
<v Speaker 1>But llms do not have thoughts or feelings of knowledge.

0:03:29.919 --> 0:03:32.600
<v Speaker 1>They are generating based on the parameters of their training data,

0:03:32.760 --> 0:03:35.160
<v Speaker 1>which means that all of their code is at best

0:03:35.280 --> 0:03:38.480
<v Speaker 1>an abstraction of somebody else's back and forth with a chatbot.

0:03:39.280 --> 0:03:42.000
<v Speaker 1>The code is not written efficiently or with any consideration

0:03:42.040 --> 0:03:43.720
<v Speaker 1>of who might have to work with it in the future,

0:03:43.960 --> 0:03:47.560
<v Speaker 1>or indeed what other things might be involved. Lms only

0:03:47.600 --> 0:03:49.640
<v Speaker 1>know what they are fed or what they're connected to.

0:03:49.920 --> 0:03:52.520
<v Speaker 1>They don't know the nuances of code. They don't know

0:03:52.600 --> 0:03:56.200
<v Speaker 1>the nuances of software engineering or architecture, which means that

0:03:56.240 --> 0:03:59.880
<v Speaker 1>they only know so much based on their training data

0:04:00.040 --> 0:04:02.920
<v Speaker 1>and the environment around them. Kind of. They don't know

0:04:03.200 --> 0:04:07.000
<v Speaker 1>the nuances of how service let's say Meta or Microsoft's

0:04:07.080 --> 0:04:10.200
<v Speaker 1>has been built over decades, and indeed, the more of

0:04:10.200 --> 0:04:13.120
<v Speaker 1>this code that's used to build those services, the less

0:04:13.120 --> 0:04:16.600
<v Speaker 1>that these companies know about how their actual fucking software works.

0:04:17.680 --> 0:04:20.960
<v Speaker 1>This is setting up the software industry for disaster after disaster,

0:04:21.200 --> 0:04:23.960
<v Speaker 1>and it's already started to happen. To quote the information

0:04:24.000 --> 0:04:28.120
<v Speaker 1>from this week, according to internal Meta communications and an

0:04:28.120 --> 0:04:31.479
<v Speaker 1>incident report seen by the Information A major security alert

0:04:31.480 --> 0:04:34.240
<v Speaker 1>occurred last week after a meta software engineer used an

0:04:34.279 --> 0:04:37.359
<v Speaker 1>in house agent tool similar to openclaw to analyze the

0:04:37.360 --> 0:04:40.839
<v Speaker 1>technical question that another employee had posted on an internal

0:04:40.839 --> 0:04:44.640
<v Speaker 1>discussion forum. After doing the analysis, the AI agent posted

0:04:44.680 --> 0:04:47.000
<v Speaker 1>a response in the discussion forum to the original question,

0:04:47.320 --> 0:04:50.920
<v Speaker 1>offering advice on the technical issue. According to internal communications,

0:04:51.320 --> 0:04:54.120
<v Speaker 1>the agent did so without approval from the employee. It's

0:04:54.120 --> 0:04:57.320
<v Speaker 1>so cool that this is happening. It's so cool, it's great,

0:04:57.440 --> 0:05:02.000
<v Speaker 1>it's actually brilliant. How fucking insan what? And I'm I'm

0:05:02.160 --> 0:05:04.599
<v Speaker 1>kindly going to assume that the person using this knew

0:05:04.600 --> 0:05:07.400
<v Speaker 1>what they were doing. But the idea that we have

0:05:07.600 --> 0:05:09.480
<v Speaker 1>and I think this is what's happening with open source too.

0:05:09.680 --> 0:05:11.720
<v Speaker 1>We have people with llms who are like, yeah, well

0:05:11.720 --> 0:05:13.400
<v Speaker 1>the LM tells me I'm good, so I must be.

0:05:13.600 --> 0:05:16.160
<v Speaker 1>Let me just run this LM pasture problems. It's why

0:05:16.160 --> 0:05:19.159
<v Speaker 1>we're getting all these junk pull requests on GitHub on

0:05:19.240 --> 0:05:22.240
<v Speaker 1>open source projects. People that think they're competent because an

0:05:22.320 --> 0:05:26.200
<v Speaker 1>LM told them to are fucking up the entire software world.

0:05:26.400 --> 0:05:29.560
<v Speaker 1>And according to the Information, Meta systems storing large amounts

0:05:29.600 --> 0:05:32.640
<v Speaker 1>of company and user related data were accessible to engineers

0:05:32.680 --> 0:05:34.960
<v Speaker 1>who didn't have permission to see them, and this was

0:05:35.000 --> 0:05:37.800
<v Speaker 1>marked as second one incident, the second highest level of

0:05:37.839 --> 0:05:40.760
<v Speaker 1>severity on an internal scale that Meta uses to rank

0:05:40.800 --> 0:05:44.400
<v Speaker 1>security incidents. And again that's quoting the information. The incident

0:05:44.440 --> 0:05:48.480
<v Speaker 1>follows multiple problems caused to Amazon by its Kero and qllms.

0:05:48.839 --> 0:05:52.839
<v Speaker 1>I quote Business Inside is Eugene Kim. On March second,

0:05:53.160 --> 0:05:57.119
<v Speaker 1>customers across Amazon marketplaces saw incorrect delivery times when adding

0:05:57.160 --> 0:05:59.960
<v Speaker 1>items to their carts. The incident led to nearly one

0:06:00.240 --> 0:06:03.400
<v Speaker 1>hundred and twenty thousand lost orders and roughly one point

0:06:03.440 --> 0:06:08.000
<v Speaker 1>six million website errors. Amazon's AI tool Q was one

0:06:08.040 --> 0:06:10.800
<v Speaker 1>of the primary contributors that trigger the event, according to

0:06:10.839 --> 0:06:14.640
<v Speaker 1>an internal review. On March fifth, another outage caused a

0:06:14.800 --> 0:06:18.400
<v Speaker 1>ninety nine percent drop in orders across Amazon's North American

0:06:18.440 --> 0:06:22.200
<v Speaker 1>market places, resulting in six point three million lost orders.

0:06:22.440 --> 0:06:25.760
<v Speaker 1>One of the internal documents stated one key factor was

0:06:25.800 --> 0:06:28.719
<v Speaker 1>a production change that was deployed without using a formal

0:06:28.760 --> 0:06:34.240
<v Speaker 1>documentation on an approval process called Model Change Management. Very cool.

0:06:35.120 --> 0:06:36.919
<v Speaker 1>I also want to be clear that it appears that

0:06:36.960 --> 0:06:39.200
<v Speaker 1>these incidents were created by use of these tools. By

0:06:39.279 --> 0:06:42.520
<v Speaker 1>actual software engineers, people that ostensibly know how code and

0:06:42.560 --> 0:06:46.680
<v Speaker 1>software architecture works. Reliance on large language models, especially at

0:06:46.680 --> 0:06:49.400
<v Speaker 1>a time when executives are putting more pressure on engineers

0:06:49.400 --> 0:06:51.800
<v Speaker 1>to deliver more features and ship more code, means that

0:06:51.839 --> 0:06:55.080
<v Speaker 1>software engineers are being incentivized to be sloppy and to

0:06:55.120 --> 0:06:59.960
<v Speaker 1>ship slop itself. There is nothing inherently good about automating code,

0:07:00.320 --> 0:07:02.840
<v Speaker 1>nor is there any inherent value in shipping a lot

0:07:02.839 --> 0:07:06.120
<v Speaker 1>of it fast. Lms convince you that what you're writing

0:07:06.160 --> 0:07:08.039
<v Speaker 1>is good and stable and does the thing you want

0:07:08.080 --> 0:07:10.320
<v Speaker 1>it to. And if your skim reading the outputs, or

0:07:10.400 --> 0:07:12.520
<v Speaker 1>of course unable to read them at all, it's easy

0:07:12.560 --> 0:07:14.480
<v Speaker 1>for you to assume that because you asked the model

0:07:14.480 --> 0:07:16.440
<v Speaker 1>that does not have thoughts whether it thinks you got

0:07:16.440 --> 0:07:18.920
<v Speaker 1>something right, that you actually did so, and that it

0:07:19.080 --> 0:07:22.720
<v Speaker 1>got it right. To be explicit, allowing an LM to

0:07:22.760 --> 0:07:24.520
<v Speaker 1>write all of your code means that you are no

0:07:24.600 --> 0:07:27.840
<v Speaker 1>longer developing code, nor are you learning how to develop code,

0:07:28.080 --> 0:07:30.360
<v Speaker 1>nor are you going to become a better software engineer

0:07:30.360 --> 0:07:33.440
<v Speaker 1>as a result, nor are you solving actual problems. You

0:07:33.520 --> 0:07:37.120
<v Speaker 1>are just handing work over to something and taking dog

0:07:37.160 --> 0:07:40.280
<v Speaker 1>shit out. I'm not saying that all code is using

0:07:40.400 --> 0:07:44.280
<v Speaker 1>lms are inherently bankrupt or anything. But I hear these

0:07:44.360 --> 0:07:47.440
<v Speaker 1>stories about writing all the code, and I'm they give

0:07:47.480 --> 0:07:50.760
<v Speaker 1>me the willies, And I know what I'm saying sounds

0:07:50.800 --> 0:07:53.600
<v Speaker 1>like an insult or hyperbole. I don't mean it in

0:07:53.640 --> 0:07:56.240
<v Speaker 1>that way. If you are just a person looking at code,

0:07:56.280 --> 0:07:58.520
<v Speaker 1>you're only as good as the code the model makes.

0:07:58.560 --> 0:08:01.840
<v Speaker 1>And as Mobitar recently this discussed, these models are built

0:08:01.840 --> 0:08:03.880
<v Speaker 1>to galvanize you, glaze you, and tell you that you

0:08:03.920 --> 0:08:07.600
<v Speaker 1>are remarkable as you barely glance at globs of overwritten

0:08:07.640 --> 0:08:10.680
<v Speaker 1>code that, even if it functions, eventually grows to a

0:08:10.720 --> 0:08:13.400
<v Speaker 1>whole built with no intention or purpose other than what

0:08:13.520 --> 0:08:17.480
<v Speaker 1>the model generated from your prompts. I'm sure there are

0:08:17.520 --> 0:08:20.560
<v Speaker 1>software engineers using these models ethically, who read all the code,

0:08:20.560 --> 0:08:22.920
<v Speaker 1>who have complete industry over it, and use it like

0:08:22.960 --> 0:08:26.240
<v Speaker 1>a glorified auto complain. I can see the value. I'm

0:08:26.240 --> 0:08:28.280
<v Speaker 1>also sure that there are some that are just asking

0:08:28.320 --> 0:08:30.800
<v Speaker 1>it to do stuff, glancing at the code and shipping it.

0:08:30.800 --> 0:08:33.520
<v Speaker 1>It's impossible to measure how many of each camp there are,

0:08:33.679 --> 0:08:37.120
<v Speaker 1>but hearing Spotify's CEO so that its top developers are

0:08:37.120 --> 0:08:40.520
<v Speaker 1>basically not writing code anymore makes me deeply worried, because

0:08:40.559 --> 0:08:44.479
<v Speaker 1>this shit isn't replacing software engineering at all. It's mindlessly

0:08:44.559 --> 0:08:47.280
<v Speaker 1>removing friction and putting the burden of good or right

0:08:47.520 --> 0:08:50.760
<v Speaker 1>on a user that it's intentionally gassing up. And ultimately,

0:08:50.840 --> 0:08:53.679
<v Speaker 1>this entire era is a test of a person's ability

0:08:53.679 --> 0:08:57.800
<v Speaker 1>to understand and appreciate friction. Friction can be a very

0:08:57.840 --> 0:09:00.400
<v Speaker 1>good thing. When I don't understand something, I make an

0:09:00.400 --> 0:09:02.679
<v Speaker 1>effort to do so, and the moment it clicks is magical.

0:09:03.120 --> 0:09:05.000
<v Speaker 1>In the last three years, I've had to teach myself

0:09:05.000 --> 0:09:08.240
<v Speaker 1>a great deal about finance accountancy in the greater technology industry,

0:09:08.360 --> 0:09:10.120
<v Speaker 1>and there have been so many moments where I've walked

0:09:10.120 --> 0:09:13.120
<v Speaker 1>away from the page, frustrated student self doubt that I'd

0:09:13.120 --> 0:09:17.800
<v Speaker 1>never understand something. I eventually did. It took time, It

0:09:17.880 --> 0:09:21.199
<v Speaker 1>really took time, and really, that luxury of time is important,

0:09:21.400 --> 0:09:25.040
<v Speaker 1>and sadly, many software engineers face increasingly deranged deadlines set

0:09:25.080 --> 0:09:27.520
<v Speaker 1>by bosses that don't understand a single fucking thing about

0:09:27.520 --> 0:09:30.480
<v Speaker 1>their job or the software industry itself, let alone what

0:09:30.640 --> 0:09:34.640
<v Speaker 1>lms are capable of, or what responsible software engineering might be.

0:09:35.640 --> 0:09:38.079
<v Speaker 1>The push from above to use these models because they

0:09:38.120 --> 0:09:40.360
<v Speaker 1>can and I quote write code faster than a human

0:09:40.720 --> 0:09:44.440
<v Speaker 1>is a disastrous conflation of fast and good, all because

0:09:44.480 --> 0:09:47.200
<v Speaker 1>of flimsy myths peddled by venture capitalists in the media

0:09:47.440 --> 0:09:52.560
<v Speaker 1>about LM's being able to replace software engineers. It's fucking stupid.

0:09:52.840 --> 0:09:56.120
<v Speaker 1>It's a disgrace, and there are real problems that are

0:09:56.160 --> 0:09:59.520
<v Speaker 1>going to happen as a result. The problem is that

0:09:59.679 --> 0:10:04.680
<v Speaker 1>lms can write all code. Theoretically, they can just put

0:10:04.720 --> 0:10:07.040
<v Speaker 1>the code out that you might have rent yourself. Doesn't

0:10:07.040 --> 0:10:09.000
<v Speaker 1>mean the code is good, or that somebody can read

0:10:09.000 --> 0:10:11.760
<v Speaker 1>it and understand its intention, or that it works, that

0:10:11.840 --> 0:10:13.679
<v Speaker 1>it will work in the future, or that you can

0:10:13.679 --> 0:10:17.840
<v Speaker 1>build any kind of sustainable or I don't know, like

0:10:18.360 --> 0:10:21.280
<v Speaker 1>stable in any way organization on top of it, or

0:10:21.320 --> 0:10:22.959
<v Speaker 1>even that having a lot of code is a good thing,

0:10:23.000 --> 0:10:25.160
<v Speaker 1>both in the present and in the future of any

0:10:25.160 --> 0:10:29.040
<v Speaker 1>company built using this generative code. Adding the variable of

0:10:29.120 --> 0:10:32.079
<v Speaker 1>code written by people who quite literally do not understand

0:10:32.120 --> 0:10:36.040
<v Speaker 1>it guarantee something severe and calamitous in the future, though

0:10:36.000 --> 0:10:38.640
<v Speaker 1>why i'd argue that was the case without their influence.

0:10:39.880 --> 0:10:42.880
<v Speaker 1>Increasing the volume of code contributed to a company naturally

0:10:42.880 --> 0:10:46.080
<v Speaker 1>increases the amount of time needed to read it. And

0:10:46.160 --> 0:10:48.840
<v Speaker 1>the amount of effort needed to maintain it, which naturally

0:10:49.000 --> 0:10:54.800
<v Speaker 1>encourages people to use llms to summarize it. And then well,

0:10:54.840 --> 0:10:56.600
<v Speaker 1>you have to rely on the lms to tell you

0:10:56.640 --> 0:10:58.559
<v Speaker 1>what good looks like, and they don't know a sincle

0:10:58.600 --> 0:11:01.800
<v Speaker 1>fucking thing. And it also creates a new burden on

0:11:01.840 --> 0:11:03.880
<v Speaker 1>the technical workers to have to clean up the slop

0:11:04.080 --> 0:11:07.600
<v Speaker 1>in their day to day lives. Generative code is a

0:11:07.640 --> 0:11:11.400
<v Speaker 1>digital ecological disaster, one that will take years to repair

0:11:11.640 --> 0:11:14.160
<v Speaker 1>thanks to company remits to write as much code as

0:11:14.200 --> 0:11:17.520
<v Speaker 1>fast as possible and use llms as much as possible too.

0:11:18.679 --> 0:11:22.840
<v Speaker 1>Every single person responsible must be held accountable, especially for

0:11:22.840 --> 0:11:25.920
<v Speaker 1>the calamities to come as lazily managed software companies see

0:11:25.960 --> 0:11:30.360
<v Speaker 1>the consequences of building their software on sand I'll see

0:11:30.360 --> 0:11:31.199
<v Speaker 1>you all next week.