WEBVTT - CZM Rewind: The Truth About Software Development with Carl Brown 0:00:02.400 --> 0:00:08.239 Also media, Hello and welcome to Better Offline. I'm your 0:00:08.280 --> 0:00:22.759 host ed zet Trunk. Also today, I'm joined by Carl Brown, 0:00:22.800 --> 0:00:25.880 a veteran software engineering host of the excellent YouTube channel 0:00:25.920 --> 0:00:28.120 Internet of Bugs. Cayl, thank you for joining me. 0:00:28.400 --> 0:00:29.160 Thanks for having me. 0:00:29.720 --> 0:00:32.199 So I'm going to start with an easy one. What 0:00:32.360 --> 0:00:34.440 is the software developer like? What actually is that? 0:00:35.479 --> 0:00:35.639 So? 0:00:35.920 --> 0:00:41.640 Basically what we do is we take ideas about problems 0:00:41.640 --> 0:00:47.599 that people want to solve, generally, and we write software. 0:00:47.640 --> 0:00:52.920 We write code that tells computers instructions how to make 0:00:53.440 --> 0:00:56.960 the computer do the thing that needs to do to 0:00:57.000 --> 0:00:58.720 solve the problem the person else is to solve. 0:00:59.200 --> 0:00:59.400 Right. 0:01:01.000 --> 0:01:04.760 Gaming programming is a little bit different, but that's most 0:01:04.760 --> 0:01:06.280 software development is basically that. 0:01:06.840 --> 0:01:10.679 And this is another quite silly question, but necessary. How 0:01:10.800 --> 0:01:12.440 much of that is actually writing code? 0:01:14.400 --> 0:01:17.760 It depends on how good you you're the people that 0:01:17.800 --> 0:01:20.760 are asking for stuff. Is As a general rule, I 0:01:20.760 --> 0:01:25.200 would say maybe between ten percent and twenty five percent. 0:01:25.520 --> 0:01:28.080 Okay, just really want to be ten to twenty Even 0:01:28.080 --> 0:01:29.800 if we say thirty percent of the job, which is 0:01:29.840 --> 0:01:32.600 more than you said, that means the majority of this 0:01:32.680 --> 0:01:34.200 job is not actually writing. 0:01:33.920 --> 0:01:35.520 Code right now. 0:01:35.800 --> 0:01:40.720 That's that's largely for folks that are farther up the chain, Right, So, 0:01:40.720 --> 0:01:42.919 if you're fresh out of school and you don't really 0:01:42.959 --> 0:01:45.560 you're not in the job in the you don't understand 0:01:45.560 --> 0:01:47.560 how to manage requirements for any of that kind of stuff. 0:01:47.600 --> 0:01:49.760 Yet someone's going to basically hand you a thing to do, 0:01:50.680 --> 0:01:52.160 and in that kind of case, you're going to be 0:01:52.160 --> 0:01:53.920 spending a lot more time writing code than that. But 0:01:54.040 --> 0:01:58.600 for me, it's you know, it's far far more talking 0:01:58.640 --> 0:02:00.560 to people and stuff than actually code. 0:02:01.080 --> 0:02:03.120 Right. The reason I asked that, and the reason we're 0:02:03.160 --> 0:02:05.160 doing this as well, is the there have been a 0:02:05.160 --> 0:02:08.240 lot of stories around like LM's replacing code as LLLMS 0:02:08.280 --> 0:02:12.880 replacing engineers, claiming that junior software engineers will be a 0:02:12.880 --> 0:02:16.800 thing of the past due to LLLMS. How much validity 0:02:16.880 --> 0:02:17.400 is there in. 0:02:17.320 --> 0:02:20.480 That, Well, when it comes to the really really really 0:02:20.560 --> 0:02:22.560 fresh out of school kids, right. 0:02:23.960 --> 0:02:25.799 That you have to basically break everything. 0:02:25.480 --> 0:02:28.040 Down in hand, the little chunks of work, and LM 0:02:28.080 --> 0:02:31.320 can kind of do that, although the kid will get 0:02:31.360 --> 0:02:34.000 better over time and the LLM is pretty much fixed 0:02:34.200 --> 0:02:39.720 right right, But past that it doesn't do a good 0:02:39.800 --> 0:02:42.280 job of being able to do any kind of long 0:02:42.360 --> 0:02:46.400 term thinking, and that's largely the job, right, I mean 0:02:46.520 --> 0:02:49.200 this is this is not a set of you know, 0:02:50.280 --> 0:02:52.440 I come in today, I do a thing today, I 0:02:52.520 --> 0:02:55.480 come in tomorrow and having no understanding of what happened yesterday, 0:02:55.800 --> 0:02:58.079 and do another self contained thing and so on and 0:02:58.160 --> 0:02:58.520 so forth. 0:02:58.600 --> 0:02:59.480 Right, that's not the job. 0:02:59.520 --> 0:03:03.160 The job is a long sequence of building up on things, 0:03:03.360 --> 0:03:05.200 day after day after day after day until we get 0:03:05.240 --> 0:03:08.240 to the point where the whole thing together works, And 0:03:08.280 --> 0:03:09.200 that's what it's supposed to do. 0:03:09.880 --> 0:03:13.440 So I think that I've known, and one of the 0:03:13.480 --> 0:03:15.480 reasons I had you on as well, is that really 0:03:15.600 --> 0:03:17.600 there are so many of these stories that claiming that, 0:03:17.800 --> 0:03:21.440 like this software engineer's job is gone, that these companies 0:03:21.520 --> 0:03:24.400 be writing all of their code with AI, and it 0:03:24.680 --> 0:03:26.960 doesn't even seem like that is possible. One of your videos, 0:03:27.000 --> 0:03:28.880 you did a really good thing around like the twenty 0:03:28.960 --> 0:03:30.720 to thirty percent a link to this in the notes, 0:03:30.720 --> 0:03:33.799 twenty to thirty percent of code that behind meta and 0:03:33.880 --> 0:03:37.120 I think Google it was is written by AI. Now, again, 0:03:37.160 --> 0:03:38.720 how much valuidity is there to them? 0:03:39.040 --> 0:03:42.600 Well, I mean so if one of the quotes was 0:03:42.680 --> 0:03:45.080 something to the effect of thirty percent of the code 0:03:45.280 --> 0:03:49.040 is suggestions that were given by autocomplete that a human accepted, right, 0:03:49.840 --> 0:03:53.840 which could be as much as you know, the thing said, 0:03:53.960 --> 0:03:55.200 oh wait, you spelled this wrong. 0:03:55.320 --> 0:03:57.000 Let me give you a suggestion about how to spell 0:03:57.040 --> 0:03:57.400 it correctly. 0:03:57.480 --> 0:04:01.160 Right, right, I mean how much of the actual text 0:04:01.240 --> 0:04:04.880 that you write is you know, it is corrected by 0:04:04.920 --> 0:04:05.560 a spell checker? 0:04:05.640 --> 0:04:05.760 Right? 0:04:05.800 --> 0:04:07.880 If all that counts as AI, then what percentage of 0:04:07.880 --> 0:04:08.960 your stuff is written by AI? 0:04:09.080 --> 0:04:09.200 Right? 0:04:09.320 --> 0:04:12.560 Well, in my case, absolutely nothing. But that's just kind 0:04:12.600 --> 0:04:14.560 of a free I'm a just a complete free But no, 0:04:14.680 --> 0:04:17.800 I get your point, and it's without being a code 0:04:17.800 --> 0:04:21.159 of myself. It's something I've really noticed across these stories 0:04:21.200 --> 0:04:23.960 where people just kind of blindly push them out and 0:04:24.040 --> 0:04:25.600 they say, oh, it's twenty to thirty percent of the 0:04:25.640 --> 0:04:29.240 code is written by but there's no verifying this. And 0:04:29.360 --> 0:04:32.679 also it feels like it might create a bigger problem, 0:04:32.720 --> 0:04:36.240 which is say, we accept this idea even though I don't, 0:04:36.400 --> 0:04:38.920 and it sounds like a pretty spurious one kind of 0:04:38.920 --> 0:04:42.720 silly to do so at some point, isn't code not 0:04:42.960 --> 0:04:45.479 just the series of things that you write to make 0:04:45.520 --> 0:04:48.320 a program work. It's connected to a bazillion other things, 0:04:48.400 --> 0:04:51.960 which if you don't know why that was written because 0:04:51.960 --> 0:04:54.840 you had something generated. Is that not a huge problem? 0:04:55.800 --> 0:04:56.040 Yes? 0:04:56.279 --> 0:05:01.440 But worse, what what we're finding when code gets generated 0:05:01.560 --> 0:05:03.800 is that basically you end up doing the same thing 0:05:03.880 --> 0:05:06.280 in a bunch of different places, but in each one 0:05:06.320 --> 0:05:07.640 of those different places. 0:05:07.360 --> 0:05:08.320 You do it a different way. 0:05:08.839 --> 0:05:09.840 Can you give me an example. 0:05:10.520 --> 0:05:14.240 So, for example, when you need to go fetch a 0:05:14.279 --> 0:05:16.600 thing from a server, right, well, you overhear in this 0:05:16.680 --> 0:05:18.280 code you fetch a thing from a server. Over Here 0:05:18.320 --> 0:05:21.000 in the code you fetch a different thing from the server. Normally, 0:05:21.040 --> 0:05:22.920 you'd be able to use the same block of code 0:05:22.920 --> 0:05:24.440 to do that, so that if there's a mistake in it, 0:05:24.520 --> 0:05:26.720 you can change it once and it's fixed everywhere, right, Right, 0:05:26.920 --> 0:05:28.800 But the way the llms work is you say, hey, 0:05:28.839 --> 0:05:29.920 I want to fetch a thing from the server, and 0:05:29.960 --> 0:05:31.480 it says cool, and it writes a whole thing. 0:05:31.440 --> 0:05:32.920 For you that may or may not work the same 0:05:32.920 --> 0:05:33.880 way as the previous one. 0:05:33.960 --> 0:05:38.160 Right, And so now you find, okay, under some circumstances, 0:05:38.240 --> 0:05:40.120 we're having a problem fetching things from the server. I 0:05:40.160 --> 0:05:42.120 don't know which one of these twelve implementations that go 0:05:42.200 --> 0:05:44.040 fetch from the server is the one that's actually causing 0:05:44.080 --> 0:05:44.520 the problem. 0:05:45.000 --> 0:05:48.239 Right, Well, so isn't there isn't there a security issue 0:05:48.279 --> 0:05:53.120 of having large language models, Like wouldn't all the code 0:05:53.160 --> 0:05:55.840 be quite similar or at least more similar depending on 0:05:56.040 --> 0:05:58.920 if everyone's using Claude or everyone's using well get hub copile. 0:05:58.960 --> 0:06:00.839 I guess this Claude now, No, not really. 0:06:01.200 --> 0:06:03.240 It basically kind of picks a random number at the 0:06:03.240 --> 0:06:05.640 beginning and goes okay, So that's the I think if 0:06:05.640 --> 0:06:07.200 it kind of like you deal a deck of cards, right, 0:06:07.279 --> 0:06:09.600 whichever deck of card gets turned over first, that's the 0:06:09.720 --> 0:06:12.640 beginning of the autocomplete that it starts. And so depending 0:06:12.680 --> 0:06:15.360 on which example it's I don't want to say thinking of, 0:06:15.800 --> 0:06:20.240 but depending on which example represents that, I'm grastically over simplifying. 0:06:20.240 --> 0:06:22.320 But depending on which example is represented by that card, 0:06:22.560 --> 0:06:24.200 it's going to go down one path or another. 0:06:24.680 --> 0:06:28.360 Right, And so what they actually what these large language 0:06:28.400 --> 0:06:30.640 model coding tools actually good for? Because I get a 0:06:30.680 --> 0:06:33.400 lot of people who who respond by saying, this is 0:06:33.560 --> 0:06:36.240 proof that AI is a big deal, and I'm just 0:06:36.320 --> 0:06:38.479 kind of like, I'm not even looking for a particular answer, 0:06:38.600 --> 0:06:40.160 just truly what's useful about them? 0:06:40.839 --> 0:06:44.480 So they are decent at when you know what you 0:06:44.680 --> 0:06:48.000 want and what you want is a fairly simple, self 0:06:48.080 --> 0:06:52.160 contained thing, and you know how to tell whether or 0:06:52.160 --> 0:06:53.880 not in the self can think thing does what you want, 0:06:54.680 --> 0:06:57.600 It can type it faster than you can, like wells 0:06:57.600 --> 0:07:01.800 so correct. Basically, Yes, it's like auto complete if you 0:07:01.960 --> 0:07:03.640 if you know exactly what you want. Yeah, I mean 0:07:03.880 --> 0:07:06.280 so I use it a lot because I programm in 0:07:06.320 --> 0:07:09.400 a bunch of different programming languages a lot, right on 0:07:09.440 --> 0:07:12.120 different projects at the same time or on the same 0:07:12.200 --> 0:07:14.360 day or the same week, And it's really easy for 0:07:14.440 --> 0:07:16.280 me to go, Okay, wait, which language am I in? Right? 0:07:16.320 --> 0:07:16.400 Now? 0:07:16.440 --> 0:07:18.160 Okay, how do I do this in this language? 0:07:18.200 --> 0:07:18.320 Right? 0:07:18.400 --> 0:07:19.240 So it's kind of you can. 0:07:19.160 --> 0:07:21.440 Actually understand the generation when it comes. 0:07:21.320 --> 0:07:23.200 To Yeah, it's like I know what kind of loop 0:07:23.240 --> 0:07:24.640 I want, but I don't remember the syntax for this 0:07:24.680 --> 0:07:26.520 particular language where I don't want to. So I use 0:07:26.560 --> 0:07:28.840 it kind of like a Google Translate kind of thing 0:07:28.920 --> 0:07:31.040 to go from one programming language to another sometimes. 0:07:32.160 --> 0:07:36.680 But you wouldn't trust it to build a full software package. 0:07:36.480 --> 0:07:37.080 Oh not at all? 0:07:37.720 --> 0:07:38.040 Why not? 0:07:38.800 --> 0:07:40.080 Well it wouldn't work to start with. 0:07:40.760 --> 0:07:41.600 Why wouldn't it work? 0:07:42.000 --> 0:07:43.840 Well, I mean, so I've done some experimentation on that 0:07:44.080 --> 0:07:48.280 where I've taken fairly complicated. 0:07:50.160 --> 0:07:50.680 Challenges. 0:07:51.000 --> 0:07:54.120 Challenges are intended for programmers to basically get better at 0:07:54.160 --> 0:07:56.280 their craft and that kind of thing. And I've run 0:07:56.440 --> 0:07:58.800 ai you told it, you know, step by step, okay, 0:07:59.640 --> 0:08:01.520 the the challenge says, this is your next step. 0:08:01.560 --> 0:08:01.760 Do this. 0:08:01.840 --> 0:08:03.520 The challenge says, this is your next step. 0:08:03.600 --> 0:08:07.520 Do this on really simple challenges in programming languages like 0:08:07.600 --> 0:08:09.440 Python that it's got a lot and a lot of 0:08:09.480 --> 0:08:13.080 examples for it does okay past the point where you're 0:08:13.160 --> 0:08:16.040 in the really simple kind of language, things they just 0:08:16.240 --> 0:08:18.080 they sometimes get to the point where they can't even 0:08:18.600 --> 0:08:19.840 create anything that builds it all. 0:08:20.680 --> 0:08:24.440 Huh, why is there? Why does so many engineers swear 0:08:24.520 --> 0:08:24.960 by it? Then? 0:08:27.240 --> 0:08:29.640 Honestly, I'm not sure to what extent the engineers are 0:08:29.640 --> 0:08:31.679 swearing by it. I've talked to a lot of folks 0:08:31.680 --> 0:08:35.400 who are like, you know, my group, you know this 0:08:35.640 --> 0:08:39.640 big bank, you know friend of mine, My group is 0:08:39.679 --> 0:08:41.959 getting copilot gemmed on our throats whether we want it 0:08:42.040 --> 0:08:44.559 or not. And the executives are all really excited about it, 0:08:44.760 --> 0:08:47.120 and none of us are interesting. 0:08:47.400 --> 0:08:50.559 So it's executive. I've I've personally had this theory that 0:08:50.600 --> 0:08:54.520 it's like executive pushed, and that it's all about it's 0:08:54.559 --> 0:08:56.800 all about what the bosses want to see rather than 0:08:57.679 --> 0:08:58.079 even do. 0:08:58.840 --> 0:09:02.960 Sorry, there's a lot of wish fulfillment. 0:09:03.040 --> 0:09:04.880 There's a lot of like, we want to not have 0:09:05.000 --> 0:09:07.480 to deal with these programmers anymore, so we would rather 0:09:07.600 --> 0:09:09.360 deal with the AI thing, and we're just gonna hope 0:09:09.400 --> 0:09:11.120 that the AI thing is going to be, you know, 0:09:11.360 --> 0:09:13.000 just as good as the programmers are close to us, 0:09:13.040 --> 0:09:16.199 just as good as the programmers, and not nearly as annoying. 0:09:16.720 --> 0:09:20.559 Seems like a definitional well maybe that's not the right word. 0:09:20.800 --> 0:09:22.959 Seems like the difference between a software engineer and a 0:09:23.000 --> 0:09:26.400 software developer almost because it's not just about flopping code out, 0:09:26.440 --> 0:09:28.240 it's about making sure the code does stuff. 0:09:28.920 --> 0:09:33.719 Yeah, I mean, those terms get mashed to keep yeah, 0:09:33.760 --> 0:09:33.960 I mean. 0:09:34.080 --> 0:09:35.880 So part of the problem is that in like I 0:09:35.960 --> 0:09:37.839 live in Texas, and in Texas you're not allowed to 0:09:37.840 --> 0:09:41.520 call yourself an engineer unless you passed the engineering exam. Right, 0:09:41.679 --> 0:09:45.079 So I can't literally, I literally can't call myself a 0:09:45.120 --> 0:09:47.319 software engineer legally in Texas As. I understand it. I'm 0:09:47.360 --> 0:09:49.520 not a lawyer, but that's my understanding. So it's like 0:09:49.600 --> 0:09:51.439 the terms get all confused. 0:09:51.640 --> 0:09:55.120 Right, So somewhat related, what is it that people misunderstand 0:09:55.160 --> 0:09:55.719 about the job? 0:09:55.840 --> 0:09:58.400 Then, well, I mean, so one of it is what 0:09:58.440 --> 0:10:01.760 you said earlier, which is that a very small percentage 0:10:01.760 --> 0:10:03.959 of the job is actually slinging code. A lot of 0:10:03.960 --> 0:10:05.599 it is basically trying to figure out what it is 0:10:05.640 --> 0:10:08.360 the code should do based on what you've been told 0:10:08.400 --> 0:10:11.120 that the problem, you know, the solution of the problem 0:10:11.120 --> 0:10:11.920 that you're trying to solve. 0:10:13.679 --> 0:10:14.680 Another thing is that. 0:10:16.280 --> 0:10:20.760 A lot of the problem with the job is that 0:10:21.600 --> 0:10:25.559 every little decision builds up over time, and at some 0:10:25.800 --> 0:10:29.400 point a bug is going to happen. They're inevitable, and 0:10:29.520 --> 0:10:33.439 when that happens, basically there's this process where what you 0:10:33.559 --> 0:10:36.640 need to do, if you're being competent, is roll back 0:10:36.679 --> 0:10:40.040 through the series of decisions, figure out what caused that bug, 0:10:40.600 --> 0:10:42.920 and then figure out what other bugs are likely to 0:10:43.000 --> 0:10:45.000 have been caused by that same set of decisions, and 0:10:45.040 --> 0:10:47.920 then fix not just the bug you're working on, but 0:10:48.200 --> 0:10:50.319 the bugs that you know, not just the bug that's 0:10:50.360 --> 0:10:53.280 been reported, but the bugs that might have also been 0:10:53.400 --> 0:10:55.760 caused by the same problem. Right, And that kind of 0:10:55.840 --> 0:10:57.760 long term thinking is not a thing I've ever seen 0:10:58.559 --> 0:11:01.920 LM exhibit at all. I talk about it like l 0:11:02.000 --> 0:11:05.679 limbs or generative AI is good at solving riddles, but 0:11:05.880 --> 0:11:07.880 actual software development is more like solving a murder. 0:11:08.320 --> 0:11:11.480 Yes, you said that in that wonderful video. Yeah, I 0:11:11.880 --> 0:11:16.080 And it almost feels as if we are building towards 0:11:16.120 --> 0:11:19.360 an actual calamity of sorts, maybe not an immediate one. 0:11:19.400 --> 0:11:22.360 Maybe it'll be kind of sectioned off into areas because 0:11:22.360 --> 0:11:25.480 you've got a new generation of young people coming into 0:11:25.520 --> 0:11:28.800 software engineering or what have you, learning to use AI 0:11:29.040 --> 0:11:32.559 tools rather than your videos definitely talk about this as well, 0:11:33.400 --> 0:11:35.960 actually how to develop software and make sure it works, 0:11:36.000 --> 0:11:38.480 and make sure that it has the infrastructural side inline, 0:11:38.559 --> 0:11:41.319 and also that you're building it with the long term 0:11:41.400 --> 0:11:44.240 thinking of someone else might need to understand how this works, 0:11:44.559 --> 0:11:46.760 and they're not learning that, So you've just got a 0:11:46.880 --> 0:11:50.120 generation of kind of pumping the internet and organizations with 0:11:50.600 --> 0:11:51.520 sloppier code. 0:11:52.720 --> 0:11:54.839 Yes, although I mean one of the problems we're having 0:11:54.840 --> 0:11:58.360 at the moment is that the hiring process for really 0:11:58.480 --> 0:12:01.839 junior engineers is actually pretty broken at the moment, and 0:12:01.960 --> 0:12:03.880 a lot of people are not hiring people that are 0:12:03.880 --> 0:12:07.040 fresh out of school because they're expecting that the AI 0:12:07.480 --> 0:12:08.800 will be able to do. 0:12:09.720 --> 0:12:12.600 Basically, a senior or a mid level developer with. 0:12:12.800 --> 0:12:16.280 The benefit of AI, with the benefit of AI that's 0:12:16.280 --> 0:12:19.400 in air quotes, will be able to do the work 0:12:19.600 --> 0:12:22.079 of that person plus a couple of fresh outs that 0:12:22.120 --> 0:12:23.960 they normally would have hired, but they're not hiring at 0:12:23.960 --> 0:12:27.240 the moment. There's some statistics about how the people that 0:12:27.280 --> 0:12:33.240 are fresh out of school these days are historically underemployed 0:12:33.320 --> 0:12:35.720 relative to the general population, at least in the US 0:12:35.760 --> 0:12:36.200 where I live. 0:12:37.240 --> 0:12:39.560 It also feels like there's no intention behind the code, 0:12:40.160 --> 0:12:42.120 like it's just if you're just generating it. You don't 0:12:42.160 --> 0:12:44.320 really know why you made any patity. You could say 0:12:44.400 --> 0:12:47.839 I chose these lines, But is that at some point 0:12:48.040 --> 0:12:51.839 if you have large amounts of software developers using it, 0:12:52.320 --> 0:12:55.240 however large, But the young people in an organization using 0:12:55.280 --> 0:12:58.280 it to generate their code, they're neither learning to write 0:12:58.320 --> 0:13:01.080 better code, nor are they learning how to just learning 0:13:01.120 --> 0:13:04.680 how to fill in blocks they'll kin within the job. 0:13:05.040 --> 0:13:05.640 Yeah, I mean. 0:13:05.800 --> 0:13:08.559 The the trick is that those of us that have 0:13:08.720 --> 0:13:12.120 spent a whole lot of time debugging software right and 0:13:12.240 --> 0:13:14.320 like finding the problems and digging into them and trying 0:13:14.320 --> 0:13:17.920 to figure out what's going on that kind of stuff. 0:13:18.960 --> 0:13:21.599 It's going to be really hard for younger folks to 0:13:21.800 --> 0:13:25.720 get hired into those jobs so that they have time 0:13:25.840 --> 0:13:28.559 to build the experience to be able to do that. 0:13:29.080 --> 0:13:30.880 And I'm afraid we're going to end up with basically 0:13:31.240 --> 0:13:34.520 an older generation or generations retiring and a newer generation 0:13:34.640 --> 0:13:37.360 that hasn't had the experience of doing that kind of debugging. 0:13:38.480 --> 0:13:41.040 And then it's going to be a real mess, especially 0:13:41.120 --> 0:13:44.160 since from what I can tell, the code that the 0:13:44.240 --> 0:13:47.880 AI's generated are a lot buggier and buggier in weirder 0:13:48.240 --> 0:13:51.360 like randomish kind of ways. Stuff just kind of comes 0:13:51.400 --> 0:13:53.000 out of nowhere in a way that I don't. I mean, 0:13:53.280 --> 0:13:55.520 I've debugged code from people that don't speak the same 0:13:55.600 --> 0:13:57.520 languages as I do, you know, all that kind of 0:13:57.559 --> 0:14:01.680 stuff AI code is different. It's just like, Okay, why 0:14:01.800 --> 0:14:04.400 would anyone want to put that block there that doesn't 0:14:04.440 --> 0:14:05.839 have anything to do with what we're trying to do 0:14:05.920 --> 0:14:06.600 at the moment. 0:14:07.040 --> 0:14:10.640 And why is that? Is it just because it's probabilistic. 0:14:10.920 --> 0:14:11.360 I guess so. 0:14:11.960 --> 0:14:14.599 I mean it's hard to say why. I mean, the 0:14:15.320 --> 0:14:17.520 idea of why an LLN does what it does is 0:14:17.640 --> 0:14:20.400 kind of a you know, anybody's guess. 0:14:21.600 --> 0:14:24.960 Yeah, it's just I keep thinking of the word calamity 0:14:25.040 --> 0:14:28.280 because you sent me these studies as well about how 0:14:28.760 --> 0:14:32.040 they found like a downward pressure on the quality of 0:14:32.160 --> 0:14:34.920 code on GitHub. Would you mind walking me through what 0:14:35.040 --> 0:14:35.440 that means? 0:14:35.920 --> 0:14:38.080 Yeah, So basically what that study found, there were there 0:14:38.080 --> 0:14:39.720 have been a couple of them, but what that particular 0:14:39.720 --> 0:14:43.440 study found is that there's what they call code churn 0:14:43.520 --> 0:14:46.480 has gone up. And code churn is basically when you 0:14:46.760 --> 0:14:49.680 push something you like, add a line of code, you 0:14:49.840 --> 0:14:52.840 push it into two test or to production, and then 0:14:53.000 --> 0:14:54.960 in a short period of time, like I don't remember 0:14:55.000 --> 0:14:57.560 exactly what the definition was, like in a month or 0:14:57.600 --> 0:15:01.120 two months, that line of code changes, right. So basically 0:15:01.200 --> 0:15:03.000 what that means is that the line of code that 0:15:03.120 --> 0:15:08.240 got created, somebody decided after it got put in, oh wait, no, 0:15:08.400 --> 0:15:10.160 that doesn't work right, we don't We're not happy with that. 0:15:10.200 --> 0:15:12.080 We're going to change it to be something else, right, right, 0:15:12.600 --> 0:15:15.080 And the percentage of lines or the number of lines 0:15:15.280 --> 0:15:20.440 that that get changed fairly quickly after they get submitted, 0:15:21.040 --> 0:15:26.120 has gone way up since the since the implementation of 0:15:26.440 --> 0:15:30.120 get hub copilot. So and this is across like most 0:15:30.200 --> 0:15:32.800 of the giant you know, millions of lines of codes 0:15:32.840 --> 0:15:33.760 on GitHub and. 0:15:33.840 --> 0:15:37.280 For a simpleton me, why does it being changed? Why 0:15:37.400 --> 0:15:38.960 is changing it so often bad? 0:15:39.960 --> 0:15:43.000 Like? Well, I mean, so, I mean if you do 0:15:43.080 --> 0:15:44.640 it right the first time, you can move on to 0:15:44.720 --> 0:15:45.240 the next thing. 0:15:45.640 --> 0:15:45.840 Ah. 0:15:46.360 --> 0:15:46.520 Right. 0:15:46.920 --> 0:15:49.120 If it's like you know, if you're writing a document 0:15:49.240 --> 0:15:51.320 and you put put the document in there and then 0:15:51.360 --> 0:15:54.680 you like you're get you're in in Google Docs and 0:15:54.720 --> 0:15:57.200 you're like tracking changes and it's like, Okay, this sentence 0:15:57.240 --> 0:15:58.440 has changed seventeen times. 0:15:58.480 --> 0:16:01.280 Obviously the person isn't happy with that, right. 0:16:01.520 --> 0:16:04.240 So the generative code isn't good, right, and so people 0:16:04.280 --> 0:16:05.200 see you need to change. 0:16:05.560 --> 0:16:07.560 That's the presumption, yes, and. 0:16:08.960 --> 0:16:11.600 So I it also said the code quality itself? Is 0:16:11.640 --> 0:16:14.200 that the only way they is that the only way 0:16:14.240 --> 0:16:16.400 they measured it? Or is it there were other things 0:16:16.440 --> 0:16:18.160 as well, So. 0:16:18.320 --> 0:16:21.520 They measured that they measured. 0:16:23.120 --> 0:16:25.720 Uh like moved code. 0:16:26.040 --> 0:16:27.520 Yeah, that the move code. 0:16:27.800 --> 0:16:31.360 The thing I was talking about earlier, where the uh 0:16:31.600 --> 0:16:33.960 you've got a bunch of different places in the code 0:16:34.240 --> 0:16:38.000 that all do the same You try to do the 0:16:38.040 --> 0:16:39.000 same function, but they do it. 0:16:39.000 --> 0:16:42.520 In different ways. Normally, what would happen is you'd have 0:16:42.760 --> 0:16:43.400 your you do it. 0:16:43.480 --> 0:16:45.880 You do a thing here, right, and then at some 0:16:45.960 --> 0:16:47.360 point in the future you need to do that thing 0:16:47.480 --> 0:16:49.320 again in a different place. And so what you do 0:16:49.480 --> 0:16:52.280 is you would move that original block that does the 0:16:52.360 --> 0:16:55.280 thing someplace else, and then you would call that block 0:16:55.320 --> 0:16:57.800 from both places because it already works, right, And then 0:16:57.880 --> 0:17:00.040 that way you've got you know, however, you go that 0:17:00.160 --> 0:17:01.960 stuff from the server, you're fetching it the same way, 0:17:03.360 --> 0:17:06.080 but with this thing. Basically instead of doing that, you've 0:17:06.080 --> 0:17:08.199 got copy paste. Okay, when we put another one here, 0:17:08.240 --> 0:17:09.680 and we put another one here, let me put another 0:17:09.720 --> 0:17:11.560 one here, and it's a it's a maintenance nightmare. 0:17:23.560 --> 0:17:26.719 So for the for the audience as well, how does 0:17:26.760 --> 0:17:29.720 the software developer actually use GitHub? Like really simple stuff? 0:17:29.720 --> 0:17:31.560 I realized, but I think it's important for people to 0:17:31.840 --> 0:17:33.480 It just occurred to be like, this may be something 0:17:33.520 --> 0:17:35.560 that most listeners don't know, which is good to I 0:17:35.600 --> 0:17:36.120 think it's good. 0:17:36.320 --> 0:17:36.560 Yeah. 0:17:36.960 --> 0:17:40.000 So so what we do is we basically make changes 0:17:40.040 --> 0:17:43.680 to code. We get to the point where we the developer, 0:17:43.720 --> 0:17:46.439 are happy with the way it's set up on our machine, 0:17:46.520 --> 0:17:47.920 and then we do what it's called a push, and 0:17:48.040 --> 0:17:50.200 we basically send all that code, submit all that code 0:17:50.280 --> 0:17:54.520 up to get hub, and then theoretically, you know, there 0:17:54.560 --> 0:17:57.200 can be automatic processes that kick in that like check 0:17:57.280 --> 0:17:59.359 that code for particular things and run tests on and 0:17:59.400 --> 0:18:01.639 that kind of stuff. And then at some point we 0:18:01.720 --> 0:18:03.840 have a thing called a poor request, which is basically 0:18:03.880 --> 0:18:05.280 a thing that says, Okay, I would like this to 0:18:05.359 --> 0:18:08.160 go into production now, or more or less, I would 0:18:08.240 --> 0:18:10.919 like this to get promoted into the next phase now, 0:18:11.080 --> 0:18:13.320 and then someone theoretically will look at it and go okay, 0:18:13.400 --> 0:18:15.600 that's fine, and then click the yes button or say, hey, 0:18:15.960 --> 0:18:17.639 you forgot about this, go look at this or that 0:18:17.720 --> 0:18:20.320 kind of thing, right, And the poor requests is kind 0:18:20.359 --> 0:18:22.159 of the unit of work kind of. 0:18:23.920 --> 0:18:25.600 So with get hub you almost use it like an 0:18:25.720 --> 0:18:30.800 organizational code dump, centralize all the code. Sorry just for 0:18:30.880 --> 0:18:35.320 the non coding as well. And I think it's I 0:18:35.440 --> 0:18:38.360 think that the LLLM industry has done a really good 0:18:38.480 --> 0:18:41.280 job of dancing around these terms and selling them to 0:18:41.320 --> 0:18:44.080 people like me. While they weren't selling they didn't work 0:18:44.119 --> 0:18:47.120 on me. I am too stupid, But it's where they've 0:18:47.240 --> 0:18:49.680 just like been like, okay, yeah, well lots of people 0:18:49.840 --> 0:18:52.680 use co pilot, that's good, and this is good because 0:18:53.040 --> 0:18:56.760 software is coding. But it kind of feels like, I 0:18:56.840 --> 0:18:59.400 don't know, all of this is taking the one thing, 0:18:59.800 --> 0:19:04.159 like one major pot out of software development and ruining it. 0:19:04.320 --> 0:19:06.240 And I don't even mean coding. I mean it's the 0:19:06.359 --> 0:19:11.920 intentionality behind software design and infrastructure and maintain Like there's 0:19:12.200 --> 0:19:16.159 it seems like they're removing intention in multiple parts. 0:19:19.680 --> 0:19:21.600 So the way I would say it is when they 0:19:21.720 --> 0:19:23.760 talk about the AI being able to do the work 0:19:23.760 --> 0:19:27.239 of a programmer, what they're doing is they're devaluing all 0:19:27.280 --> 0:19:32.119 of the stuff that's not just hacking code, right, And 0:19:32.240 --> 0:19:34.080 so what they're saying is that basically the job of 0:19:34.119 --> 0:19:38.800 a developer is basically just you know, typing, basically, right, 0:19:39.200 --> 0:19:40.840 And that all of the work that we do to 0:19:41.040 --> 0:19:43.600 understand what the problem actually is and how it needs 0:19:43.640 --> 0:19:47.120 to work, and you know what other problems are likely 0:19:47.200 --> 0:19:49.040 to show up when we try to do that, and 0:19:49.119 --> 0:19:51.200 how to avoid those things as we go and that 0:19:51.320 --> 0:19:54.200 kind of thing, all that work is basically not important. 0:19:56.200 --> 0:20:01.560 And I mean I two words which would probably annoy 0:20:01.560 --> 0:20:04.399 you this is I feel like vibe coating is the 0:20:04.520 --> 0:20:08.000 other part of this. So if I'm correct, correct me 0:20:08.000 --> 0:20:10.680 if I'm wrong, vibe coating is just typing stuff into 0:20:10.680 --> 0:20:13.080 an LEM and software comes out and hopefully it works. 0:20:13.520 --> 0:20:18.359 Yeah. Vibe coating is basically when you intentionally try it. 0:20:18.480 --> 0:20:21.360 Well, I don't know, but intentionally, but basically you make 0:20:21.400 --> 0:20:23.840 a point of not digging into the code and looking 0:20:23.840 --> 0:20:26.440 at what the LM is doing, and you basically say, Okay, 0:20:26.640 --> 0:20:28.920 I would like something that does X right. I would 0:20:29.000 --> 0:20:32.200 like a game where I fly airplanes around a city 0:20:32.359 --> 0:20:35.720 or something right, And then you get what it spits out, 0:20:35.920 --> 0:20:38.600 and then you say, you know, okay, let me try it. Okay, well, 0:20:38.800 --> 0:20:40.760 can we have more airplanes? And okay, can we have 0:20:40.840 --> 0:20:42.800 some balloons with you know, signs on them now? And 0:20:42.880 --> 0:20:44.520 can we do this kind of thing? And then you 0:20:44.680 --> 0:20:47.200 don't think about what the side effects are. You don't 0:20:47.240 --> 0:20:49.639 think about what things could go wrong, You don't think 0:20:49.680 --> 0:20:51.879 about air conditions that kind of stuff, and you just 0:20:52.000 --> 0:20:55.600 hope that this whatever you look at and has the 0:20:55.720 --> 0:20:58.320 right vibe and that you know, if it if it 0:20:58.400 --> 0:21:01.040 looks like kind of what you wanted, that probably it's 0:21:01.080 --> 0:21:02.880 going to be fine, or hopefully it's going to be fine. 0:21:03.240 --> 0:21:04.680 How do you feel about vibe coating? 0:21:05.800 --> 0:21:06.720 So I do it. 0:21:06.800 --> 0:21:09.320 Sometimes vicoding is great for a thing that you're going 0:21:09.400 --> 0:21:11.040 to do once and then throw away. 0:21:12.040 --> 0:21:12.280 Yeah. 0:21:12.520 --> 0:21:14.280 Right, So if it's like, you know, okay, I want 0:21:14.280 --> 0:21:15.560 to I want to do a thing. I want to 0:21:15.600 --> 0:21:17.840 translate this thing to you know, I want to make 0:21:17.920 --> 0:21:19.720 this table go into this format over here or that 0:21:19.800 --> 0:21:21.439 kind of thing. You do it, you get the output 0:21:21.480 --> 0:21:23.320 you want, you throw the code away. No big deal, right, 0:21:23.400 --> 0:21:27.639 like a prototype almost, yeah, basically, And so you know, 0:21:28.080 --> 0:21:30.720 we call them spikes or tracer bullets. Sometimes it's like 0:21:30.760 --> 0:21:33.159 a you know, let me get a thing that works 0:21:33.880 --> 0:21:36.080 at all, right, and then let me see what I 0:21:36.119 --> 0:21:39.640 can learn from that to move into my big maintainable project. 0:21:40.840 --> 0:21:43.600 But for anything that's like, you know, this thing needs 0:21:43.640 --> 0:21:45.359 to run for a while, this thing's needs to not 0:21:45.480 --> 0:21:46.000 get hacked. 0:21:46.119 --> 0:21:49.000 This thing needs to you know, not crash. It's a 0:21:49.040 --> 0:21:49.760 really bad idea. 0:21:50.520 --> 0:21:53.760 Yeah, And at some point I feel like someone building 0:21:53.800 --> 0:21:56.240 a product that they don't really understand the working So 0:21:56.240 --> 0:21:59.760 of it's kind of almost identical to generating a story 0:21:59.840 --> 0:22:03.600 with at GPT, except kind of more complex and more 0:22:03.680 --> 0:22:04.480 prone to errors. 0:22:05.320 --> 0:22:05.520 Yeah. 0:22:06.760 --> 0:22:10.040 And the other thing is that there's an adversarial component, right, 0:22:10.800 --> 0:22:14.959 so people will intentionally try to go hack that thing 0:22:15.000 --> 0:22:17.560 that's sitting on the internet, right, Yeah, in a way 0:22:17.600 --> 0:22:19.560 that they don't intentionally try to go mess with the 0:22:19.600 --> 0:22:23.000 story that you wrote, right, right, And so even if 0:22:23.080 --> 0:22:25.240 it works all by itself, that doesn't mean it's going 0:22:25.320 --> 0:22:27.480 to work when somebody starts pounding on it intentionally trying 0:22:27.520 --> 0:22:30.240 to break it. And if they can break it, then 0:22:30.320 --> 0:22:32.359 that's a whole other set of problems that you now have. 0:22:33.160 --> 0:22:36.159 It feels like quality assurance is just never part. Oh no, 0:22:36.280 --> 0:22:38.920 they are they claiming they're going to do quality assurance 0:22:39.000 --> 0:22:41.399 with large language models. Yet they must some. 0:22:41.480 --> 0:22:44.680 People are Yeah, I mean, to be honest, a lot 0:22:44.760 --> 0:22:48.960 of companies have just been getting rid of quality assurants 0:22:49.119 --> 0:22:49.720 over the years. 0:22:49.840 --> 0:22:49.959 Right. 0:22:50.240 --> 0:22:52.480 Really, when I worked at IBM, we didn't have quality 0:22:52.480 --> 0:22:56.200 assurance at all. They would, no, seriously, they would do this. 0:22:56.359 --> 0:22:58.000 I was in IBM's Cloud group and they would do 0:22:58.080 --> 0:23:02.000 these these what do they call them, uh, packathon kind 0:23:02.040 --> 0:23:03.000 of things that they didn't call that. 0:23:03.040 --> 0:23:03.720 I don't know what they called it. 0:23:03.800 --> 0:23:07.200 But basically everybody in all the other development groups would 0:23:07.200 --> 0:23:09.680 get together and basically bang on the code that was 0:23:09.720 --> 0:23:11.879 about to get released from some other group to try 0:23:11.920 --> 0:23:13.840 to see if they could break it. Right, But they 0:23:13.880 --> 0:23:16.680 didn't have dedicated testers anymore because they decided I guess 0:23:16.760 --> 0:23:19.280 that they didn't think they were worth the money. 0:23:19.359 --> 0:23:22.280 I don't know, but we had some issues because of that. 0:23:22.800 --> 0:23:24.520 When spinal movement happened. 0:23:25.760 --> 0:23:27.560 I was in I don't know, so I was at 0:23:27.760 --> 0:23:31.639 IBM in like twenty seventeen, twenty eighteen, right, so it 0:23:31.680 --> 0:23:32.800 would have been sometime prior to that. 0:23:32.880 --> 0:23:34.840 When I got there, they didn't have any QA folks. 0:23:35.480 --> 0:23:38.520 Really just feels like the it's the management problem as well. 0:23:38.640 --> 0:23:40.679 It's the management cotton people. 0:23:40.920 --> 0:23:41.399 I would think. 0:23:41.480 --> 0:23:46.240 So it's a real shame as well. And I forgive 0:23:46.320 --> 0:23:49.199 me if I'm forgetting exactly where You've mentioned as well 0:23:49.240 --> 0:23:52.480 that there is like compound scar tissue from AI generated code, 0:23:52.560 --> 0:23:55.879 a larger problem of lots of this code being generated 0:23:55.920 --> 0:23:56.280 with AI. 0:23:57.160 --> 0:24:03.840 Well that's that's my expert, right, Yeah, just a potential worry, right. 0:24:03.840 --> 0:24:05.840 Right, so that the more of this we get and 0:24:05.960 --> 0:24:09.520 the more issues that we have, the more stuff we're 0:24:09.520 --> 0:24:11.600 gonna have to dig out of, right, And what I'm 0:24:11.680 --> 0:24:14.000 honestly envisioning at some point in the I don't know 0:24:14.080 --> 0:24:14.800 how long it will take. 0:24:14.880 --> 0:24:17.240 The crypto bubble took way longer to pop than I expected. 0:24:17.280 --> 0:24:18.399 So I don't know how long it's going to be 0:24:18.480 --> 0:24:21.479 before this one does, but I'm expecting that there's going 0:24:21.560 --> 0:24:23.679 to be this big push to try to clean up 0:24:23.680 --> 0:24:26.120 a bunch of this crap here in a few years, 0:24:26.160 --> 0:24:28.280 once people realize that a lot of the code that's 0:24:28.280 --> 0:24:31.960 being written and generated right now is has all of 0:24:32.000 --> 0:24:35.520 these vulnerabilities that nobody's bothering to check for at the moment. 0:24:35.920 --> 0:24:39.800 Right, and those vulnerabilities again non technical way, I read 0:24:39.840 --> 0:24:42.320 that it was like the call upon things on GitHub 0:24:42.400 --> 0:24:45.400 that don't exist, so bad actors create something that resembles 0:24:45.480 --> 0:24:46.240 what it's pulling from. 0:24:46.600 --> 0:24:50.080 That's so that's that's a more specific kind of one. 0:24:50.240 --> 0:24:52.159 I mean, there are a lot of things. I mean, 0:24:52.760 --> 0:24:55.600 so there have been computer viruses since the eighties, right right, 0:24:56.359 --> 0:24:58.160 you know, the Morris worm and that kind of stuff. 0:24:58.160 --> 0:25:03.560 And basically there are no own ways that code if 0:25:03.600 --> 0:25:05.639 you have to write it in a particular way in 0:25:05.760 --> 0:25:07.800 order for it to be secure, right right, And even 0:25:07.880 --> 0:25:10.320 then sometimes people come up with novel ways of making 0:25:10.400 --> 0:25:11.639 something not secure, and. 0:25:11.720 --> 0:25:13.159 How how do you have to write it to make 0:25:13.200 --> 0:25:15.600 it secure? If it's possible to explain. 0:25:15.400 --> 0:25:17.600 Well, I mean, there's a big, long list of rules, right. 0:25:17.720 --> 0:25:19.320 I mean, one thing you can do is you can 0:25:19.440 --> 0:25:22.960 use languages that are what they call safer. But still 0:25:23.040 --> 0:25:24.879 you have to make sure that any input that you 0:25:25.040 --> 0:25:27.479 get from the network, you're really really careful to make 0:25:27.480 --> 0:25:30.560 sure that it doesn't get to overwrite parts of your 0:25:30.640 --> 0:25:34.360 program that actually execute things. You have to make sure 0:25:34.400 --> 0:25:37.040 that it doesn't have the opportunity to be able to 0:25:37.080 --> 0:25:39.119 write to places on your disc that it shouldn't be 0:25:39.160 --> 0:25:42.080 able to write to. You have to be able to 0:25:42.080 --> 0:25:44.280 make sure that it doesn't have access to read data 0:25:44.320 --> 0:25:46.280 that it shouldn't be able to read, you know, all 0:25:46.359 --> 0:25:48.560 that kind of stuff. And when those things don't happen, 0:25:48.720 --> 0:25:52.480 you end up with you know, so and so got hacked. 0:25:52.960 --> 0:25:56.080 You know, turns out that somebody, we think maybe China, 0:25:56.200 --> 0:25:59.800 is reading the email of the you know, people in Congress. 0:26:01.600 --> 0:26:03.560 You get another letter in the mail that says your 0:26:03.600 --> 0:26:07.800 social Security number has been you know, leaked by you know, 0:26:07.960 --> 0:26:10.440 some credit checking firm or something like that. 0:26:10.840 --> 0:26:12.840 Even even like I think it was what the big 0:26:12.920 --> 0:26:15.920 hot target data breach from a while back was through 0:26:15.920 --> 0:26:20.479 the HVAC system. It was it was it's just except 0:26:20.520 --> 0:26:23.399 now we've got and that was with humans writing the 0:26:23.720 --> 0:26:27.800 code right, imagine if we didn't know. Oh god, it 0:26:27.880 --> 0:26:31.600 really does feel like the young people are going like that. Actually, no, 0:26:31.680 --> 0:26:34.240 I take it back. You were talking about agile the 0:26:34.280 --> 0:26:35.760 other day. I'm going to ask you to explain that 0:26:35.800 --> 0:26:38.760 in a second. But it's like, it sounds like for 0:26:39.160 --> 0:26:42.680 almost decades they've been gnawing away at management's been gnawing 0:26:42.720 --> 0:26:45.520 away at the sides of building good software and building 0:26:45.560 --> 0:26:46.600 good software culture. 0:26:48.920 --> 0:26:51.399 Yes, I mean, there's an argument that says we never 0:26:51.440 --> 0:26:52.760 got it right in the first place. But I mean, 0:26:53.520 --> 0:26:55.600 I mean, if you think about it, software has been 0:26:55.640 --> 0:26:59.600 a thing for what fifty years, sixty years, seventy years, right, 0:26:59.680 --> 0:27:02.760 I mean compare that to like construction engineering or bridge 0:27:02.800 --> 0:27:05.159 building or that kind of stuff. Right, we're still, you know, 0:27:05.480 --> 0:27:08.159 relatively speaking, in the infancy, In our infancy as a 0:27:08.280 --> 0:27:12.119 as an industry. You know, it's been a it's been 0:27:12.160 --> 0:27:15.040 a constant evolution, and a lot of times the things 0:27:15.160 --> 0:27:19.560 that were the things that we did to solve a 0:27:19.680 --> 0:27:21.840 problem that we had ended up causing other problems. 0:27:21.960 --> 0:27:22.080 Right. 0:27:22.600 --> 0:27:25.639 So, going back to agile, in the long long ago, right, 0:27:25.840 --> 0:27:28.040 we used to manage software projects the same way we 0:27:28.160 --> 0:27:31.800 manage like build you know, bridge building and building building project, 0:27:32.240 --> 0:27:35.720 you know, construction projects. And it turns out that when 0:27:35.800 --> 0:27:39.320 you're going to build a bridge, you know beforehand what 0:27:39.520 --> 0:27:41.400 you need to build a bridge to do. When you're 0:27:41.400 --> 0:27:43.679 building software, a lot of times people are changing their 0:27:43.680 --> 0:27:45.639 minds as you go. Right, and you build a thing 0:27:45.680 --> 0:27:46.959 and you show it to them and they're like, oh, 0:27:47.200 --> 0:27:48.560 why don't we put this over here, and why don't 0:27:48.560 --> 0:27:50.280 we change this? And that kind of thing right right, 0:27:50.560 --> 0:27:53.040 because you don't have the same kind of constraints physical 0:27:53.040 --> 0:27:54.720 constraints that you do when you're trying to build a bridge. 0:27:55.040 --> 0:27:58.359 And so we gotten this problem where you would create 0:27:58.400 --> 0:28:00.119 these project plans about how you were going to to 0:28:00.160 --> 0:28:01.960 build this thing, and you would never be anywhere close 0:28:02.000 --> 0:28:04.000 to on time because things would change the whole time. 0:28:04.800 --> 0:28:07.320 And so they created this thing called the agile methodology. 0:28:07.480 --> 0:28:10.439 I'm drastically simplifying. There were steps in the middle, but basically, 0:28:10.560 --> 0:28:15.240 so this agile thing is where we instead of saying, okay, 0:28:15.280 --> 0:28:16.720 so this is what the whole project's going to look like, 0:28:16.720 --> 0:28:18.879 we're gonna be doing. We're going to be done in 0:28:18.920 --> 0:28:20.879 six months, and then things changing along the way, we 0:28:21.000 --> 0:28:23.080 basically block off a thing called a sprint. It's a 0:28:23.119 --> 0:28:25.480 week or two or a month, maybe it depends. And 0:28:25.600 --> 0:28:27.840 then you know, everybody picks their own sprint length and 0:28:27.880 --> 0:28:29.960 then you go, okay, I'm only going to talk about 0:28:30.040 --> 0:28:32.439 what's going to happen in the next sprint or two, right, 0:28:32.560 --> 0:28:33.800 And then you get to the end of that two 0:28:33.800 --> 0:28:35.359 weeks and you go, okay, cool, this is what we 0:28:35.440 --> 0:28:37.119 got done. What do we want to do next? And 0:28:37.160 --> 0:28:38.560 then okay, that's what we got done, and what do 0:28:38.640 --> 0:28:39.800 we want to do next? And that kind of thing, 0:28:40.000 --> 0:28:43.200 and that way, as you go, you have the opportunity 0:28:43.240 --> 0:28:46.080 to change things. You have an opportunity to roll changes 0:28:46.200 --> 0:28:48.880 into the process, that kind of thing. Right. The problem 0:28:48.960 --> 0:28:52.080 with that is kind of the same way that dates 0:28:52.080 --> 0:28:55.520 always ran out in water in waterfall, land projects can 0:28:55.600 --> 0:28:57.880 go way way longer than they were expected to at 0:28:57.880 --> 0:29:00.479 the beginning because everybody's focused on just two weeks at 0:29:00.480 --> 0:29:02.280 a time, and you never kind of take a big 0:29:02.360 --> 0:29:04.880 step back like you ought to and go, okay, wait, 0:29:05.160 --> 0:29:07.480 you know we were supposed to be done, you know, 0:29:07.720 --> 0:29:08.400 two months ago. 0:29:09.320 --> 0:29:10.520 When are we going to wrap this up? 0:29:11.640 --> 0:29:14.840 Right? And how has that led to things agting worse? 0:29:14.960 --> 0:29:17.560 Is it that just software culture software development culture has 0:29:17.600 --> 0:29:21.560 been focused on short term perpetually. 0:29:21.200 --> 0:29:23.160 Is the short term is part of it. 0:29:23.800 --> 0:29:27.800 Part of it is there are you know, unscrupulous developers 0:29:27.840 --> 0:29:31.480 out there that basically want to extend the length of 0:29:31.560 --> 0:29:33.160 the project so they can get more money out of. 0:29:33.160 --> 0:29:36.920 It, right right, That's that's always the case. But the 0:29:37.000 --> 0:29:39.800 other thing is that you end up with a real. 0:29:42.200 --> 0:29:43.400 A lot of times, you end up with a real 0:29:43.560 --> 0:29:46.360 lack of like long term planning and long term understanding, 0:29:46.840 --> 0:29:48.080 right right, because everybody's you. 0:29:48.080 --> 0:29:50.920 Know, some kind of thing. You know, companies are only 0:29:50.960 --> 0:29:52.280 worried about what happens next quarter. 0:29:52.440 --> 0:29:52.520 Right. 0:29:52.960 --> 0:29:54.400 If you're only worried about what's going to happen the 0:29:54.440 --> 0:29:57.920 next week or the next four weeks, the things that 0:29:58.000 --> 0:30:01.480 you look on, look at, you know, tends not to 0:30:01.680 --> 0:30:04.600 have the longer term implications that sometimes you need. Right 0:30:05.000 --> 0:30:06.480 And there are times you get close to the end 0:30:06.600 --> 0:30:10.000 and you're like, oh, you know, we didn't think about 0:30:10.040 --> 0:30:11.800 this problem yeaheah. 0:30:12.560 --> 0:30:15.240 And also if you're in a two or three week thing, 0:30:15.480 --> 0:30:18.880 you're probably not thinking even what you did last sprint like. 0:30:18.960 --> 0:30:22.520 It maybe last one, but not like two or three, 0:30:22.840 --> 0:30:23.760 two or three ones ago. 0:30:24.320 --> 0:30:27.320 Is this a problem throughout organizations of all sizes? Is 0:30:27.360 --> 0:30:30.200 this a consultancy problem. Is it everywhere? 0:30:31.800 --> 0:30:35.800 It's most places, Huh, there are, there are some places 0:30:35.880 --> 0:30:36.280 that are. 0:30:38.560 --> 0:30:44.280 Usually in startups, we're a lot more ad hoc and 0:30:44.400 --> 0:30:47.760 we're a lot more you know, focused on trying to 0:30:47.800 --> 0:30:52.440 get things done. The basically, the the the idea is 0:30:52.680 --> 0:30:56.280 the larger you get as an organization, and the more 0:30:56.360 --> 0:30:59.000 money you're throwing at it, and the more management control 0:30:59.120 --> 0:31:02.640 you want, the more of this overhead you put in place, 0:31:02.720 --> 0:31:05.000 and the more complicated things get as just as a 0:31:05.080 --> 0:31:06.479 as a management structure kind of thing. 0:31:07.400 --> 0:31:09.760 And this in the big So this is something you'd 0:31:09.800 --> 0:31:11.640 seem like a Google and an Amazon as well. 0:31:11.760 --> 0:31:13.560 Oh absolutely so. 0:31:13.760 --> 0:31:16.880 Do you do you think it has the same organizational effects. 0:31:16.600 --> 0:31:20.720 Or largely yes. 0:31:21.640 --> 0:31:27.920 The so those organizations tend to be well, those organizations 0:31:28.080 --> 0:31:29.560 historically have tended to be. 0:31:31.080 --> 0:31:33.840 Before the recent like in shitification wave. 0:31:34.760 --> 0:31:37.800 Those I'm assuming I can swear on this, Yeah, yeah, 0:31:41.280 --> 0:31:46.640 those organizations have historically been fairly more engineering driven, which 0:31:46.760 --> 0:31:51.080 means that you typically have people higher in the organization 0:31:51.400 --> 0:31:55.640 that are technical and have been programmers and who understand 0:31:55.680 --> 0:31:58.680 some of the implications, and so they tend to try 0:31:58.760 --> 0:32:01.600 at least we try to run interference with management and 0:32:01.720 --> 0:32:03.920 to try to, you know, make sure everybody's on the 0:32:03.920 --> 0:32:06.680 same page and that kind of stuff. A lot of 0:32:08.040 --> 0:32:09.760 a lot, not all, but a lot of problems can 0:32:09.840 --> 0:32:13.960 get get lessened if you have people in the organization 0:32:14.040 --> 0:32:16.160 that are at higher level whose job is not to 0:32:16.320 --> 0:32:18.960 manage people, but whose job is basically to keep track 0:32:19.040 --> 0:32:21.640 and coordinate between different groups that are doing different technical 0:32:21.720 --> 0:32:22.520 things right, to. 0:32:22.520 --> 0:32:24.680 Make sure people aren't building the same thing I'm guessing, 0:32:24.880 --> 0:32:26.960 or are building the right thing in the right way. 0:32:27.320 --> 0:32:31.480 Yeah, and that how what this group is building is 0:32:31.560 --> 0:32:33.440 going to impact what this group is building at some 0:32:33.520 --> 0:32:35.560 point in the future. And making sure that when you 0:32:35.640 --> 0:32:37.040 get to the point where those two things need to 0:32:37.040 --> 0:32:39.000 talk to each other, they're both aware enough of what 0:32:39.080 --> 0:32:40.680 the other one is doing that the two things hook 0:32:40.720 --> 0:32:41.440 together correctly. 0:32:42.720 --> 0:32:46.520 Yes, So, based on my analysis at these companies, that's 0:32:46.600 --> 0:32:51.160 definitely gone out the window. I mean, even with LLLM integrations. 0:32:51.200 --> 0:32:53.280 So there was a Johnson and Johnson story that went 0:32:53.280 --> 0:32:55.040 out Wall Street General a couple of weeks ago where 0:32:55.040 --> 0:32:58.120 it was like they had eight hundred and ninety LM 0:32:58.240 --> 0:33:02.640 project Generative AI project, of which Taitla the Pereto principle 0:33:02.680 --> 0:33:05.200 wins again ten fifteen percent of them were actually useful. 0:33:05.800 --> 0:33:07.680 And the thing that stunned me about that of them, 0:33:07.680 --> 0:33:10.840 the fact to confirmed my biases, which I love, was 0:33:10.920 --> 0:33:13.520 the fact that they were eight hundred and ninety at 0:33:13.520 --> 0:33:16.400 the fucking things and no one was like should we 0:33:16.560 --> 0:33:20.400 have this many that There was no like like selfare 0:33:20.440 --> 0:33:22.960 engineering culture that was like, hey, are we all chasing 0:33:23.040 --> 0:33:25.640 our tails? Is this useless? But it sounds like they 0:33:25.680 --> 0:33:27.840 were all focused on their little boxes. 0:33:28.160 --> 0:33:29.560 Yeah, I mean so the other thing. 0:33:29.760 --> 0:33:34.400 So understand that again greatly oversimplifying a lot of the 0:33:34.560 --> 0:33:39.080 new stuff that's happened with large language machines, large language models, 0:33:39.240 --> 0:33:44.640 and generative AI. People didn't expect, right. It was kind 0:33:44.680 --> 0:33:47.120 of a surprise when you throw a whole bunch more 0:33:47.200 --> 0:33:49.320 data at a large language model and it started spitting 0:33:49.360 --> 0:33:52.600 out text in a way that nobody really There was 0:33:52.680 --> 0:33:55.480 no like mathematical reason to expect it to be able 0:33:55.520 --> 0:33:58.360 to be as good at generating rottocomplete. 0:33:57.760 --> 0:33:58.360 Stuff as it is. 0:33:58.440 --> 0:34:02.880 It was, right, And so there's this belief that if 0:34:03.040 --> 0:34:06.200 we did the thing and we unexpectedly got more than 0:34:06.280 --> 0:34:08.279 we asked for, if we do more of the thing, 0:34:08.400 --> 0:34:11.000 maybe we'll unexpectedly get more of what we wanted, right 0:34:11.640 --> 0:34:14.680 that hasn't seemed to really pan out the last couple 0:34:14.719 --> 0:34:19.239 of years from what I can see, but that we 0:34:19.400 --> 0:34:21.640 don't really understand enough about this to know whether it's 0:34:21.680 --> 0:34:23.600 going to work, So we might as well throw spaghetti 0:34:23.600 --> 0:34:25.279 at the wall and see if it sticks, because it might. 0:34:26.640 --> 0:34:29.719 Kind of mentality is kind of pervasive at the moment, 0:34:30.520 --> 0:34:32.719 and everybody's there's a lot of fomo. There's a lot 0:34:32.800 --> 0:34:35.400 of like, you know, well, our competitors are probably doing this, 0:34:35.600 --> 0:34:38.880 and so we don't want to get left behind. It 0:34:39.080 --> 0:34:42.000 kind of reminds me of the rumors that they talked 0:34:42.000 --> 0:34:44.120 about back in the eighties when the CIA was doing 0:34:44.160 --> 0:34:46.880 all this psychic research, because supposedly the Russians were doing 0:34:46.920 --> 0:34:49.640 psychic research and it was all complete crap, but both 0:34:49.680 --> 0:34:52.200 sides were convinced that the other side was making some progress, 0:34:52.200 --> 0:34:54.520 and so everybody was dumping a ton of money into it. 0:34:55.080 --> 0:35:14.799 LMM Kultrum exactly. Yes, the title of the episode, So, okay, 0:35:15.440 --> 0:35:20.040 koltr aside, is this something you're seeing in software development though, 0:35:20.400 --> 0:35:22.400 because I know I've seen that in management or it's 0:35:22.400 --> 0:35:23.759 just going to like shut the ship in there. This 0:35:23.880 --> 0:35:26.320 seems like it's an important thing, right or is this 0:35:26.480 --> 0:35:28.080 Are you seeing it within software development? 0:35:29.080 --> 0:35:30.959 So I am seeing it within. 0:35:32.280 --> 0:35:36.879 Software planning, right, So when managers are sitting down and saying, Okay, 0:35:36.920 --> 0:35:38.400 we need to build this new thing, we need to 0:35:38.440 --> 0:35:40.439 create a new group, we need to split this group apart, 0:35:40.640 --> 0:35:42.120 we need to decide what our headcount is going to 0:35:42.160 --> 0:35:44.800 be for next year, there's a lot of okay, and 0:35:45.000 --> 0:35:46.320 what do we think the AI is going to do 0:35:46.400 --> 0:35:46.719 next year? 0:35:46.760 --> 0:35:48.160 And how many headcount do we think that's going to 0:35:48.200 --> 0:35:48.520 save us? 0:35:48.520 --> 0:35:50.880 In that kind of thing, right, There are some companies 0:35:51.239 --> 0:35:56.879 do a Lingo is one, Klarna is one OP sorry, BP, 0:35:57.400 --> 0:36:00.680 the former British Petroleum of what last year had a 0:36:00.719 --> 0:36:02.520 thing where they said they were cutting seventy percent of 0:36:02.560 --> 0:36:04.440 their contract software developers. 0:36:04.480 --> 0:36:06.640 And in most of these they've kind of rolled them 0:36:06.680 --> 0:36:07.279 back as well. 0:36:07.680 --> 0:36:09.480 And I don't think dual lingo has yet. 0:36:10.760 --> 0:36:12.640 This is just being unfit to you. They like a 0:36:12.760 --> 0:36:15.839 day ago, really just like that would kind of It's 0:36:15.880 --> 0:36:18.320 so funny. It's so funny. It's so funny that this 0:36:18.480 --> 0:36:20.680 just keeps happening in exactly the same way. It's like, oh, 0:36:20.719 --> 0:36:23.680 what a surprise, human beings to do stuff. Yeah, but 0:36:23.800 --> 0:36:25.239 it kind of gets back to I think what you've 0:36:25.239 --> 0:36:27.759 said about everything with l lams. It's like you can 0:36:27.880 --> 0:36:30.719 teach something to say, Yeah, I think the right The 0:36:30.760 --> 0:36:32.359 thing you're looking for is this, but you can't teach 0:36:32.400 --> 0:36:34.480 it context. And that's been a point you've made again 0:36:34.480 --> 0:36:37.400 and again, Like it seems the job of a software 0:36:37.440 --> 0:36:41.439 engineer is highly contextual, unless you're like in the earlier days. 0:36:41.840 --> 0:36:43.080 Yeah, and I like it. 0:36:43.120 --> 0:36:45.960 It sometimes to the Memento guy from the Momento movie, right, 0:36:46.040 --> 0:36:48.640 where like can't form long term memories. Then do you 0:36:48.800 --> 0:36:50.360 do you really want the Momento guy to be the 0:36:50.400 --> 0:36:53.520 person that's building the software that makes the seven thirty 0:36:53.560 --> 0:36:59.480 seven max be able to compensate for its control input. Yeah. 0:37:00.280 --> 0:37:03.719 Well, the thing is, though, with that argument, they would argue, 0:37:03.760 --> 0:37:05.799 and I know that there is a better argument here. 0:37:05.880 --> 0:37:07.640 They would argue, well, what if we just give it 0:37:07.719 --> 0:37:10.600 everything that's ever happened? What if we just show every 0:37:10.680 --> 0:37:13.400 single thing we've ever done in GitHub? Surely then it 0:37:13.400 --> 0:37:14.240 would understand. 0:37:16.360 --> 0:37:20.640 So the what I have seen from the papers that 0:37:20.719 --> 0:37:25.720 I have read is that lllms have a basically squishy 0:37:25.760 --> 0:37:28.640 middle context problem kind of the way that you do right. So, 0:37:28.760 --> 0:37:31.680 if somebody gives you a big document to read or 0:37:31.719 --> 0:37:33.680 a big long documentary to watch or something, and then 0:37:33.680 --> 0:37:35.480 they ask you questions. What they're going to find is 0:37:35.560 --> 0:37:37.520 that you remember a lot more from the beginning of 0:37:37.560 --> 0:37:39.040 that and the end of that than you do from 0:37:39.080 --> 0:37:41.920 the middle of that. Right, and lllms have the same 0:37:42.000 --> 0:37:43.640 kind of problem. Right, And the other problem that the 0:37:43.760 --> 0:37:46.759 LMS seem to have is that when you give them 0:37:46.880 --> 0:37:49.920 a whole bunch of instructions, just instructions, polled on instructions, 0:37:49.960 --> 0:37:54.760 pulled on instructions, they can either get confused and forget 0:37:54.840 --> 0:37:57.560 some of the instructions, or they deadlock, or they just 0:37:57.880 --> 0:38:00.160 start going, Okay, I can't satisfy all of these I'm 0:38:00.160 --> 0:38:01.759 not even going to bother to satisfy any of them, 0:38:02.480 --> 0:38:05.840 or they'll pick one or two. The fact that you 0:38:06.000 --> 0:38:10.680 can take a million tokens and you can stick that 0:38:10.960 --> 0:38:16.080 in the memory block that the the GPU is going 0:38:16.120 --> 0:38:21.480 to process, doesn't necessarily believe, doesn't necessarily mean that all 0:38:21.719 --> 0:38:24.120 of the tokens in that memory block are actually going 0:38:24.200 --> 0:38:25.920 to be treated equally and going. 0:38:25.840 --> 0:38:34.240 To be understood. Right in theory, maybe if you could. 0:38:35.840 --> 0:38:40.399 Train your if you could like custom train an LLM 0:38:41.120 --> 0:38:43.680 and modify all of its weights based on exactly what 0:38:43.920 --> 0:38:46.560 your stuff was, and do that like day after day 0:38:46.600 --> 0:38:50.160 after day after day. As things changed, you would theoretically 0:38:50.280 --> 0:38:53.680 get better. I still don't think it would be you know, 0:38:54.400 --> 0:38:57.400 I still think would understand the context as well, but 0:38:57.520 --> 0:38:59.880 that would be ridiculously expensive. 0:39:00.360 --> 0:39:05.480 Yeah, and at that point you could train a person, yes. 0:39:05.480 --> 0:39:07.400 I mean the person would probably be more annoying. So 0:39:07.920 --> 0:39:10.880 that's I mean the point. I mean, a lot of 0:39:10.960 --> 0:39:14.200 this is seems to be really you know, we don't 0:39:14.280 --> 0:39:16.799 like dealing with the Prima Donna programmer kind of thing, 0:39:16.920 --> 0:39:22.480 right that there's this you know, I mean not just programmers, right, 0:39:22.520 --> 0:39:25.000 we don't also don't want to deal with the Prima 0:39:25.040 --> 0:39:28.080 Donna reporters or the Prima Donna illustrators or just want 0:39:28.080 --> 0:39:28.480 to get rid. 0:39:28.440 --> 0:39:31.760 Of these people. Right. He's annoying. They ask for stuff, 0:39:31.800 --> 0:39:32.719 they want money. 0:39:33.280 --> 0:39:37.719 Yeah, and days off and sickly even you know, healthcare, and. 0:39:38.280 --> 0:39:42.960 It's just disgusting, how Dad. It's so it's frustrating as 0:39:43.000 --> 0:39:47.279 well because across software development and everything, but especially with 0:39:47.320 --> 0:39:50.840 self A developers, it feels just very insulting because it 0:39:50.960 --> 0:39:53.839 doesn't seem like this stuff. Actually, here's a better question. 0:39:54.560 --> 0:39:58.239 Have you seen much of an improvement with like one 0:39:58.280 --> 0:40:00.200 to oh three like these reasoning Do you I think 0:40:00.440 --> 0:40:03.880 the reasoning models change things for the beta. If so, how. 0:40:05.239 --> 0:40:12.120 So a little that they don't make as many stupid mistakes. 0:40:13.239 --> 0:40:15.399 It is basically what it what what it boils down 0:40:15.440 --> 0:40:18.400 to going back to your your first thing, though, right, 0:40:18.520 --> 0:40:21.359 I mean so. There was a piece, actually a couple 0:40:21.360 --> 0:40:24.360 of pieces recently. One of them was about, you know, 0:40:24.440 --> 0:40:26.440 tech workers are just like the rest of us, They're miserable. 0:40:27.040 --> 0:40:29.239 There I'll I'll give you blinks to these. The other 0:40:29.320 --> 0:40:32.440 one was a Corey doctor opiece that was like the 0:40:32.760 --> 0:40:37.000 future of Amazon coders is the present of Amazon warehouse 0:40:37.040 --> 0:40:42.160 workers or vice versa. There's there's a lot of there 0:40:42.200 --> 0:40:46.520 has been a lot of deference given to software developers 0:40:47.680 --> 0:40:51.560 over time, because you know, we have been kind of 0:40:51.640 --> 0:40:54.439 the engine that's made a lot of the last twenty 0:40:54.480 --> 0:40:58.640 thirty years work, and there's a desire to make that 0:40:58.960 --> 0:41:01.680 not so anymore, and to make us just as interchangeable 0:41:01.719 --> 0:41:06.399 as everybody else. I guess, you know, from a from 0:41:06.440 --> 0:41:09.080 a economic standpoint, I kind of don't blame them. 0:41:09.160 --> 0:41:10.960 I understand why they're trying to do what they're doing. 0:41:11.040 --> 0:41:13.440 I don't I mean, I don't think that the warehouse 0:41:13.440 --> 0:41:15.520 workers should be treated the way the warehouse workers are treated, 0:41:15.920 --> 0:41:17.920 you know, much less everybody else gets treated that way. 0:41:19.080 --> 0:41:23.920 And it's been a lot worse since the giant layoffs 0:41:24.000 --> 0:41:28.799 at Twitter now X. When that happened and the thing 0:41:28.960 --> 0:41:33.640 didn't crash and completely burn like everybody was or not everybody, 0:41:33.680 --> 0:41:36.520 but a lot of people were expecting it to, the 0:41:37.320 --> 0:41:41.720 the sentiment became, well, maybe all this, all these software 0:41:41.719 --> 0:41:44.400 developers aren't as important as they you know, we've always 0:41:44.400 --> 0:41:44.920 thought they were. 0:41:46.040 --> 0:41:49.600 And you know, we will see over time what the 0:41:50.000 --> 0:41:52.560 end result of that is. My guess is it's going 0:41:52.640 --> 0:41:53.600 to be end up being a mess. 0:41:54.360 --> 0:41:57.880 But you know, I'm a I'm a software developer, right, 0:41:57.920 --> 0:42:01.400 I'm gonna it behooves me for it to be a mess, right, 0:42:01.560 --> 0:42:04.080 So it might just be my bias that's getting in 0:42:04.120 --> 0:42:04.399 the way. 0:42:04.719 --> 0:42:06.920 I actually I think that you're right though, because I 0:42:07.000 --> 0:42:10.359 remember back in twenty twenty one and onward the kind 0:42:10.400 --> 0:42:13.759 of post remote work, the remote works. There was the 0:42:13.800 --> 0:42:16.640 whole anti remote work push, but there was the whole 0:42:16.719 --> 0:42:19.080 quiet quitting and things like that. That's twenty twenty two 0:42:19.160 --> 0:42:22.640 where it's like software engineering, they just they expect to 0:42:22.680 --> 0:42:24.960 be treated so well because twenty twenty one's all the 0:42:25.080 --> 0:42:28.760 insane hiring, right. You saw tech companies like parking software workers. 0:42:29.239 --> 0:42:31.319 I think that played into it as well, where all 0:42:31.400 --> 0:42:34.160 of these companies who chose to pay these software engineers, 0:42:34.200 --> 0:42:36.680 they were the ones that made the offers, got pissed 0:42:36.719 --> 0:42:39.680 off that they'd done. Someone thought we should cut all 0:42:39.800 --> 0:42:44.360 labor down to size, and then along comes coding. Almost 0:42:44.400 --> 0:42:46.560 makes me wonder if most of these venture capitalists talking 0:42:46.560 --> 0:42:50.720 about this don't know how to code themselves. Yeah, gotta wonder. 0:42:51.120 --> 0:42:54.960 I don't know many that do. Yeah, I know some 0:42:55.120 --> 0:42:56.560 that have at some point. 0:42:56.760 --> 0:43:01.279 But the best thing it's at some point it's like 0:43:01.360 --> 0:43:05.000 they're not part of modern software development culture, which I 0:43:05.080 --> 0:43:07.560 know sounds kind of wanky, but I mean, just how 0:43:07.640 --> 0:43:11.520 an organization builds software feels like something they should know. 0:43:11.920 --> 0:43:13.200 But then again, they don't know how to build a 0:43:13.200 --> 0:43:15.879 real organization ethos. Who the fuck? Yeah? 0:43:16.320 --> 0:43:18.360 Well, I mean, honestly a lot of it. 0:43:20.000 --> 0:43:24.280 I've been in organizations that VC's basically killed, right because 0:43:24.800 --> 0:43:27.800 you know, we built a thing. That thing was, you know, 0:43:28.719 --> 0:43:32.880 a reasonable business, But vcs don't want a reasonable business. 0:43:33.000 --> 0:43:34.759 They want either one hundred ex return or they want 0:43:34.800 --> 0:43:38.200 to tax write off, and they don't want anything in between, right, yeah, 0:43:38.600 --> 0:43:41.000 So I mean what what they're looking for is really 0:43:41.080 --> 0:43:43.799 I mean, they're not trying to run a regular business, right, 0:43:43.840 --> 0:43:46.600 They're not trying to do the normal process. They're trying 0:43:46.640 --> 0:43:49.440 to either you know, hit one out of the park 0:43:49.880 --> 0:43:53.480 or throw it away and move on. And so they're 0:43:53.560 --> 0:43:55.640 they're the rules for them are different because what they're 0:43:55.640 --> 0:43:57.480 trying to accomplish is not what the rest of us. 0:43:57.400 --> 0:43:58.920 Are trying to accomplish. As a general rule. 0:44:00.080 --> 0:44:02.440 The theme of the fucking show, it's just like, it's 0:44:02.560 --> 0:44:04.680 just like you have these people that don't code saying 0:44:04.719 --> 0:44:07.279 how cod is should code, like Dario amat Day the 0:44:07.360 --> 0:44:09.719 other day saying that this year we're going to have 0:44:09.800 --> 0:44:14.160 the first one person software company with a billion dollars 0:44:14.400 --> 0:44:17.720 revenue or something like that, and it's just I feel 0:44:17.760 --> 0:44:20.120 like there are some people who should not be allowed 0:44:20.160 --> 0:44:23.800 to speak as much sometimes, but it's just frustrating and insulting. 0:44:23.840 --> 0:44:25.960 And it's but now that you've got me thinking about it, 0:44:26.080 --> 0:44:28.080 it does feel like this is an attempt to reset 0:44:28.160 --> 0:44:31.279 the labor market finally coming for software developers. And I 0:44:31.320 --> 0:44:33.040 don't mean finally in a good way, right. 0:44:33.719 --> 0:44:36.000 I mean, it feels like that being in the being 0:44:36.080 --> 0:44:39.319 in that organ being in that industry at the moment, 0:44:39.400 --> 0:44:40.400 it really feels like that. 0:44:40.960 --> 0:44:41.960 Is it scary right now? 0:44:43.120 --> 0:44:44.080 Is it scary right now? 0:44:45.360 --> 0:44:50.000 Not for me because I'm old enough to be semi retired, right, 0:44:50.840 --> 0:44:53.400 But I mean, I've been talking to a lot of folks. 0:44:54.040 --> 0:44:57.279 I've been having a interviewing how much folks that are 0:44:57.440 --> 0:45:00.000 that are listeners from my channel and kind of trying 0:45:00.120 --> 0:45:02.839 to get a feel for what's going on. And I've 0:45:02.920 --> 0:45:05.520 talked to folks that are you know, like I said that, 0:45:05.840 --> 0:45:07.160 I talked to some folks that were like, you know, 0:45:07.200 --> 0:45:09.319 I work for a big bank. They're cramming copilot, dinner 0:45:09.360 --> 0:45:12.120 throat or eeveryoneted or not. I've talked to some folks 0:45:12.200 --> 0:45:13.919 that are like, every time I sit down with my boss, 0:45:13.960 --> 0:45:15.440 I'm thinking that, you know, this is going to be 0:45:15.480 --> 0:45:17.279 the day that I'm going to find out that my 0:45:17.400 --> 0:45:19.360 group is getting cut the way the other three groups 0:45:19.400 --> 0:45:21.560 in the company is getting cut. 0:45:22.000 --> 0:45:22.880 There's a lot of. 0:45:24.840 --> 0:45:29.640 Artificial productivity requirement increases kind of thing, which is like 0:45:30.000 --> 0:45:34.240 like one, just you know, we you know, we expect 0:45:34.280 --> 0:45:37.719 more tickets closed per you know, two week period than 0:45:38.040 --> 0:45:39.960 you know we've had before because we were giving you 0:45:40.040 --> 0:45:42.080 this AI now, so you ought to be more productive 0:45:42.160 --> 0:45:42.719 that kind of thing. 0:45:43.600 --> 0:45:46.480 Would the ticket necessarily be something that you just write 0:45:46.520 --> 0:45:48.120 toad for it more than just. 0:45:48.760 --> 0:45:51.640 Well, so generally it's more than just that. But but 0:45:51.840 --> 0:45:53.320 generally the ticket. 0:45:54.120 --> 0:45:56.120 That's kind of the way that we track the work 0:45:56.200 --> 0:45:58.920 that we do in a lot of organizations, right, And 0:45:59.120 --> 0:46:01.840 some tickets are like, I'm building a new thing, and 0:46:01.920 --> 0:46:04.320 those are kind of easier to predict. And some tickets 0:46:04.360 --> 0:46:07.160 are this thing isn't behaving right, go figure out where 0:46:07.160 --> 0:46:09.080 the bug is. And those are a lot harder to predict, 0:46:09.560 --> 0:46:12.919 but they have these things. Agile has this thing called 0:46:12.920 --> 0:46:15.800 a velocity graph where basically you see how many tickets 0:46:15.840 --> 0:46:18.400 per person get closed over time, and people want to 0:46:18.400 --> 0:46:22.240 see the slope of that line change because they're giving 0:46:22.280 --> 0:46:22.640 you AI. 0:46:24.320 --> 0:46:26.320 And I'm guessing the people telling you to change that 0:46:26.440 --> 0:46:27.640 don't know what they're talking about. 0:46:28.200 --> 0:46:29.279 That seems to be the case. 0:46:29.800 --> 0:46:35.080 Great, so I mean the good news in theory, right, 0:46:35.200 --> 0:46:36.959 I don't know to what extent this is going to happen, 0:46:37.040 --> 0:46:41.120 but in theory, if they keep telling people, you know, 0:46:41.560 --> 0:46:43.600 that slope of that line should be changing because you 0:46:43.680 --> 0:46:46.280 have AI. Now, over time, if we see the slope 0:46:46.320 --> 0:46:50.840 of that line not changing though, right, then theoretically it 0:46:50.920 --> 0:46:54.680 will be proof that the AI is not providing the 0:46:54.760 --> 0:46:58.399 return that people expect it. Well, you're not using it, right, Well, yes, 0:46:58.480 --> 0:46:59.920 there's always that you're not prompting it, right. 0:47:00.040 --> 0:47:04.640 That is that is basically what I am people. One 0:47:04.680 --> 0:47:06.560 of the many reasons what you want is like, I 0:47:06.680 --> 0:47:09.239 want to have people that actually code on to talk 0:47:09.280 --> 0:47:12.480 about this stuff, because it's really easy as a layman 0:47:12.560 --> 0:47:15.680 myself and for others to just be like, oh but 0:47:15.880 --> 0:47:18.879 this does replace coding, right, and it does? It sounds 0:47:18.920 --> 0:47:20.440 like it really doesn't. 0:47:20.520 --> 0:47:21.160 Like it can help. 0:47:21.200 --> 0:47:24.000 It can be like a force multiplier to an extent, 0:47:24.120 --> 0:47:27.680 but even past the initial steps, it just isn't there. 0:47:28.200 --> 0:47:31.600 Well, I mean, so the best analogy I've always found 0:47:31.800 --> 0:47:35.319 to writing code is actually just writing, right. I Mean, 0:47:35.400 --> 0:47:37.520 you can get chat GBT to spit out a few 0:47:37.560 --> 0:47:42.160 paragraphs for you, right, but you know, you end up with, 0:47:42.480 --> 0:47:44.399 you know, the legal briefs that have the story that's 0:47:44.440 --> 0:47:47.200 made up or the you know, just things that aren't 0:47:47.320 --> 0:47:49.759 connected to reality or stuff that you know, when people 0:47:49.800 --> 0:47:53.200 read them, they're like, I mean, you you can tell 0:47:53.200 --> 0:47:55.800 the difference between AI slot generated you know, like the 0:47:56.200 --> 0:48:01.000 stupid the insert from the Chicago Sun, Yeah, yeah, the 0:48:01.040 --> 0:48:04.200 Philadelphia Inquirer. You know, all the books, all the things 0:48:04.239 --> 0:48:06.279 you can do this summer, right that like made up 0:48:06.320 --> 0:48:07.840 books and all that kind of I mean, like, but 0:48:08.000 --> 0:48:11.279 even even the articles that weren't the ones that we're 0:48:11.320 --> 0:48:13.759 making up stuff. You read the you know, this is 0:48:13.800 --> 0:48:15.200 what's going to be happening this summer. This is what 0:48:15.239 --> 0:48:16.520 the weather's going to be like or whatever. And you're 0:48:16.560 --> 0:48:20.480 reading and you're like this this there's no like insight here, 0:48:20.600 --> 0:48:24.359 there's no thought here, there's no you know, there's nothing 0:48:24.440 --> 0:48:26.160 in here that I get to the end of this. 0:48:26.239 --> 0:48:28.080 I've read the whole thing. I understand the whole thing, 0:48:28.160 --> 0:48:30.240 but I don't have anything I can walk away. 0:48:30.040 --> 0:48:34.080 With, right and I AI agents aren't coming along to 0:48:34.200 --> 0:48:37.040 replace software. But that you're not scared of Devin. 0:48:37.719 --> 0:48:42.080 I am not scared of Devon, so I well, actually 0:48:42.280 --> 0:48:44.160 I kind of am. I am scared that Devon is 0:48:44.200 --> 0:48:46.000 going to make a mess of things and then more 0:48:46.040 --> 0:48:47.680 things are going to get hacked, and that's going to 0:48:47.760 --> 0:48:49.279 end up being worse for everybody. 0:48:48.960 --> 0:48:50.360 On the unit. Right, how would it do that? 0:48:51.360 --> 0:48:53.200 By I mean like we were talking about before, right, 0:48:53.320 --> 0:48:56.400 So when you write code that isn't secure, right, and 0:48:56.520 --> 0:48:59.279 you write code that you know uses a library that's 0:48:59.320 --> 0:49:01.279 got an old version of a thing that they that 0:49:01.360 --> 0:49:03.160 there's a known bug in it, but you don't bother 0:49:03.280 --> 0:49:05.719 to check to see if there's a fix for that bug. 0:49:06.239 --> 0:49:09.000 Or you don't use best practices when it comes to 0:49:09.040 --> 0:49:12.080 writing code and that kind of thing, or you don't 0:49:12.120 --> 0:49:15.799 think about the the kinds of maintainability issues that you're 0:49:15.840 --> 0:49:18.279 going to have, and you do things like you ship 0:49:18.360 --> 0:49:21.359 out code in a in an Internet of things thing 0:49:21.400 --> 0:49:27.560 a light bulb right, or Internet Wi Fi router that 0:49:27.760 --> 0:49:30.279 cannot be patched over the Internet that has a bug 0:49:30.360 --> 0:49:32.960 in it, right, And now it's like that thing is 0:49:33.000 --> 0:49:35.480 going to have a bug in it forever, and you're 0:49:35.480 --> 0:49:37.440 gonna have to find all the ones on the on 0:49:37.560 --> 0:49:40.880 the earth and turn them off before someone's not going 0:49:40.920 --> 0:49:42.040 to be able to take them and be able to 0:49:42.080 --> 0:49:44.080 hack them and use them to attack somebody else from there. 0:49:44.719 --> 0:49:47.000 I mean, IoT is a huge probe low. Oh yeah, 0:49:47.080 --> 0:49:50.160 but the cheap ones have like the spywab stuff and 0:49:50.640 --> 0:49:52.319 panto mining it. 0:49:52.480 --> 0:49:54.719 Just but yeah, the ones, the ones that have they 0:49:54.760 --> 0:49:57.279 have like really nasty vulnerabilities, and they have no way 0:49:57.320 --> 0:50:01.000 of being updated once they leave the factory, right, and 0:50:01.080 --> 0:50:03.200 it's just as long as they're out there, they're going 0:50:03.239 --> 0:50:05.360 to be a problem literally for everybody on the internet. 0:50:06.080 --> 0:50:11.200 Jesus, Well, what can to wrap us up? What can 0:50:11.480 --> 0:50:14.759 a new engineer, someone new to software development? What can 0:50:14.840 --> 0:50:16.839 they learn right now? You've kind of done a video 0:50:16.880 --> 0:50:18.000 on this, but I think it's a good place to 0:50:18.040 --> 0:50:20.920 wrap us up. What can they start learning to actually 0:50:20.960 --> 0:50:22.640 get ahead, to actually prepare for all of this. 0:50:23.440 --> 0:50:26.800 That's a really good question. So you can't these days, 0:50:27.920 --> 0:50:31.920 you can't really be able to be an engineer. You 0:50:31.960 --> 0:50:36.200 can't get hired as an engineer without some ability to 0:50:36.320 --> 0:50:40.520 talk about being able to do prompts and use you know, 0:50:40.719 --> 0:50:42.680 some kind of AI code editor or that kind of thing. 0:50:43.200 --> 0:50:45.200 It's just an expectation of the job. Now, whether it 0:50:45.239 --> 0:50:48.920 should be or not a different thing. The I mean, 0:50:49.120 --> 0:50:52.560 like I said before, there are situations where you tell 0:50:52.600 --> 0:50:54.200 it what you want and it will type faster than 0:50:54.200 --> 0:50:57.200 you possibly can. So you know that's not necessarily bad. 0:50:57.239 --> 0:51:01.440 You need to understand that you need to figure out Well, okay, 0:51:01.760 --> 0:51:03.960 I'll get back to something else. You need to figure 0:51:03.960 --> 0:51:07.359 out basically how to test the thing, right, So how 0:51:07.400 --> 0:51:09.240 do you make sure that the code that it spits 0:51:09.280 --> 0:51:12.520 out does what you meant it to do. And what 0:51:12.600 --> 0:51:14.600 I'm expecting is that we're going to spend more time 0:51:14.920 --> 0:51:18.319 thinking about testing and thinking more about, you know, trying 0:51:18.360 --> 0:51:20.279 to find exceptions and that kind of thing than we 0:51:20.400 --> 0:51:22.520 have in the past, because the code that's actually being 0:51:22.600 --> 0:51:24.440 generated is going to be less likely to be quality 0:51:24.480 --> 0:51:28.440 than it was in the past. Right. The problem is 0:51:29.680 --> 0:51:32.320 it has become the case in the in the programming 0:51:32.360 --> 0:51:34.799 industry that the things you need to do to get 0:51:34.840 --> 0:51:37.240 through the interview to get hired have very little resemblance 0:51:37.280 --> 0:51:38.759 to the things that you actually do on the job 0:51:38.840 --> 0:51:41.239 that you need to actually do a good job. And 0:51:41.360 --> 0:51:44.279 so that's a whole different We could probably have a 0:51:44.360 --> 0:51:47.360 whole other podcast episode just about the interviewing problem. 0:51:48.280 --> 0:51:53.759 But the main thing right now, it's so right now 0:51:53.920 --> 0:51:55.120 the whole hiring thing. 0:51:55.200 --> 0:51:57.520 And this isn't I don't think true for just programmers, 0:51:57.560 --> 0:52:01.400 but it's especially true for programmers. Is all you know, 0:52:02.280 --> 0:52:05.520 bots that customize your resume and write a custom color 0:52:05.640 --> 0:52:08.880 cover letter and then send them over to the submit 0:52:08.960 --> 0:52:11.520 the thing to the bot that's screening the resume and 0:52:11.600 --> 0:52:14.600 screening right, and that getting it to the point where 0:52:14.600 --> 0:52:16.080 you can actually talk to a human is a nightmare 0:52:16.120 --> 0:52:18.000 right now, So the whole hiring system is kind of broken, 0:52:19.920 --> 0:52:21.759 so that the actually getting to the point where you 0:52:21.800 --> 0:52:24.759 can get hired is a nightmare at the moment. But 0:52:25.400 --> 0:52:29.320 the thing that you can do is figure out what 0:52:29.640 --> 0:52:32.399 kinds of things that AI are good at is good at. 0:52:33.280 --> 0:52:34.759 And one of the things that AI is pretty good 0:52:34.760 --> 0:52:37.680 at is things that don't matter as much, right. So, 0:52:37.920 --> 0:52:41.200 like you know, AI can pick the layout of a 0:52:41.280 --> 0:52:43.319 site potentially right, and you could have it picked two 0:52:43.400 --> 0:52:45.759 or three of them, and you can basically do what's 0:52:45.800 --> 0:52:47.880 called an A B test, and you can randomly assign 0:52:47.920 --> 0:52:49.200 people to it. You can figure out which one of 0:52:49.200 --> 0:52:50.880 them performs better, and you can throw the rest. 0:52:50.760 --> 0:52:51.160 Of the money. 0:52:51.320 --> 0:52:53.359 And even then at some point you will probably want 0:52:53.400 --> 0:52:54.560 the design customized. 0:52:55.080 --> 0:52:56.160 Yeah, I mean, but but. 0:52:58.640 --> 0:53:00.799 I think there will be a lot of things where 0:53:02.160 --> 0:53:05.839 people can kind of get something that's kind of good 0:53:05.960 --> 0:53:09.759 enough to get started right, right. And I think that 0:53:09.960 --> 0:53:11.719 to some extent this is going to be kind of 0:53:11.760 --> 0:53:14.319 a boon for the industry in the longer term where 0:53:14.520 --> 0:53:17.719 somebody who can't program right now, but who has some 0:53:17.840 --> 0:53:19.719 idea of kind of what they want can do like 0:53:19.800 --> 0:53:22.880 a vibe coding thing. They can validate that the market 0:53:23.040 --> 0:53:27.279 that they want to try to attack exists, right, and 0:53:27.360 --> 0:53:29.239 that people want to use the kind of thing that 0:53:29.320 --> 0:53:31.120 they built, and then they can bring in somebody to 0:53:31.160 --> 0:53:34.160 actually build it, right, you know what I mean. And 0:53:34.280 --> 0:53:36.360 those kinds of things wouldn't necessarily have been able to 0:53:36.480 --> 0:53:39.319 happen in the complete absence of AI. 0:53:39.400 --> 0:53:41.040 So it's not, I don't think, completely useless. 0:53:41.719 --> 0:53:44.840 And there there's times when as a as a developer, 0:53:45.000 --> 0:53:47.440 there are things that we're not good at, like you know, 0:53:47.520 --> 0:53:49.640 writing marketing copy and that kind of stuff that if 0:53:49.680 --> 0:53:51.680 we're trying to do a project for ourselves, you know, 0:53:51.760 --> 0:53:53.960 a lot of that stuff we can just outsource to 0:53:54.000 --> 0:53:56.680 the AI because it's not the thing that keeps the 0:53:57.200 --> 0:53:59.759 project from actually breaking and getting hacked in that kind 0:53:59.760 --> 0:54:02.040 of thing, right. So it's kind of like there's this 0:54:02.160 --> 0:54:04.839 concept where you need to keep the things that are 0:54:04.960 --> 0:54:07.759 part of your competitive advantage in house, and everything else 0:54:07.840 --> 0:54:10.080 you can kind of outsource to somebody else. The kinds 0:54:10.080 --> 0:54:11.719 of things you can outsource to somebody else are the 0:54:11.760 --> 0:54:13.239 kinds of things that you potentially you could throw an 0:54:13.239 --> 0:54:14.440 AI at because they're. 0:54:14.239 --> 0:54:17.400 Not even even then it's like, it doesn't seem like 0:54:17.480 --> 0:54:20.520 that's a ton of things right now or will. 0:54:20.440 --> 0:54:25.120 Be again, it's the so it's basically two things. 0:54:25.160 --> 0:54:27.840 It's things that where the quality of the thing doesn't 0:54:27.880 --> 0:54:32.120 matter really, right, which every business has those kinds of things, right, 0:54:32.400 --> 0:54:36.280 And they're the kinds of things where you can define 0:54:36.320 --> 0:54:39.520 a metric that you can test the AI against and 0:54:39.640 --> 0:54:41.080 let it try over and over and over and over 0:54:41.080 --> 0:54:42.440 and over again until it gets to the point where 0:54:42.440 --> 0:54:43.000 it's good enough. 0:54:43.960 --> 0:54:44.160 Yeah. 0:54:44.200 --> 0:54:47.200 Right, So if your metric is more people click on 0:54:47.320 --> 0:54:51.640 this button than the button before, right, then you can 0:54:51.719 --> 0:54:53.560 have the AI create a whole bunch of different ways 0:54:53.600 --> 0:54:56.719 to skin that button, right, and then you can say, okay, 0:54:56.840 --> 0:54:58.560 so the one that tested best is the one we're 0:54:58.560 --> 0:55:00.279 going to keep. That's the thing you can throw an 0:55:00.280 --> 0:55:03.000 AI at, right, because you've got a well defined way 0:55:03.040 --> 0:55:05.680 of checking in no telling how long it's going to take, 0:55:05.880 --> 0:55:07.360 but you have a well defined way of checking to 0:55:07.400 --> 0:55:08.520 see if it's working right or not. 0:55:09.480 --> 0:55:12.760 So yeah, I mean, for years I've had the theory 0:55:12.840 --> 0:55:15.920 that this industry was a twenty to twenty five billion 0:55:15.960 --> 0:55:19.520 dollar total addressable market pretending to be a trillion dollar one. 0:55:19.719 --> 0:55:21.759 And everything you're saying really is just it's like you're 0:55:21.800 --> 0:55:24.919 describing things like platform as a service. They like like, yeah, 0:55:25.160 --> 0:55:28.440 things that you use in tandem with very real people 0:55:28.480 --> 0:55:29.720 in intentional ideas. 0:55:31.200 --> 0:55:33.839 Yeah, this is I don't see a world in which 0:55:33.960 --> 0:55:37.160 this is a we replace all the humans. You know 0:55:37.560 --> 0:55:40.560 that the whole Like, you know, this is going to 0:55:40.640 --> 0:55:42.919 displace eighty percent of the white color workers in the world. 0:55:43.600 --> 0:55:47.839 I just you know that the only the only people 0:55:47.880 --> 0:55:50.759 that are really going to be replaced anytime soon are 0:55:50.840 --> 0:55:54.239 people that either weren't doing a great job to start with, 0:55:54.560 --> 0:55:59.080 or people whose bosses don't understand what they were doing 0:55:59.360 --> 0:56:01.560 to the point the boss thought that what they were 0:56:01.600 --> 0:56:04.200 doing mattered. And my guess is that there's going to 0:56:04.239 --> 0:56:06.319 be regret at that point and that at some point 0:56:06.360 --> 0:56:07.759 they're gonna have to bring those people back. 0:56:08.840 --> 0:56:12.480 Well, Carle, this has been such a wonderful conversation. Where 0:56:12.480 --> 0:56:13.239 can people find you? 0:56:14.040 --> 0:56:16.600 I am Internet of Bugs at YouTube is probably the 0:56:16.640 --> 0:56:19.440 easiest place to find me, and then there are links 0:56:19.560 --> 0:56:21.359 on that channel to point at other things. 0:56:21.880 --> 0:56:23.680 And you've been listening to me at Zichron you've been 0:56:23.680 --> 0:56:26.839 listening to Better Offline. Thank you everyone for listening, and yeah, 0:56:26.880 --> 0:56:27.759 we'll catch you next week. 0:56:35.920 --> 0:56:37.440 Thank you for listening to Better Offline. 0:56:37.719 --> 0:56:40.120 The editor and composer of the Better Offline theme song 0:56:40.200 --> 0:56:42.839 is Mattawsowski. You can check out more of his music 0:56:42.880 --> 0:56:46.360 and audio projects at Mattasowski dot com, m A T 0:56:46.520 --> 0:56:50.960 T O S O W s ki dot com. You 0:56:51.040 --> 0:56:53.439 can email me at easy at Better offline dot com 0:56:53.640 --> 0:56:55.920 or visit better Offline dot com to find more podcast 0:56:56.000 --> 0:56:59.320 links and of course, my newsletter. I also really recommend 0:56:59.320 --> 0:57:01.279 you go to chat Where's your Head dot at to 0:57:01.360 --> 0:57:04.160 visit the discord, and go to our slash Better Offline 0:57:04.200 --> 0:57:07.400 to check out I'll Reddit. Thank you so much for listening. 0:57:08.280 --> 0:57:10.920 Better Offline is a production of cool Zone Media. For 0:57:11.080 --> 0:57:14.680 more from cool Zone Media, visit our website Coolzonemedia dot com, 0:57:15.200 --> 0:57:18.040 or check us out on the iHeartRadio app, Apple Podcasts, 0:57:18.160 --> 0:57:19.600 or wherever you get your podcasts