WEBVTT - Why Corporate America Still Runs on Ancient Software That Breaks 0:00:10.119 --> 0:00:14.360 Hello, and welcome to another episode of the Odd Locks Podcast. 0:00:14.440 --> 0:00:18.360 I'm Joe Wisenthal and I'm Tracy Hallo. Tracy, I forgot 0:00:18.360 --> 0:00:20.800 to ask you, and it's kind of embarrassing like this 0:00:20.920 --> 0:00:24.400 late in January, but how is your how is your 0:00:24.400 --> 0:00:25.880 New Year's? Like? How is your holidays? Do you have 0:00:25.920 --> 0:00:29.320 a good Christmas? And stuff? Thanks? Thanks, Joe. I had 0:00:29.360 --> 0:00:32.559 excellent Christmas. I stayed at home for a week with 0:00:32.600 --> 0:00:35.960 my husband and my dog and we did hardly anything 0:00:36.040 --> 0:00:40.000 and it was absolutely glorious. How about you? It was 0:00:40.040 --> 0:00:41.920 all right. So the thing was I was down in 0:00:42.000 --> 0:00:45.080 Texas visiting a family, which is but there was that 0:00:45.280 --> 0:00:50.040 huge cold blast and it's worse in It's worth when 0:00:50.159 --> 0:00:52.239 when the weather gets really cold, it's worse in a 0:00:52.280 --> 0:00:55.200 place like Texas because like none of the buildings are 0:00:55.240 --> 0:00:58.120 insulated particularly well, none of them have that heat. So 0:00:58.160 --> 0:01:00.640 it's like, when it's really cold, is that like better 0:01:00.720 --> 0:01:03.279 to be in a cold place where people are used 0:01:03.320 --> 0:01:05.880 to it. So it's a little uncomfortable. But the good 0:01:05.880 --> 0:01:09.000 news is somehow we managed to travel back and forth 0:01:09.240 --> 0:01:14.200 without getting having any major airline disruptions. Right, So this 0:01:14.280 --> 0:01:17.360 is the key thing that happened right before Christmas, which 0:01:17.400 --> 0:01:21.040 is we had that very big winter storm, the Arctic 0:01:21.160 --> 0:01:25.839 bomb blast, and it disrupted a ton of flights first off, 0:01:25.840 --> 0:01:28.640 because of the weather, But then what happened is you 0:01:28.680 --> 0:01:31.639 had this sort of cascade effect because the weather event 0:01:31.720 --> 0:01:35.520 was so large, a number of airlines, but one airline 0:01:35.640 --> 0:01:39.520 in particular experienced a lot of problems with its software 0:01:39.520 --> 0:01:42.479 and Southwest had to cancel I think in the end 0:01:42.520 --> 0:01:46.400 it was something like sixteen thousand flights. You had millions 0:01:46.400 --> 0:01:49.680 of passengers affected, and you had disruptions that you know, 0:01:49.760 --> 0:01:52.800 a weather related disruption that lasted one or two days 0:01:53.360 --> 0:01:57.440 ended up lasting I think more than a week because 0:01:57.680 --> 0:02:01.720 of the impact of the computer litches. I guess, yeah, yeah, 0:02:01.800 --> 0:02:04.760 that's right. Like you know, airline traveled like it always 0:02:04.840 --> 0:02:07.640 sort of cascades and ripples out right, because it canceled 0:02:07.680 --> 0:02:09.840 flight is going to affect other flights and so forth. 0:02:10.160 --> 0:02:13.840 But it seemed like Southwest experienced something unique, which is 0:02:13.880 --> 0:02:17.400 that there's turned into this major software problem. And it's 0:02:17.440 --> 0:02:19.800 sort of like a reminder that like, okay, when we 0:02:19.880 --> 0:02:22.040 use the Internet, when we use like sort of like 0:02:22.160 --> 0:02:25.280 modern consumer software, it's all very zippy and quick, and 0:02:25.360 --> 0:02:28.560 it has nice interfaces and then when you use like 0:02:28.600 --> 0:02:32.600 sort of like back end corporate software, particularly at large 0:02:32.840 --> 0:02:39.200 legacy institutions, it's just like it's nothing like the consumer Internet. 0:02:39.240 --> 0:02:43.000 It's like clunky. It's like we all know what it's like, right. 0:02:43.080 --> 0:02:44.919 So this is something that came up quite a lot. 0:02:44.960 --> 0:02:48.200 I used to cover the big banks, and one of 0:02:48.240 --> 0:02:51.840 the crazy things that I learned relatively early on while 0:02:51.840 --> 0:02:54.120 I was doing this was just how much of their 0:02:54.160 --> 0:02:59.080 I T system was still these old, creaky, big iron mainframes, 0:02:59.400 --> 0:03:01.800 some of them still running on Cobalt, which is the 0:03:01.840 --> 0:03:03.959 programming language that I think it dates back to the 0:03:04.040 --> 0:03:08.000 nineteen fifties or nineteen sixties. And I remember, you know, 0:03:08.120 --> 0:03:10.200 you hear this, You hear like, oh, I can't believe 0:03:10.200 --> 0:03:13.799 that these big banks, our entire financial system in some respects, 0:03:13.840 --> 0:03:17.080 is still running on these legacy computer systems. But on 0:03:17.120 --> 0:03:19.440 the other hand, if you look at what a big 0:03:19.480 --> 0:03:25.160 bank is, it's structurally a series of mergers and acquisitions. 0:03:25.200 --> 0:03:30.000 There used to be the no, no, that's there used 0:03:30.040 --> 0:03:33.160 to be this great flow chart that showed the formation 0:03:33.360 --> 0:03:35.480 of like a JP Morgan or a Bank of America, 0:03:35.600 --> 0:03:37.400 and you can just see it's like a series of 0:03:37.480 --> 0:03:40.320 roll ups of smaller banks, and you think every time 0:03:40.360 --> 0:03:43.720 they acquire a new bank, they have to integrate another 0:03:43.840 --> 0:03:46.640 system into their own system, and in the end you 0:03:46.720 --> 0:03:50.320 kind of end up with this just incredibly like complex 0:03:50.320 --> 0:03:53.720 and kind of patchy i T structure that in some 0:03:53.800 --> 0:03:57.960 ways very much resembles the amalgamation of all these smaller 0:03:57.960 --> 0:04:00.560 banks into a larger bank. There's too the right, like 0:04:00.640 --> 0:04:03.280 you think of like banks and when they merge, it 0:04:03.400 --> 0:04:05.760 just like okay, like you know, the capitals all merged 0:04:05.800 --> 0:04:09.080 together and the assets, etcetera. But like, right, like none 0:04:09.120 --> 0:04:11.000 of them are going to have like i T systems 0:04:11.000 --> 0:04:14.000 that work perfectly together, so they're like glued together with 0:04:14.000 --> 0:04:17.360 like duct tape and just over time. I think technical 0:04:17.440 --> 0:04:19.960 debt is a term that software engineers, and so you 0:04:19.960 --> 0:04:23.000 can to relate all this technical debt. And I'm surely 0:04:23.120 --> 0:04:26.640 nobody likes it inside the bank at any level. But 0:04:26.680 --> 0:04:29.760 I've always been curious, like, what are the economics such 0:04:29.880 --> 0:04:37.000 that it is just so impossible for these legacy institutions, banking, airlines, 0:04:37.080 --> 0:04:42.120 absolute public sector to like get with the times, you know, absolutely, 0:04:42.240 --> 0:04:44.160 and when it comes to Southwest, So first of all, 0:04:44.240 --> 0:04:47.320 let me declare a small self interest in Southwest which 0:04:47.400 --> 0:04:50.080 is my My dad was a pilot for Southwest for 0:04:50.120 --> 0:04:52.160 a long time and I remember when the outages were 0:04:52.200 --> 0:04:54.680 happening this Christmas. I sent him a text going like, well, 0:04:54.680 --> 0:04:56.880 what do you think about this? And he just said, well, 0:04:56.880 --> 0:04:58.760 all the crew are out of position and they need 0:04:58.800 --> 0:05:02.359 to get them back. So very straightforward ex pilot answer 0:05:02.480 --> 0:05:06.040 about what was going on. Not that helpful for this podcast. 0:05:06.320 --> 0:05:09.599 But I do think in the case of Southwest, this 0:05:09.640 --> 0:05:12.320 has been a long running issue and you have had, 0:05:12.600 --> 0:05:14.960 you know, people talking about it at various times, the 0:05:15.040 --> 0:05:18.680 need to upgrade the infrastructure, of the technological infrastructure, and 0:05:18.760 --> 0:05:21.200 yet it hasn't happened. And so the question is why. 0:05:21.520 --> 0:05:24.240 The question is whether or not is it cheaper just 0:05:24.320 --> 0:05:27.280 to keep running these old mainframes and assume that you 0:05:27.320 --> 0:05:30.840 are going to have these outages during major events, cheaper 0:05:30.920 --> 0:05:34.320 than it would cost to actually upgrade it. No. Absolutely, Well, 0:05:34.640 --> 0:05:38.800 let's let's talk to somebody who knows about software and 0:05:38.880 --> 0:05:41.280 knows about economics and can walk us through and help 0:05:41.360 --> 0:05:45.600 us understand the problem. We're gonna be speaking with Patrick mackenzie. 0:05:46.040 --> 0:05:49.960 He's an expert on software and infrastructure. He's a writer 0:05:50.080 --> 0:05:52.640 of the Bits about Money newsletter and knows a lot 0:05:52.680 --> 0:05:56.839 about finance, and he just left a stripe after six years. 0:05:56.880 --> 0:06:00.200 He's still an adviser there. Been reading his stuff for 0:06:00.240 --> 0:06:02.039 a long time. One of these people I was sort 0:06:02.040 --> 0:06:04.839 of trust on almost any topic. Patrick, Thank you so 0:06:04.920 --> 0:06:07.560 much for coming on odd lots. Thanks very much for 0:06:07.600 --> 0:06:10.320 having me. What is the deal? Let's just start like 0:06:10.520 --> 0:06:13.680 straight up, like why do you like, I mean Tracy 0:06:13.800 --> 0:06:17.360 mentioned like all these institutions, they just like they swish 0:06:17.400 --> 0:06:20.080 it all together. They sort of like tie it together 0:06:20.200 --> 0:06:23.360 with duct tape and so forth. You get these unwieldy things. 0:06:24.080 --> 0:06:26.159 Just like give us the high level view of like 0:06:26.600 --> 0:06:29.880 why is it so difficult from at an abstract level 0:06:30.240 --> 0:06:35.560 to modernize legacy software? So I'll start with a disclaimer, 0:06:35.560 --> 0:06:38.880 which is sort of mandatory and engineering culture, we have 0:06:39.000 --> 0:06:40.880 this thing that we've come up with over the last 0:06:41.000 --> 0:06:44.799 course of two decades, or so called blameless postmortems, where 0:06:45.320 --> 0:06:48.440 when there is a failure within a company where you know, 0:06:48.640 --> 0:06:50.840 planes cannot be up in the air for a week 0:06:50.880 --> 0:06:53.120 at a time. Rather than like trying to point the 0:06:53.160 --> 0:06:55.359 finger at someone and say it was your decision or 0:06:55.400 --> 0:06:58.680 your inaction that caused this event, we as engineers want 0:06:58.800 --> 0:07:01.200 to like look at the rejective reality of the system, 0:07:01.279 --> 0:07:03.719 figure out what went wrong for the benefit of both 0:07:03.720 --> 0:07:05.880 that organization and for the larger community. And so this 0:07:05.920 --> 0:07:08.400 isn't too grind their nose in it, but just you know, 0:07:08.720 --> 0:07:13.040 as an engineering matter, what probably happened in an ideal world, 0:07:13.320 --> 0:07:15.840 And if this had happened at a Google, for example, 0:07:15.920 --> 0:07:18.880 the engineering teams would push, because of the culture of 0:07:18.880 --> 0:07:23.040 these things, to do a very public postmortem of what 0:07:23.200 --> 0:07:26.320 the decisions were, what the background is, etcetera, etcetera. In 0:07:26.360 --> 0:07:29.360 more traditional industries, I don't think that culture is fully 0:07:29.680 --> 0:07:32.800 baked yet as it were, although they're very well might 0:07:32.840 --> 0:07:35.920 be a postmortem by uh, you know, the FA and 0:07:35.960 --> 0:07:39.520 federal regulators, because you know, liveness constraint is a real 0:07:39.560 --> 0:07:43.920 thing for extremely important economic systems like airlines. Anyhowp what 0:07:44.240 --> 0:07:48.240 probably happened. It wouldn't surprise anyone in the bowels of 0:07:48.320 --> 0:07:51.640 an airline that if you put off maintenance on airplanes 0:07:51.680 --> 0:07:54.080 for decades at a time, that eventually bad things would 0:07:54.120 --> 0:07:58.400 happen and no one would countenance that. However, software systems 0:07:58.400 --> 0:08:01.800 are quite similar. Where aren't a build once and then 0:08:01.920 --> 0:08:04.200 run for the rest of eternity sort of things? There 0:08:04.240 --> 0:08:06.960 were some decisions made early in their lifetimes which share 0:08:07.040 --> 0:08:09.320 no longer accurate for the world we live in. They 0:08:09.360 --> 0:08:12.600 do suffer from something engineers you from misply called bit 0:08:12.720 --> 0:08:16.680 rot where software which worked back in the past will 0:08:16.680 --> 0:08:19.640 tend to like succumbed entropy over time and not work 0:08:19.680 --> 0:08:22.760 exactly perfectly for all time afterwards, and so you need 0:08:22.800 --> 0:08:27.320 to be doing a ongoing program of maintenance for your software, 0:08:27.400 --> 0:08:30.840 just like you would for your airplanes. That bluntly was 0:08:30.960 --> 0:08:34.920 not done, and it seems to be credibly reported that's 0:08:35.320 --> 0:08:38.320 the sort of cultural factors at Southwest that caused that 0:08:38.360 --> 0:08:40.959 to not be done. Might have been caused by a 0:08:41.120 --> 0:08:43.760 sort of like overly accounting the slash penny pinching focused 0:08:43.760 --> 0:08:46.400 management culture, which thought, well, it costs money and the 0:08:46.440 --> 0:08:49.320 short term to do to do maintenance, we can like 0:08:49.480 --> 0:08:51.560 cram down or engineering costs by doing less of this 0:08:51.600 --> 0:08:54.640 and relying more on external vendors, etcetera, etcetera. And then 0:08:54.840 --> 0:08:59.000 when stuff HiT's the oscillating plate, the right people have 0:08:59.080 --> 0:09:01.040 not done the years of work that are required to 0:09:01.880 --> 0:09:04.560 get to a posture where you can like quickly recover 0:09:04.640 --> 0:09:07.680 from failures, and so they were left in a part 0:09:07.760 --> 0:09:11.000 where we use euphemistically in the in street called heroics 0:09:11.440 --> 0:09:15.720 was required from folks in operations trying to you know, 0:09:15.800 --> 0:09:18.839 contact thousands and tens of thousands of employees by fille 0:09:18.880 --> 0:09:22.200 and pass around their information and probably spreadsheets to figure 0:09:22.200 --> 0:09:24.720 out where the crew is actually where to be able 0:09:24.720 --> 0:09:27.679 to take the boxes that are required due to regulation 0:09:27.760 --> 0:09:30.280 to allow people to get back and up in the air. 0:09:31.120 --> 0:09:33.720 So that's probably the high level, like root cause of 0:09:33.720 --> 0:09:37.080 what happened. But like a reason to conduct these postmortems 0:09:37.200 --> 0:09:40.720 is it's there for one, like one single decision made 0:09:40.720 --> 0:09:44.280 by one person. It's the result of cultural factors, business 0:09:44.320 --> 0:09:47.440 decisions made over a course of probably decades in this place, 0:09:47.760 --> 0:09:49.880 and we want to like keys out the various nuances 0:09:49.880 --> 0:09:53.520 there and like make both Southwest and other organizations where 0:09:53.520 --> 0:09:57.040 of that so that they don't suffer critical systemic failures 0:09:57.040 --> 0:10:00.960 in their own places. So, first of all, bit brought 0:10:01.240 --> 0:10:03.080 is a fantastic term that I am going to have 0:10:03.160 --> 0:10:06.120 to try to work into all my conversations going forward. 0:10:06.520 --> 0:10:09.480 But secondly, just to step back a bit, can you 0:10:09.520 --> 0:10:14.199 maybe explain, you know, when we talk about mainframe computer systems, 0:10:14.640 --> 0:10:17.480 what exactly are we talking about? And like what is 0:10:17.520 --> 0:10:21.280 the counterpoint to mainframe. I'm assuming it's more like cloud 0:10:21.280 --> 0:10:23.800 based applications and things like that, but could you maybe 0:10:24.120 --> 0:10:28.000 define those that basic term very quickly. And then secondly, 0:10:28.640 --> 0:10:33.320 my understanding of the Southwest debacle was that they had 0:10:33.480 --> 0:10:37.600 this in house software. I think it was called sky Solver, 0:10:38.320 --> 0:10:41.679 but it was based on an application that g E 0:10:42.040 --> 0:10:45.480 had been selling, and then Southwest kind of customized it. 0:10:45.880 --> 0:10:48.839 And I guess my question is how endemic is that 0:10:48.880 --> 0:10:51.719 type of software where you get something off the shelf, 0:10:51.760 --> 0:10:53.760 but then you customize it in such a way that 0:10:53.800 --> 0:10:57.680 it becomes um, I guess special to you and therefore 0:10:57.920 --> 0:11:01.680 your problem when it goes awry. So let's talk about 0:11:01.720 --> 0:11:04.080 the the main frame first, and then we'll talk about 0:11:04.080 --> 0:11:07.120 the problem of like who owns the problem, whether it's 0:11:07.120 --> 0:11:09.280 your problem or some of the vendor's problem, and the 0:11:09.840 --> 0:11:12.439 way that balls tend to end uh in the middle 0:11:12.480 --> 0:11:16.320 of people and things get dropped. So mainframes back in 0:11:16.360 --> 0:11:19.880 the day, many many decades ago, computers were approximately the 0:11:19.920 --> 0:11:22.160 size of a room. This is the before the personal 0:11:22.160 --> 0:11:25.360 computer revolution and banks, which were some of the earliest 0:11:25.360 --> 0:11:28.840 adopters of computers for sort of scaled usage in industry. 0:11:29.040 --> 0:11:32.360 And it's funny like the earliest users and computers tended 0:11:32.400 --> 0:11:35.400 to be either the financial industry or the military, either 0:11:35.520 --> 0:11:38.880 attempting to like move numbers which represented money, or numbers 0:11:38.880 --> 0:11:42.120 which represented like literally artillery shells flying through the air, 0:11:42.440 --> 0:11:45.760 and both were a very important society for better or worse, 0:11:45.920 --> 0:11:49.520 Like banks because they standardized on what was the best 0:11:49.520 --> 0:11:51.600 available technology at the time, ended up with a lot 0:11:51.640 --> 0:11:53.880 of main frames, and they have kept those main frames 0:11:53.960 --> 0:11:56.120 running for a good portion of seventy years now. In 0:11:56.160 --> 0:11:59.720 some cases, as you mentioned earlier, the alternative to main frames. 0:12:00.040 --> 0:12:02.480 Cloud is a bit of a buzzword. So there's the 0:12:02.480 --> 0:12:05.480 personal computer form factor that you're familiar with, where something 0:12:05.480 --> 0:12:08.600 sits on your desk. There are servers which would typically 0:12:08.640 --> 0:12:11.200 sit in a server X somewhere, uh. And the difference 0:12:11.240 --> 0:12:14.120 between like servers and quote unquote the cloud is like 0:12:14.440 --> 0:12:16.800 the traditional way to manage servers would be you would 0:12:16.840 --> 0:12:19.320 have a data center that would be owned or at 0:12:19.360 --> 0:12:21.440 least by yourself, and you would put hardware that you 0:12:21.559 --> 0:12:24.360 owned in that data center and in the cloud. The 0:12:24.400 --> 0:12:27.280 cloud case, the data center is owned by Amazon or 0:12:27.320 --> 0:12:30.760 Google or Microsoft. You have rented access to a machine 0:12:30.800 --> 0:12:34.280 there which sits on their balance sheet and it's probably 0:12:34.280 --> 0:12:37.400 not one machine, and it's probably like an awful lot 0:12:37.440 --> 0:12:41.480 of machines potentially with some virtualization layer, and your engineers 0:12:41.520 --> 0:12:44.920 can cause systems to scale up or scale down based 0:12:44.960 --> 0:12:47.280 on how many machines you need on a you know, 0:12:47.360 --> 0:12:49.880 minute by minute or second by second basis. Is sort 0:12:49.880 --> 0:12:51.880 of like the high level version of the sales pitch 0:12:51.920 --> 0:12:55.199 that cloud vendors will give you. So that is a 0:12:55.280 --> 0:12:58.240 hey quick run through between like four generations of the 0:12:58.240 --> 0:13:01.160 main stay of how technology gets done at scale the 0:13:01.240 --> 0:13:04.839 question of where software gets written. So it is quite 0:13:04.880 --> 0:13:07.800 common for businesses in the traditional economy to not have 0:13:08.000 --> 0:13:12.360 an internal software engineering competence, and as a result, they'll 0:13:12.400 --> 0:13:15.560 go to vendors. Like in this case, you probably know 0:13:15.600 --> 0:13:18.599 this better than me. But for example g and the 0:13:19.200 --> 0:13:22.240 vendor will sell quote unquote package software which might not 0:13:22.280 --> 0:13:24.320 be fully responsive to the needs of the business, and 0:13:24.360 --> 0:13:27.800 then some customization happens. Um what could happen is the 0:13:27.800 --> 0:13:30.680 customization might happen within the business if they have some 0:13:30.880 --> 0:13:34.160 level of software engineers internally. What happens more frequently in 0:13:34.200 --> 0:13:36.400 a lot of places, particular in depend where I live, 0:13:36.800 --> 0:13:39.520 is that the business will contract with we call them 0:13:39.520 --> 0:13:42.280 system integrators here that may be called consultancies in America, 0:13:42.720 --> 0:13:47.880 you know a Deloitte or a another large consulting firm 0:13:47.960 --> 0:13:52.280 like that Accenture and say, okay, we have this package system, 0:13:52.360 --> 0:13:55.760 we have these business requirements. Clearly some software engineers have 0:13:55.800 --> 0:13:57.800 to be involved. We don't have enough on our staff. 0:13:57.800 --> 0:14:00.000 Can you like figure out the missing link for us? 0:14:00.400 --> 0:14:03.600 And so there is an ex sense of contract negotiation. 0:14:04.000 --> 0:14:07.480 Consultancy goes off and does the customizations, and they delivered 0:14:07.480 --> 0:14:10.720 the work to the organization. And then there are various 0:14:10.720 --> 0:14:13.120 different ways to do maintenance, but often that will involve 0:14:13.160 --> 0:14:16.720 the original team standing down for a while, which that 0:14:16.760 --> 0:14:19.160 always sounds like a great idea when it's pitched, because 0:14:19.200 --> 0:14:21.320 it's like, oh great, I don't have to pay expensive 0:14:21.320 --> 0:14:24.080 engineers every week to just sit around waiting for something 0:14:24.120 --> 0:14:27.840 to happen. And then when something actually happens, you sort 0:14:27.880 --> 0:14:31.080 of like realize that the other end of the option 0:14:31.160 --> 0:14:34.600 value there where oh not so great. I don't have 0:14:34.680 --> 0:14:37.400 a team of like experts who understand the system, ready 0:14:37.400 --> 0:14:39.440 to call up at a moment's notice and bring in 0:14:39.480 --> 0:14:43.040 to debug the problems that we're seeing right now. And 0:14:43.080 --> 0:14:45.360 so I have no specific knowledge about went in this 0:14:45.600 --> 0:14:48.600 individual instance. But a thing that happens during a lot 0:14:48.640 --> 0:14:52.240 of outages is uh a, you know, quick look into 0:14:52.280 --> 0:14:55.480 the history of the system to say, hey, wait, who 0:14:55.600 --> 0:14:57.920 actually built this for us? Are they still in business? 0:14:58.120 --> 0:15:00.720 Can we get on their calendar immediately? Is there an 0:15:00.720 --> 0:15:03.200 engineering team there that is ready to hop in and 0:15:03.200 --> 0:15:06.480 work at this? Like goodness, they're going to charge us 0:15:06.600 --> 0:15:09.320 I bleeding rates to to do it, but we're going 0:15:09.360 --> 0:15:11.320 to have to pay that. That's a you know, tertiary 0:15:11.320 --> 0:15:14.280 consideration at this point. The bigger consideration is like, how 0:15:14.360 --> 0:15:17.720 quickly can we get our you know, organization to organization 0:15:17.960 --> 0:15:21.280 like legal paperwork, etcetera, spun up? And then how quickly 0:15:21.320 --> 0:15:23.320 can you get they get the engineering team spun up 0:15:23.320 --> 0:15:26.720 to address the system in real time? And this sort 0:15:26.760 --> 0:15:30.120 of thing is why the sort of like major scaled 0:15:30.560 --> 0:15:34.440 software companies and the economy that Google, Microsoft, etcetera, etcetera, 0:15:34.480 --> 0:15:37.400 you all know the names, largely treat engineering as an 0:15:37.400 --> 0:15:40.520 internal competence rather than you know, the software for the 0:15:40.560 --> 0:15:43.240 iPhone isn't built by Debloid, It's built by Apple, and 0:15:43.520 --> 0:15:46.000 almost everything in the critical path will either be open 0:15:46.000 --> 0:15:48.480 source or something that there is a team on Apple 0:15:48.560 --> 0:15:50.720 that owns the sort of totality of the experience for 0:15:50.800 --> 0:16:10.720 a goal. So something that I sort of hinted at 0:16:10.800 --> 0:16:13.320 in the beginning, and we talked about this a little bit. 0:16:13.400 --> 0:16:16.320 We did an episode on petroleum engineering. But I'm really 0:16:16.360 --> 0:16:20.600 curious about like the distribution of engineering talent. Like in 0:16:20.800 --> 0:16:23.160 my mind, I would have to imagine that a talented 0:16:23.200 --> 0:16:27.120 engineer would be more excited to work for Stripe than Google, 0:16:27.560 --> 0:16:30.840 and we're excited to work for Google than Deloitte, and 0:16:30.840 --> 0:16:35.160 we're excited to work for Deloitte than Southwest. And I'm 0:16:35.240 --> 0:16:37.640 just sort of and I'm curious, like a if that's 0:16:38.040 --> 0:16:41.920 the case, and whether like this is a problem, and 0:16:42.000 --> 0:16:44.160 whether like it would be better, whether it be better 0:16:44.320 --> 0:16:49.360 if like there were just more high tech experts, software 0:16:49.400 --> 0:16:52.680 experts who would want to work for a Southwest directly 0:16:53.120 --> 0:16:56.200 or work for a bank directly, whether this sort of 0:16:56.240 --> 0:17:00.400 like distribution of engineering talent contributes to software body all act. 0:17:00.920 --> 0:17:03.960 So I have nuanced thoughts that engineering talent, on one hand, 0:17:04.080 --> 0:17:06.520 like that is certainly thing that exists in the world, 0:17:06.600 --> 0:17:12.040 and like talent is not necessarily distributed equally among different industries, individuals, etcetera. 0:17:12.440 --> 0:17:14.439 On the other hand, I think that Silicon Valley and 0:17:14.520 --> 0:17:18.440 ecosystem to which I owe a lot occasionally has an 0:17:18.480 --> 0:17:21.119 overly enamored view of itself, thinking that, oh, yes, we 0:17:21.240 --> 0:17:23.639 have the best engineers in the world, and all the 0:17:23.680 --> 0:17:26.040 engineers who made every other system that we rely on 0:17:26.119 --> 0:17:28.760 in our lives are sort of second rate, And clearly 0:17:28.800 --> 0:17:32.960 that's not true. Like the phone system works, airplanes like 0:17:33.080 --> 0:17:35.240 what's the old Lui kiss came on a love like 0:17:35.480 --> 0:17:38.280 it flies through the sky and then teleports beams up 0:17:38.280 --> 0:17:41.320 to space like none of that happened by accident, And 0:17:41.400 --> 0:17:47.399 so it's important to not focus on like the You know, 0:17:47.520 --> 0:17:50.840 engineers working traditional industry are worse engineers that engineers their 0:17:50.880 --> 0:17:54.679 work in software companies. The larger problement they have is 0:17:54.880 --> 0:17:57.879 one they don't drive the bus. They have less ability 0:17:58.000 --> 0:18:01.840 to control the situations within their organizations and control like 0:18:02.000 --> 0:18:06.000 larger decisions made such that they have influence over decisions 0:18:06.040 --> 0:18:08.320 on like what the main and schedule would look like, 0:18:08.520 --> 0:18:11.240 or who gets to make decisions with respect to whether 0:18:11.280 --> 0:18:13.960 something ships or not. And there is a little bit 0:18:14.000 --> 0:18:16.560 of a sort of like the life cycle of engineers 0:18:16.960 --> 0:18:19.959 thing that happens where the original architects of the system 0:18:20.080 --> 0:18:22.760 did it forty years ago. A careers are about as 0:18:22.800 --> 0:18:24.800 long as they are. Many of the original architects of 0:18:24.840 --> 0:18:27.360 the system will sort of be aging into the retirement 0:18:27.440 --> 0:18:29.480 years at this point, and so there is a question 0:18:29.520 --> 0:18:32.040 of did the organization put in the work over the 0:18:32.119 --> 0:18:35.400 years to recruit newer engineers to inculcate them into how 0:18:35.400 --> 0:18:38.600 the system is made, etcetera, etcetera, Or do they just 0:18:38.720 --> 0:18:40.720 allow all of the knowledge to walk out the door 0:18:41.119 --> 0:18:43.480 and did they Like I think that comes up a 0:18:43.480 --> 0:18:46.480 lot is do they put enough work on a day 0:18:46.480 --> 0:18:49.080 to day basis to maintain an engineering brand so that 0:18:49.119 --> 0:18:51.600 they can get new talent engineers to join them in 0:18:52.400 --> 0:18:55.840 three such that those engineers are the word used as 0:18:55.840 --> 0:18:59.040 often gray beard, but you know that the wise and 0:18:59.160 --> 0:19:01.280 veteran thirty year is from now, so that when something 0:19:01.280 --> 0:19:04.280 happens in there are people who have been around the 0:19:04.280 --> 0:19:08.000 black that know where the skeletons are buried in the system. Interestingly, 0:19:08.119 --> 0:19:11.320 traditional industry is getting better over the years at having 0:19:11.320 --> 0:19:14.919 an engineering brand, sort of moving away from this this 0:19:15.000 --> 0:19:17.639 world where engineering was largely seen as a cost center, 0:19:18.040 --> 0:19:20.200 where like the goal is just too cram down the 0:19:20.200 --> 0:19:22.680 amount of money you spend on it and improve your margins. 0:19:23.080 --> 0:19:25.600 I think there's a few things that played into that. 0:19:25.680 --> 0:19:29.040 One of them was, particularly in the Internet age, it 0:19:29.280 --> 0:19:32.000 became obvious that you know, in finance, you talk about 0:19:32.080 --> 0:19:34.680 like the front office and the back office. Engineering used 0:19:34.680 --> 0:19:36.160 to be in the back office, and the front office, 0:19:36.160 --> 0:19:38.959 where the sales people live, is the one that generates 0:19:39.000 --> 0:19:42.240 all the money for the bank. And increasingly, because like 0:19:42.800 --> 0:19:45.199 experiences that were in the palm of the users hands 0:19:45.240 --> 0:19:47.520 were the thing that we're actually generating in the sales, 0:19:48.040 --> 0:19:53.320 those experiences became sort of institutionally important within banks and 0:19:53.359 --> 0:19:57.160 airlines and other firms. And then when those experiences became 0:19:57.200 --> 0:20:00.800 important took a while, but gradually the people and teams 0:20:00.840 --> 0:20:03.840 that build those experiences became more institutionally important than they 0:20:03.840 --> 0:20:06.479 had previously been. And so like, if you look at 0:20:06.480 --> 0:20:08.840 the large money center banks in the US, they certainly 0:20:08.880 --> 0:20:12.440 have no small number of technical challenges. But the thing 0:20:12.480 --> 0:20:16.560 that is like largely true which was not true, is 0:20:16.600 --> 0:20:19.200 that their mobile apps are actually kind of good these days. 0:20:19.359 --> 0:20:22.399 Like if you download, you know, not to endorse anybody 0:20:22.400 --> 0:20:25.119 in particular, but like Chase or Capital One, when you 0:20:25.119 --> 0:20:26.919 play with their mobile apps, like, oh, this kind of 0:20:26.920 --> 0:20:29.879 feels like a mobile app made in Silicon Valley. And 0:20:29.920 --> 0:20:31.840 the reason is, well, yeah, they hired a lot of 0:20:31.840 --> 0:20:34.720 people who made the apps from Silicon Valley, and those 0:20:34.760 --> 0:20:37.720 folks brought their skills and sort of like level of 0:20:37.720 --> 0:20:40.720 competence with this and their taste and now exercise and 0:20:40.760 --> 0:20:43.760 on behalf of old world companies. One hopes, knock on wood, 0:20:44.000 --> 0:20:46.720 that that seeps into the back end of these systems 0:20:46.760 --> 0:20:50.119 to where Apple, Google, Microsoft, they have very talented teams 0:20:50.119 --> 0:20:51.680 on the front end of their systems, but they also 0:20:51.720 --> 0:20:54.240 have very talented teams on the back end of the systems. 0:20:54.240 --> 0:20:57.280 And that's bluntly why you don't see like core systems 0:20:57.280 --> 0:20:58.920 that Google going down for a week at a time 0:20:59.760 --> 0:21:02.720 that is almost unimaginable. You know how much of capitalism 0:21:02.800 --> 0:21:05.280 would break if Google Docs was just down for a week. 0:21:05.800 --> 0:21:07.960 Given that there are many parts of the world where 0:21:09.320 --> 0:21:12.600 there is a liveness constraint, we would like all else 0:21:12.680 --> 0:21:15.159 being equal. If it is safe to fly airplanes, we 0:21:15.160 --> 0:21:17.600 would prefer that there be airplanes flying versus all the 0:21:17.640 --> 0:21:20.879 airplanes being on the ground, because airplanes generated valley for 0:21:20.960 --> 0:21:23.840 human society. If that is true, then it must be 0:21:23.880 --> 0:21:27.399 the case that the back end systems of airlines that 0:21:27.720 --> 0:21:30.680 control whether airplanes are allowed to fly at a given time. 0:21:31.160 --> 0:21:33.440 That has to be at least as important as you know, 0:21:33.640 --> 0:21:36.280 Google docs is, which implies that the airlines need to 0:21:36.320 --> 0:21:39.040 put at least as much work as Google does, roughly 0:21:39.480 --> 0:21:42.239