WEBVTT - CrowdStruck 0:00:01.720 --> 0:00:05.760 A media Hello one, Welcome to a very special emergency 0:00:05.800 --> 0:00:08.600 episode of Better Offline. I'm at Zeitron, I'm your host, 0:00:08.680 --> 0:00:10.600 and I'm recording this from inside a closet in a 0:00:10.640 --> 0:00:25.639 hotel in San Francisco. You're very important to me. On 0:00:25.720 --> 0:00:28.360 Friday afternoon, I sat at my desk and just started 0:00:28.360 --> 0:00:31.120 writing about any clear aim or objective other than a 0:00:31.200 --> 0:00:34.400 desire to wrap my head around probably the most cataclystic 0:00:34.479 --> 0:00:38.000 technological meltdown that I've seen in my career, And of 0:00:38.080 --> 0:00:41.839 course I'm referring to the CrowdStrike situation. How was it 0:00:41.920 --> 0:00:45.000 the piece of software, one that few people understood, made 0:00:45.040 --> 0:00:47.800 by a company that people really didn't know was able 0:00:47.840 --> 0:00:51.400 to shut down our banking system, mayor travel, TV logistics chains, 0:00:51.479 --> 0:00:54.600 those weird screens that you see around and of course hospitals. 0:00:55.400 --> 0:00:58.120 And as I wrote this script, I found myself returning 0:00:58.200 --> 0:00:59.720 to some of the themes that I wrote about in 0:00:59.720 --> 0:01:02.360 The Rock Economy, in the Shareholder Supremacy, and many other 0:01:02.400 --> 0:01:05.520 pieces that speak to a larger problem in the tech industry, 0:01:06.120 --> 0:01:10.360 a complete misalignment in the incentives of most major tech companies, 0:01:10.680 --> 0:01:14.039 which has become less about building new technologies and maintaining 0:01:14.040 --> 0:01:16.120 them and then selling them to people who would then 0:01:16.240 --> 0:01:19.720 use them over time, and more about capturing monopolies and 0:01:19.760 --> 0:01:24.240 gearing organizations to extract value from the things around them. 0:01:24.760 --> 0:01:28.360 Every problem you see is a result of the tech industry, 0:01:28.760 --> 0:01:31.360 from the people funding the earliest startups to the trillion 0:01:31.400 --> 0:01:34.560 dollar juggernauts that dominate our lives, and the fact that 0:01:34.600 --> 0:01:37.360 it's no longer focused on the creation of technology with 0:01:37.400 --> 0:01:42.839 a purpose and organizations driven towards said purpose. Everything's about 0:01:42.880 --> 0:01:46.200 expressing growth and about showing how you will dominate an 0:01:46.240 --> 0:01:49.720 industry rather than serve it, and providing metrics that speak 0:01:49.760 --> 0:01:52.720 to the paradoxical notion that you'll grow forever without any 0:01:52.800 --> 0:01:57.920 consideration of how you'll actually live that long. Legacies are 0:01:57.960 --> 0:02:02.320 now subordinate to monopolies, current customers are subordinate to new customers, 0:02:02.320 --> 0:02:05.480 and products well, they're considered the means to introduce a 0:02:05.480 --> 0:02:07.960 customer to a form of parasite designed to punish the 0:02:08.080 --> 0:02:12.760 user for even thinking about moving to a competitor. The 0:02:12.840 --> 0:02:16.280 key difference between what happened on Friday with CrowdStrike and 0:02:16.320 --> 0:02:18.079 by the way, it's still being fixed, and as I'll 0:02:18.120 --> 0:02:21.160 explain later, will really take some time to be fully resolved, 0:02:21.400 --> 0:02:24.320 and my criticisms of other companies like Facebook and Google 0:02:24.919 --> 0:02:29.280 is the sheer violent nature of this failure, the decline 0:02:29.320 --> 0:02:31.799 of search and social tools we use in it is 0:02:31.880 --> 0:02:36.040 kind of a gradual, incremental kind of rot. CrowdStrike, meanwhile, 0:02:36.200 --> 0:02:38.639 was a demonstration of what happens when the rod fully 0:02:38.720 --> 0:02:44.000 consumes the timber holding up the building. What's happened with 0:02:44.160 --> 0:02:48.880 CrowdStrike is completely unprecedented. I'll get to why shortly, and 0:02:48.960 --> 0:02:51.040 on the scale of the much feared why to Q 0:02:51.280 --> 0:02:53.799 bug that threatened to ground the entirety of the world's 0:02:53.800 --> 0:02:58.280 computer based infrastructure once the year two thousand began. You'll 0:02:58.320 --> 0:03:01.360 note that I'm not saying that White was over hyapt 0:03:01.440 --> 0:03:04.880 or dismissing the scale, because ydo K was a huge 0:03:04.919 --> 0:03:08.520 society threatening calamity waiting to happen, and said calamity was 0:03:08.520 --> 0:03:12.079 averted not through any kind of magical thinking, but through 0:03:12.080 --> 0:03:15.320 a remarkable half trillion dollar industrial effort that took a 0:03:15.360 --> 0:03:19.040 decade to manifest. Because the seriousness of such a significant 0:03:19.040 --> 0:03:22.280 single point of failure would have likely crippled governments, banks, 0:03:22.320 --> 0:03:26.519 and airlines, people laughed when nothing happened on January first, 0:03:26.560 --> 0:03:29.160 two thousand, Assuming that all that money and time had 0:03:29.160 --> 0:03:31.880 been wasted. All of the media was just being hysterical 0:03:32.160 --> 0:03:35.640 rather than being grateful that an infrastructural weakness was identified 0:03:35.720 --> 0:03:38.800 taken seriously, and that a single point of failure was 0:03:38.840 --> 0:03:41.640 dealt with, and that the crisis was averted by investing 0:03:41.680 --> 0:03:47.200 in stopping bad staff happening before it does. Crazy goddamn idea. Huh. 0:03:47.240 --> 0:03:49.960 But as we speak, millions or even hundreds of millions 0:03:50.000 --> 0:03:52.640 of different Windows based computers are now stuck in a 0:03:52.680 --> 0:03:55.560 doom loop, repeatedly showing us as the fame blue screen 0:03:55.600 --> 0:03:57.720 of death, thanks to a single point of failure in 0:03:57.760 --> 0:04:01.160 a company called CrowdStrike, the developed of a globally adopted 0:04:01.200 --> 0:04:05.560 cybersecurity product designed ironically to prevent the kinds of disruption 0:04:05.680 --> 0:04:08.400 that we witnessed on Friday end. We're still witnessing today, 0:04:09.560 --> 0:04:12.040 and for reasons we'll get into shortly, this nightmare is 0:04:12.080 --> 0:04:14.320 going to drag on for several days, if not weeks 0:04:14.320 --> 0:04:19.080 to come. The product called CrowdStrike Falcon Sensor is an 0:04:19.200 --> 0:04:23.080 EDR system which stands for endpoint Detection and Response. If 0:04:23.080 --> 0:04:25.479 you aren't a security professional and your eyes are glazing over, 0:04:25.760 --> 0:04:28.839 I'll keep it brief an EDR system is designed to 0:04:28.880 --> 0:04:32.280 identify hacking attempts, to remediate them prevent them. They're big, 0:04:32.320 --> 0:04:34.719 sophisticated and complicated products, and they do a lot of 0:04:34.760 --> 0:04:37.480 things that's quite hard to build with the standard tools 0:04:37.520 --> 0:04:40.600 available to Windows developers. But as I'll get to later, 0:04:40.880 --> 0:04:45.240 not Microsoft, and so to make Falcon sensor work, CrowdStrike 0:04:45.279 --> 0:04:48.599 had to build its own internal kernel driver. Now, kernel 0:04:48.680 --> 0:04:51.440 drivers operate at the lowest level in the computer. They 0:04:51.440 --> 0:04:54.400 have the highest possible permissions, but they operate with the 0:04:54.400 --> 0:04:57.880 fewest amount of guardrails because massive control and they're very 0:04:57.920 --> 0:05:00.240 important to the system. Very technical people can to hear 0:05:00.240 --> 0:05:01.800 that and be like, that's not the right way to 0:05:01.800 --> 0:05:04.640 put it. Get out not your podcast. But if you've 0:05:04.640 --> 0:05:07.560 ever built your own computer, or you remember what computers 0:05:07.560 --> 0:05:09.839 were like in the dark days of Windows ninety eight, 0:05:10.279 --> 0:05:12.880 you know that a single faulty kernel driver can wreak 0:05:12.920 --> 0:05:16.560 havoc on the stability of your system. The problem here 0:05:16.640 --> 0:05:19.880 is that CrowdStrike pushed out an evidently broken kernel driver 0:05:19.960 --> 0:05:22.880 that locked whatever system that installed it in a permanent bootloop, 0:05:22.960 --> 0:05:26.120 meaning that you just started Blue Screen of Death, restarted 0:05:26.320 --> 0:05:29.400 kept doing him, the system would start loading Windows Encounter 0:05:29.440 --> 0:05:32.880 a fatal error and reboot, and then reboot, and then 0:05:32.960 --> 0:05:36.880 reboot again and again and again, in essence rendering the 0:05:36.880 --> 0:05:41.599 machine useless. It's convenient to blame CrowdStrike here, and perhaps 0:05:41.680 --> 0:05:44.040 that's fair, and I intend to do so several times. 0:05:44.960 --> 0:05:47.560 This should not have happened on a basic level. Whenever 0:05:47.560 --> 0:05:49.880 you write or update a kernel driver, you need to 0:05:49.920 --> 0:05:54.680 know it's actually robust and won't shit the bed immediately. Regrettably, 0:05:54.800 --> 0:05:59.040 CrowdStrike seemed to borrow Boeing's approach to quality control, except 0:05:59.080 --> 0:06:01.159 instead of building plane where the doors fly off and 0:06:01.200 --> 0:06:03.360 Boeing is the noise it makes when they fly off 0:06:03.560 --> 0:06:06.440 at the most inopportune times, it released a piece of 0:06:06.480 --> 0:06:09.080 software that blew up the transportation and banking sectors. The 0:06:09.200 --> 0:06:12.560 name just a few. It created a global IT outage 0:06:12.680 --> 0:06:16.200 that as grounded flights and broken banking services. It took 0:06:16.279 --> 0:06:20.159 down the BBC's flagship TV channel for kids, infuriating parents 0:06:20.160 --> 0:06:22.600 across the British isles, as well as Sky News, which, 0:06:23.000 --> 0:06:25.719 when it was able to resume life broadcasts, was forced 0:06:25.720 --> 0:06:29.159 to do so without graphics. In essence, it was forced 0:06:29.200 --> 0:06:32.040 back to the nineteen fifties, giving an esthetic that matches 0:06:32.040 --> 0:06:34.760 the politics of its founder and former owner, Rupert Murdoch. 0:06:35.200 --> 0:06:38.799 By no means is this exhaustive list of those affected. Either. 0:06:39.800 --> 0:06:42.400 The scale and disruption caused by this incident is unlike 0:06:42.440 --> 0:06:47.200 anything we've ever seen before. Previous instances like this, particularly 0:06:47.279 --> 0:06:51.960 rival ransomware outbreaks like Wanna Craze, simply can't compare, especially 0:06:51.960 --> 0:06:54.200 when we're looking at the disruption at the sheer scale 0:06:54.680 --> 0:06:57.880 of this problem. Still, if your day has been ruined 0:06:57.880 --> 0:06:59.920 by this outage, at least spare a thought for those 0:07:00.040 --> 0:07:03.279 will have to actually fix it, because those machines affected 0:07:03.320 --> 0:07:05.479 are now locked in this boot loop. It's not like 0:07:05.520 --> 0:07:08.240 CrowdStrike and just release a new software patch and call 0:07:08.279 --> 0:07:11.320 it a day on Doing this update requires some users 0:07:11.360 --> 0:07:14.480 to have to individually go to each computer, loading up 0:07:14.520 --> 0:07:17.000 safe mode or limited version of Windows with most non 0:07:17.080 --> 0:07:21.080 essential software and drivers disabled, and manually remove the faulty code. 0:07:21.440 --> 0:07:23.800 And if you have encrypted your computer, that process gets 0:07:23.800 --> 0:07:27.440 a lot harder. Servers running on cloud services like Amazon 0:07:27.480 --> 0:07:29.920 Web Services and Microsoft Azure, you know, the way that 0:07:30.000 --> 0:07:35.160 most of the Internet's infrastructure works, requires an entirely different 0:07:35.320 --> 0:07:40.360 and much more annoying, separate series of actions. If you're 0:07:40.400 --> 0:07:43.000 on a small item team, and you're supporting hundreds of 0:07:43.000 --> 0:07:47.040 workstations across several far flung locations, which really isn't unusual 0:07:47.080 --> 0:07:50.000 these days, especially in sectors like retail and social care. 0:07:50.720 --> 0:07:54.320 You're especially fucked. Say goodbye to your weekend, your evenings, 0:07:54.440 --> 0:07:56.800 Say goodbye to your spouse, your kids. You won't be 0:07:56.800 --> 0:07:59.000 seeing them for a while, and I'm really sorry. I'll 0:07:59.000 --> 0:08:01.280 buy you a drink some time. Your life will be 0:08:01.320 --> 0:08:04.080 driving from site to site, applying the figs and moving on. 0:08:04.360 --> 0:08:06.240 Forget about sleeping in your own bed or eating a 0:08:06.240 --> 0:08:10.120 meal that wasn't brought to you by door dash, Good luck, godspeed, 0:08:10.240 --> 0:08:13.960 God bless. I do not envy you. I so gratefully 0:08:14.040 --> 0:08:18.440 have a fake job. You know what do envy? I 0:08:18.480 --> 0:08:21.680 was buying the products that follow this utterly seamless ad break, 0:08:21.680 --> 0:08:25.600 which will likely echo my exact sentiments on literally every 0:08:25.720 --> 0:08:36.240 issue ever. And we're back. The significance of this failure, 0:08:36.280 --> 0:08:38.960 which isn't a breach, by the way, and in many respects, 0:08:39.000 --> 0:08:42.160 is far worse, at least with destruction it courst is 0:08:42.200 --> 0:08:44.520 not its damage to individual users, but to the amount 0:08:44.559 --> 0:08:47.800 of technical infrastructure that runs on Windows, and that so 0:08:47.920 --> 0:08:51.400 much of our global infrastructure relies on automated enterprise software 0:08:51.400 --> 0:08:56.240 that when it goes wrong, breaks everything. It isn't about 0:08:56.240 --> 0:08:58.360 the number of computers, but the amount of them that 0:08:58.480 --> 0:09:01.960 underpin things like security checkpoints or systems that run airlines 0:09:02.040 --> 0:09:05.400 or banks or hospitals, all running as much automated software 0:09:05.400 --> 0:09:08.240 as possible so that the costs can be kept down. 0:09:08.720 --> 0:09:12.400 Hey remember the raw economy. Jesus fucking The problem here 0:09:12.440 --> 0:09:15.320 is systemic that there's a company that the majority of 0:09:15.320 --> 0:09:18.240 people affected by the outage had no idea existed until 0:09:18.679 --> 0:09:22.280 well a day or two ago, that Microsoft trusted to 0:09:22.280 --> 0:09:23.959 the extent that they were able to push an update 0:09:24.000 --> 0:09:25.840 that broke the back of a chunk of the world's 0:09:25.880 --> 0:09:30.920 digital infrastructure. Microsoft a company, instead of building the kind 0:09:30.920 --> 0:09:33.960 of rigorous security protocols that would say, I don't know, 0:09:34.120 --> 0:09:36.480 rigorously tests something that connects to what seems to be 0:09:36.480 --> 0:09:41.200 a huge portion of Windows computers, Well, they just chose 0:09:41.240 --> 0:09:44.520 to do something else. They've just screwed the fuck up. 0:09:44.800 --> 0:09:47.920 As pointed out by Whir, the company vets and cryptographically 0:09:48.000 --> 0:09:50.920 signs all kernel drivers, which is sensible and good because 0:09:50.960 --> 0:09:53.360 kernel drivers have an incredible amount of access and thus 0:09:53.360 --> 0:09:57.079 can inflict serious harm. With this testing process, usually taking 0:09:57.120 --> 0:10:01.840 several weeks. What happened Microsoft? How did this slip through 0:10:01.880 --> 0:10:06.439 Microsoft's fingers? Well, for this to have happened, two companies 0:10:06.480 --> 0:10:09.920 needed to screw up epically in boy, fucking howardy did they? 0:10:11.000 --> 0:10:13.840 What we're seeing isn't just one major fuck up, but 0:10:13.880 --> 0:10:17.160 the first of what will be many systemic failures, some small, 0:10:17.440 --> 0:10:20.679 some potentially larger, that are the natural byproduct of the 0:10:20.720 --> 0:10:23.120 growth of all costs ecosystem, where any attempt to save 0:10:23.200 --> 0:10:26.120 money by outsourcing major systems is one that must simply 0:10:26.160 --> 0:10:30.000 be taken to please the beautiful, sexy shareholder that they 0:10:30.040 --> 0:10:33.160 all love so much. And this is a problem with 0:10:33.200 --> 0:10:36.720 the digitization of society, or more specifically, the automation of 0:10:36.760 --> 0:10:40.480 once manual tasks. It introduces a single point of failure, 0:10:40.600 --> 0:10:44.280 or rather several of them, or clustered together like a 0:10:44.360 --> 0:10:48.320 rat king or a Katamari. Our world, our lifestyle, and 0:10:48.400 --> 0:10:52.680 our economy is dependent on automation and computerization, with these 0:10:52.679 --> 0:10:56.440 systems in turn dependent on other systems to work, and 0:10:56.520 --> 0:11:00.240 if one of those systems breaks, the effects rick shay 0:11:00.280 --> 0:11:02.319 outwards like ripples mean you cast a rock in a 0:11:02.400 --> 0:11:06.200 lake or throw a body in. For some listeners, Freddy's 0:11:06.240 --> 0:11:09.200 CrowdStrike cockup is just the latest example of this, but 0:11:09.240 --> 0:11:11.720 it isn't the only one. Some of you might remember 0:11:11.720 --> 0:11:14.040 the Solar Winds hacked back in twenty twenty, where Russian 0:11:14.080 --> 0:11:17.160 state link hackers gained access to an estimate eighteen thousand 0:11:17.240 --> 0:11:21.120 companies in public sector organizations including NATO, the European Parliament, 0:11:21.120 --> 0:11:24.000 the US Treasury Department, and the UK's National Health Service 0:11:24.000 --> 0:11:28.680 by compromising just one service, Solar wins Oryan Remember when 0:11:28.760 --> 0:11:31.480 Octa some of you might know Octa is a company 0:11:31.480 --> 0:11:34.160 that makes software that handles authentication for a bunch of websites, 0:11:34.200 --> 0:11:36.640 governments and businesses. Well, when they got hacked in twenty 0:11:36.640 --> 0:11:40.080 twenty three, they then lied about the scale of the breach. Hey, 0:11:40.080 --> 0:11:42.400 do you remember when those hackers leap frogged from Octa 0:11:42.440 --> 0:11:45.360 to a bunch of other companies like cloud Flare. Yeah, 0:11:45.400 --> 0:11:49.280 they provide the content delivery services and the services that 0:11:49.320 --> 0:11:52.320 protect websites from being well brought down by a bunch 0:11:52.360 --> 0:11:56.000 of bots. From much the entire Internet, everything feels like 0:11:56.080 --> 0:11:59.640 it's being held up by like twigs and chewing gum. 0:12:00.040 --> 0:12:02.080 You probably know the quote no man is an island, 0:12:02.200 --> 0:12:04.560 and it's especially true when we're talking about tech, because 0:12:04.600 --> 0:12:07.960 when you scratch beneath the surface, every system that looks 0:12:08.000 --> 0:12:11.600 like it's independent is actually heavily, heavily dependent on services 0:12:11.600 --> 0:12:14.760 and software provided by a very small number of companies, 0:12:14.800 --> 0:12:17.640 many of whom are not particularly good. And this is 0:12:17.679 --> 0:12:20.440 as much a cultural failing as it is a technological one, 0:12:20.480 --> 0:12:23.680 the result of a management culture geared towards value extraction, 0:12:23.840 --> 0:12:27.439 building systems that build monopolies by attaching themselves to other monopolies. 0:12:28.280 --> 0:12:31.720 CrowdStrike went public in twenty nineteen and immediately popped on 0:12:31.920 --> 0:12:34.560 its first day of trading thanks to wall streets appreciation 0:12:34.640 --> 0:12:37.480 of them moving away from a focused approach to serving 0:12:37.559 --> 0:12:41.120 large enterprise clients, building products now for small and medium 0:12:41.160 --> 0:12:44.880 sized businesses by selling through channel partners, in effect outsourcing 0:12:44.920 --> 0:12:47.760 both product sales and the relationship with the client that 0:12:47.880 --> 0:12:51.120 would tailor a business a solution to said client. Especially 0:12:51.120 --> 0:12:54.120 when something is so serious like this, I want you 0:12:54.200 --> 0:12:57.079 to really think about this and think about this problem, 0:12:57.120 --> 0:12:59.640 because the problem isn't so much selling to small businesses 0:12:59.679 --> 0:13:02.640 or media businesses. It's the fact that CrowdStrike made its 0:13:02.720 --> 0:13:06.880 money selling to the enterprise and specializing in that, and 0:13:06.920 --> 0:13:09.199 that's the thing. When you broaden out, when you must 0:13:09.240 --> 0:13:12.400 grow in all directions, at all times, in all ways 0:13:12.440 --> 0:13:16.720 to please the horny beasts of Wall Street, you lose 0:13:16.760 --> 0:13:21.040 your focus. But that isn't the only problem, because Crowdstrike's 0:13:21.040 --> 0:13:24.720 culture appears to also fucking suck. A recent Glassloor entry 0:13:24.760 --> 0:13:28.440 referred to CrowdStrike as great tech with terrible culture with 0:13:28.559 --> 0:13:31.480 no work life balance, with leadership that does not care 0:13:31.520 --> 0:13:35.000 about employee well being. Another from June twenty twenty four 0:13:35.040 --> 0:13:38.080 claim that CrowdStrike was changing its culture for the street 0:13:38.160 --> 0:13:40.640 with KPIs as in metrics related to your success at 0:13:40.679 --> 0:13:44.640 the company, driving behavior more than building relationships, with a 0:13:44.679 --> 0:13:48.000 serious lack of experience in the public sector in senior management. 0:13:49.080 --> 0:13:51.920 So glad that this company is selling intellect government anyway. 0:13:51.960 --> 0:13:55.880 Moving on, others complained of micromanagement, with one claiming that 0:13:55.960 --> 0:13:59.000 management is the biggest issue, with managers asking way too 0:13:59.080 --> 0:14:01.440 much of you and it doesn't matter if you do 0:14:01.480 --> 0:14:03.800 what they ask since they're not even around to check 0:14:03.800 --> 0:14:07.080 on you, and another saying that management is arrogant and 0:14:07.280 --> 0:14:11.040 needed to stop lying to the market on product capability. 0:14:11.640 --> 0:14:14.200 That's what I love to see, we all love to 0:14:14.240 --> 0:14:17.719 see that. I'm very happy to read that, And while 0:14:17.760 --> 0:14:20.520 I can't say for sure, I'd imagine an organization with 0:14:20.640 --> 0:14:23.720 such powerful signs of growth at all costs thinking a 0:14:23.720 --> 0:14:26.320 place where you and I quote have to get used 0:14:26.320 --> 0:14:28.840 to the pressure, that's a clique that you're not in. 0:14:29.480 --> 0:14:32.240 Likely isn't giving its quality assurance teams the time and 0:14:32.280 --> 0:14:34.680 the space to make sure that there aren't any Kaiju 0:14:34.800 --> 0:14:39.640 level security threats baked into an update. And that assumes 0:14:39.680 --> 0:14:42.400 it actually has a significant QA team in house and 0:14:42.480 --> 0:14:45.280 hasn't just this with many companies outsourced the work to 0:14:45.360 --> 0:14:48.840 a body shop like Wypro or Emphasis or Tartar Consultancy. 0:14:50.320 --> 0:14:52.000 But for a moment, I'm going to change gears a 0:14:52.000 --> 0:14:54.480 little to try and explain what actually happened and why. 0:14:54.520 --> 0:14:57.160 It suggests that the issue is likely the product of 0:14:57.200 --> 0:15:01.240 cost cutting and institutional failure within CrowdStrike. In the aftermath 0:15:01.280 --> 0:15:04.640 of Friday's incident, we've seen some analyses about what actually 0:15:04.680 --> 0:15:08.200 went down with them first some throat clearing. I haven't 0:15:08.320 --> 0:15:11.240 verified this stuff independently. From what I've read, though, and 0:15:11.240 --> 0:15:14.920 from speaking to developers, this all seems relatively plausible, but 0:15:15.880 --> 0:15:18.560 maybe worth googling this a little yourself. But I'm going 0:15:18.640 --> 0:15:21.720 to give it a go. So the kernel driver at 0:15:21.720 --> 0:15:24.960 fort was written with a programming language called C plus plus. 0:15:25.000 --> 0:15:27.120 This language was developed in the nineteen eighties and it's 0:15:27.200 --> 0:15:30.520 very good for writing high performance applications, anything where you're 0:15:30.560 --> 0:15:33.480 concerned about speed, like the Interenno's operating system or a 0:15:33.560 --> 0:15:37.240 video game. It's pretty popular for that, and it's so 0:15:37.360 --> 0:15:41.080 pretty dangerous too, so dangerous in fact, that it's often 0:15:41.120 --> 0:15:45.520 referred to as an unsafe language. Without getting two into 0:15:45.520 --> 0:15:48.000 the weeds. C plus plus makes it incredibly easy to 0:15:48.000 --> 0:15:49.880 shoot yourself and the foot, the ars, and the dick. 0:15:49.920 --> 0:15:52.760 At the same time, it's big, complex and has few 0:15:52.800 --> 0:15:55.920 safeguards while providing many opportunities for developers to screw up 0:15:56.480 --> 0:16:01.400 very badly. Like the languages derived from C, it forces 0:16:01.480 --> 0:16:03.800 developers to deal with a lot of low level stuff 0:16:04.240 --> 0:16:06.560 like handling memory allocation that you don't really have to 0:16:06.600 --> 0:16:09.600 deal with in many popular languages like Python, Java, Russ, 0:16:09.600 --> 0:16:12.640 Swift or Sea sharp. And this matters because if you 0:16:12.680 --> 0:16:15.800 screw this up, your code will break, or I don't know, 0:16:15.960 --> 0:16:19.840 it might introduce some kind of potentially disastrous security vulnerability. 0:16:21.160 --> 0:16:24.240 In twenty nineteen, Microsoft researchers said that seventy percent of 0:16:24.280 --> 0:16:27.560 all security vulnerabilities were the result of memory management issues, 0:16:27.920 --> 0:16:30.040 and I doubt that figure has changed much since then. 0:16:30.800 --> 0:16:33.240 And earlier this year, the White House Office of the 0:16:33.320 --> 0:16:37.280 National Cyber Director urged developers to stop using unsafe languages 0:16:37.640 --> 0:16:40.160 like C and C plus plus and start using modern 0:16:40.200 --> 0:16:44.520 and safer alternatives like Rust. With me so far, ah, So, 0:16:44.680 --> 0:16:47.400 from what I've read, the CrowdStrike Falcon sensor kernel driver 0:16:47.520 --> 0:16:51.160 crash because it had something called a null pointer error. Essentially, 0:16:51.240 --> 0:16:53.760 the developer wrote some code that told the program to 0:16:53.800 --> 0:16:56.520 look for a memory location that didn't exist, and didn't 0:16:56.520 --> 0:16:59.840 write any safeguards to protect against them. When this happened, 0:17:00.080 --> 0:17:03.920 the driver and so the operating system crashed. This is 0:17:03.920 --> 0:17:06.280 a rookie mistake, and I've talked to multiple developers that 0:17:06.320 --> 0:17:08.399 have backed this up. If you take an introductory C 0:17:08.480 --> 0:17:11.280 plus plus programming class at university, they'll cover this in 0:17:11.320 --> 0:17:14.520 the first year. Kind of boggles the mind how trivial 0:17:14.520 --> 0:17:16.280 a mistake this is, and how it made it into 0:17:16.320 --> 0:17:19.160 production code, which is the code that goes out into 0:17:19.200 --> 0:17:21.400 the real world, and how it wasn't caught either by 0:17:21.400 --> 0:17:25.840 CrowdStrike or by Microsoft, who are supposedly obligated to vet 0:17:25.880 --> 0:17:29.280 this driver, and if the reports are true, someone really 0:17:29.359 --> 0:17:33.760 really really screwed up, really badly. But if you don't 0:17:33.800 --> 0:17:36.720 want to screw up, if you want to really do 0:17:36.840 --> 0:17:39.480 well in life, I advise you to buy one of 0:17:39.520 --> 0:17:43.800 the following products or services, which I of course fully understand, 0:17:43.920 --> 0:17:51.639 know all about and won't be embarrassed by. 0:17:53.720 --> 0:17:54.440 And we're back. 0:17:54.560 --> 0:17:56.120 And to be clear, I don't want you to think 0:17:56.119 --> 0:17:59.920 that I'm letting Microsoft off the hook either, assuming the 0:18:00.040 --> 0:18:02.679 kernel driver testing roles are still being done in house. 0:18:03.040 --> 0:18:05.280 Do you think that these testers who have likely seen 0:18:05.320 --> 0:18:07.879 their friends laid off at a time when Microsoft was 0:18:07.960 --> 0:18:11.439 highly profitable and denied raises, when their welfared CEO probably 0:18:11.480 --> 0:18:14.399 took home over one hundred million dollars in salary for 0:18:14.440 --> 0:18:16.800 a job he's eminently bad at. Do you think these 0:18:16.840 --> 0:18:19.000 people doing their best work? Do you think they go 0:18:19.080 --> 0:18:22.280 into a jazz full of piss and vinegar ready to 0:18:22.359 --> 0:18:25.600 save the world, or do you think they hate their 0:18:25.680 --> 0:18:28.199 job and they're being forced to do too much and 0:18:28.200 --> 0:18:30.680 they're miserable, And the people that knew what the fuck 0:18:30.840 --> 0:18:33.200 was going on haven't been fired, and the people who 0:18:33.200 --> 0:18:35.400 managed those people and the people that wrote the code 0:18:35.400 --> 0:18:38.399 that they're edited. Do you think anyone knows what the 0:18:38.400 --> 0:18:44.600 hell is going on? No, they don't, And this is 0:18:44.640 --> 0:18:48.399 the culture that's poisoned almost the entirety of Silicon Valley. 0:18:49.400 --> 0:18:52.320 What we're seeing now is the societal cost of moving 0:18:52.400 --> 0:18:56.600 fast and breaking things of people like Mark Andresen considering 0:18:56.720 --> 0:19:00.480 risk management the enemy of hiring and firing things. Thousands 0:19:00.520 --> 0:19:03.359 of people, tens of thousands in some case, to please 0:19:03.400 --> 0:19:06.639 Wall Street, are seeking as many new possible ways to 0:19:06.640 --> 0:19:09.879 make as much money as possible, to show shareholders that 0:19:10.000 --> 0:19:13.000 you'll grow, even if doing so means growing at a 0:19:13.040 --> 0:19:17.439 pace that makes it impossible to sustain organizational and cultural stability. 0:19:18.680 --> 0:19:21.320 When you aren't intentional on the people you hire and retain, 0:19:21.680 --> 0:19:23.800 the people you fire, the things that you build, the 0:19:23.840 --> 0:19:27.720 way that they are deployed, maintaining your systems, understanding how 0:19:27.800 --> 0:19:32.040 and why things were written, the decisions that were made five, ten, 0:19:32.080 --> 0:19:35.080 and fifteen years ago, you're going to lose the people 0:19:35.119 --> 0:19:37.439 to understand the problems they're solving, and thus lack the 0:19:37.520 --> 0:19:40.879 organizational ability to understand the ways the problems might be 0:19:40.920 --> 0:19:44.919 solved in the future, or disasters might be averted. This 0:19:45.040 --> 0:19:48.400 is dangerous, and it's also a dark warning for the future. 0:19:49.320 --> 0:19:52.120 Do you think the Facebook or Microsoft or Google, all 0:19:52.160 --> 0:19:54.280 of whom have laid off over ten thousand people in 0:19:54.320 --> 0:19:57.200 the last year, have done so in a conscientious way, 0:19:57.440 --> 0:19:59.960 in a knowledgeable way, a people focused a way, in organized, 0:20:00.040 --> 0:20:02.639 zationally rigorous way that means that the people are left 0:20:03.000 --> 0:20:06.240 who understand how their systems run and the inherent issues 0:20:06.280 --> 0:20:09.920 built into them. Do you think the management types obsessed 0:20:09.960 --> 0:20:13.800 with unsustainable AI bullshit are investing heavily in making sure 0:20:13.800 --> 0:20:17.680 that their organizations are rigorously protected against, say, one bad 0:20:17.760 --> 0:20:21.280 line of code or one dipshit error. Did they even 0:20:21.359 --> 0:20:23.679 know who wrote the code of their current systems? Is 0:20:23.720 --> 0:20:26.240 that person still there? Do they have their email and 0:20:26.280 --> 0:20:30.040 their phone number? Is that person at least contracted to 0:20:30.080 --> 0:20:32.240 make sure that something nuanced about the system in question 0:20:32.280 --> 0:20:37.560 isn't mistakenly removed or changed or quote fixed. No, now 0:20:37.560 --> 0:20:40.720 they're not, They're gone. They're not there anymore. Only a 0:20:40.800 --> 0:20:43.240 few months ago, Google laid off two hundred employees in 0:20:43.280 --> 0:20:46.320 the core of its organization, outsourcing their roles to Mexico 0:20:46.359 --> 0:20:49.680 and India in a cost cutting measure. The quarter after 0:20:49.760 --> 0:20:53.840 the company made twenty three billion dollars in profit I'm 0:20:53.920 --> 0:20:56.840 jumping to Google because they're just probably next in one 0:20:56.880 --> 0:21:02.040 of these horrible breaches or sorry, not breaches. Silicon Valley 0:21:02.040 --> 0:21:04.720 in big tech writ large is not built to protect 0:21:04.720 --> 0:21:07.159 against situations like the one we saw on Friday and 0:21:07.200 --> 0:21:10.040 the damage we're going to get from CrowdStrike because the 0:21:10.080 --> 0:21:14.000 culture's cancer. He values growth or costs with no respect 0:21:14.000 --> 0:21:17.159 for the human capital that empowers organizations or the value 0:21:17.200 --> 0:21:21.080 of building rigorous, quality focused products that are maintained over time. 0:21:22.520 --> 0:21:24.879 You know me, I'm a nasty little bitch. What are 0:21:24.880 --> 0:21:28.000 more on the nose? Example, George Kurtz, the CEO and 0:21:28.040 --> 0:21:30.840 co founder of CrowdStrike, said in twenty twenty that not 0:21:31.080 --> 0:21:34.679 one time has he regretted firing someone too fast, in 0:21:34.720 --> 0:21:37.520 a conversation where he argued that tech executives were becoming 0:21:37.560 --> 0:21:42.040 too obsessed with culture, and in a stunning act of foreshadowing, 0:21:42.359 --> 0:21:44.880 when he was the chief technology officer at McAfee, best 0:21:44.920 --> 0:21:47.239 known as the company that makes antivirus software that they 0:21:47.280 --> 0:21:49.840 sell to your granddad and that they ship with computers 0:21:49.880 --> 0:21:52.919 and you immediately uninstalled, while he oversaw an update that 0:21:52.920 --> 0:21:55.199 treated in the central part of Windows XP as a 0:21:55.280 --> 0:21:58.680 virus quarantining it and sending the computer into a boot loop. 0:21:58.880 --> 0:22:01.520 It's almost a little too on the nose. They're calling 0:22:01.560 --> 0:22:04.720 him the prabagar Ragavan of security. It's a very bad deal. 0:22:05.280 --> 0:22:08.760 But dear listener, this is just the beginning. Big Tech is, 0:22:08.800 --> 0:22:11.720 to quote trivium, in the throes of perdition, teetering over 0:22:11.720 --> 0:22:14.800 the edge of the abyss, finally paying the harsh cost 0:22:14.800 --> 0:22:18.439 of building systems as fast as possible. But let's be honest, 0:22:18.960 --> 0:22:23.040 they're not paying the cost we are. This isn't simply 0:22:23.119 --> 0:22:26.200