WEBVTT - TechStuff Classic: Bad Computer Bugs 0:00:04.440 --> 0:00:12.239 Welcome to tech Stuff, a production from iHeartRadio. Hey there, 0:00:12.320 --> 0:00:15.800 and welcome to tech Stuff. I'm your host, Jonathan Strickland. 0:00:15.840 --> 0:00:19.079 I'm an executive producer with iHeartRadio, and how the tech 0:00:19.120 --> 0:00:22.480 are you? It is time for a tech Stuff classics episode. 0:00:22.880 --> 0:00:27.920 This episode originally published on December seventh, twenty sixteen. It 0:00:28.000 --> 0:00:33.000 is called bad Computer Bugs. No, it was twenty sixteen, 0:00:33.080 --> 0:00:36.680 the year where like bedbugs became a big news item. 0:00:36.840 --> 0:00:40.560 Like there was a year where it just bed bugs 0:00:40.600 --> 0:00:43.000 were in the news, and I'm wondering if that was 0:00:43.000 --> 0:00:45.559 twenty sixteen. At this point it goes to show that 0:00:45.600 --> 0:00:48.800 I haven't actually listened back to this episode, but yeah, 0:00:48.840 --> 0:00:53.279 Bad Computer Bugs. It originally published December seventh, twenty sixteen. 0:00:53.760 --> 0:00:57.400 Let's take a listen, now, shall we. I have to 0:00:57.480 --> 0:01:02.760 address a bit of apocryphal history, and regrettably it's a 0:01:02.800 --> 0:01:05.760 story that we've repeated on tech Stuff. So I'm sad 0:01:05.800 --> 0:01:10.800 to admit that I was complicit, although unknowingly, in the 0:01:10.840 --> 0:01:15.160 spread of misinformation, and that all has to do with 0:01:15.200 --> 0:01:18.360 the origin of the term bug to describe a flaw 0:01:18.480 --> 0:01:22.720 in programming. So here's the popular story, the one that 0:01:22.959 --> 0:01:29.760 we have accidentally promoted on tech stuff without knowing that 0:01:29.800 --> 0:01:33.120 we were in the wrong. It goes that Grace Hopper, 0:01:33.440 --> 0:01:35.959 who was an early computer scientist who rose to the 0:01:36.040 --> 0:01:39.480 rank of Rear admiral in the US Navy, coined the 0:01:39.560 --> 0:01:43.760 phrase bug after discovering a moth guming up Harvard's Mark 0:01:43.880 --> 0:01:49.000 II calculator, a literal bug. Generally speaking, the story tends 0:01:49.040 --> 0:01:53.080 to be set in nineteen forty five, and there is 0:01:53.200 --> 0:01:56.280 even a note in the logbook that reads first actual 0:01:56.400 --> 0:02:00.240 case of bug being found that's attributed to Grace Hopper. 0:02:01.200 --> 0:02:05.559 But there are several points that are wrong in this story. First, 0:02:05.640 --> 0:02:08.480 the year. It didn't happen in nineteen forty five. It 0:02:08.520 --> 0:02:11.520 happened on September ninth, nineteen forty seven. We know because 0:02:11.560 --> 0:02:15.480 there's a logbook. At the logbook that marks the incident 0:02:15.760 --> 0:02:18.680 not only has the notes, it actually has the moth 0:02:18.840 --> 0:02:23.760 taped into the book itself. It's taped onto the page. Second, 0:02:23.840 --> 0:02:27.240 Grace Hopper wasn't the person to discover the moth or 0:02:27.440 --> 0:02:31.640 make that log entry. She did tell the story about 0:02:31.680 --> 0:02:35.040 the moth several times, but it wasn't in the context 0:02:35.120 --> 0:02:38.880 of finding it or logging it. She just told the 0:02:38.880 --> 0:02:41.040 story that, yeah, we really did have a bug in 0:02:41.080 --> 0:02:44.920 the system, and most importantly, the word bug had already 0:02:44.960 --> 0:02:49.200 been used to describe design flaws for decades before the 0:02:49.240 --> 0:02:52.679 Mark II was even designed. In fact, if you look 0:02:52.680 --> 0:02:55.840 at the logbook, this makes sense. It says first actual 0:02:55.919 --> 0:02:59.880 case of bug being found. That sentence doesn't make sense 0:03:00.200 --> 0:03:04.399 unless you've already used the word bug to describe a flaw, 0:03:04.880 --> 0:03:08.760 because you wouldn't say first actual case of bug being found. 0:03:09.320 --> 0:03:13.920 The wording doesn't make any sense. The context makes no sense. Sadly, 0:03:13.960 --> 0:03:17.639 there are documented quotes dating back to the nineteenth century 0:03:18.120 --> 0:03:21.480 using the word bug to mean a design fault, and 0:03:21.520 --> 0:03:25.360 it could go back even further than that. So is 0:03:25.400 --> 0:03:28.200 with much regret that I admit I have unwittingly contributed 0:03:28.240 --> 0:03:31.080 to a bit of misleading folklore making the rounds. But 0:03:31.120 --> 0:03:33.480 I'm glad I can take this opportunity to address it. 0:03:34.120 --> 0:03:36.640 All Right, So let's talk about design bugs, and I'll 0:03:36.680 --> 0:03:41.520 be covering several goofs, mistakes, flubs, flaws, and outright catastrophes 0:03:41.640 --> 0:03:45.640 in this episode. But one thing I'm not necessarily going 0:03:45.680 --> 0:03:49.600 to cover our software vulnerabilities that were later exploited either 0:03:49.680 --> 0:03:53.280 by opportunistic hackers or white hats who are just trying 0:03:53.320 --> 0:03:57.720 to improve system security. Those vulnerabilities are common in many 0:03:57.760 --> 0:04:01.000 types of software and arise not just through mistakes, but 0:04:01.480 --> 0:04:05.720 sometimes simple oversights. And I think it might be more 0:04:05.760 --> 0:04:08.080 fun to look at some real bugs, like stuff that 0:04:08.520 --> 0:04:11.320 made things go wrong, stuff that may have rendered a 0:04:11.320 --> 0:04:15.080 program defunct or otherwise caused headaches. Now I'm going to 0:04:15.120 --> 0:04:17.400 make an exception to this. I'm going to start off 0:04:17.440 --> 0:04:21.000 with the Ping of death. And I only mention it 0:04:21.040 --> 0:04:24.800 because it has an awesome name. Now, this flaw caused 0:04:24.839 --> 0:04:27.680 headaches back in nineteen ninety five and ninety six. It 0:04:27.760 --> 0:04:32.640 was a flawed IP fragmentation reassembly code, and it became 0:04:32.680 --> 0:04:35.560 possible to crash lots of different types of computers using 0:04:35.560 --> 0:04:39.760 different operating systems, although Windows machines were particularly vulnerable, and 0:04:39.839 --> 0:04:42.680 this particular flaw would make a Windows machine revert to 0:04:42.760 --> 0:04:46.200 the dreaded blue screen of death. And it all happened 0:04:46.200 --> 0:04:49.800 by sending a special ping packet over the Internet. So, 0:04:49.920 --> 0:04:52.120 for those of you who aren't familiar with what that is, 0:04:52.200 --> 0:04:55.680 a ping is essentially a simple message that checks for 0:04:55.800 --> 0:04:59.320 a connection between two computers. You send one ping from 0:04:59.320 --> 0:05:01.520 a computer to a another one and look for a 0:05:01.560 --> 0:05:04.479 response so that way you verify there is in fact 0:05:04.520 --> 0:05:07.240 a connection. You can also tell other things like how 0:05:07.320 --> 0:05:10.800 fast is that connection between those two computers. Now, in 0:05:10.839 --> 0:05:13.800 this case, you would have to actually design a malformed 0:05:13.920 --> 0:05:17.240 ping request and send that to a target and it 0:05:17.279 --> 0:05:21.520 would bring that target down. That's the only security vulnerability 0:05:21.560 --> 0:05:24.200 story I really wanted to focus on. The others are 0:05:24.240 --> 0:05:27.960 all just design flaws. And let's begin with the bug 0:05:27.960 --> 0:05:30.120 that inspired me to do this episode in the first place, 0:05:30.200 --> 0:05:33.720 that Spotify bug I mentioned earlier. Ours Tetnica wrote a 0:05:33.760 --> 0:05:36.479 piece on it in November twenty sixteen, but the problem 0:05:36.560 --> 0:05:39.560 seems to date back at least as far as June 0:05:39.640 --> 0:05:43.760 twenty sixteen, and that's when a few savvy Spotify users 0:05:43.880 --> 0:05:47.680 noticed some unusual activities on their computers, and it took 0:05:47.680 --> 0:05:49.760 a little bit of detective work, but they discovered that 0:05:49.800 --> 0:05:53.600 Spotify was apparently generating a huge amount of data on 0:05:53.640 --> 0:05:58.760 a daily basis, like gigabytes of data per day. And 0:05:58.800 --> 0:06:02.039 the culprit turned out to be a vacuum process for 0:06:02.120 --> 0:06:06.400 a database file containing the string mercury dot dB. Now, 0:06:06.400 --> 0:06:10.120 the vacuum process is the digital equivalent of vacuum ceiling. 0:06:10.240 --> 0:06:12.960 It's meant to repack data so that it takes up 0:06:13.040 --> 0:06:16.200 less space on a drive. Now, this involves building a 0:06:16.240 --> 0:06:19.640 new file to maximize efficiency, which is a good thing 0:06:19.800 --> 0:06:26.040 generally speaking. The problem was that Spotify's version was making 0:06:26.040 --> 0:06:28.400 it happen way too frequently, like on the order of 0:06:28.440 --> 0:06:32.560 once every few minutes. So that's not generally necessary. You 0:06:32.560 --> 0:06:35.400 don't need to rebuild a database file every few minutes 0:06:35.440 --> 0:06:38.520 to make sure it's the most efficient size it can be. 0:06:39.640 --> 0:06:42.919 So each rebuild represented a relatively small amount of data, 0:06:43.000 --> 0:06:45.960 but over time it added up, which meant that if 0:06:46.000 --> 0:06:48.839 you had Spotify on on your computer, even if it 0:06:48.880 --> 0:06:51.640 was just running in the background, it would be generating 0:06:51.640 --> 0:06:55.760 gigabytes worth of information rewriting this file over and over. Now, 0:06:55.800 --> 0:06:58.600 it wasn't filling up a hard drive. It was just 0:06:58.839 --> 0:07:04.159 overwriting the same file. Now, if it had been filling 0:07:04.240 --> 0:07:06.240 up a hard drive, people would have noticed much earlier, 0:07:06.279 --> 0:07:09.320 and it wouldn't have just been savvy Spotify users, because 0:07:09.320 --> 0:07:12.200 you would suddenly notice, hey, I can't save anything to 0:07:12.200 --> 0:07:15.840 my hard drive because everything's filling up. Instead, again, it 0:07:15.920 --> 0:07:18.480 was just sort of writing and deleting, and writing and 0:07:18.520 --> 0:07:21.320 deleting the same file over and over again, and that 0:07:21.360 --> 0:07:23.440 probably doesn't sound like a big deal, but it is 0:07:23.480 --> 0:07:27.160 a problem if you're using a solid state drive or SSD. 0:07:28.040 --> 0:07:30.640 So one of the drawbacks of an SSD is that 0:07:30.720 --> 0:07:34.560 over time it loses storage capacity. Like you can store 0:07:34.960 --> 0:07:39.040 less data on an SSD over time. Now, by overtime 0:07:39.080 --> 0:07:41.280 I generally mean over a great deal of time and 0:07:41.320 --> 0:07:44.040 a lot of different data being written to it and overwritten. 0:07:45.880 --> 0:07:49.080 Generally speaking, most of us end up replacing our drives 0:07:49.120 --> 0:07:51.640 before we get to a point where the loss of 0:07:51.720 --> 0:07:55.240 capacity is a real issue. But similar in a way 0:07:55.280 --> 0:07:57.400 to how a battery can lose its ability to hold 0:07:57.400 --> 0:08:01.200 a full charge after you've gone through lots of charging 0:08:01.240 --> 0:08:04.520 and discharging cycles, you know how a battery won't be 0:08:04.600 --> 0:08:07.200 able to hold as much even if it says it's 0:08:07.280 --> 0:08:09.560 up to one hundred percent, But one hundred percent doesn't 0:08:09.640 --> 0:08:12.280 last you as long as it used to. That's because 0:08:12.440 --> 0:08:15.400 its capacity to hold a full charge has decreased over time. 0:08:16.920 --> 0:08:19.600 But let's say you've got a program that's just constantly 0:08:19.680 --> 0:08:23.480 overwriting data to your drive, you might discover that your 0:08:23.520 --> 0:08:28.280 SSD's useful lifespan has been drastically reduced. So as I 0:08:28.360 --> 0:08:32.360 record this episode, Spotify has already rolled out an updated 0:08:32.440 --> 0:08:35.000 version of its desktop application, and that, by the way, 0:08:35.080 --> 0:08:37.560 is the only version of Spotify that was affected. If 0:08:37.600 --> 0:08:41.360 you use web based Spotify or mobile Spotify, you're in 0:08:41.400 --> 0:08:43.959 the clear already. If you use a desktop version, as 0:08:43.960 --> 0:08:46.960 long as you have version one point zero point four 0:08:47.000 --> 0:08:51.880 to two or later, you are fine. But if you 0:08:51.920 --> 0:08:54.280 did have that earlier version and you just had Spotify 0:08:54.360 --> 0:08:57.360 running on in the background, chances are it was writing 0:08:57.360 --> 0:09:00.680 to your hard drive like crazy. So what about some 0:09:00.720 --> 0:09:03.440 of the other big bugs in computer history. Well, some 0:09:03.480 --> 0:09:06.280 of the real doozies involve our attempts to explore the 0:09:06.320 --> 0:09:10.080 final frontier. So we'll be talking about space a few 0:09:10.080 --> 0:09:12.520 times in this episode, and we'll start with an early 0:09:12.760 --> 0:09:16.720 US satellite. So first up is a nineteen sixty two 0:09:16.840 --> 0:09:20.760 blunder involving the Mariner one. So some backstory on this one. 0:09:21.720 --> 0:09:23.560 We're going to talk a lot about the Soviet Union 0:09:23.559 --> 0:09:26.560 in this episode too. It takes a couple of roles 0:09:26.640 --> 0:09:29.280 as we go on, But in this case, the then 0:09:29.480 --> 0:09:33.440 USSR had launched Sputnik into orbit in nineteen fifty seven, 0:09:33.440 --> 0:09:36.280 which really kicked off the space race and also was 0:09:36.400 --> 0:09:39.000 a big shot in the Cold War because the Soviet 0:09:39.080 --> 0:09:42.360 Union was essentially saying, hey, we can launch this into space, 0:09:42.400 --> 0:09:46.200 we could also launch something at you. In response, the 0:09:46.280 --> 0:09:48.200 US done sort of the same thing. They had launched 0:09:48.240 --> 0:09:51.360 some satellites into space, and the Mariner one was going 0:09:51.440 --> 0:09:54.240 to be a big, big feather in the cap of 0:09:54.320 --> 0:09:56.439 the US. The whole idea was to launch a probe 0:09:56.440 --> 0:09:59.400 that would be a flyby probe and it would go 0:09:59.440 --> 0:10:05.280 by Venus. So NASA, which was newly formed in nineteen 0:10:05.360 --> 0:10:08.360 sixty two, was taking control of this, and the budget 0:10:08.400 --> 0:10:12.000 for this particular project was eighteen point five million dollars, which, 0:10:12.080 --> 0:10:14.840 if you were to adjust for inflation, would be almost 0:10:14.880 --> 0:10:19.079 one hundred and fifty million dollars today, So one hundred 0:10:19.080 --> 0:10:21.920 and fifty million dollar project to launch the Mariner one 0:10:21.960 --> 0:10:25.200 and have it fly by Venus. But, as I'm sure 0:10:25.240 --> 0:10:27.960 you guys have figured out by now based upon the 0:10:28.120 --> 0:10:31.679 topic of this podcast, not all went according to plan. 0:10:33.000 --> 0:10:36.959 Not long at all. After the rocket launched from the 0:10:37.040 --> 0:10:40.680 launch pad, it began to veer off course, and neither 0:10:40.720 --> 0:10:43.600 the computer controls on the rocket or manual controls back 0:10:43.640 --> 0:10:47.880 at HQ could correct for the problem. The rocket's course 0:10:48.440 --> 0:10:50.360 was such that it was going to take it over 0:10:50.480 --> 0:10:54.000 shipping lanes, which meant there could be a potential catastrophe, 0:10:54.400 --> 0:10:58.040 and so a range safety officer made the difficult call 0:10:58.160 --> 0:11:00.800 and issued the command to blow the whole thing up 0:11:01.480 --> 0:11:05.080 just shy of three hundred seconds after it launched. So 0:11:05.120 --> 0:11:07.440 what happened? What why did it go off course in 0:11:07.440 --> 0:11:09.600 the first place? Well, there was a flaw in the 0:11:09.600 --> 0:11:13.400 spacecraft's guidance software which diverted the rocket, and no amount 0:11:13.440 --> 0:11:16.520 of commands from ground control could correct for it. After 0:11:16.559 --> 0:11:21.520 a lengthy investigation, NASA discovered the error was the result 0:11:21.600 --> 0:11:27.520 of a mistake transcribing handwritten notes into computer code. So 0:11:27.559 --> 0:11:32.160 someone just took some handwritten notes and misinterpreted one of them, 0:11:32.880 --> 0:11:38.720 and that one mistake was enough to crash the rocket 0:11:38.800 --> 0:11:44.760 or to necessitate it being destroyed. The great science fiction 0:11:45.280 --> 0:11:49.440 author Arthur C. Clark wrote that the Mariner one was 0:11:49.600 --> 0:11:54.360 wrecked by the most expensive hyphen in history, which isn't 0:11:54.400 --> 0:11:57.560 quite right, but it's pretty funny, I mean, come on, 0:11:57.679 --> 0:12:01.880 its humorous phrase. So the actual punctuation mark that caused 0:12:01.920 --> 0:12:05.040 the problem was not technically a hyphen. It was a 0:12:05.080 --> 0:12:09.440 superscript bar. Superscript bars, by the way, not a place 0:12:09.480 --> 0:12:12.760 where playwrights hang out. To get tore up. A superscript 0:12:12.760 --> 0:12:16.040 bar just means it's a horizontal bar that is above 0:12:16.200 --> 0:12:19.520 some other symbol. In this case, it was a radius symbol, 0:12:20.160 --> 0:12:23.760 and that was a symbol along with the superscript bar 0:12:24.040 --> 0:12:28.080 to describe a smoothing function, which means the formula was 0:12:28.120 --> 0:12:32.040 meant to calculate smoothed values over the time derivative of 0:12:32.080 --> 0:12:37.520 a radius. Now, without the smoothing function, tiny deviations in 0:12:37.640 --> 0:12:40.880 course sent commands to the rocket's thrusters to kick in 0:12:40.920 --> 0:12:44.760 big time to overcorrect for that problem. As an analogy, 0:12:44.840 --> 0:12:47.559 imagine you're driving a vehicle and you see a pothole 0:12:47.960 --> 0:12:50.560 in the road and you're approaching it, and instead of 0:12:51.120 --> 0:12:55.320 gently steering out of the way, you wrench the wheel 0:12:55.559 --> 0:12:57.680 really hard to the left or to the right in 0:12:57.760 --> 0:13:00.280 order to try and get around this pothole. That's kind 0:13:00.280 --> 0:13:02.360 of what was happening with the rocket. It didn't have 0:13:02.400 --> 0:13:05.439 the smoothing function and so as a result, it was 0:13:05.520 --> 0:13:09.640 having these wild deviations and course. So it wasn't a 0:13:09.720 --> 0:13:14.000 hyphen that caused the problem, was close enough. Our next 0:13:14.040 --> 0:13:16.480 space story takes place in nineteen ninety six with the 0:13:16.480 --> 0:13:22.480 European Space Agency's Ari Anne five flight five oh one rocket. Now, 0:13:22.520 --> 0:13:24.880 this rocket was to launch into space on June fourth, 0:13:25.040 --> 0:13:30.040 nineteen ninety six, and instead the rocket disintegrated forty seconds 0:13:30.080 --> 0:13:33.760 after taking off. So what the heck happened? Well, it 0:13:33.840 --> 0:13:37.760 largely had to do with the ESA reusing old work. 0:13:38.120 --> 0:13:40.880 This actually becomes a theme in this episode. One of 0:13:40.920 --> 0:13:44.960 the morals the of this entire podcast is, if you're 0:13:45.000 --> 0:13:50.040 designing something a successor to an earlier product, and you'd 0:13:50.080 --> 0:13:53.480 want to reuse some of the features that you created 0:13:53.520 --> 0:13:57.559 in your previous product, test the heck out of it 0:13:57.720 --> 0:14:01.160 in its new form, factor, because it could be that 0:14:01.200 --> 0:14:04.960 things that worked perfectly fine in the earlier model will 0:14:05.040 --> 0:14:10.160 go awry in the new one. That's what happened here. So, 0:14:10.400 --> 0:14:13.320 as you might guess from the name, the Ariyan five 0:14:13.600 --> 0:14:17.240 marked the fifth generation of launch vehicles under that name. 0:14:17.440 --> 0:14:22.200 The Arian four's inertial reference system would convert sixty four 0:14:22.240 --> 0:14:26.080 bit floating point numbers into a sixteen bit signed integer, 0:14:27.120 --> 0:14:32.600 and it worked just fine. But the Arian five's stats 0:14:32.640 --> 0:14:36.680 were beefier than its predecessor with faster engines, and that 0:14:36.880 --> 0:14:40.600 was where the problem really started. The engine output meant 0:14:40.720 --> 0:14:45.280 those sixty four bit floating point numbers were significantly larger 0:14:45.320 --> 0:14:48.640 than the ones generated by the engines on the Arean four. 0:14:48.880 --> 0:14:52.960 They didn't anticipate this, so during the conversion process there 0:14:53.040 --> 0:14:57.200 was actually data overflow, and that overflow caused both the 0:14:57.240 --> 0:15:00.760 backup computer and the primary computer board the Area N 0:15:00.880 --> 0:15:03.520 five to crash, and they crashed in that order. The 0:15:03.520 --> 0:15:07.080 backup computer crashed first, followed by the primary computer a 0:15:07.120 --> 0:15:10.120 couple of seconds later. The whole thing took less than 0:15:10.160 --> 0:15:15.680 a minute to go from launch to disintegration. Oops, now 0:15:15.720 --> 0:15:17.880 we're going to stick with space. But jump forward to 0:15:18.000 --> 0:15:23.480 nineteen ninety eight and the Mars Climate Orbiter. This was 0:15:24.360 --> 0:15:28.480 an unfortunate problem. So this particular spacecraft was meant to 0:15:28.520 --> 0:15:32.080 study Mars's climate, atmosphere and surface changes, and it was 0:15:32.120 --> 0:15:34.600 also supposed to be a kind of relay station for 0:15:34.720 --> 0:15:38.760 landers that would explore Mars's surface, but none of that 0:15:38.800 --> 0:15:42.560 would last because of some pretty significant goofs. So on 0:15:42.600 --> 0:15:46.720 September twenty third, nineteen ninety nine, the orbiter passed into 0:15:46.760 --> 0:15:51.120 the upper atmosphere of Mars and did so at a 0:15:51.200 --> 0:15:54.400 pretty low altitude. And this is what folks in the 0:15:54.440 --> 0:15:58.000 space industry call a bad thing. The drag on the 0:15:58.000 --> 0:16:02.120 spacecraft was significant, it began to fall apart and it 0:16:02.240 --> 0:16:08.880 was destroyed upon entering Mars's atmosphere. That's what happened. So 0:16:08.960 --> 0:16:13.280 the software guiding the orbiter was to blame, and it's 0:16:13.320 --> 0:16:16.880 a dumb, dumb mistake. It was supposed to make adjustments 0:16:16.880 --> 0:16:21.520 to the orbiter's flight in SI units, specifically in Newton's seconds. 0:16:22.600 --> 0:16:27.160 That's what the contract with Lockheed and NASA said, Newton seconds, 0:16:27.360 --> 0:16:31.160 use Si units for all of your all of your calculations. 0:16:31.840 --> 0:16:35.840 But the software instead made calculations in non SI units, 0:16:35.920 --> 0:16:42.080 namely pounds seconds. So Lockheed software gave information to NASA's 0:16:42.120 --> 0:16:46.920 systems using the wrong units of measure. NASA systems then 0:16:46.960 --> 0:16:50.320 took that information, assuming it was with the right units 0:16:50.320 --> 0:16:56.200 of measure, and executed commands based upon that. So this 0:16:56.400 --> 0:16:59.200 is why if you're ever in a math course and 0:16:59.840 --> 0:17:03.360 the teacher makes you stop in the middle of writing 0:17:03.360 --> 0:17:05.720 a problem on the board and says, where are your units? 0:17:06.400 --> 0:17:09.040 This is why you have to make sure you're using 0:17:09.040 --> 0:17:12.800 the right units, because if you're saying a number and 0:17:12.880 --> 0:17:15.800 you don't associate a unit with it, someone could make 0:17:15.840 --> 0:17:19.200 an incorrect decision based on that, and it could be disastrous, 0:17:19.320 --> 0:17:22.160 as it was with the case of this orbiter, the 0:17:22.160 --> 0:17:25.639 thrusters fired at four point four or five times the 0:17:25.760 --> 0:17:28.960 power they were supposed to, and the orbiter didn't stand 0:17:28.960 --> 0:17:32.480 a chance. And this was a pre expensive mistake. That 0:17:32.600 --> 0:17:34.960 mission's cost came in at three hundred and twenty seven 0:17:35.000 --> 0:17:38.720 point six million dollars. But on the bright side, with 0:17:38.840 --> 0:17:41.520 all of these stories, at least no human lives were 0:17:41.560 --> 0:17:44.320 ever in real danger as a result of the mistake. 0:17:45.440 --> 0:17:47.640 We're gonna take a quick break, and when we come back, 0:17:48.160 --> 0:18:01.800 we'll talk more about bad computer bugs. All right, Now, 0:18:01.880 --> 0:18:04.320 let's make a switch to AT and T, which is 0:18:04.359 --> 0:18:06.840 a company that had a pretty big problem with switches 0:18:07.040 --> 0:18:09.720 once upon a time. I'm talking about an issue that 0:18:09.720 --> 0:18:13.240 popped up on January fifteenth, nineteen ninety. That's when AT 0:18:13.280 --> 0:18:15.960 and T long distance customers discovered they were unable to 0:18:15.960 --> 0:18:19.600 make any long distance calls. Why why could they no 0:18:19.680 --> 0:18:24.359 longer reach anybody? Well? AT and t's long distance switches, 0:18:25.080 --> 0:18:29.520 which control that and allow for the actual connections to 0:18:29.560 --> 0:18:32.119 be made, were on the frints. They were trying to 0:18:32.160 --> 0:18:34.840 reboot over and over again. They were just stuck in 0:18:34.880 --> 0:18:39.040 a reboot cycle. Now, initially the company thought it was 0:18:39.080 --> 0:18:41.679 being hacked, but like I said at the top of 0:18:41.680 --> 0:18:44.920 the show, I'm not covering stories about hackers here. I'm 0:18:44.920 --> 0:18:48.920 talking about big design flaws that caused problems. So they 0:18:48.960 --> 0:18:52.320 weren't getting hacked. That's not what was going on with 0:18:52.320 --> 0:18:55.400 those one hundred and fourteen long distance switches. No, there 0:18:55.440 --> 0:18:58.560 was a design problem at fault. So what had happened 0:18:58.600 --> 0:19:00.919 was AT and T had rolled out an update to 0:19:01.000 --> 0:19:03.840 the code that managed the switches, and it was meant 0:19:03.880 --> 0:19:06.959 to increase the efficiency. It was meant to speed things up. 0:19:07.000 --> 0:19:09.720 But the problem was it sped things up so much 0:19:09.760 --> 0:19:13.000 that the system got caught up in itself. It gets 0:19:13.000 --> 0:19:14.679 pretty technical, but I can give you kind of an 0:19:15.119 --> 0:19:18.359 overview of what the problem was. All right, So each 0:19:18.400 --> 0:19:23.000 switch had a function that allowed it to alert the 0:19:23.040 --> 0:19:26.520 next switch down the line if things were starting to 0:19:26.520 --> 0:19:31.200 get hairy. So imagine that switch number one is handling traffic, 0:19:31.320 --> 0:19:34.080 but it's getting really close to capacity. So it sends 0:19:34.119 --> 0:19:36.320 a message over to switch number two and says, I 0:19:36.359 --> 0:19:39.520 can't take on any more work because if I do, 0:19:39.640 --> 0:19:43.680 I'll be overloaded. Switch too then says, no problem, I'll 0:19:43.720 --> 0:19:47.800 take on any oncoming work for you and we'll handle 0:19:47.840 --> 0:19:50.679 it from there. And if switched number two were to 0:19:50.760 --> 0:19:53.200 get into the same sort of situation, it would say 0:19:53.200 --> 0:19:55.119 the same thing to switch number three, and so on 0:19:55.200 --> 0:19:59.040 and so forth. Now, eventually each switch will contact the 0:19:59.040 --> 0:20:01.760 one below it and say, hey, how you doing there, 0:20:02.160 --> 0:20:05.520 And if the answer is okay, then everything switches back 0:20:05.560 --> 0:20:09.040 and you go back to normal operation. That's how it's 0:20:09.040 --> 0:20:13.800 supposed to work up. But AT and t's updated code 0:20:13.880 --> 0:20:18.000 sped things up so much it caused some real issues, 0:20:18.080 --> 0:20:21.439 and there was some poor timing, just coincidental timing that 0:20:21.480 --> 0:20:25.080 made things worse. So switch number one starts to get 0:20:25.080 --> 0:20:27.840 overwhelmed and sends a message over to switch number two, 0:20:27.920 --> 0:20:30.480 But switch number two was just in the middle of 0:20:30.520 --> 0:20:34.720 resetting itself, So switch number two goes into reset mode, 0:20:34.720 --> 0:20:37.000 which says do not disturb. Sends a message over to 0:20:37.000 --> 0:20:40.639 switch number three. That prompted switch number three to overload 0:20:40.920 --> 0:20:42.960 and put up a do not disturb sign. Move that 0:20:43.000 --> 0:20:46.000 down to switch number four. This whole thing goes down 0:20:46.040 --> 0:20:48.680 the entire line of one hundred and fourteen switches. They 0:20:48.720 --> 0:20:51.360 all end up getting overloaded as a result of this, 0:20:51.760 --> 0:20:54.920 and all go into reset mode and they get stuck there. 0:20:56.480 --> 0:21:00.520 That problem lasted for nine hours before be for AT 0:21:00.600 --> 0:21:03.240 and T was finally able to address the message load 0:21:03.359 --> 0:21:05.879 on the entire system and get the switches back to normal. 0:21:06.359 --> 0:21:09.520 The estimated cost of lost revenue for that time was 0:21:09.560 --> 0:21:13.840 about sixty million dollars in long distance calls, and there 0:21:13.840 --> 0:21:16.000 were a lot of angry customers to boot, so to 0:21:16.119 --> 0:21:19.919 placate them, AT and T offered reduced long distance rates 0:21:19.960 --> 0:21:25.040 on Valentine's Day. Pretty ugly, but AT and T tried 0:21:25.080 --> 0:21:27.080 to handle it, at least in a way that didn't 0:21:27.119 --> 0:21:30.400 turn it into a pr nightmare. Not so with Intel. 0:21:31.080 --> 0:21:33.880 That's what brings us to the Pendium problem. I don't 0:21:33.920 --> 0:21:36.760 know if you guys remember when pentium processors first came out, 0:21:36.760 --> 0:21:41.080 but they were a big deal. It was a redesign 0:21:41.160 --> 0:21:43.480 of the architecture of the microprocessor and it was meant 0:21:43.520 --> 0:21:47.159 to really speed things up. Well, Intel had a massive 0:21:47.240 --> 0:21:51.159 nightmare in nineteen ninety four thanks to a flaw in 0:21:51.240 --> 0:21:56.439 the entire first generation of Pentium processors. Now, when you 0:21:56.480 --> 0:21:59.920 break it all down, a CPU is all about performing math, 0:22:00.080 --> 0:22:02.760 medical operations on data, so it's kind of important that 0:22:02.760 --> 0:22:07.200 it does this correctly. Unfortunately, the flaw on the Pentium 0:22:07.280 --> 0:22:10.200 processors kind of messed that up, and the issue has 0:22:10.240 --> 0:22:13.600 to do with floating point operations. So the predecessor to 0:22:13.640 --> 0:22:16.560 the Pentium, the four eighty six, used a shift and 0:22:16.640 --> 0:22:21.040 subtract algorithm for floating point operations, which was effective but 0:22:21.560 --> 0:22:25.080 relatively slow compared to what Intel thought they could do 0:22:25.680 --> 0:22:32.280 by totally redesigning that structure and using a lookup table approach. Now, 0:22:32.320 --> 0:22:35.040 the table was supposed to have one thousand and sixty 0:22:35.160 --> 0:22:39.080 six entries programmed directly onto the logic array of the 0:22:39.080 --> 0:22:43.800 Pentium processor, but for some reason only one thousand and 0:22:43.920 --> 0:22:48.280 sixty one entries made it. Five entries went missing and 0:22:48.640 --> 0:22:52.520 essentially returned an answer of zero instead of what they 0:22:52.520 --> 0:22:56.359 were supposed to say, so if a calculation accessed one 0:22:56.359 --> 0:22:58.720 of those missing cells, it got zero, even though that's 0:22:58.760 --> 0:23:02.240 not the correct answer. All the first generation pentiums went 0:23:02.280 --> 0:23:04.600 out with this error because it was so minor that 0:23:04.680 --> 0:23:07.679 it wasn't even picked up by Intel's quality control at 0:23:07.680 --> 0:23:12.560 the time. Now, processes worked just fine up to the 0:23:12.680 --> 0:23:16.399 eighth decimal point. Beyond that things got messy, but for 0:23:16.480 --> 0:23:19.720 most folks that wasn't a problem because they weren't doing 0:23:20.000 --> 0:23:24.159 mathematical calculations that needed that level of precision. It just 0:23:24.400 --> 0:23:26.560 wasn't a thing. In fact, there was only a one 0:23:26.560 --> 0:23:30.000 in three hundred and sixty billion chance that this error 0:23:30.040 --> 0:23:34.000 would cause a big enough problem to reach up to 0:23:34.080 --> 0:23:38.000 the fourth decimal place. So most calculations that were simple 0:23:38.040 --> 0:23:41.840 were bulletproof. You were fine. But if you needed that precision, 0:23:42.440 --> 0:23:45.760 if you needed that really fine degree, that's when you 0:23:45.800 --> 0:23:50.400 would encounter the flaw. And that happened because they're math 0:23:50.480 --> 0:23:54.280 professors in this world, and one of those, Thomas Nicely 0:23:55.040 --> 0:23:58.000 discovered in October nineteen ninety four that he was getting 0:23:58.080 --> 0:24:01.280 errors because of this issue. He needed the processor to 0:24:01.320 --> 0:24:05.639 work correctly, and so he contacted Intel about the problem. 0:24:06.280 --> 0:24:09.480 And this is where we take a moment to acknowledge 0:24:09.480 --> 0:24:11.960 there's a right way and a wrong way to handle 0:24:12.680 --> 0:24:16.240 an issue. That's your fault. Intel decided to go the 0:24:16.320 --> 0:24:19.280 wrong way. My opinion is if you make a mistake, 0:24:19.320 --> 0:24:21.719 it's usually a good idea to just own up to 0:24:21.760 --> 0:24:25.000 it and try to make it better. But Intel's response 0:24:25.119 --> 0:24:26.840 was more along the lines of, yeah, we didn't think 0:24:26.840 --> 0:24:29.480