WEBVTT - Bad Computer Bugs 0:00:04.160 --> 0:00:07.160 Get in touch with technology with tech Stuff from how 0:00:07.240 --> 0:00:13.800 Stuff works dot com. Hey guys, welcome to tech Stuff. 0:00:13.880 --> 0:00:18.400 I'm your host Jonathan strickland Um with super producer Dylan 0:00:18.880 --> 0:00:22.239 in the teeny Tiny Stuff You Should Know Studio. Pretty soon, 0:00:22.280 --> 0:00:24.239 I'm going to demand it be called the Stuff You 0:00:24.280 --> 0:00:28.319 Should Know in Tech Stuff Studio, because who's gonna stop me. 0:00:29.000 --> 0:00:30.960 Today I wanted to take a look at some of 0:00:31.000 --> 0:00:35.760 the most spectacular computer bugs ever made, or at least 0:00:35.840 --> 0:00:37.839 some of the more notable ones, and I got the 0:00:38.000 --> 0:00:41.400 inspiration for this episode after researching a bug affecting the 0:00:41.440 --> 0:00:44.640 Spotify desktop app. More on that in just a second, 0:00:45.880 --> 0:00:50.480 but first I have to address a bit of apocryphal history, 0:00:50.800 --> 0:00:55.200 and regrettably it's a story that we've repeated on tech Stuff. 0:00:55.240 --> 0:00:59.840 So I'm sad to admit that I was complicit, although unknowingly, 0:01:00.600 --> 0:01:04.960 in the spread of misinformation and that all has to 0:01:05.000 --> 0:01:07.960 do with the origin of the term bug to describe 0:01:07.959 --> 0:01:12.280 a flaw in programming. So here's the popular story, the 0:01:12.319 --> 0:01:19.039 one that we have accidentally, uh promoted on tech Stuff 0:01:19.040 --> 0:01:22.280 without knowing that we were in the wrong. It goes 0:01:22.319 --> 0:01:25.560 that Grace Hopper, who was an early computer scientist who 0:01:25.640 --> 0:01:28.000 rose to the rank of Rear admiral in the U. S. 0:01:28.080 --> 0:01:32.759 Navy coined the phrase bug after discovering a moth coming 0:01:32.840 --> 0:01:38.200 up Harvard's Mark two calculator, a literal bug. Generally speaking, 0:01:38.240 --> 0:01:43.000 the story tends to be set in nineteen and there 0:01:43.080 --> 0:01:45.400 is even a note in the log book that reads 0:01:45.560 --> 0:01:49.480 first actual case of bug being found that's attributed to 0:01:49.520 --> 0:01:53.200 Grace Hopper. But there are several points that are wrong 0:01:53.320 --> 0:01:57.160 in this story. First, the year it didn't happen in ninet. 0:01:58.480 --> 0:02:02.000 It happened on September. N We know because there's a 0:02:02.080 --> 0:02:05.560 log book. At the log book that marks the incident 0:02:05.760 --> 0:02:08.720 not only has the notes, it actually has the moth 0:02:08.880 --> 0:02:13.880 taped into the book itself. It's taped onto the page. Second, 0:02:13.880 --> 0:02:17.280 Grace Hopper wasn't the person to discover the moth or 0:02:17.480 --> 0:02:21.720 make that log entry. She did tell the story about 0:02:21.720 --> 0:02:25.080 the moth several times, but it wasn't in the context 0:02:25.200 --> 0:02:28.919 of finding it or logging it. She just told the 0:02:28.960 --> 0:02:31.080 story that, yeah, we really did have a bug in 0:02:31.120 --> 0:02:34.960 the system. And most importantly, the word bug had already 0:02:35.000 --> 0:02:39.280 been used to describe design flaws for decades before the 0:02:39.280 --> 0:02:42.720 Mark two was even designed. In fact, if you look 0:02:42.760 --> 0:02:45.359 at the log book. This makes sense. It says first 0:02:45.400 --> 0:02:49.519 actual case of bug being found. That sentence doesn't make 0:02:49.680 --> 0:02:53.440 sense unless you've already used the word bug to describe 0:02:53.520 --> 0:02:57.920 a flaw, because you wouldn't say first actual case of 0:02:57.919 --> 0:03:00.880 bug being found. That the wording doesn't make any sense. 0:03:00.880 --> 0:03:05.400 The context makes no sense. Sadly, there are documented quotes 0:03:05.760 --> 0:03:09.040 dating back to the nineteenth century using the word bug 0:03:09.080 --> 0:03:12.239 to mean a design fault, and it could go back 0:03:12.240 --> 0:03:16.239 even further than that. So is with much regret that 0:03:16.280 --> 0:03:18.840 I admit I have unwittingly contributed to a bit of 0:03:18.880 --> 0:03:21.799 misleading folklore making the rounds. But I'm glad I can 0:03:21.800 --> 0:03:24.880 take this opportunity to address it. All Right, So let's 0:03:24.919 --> 0:03:29.799 talk about design bugs, and I'll be covering several goofs, mistakes, flubs, flaws, 0:03:29.840 --> 0:03:34.040 and outright catastrophes in this episode. But one thing I'm 0:03:34.080 --> 0:03:38.240 not necessarily going to cover our software vulnerabilities that were 0:03:38.320 --> 0:03:42.680 later exploited, either by opportunistic hackers or white hats who 0:03:42.680 --> 0:03:46.880 are just trying to improve system security. Those vulnerabilities are 0:03:46.960 --> 0:03:50.000 common in many types of software and arise not just 0:03:50.080 --> 0:03:55.240 through mistakes, but sometimes simple oversights, and I think it 0:03:55.320 --> 0:03:57.440 might be more fun to look at some real bugs, 0:03:57.440 --> 0:04:00.760 like stuff that made things go wrong, stuff that may 0:04:00.800 --> 0:04:04.800 have rendered a program defunct or otherwise caused headaches. Now 0:04:04.800 --> 0:04:06.840 I'm gonna make an exception to this. I'm going to 0:04:06.960 --> 0:04:10.560 start off with the ping of death, and I only 0:04:10.600 --> 0:04:14.240 mention it because it has an awesome name. Now, this 0:04:14.280 --> 0:04:18.000 flaw caused headaches back in ninety six. It was a 0:04:18.000 --> 0:04:23.160 flawed i P fragmentation reassembly code, and it became possible 0:04:23.160 --> 0:04:25.919 to crash lots of different types of computers using different 0:04:25.960 --> 0:04:30.000 operating systems, although Windows machines were particularly vulnerable, and this 0:04:30.080 --> 0:04:32.880 particular flaw would make a Windows machine revert to the 0:04:33.000 --> 0:04:36.400 dreaded blue screen of death. And it all happened by 0:04:36.400 --> 0:04:40.039 sending a special ping packet over the internet. So, for 0:04:40.160 --> 0:04:42.120 those of you who aren't familiar with what that is, 0:04:42.279 --> 0:04:45.800 a ping is essentially a simple message that checks for 0:04:45.880 --> 0:04:49.400 a connection between two computers. You send one ping from 0:04:49.400 --> 0:04:52.200 a computer to another one and look for a response, 0:04:52.480 --> 0:04:55.080 so that way you verify there is in fact a connection. 0:04:55.120 --> 0:04:57.840 You can also tell other things like how fast is 0:04:57.880 --> 0:05:01.400 that connection between those two computers. Now, in this case, 0:05:01.720 --> 0:05:04.960 you would have to actually design a malformed ping request 0:05:05.040 --> 0:05:07.840 and send that to a target, and it would bring 0:05:07.880 --> 0:05:12.000 that target down. That's the only security vulnerability story I 0:05:12.040 --> 0:05:15.040 really wanted to focus on. The others are all just 0:05:15.760 --> 0:05:18.640 design flaws. And let's begin with the bug that inspired 0:05:18.680 --> 0:05:20.320 me to do this episode in the first place, That 0:05:20.400 --> 0:05:24.080 Spotify bug I mentioned earlier. Ours Technica wrote a piece 0:05:24.120 --> 0:05:26.240 on it in November two thousand and sixteen, but the 0:05:26.240 --> 0:05:29.120 problem seems to date back at least as far as 0:05:29.240 --> 0:05:32.760 June two thousand sixteen, and that's when a few savvy 0:05:32.839 --> 0:05:37.440 Spotify users noticed some unusual activities on their computers. And 0:05:37.480 --> 0:05:39.240 it took a little bit of detective work, but they 0:05:39.240 --> 0:05:43.120 discovered that Spotify was apparently generating a huge amount of 0:05:43.200 --> 0:05:48.560 data on a daily basis, like gigabytes of data per day. 0:05:48.720 --> 0:05:51.960 And the culprit turned out to be a vacuum process 0:05:52.040 --> 0:05:56.440 for a database file containing the string mercury dot dB. Now, 0:05:56.480 --> 0:06:00.279 the vacuum process is the digital equivalent of vacuum c link. 0:06:00.320 --> 0:06:03.000 It's meant to repack data so that it takes up 0:06:03.120 --> 0:06:06.280 less space on a drive. Now, this involves building a 0:06:06.279 --> 0:06:09.719 new file to maximize efficiency, which is a good thing 0:06:09.839 --> 0:06:16.080 generally speaking. The problem was that Spotify's version was making 0:06:16.120 --> 0:06:18.440 it happen way too frequently, like on the order of 0:06:18.520 --> 0:06:22.640 once every few minutes, so that's not generally necessary. You 0:06:22.640 --> 0:06:25.440 don't need to rebuild a database file every few minutes 0:06:25.480 --> 0:06:28.440 to make sure it's the most efficient size it can be. 0:06:29.720 --> 0:06:33.000 So each rebuild represented a relatively small amount of data, 0:06:33.080 --> 0:06:36.039 but over time it added up, which meant that if 0:06:36.080 --> 0:06:38.880 you had Spotify on on your computer, even if it 0:06:38.960 --> 0:06:41.680 was just running in the background, it would be generating 0:06:41.720 --> 0:06:45.400 gigabytes worth of information rewriting this file over and over. 0:06:45.720 --> 0:06:48.479 Now it wasn't filling up a hard drive. It was 0:06:48.520 --> 0:06:53.920 just overwriting the same file. Now, if it had been 0:06:53.920 --> 0:06:56.320 filling up a hard drive, people would have noticed much earlier, 0:06:56.320 --> 0:06:59.360 and it wouldn't have just been savvy Spotify users, because 0:06:59.360 --> 0:07:01.800 you would suddenly notice, hey, I don't I can't save 0:07:01.800 --> 0:07:05.800 anything to my hard drive because everything is filling up. Instead. Again, 0:07:05.839 --> 0:07:08.480 it was just sort of writing and deleting and writing 0:07:08.480 --> 0:07:11.280 and deleting the same file over and over again. And 0:07:11.320 --> 0:07:13.240 that probably doesn't sound like a big deal, but it 0:07:13.440 --> 0:07:15.800 is a problem if you're using a solid state drive 0:07:16.040 --> 0:07:19.240 or s s D. So one of the drawbacks of 0:07:19.240 --> 0:07:23.600 an ss D is that over time it loses storage capacity, 0:07:23.720 --> 0:07:27.960 like you can store less data on an SSD over time. Now, 0:07:28.400 --> 0:07:30.960 by overtime I generally mean over a great deal of 0:07:31.040 --> 0:07:33.160 time and a lot of different data being written to 0:07:33.200 --> 0:07:37.360 it and overwritten. Uh. Generally speaking, most of us end 0:07:37.440 --> 0:07:40.320 up replacing our drives before we get to a point 0:07:40.320 --> 0:07:44.320 where the loss of capacity is a real issue. But 0:07:44.520 --> 0:07:46.560 similar in a way to how a battery can lose 0:07:46.600 --> 0:07:49.400 its ability to hold a full charge after you've gone 0:07:49.440 --> 0:07:53.560 through lots of charging and discharging cycles, you know how 0:07:53.600 --> 0:07:55.960 a battery won't be able to to hold as much 0:07:56.000 --> 0:07:58.720 even if it says it's up to a but a 0:07:58.760 --> 0:08:01.120 hundred percent doesn't last you as long as it used to. 0:08:01.680 --> 0:08:04.400 That's because its capacity to hold a full charge has 0:08:04.440 --> 0:08:08.560 decreased over time. But let's say you've got a program 0:08:08.640 --> 0:08:12.680 that's just constantly overwriting data to your drive, you might 0:08:12.720 --> 0:08:15.560 discover that your ss D s useful lifespan has been 0:08:15.640 --> 0:08:20.560 drastically reduced. So as I record this episode, Spotify has 0:08:20.600 --> 0:08:24.360 already rolled out an updated version of its desktop application, 0:08:24.400 --> 0:08:26.280 and that, by the way, is the only version of 0:08:26.320 --> 0:08:29.520 Spotify that was affected. If you use web based Spotify 0:08:29.800 --> 0:08:32.719 or mobile Spotify, you're in the clear already if you 0:08:32.800 --> 0:08:35.480 use a desktop version, as long as you have version 0:08:35.600 --> 0:08:40.160 one point zero point four two or later, you are fine. 0:08:41.280 --> 0:08:43.520 But if you did have that earlier version and you 0:08:43.640 --> 0:08:46.000 just had Spotify running on in the background, chances are 0:08:46.160 --> 0:08:50.160 it was writing to your hard drive like crazy. So 0:08:50.200 --> 0:08:53.400 what about some of the other big bugs in computer history. Well, 0:08:53.440 --> 0:08:56.120 some of the real doozies involve our attempts to explore 0:08:56.200 --> 0:08:59.839 the final frontier. So we'll be talking about space a 0:09:00.000 --> 0:09:02.200 few times in this episode, and we'll start with an 0:09:02.200 --> 0:09:06.640 early US satellite. So first up is a nineteen sixty 0:09:06.679 --> 0:09:10.400 two blunder involving the Mariner one. So some backstory on 0:09:10.440 --> 0:09:12.880 this one. Uh, We're gonna talk a lot about the 0:09:12.880 --> 0:09:16.040 Soviet Union in this episode two. It takes a couple 0:09:16.080 --> 0:09:18.880 of roles as we go on. But in this case, 0:09:18.960 --> 0:09:22.880 the then USSR had launched Sputnik into orbit in nineteen 0:09:22.920 --> 0:09:25.280 fifty seven, which really kicked off the space race and 0:09:25.360 --> 0:09:28.640 also was a big shot in the Cold War because 0:09:28.679 --> 0:09:31.640 of so Union was essentially saying, hey, we can launch 0:09:31.720 --> 0:09:34.320 this into space, we could also launch something at you. 0:09:35.520 --> 0:09:37.760 In response, the US and done sort of the same thing. 0:09:37.760 --> 0:09:40.840 They had launched some satellites into space and the Mariner 0:09:40.920 --> 0:09:43.880 one was going to be a big, big feather in 0:09:43.920 --> 0:09:45.599 the cap of the US. The whole idea was to 0:09:45.679 --> 0:09:48.440 launch a probe that would be a fly by probe 0:09:48.559 --> 0:09:53.640 and it would go by Venus. So uh NASA, which 0:09:53.679 --> 0:09:57.400 was newly formed in nineteen sixty two, was taking control 0:09:57.440 --> 0:09:59.720 of this and the budget for this particular project was 0:09:59.800 --> 0:10:02.600 eight team point five million dollars, which if you were 0:10:02.640 --> 0:10:06.440 to adjust for inflation, would be almost a hundred fifty 0:10:06.520 --> 0:10:10.520 million dollars today, So a hundred fifty million dollar project 0:10:10.559 --> 0:10:14.200 to launch the Mariner one and have it fly by Venus. 0:10:14.240 --> 0:10:16.760 But as I'm sure you guys have figured out by 0:10:16.760 --> 0:10:20.520 now based upon the topic of this podcast, not all 0:10:20.600 --> 0:10:24.760 went according to plan. Not long at all. After the 0:10:24.880 --> 0:10:28.800 rocket launched from the launch pad, it began to veer 0:10:28.880 --> 0:10:32.440 off course, and neither the computer controls on the rocket 0:10:32.559 --> 0:10:37.000 or manual controls back at HQ could correct for the problem. 0:10:37.080 --> 0:10:39.760 The rockets course was such that it was going to 0:10:39.840 --> 0:10:42.640 take it over shipping lanes, which meant there could be 0:10:42.640 --> 0:10:47.280 a potential catastrophe, and so Arrange safety officer made the 0:10:47.320 --> 0:10:50.280 difficult call and issued the command to blow the whole 0:10:50.360 --> 0:10:54.480 thing up, just shy of three hundred seconds after it launched. 0:10:55.000 --> 0:10:57.400 So what happened? What why did it go off course 0:10:57.400 --> 0:10:59.520 in the first place? Well, there was a flaw in 0:10:59.559 --> 0:11:02.880 the space crafts guidance software which diverted the rocket, and 0:11:02.960 --> 0:11:05.840 no amount of commands from ground control could correct for it. 0:11:06.280 --> 0:11:10.800 After a lengthy investigation, NASA discovered the era error was 0:11:11.080 --> 0:11:16.560 the result of a mistake transcribing handwritten notes into computer code. 0:11:17.360 --> 0:11:21.880 So someone just took some handwritten notes and misinterpreted one 0:11:21.880 --> 0:11:28.000 of them, and that one mistake was enough to crash 0:11:28.280 --> 0:11:33.680 the rocket, or to to necessitate it being destroyed. The 0:11:33.760 --> 0:11:38.400 great science fiction author Arthur C. Clark wrote that the 0:11:38.480 --> 0:11:43.800 Mariner one was wrecked by the most expensive hyphen in history, 0:11:43.960 --> 0:11:47.120 which isn't quite right, but it's pretty funny. I mean, 0:11:47.280 --> 0:11:51.480 come on, it's humorous phrase. So the actual punctuation mark 0:11:51.520 --> 0:11:54.840 that caused the problem was not technically a hyphen. It 0:11:54.920 --> 0:11:59.120 was a superscript bar. Superscript bars, by the way, not 0:11:59.200 --> 0:12:01.319 a place where player rights hang out to get tore up. 0:12:02.120 --> 0:12:05.440 A superscript bar just means it's a horizontal bar that 0:12:05.600 --> 0:12:08.360 is above some other symbol. In this case, it was 0:12:08.400 --> 0:12:12.920 a radius symbol, and that was a symbol along with 0:12:12.960 --> 0:12:17.280 the superscript bar to describe a smoothing function, which means 0:12:17.320 --> 0:12:20.920 the formula was meant to calculate smoothed values over the 0:12:20.960 --> 0:12:25.760 time derivative of a radius. Now, without the smoothing function, 0:12:26.320 --> 0:12:30.439 tiny deviations in course sent commands to the rockets thrusters 0:12:30.440 --> 0:12:34.000 to kick in big time to overcorrect for that problem. 0:12:34.040 --> 0:12:36.760 As an analogy, imagine you're driving a vehicle and you 0:12:36.800 --> 0:12:39.600 see a pothole in the road and you're approaching it, 0:12:39.720 --> 0:12:44.160 and instead of gently steering out of the way, you 0:12:44.400 --> 0:12:47.160 wrenched the wheel really hard to the left or to 0:12:47.200 --> 0:12:49.440 the right in order to try and get around this pothole. 0:12:49.880 --> 0:12:51.920 That's kind of what was happening with the rocket. It 0:12:51.960 --> 0:12:55.120 didn't have the smoothing function and so as a result, 0:12:55.200 --> 0:12:59.240 it was having these wild deviations and course. So it 0:12:59.360 --> 0:13:02.000 wasn't a high fin that caused the problem, but is 0:13:02.040 --> 0:13:05.800 close enough. Our next space story takes place in nineteen 0:13:06.400 --> 0:13:10.640 with the European Space Agencies RAN five Flight five O 0:13:10.880 --> 0:13:14.200 one rocket. Now, this rocket was to launch into space 0:13:14.240 --> 0:13:19.400 on June four nine, and instead the rocket disintegrated forty 0:13:19.559 --> 0:13:23.640 seconds after taking off. So what the heck happened? Well, 0:13:23.679 --> 0:13:25.640 it largely had to do with the e s A 0:13:26.080 --> 0:13:30.239 reusing old work. This actually becomes a theme in this episode. 0:13:30.640 --> 0:13:34.480 One of the morals of of this entire podcast is 0:13:34.600 --> 0:13:38.440 if you're designing something a successor to an earlier product, 0:13:39.720 --> 0:13:42.800 and you'd want to reuse some of the features that 0:13:42.880 --> 0:13:47.320 you created in your previous product, test the heck out 0:13:47.360 --> 0:13:50.760 of it in its new form factor, because it could 0:13:50.840 --> 0:13:53.760 be that things that worked perfectly fine in the earlier 0:13:53.800 --> 0:13:58.240 model will go awry in the new one. That's what 0:13:58.360 --> 0:14:01.760 happened here. So as you might guess from the name, 0:14:02.000 --> 0:14:06.360 the Aeryan five marked the fifth generation of launch vehicles 0:14:06.520 --> 0:14:11.160 under that name. The Arian four's inertial reference system would 0:14:11.160 --> 0:14:14.920 convert sixty four bit floating point numbers into a sixteen 0:14:14.960 --> 0:14:20.680 bits signed integer, and it worked just fine. But the 0:14:20.760 --> 0:14:26.400 Arian five stats were beefier than its predecessor with faster engines, 0:14:26.440 --> 0:14:29.640 and that was where the problem really started. The engine 0:14:29.640 --> 0:14:33.480 output meant those sixty four bit floating point numbers were 0:14:33.560 --> 0:14:37.520 significantly larger than the ones generated by the engines on 0:14:37.560 --> 0:14:41.680 the Arian four. They didn't anticipate this, so during the 0:14:41.680 --> 0:14:46.400 conversion process there was actually data overflow, and that overflow 0:14:46.520 --> 0:14:50.360 caused both the backup computer and the primary computer aboard 0:14:50.360 --> 0:14:53.440 the Arian five to crash, and they crashed in that order. 0:14:53.480 --> 0:14:57.040 The backup computer crashed first, followed by the primary computer 0:14:57.080 --> 0:14:59.960 a couple of seconds later. The whole thing took less 0:15:00.000 --> 0:15:04.120 than a minute to go from launch to disintegration. Oops, 0:15:05.600 --> 0:15:07.920 now we're gonna stick with space. But jumped forward to 0:15:10.240 --> 0:15:16.200 and the Mars Climate Orbiter. This was an unfortunate problem. 0:15:16.560 --> 0:15:19.960 So this particular spacecraft was meant to study Mars's climate, 0:15:20.080 --> 0:15:22.920 atmosphere and surface changes, and it was also supposed to 0:15:22.960 --> 0:15:26.200 be a kind of relay station for landers that would 0:15:26.240 --> 0:15:30.320 explore Mars's surface, but none of that would last because 0:15:30.440 --> 0:15:36.040 of some pretty significant goofs. So on September, the orbiter 0:15:36.160 --> 0:15:40.680 passed into the upper atmosphere of Mars and did so 0:15:40.880 --> 0:15:44.320 at a pretty low altitude. And this is what folks 0:15:44.320 --> 0:15:47.920 in the space industry called a bad thing. The Dragon, 0:15:48.000 --> 0:15:51.800 the spacecraft was significant. It began to fall apart and 0:15:51.960 --> 0:15:57.680 it was destroyed upon entering Mars's atmosphere. That's what happened. 0:15:58.840 --> 0:16:02.280 So the software guide the orbiter was to blame, and 0:16:03.160 --> 0:16:06.360 it's a dumb, dumb mistake. It was supposed to make 0:16:06.360 --> 0:16:10.600 adjustments to the orbiter's flight in SI units, specifically in 0:16:10.640 --> 0:16:15.560 Newton seconds. That's what the contract with Lockheed and NASA said, 0:16:16.280 --> 0:16:20.040 Newton seconds, use SI units for all of your all 0:16:20.080 --> 0:16:24.400 of your calculations, but the software instead made calculations in 0:16:24.520 --> 0:16:30.840 non SI units, namely pounds seconds. So Lockheeds software gave 0:16:30.920 --> 0:16:34.960 information to NASA's systems using the wrong units of measure. 0:16:36.040 --> 0:16:39.240 NASA systems then took that information, assuming it was with 0:16:39.360 --> 0:16:43.800 the right units of measure, and executed commands based upon that. 0:16:45.000 --> 0:16:47.760 Uh So, this is why if you're ever in a 0:16:47.840 --> 0:16:52.600 math course and the teacher makes you stop in the 0:16:52.640 --> 0:16:54.760 middle of writing a problem on the board and says, 0:16:54.800 --> 0:16:57.920 where are your units? This is why you have to 0:16:58.000 --> 0:17:01.520 make sure you're using the right units, because if you're 0:17:01.560 --> 0:17:04.800 saying a number and you don't associate a unit with it, 0:17:05.080 --> 0:17:08.040 someone could make an incorrect decision based on that, and 0:17:08.080 --> 0:17:10.639 it could be disastrous, as it was with the case 0:17:10.760 --> 0:17:14.360 of this orbiter. The thrusters fired at four point four 0:17:14.440 --> 0:17:17.960 or five times the power they were supposed to, and 0:17:18.000 --> 0:17:20.760 the orbiter didn't stand a chance. And this was a 0:17:20.760 --> 0:17:24.080 pretty expensive mistake. That mission's cost came in at three 0:17:24.800 --> 0:17:28.639 seven point six million dollars, but on the bright side 0:17:28.680 --> 0:17:31.359 with all of these stories, at least no human lives 0:17:31.440 --> 0:17:36.000 were ever in real danger as a result of the mistake. Now, 0:17:36.080 --> 0:17:39.840 I've got a lot more to say about bugs, but 0:17:40.040 --> 0:17:42.240 before I get into that, let's take a quick break 0:17:42.400 --> 0:17:53.000 to thank our sponsor. All right, Now, let's make a 0:17:53.040 --> 0:17:55.479 switch to A T and T, which is a company 0:17:55.480 --> 0:17:58.160 that had a pretty big problem with switches once upon 0:17:58.200 --> 0:18:00.920 a time. I'm talking about issue that popped up on 0:18:01.000 --> 0:18:04.560 January nine nine. That's when A T and T long 0:18:04.600 --> 0:18:07.200 distance customers discovered they were unable to make any long 0:18:07.240 --> 0:18:11.879 distance calls. Why why could they no longer reach anybody? Well, 0:18:12.440 --> 0:18:17.119 A T and T s long distance switches, which control 0:18:17.240 --> 0:18:20.639 that and allow for the actual connections to be made, 0:18:20.960 --> 0:18:23.840 were on the fritz. They were trying to reboot over 0:18:23.880 --> 0:18:27.560 and over again. They were just stuck in a reboot cycle. Now, 0:18:27.640 --> 0:18:31.520 initially the company thought it was being hacked, But like 0:18:31.560 --> 0:18:33.000 I said at the top of the show, I'm not 0:18:33.119 --> 0:18:36.680 covering stories about hackers here. I'm talking about big design 0:18:36.720 --> 0:18:41.680 flaws that caused problems. So they weren't getting hacked. That's 0:18:41.920 --> 0:18:44.320 not what was going on with those D fourteen long 0:18:44.320 --> 0:18:47.600 distance switches. No, there was a design problem at fault. 0:18:48.440 --> 0:18:49.959 So what had happened was a T and D had 0:18:50.040 --> 0:18:53.480 rolled out an update to the code that managed the switches, 0:18:53.880 --> 0:18:56.080 and it was meant to increase the efficiency. It was 0:18:56.119 --> 0:18:58.879 meant to speed things up, but the problem was it 0:18:59.040 --> 0:19:01.680 sped things up so much that the system got caught 0:19:01.800 --> 0:19:04.679 up in itself. It gets pretty technical, but I can 0:19:04.680 --> 0:19:08.439 give you kind of a overview of what the problem was. Alright, 0:19:08.480 --> 0:19:12.560 so each switch had a function that allowed it to 0:19:12.760 --> 0:19:16.800 alert the next switch down the line if things were 0:19:16.800 --> 0:19:20.560 starting to get hairy. So imagine that switch number one 0:19:20.960 --> 0:19:24.200 is handling traffic, but it's getting really close to capacity. 0:19:24.240 --> 0:19:26.320 So it sends a message over to switch number two 0:19:26.359 --> 0:19:29.680 and says, I can't take on any more work because 0:19:29.720 --> 0:19:33.200 if I do, I'll be overloaded. Switch to then says, 0:19:33.400 --> 0:19:36.679 no problem, I'll take on the any oncoming work for 0:19:36.720 --> 0:19:40.159 you and we'll handle it from there. And if switched 0:19:40.200 --> 0:19:43.000 number two order to get into the same source situation, 0:19:43.359 --> 0:19:45.240 it would say the same thing to switch number three, 0:19:45.320 --> 0:19:48.840 and so on and so forth. Now, eventually each switch 0:19:48.920 --> 0:19:51.679 will contact the one below it and say, hey, how 0:19:51.680 --> 0:19:54.440 are you doing there? And if the answer is okay, 0:19:54.520 --> 0:19:58.240 then everything switches back and you go back to normal operation. 0:19:59.200 --> 0:20:02.679 That's how it's suppo was to work up. But A 0:20:02.800 --> 0:20:05.560 T and t s updated code sped things up so 0:20:05.640 --> 0:20:09.360 much it caused some real issues, and there was some 0:20:09.560 --> 0:20:14.080 poor timing, just coincidental timing that made things worse. So 0:20:14.440 --> 0:20:17.159 switch number one starts to get overwhelmed and sends a 0:20:17.200 --> 0:20:19.679 message over to switch number two, but switched number two 0:20:19.920 --> 0:20:23.840 was just in the middle of resetting itself. So switch 0:20:23.920 --> 0:20:26.680 number two goes into reset mode, which says do not disturb. 0:20:26.760 --> 0:20:29.680 Sends a message over to switch number three. That prompted 0:20:29.720 --> 0:20:32.239 switch number three to overload and put up I do 0:20:32.280 --> 0:20:34.760 not disturb sign. Move that down to switch number four. 0:20:35.119 --> 0:20:39.120 This whole thing goes down the entire line of on switches. 0:20:39.200 --> 0:20:42.040 They all end up getting overloaded as a result of this, 0:20:42.440 --> 0:20:45.600 and I'll go into reset mode and they get stuck there. 0:20:47.160 --> 0:20:51.359 That problem lasted for nine hours before A T and 0:20:51.359 --> 0:20:54.080 T was finally able to address the message load on 0:20:54.119 --> 0:20:56.520 the entire system and get the switches back to normal. 0:20:57.040 --> 0:21:00.560 The estimated cost of lost revenue for that was about 0:21:00.560 --> 0:21:04.600 sixty million dollars in long distance calls, and there were 0:21:04.640 --> 0:21:07.560 a lot of angry customers to boot, so to placate them, 0:21:07.600 --> 0:21:10.719 A T and T offered reduced long distance rates on 0:21:10.840 --> 0:21:15.440 Valentine's Day pretty ugly. By the A T and T 0:21:15.440 --> 0:21:17.280 tried to handle it, at least in a way that 0:21:17.440 --> 0:21:21.040 didn't turn it into a pr nightmare. Not so with Intel. 0:21:21.760 --> 0:21:24.320 That's what it brings us to the Pendium problem. I 0:21:24.359 --> 0:21:27.040 don't know if you guys remember when Pentium processors first 0:21:27.080 --> 0:21:29.720 came out, but they were a big deal. It was 0:21:30.680 --> 0:21:33.800 a redesign of the architecture of the microprocessor and it 0:21:33.840 --> 0:21:37.239 was meant to really speed things up. Well, Intel had 0:21:37.280 --> 0:21:41.840 a massive nightmare in nine thanks to a flaw in 0:21:41.880 --> 0:21:47.080 the entire first generation of Pentium processors. Now, when you 0:21:47.119 --> 0:21:50.400 break it all down, a CPU is all about performing 0:21:50.440 --> 0:21:53.399 mathematical operations on data, so it's kind of important that 0:21:53.440 --> 0:21:57.920 it does this correctly. Unfortunately, the flaw and the Pentium 0:21:57.920 --> 0:22:00.840 processors kind of messed that up. And the issue has 0:22:00.880 --> 0:22:04.280 to do with floating point operations. So the predecessor to 0:22:04.280 --> 0:22:07.879 the Pentium, the four six, used a shift and subtract 0:22:07.960 --> 0:22:12.800 algorithm for floating point operations, which was effective but relatively 0:22:12.880 --> 0:22:17.000 slow compared to what Intel thought they could do by 0:22:17.280 --> 0:22:22.960 totally redesigning that structure and using a look up table approach. Now, 0:22:22.960 --> 0:22:26.159 the table was supposed to have one thousand sixty six 0:22:26.440 --> 0:22:30.800 entries programmed directly onto the logic array of the Pentium processor, 0:22:31.920 --> 0:22:36.320 but for some reason only one thousand sixty one entries 0:22:36.400 --> 0:22:40.800 made it. Five entries went missing and essentially returned an 0:22:40.800 --> 0:22:45.400 answer of zero instead of what they were supposed to say, 0:22:45.480 --> 0:22:47.919 So if a calculation accessed one of those missing cells, 0:22:47.960 --> 0:22:50.280 it got zero, even though that's not the correct answer. 0:22:51.200 --> 0:22:53.720 All the first generation pentiums went out with this error 0:22:53.800 --> 0:22:56.200 because it was so minor that it wasn't even picked 0:22:56.320 --> 0:23:01.680 up by Intel's quality control at the time. Now, processes 0:23:01.680 --> 0:23:05.159 work just fine up to the eighth decimal point. Beyond 0:23:05.240 --> 0:23:08.360 that things got messy. But for most folks that wasn't 0:23:08.359 --> 0:23:12.959 a problem because they weren't doing mathematical calculations that needed 0:23:12.960 --> 0:23:16.400 that level of precision. It just wasn't a thing. In fact, 0:23:16.440 --> 0:23:19.399 there was only a one in three sixty billion chance 0:23:19.880 --> 0:23:23.320 that this error would cause a a big enough problem 0:23:23.560 --> 0:23:27.480 to reach up to the fourth decimal place. So most 0:23:27.480 --> 0:23:31.000 calculations that were simple were bulletproof. You you're fine, But 0:23:31.080 --> 0:23:34.439 if you needed that precision, if you needed that really 0:23:34.560 --> 0:23:38.399 fine degree, that's when you would encounter the flaw, and 0:23:38.560 --> 0:23:42.439 that happened because there are math professors in this world, 0:23:42.920 --> 0:23:48.040 and one of those, Thomas nicely discovered in October that 0:23:48.119 --> 0:23:51.200 he was getting errors because of this issue. He needed 0:23:51.240 --> 0:23:54.840 the processor to work correctly, and so he contacted Intel 0:23:54.880 --> 0:23:58.800 about the problem. And this is where we take a 0:23:58.880 --> 0:24:01.639 moment to acknowledge there's a right way and a wrong 0:24:01.680 --> 0:24:06.479 way to handle an issue. That's your fault until decided 0:24:06.520 --> 0:24:09.199 to go the wrong way. My opinion is, if you 0:24:09.240 --> 0:24:11.520 make a mistake, it's usually a good idea to just 0:24:11.720 --> 0:24:13.920 own up to it and try to make it better. 0:24:14.600 --> 0:24:16.960 But Intel's response was more along the lines of yeah, 0:24:17.040 --> 0:24:19.200 we didn't think it was a big deal. And then 0:24:19.240 --> 0:24:23.800 Intel made other pr blenders. But because people began to hear, hey, 0:24:23.880 --> 0:24:27.119 that Pentium processor in your computer that you just bought, 0:24:27.880 --> 0:24:31.480 it doesn't work properly. So people wanted to get replacements. 0:24:32.160 --> 0:24:34.520