WEBVTT - Rerun: Bad Computer Bugs 0:00:04.240 --> 0:00:07.240 Welcome to tech Stuff, a production of I Heart Radios 0:00:07.320 --> 0:00:13.960 How Stuff Works. Hey there, and welcome to tech Stuff. 0:00:14.000 --> 0:00:16.360 I'm your host, Jonathan Strickland. I'm an executive producer with 0:00:16.440 --> 0:00:18.920 iHeart Radio and I love all things tech and I'm 0:00:18.920 --> 0:00:22.680 probably still running all over the United States right now 0:00:23.000 --> 0:00:26.040 on special projects. And for that reason, we are going 0:00:26.079 --> 0:00:30.600 to enjoy another classic episode of tech Stuff. This episode 0:00:30.840 --> 0:00:34.440 is called bad Computer Bugs, and it's all about software 0:00:34.479 --> 0:00:37.879 bugs that were some of the worst to ever go 0:00:37.960 --> 0:00:41.960 out with shipped projects. So let's go back and listen 0:00:41.960 --> 0:00:45.200 to this classic episode. I have to address a bit 0:00:45.240 --> 0:00:50.640 of apocryphal history, and regrettably it's a story that we've 0:00:50.680 --> 0:00:53.960 repeated on tech Stuff. So I'm sad to admit that 0:00:54.080 --> 0:00:59.400 I was complicit, although unknowingly, in the spread of misinformation 0:01:00.960 --> 0:01:02.920 and that all has to do with the origin of 0:01:02.960 --> 0:01:07.399 the term bug to describe a flaw in programming. So 0:01:07.440 --> 0:01:11.960 here's the popular story, the one that we have accidentally, 0:01:13.800 --> 0:01:17.080 uh promoted on tech Stuff without knowing that we were 0:01:17.319 --> 0:01:20.760 in the wrong. It goes that Grace Hopper, who was 0:01:20.800 --> 0:01:23.399 an early computer scientist who rose to the rank of 0:01:23.480 --> 0:01:26.920 rear admiral in the U. S. Navy coined the phrase 0:01:27.040 --> 0:01:31.800 bug after discovering a moth coming up Harvard's Market two calculator, 0:01:31.959 --> 0:01:36.320 a literal bug. Generally speaking, the story tends to be 0:01:36.440 --> 0:01:40.920 set in nineteen, and there is even a note in 0:01:40.959 --> 0:01:44.240 the log book that reads first actual case of bug 0:01:44.319 --> 0:01:48.560 being found that's attributed to Grace Hopper. But there are 0:01:48.560 --> 0:01:52.640 several points that are wrong in this story. First, the 0:01:52.720 --> 0:01:56.280 year it didn't happen in nineteen. It happened on September. 0:01:57.680 --> 0:02:01.520 We know because there's a log book. The log book 0:02:01.560 --> 0:02:04.320 that marks the incident not only has the notes, it 0:02:04.400 --> 0:02:08.359 actually has the moth taped into the book itself. It's 0:02:08.400 --> 0:02:12.560 taped onto the page. Second, Grace Hopper wasn't the person 0:02:12.680 --> 0:02:16.520 to discover the moth or make that log entry. She 0:02:16.639 --> 0:02:20.600 did tell the story about the moth several times, but 0:02:20.680 --> 0:02:24.920 it wasn't in the context of finding it or logging it. 0:02:25.080 --> 0:02:27.160 She just told the story that, yeah, we really did 0:02:27.200 --> 0:02:30.680 have a bug in the system. And most importantly, the 0:02:30.680 --> 0:02:34.360 word bug had already been used to describe design flaws 0:02:34.440 --> 0:02:39.160 for decades before the Mark two was even designed. In fact, 0:02:39.160 --> 0:02:41.280 if you look at the log book, this makes sense. 0:02:41.320 --> 0:02:45.000 It says first actual case of bug being found that 0:02:45.120 --> 0:02:48.959 sentence doesn't make sense unless you've already used the word 0:02:49.040 --> 0:02:54.040 bug to describe a flaw, because you wouldn't say, first 0:02:54.040 --> 0:02:57.239 actual case of bug being found that the wording doesn't 0:02:57.240 --> 0:03:01.040 make a sense, the context makes no sense. Sadly, there 0:03:01.040 --> 0:03:05.320 are documented quotes dating back to the nineteenth century using 0:03:05.320 --> 0:03:08.560 the word bug to mean a design fault, and it 0:03:08.560 --> 0:03:12.440 could go back even further than that. So is with 0:03:12.520 --> 0:03:15.240 much regret that I admit I have unwillingly contributed to 0:03:15.320 --> 0:03:18.200 a bit of misleading folklore making the rounds. But I'm 0:03:18.240 --> 0:03:21.400 glad I can take this opportunity to address it. All Right, 0:03:21.480 --> 0:03:24.040 so let's talk about design bugs, and I'll be covering 0:03:24.080 --> 0:03:29.440 several goofs mistakes, flubs, flaws, and outright catastrophes in this episode. 0:03:30.080 --> 0:03:33.400 But one thing I'm not necessarily going to cover our 0:03:33.400 --> 0:03:38.200 software vulnerabilities that were later exploited, either by opportunistic hackers 0:03:38.360 --> 0:03:42.280 or white hats who are just trying to improve system security. 0:03:42.520 --> 0:03:46.040 Those vulnerabilities are common in many types of software and 0:03:46.080 --> 0:03:51.600 arise not just through mistakes but sometimes simple oversights, and 0:03:51.680 --> 0:03:53.320 I think it might be more fun to look at 0:03:53.360 --> 0:03:56.920 some real bugs, like stuff that made things go wrong, 0:03:57.040 --> 0:03:59.800 stuff that may have rendered a program defunct or otherwise 0:03:59.800 --> 0:04:03.160 cau headaches. Now I'm gonna make an exception to this. 0:04:03.400 --> 0:04:06.000 I'm going to start off with the Ping of death, 0:04:06.840 --> 0:04:10.960 and I only mention it because it has an awesome name. Now, 0:04:10.960 --> 0:04:14.760 this flaw caused headaches back in ninety six. It was 0:04:14.800 --> 0:04:19.560 a flawed i P fragmentation reassembly code, and it became 0:04:19.600 --> 0:04:22.480 possible to crash lots of different types of computers using 0:04:22.520 --> 0:04:26.719 different operating systems, although Windows machines were particularly vulnerable, and 0:04:26.760 --> 0:04:29.640 this particular flaw would make a Windows machine revert to 0:04:29.680 --> 0:04:33.159 the dreaded blue screen of death. And it all happened 0:04:33.160 --> 0:04:36.719 by sending a special ping packet over the Internet. So, 0:04:36.839 --> 0:04:39.000 for those of you who aren't familiar with what that is, 0:04:39.160 --> 0:04:42.520 a ping is essentially a simple message that checks for 0:04:42.760 --> 0:04:46.280 a connection between two computers. You send one ping from 0:04:46.279 --> 0:04:49.080 a computer to another one and look for a response, 0:04:49.360 --> 0:04:51.960 so that way you verify there is in fact a connection. 0:04:52.000 --> 0:04:54.720 You can also tell other things like how fast is 0:04:54.760 --> 0:04:58.279 that connection between those two computers. Now, in this case, 0:04:58.600 --> 0:05:01.800 you would have to actually design a malformed ping request 0:05:01.920 --> 0:05:04.720 and send that to a target and it would bring 0:05:04.760 --> 0:05:08.880 that target down. That's the only security vulnerability story I 0:05:08.920 --> 0:05:11.920 really wanted to focus on. The others are all just 0:05:12.640 --> 0:05:15.520 design flaws. And let's begin with the bug that inspired 0:05:15.560 --> 0:05:17.200 me to do this episode in the first place. That 0:05:17.279 --> 0:05:20.960 Spotify bug I mentioned earlier. Ours Technica wrote a piece 0:05:21.000 --> 0:05:23.120 on it in November two thousand and sixteen, but the 0:05:23.120 --> 0:05:26.000 problem seems to date back at least as far as 0:05:26.120 --> 0:05:29.640 June two thousand sixteen, and that's when a few savvy 0:05:29.720 --> 0:05:34.320 Spotify users noticed some unusual activities on their computers. And 0:05:34.360 --> 0:05:36.080 it took a little bit of detective work, but they 0:05:36.120 --> 0:05:40.000 discovered that Spotify was apparently generating a huge amount of 0:05:40.080 --> 0:05:45.400 data on a daily basis, like gigabytes of data per day. 0:05:45.600 --> 0:05:48.840 And the culprit turned out to be a vacuum process 0:05:48.920 --> 0:05:53.320 for a database file containing the string mercury dot d b. Now, 0:05:53.360 --> 0:05:57.080 the vacuum process is the digital equivalent of vacuum ceiling. 0:05:57.200 --> 0:05:59.880 It's meant to repack data so that it takes up 0:06:00.040 --> 0:06:03.160 a space on a drive. Now, this involves building a 0:06:03.160 --> 0:06:06.599 new file to maximize efficiency, which is a good thing 0:06:06.720 --> 0:06:12.960 generally speaking. The problem was that Spotify's version was making 0:06:13.000 --> 0:06:15.320 it happen way too frequently, like on the order of 0:06:15.400 --> 0:06:19.520 once every few minutes, so that's not generally necessary. You 0:06:19.520 --> 0:06:22.320 don't need to rebuild a database file every few minutes 0:06:22.360 --> 0:06:25.320 to make sure it's the most efficient size it can be. 0:06:26.600 --> 0:06:29.840 So each rebuild represented a relatively small amount of data, 0:06:29.960 --> 0:06:32.880 but over time it added up, which meant that if 0:06:32.960 --> 0:06:35.760 you had Spotify on on your computer, even if it 0:06:35.800 --> 0:06:38.560 was just running in the background, it would be generating 0:06:38.600 --> 0:06:42.320 gigabytes worth of information rewriting this file over and over. 0:06:42.600 --> 0:06:45.359 Now it wasn't filling up a hard drive. It was 0:06:45.400 --> 0:06:50.800 just overriding the same file. Now, if it had been 0:06:50.800 --> 0:06:53.200 filling up a hard drive, people would have noticed much earlier, 0:06:53.200 --> 0:06:56.279 and it wouldn't have just been savvy Spotify users, because 0:06:56.279 --> 0:06:58.640 you would suddenly notice, hey, I don't I can't save 0:06:58.680 --> 0:07:02.680 anything to my hard drive because things filling up. Instead, Again, 0:07:02.720 --> 0:07:05.320 it was just sort of writing and deleting and writing 0:07:05.360 --> 0:07:08.159 and deleting the same file over and over again. And 0:07:08.200 --> 0:07:10.120 that probably doesn't sound like a big deal, but it 0:07:10.320 --> 0:07:12.680 is a problem if you're using a solid state drive 0:07:12.920 --> 0:07:16.120 or s s D. So one of the drawbacks of 0:07:16.120 --> 0:07:20.480 an ss D is that over time it loses storage capacity. 0:07:20.600 --> 0:07:23.560 Like you can store less data on an ss D 0:07:23.880 --> 0:07:27.280 over time. Now by overtime, I generally mean over a 0:07:27.320 --> 0:07:29.400 great deal of time and a lot of different data 0:07:29.440 --> 0:07:33.760 being written to it and overwritten. Uh. Generally speaking, most 0:07:33.760 --> 0:07:36.680 of us end up replacing our drives before we get 0:07:36.720 --> 0:07:39.520 to a point where the loss of capacity is a 0:07:39.560 --> 0:07:42.600 real issue. But similar in a way to how a 0:07:42.600 --> 0:07:45.120 battery can lose its ability to hold a full charge 0:07:45.560 --> 0:07:49.520 after you've gone through lots of charging and discharging cycles, 0:07:49.960 --> 0:07:52.160 you know how a battery won't be able to to 0:07:52.240 --> 0:07:54.360 hold as much even if it says it's up to 0:07:54.400 --> 0:07:57.120 a percent, but a hundred percent doesn't last you as 0:07:57.160 --> 0:08:00.160 long as it used to. That's because it's capacity need 0:08:00.200 --> 0:08:04.040 to hold a full charge has decreased over time. But 0:08:04.160 --> 0:08:07.160 let's say you've got a program that's just constantly overwriting 0:08:07.240 --> 0:08:10.720 data to your drive, you might discover that your ss 0:08:10.800 --> 0:08:15.120 D S useful lifespan has been drastically reduced. So as 0:08:15.160 --> 0:08:18.760 I record this episode, Spotify has already rolled out an 0:08:18.840 --> 0:08:21.960 updated version of its desktop application, and that, by the way, 0:08:22.040 --> 0:08:24.520 is the only version of Spotify that was affected. If 0:08:24.560 --> 0:08:28.280 you use web based Spotify or mobile Spotify, you're in 0:08:28.320 --> 0:08:30.920 the clear already. If you use a desktop version, as 0:08:30.920 --> 0:08:34.000 long as you have version one point zero point four 0:08:34.040 --> 0:08:39.040 two or later, you are fine. But if you did 0:08:39.080 --> 0:08:41.600 have that earlier version and you just had Spotify running 0:08:41.600 --> 0:08:44.400 on in the background, chances are it was writing to 0:08:44.440 --> 0:08:47.760 your hard drive like crazy. So what about some of 0:08:47.760 --> 0:08:50.520 the other big bugs in computer history? Well, some of 0:08:50.520 --> 0:08:54.239 the real doozies involve our attempts to explore the final frontier. 0:08:55.040 --> 0:08:57.400 So we'll be talking about space a few times in 0:08:57.400 --> 0:09:00.920 this episode, and we'll start with an early US satellite. 0:09:01.480 --> 0:09:04.679 So first up is a nineteen sixty two blunder involving 0:09:04.679 --> 0:09:08.560 the Mariner one. So some backstory on this one. Uh, 0:09:08.640 --> 0:09:10.600 We're gonna talk a lot about the Soviet Union in 0:09:10.600 --> 0:09:13.800 this episode too. It takes a couple of roles as 0:09:13.800 --> 0:09:17.160 we go on. But in this case, the then USSR 0:09:17.360 --> 0:09:20.640 had launched Sputnik into orbit in nineteen fifty seven, which 0:09:20.640 --> 0:09:23.480 really kicked off the space race and also was a 0:09:23.520 --> 0:09:26.280 big shot in the Cold War because of so Union 0:09:26.320 --> 0:09:29.280 was essentially saying, hey, we can launch this into space, 0:09:29.360 --> 0:09:33.079 we could also launch something at you in response to 0:09:33.080 --> 0:09:34.760 the US and done sort of the same thing. They 0:09:34.760 --> 0:09:37.959 had launched some satellites into space, and the Mariner one 0:09:38.080 --> 0:09:40.880 was going to be a big, big, feather in the 0:09:40.920 --> 0:09:42.839 cap of the US. The whole idea was to launch 0:09:42.840 --> 0:09:45.520 a probe that would be a fly by probe and 0:09:45.600 --> 0:09:50.640 it would go by Venus. So uh NASA, which was 0:09:50.679 --> 0:09:54.400 newly formed in nineteen sixty two, was taking control of 0:09:54.440 --> 0:09:57.120 this and the budget for this particular project was eighteen 0:09:57.160 --> 0:10:00.000 point five million dollars, which if you were to adjust 0:10:00.040 --> 0:10:05.280 for inflation, would be almost a hundred fifty million dollars today, 0:10:05.480 --> 0:10:08.120 So a hundred fifty million dollar project to launch the 0:10:08.160 --> 0:10:11.640 Mariner one and have it fly by Venus. But as 0:10:11.679 --> 0:10:14.439 I'm sure you guys have figured out by now based 0:10:14.520 --> 0:10:18.160 upon the topic of this podcast, not all went according 0:10:18.200 --> 0:10:23.079 to plan. Not long at all. After the rocket launched 0:10:23.559 --> 0:10:27.280 from the launchpad, it began to veer off course, and 0:10:27.360 --> 0:10:30.360 neither the computer controls on the rocket or manual controls 0:10:30.360 --> 0:10:34.520 back at HQ could correct for the problem. The rockets 0:10:34.520 --> 0:10:37.040 course was such that it was going to take it 0:10:37.120 --> 0:10:40.880 over shipping lanes, which meant there could be a potential catastrophe, 0:10:41.320 --> 0:10:45.200 and so Arrange safety officer made the difficult call and 0:10:45.280 --> 0:10:48.360 issued the command to blow the whole thing up just 0:10:48.559 --> 0:10:52.640 shy of three hundred seconds after it launched. So what happened. 0:10:52.679 --> 0:10:55.520 What why did it go off course in the first place, Well, 0:10:55.520 --> 0:10:58.560 there was a flaw in the spacecraft's guidance software which 0:10:58.600 --> 0:11:01.360 diverted the rocket, and no amount of commands from ground 0:11:01.360 --> 0:11:06.080 control could correct for it. After a lengthy investigation, NASA 0:11:06.120 --> 0:11:09.320 discovered the era error was the result of a mistake 0:11:09.760 --> 0:11:15.840 transcribing handwritten notes into computer code. So someone just took 0:11:15.920 --> 0:11:20.199 some handwritten notes and misinterpreted one of them, and that 0:11:20.600 --> 0:11:26.080 one mistake was enough to crash the rocket or to 0:11:26.080 --> 0:11:32.520 to necessitate it being destroyed. The great science fiction author 0:11:32.760 --> 0:11:36.960 Arthur C. Clark wrote that the Mariner one was wrecked 0:11:37.080 --> 0:11:41.800 by the most expensive hyphen in history, which isn't quite right, 0:11:41.840 --> 0:11:46.520 but it's pretty funny. I mean, come on, it's humorous phrase. 0:11:46.960 --> 0:11:49.600 So the actual punctuation mark that caused the problem was 0:11:49.679 --> 0:11:55.120 not technically a hyphen. It was a superscript bar. Superscript 0:11:55.120 --> 0:11:57.120 of bars, by the way, not a place where playwrights 0:11:57.120 --> 0:12:00.360 hang out to get tore up. A superscript barge means 0:12:00.400 --> 0:12:04.080 it's a horizontal bar that is above some other symbol. 0:12:04.120 --> 0:12:07.920 In this case, it was a radius symbol, and that 0:12:08.040 --> 0:12:11.679 was a symbol along with the superscript bar to describe 0:12:11.720 --> 0:12:15.320 a smoothing function, which means the formula was meant to 0:12:15.360 --> 0:12:21.240 calculate smoothed values over the time derivative of a radius. Now, 0:12:21.240 --> 0:12:25.760 without the smoothing function, tiny deviations in course sent commands 0:12:25.800 --> 0:12:28.480 to the rockets thrusters to kick in big time to 0:12:28.760 --> 0:12:32.560 overcorrect for that problem. As an analogy, imagine you're driving 0:12:32.600 --> 0:12:35.440 a vehicle and you see a pothole in the road 0:12:35.520 --> 0:12:39.360 and you're approaching it, and instead of gently steering out 0:12:39.440 --> 0:12:43.360 of the way, you wrenched the wheel really hard to 0:12:43.400 --> 0:12:45.199 the left or to the right in order to try 0:12:45.200 --> 0:12:47.560 and get around this pothole. That's kind of what was 0:12:47.559 --> 0:12:50.360 happening with the rocket. It didn't have the smoothing function, 0:12:50.840 --> 0:12:53.320 and so as a result, it was having these wild 0:12:53.360 --> 0:12:57.760 deviations and course. So it wasn't a hyphen that caused 0:12:57.760 --> 0:13:01.640 the problem, but is close enough. Our next space story 0:13:01.720 --> 0:13:05.720 takes place in nine with the European Space Agencies ARIANE 0:13:06.000 --> 0:13:10.120 five Flight five O one rocket. Now, this rocket was 0:13:10.160 --> 0:13:14.040 to launch into space on June four nine and instead 0:13:14.440 --> 0:13:18.439 the rocket disintegrated forty seconds after taking off. So what 0:13:18.520 --> 0:13:21.720 the heck happened? Well, it largely had to do with 0:13:21.800 --> 0:13:25.920 the E s A reusing old work. This actually becomes 0:13:25.920 --> 0:13:28.680 a theme in this episode. One of the morals of 0:13:28.920 --> 0:13:33.160 of this entire podcast is if you're designing something a 0:13:33.240 --> 0:13:38.000 successor to an earlier product, and you'd want to reuse 0:13:38.320 --> 0:13:41.840 some of the features that you created in your previous product, 0:13:42.960 --> 0:13:46.120 test the heck out of it in its new form factor, 0:13:46.920 --> 0:13:49.760 because it could be that things that worked perfectly fine 0:13:49.800 --> 0:13:53.920 in the earlier model will go awry in the new one. 0:13:54.760 --> 0:13:58.280 That's what happened here, So as you might guess from 0:13:58.320 --> 0:14:02.280 the name, the ARIANE five marked the fifth generation of 0:14:02.400 --> 0:14:06.880 launch vehicles under that name. The Arian four's inertial reference 0:14:06.920 --> 0:14:10.960 system would convert sixty four bit floating point numbers into 0:14:11.040 --> 0:14:14.959 a sixteen bits signed integer, and it worked just fine. 0:14:16.240 --> 0:14:21.800 But the Arian five stats were beefier than its predecessor 0:14:22.120 --> 0:14:25.920 with faster engines, and that was where the problem really started. 0:14:26.000 --> 0:14:29.320 The engine output meant those sixty four bit floating point 0:14:29.360 --> 0:14:33.760 numbers were significantly larger than the ones generated by the 0:14:33.800 --> 0:14:38.160 engines on the Arian four. They didn't anticipate this, so 0:14:38.240 --> 0:14:42.360 during the conversion process there was actually data overflow, and 0:14:42.440 --> 0:14:46.320 that overflow caused both the backup computer and the primary 0:14:46.320 --> 0:14:49.160 computer aboard the are N five to crash, and they 0:14:49.200 --> 0:14:52.920 crashed in that order. The backup computer crashed first, followed 0:14:52.920 --> 0:14:55.840 by the primary computer a couple of seconds later. The 0:14:55.920 --> 0:14:58.040 whole thing took less than a minute to go from 0:14:58.120 --> 0:15:03.960 launch to disintegration. Oops, now we're gonna stick with space. 0:15:04.000 --> 0:15:10.160 But jumped forward to and the Mars Climate Orbiter. This 0:15:10.320 --> 0:15:15.320 was an unfortunate problem. So this particular spacecraft was meant 0:15:15.320 --> 0:15:18.880 to study Mars's climate, atmosphere and surface changes, and it 0:15:18.960 --> 0:15:21.360 was also supposed to be a kind of relay station 0:15:21.440 --> 0:15:25.600 for landers that would explore Mars's surface, but none of 0:15:25.640 --> 0:15:29.360 that would last because of some pretty significant goofs. So 0:15:29.400 --> 0:15:34.760 on September, the orbiter passed into the upper atmosphere of 0:15:34.800 --> 0:15:40.200 Mars and did so at a pretty low altitude. And 0:15:40.280 --> 0:15:42.600 this is what folks in the space industry called a 0:15:42.720 --> 0:15:47.120 bad thing. The drag on the spacecraft was significant. It 0:15:47.200 --> 0:15:51.840 began to fall apart and it was destroyed upon entering 0:15:51.880 --> 0:15:57.000 Mars's atmosphere. That's what happened. So the software guiding the 0:15:57.120 --> 0:16:02.160 orbiter was to blame, and it's a dumb, dumb mistake. 0:16:02.400 --> 0:16:04.840 It was supposed to make adjustments to the orbiter's flight 0:16:05.400 --> 0:16:10.000 in SI units specifically in Newton seconds. That's what the 0:16:10.040 --> 0:16:14.880 contract with Lockheed and NASA said, Newton seconds use SI 0:16:15.000 --> 0:16:18.840 units for all of your all of your calculations, but 0:16:18.920 --> 0:16:23.200 the software instead made calculations in non SI units, namely 0:16:23.520 --> 0:16:29.600 pounds seconds. So Lockheeds software gave information to NASA's systems 0:16:29.840 --> 0:16:34.160 using the wrong units of measure. NASA systems then took 0:16:34.160 --> 0:16:37.720 the information, assuming it was with the right units of measure, 0:16:38.160 --> 0:16:43.440 and executed commands based upon that. Uh So, this is 0:16:43.480 --> 0:16:47.480 why if you're ever in a math course and the 0:16:47.480 --> 0:16:50.360 teacher makes you stop in the middle of writing a 0:16:50.440 --> 0:16:52.640 problem on the board and says, where are your units? 0:16:53.320 --> 0:16:55.960 This is why you have to make sure you're using 0:16:56.000 --> 0:16:59.680 the right units, because if you're saying a number and 0:16:59.760 --> 0:17:02.760 you don't associate a unit with it, someone could make 0:17:02.800 --> 0:17:06.160 an incorrect decision based on that and it could be disastrous, 0:17:06.280 --> 0:17:09.080 as it was with the case of this orbiter. The 0:17:09.119 --> 0:17:12.560 thrusters fired at four point four or five times the 0:17:12.640 --> 0:17:15.880 power they were supposed to, and the orbiter didn't stand 0:17:15.880 --> 0:17:19.399 a chance. And this was a pretty expensive mistake. That 0:17:19.520 --> 0:17:23.840 mission's cost came in at three seven point six million dollars. 0:17:24.520 --> 0:17:26.520 But on the bright side, with all of these stories, 0:17:26.560 --> 0:17:29.720 at least no human lives were ever in real danger 0:17:29.800 --> 0:17:33.399 as a result of the mistake. Now I've got a 0:17:33.400 --> 0:17:37.560 lot more to say about bugs, but before I get 0:17:37.560 --> 0:17:40.400 into that, let's take a quick break to thank our sponsor. 0:17:48.880 --> 0:17:50.720 All right, Now, let's make a switch to A T 0:17:50.880 --> 0:17:53.040 and T, which is a company that had a pretty 0:17:53.119 --> 0:17:56.560 big problem with switches once upon a time. I'm talking 0:17:56.600 --> 0:18:00.520 about an issue that popped up on January. That's when 0:18:00.560 --> 0:18:03.000 A T and T long distance customers discovered they were 0:18:03.040 --> 0:18:06.840 unable to make any long distance calls. Why why could 0:18:06.840 --> 0:18:09.920 they no longer reach anybody? Well, A T and T 0:18:10.200 --> 0:18:15.159 s long distance switches, which control that and allow for 0:18:15.320 --> 0:18:18.680 the actual connections to be made, were on the fritz. 0:18:19.040 --> 0:18:21.639 They were trying to reboot over and over again. They 0:18:21.640 --> 0:18:25.399 were just stuck in a reboot cycle. Now, initially the 0:18:25.480 --> 0:18:28.760 company thought it was being hacked, But like I said 0:18:28.760 --> 0:18:30.800 at the top of the show, I'm not covering stories 0:18:30.840 --> 0:18:34.040 about hackers here. I'm talking about big design flaws that 0:18:34.119 --> 0:18:39.280 caused problems. So they weren't getting hacked. That's not what 0:18:39.320 --> 0:18:42.600 was going on with those D fourteen long distance switches. No, 0:18:42.760 --> 0:18:45.760 there was a design problem at fault, so What had 0:18:45.760 --> 0:18:47.439 happened was a T and D had rolled out an 0:18:47.520 --> 0:18:50.960 update to the code that managed the switches, and it 0:18:51.040 --> 0:18:53.280 was meant to increase the efficiency. It was meant to 0:18:53.359 --> 0:18:56.480 speed things up. But the problem was it sped things 0:18:56.560 --> 0:18:59.840 up so much that the system got caught up in itself. 0:19:00.240 --> 0:19:02.000 It gets pretty technical, but I can give you kind 0:19:02.000 --> 0:19:05.479 of a overview of what the problem was. Alright, so 0:19:05.640 --> 0:19:10.120 each switch had a function that allowed it to alert 0:19:10.359 --> 0:19:13.960 the next switch down the line if things were starting 0:19:13.960 --> 0:19:17.919 to get harry. So imagine the switch number one is 0:19:17.960 --> 0:19:21.239 handling traffic, but it's getting really close to capacity. So 0:19:21.280 --> 0:19:23.560 it sends a message over to switch number two and says, 0:19:23.720 --> 0:19:27.040 I can't take on any more work because if I do, 0:19:27.160 --> 0:19:31.240 I'll be overloaded. Switch to then says, no problem, I'll 0:19:31.240 --> 0:19:35.040 take on the any oncoming work for you and we'll 0:19:35.080 --> 0:19:38.000 handle it from there. And if switched number two order 0:19:38.080 --> 0:19:40.720 to get into the same source situation, it would say 0:19:40.760 --> 0:19:42.679 the same thing to switch number three, and so on 0:19:42.720 --> 0:19:46.600 and so forth. Now, eventually each switch will contact the 0:19:46.600 --> 0:19:49.320 one below it and say, hey, how are you doing there? 0:19:49.720 --> 0:19:53.040 And if the answer is okay, then everything switches back 0:19:53.080 --> 0:19:56.600 and you go back to normal operation. That's how it's 0:19:56.600 --> 0:20:01.000 supposed to work up. But A T and He's updated 0:20:01.040 --> 0:20:05.520 code sped things up so much it caused some real issues, 0:20:05.600 --> 0:20:09.000 and there was some poor timing, just coincidental timing that 0:20:09.040 --> 0:20:12.600 made things worse. So switch number one starts to get 0:20:12.640 --> 0:20:15.480 overwhelmed and sends a message over to switch number two, 0:20:15.480 --> 0:20:17.960 But switched number two was just in the middle of 0:20:18.080 --> 0:20:22.240 resetting itself. So switch number two goes into reset mode, 0:20:22.280 --> 0:20:24.520 which says do not disturb. Sends a message over to 0:20:24.560 --> 0:20:28.200 switch number three. That prompted switched number three to overload 0:20:28.480 --> 0:20:30.520 and put up a do not disturb sign. Move that 0:20:30.520 --> 0:20:33.520 down to switch number four. This whole thing goes down 0:20:33.560 --> 0:20:36.919 the entire line of on switches. They all end up 0:20:36.920 --> 0:20:39.760 getting overloaded as a result of this, and I'll go 0:20:39.800 --> 0:20:44.600 into reset mode and they get stuck there. That problem 0:20:44.680 --> 0:20:48.560 lasted for nine hours before A T and T was 0:20:48.600 --> 0:20:51.480 finally able to address the message load on the entire 0:20:51.560 --> 0:20:54.639 system and get the switches back to normal. The estimated 0:20:54.760 --> 0:20:57.840 cost of lost revenue for that time was about sixty 0:20:57.960 --> 0:21:01.560 million dollars in long distance calls, and there were a 0:21:01.600 --> 0:21:04.440 lot of angry customers to boot, so to placate them, 0:21:04.480 --> 0:21:07.600 A T and T offered reduced long distance rates on 0:21:07.720 --> 0:21:12.320 Valentine's Day pretty ugly by the A, T and T 0:21:12.320 --> 0:21:14.159 tried to handle it, at least in a way that 0:21:14.320 --> 0:21:17.920 didn't turn it into a pr nightmare. Not so with Intel. 0:21:18.600 --> 0:21:21.199 That's what it brings us to the Pendium problem. I 0:21:21.240 --> 0:21:23.880 don't know if you guys remember when Penium processors first 0:21:23.960 --> 0:21:26.600 came out, but they were a big deal. It was 0:21:27.560 --> 0:21:30.679 a redesign of the architecture of the microprocessor and it 0:21:30.720 --> 0:21:34.119 was meant to really speed things up. Well, Intel had 0:21:34.160 --> 0:21:38.679 a massive nightmare in n thanks to a flaw in 0:21:38.760 --> 0:21:43.960 the entire first generation of Pentium processors. Now, when you 0:21:44.000 --> 0:21:47.280 break it all down, a CPU is all about performing 0:21:47.320 --> 0:21:50.320 mathematical operations on data, so it's kind of important that 0:21:50.440 --> 0:21:55.480 does this correctly. Unfortunately, the flaw and the Pentium processors 0:21:55.520 --> 0:21:57.840 kind of messed that up. And the issue has to 0:21:57.840 --> 0:22:01.760 do with floating point operations. So the predecessor to the Pentium, 0:22:01.840 --> 0:22:05.639 the four six, used a shift and subtract algorithm for 0:22:05.680 --> 0:22:10.840 floating point operations, which was effective but relatively slow compared 0:22:10.880 --> 0:22:15.480 to what Intel thought they could do by totally redesigning 0:22:15.480 --> 0:22:19.840 that structure and using a look up table approach. Now, 0:22:19.840 --> 0:22:23.040 the table was supposed to have one thousand sixty six 0:22:23.320 --> 0:22:27.679 entries programmed directly onto the logic array of the Pentium processor, 0:22:28.800 --> 0:22:33.200 but for some reason only one thousand sixty one entries 0:22:33.280 --> 0:22:37.680 made it. Five entries went missing and essentially returned an 0:22:37.680 --> 0:22:42.280 answer of zero instead of what they were supposed to say, 0:22:42.359 --> 0:22:44.800 so if a calculation accessed one of those missing cells, 0:22:44.840 --> 0:22:47.159 it got zero, even though that's not the correct answer. 0:22:48.080 --> 0:22:50.600 All the first generation pentiums went out with this error 0:22:50.640 --> 0:22:53.080 because it was so minor that it wasn't even picked 0:22:53.200 --> 0:22:58.560 up by Intel's quality control at the time. Now, processes 0:22:58.560 --> 0:23:02.040 worked just fine up to the eighth decimal point. Beyond 0:23:02.119 --> 0:23:05.240 that things got messy, But for most folks that wasn't 0:23:05.240 --> 0:23:09.800 a problem because they weren't doing mathematical calculations that needed 0:23:09.840 --> 0:23:13.280 that level of precision. It just wasn't a thing. In fact, 0:23:13.320 --> 0:23:16.280 there was only a one in three sixty billion chance 0:23:16.760 --> 0:23:20.520 that this error would cause a big enough problem to 0:23:20.640 --> 0:23:25.040 reach up to the fourth decimal place. So most calculations 0:23:25.040 --> 0:23:27.879 that were simple were bulletproof. You you were fine. But 0:23:27.960 --> 0:23:31.320 if you needed that precision, if you needed that really 0:23:31.440 --> 0:23:35.320 fine degree, that's when you would encounter the flaw, and 0:23:35.400 --> 0:23:39.320 that happened because there are math professors in this world, 0:23:39.760 --> 0:23:44.919 and one of those, Thomas Nicely, discovered in October that 0:23:45.000 --> 0:23:48.080 he was getting errors because of this issue. He needed 0:23:48.080 --> 0:23:51.720 the processor to work correctly, and so he contacted Intel 0:23:51.760 --> 0:23:55.679 about the problem. And this is where we take a 0:23:55.720 --> 0:23:58.520 moment to acknowledge there's a right way and a wrong 0:23:58.560 --> 0:24:03.360 way to handle an issue. That's your fault until decided 0:24:03.400 --> 0:24:06.080 to go the wrong way. My opinion is, if you 0:24:06.119 --> 0:24:08.400 make a mistake, it's usually a good idea to just 0:24:08.600 --> 0:24:10.800 own up to it and try to make it better. 0:24:11.480 --> 0:24:13.840 But Intel's response was more along the lines of, yeah, 0:24:13.920 --> 0:24:16.080 we didn't think it was a big deal. And then 0:24:16.119 --> 0:24:20.680 Intel made other pr blenders. But because people began to hear, hey, 0:24:20.760 --> 0:24:24.000 that pentium processor in your computer that you just bought, 0:24:24.760 --> 0:24:28.359 it doesn't work properly. So people wanted to get replacements. 0:24:29.040 --> 0:24:31.399