Speaker 1: Get in touch with technology with tech Stuff from how
Speaker 1: Stuff works dot com. Hey guys, welcome to tech Stuff.
Speaker 1: I'm your host Jonathan strickland Um with super producer Dylan
Speaker 1: in the teeny Tiny Stuff You Should Know Studio. Pretty soon,
Speaker 1: I'm going to demand it be called the Stuff You
Speaker 1: Should Know in Tech Stuff Studio, because who's gonna stop me.
Speaker 1: Today I wanted to take a look at some of
Speaker 1: the most spectacular computer bugs ever made, or at least
Speaker 1: some of the more notable ones, and I got the
Speaker 1: inspiration for this episode after researching a bug affecting the
Speaker 1: Spotify desktop app. More on that in just a second,
Speaker 1: but first I have to address a bit of apocryphal history,
Speaker 1: and regrettably it's a story that we've repeated on tech Stuff.
Speaker 1: So I'm sad to admit that I was complicit, although unknowingly,
Speaker 1: in the spread of misinformation and that all has to
Speaker 1: do with the origin of the term bug to describe
Speaker 1: a flaw in programming. So here's the popular story, the
Speaker 1: one that we have accidentally, uh promoted on tech Stuff
Speaker 1: without knowing that we were in the wrong. It goes
Speaker 1: that Grace Hopper, who was an early computer scientist who
Speaker 1: rose to the rank of Rear admiral in the U. S.
Speaker 1: Navy coined the phrase bug after discovering a moth coming
Speaker 1: up Harvard's Mark two calculator, a literal bug. Generally speaking,
Speaker 1: the story tends to be set in nineteen and there
Speaker 1: is even a note in the log book that reads
Speaker 1: first actual case of bug being found that's attributed to
Speaker 1: Grace Hopper. But there are several points that are wrong
Speaker 1: in this story. First, the year it didn't happen in ninet.
Speaker 1: It happened on September. N We know because there's a
Speaker 1: log book. At the log book that marks the incident
Speaker 1: not only has the notes, it actually has the moth
Speaker 1: taped into the book itself. It's taped onto the page. Second,
Speaker 1: Grace Hopper wasn't the person to discover the moth or
Speaker 1: make that log entry. She did tell the story about
Speaker 1: the moth several times, but it wasn't in the context
Speaker 1: of finding it or logging it. She just told the
Speaker 1: story that, yeah, we really did have a bug in
Speaker 1: the system. And most importantly, the word bug had already
Speaker 1: been used to describe design flaws for decades before the
Speaker 1: Mark two was even designed. In fact, if you look
Speaker 1: at the log book. This makes sense. It says first
Speaker 1: actual case of bug being found. That sentence doesn't make
Speaker 1: sense unless you've already used the word bug to describe
Speaker 1: a flaw, because you wouldn't say first actual case of
Speaker 1: bug being found. That the wording doesn't make any sense.
Speaker 1: The context makes no sense. Sadly, there are documented quotes
Speaker 1: dating back to the nineteenth century using the word bug
Speaker 1: to mean a design fault, and it could go back
Speaker 1: even further than that. So is with much regret that
Speaker 1: I admit I have unwittingly contributed to a bit of
Speaker 1: misleading folklore making the rounds. But I'm glad I can
Speaker 1: take this opportunity to address it. All Right, So let's
Speaker 1: talk about design bugs, and I'll be covering several goofs, mistakes, flubs, flaws,
Speaker 1: and outright catastrophes in this episode. But one thing I'm
Speaker 1: not necessarily going to cover our software vulnerabilities that were
Speaker 1: later exploited, either by opportunistic hackers or white hats who
Speaker 1: are just trying to improve system security. Those vulnerabilities are
Speaker 1: common in many types of software and arise not just
Speaker 1: through mistakes, but sometimes simple oversights, and I think it
Speaker 1: might be more fun to look at some real bugs,
Speaker 1: like stuff that made things go wrong, stuff that may
Speaker 1: have rendered a program defunct or otherwise caused headaches. Now
Speaker 1: I'm gonna make an exception to this. I'm going to
Speaker 1: start off with the ping of death, and I only
Speaker 1: mention it because it has an awesome name. Now, this
Speaker 1: flaw caused headaches back in ninety six. It was a
Speaker 1: flawed i P fragmentation reassembly code, and it became possible
Speaker 1: to crash lots of different types of computers using different
Speaker 1: operating systems, although Windows machines were particularly vulnerable, and this
Speaker 1: particular flaw would make a Windows machine revert to the
Speaker 1: dreaded blue screen of death. And it all happened by
Speaker 1: sending a special ping packet over the internet. So, for
Speaker 1: those of you who aren't familiar with what that is,
Speaker 1: a ping is essentially a simple message that checks for
Speaker 1: a connection between two computers. You send one ping from
Speaker 1: a computer to another one and look for a response,
Speaker 1: so that way you verify there is in fact a connection.
Speaker 1: You can also tell other things like how fast is
Speaker 1: that connection between those two computers. Now, in this case,
Speaker 1: you would have to actually design a malformed ping request
Speaker 1: and send that to a target, and it would bring
Speaker 1: that target down. That's the only security vulnerability story I
Speaker 1: really wanted to focus on. The others are all just
Speaker 1: design flaws. And let's begin with the bug that inspired
Speaker 1: me to do this episode in the first place, That
Speaker 1: Spotify bug I mentioned earlier. Ours Technica wrote a piece
Speaker 1: on it in November two thousand and sixteen, but the
Speaker 1: problem seems to date back at least as far as
Speaker 1: June two thousand sixteen, and that's when a few savvy
Speaker 1: Spotify users noticed some unusual activities on their computers. And
Speaker 1: it took a little bit of detective work, but they
Speaker 1: discovered that Spotify was apparently generating a huge amount of
Speaker 1: data on a daily basis, like gigabytes of data per day.
Speaker 1: And the culprit turned out to be a vacuum process
Speaker 1: for a database file containing the string mercury dot dB. Now,
Speaker 1: the vacuum process is the digital equivalent of vacuum c link.
Speaker 1: It's meant to repack data so that it takes up
Speaker 1: less space on a drive. Now, this involves building a
Speaker 1: new file to maximize efficiency, which is a good thing
Speaker 1: generally speaking. The problem was that Spotify's version was making
Speaker 1: it happen way too frequently, like on the order of
Speaker 1: once every few minutes, so that's not generally necessary. You
Speaker 1: don't need to rebuild a database file every few minutes
Speaker 1: to make sure it's the most efficient size it can be.
Speaker 1: So each rebuild represented a relatively small amount of data,
Speaker 1: but over time it added up, which meant that if
Speaker 1: you had Spotify on on your computer, even if it
Speaker 1: was just running in the background, it would be generating
Speaker 1: gigabytes worth of information rewriting this file over and over.
Speaker 1: Now it wasn't filling up a hard drive. It was
Speaker 1: just overwriting the same file. Now, if it had been
Speaker 1: filling up a hard drive, people would have noticed much earlier,
Speaker 1: and it wouldn't have just been savvy Spotify users, because
Speaker 1: you would suddenly notice, hey, I don't I can't save
Speaker 1: anything to my hard drive because everything is filling up. Instead. Again,
Speaker 1: it was just sort of writing and deleting and writing
Speaker 1: and deleting the same file over and over again. And
Speaker 1: that probably doesn't sound like a big deal, but it
Speaker 1: is a problem if you're using a solid state drive
Speaker 1: or s s D. So one of the drawbacks of
Speaker 1: an ss D is that over time it loses storage capacity,
Speaker 1: like you can store less data on an SSD over time. Now,
Speaker 1: by overtime I generally mean over a great deal of
Speaker 1: time and a lot of different data being written to
Speaker 1: it and overwritten. Uh. Generally speaking, most of us end
Speaker 1: up replacing our drives before we get to a point
Speaker 1: where the loss of capacity is a real issue. But
Speaker 1: similar in a way to how a battery can lose
Speaker 1: its ability to hold a full charge after you've gone
Speaker 1: through lots of charging and discharging cycles, you know how
Speaker 1: a battery won't be able to to hold as much
Speaker 1: even if it says it's up to a but a
Speaker 1: hundred percent doesn't last you as long as it used to.
Speaker 1: That's because its capacity to hold a full charge has
Speaker 1: decreased over time. But let's say you've got a program
Speaker 1: that's just constantly overwriting data to your drive, you might
Speaker 1: discover that your ss D s useful lifespan has been
Speaker 1: drastically reduced. So as I record this episode, Spotify has
Speaker 1: already rolled out an updated version of its desktop application,
Speaker 1: and that, by the way, is the only version of
Speaker 1: Spotify that was affected. If you use web based Spotify
Speaker 1: or mobile Spotify, you're in the clear already if you
Speaker 1: use a desktop version, as long as you have version
Speaker 1: one point zero point four two or later, you are fine.
Speaker 1: But if you did have that earlier version and you
Speaker 1: just had Spotify running on in the background, chances are
Speaker 1: it was writing to your hard drive like crazy. So
Speaker 1: what about some of the other big bugs in computer history. Well,
Speaker 1: some of the real doozies involve our attempts to explore
Speaker 1: the final frontier. So we'll be talking about space a
Speaker 1: few times in this episode, and we'll start with an
Speaker 1: early US satellite. So first up is a nineteen sixty
Speaker 1: two blunder involving the Mariner one. So some backstory on
Speaker 1: this one. Uh, We're gonna talk a lot about the
Speaker 1: Soviet Union in this episode two. It takes a couple
Speaker 1: of roles as we go on. But in this case,
Speaker 1: the then USSR had launched Sputnik into orbit in nineteen
Speaker 1: fifty seven, which really kicked off the space race and
Speaker 1: also was a big shot in the Cold War because
Speaker 1: of so Union was essentially saying, hey, we can launch
Speaker 1: this into space, we could also launch something at you.
Speaker 1: In response, the US and done sort of the same thing.
Speaker 1: They had launched some satellites into space and the Mariner
Speaker 1: one was going to be a big, big feather in
Speaker 1: the cap of the US. The whole idea was to
Speaker 1: launch a probe that would be a fly by probe
Speaker 1: and it would go by Venus. So uh NASA, which
Speaker 1: was newly formed in nineteen sixty two, was taking control
Speaker 1: of this and the budget for this particular project was
Speaker 1: eight team point five million dollars, which if you were
Speaker 1: to adjust for inflation, would be almost a hundred fifty
Speaker 1: million dollars today, So a hundred fifty million dollar project
Speaker 1: to launch the Mariner one and have it fly by Venus.
Speaker 1: But as I'm sure you guys have figured out by
Speaker 1: now based upon the topic of this podcast, not all
Speaker 1: went according to plan. Not long at all. After the
Speaker 1: rocket launched from the launch pad, it began to veer
Speaker 1: off course, and neither the computer controls on the rocket
Speaker 1: or manual controls back at HQ could correct for the problem.
Speaker 1: The rockets course was such that it was going to
Speaker 1: take it over shipping lanes, which meant there could be
Speaker 1: a potential catastrophe, and so Arrange safety officer made the
Speaker 1: difficult call and issued the command to blow the whole
Speaker 1: thing up, just shy of three hundred seconds after it launched.
Speaker 1: So what happened? What why did it go off course
Speaker 1: in the first place? Well, there was a flaw in
Speaker 1: the space crafts guidance software which diverted the rocket, and
Speaker 1: no amount of commands from ground control could correct for it.
Speaker 1: After a lengthy investigation, NASA discovered the era error was
Speaker 1: the result of a mistake transcribing handwritten notes into computer code.
Speaker 1: So someone just took some handwritten notes and misinterpreted one
Speaker 1: of them, and that one mistake was enough to crash
Speaker 1: the rocket, or to to necessitate it being destroyed. The
Speaker 1: great science fiction author Arthur C. Clark wrote that the
Speaker 1: Mariner one was wrecked by the most expensive hyphen in history,
Speaker 1: which isn't quite right, but it's pretty funny. I mean,
Speaker 1: come on, it's humorous phrase. So the actual punctuation mark
Speaker 1: that caused the problem was not technically a hyphen. It
Speaker 1: was a superscript bar. Superscript bars, by the way, not
Speaker 1: a place where player rights hang out to get tore up.
Speaker 1: A superscript bar just means it's a horizontal bar that
Speaker 1: is above some other symbol. In this case, it was
Speaker 1: a radius symbol, and that was a symbol along with
Speaker 1: the superscript bar to describe a smoothing function, which means
Speaker 1: the formula was meant to calculate smoothed values over the
Speaker 1: time derivative of a radius. Now, without the smoothing function,
Speaker 1: tiny deviations in course sent commands to the rockets thrusters
Speaker 1: to kick in big time to overcorrect for that problem.
Speaker 1: As an analogy, imagine you're driving a vehicle and you
Speaker 1: see a pothole in the road and you're approaching it,
Speaker 1: and instead of gently steering out of the way, you
Speaker 1: wrenched the wheel really hard to the left or to
Speaker 1: the right in order to try and get around this pothole.
Speaker 1: That's kind of what was happening with the rocket. It
Speaker 1: didn't have the smoothing function and so as a result,
Speaker 1: it was having these wild deviations and course. So it
Speaker 1: wasn't a high fin that caused the problem, but is
Speaker 1: close enough. Our next space story takes place in nineteen
Speaker 1: with the European Space Agencies RAN five Flight five O
Speaker 1: one rocket. Now, this rocket was to launch into space
Speaker 1: on June four nine, and instead the rocket disintegrated forty
Speaker 1: seconds after taking off. So what the heck happened? Well,
Speaker 1: it largely had to do with the e s A
Speaker 1: reusing old work. This actually becomes a theme in this episode.
Speaker 1: One of the morals of of this entire podcast is
Speaker 1: if you're designing something a successor to an earlier product,
Speaker 1: and you'd want to reuse some of the features that
Speaker 1: you created in your previous product, test the heck out
Speaker 1: of it in its new form factor, because it could
Speaker 1: be that things that worked perfectly fine in the earlier
Speaker 1: model will go awry in the new one. That's what
Speaker 1: happened here. So as you might guess from the name,
Speaker 1: the Aeryan five marked the fifth generation of launch vehicles
Speaker 1: under that name. The Arian four's inertial reference system would
Speaker 1: convert sixty four bit floating point numbers into a sixteen
Speaker 1: bits signed integer, and it worked just fine. But the
Speaker 1: Arian five stats were beefier than its predecessor with faster engines,
Speaker 1: and that was where the problem really started. The engine
Speaker 1: output meant those sixty four bit floating point numbers were
Speaker 1: significantly larger than the ones generated by the engines on
Speaker 1: the Arian four. They didn't anticipate this, so during the
Speaker 1: conversion process there was actually data overflow, and that overflow
Speaker 1: caused both the backup computer and the primary computer aboard
Speaker 1: the Arian five to crash, and they crashed in that order.
Speaker 1: The backup computer crashed first, followed by the primary computer
Speaker 1: a couple of seconds later. The whole thing took less
Speaker 1: than a minute to go from launch to disintegration. Oops,
Speaker 1: now we're gonna stick with space. But jumped forward to
Speaker 1: and the Mars Climate Orbiter. This was an unfortunate problem.
Speaker 1: So this particular spacecraft was meant to study Mars's climate,
Speaker 1: atmosphere and surface changes, and it was also supposed to
Speaker 1: be a kind of relay station for landers that would
Speaker 1: explore Mars's surface, but none of that would last because
Speaker 1: of some pretty significant goofs. So on September, the orbiter
Speaker 1: passed into the upper atmosphere of Mars and did so
Speaker 1: at a pretty low altitude. And this is what folks
Speaker 1: in the space industry called a bad thing. The Dragon,
Speaker 1: the spacecraft was significant. It began to fall apart and
Speaker 1: it was destroyed upon entering Mars's atmosphere. That's what happened.
Speaker 1: So the software guide the orbiter was to blame, and
Speaker 1: it's a dumb, dumb mistake. It was supposed to make
Speaker 1: adjustments to the orbiter's flight in SI units, specifically in
Speaker 1: Newton seconds. That's what the contract with Lockheed and NASA said,
Speaker 1: Newton seconds, use SI units for all of your all
Speaker 1: of your calculations, but the software instead made calculations in
Speaker 1: non SI units, namely pounds seconds. So Lockheeds software gave
Speaker 1: information to NASA's systems using the wrong units of measure.
Speaker 1: NASA systems then took that information, assuming it was with
Speaker 1: the right units of measure, and executed commands based upon that.
Speaker 1: Uh So, this is why if you're ever in a
Speaker 1: math course and the teacher makes you stop in the
Speaker 1: middle of writing a problem on the board and says,
Speaker 1: where are your units? This is why you have to
Speaker 1: make sure you're using the right units, because if you're
Speaker 1: saying a number and you don't associate a unit with it,
Speaker 1: someone could make an incorrect decision based on that, and
Speaker 1: it could be disastrous, as it was with the case
Speaker 1: of this orbiter. The thrusters fired at four point four
Speaker 1: or five times the power they were supposed to, and
Speaker 1: the orbiter didn't stand a chance. And this was a
Speaker 1: pretty expensive mistake. That mission's cost came in at three
Speaker 1: seven point six million dollars, but on the bright side
Speaker 1: with all of these stories, at least no human lives
Speaker 1: were ever in real danger as a result of the mistake. Now,
Speaker 1: I've got a lot more to say about bugs, but
Speaker 1: before I get into that, let's take a quick break
Speaker 1: to thank our sponsor. All right, Now, let's make a
Speaker 1: switch to A T and T, which is a company
Speaker 1: that had a pretty big problem with switches once upon
Speaker 1: a time. I'm talking about issue that popped up on
Speaker 1: January nine nine. That's when A T and T long
Speaker 1: distance customers discovered they were unable to make any long
Speaker 1: distance calls. Why why could they no longer reach anybody? Well,
Speaker 1: A T and T s long distance switches, which control
Speaker 1: that and allow for the actual connections to be made,
Speaker 1: were on the fritz. They were trying to reboot over
Speaker 1: and over again. They were just stuck in a reboot cycle. Now,
Speaker 1: initially the company thought it was being hacked, But like
Speaker 1: I said at the top of the show, I'm not
Speaker 1: covering stories about hackers here. I'm talking about big design
Speaker 1: flaws that caused problems. So they weren't getting hacked. That's
Speaker 1: not what was going on with those D fourteen long
Speaker 1: distance switches. No, there was a design problem at fault.
Speaker 1: So what had happened was a T and D had
Speaker 1: rolled out an update to the code that managed the switches,
Speaker 1: and it was meant to increase the efficiency. It was
Speaker 1: meant to speed things up, but the problem was it
Speaker 1: sped things up so much that the system got caught
Speaker 1: up in itself. It gets pretty technical, but I can
Speaker 1: give you kind of a overview of what the problem was. Alright,
Speaker 1: so each switch had a function that allowed it to
Speaker 1: alert the next switch down the line if things were
Speaker 1: starting to get hairy. So imagine that switch number one
Speaker 1: is handling traffic, but it's getting really close to capacity.
Speaker 1: So it sends a message over to switch number two
Speaker 1: and says, I can't take on any more work because
Speaker 1: if I do, I'll be overloaded. Switch to then says,
Speaker 1: no problem, I'll take on the any oncoming work for
Speaker 1: you and we'll handle it from there. And if switched
Speaker 1: number two order to get into the same source situation,
Speaker 1: it would say the same thing to switch number three,
Speaker 1: and so on and so forth. Now, eventually each switch
Speaker 1: will contact the one below it and say, hey, how
Speaker 1: are you doing there? And if the answer is okay,
Speaker 1: then everything switches back and you go back to normal operation.
Speaker 1: That's how it's suppo was to work up. But A
Speaker 1: T and t s updated code sped things up so
Speaker 1: much it caused some real issues, and there was some
Speaker 1: poor timing, just coincidental timing that made things worse. So
Speaker 1: switch number one starts to get overwhelmed and sends a
Speaker 1: message over to switch number two, but switched number two
Speaker 1: was just in the middle of resetting itself. So switch
Speaker 1: number two goes into reset mode, which says do not disturb.
Speaker 1: Sends a message over to switch number three. That prompted
Speaker 1: switch number three to overload and put up I do
Speaker 1: not disturb sign. Move that down to switch number four.
Speaker 1: This whole thing goes down the entire line of on switches.
Speaker 1: They all end up getting overloaded as a result of this,
Speaker 1: and I'll go into reset mode and they get stuck there.
Speaker 1: That problem lasted for nine hours before A T and
Speaker 1: T was finally able to address the message load on
Speaker 1: the entire system and get the switches back to normal.
Speaker 1: The estimated cost of lost revenue for that was about
Speaker 1: sixty million dollars in long distance calls, and there were
Speaker 1: a lot of angry customers to boot, so to placate them,
Speaker 1: A T and T offered reduced long distance rates on
Speaker 1: Valentine's Day pretty ugly. By the A T and T
Speaker 1: tried to handle it, at least in a way that
Speaker 1: didn't turn it into a pr nightmare. Not so with Intel.
Speaker 1: That's what it brings us to the Pendium problem. I
Speaker 1: don't know if you guys remember when Pentium processors first
Speaker 1: came out, but they were a big deal. It was
Speaker 1: a redesign of the architecture of the microprocessor and it
Speaker 1: was meant to really speed things up. Well, Intel had
Speaker 1: a massive nightmare in nine thanks to a flaw in
Speaker 1: the entire first generation of Pentium processors. Now, when you
Speaker 1: break it all down, a CPU is all about performing
Speaker 1: mathematical operations on data, so it's kind of important that
Speaker 1: it does this correctly. Unfortunately, the flaw and the Pentium
Speaker 1: processors kind of messed that up. And the issue has
Speaker 1: to do with floating point operations. So the predecessor to
Speaker 1: the Pentium, the four six, used a shift and subtract
Speaker 1: algorithm for floating point operations, which was effective but relatively
Speaker 1: slow compared to what Intel thought they could do by
Speaker 1: totally redesigning that structure and using a look up table approach. Now,
Speaker 1: the table was supposed to have one thousand sixty six
Speaker 1: entries programmed directly onto the logic array of the Pentium processor,
Speaker 1: but for some reason only one thousand sixty one entries
Speaker 1: made it. Five entries went missing and essentially returned an
Speaker 1: answer of zero instead of what they were supposed to say,
Speaker 1: So if a calculation accessed one of those missing cells,
Speaker 1: it got zero, even though that's not the correct answer.
Speaker 1: All the first generation pentiums went out with this error
Speaker 1: because it was so minor that it wasn't even picked
Speaker 1: up by Intel's quality control at the time. Now, processes
Speaker 1: work just fine up to the eighth decimal point. Beyond
Speaker 1: that things got messy. But for most folks that wasn't
Speaker 1: a problem because they weren't doing mathematical calculations that needed
Speaker 1: that level of precision. It just wasn't a thing. In fact,
Speaker 1: there was only a one in three sixty billion chance
Speaker 1: that this error would cause a a big enough problem
Speaker 1: to reach up to the fourth decimal place. So most
Speaker 1: calculations that were simple were bulletproof. You you're fine, But
Speaker 1: if you needed that precision, if you needed that really
Speaker 1: fine degree, that's when you would encounter the flaw, and
Speaker 1: that happened because there are math professors in this world,
Speaker 1: and one of those, Thomas nicely discovered in October that
Speaker 1: he was getting errors because of this issue. He needed
Speaker 1: the processor to work correctly, and so he contacted Intel
Speaker 1: about the problem. And this is where we take a
Speaker 1: moment to acknowledge there's a right way and a wrong
Speaker 1: way to handle an issue. That's your fault until decided
Speaker 1: to go the wrong way. My opinion is, if you
Speaker 1: make a mistake, it's usually a good idea to just
Speaker 1: own up to it and try to make it better.
Speaker 1: But Intel's response was more along the lines of yeah,
Speaker 1: we didn't think it was a big deal. And then
Speaker 1: Intel made other pr blenders. But because people began to hear, hey,
Speaker 1: that Pentium processor in your computer that you just bought,
Speaker 1: it doesn't work properly. So people wanted to get replacements.
Speaker 1: But Intel said, oh, we're only going to replace the
Speaker 1: ones if you can prove that the mistakes that it
Speaker 1: makes affect you in some meaningful way. So they weren't
Speaker 1: They weren't denying that there was a problem. They were
Speaker 1: just saying, hey, unless you can prove the problem affects
Speaker 1: you where we don't care. That didn't go well. If
Speaker 1: you create a product and you market it as the
Speaker 1: future of computing and then it's discovered there's a flaw
Speaker 1: on the design, and then you say we'll replace it,
Speaker 1: but only if you proof you deserve it, it doesn't
Speaker 1: tend to make your customer base very happy. So ultimately,
Speaker 1: until reverse that decision and offered to replace the processor
Speaker 1: for anyone who wanted it who had a first generation Pentium,
Speaker 1: and that mistake ended up costing the company four million dollars. Yikes.
Speaker 1: All right, now we're gonna switch Gears over to Microsoft. First.
Speaker 1: I think you could claim that all of Microsoft Bob
Speaker 1: product that was supposed to be an easy, accessible computer
Speaker 1: interface was really just a massive software bug. I mean,
Speaker 1: it introduced comic sands for goodness sakes, the cluttered organization system,
Speaker 1: the lack of meaningful security, and other numerous issues plagued
Speaker 1: that software. But we did an entire episode of Tech
Speaker 1: Stuff about Microsoft Bob a couple of years ago, so
Speaker 1: I'm not gonna dwell on it anymore, but if you
Speaker 1: want to hear more about it, go in that episode.
Speaker 1: It was a fun one. Now in two thousand seven,
Speaker 1: Microsoft experienced a massive headache when a bug on their
Speaker 1: servers notified thousands of Windows customers that they were filthy,
Speaker 1: dirty software pirates and they should be punished. These include
Speaker 1: people who actually had legitimate, legal purchase copies of Windows
Speaker 1: XP or Vista. So the problem here was Microsoft had
Speaker 1: an initiative called Windows Genuine Advantage, and it was a
Speaker 1: nice name for a strategy meant to curtail operating system piracy. Essentially,
Speaker 1: it was a component in Windows that would allow Microsoft
Speaker 1: to figure out if the copy of Windows on any
Speaker 1: given computer was legit. In other words, it was a
Speaker 1: d r M strategy. But in two thousand seven, a
Speaker 1: buggy install of software on a server misidentified thousands of legitimate,
Speaker 1: law abiding customers as pirates for nineteen hours. The software
Speaker 1: just laid on the law, and so people began to
Speaker 1: receive sternly written warnings about their choice to indulge in
Speaker 1: bad behavior. And if you were a Windows Vista customer,
Speaker 1: you had it the worst, not just because you were
Speaker 1: using Windows Vista, which I think we all agree was
Speaker 1: not one of the bright points and Microsoft's operating system history,
Speaker 1: but also because Microsoft had built in the ability for
Speaker 1: Windows genuine advantage to switch off certain operating system features
Speaker 1: in Windows Vista if it determined that the copy someone
Speaker 1: was using was a pirated version, so it was misidentifying
Speaker 1: real versions as pirated ones turning off features, and these
Speaker 1: are for people who have bought legitimate copies. This, by
Speaker 1: the way, is one of the big arguments people have
Speaker 1: against DRM. It has the tendency to punish legitimate customers.
Speaker 1: And you feel like you're stupid for buying a copy
Speaker 1: of a piece of software rather than just stealing one
Speaker 1: that has had those feet yours or those defenses removed,
Speaker 1: Like why you're you're creating more incentives for people to
Speaker 1: go outside and get a pirated copy. Alright, so imagine
Speaker 1: you've purchased this legitimate copy of Windows Vista. First of all,
Speaker 1: you you already feel bad. Then you're told you're a thief,
Speaker 1: so you feel worse. Then someone remotely switches off several
Speaker 1: features of your operating system. That was not a great
Speaker 1: pr message, So that was a real issue. They did
Speaker 1: eventually fix it after that nineteen hours, but by then
Speaker 1: people were already very upset. Also, I don't want to
Speaker 1: just you know, pile lots of abuse onto Microsoft I
Speaker 1: gotta talk about Apple here too. So the company prides
Speaker 1: itself on a high standard of quality and general it's
Speaker 1: pretty good about living up to that standard of quality,
Speaker 1: depending upon your point of view of their various products.
Speaker 1: But that hasn't stopped a few clunkers getting through and
Speaker 1: into the public hands. And that was the case in
Speaker 1: two thousand twelve with Apple Maps. If you owned an
Speaker 1: iPhone back in two thousand twelve when Apple Maps came out,
Speaker 1: you may remember this problem. It's pretty well publicized. Maps
Speaker 1: were inaccurate, sometimes leaving out important details like you know,
Speaker 1: a river or a lake between you and your destination,
Speaker 1: things that might be important if I don't know you
Speaker 1: don't drive an amphibious vehicle, might not have a road
Speaker 1: on there that's important. Might misidentify the location of a
Speaker 1: historical landmark. For instance, that thought the Washington Monument was
Speaker 1: across the street from where it is, But nope, it's
Speaker 1: just where we left it. Despite all of Roland emericks
Speaker 1: best attempts to move it or destroy it, it's still there.
Speaker 1: The real problem here was that the Apple software just
Speaker 1: wasn't ready for public unveiling. It was. It needed a
Speaker 1: lot more testing. It was trying to play catch up
Speaker 1: to Google Maps, but Google had the advantage of working
Speaker 1: with companies that have been doing mapping software for years.
Speaker 1: Google acquired the those companies and acquired the expertise of
Speaker 1: people who have been working on that software. And Apple
Speaker 1: was really just trying to create their own version and
Speaker 1: get it out as fast as it could. But it
Speaker 1: got out a little too early, and the company spent
Speaker 1: the next several months tweaking maps and trying to keep
Speaker 1: control of the situation. But by that time, many of
Speaker 1: Apple's fans, even the most devoted ones, had kind of
Speaker 1: given up and switched over to Google Maps instead. Well,
Speaker 1: that's most of the fun stuff. I've got some really
Speaker 1: serious bugs to cover. But before I do that, let's
Speaker 1: take another quick break and thank our sponsor. Now I'm
Speaker 1: going to transition into some serious bugs. These are ones
Speaker 1: that either threatened the lives of people or they contributed
Speaker 1: to people dying. The ones I've talked about now up
Speaker 1: to now rather have cost companies millions of dollars, but
Speaker 1: one's life was truly threatened. Unfortunately, that's not the case
Speaker 1: with all software bugs. Now, a couple of bugs had
Speaker 1: the potential to kill millions of people. One of those
Speaker 1: happened in night a famous, famous bug, or at least
Speaker 1: a faulty circuit, and that was a faulty circuit in
Speaker 1: nora ADS computer system which caused it to mistakenly conclude
Speaker 1: the US was under nuclear attack from the Soviet Union.
Speaker 1: So displays on nora AD systems showed seemingly random attacks,
Speaker 1: and they didn't correspond with each other. So the display
Speaker 1: might show, Hey, there two missiles heading over from the
Speaker 1: Soviet Union. No, they're two hundred. No they're fifty. No,
Speaker 1: there's three, And it wasn't consistent, and command posts around
Speaker 1: the US all had conflicting information, which led leaders to
Speaker 1: conclude the whole thing was a regrettable computer error, and
Speaker 1: they were right to do so. To be fair, they
Speaker 1: were kind of prepared for this because there was another
Speaker 1: incident that had actually happened in nineteen seventy nine that
Speaker 1: was way scarier, and in that case, someone mistakenly inserted
Speaker 1: a training scenario into the computer system that made it
Speaker 1: seem like the Soviet Union had launched an all out
Speaker 1: nuclear attack on the US. But that wasn't a bug.
Speaker 1: That was a mistake on the part of a human
Speaker 1: who had accidentally uploaded the wrong or rather executed the
Speaker 1: wrong command. It didn't have something to do with a
Speaker 1: flaw in the computer system itself, however, because that thing
Speaker 1: happened and everybody was freaked out and then was able
Speaker 1: to determine that, in fact it was a false alarm,
Speaker 1: it meant that calmer heads could prevail in the nineteen
Speaker 1: eighty incident, so the Soviets also had a close call
Speaker 1: just a few years later. It was a bug in
Speaker 1: the early warning detection software that the USSR was using
Speaker 1: in the early eighties, and on September twenty three, nine
Speaker 1: eight three and so Union received an alert that the
Speaker 1: US had launched a nuclear attack in the form of
Speaker 1: five nuclear war It's uh, technically two different attacks. The
Speaker 1: first would have been a single nuclear warhead and the
Speaker 1: second was four nuclear warheads. And this was during a
Speaker 1: particularly stressful period in the history of both countries and
Speaker 1: their relationship with each other, at the height of the
Speaker 1: Cold War nine three now fortunately, UH Soviet Air Defense
Speaker 1: Forces Lieutenant Colonel Stanislav Petrov suspected that this report was
Speaker 1: an error and that there was some sort of bug
Speaker 1: in the software or a mistake in the reporting system
Speaker 1: that caused this, he gave a command to hold off
Speaker 1: on any sort of retaliatory strike, which would have initiated
Speaker 1: a full scale nuclear war had it happened. Petrov was
Speaker 1: the officer in charge of a bunker that served as
Speaker 1: the command center for this early warning system, and he
Speaker 1: he had said afterward that his reckoning was any real
Speaker 1: attack would consist of hundreds of warheads, not five. No
Speaker 1: one would start an attack with just five warheads, so
Speaker 1: it was more likely to be an error than a
Speaker 1: genuine attack. So he gave the command to wait until
Speaker 1: the reported missiles would pass into the range of radar,
Speaker 1: which only extended as far as the horizon, so if
Speaker 1: it had in fact been a real attack, it would
Speaker 1: have potentially limited the Soviet Union's ability to respond. But
Speaker 1: no missile showed up, and he was vindicated in his decision. Now,
Speaker 1: the cause of the false alarm in this case was
Speaker 1: a combination of factors that the designers didn't anticipate, uh
Speaker 1: which largely consisted of sunlight hitting high altitude clouds at
Speaker 1: a particular angle from a particular perspective of the satellites,
Speaker 1: so the satellites misidentified that reflection as a warhead. Now,
Speaker 1: they were. The subvieys were able to address this error
Speaker 1: in the future by adding another step in which these
Speaker 1: satellites would cross referenced data from other geostationary satellites to
Speaker 1: make certain that they are identifying actual rockets as opposed
Speaker 1: to high altitude clouds. Now, there are several cases of
Speaker 1: software bugs leading to actual deaths. For example, the the
Speaker 1: RACK was such a case. Now that was a radiation
Speaker 1: therapy machine that could deliver two different modes of radiation treatments.
Speaker 1: The first was a low powered direct electron beam and
Speaker 1: the second was a mega volt X ray beam. Now,
Speaker 1: the x ray beam was far more intense and it
Speaker 1: required physicians to provide shielding to patients to limit exposure
Speaker 1: to the beam. But the therac had inherited its code
Speaker 1: from its predecessor, which had different hardware constraints. Now the
Speaker 1: new machine meant that these constraints weren't there, and it
Speaker 1: created a deadly problem if operators changed the machines mode
Speaker 1: too quickly from one to the other, it would actually
Speaker 1: send sets of instructions to the processor, one for each
Speaker 1: mode of operation, and whichever set of instructions reached the
Speaker 1: processor first. That's what the machine would switch to So
Speaker 1: let's say you've been operating the therac in the mega
Speaker 1: volt X ray mode, but now you're going to have
Speaker 1: a patient come in. You need to administer radiation therapy,
Speaker 1: so you want to switch it to low electron. Being
Speaker 1: you switch it too quickly, it sends two sets of
Speaker 1: instructions to the processor, and the one that arises the
Speaker 1: mega volt x ray instruction, so instead of switching it,
Speaker 1: you confirm to stay on the more intense, deadlier radiation.
Speaker 1: The tragic news is this did happen several times. Six
Speaker 1: patients were documented as dying from complications due to radiation
Speaker 1: poisoning from there at twenty five machines between night five
Speaker 1: and nineteen eight six, And while the machine would send
Speaker 1: error messages when these conditions were present, the documentation for
Speaker 1: the machine didn't explain what the errors meant. It didn't say, hey,
Speaker 1: if you get this error, it means that you've switched
Speaker 1: modes too quickly and you need to address this. So,
Speaker 1: since operators weren't told that this was necessarily a hazardous condition,
Speaker 1: they would just clear the error and proceed, and there
Speaker 1: were deadly results in a similar vein in Panama City, Panama,
Speaker 1: there was an incident involving a Cobalt sixty system, actually
Speaker 1: several incidents involving this Cobalt sixty system that was running
Speaker 1: therapy planning software made by a company called Multi Data
Speaker 1: Systems International. Now, the software's purpose was to calculate the
Speaker 1: amount of radiation that cancer patients should receive in radiation
Speaker 1: therapy sessions. During these radiation therapy sessions, the therapists were
Speaker 1: meant to place metal shields on the patient to protect
Speaker 1: healthy tissue from radiation damage. And the software would allow
Speaker 1: therapists to use a methodology to show where those shields
Speaker 1: were on the patient, to indicate where the shields are present.
Speaker 1: But they could only draw up to four shields, and
Speaker 1: the doctors in Panama wanted to use five shields for
Speaker 1: particular therapy sessions. They were overloaded, they had a long
Speaker 1: waiting list of patients, and they were trying to make
Speaker 1: things more efficient, and they discovered that they could kind
Speaker 1: of work around this limitation of four shields by drawing
Speaker 1: a design on the computer screen as if they were
Speaker 1: using just one large shield that has a hole in
Speaker 1: the middle of it. And so what they would do
Speaker 1: is they would arrange the five shields to essentially be
Speaker 1: in the same sort of shape with the middle of
Speaker 1: it being open so that they can have the radiation
Speaker 1: therapy passed through it. Uh, But they didn't realize that
Speaker 1: the software had a bug in it, and that bug
Speaker 1: was if you drew the whole in one direction, you
Speaker 1: get the correct dose of radiation, but if you drew
Speaker 1: it in the other direction, so like clockwise versus counterclockwise,
Speaker 1: the software would recommend a dosage twice as strong as
Speaker 1: what was needed, and the result was devastating. Eight patients
Speaker 1: died as a result of this, and another twenty received
Speaker 1: doses high enough to potentially cause health problems. Later on,
Speaker 1: the physicians were actually arrested and brought up on murder
Speaker 1: charges because they were supposed to double check all calculations
Speaker 1: by hand to ensure that they were going to give
Speaker 1: the proper dose of radiation treatment. So while the software
Speaker 1: was calculating the incorrect dose, the physicians were responsible for
Speaker 1: making sure that any dose that was calculated was in
Speaker 1: fact the correct one, and they failed to do so,
Speaker 1: or at least that was the charge. There are also
Speaker 1: bugs that involved military applications that have resulted in the
Speaker 1: loss of life. During the Persian Gulf War in Iraqi,
Speaker 1: fired scud missile hit a US base in Saudi Arabia
Speaker 1: and it killed twenty eight soldiers. Now the base had
Speaker 1: detected the missile and had launched and fired a Patriot
Speaker 1: missile in return. The purpose of the Patriot missile was
Speaker 1: to intercept and destroy incoming missiles, and the way a
Speaker 1: Patriot missile did this was to use radar pulses to
Speaker 1: guide trajectory calculations so that it would end up getting
Speaker 1: close to the incoming missile. This is harder than it
Speaker 1: sounds because both missiles are moving very very quickly, so
Speaker 1: we need a very precise information in order to adjust
Speaker 1: its trajectory properly and make sure it was on target. Now,
Speaker 1: once it gets within range, which is between five and
Speaker 1: ten meters I think uh, it would then fire out
Speaker 1: a thousand pellets from the Patriot missile at high velocity
Speaker 1: with the goal of causing the incoming warhead to explode prematurely.
Speaker 1: In this case, the Patriot missile missed and the military
Speaker 1: investigated the issue in the wake of the loss of
Speaker 1: life and found a problem with the software guiding the
Speaker 1: Patriot missile, and it was a problem that actually the
Speaker 1: military kind of knew about already. So one of the
Speaker 1: processes in the Patriots programming was to convert time into
Speaker 1: floating point operations for increased accuracy. But not all subroutines
Speaker 1: that depended on tracking time did this. Some of them
Speaker 1: remained UH clock units rather than floating point operations, which
Speaker 1: meant that they would get out of sync after a while.
Speaker 1: There'd be a disagreement in various subroutines as to what
Speaker 1: how much time had actually passed. And like I said,
Speaker 1: the military was aware of this issue and they had
Speaker 1: a work around, which was not ideal. The workaround was
Speaker 1: you would occasionally reboot the system, which would reset the
Speaker 1: clocks and synchronize them, but over time they would fall
Speaker 1: out of sync because they're not tracking time the same way.
Speaker 1: And since there was no hard and fast rule as
Speaker 1: to how frequently you'd reset the system, problems like this
Speaker 1: one where possible, and in fact, in this case it
Speaker 1: did happen. So prior to this particular incident, that specific
Speaker 1: Patriot system had been running for one hours with how
Speaker 1: to reboot, and the clock disagreement amounted to about one
Speaker 1: third of a second. Now, that seems like it's no
Speaker 1: time at all. One third of a second is so
Speaker 1: so short, But a scutt missile's top speed is about
Speaker 1: one point one miles per second or one point seven
Speaker 1: kilometers per second, which means if you take a third
Speaker 1: of a second, the missile could travel more than five
Speaker 1: And since the patriot needs to be within ten ms
Speaker 1: of a target to destroy it, that resulted in a
Speaker 1: catastrophic failure. So software bugs can be a matter of
Speaker 1: life or death. It's not all just Hey, this irritating
Speaker 1: thing meant people couldn't make long distance phone calls or uh,
Speaker 1: this issue caused my computer to start writing massive amounts
Speaker 1: of data to its hard drive. And this is why
Speaker 1: it's so important to have really qualified q A personnel
Speaker 1: go through code and make sure it's doing what it's
Speaker 1: supposed to do, because the problems that can arise can
Speaker 1: be non trivial and in fact life or death situations
Speaker 1: depending upon the application of technology. So technology is a
Speaker 1: fascinating thing. It's a wonderful thing. It has benefited us
Speaker 1: in ways that I can't even begin to describe. It's
Speaker 1: just too broad a topic, and it's something I've been
Speaker 1: tackling for, you know, eight years and I haven't haven't
Speaker 1: even gotten close to getting toward the finishing point. So
Speaker 1: I don't want to suggest that technology is bad, but
Speaker 1: we definitely have the need to check, double check, and
Speaker 1: triple check all this work to make certain things are
Speaker 1: working properly before we release them out into the wild.
Speaker 1: That particularly applies if, again, you are reusing old code
Speaker 1: or old components in a new way, because you have
Speaker 1: to make absolutely certain that there's not going to be
Speaker 1: some unintended problem that results when a new form factor
Speaker 1: is using old code. I hope you guys found this
Speaker 1: episode interesting. I plan on doing a lot of other
Speaker 1: kind of list sort of podcasts in the future, things
Speaker 1: like some of the the most successful pieces of software
Speaker 1: of all time, the most uh popular gadgets of all time,
Speaker 1: that sort of stuff. But if you have any suggestions
Speaker 1: for that kind of topic or anything else, really you
Speaker 1: should write me. My email address is tech stuff at
Speaker 1: how stuff works dot com, or you can drop me
Speaker 1: a line on Facebook for Twitter to handle it. Both
Speaker 1: of those is text stuff H s W. And I'll
Speaker 1: talk to you guys again really soon for more on
Speaker 1: this and thousands of other topics. Is it how stuff
Speaker 1: Works dot Com