1 00:00:15,410 --> 00:00:24,530 Speaker 1: Pushkin. Nobody really wants to be deliberately bombarded with radiation, 2 00:00:25,370 --> 00:00:29,450 Speaker 1: but if you have cancer, radiation therapy might just save 3 00:00:29,530 --> 00:00:33,090 Speaker 1: your life. That must have been what a young man 4 00:00:33,290 --> 00:00:37,170 Speaker 1: called Voyin Ray Cox hoped as he made visit after 5 00:00:37,290 --> 00:00:41,810 Speaker 1: visit to the East Texas Cancer Center. Just thirty three 6 00:00:41,890 --> 00:00:45,570 Speaker 1: years old, Ray, to his friends, was young to be 7 00:00:45,610 --> 00:00:49,570 Speaker 1: a cancer survivor. He had a tumor cut out of 8 00:00:49,570 --> 00:00:54,410 Speaker 1: his shoulder, and now March nineteen eighty six, he was 9 00:00:54,450 --> 00:00:58,770 Speaker 1: there for his ninth session of radiation therapy designed to 10 00:00:58,930 --> 00:01:03,890 Speaker 1: ensure that no traces of the cancer remained. Despite his 11 00:01:04,010 --> 00:01:08,650 Speaker 1: bad luck, Ray was a cheerful, resilient man. He knew 12 00:01:08,690 --> 00:01:12,490 Speaker 1: the drill, press his bare chest and stomach onto the 13 00:01:12,610 --> 00:01:17,050 Speaker 1: cold metal treatment table, chat to the operator while she 14 00:01:17,130 --> 00:01:21,290 Speaker 1: maneuvered him into position underneath the looming bulk of the 15 00:01:21,290 --> 00:01:27,850 Speaker 1: THEAK twenty five radiation therapy machine. The operator knew the 16 00:01:27,890 --> 00:01:32,610 Speaker 1: drill too. In his account of the case, the ergonomics 17 00:01:32,650 --> 00:01:38,250 Speaker 1: expert Stephen Casey calls her Mary Beth, although that's not 18 00:01:38,530 --> 00:01:44,090 Speaker 1: her real name, will do the same. Mary Beth cheerfully 19 00:01:44,130 --> 00:01:47,090 Speaker 1: caught up with Ray as she used a console control 20 00:01:47,250 --> 00:01:52,170 Speaker 1: to precisely position him under the THAK twenty five's radiation beamgun. 21 00:01:53,170 --> 00:01:56,410 Speaker 1: Then she walked down the corridor to the control room, 22 00:01:56,650 --> 00:02:01,330 Speaker 1: which was at a safe distance. Ordinarily, the control room 23 00:02:01,370 --> 00:02:04,370 Speaker 1: and the treatment room would be linked up by CCTV 24 00:02:04,490 --> 00:02:09,570 Speaker 1: and microphones, but neither the cameras nor the audio were 25 00:02:09,570 --> 00:02:14,410 Speaker 1: connected that day, and that didn't seem to matter. Normally, 26 00:02:14,450 --> 00:02:17,570 Speaker 1: they'd be useful for some reassuring chat or to give 27 00:02:17,610 --> 00:02:21,050 Speaker 1: the patient a word of instruction, and Ray, being an 28 00:02:21,090 --> 00:02:26,250 Speaker 1: old hand, didn't need any of that. Mary Beth typed 29 00:02:26,330 --> 00:02:30,490 Speaker 1: the treatment instructions into the computer a series of letters. 30 00:02:31,050 --> 00:02:35,610 Speaker 1: She pressed X to choose the mode, then straightway realized 31 00:02:35,610 --> 00:02:40,130 Speaker 1: her mistake. Ray needed the other mode. She deleted the 32 00:02:40,410 --> 00:02:44,810 Speaker 1: X and pressed E should check the instructions. They were 33 00:02:44,850 --> 00:02:49,450 Speaker 1: all correct, beam ready, the computer told her. She pressed 34 00:02:49,650 --> 00:02:54,530 Speaker 1: B to administer the treatment. Down the hall. On the 35 00:02:54,610 --> 00:02:59,130 Speaker 1: treatment table, Ray Cox heard a sizzling sound and saw 36 00:02:59,170 --> 00:03:07,090 Speaker 1: a blue flash and then agony. It was like someone 37 00:03:07,130 --> 00:03:10,650 Speaker 1: had thrust a hot skewer through his sh shoulder. This 38 00:03:10,810 --> 00:03:13,410 Speaker 1: wasn't right. He knew it couldn't be right. The last 39 00:03:13,490 --> 00:03:17,810 Speaker 1: eight treatments had been nothing like this. Back in the 40 00:03:17,850 --> 00:03:22,330 Speaker 1: control room, Mary Beth couldn't hear Ray's cry of pain, 41 00:03:23,330 --> 00:03:26,730 Speaker 1: and she couldn't see his body contorting on the treatment table. 42 00:03:27,530 --> 00:03:33,170 Speaker 1: All she saw was a bland little notification malfunction fifty four. 43 00:03:34,970 --> 00:03:38,330 Speaker 1: It wasn't clear what that meant. The machine would often 44 00:03:38,410 --> 00:03:44,450 Speaker 1: pause and produce an unexplained error code, sometimes thirty forty 45 00:03:44,650 --> 00:03:49,730 Speaker 1: fifty times a day. As one operator later commented, I 46 00:03:49,730 --> 00:03:52,370 Speaker 1: can't remember all the reasons it would stop, but there 47 00:03:52,370 --> 00:03:56,890 Speaker 1: were a lot of them. The machine indicated that Ray 48 00:03:57,050 --> 00:04:01,130 Speaker 1: had received only a tiny fraction of the intended dose, 49 00:04:02,170 --> 00:04:04,970 Speaker 1: and Mary Beth had been assured that the Therak twenty 50 00:04:05,010 --> 00:04:09,650 Speaker 1: five had so many safeguards it was almost impossible to 51 00:04:09,770 --> 00:04:14,250 Speaker 1: overdose a patient. It only took a single keypress for 52 00:04:14,290 --> 00:04:19,450 Speaker 1: her to reset the machine and try again. I'm Tim Harford, 53 00:04:19,850 --> 00:04:46,730 Speaker 1: and you're listening to cautionary tales. Mary Beth was an 54 00:04:46,770 --> 00:04:51,330 Speaker 1: experienced operator of for Ferrak twenty five. She must have 55 00:04:51,370 --> 00:04:54,450 Speaker 1: seen it crash and pop up an error message ten 56 00:04:54,730 --> 00:04:58,930 Speaker 1: thousand times or more. Although she didn't know what those 57 00:04:59,210 --> 00:05:03,770 Speaker 1: error messages meant. How could she? The machine's manual didn't 58 00:05:03,770 --> 00:05:08,810 Speaker 1: explain them, it didn't even mention them. Because she'd been 59 00:05:08,850 --> 00:05:12,410 Speaker 1: in that position so many times. It took her mere 60 00:05:12,650 --> 00:05:16,210 Speaker 1: seconds to reset the machine for another try at giving 61 00:05:16,330 --> 00:05:20,890 Speaker 1: Ray his treatment. That wasn't enough time for Ray to 62 00:05:20,930 --> 00:05:24,850 Speaker 1: get off the table. He'd rolled onto his side, but 63 00:05:24,930 --> 00:05:28,130 Speaker 1: mary Beth, of course couldn't see that, and she couldn't 64 00:05:28,170 --> 00:05:33,810 Speaker 1: hear his agonized yelling. The machine fired again, another flash 65 00:05:33,810 --> 00:05:38,770 Speaker 1: of blue light, Another sizzled, and this time a hot 66 00:05:38,810 --> 00:05:42,530 Speaker 1: skewer went through Ray's neck. He was in too much 67 00:05:42,610 --> 00:05:48,410 Speaker 1: pain even to scream. Then the agony started to fade. 68 00:05:49,010 --> 00:05:51,690 Speaker 1: He gulped in some air and blew it out again. 69 00:05:52,210 --> 00:05:56,970 Speaker 1: Tried to calm himself, Hey, are you pushing the long bottom? 70 00:05:58,530 --> 00:06:09,290 Speaker 1: But mary Beth couldn't hear him. Something had gone wrong 71 00:06:09,410 --> 00:06:16,650 Speaker 1: with the Tharak twenty five. But what loyal listeners to 72 00:06:16,810 --> 00:06:20,290 Speaker 1: cautionary tales will be familiar with the Swiss cheese model 73 00:06:20,290 --> 00:06:25,730 Speaker 1: of accidents, made famous by the psychologist James Reason. Imagine 74 00:06:25,850 --> 00:06:29,850 Speaker 1: slices of Swiss cheese with those distinctive holes in them. 75 00:06:30,250 --> 00:06:34,170 Speaker 1: Each slice represents some kind of safeguard against an accident. 76 00:06:34,730 --> 00:06:37,490 Speaker 1: Maybe it's a fail safe in the hardware of a system, 77 00:06:37,850 --> 00:06:40,650 Speaker 1: so it simply won't work if the right pieces aren't 78 00:06:40,690 --> 00:06:45,370 Speaker 1: in position. Maybe it's a subroutine in the software, monitoring 79 00:06:45,410 --> 00:06:48,610 Speaker 1: what the system's doing and shutting it down if another 80 00:06:48,690 --> 00:06:53,810 Speaker 1: part of the software happens to glitch. But no safeguard 81 00:06:53,930 --> 00:06:57,850 Speaker 1: is perfect. Every slice of cheese has holes in it. 82 00:06:58,690 --> 00:07:03,290 Speaker 1: In James Reasons model, an accident becomes possible when all 83 00:07:03,370 --> 00:07:06,930 Speaker 1: the holes line up. That means that every safeguard in 84 00:07:06,970 --> 00:07:12,370 Speaker 1: the system becomes simultananeously vulnerable to the same kind of problem. 85 00:07:12,930 --> 00:07:16,970 Speaker 1: To prevent accidents, then get extra lines of defense and 86 00:07:17,010 --> 00:07:20,370 Speaker 1: try to strengthen the defenses you already have. To put 87 00:07:20,410 --> 00:07:24,050 Speaker 1: it another way, get more slices of cheese with fewer 88 00:07:24,050 --> 00:07:29,250 Speaker 1: holes in them. So did the THEAC twenty five need 89 00:07:29,370 --> 00:07:33,370 Speaker 1: better hardware or better software to prevent the accident that 90 00:07:33,490 --> 00:07:37,970 Speaker 1: happened to ray Cox? As it turns out, yes, but 91 00:07:38,050 --> 00:07:41,770 Speaker 1: that's also the wrong question. We should be looking at 92 00:07:41,810 --> 00:07:49,850 Speaker 1: a different kind of cheese slice. Altogether. Nine months before 93 00:07:50,130 --> 00:07:56,450 Speaker 1: ray Cox's excruciating experience, in July nineteen eighty five, forty 94 00:07:56,530 --> 00:08:00,770 Speaker 1: year old Francis Hill arrived at the Ontario Cancer Foundation 95 00:08:00,970 --> 00:08:05,450 Speaker 1: Clinic for her twenty fourth round of radiation treatment for 96 00:08:05,530 --> 00:08:10,450 Speaker 1: cervical cancer. The clinic was using a twenty five machine, 97 00:08:11,090 --> 00:08:14,250 Speaker 1: but there was a problem the machine didn't seem to 98 00:08:14,250 --> 00:08:18,370 Speaker 1: be working. Every time the operator tried to fire the 99 00:08:18,490 --> 00:08:23,290 Speaker 1: radiation beam, the machine paused, produced an error message and 100 00:08:23,330 --> 00:08:27,370 Speaker 1: reported that no dose had been given. The operator hit 101 00:08:27,450 --> 00:08:30,770 Speaker 1: the pea key to proceed, and the same thing happened. 102 00:08:31,570 --> 00:08:35,930 Speaker 1: The operator hit p again, and it happened again. We've 103 00:08:35,970 --> 00:08:39,330 Speaker 1: all been there, clicking an icon on a screen, finding 104 00:08:39,370 --> 00:08:42,410 Speaker 1: that nothing seems to happen, and then clicking it again. 105 00:08:43,970 --> 00:08:48,850 Speaker 1: After four attempts, the operator called a technician who couldn't 106 00:08:48,850 --> 00:08:52,770 Speaker 1: find anything wrong with the Thearact twenty five. Francis Hill 107 00:08:52,970 --> 00:08:56,650 Speaker 1: left and the machine was used successfully on half a 108 00:08:56,770 --> 00:09:01,770 Speaker 1: dozen other patients that afternoon. That sort of thing wasn't 109 00:09:01,890 --> 00:09:06,730 Speaker 1: particularly strange, the operator reflected. The fair Act twenty five 110 00:09:06,770 --> 00:09:10,810 Speaker 1: would often seem to glitch like that, produce mysterious error messages, 111 00:09:11,290 --> 00:09:17,890 Speaker 1: and then suddenly working again for no particular reason. But 112 00:09:17,970 --> 00:09:22,930 Speaker 1: while glitches didn't seem strange to the operator, something seemed 113 00:09:22,970 --> 00:09:28,690 Speaker 1: strange to Francis. She could feel a kind of burning, tingling, 114 00:09:28,810 --> 00:09:32,730 Speaker 1: electric shock kind of sensation in her hip near where 115 00:09:32,810 --> 00:09:36,570 Speaker 1: the therapeutic beam had been aimed. When she came back 116 00:09:36,610 --> 00:09:40,050 Speaker 1: for another round of treatment three days later, her doctors 117 00:09:40,290 --> 00:09:44,650 Speaker 1: immediately diagnosed a radiation burn in her hip, which was 118 00:09:44,770 --> 00:09:50,050 Speaker 1: painfully swollen. They called the machines manufacturers to report a 119 00:09:50,130 --> 00:09:56,770 Speaker 1: suspected radiation overdose. The manufacturers were Atomic Energy of Canada 120 00:09:56,810 --> 00:10:05,130 Speaker 1: Limited AECL of radiation overdose. AECL had never heard of 121 00:10:05,170 --> 00:10:09,850 Speaker 1: anything like that before Strange. They sent a engineer along 122 00:10:09,890 --> 00:10:17,290 Speaker 1: to investigate. The FARACT twenty five could be used in 123 00:10:17,410 --> 00:10:22,530 Speaker 1: two different modes. The electron mode attacked cancer near the 124 00:10:22,570 --> 00:10:26,330 Speaker 1: surface of the patient's body. The machine emitted a beam 125 00:10:26,370 --> 00:10:30,930 Speaker 1: of electrons spread out by an array of magnets. The 126 00:10:31,130 --> 00:10:35,370 Speaker 1: X ray mode attacked cancer deep inside a patient's body. 127 00:10:36,090 --> 00:10:39,490 Speaker 1: The magnet array would be moved aside and replaced by 128 00:10:39,530 --> 00:10:43,290 Speaker 1: a device called a flattener, which focused the X ray 129 00:10:43,330 --> 00:10:48,370 Speaker 1: beam precisely on the cancer. The flattener absorbed a lot 130 00:10:48,370 --> 00:10:51,530 Speaker 1: of energy, which meant the X ray beam had to 131 00:10:51,530 --> 00:10:56,410 Speaker 1: be very powerful. The components which diffused the electron beams 132 00:10:56,530 --> 00:10:59,410 Speaker 1: or focused the X rays on the FAACT twenty five 133 00:10:59,810 --> 00:11:03,690 Speaker 1: were positioned on a turntable. As the machine was programmed 134 00:11:03,730 --> 00:11:08,490 Speaker 1: to fire either electrons or X rays at the patient's tumor, 135 00:11:08,490 --> 00:11:12,930 Speaker 1: the turntail would rotate automatically to fix the right component 136 00:11:13,090 --> 00:11:18,530 Speaker 1: into position. At least that was the idea. Two types 137 00:11:18,570 --> 00:11:22,290 Speaker 1: of radiation beam then one that needs diffusing, one that 138 00:11:22,330 --> 00:11:27,130 Speaker 1: needs focusing. If this sounds like an accident waiting to happen, 139 00:11:28,010 --> 00:11:32,690 Speaker 1: well it was. But machines like the THEAK twenty five 140 00:11:32,770 --> 00:11:37,330 Speaker 1: are expensive, and this dual purpose design meant that hospitals 141 00:11:37,370 --> 00:11:40,130 Speaker 1: got more bang for the buck. As long as the 142 00:11:40,210 --> 00:11:43,330 Speaker 1: right component was in place for the right beam, there 143 00:11:43,370 --> 00:11:49,010 Speaker 1: would be no problem. The THEAK twenty five was fully 144 00:11:49,050 --> 00:11:53,690 Speaker 1: controlled by a computer, unremarkable these days, but radical for 145 00:11:53,730 --> 00:11:58,290 Speaker 1: the mid nineteen eighties. Its predecessors, the THEAK six and 146 00:11:58,370 --> 00:12:02,970 Speaker 1: the THEAK twenty allowed a human operator to physically position 147 00:12:03,090 --> 00:12:07,090 Speaker 1: the magnets or the flattener. On the THEAK twenty five, 148 00:12:07,530 --> 00:12:12,930 Speaker 1: this manual position was replaced by servo motors, computer controlled 149 00:12:13,090 --> 00:12:19,330 Speaker 1: to quickly and precisely put everything in position. When AECL 150 00:12:19,530 --> 00:12:24,130 Speaker 1: investigated the incident with Francis Hill, they weren't actually able 151 00:12:24,250 --> 00:12:28,450 Speaker 1: to reproduce the error, but they suspected that the turntable 152 00:12:28,530 --> 00:12:33,610 Speaker 1: system hadn't worked properly. The turntable had three tiny switches 153 00:12:33,690 --> 00:12:36,650 Speaker 1: designed to measure when it was in position, but it 154 00:12:36,730 --> 00:12:40,850 Speaker 1: emerged that a single bit of error the computer glitching 155 00:12:40,890 --> 00:12:43,810 Speaker 1: and mistaking a zero for a one could produce a 156 00:12:43,930 --> 00:12:49,490 Speaker 1: faulty reading of the turntable's position. So AECL told the 157 00:12:49,530 --> 00:12:53,410 Speaker 1: clinics that used the THEAAK twenty five to visually confirm 158 00:12:53,770 --> 00:12:58,730 Speaker 1: before each procedure that the turntable was in the correct position, 159 00:12:59,530 --> 00:13:03,530 Speaker 1: just as a precaution until further notice. They tightened up 160 00:13:03,530 --> 00:13:06,410 Speaker 1: the software, making it more robust to a small error 161 00:13:06,450 --> 00:13:09,090 Speaker 1: like that. Then they got back in touch with the clinics. 162 00:13:09,810 --> 00:13:13,370 Speaker 1: No need for those visual checks anymore. We've just made 163 00:13:13,370 --> 00:13:18,970 Speaker 1: the machine five orders of magnitude safer, which in plain 164 00:13:18,970 --> 00:13:22,930 Speaker 1: English means it's about one hundred thousand times safer than before. 165 00:13:23,370 --> 00:13:27,970 Speaker 1: And it was safe already. But one thing AECL don't 166 00:13:28,010 --> 00:13:31,410 Speaker 1: seem to have done is to have notified the clinics 167 00:13:31,490 --> 00:13:34,570 Speaker 1: that an accident had happened and that a patient had 168 00:13:34,610 --> 00:13:38,330 Speaker 1: been injured by the machine. The clinics, at least say 169 00:13:38,370 --> 00:13:43,090 Speaker 1: they weren't told of any injuries. When AECL announced that 170 00:13:43,090 --> 00:13:46,970 Speaker 1: they'd fixed the problem, it was September nineteen eighty five. 171 00:13:47,890 --> 00:13:51,610 Speaker 1: A month later, they were sued by a woman named 172 00:13:51,770 --> 00:13:58,050 Speaker 1: Katie Yarborough. AECL had never heard of Katie Yarborough, who 173 00:13:58,130 --> 00:14:04,250 Speaker 1: was Katie Yarborough. Cautionary tales will return after the break. 174 00:14:13,050 --> 00:14:19,650 Speaker 1: Katie Yarborough's injury happened seven weeks before Francis Hills, early 175 00:14:19,690 --> 00:14:23,210 Speaker 1: in the summer of nineteen eighty five. She was being 176 00:14:23,250 --> 00:14:28,290 Speaker 1: treated at the Kenniston Oncology Center in Marietta, Georgia. Katie 177 00:14:28,330 --> 00:14:31,810 Speaker 1: was sixty one. She'd had a malignant tumor removed from 178 00:14:31,810 --> 00:14:35,530 Speaker 1: her breast, and now she needed follow up treatment to 179 00:14:35,610 --> 00:14:38,810 Speaker 1: destroy any secondary tumors which might have spread to the 180 00:14:38,890 --> 00:14:43,450 Speaker 1: lymph nodes under her collar bone. That treatment, of course, 181 00:14:43,970 --> 00:14:48,250 Speaker 1: would be provided by a Therak twenty five machine. But 182 00:14:48,410 --> 00:14:53,410 Speaker 1: when the technician fired up the Therak twenty five, Katie 183 00:14:53,410 --> 00:14:58,010 Speaker 1: felt an agonizing pain, a tremendous force of heat. This 184 00:14:58,410 --> 00:15:03,610 Speaker 1: red hot sensation. You burned me, Katie, exclaimed the technician, 185 00:15:04,050 --> 00:15:07,970 Speaker 1: who was puzzled. That shouldn't be possible. The technician said 186 00:15:08,570 --> 00:15:12,650 Speaker 1: the twenty five was safe, and there wasn't any sign 187 00:15:12,690 --> 00:15:16,450 Speaker 1: of a burn. Perhaps Katie's clavicle was a little warm 188 00:15:16,570 --> 00:15:21,210 Speaker 1: to the touch, but otherwise nothing seemed to be a miss. 189 00:15:21,770 --> 00:15:24,850 Speaker 1: The physicist at the Keniston Center was a man called 190 00:15:25,010 --> 00:15:29,170 Speaker 1: Tim Still. When he was informed, he was just as puzzled. 191 00:15:30,490 --> 00:15:34,170 Speaker 1: Tim knew about the two treatment modes, the powerful X 192 00:15:34,250 --> 00:15:37,770 Speaker 1: ray fire through the flattener and the gentler electron beam 193 00:15:38,210 --> 00:15:41,930 Speaker 1: fired through the magnet array. Katie Yarborough had been treated 194 00:15:41,930 --> 00:15:45,570 Speaker 1: in the electron mode, but Tim still wondered if something 195 00:15:45,610 --> 00:15:49,170 Speaker 1: had gone wrong with the array. He called the manufacturers 196 00:15:49,290 --> 00:15:53,290 Speaker 1: AECL with a question, was there any way the electron 197 00:15:53,370 --> 00:15:57,370 Speaker 1: beam could be fired directly at a patient without the 198 00:15:57,410 --> 00:16:04,170 Speaker 1: magnetarray in position? After three days, AECL replied no. They 199 00:16:04,210 --> 00:16:10,010 Speaker 1: explained that was simply impossible. But somewhere along the line 200 00:16:10,410 --> 00:16:13,850 Speaker 1: AECL didn't seem to get the message that a patient 201 00:16:13,890 --> 00:16:18,610 Speaker 1: had been injured. Maybe Tim still didn't tell them, or 202 00:16:18,610 --> 00:16:21,170 Speaker 1: maybe he did but the message didn't get through to 203 00:16:21,210 --> 00:16:25,290 Speaker 1: the key decision makers. It's quite possible that still didn't 204 00:16:25,290 --> 00:16:30,690 Speaker 1: even realize that Katie Yarborough was injured. After all. At 205 00:16:30,690 --> 00:16:35,930 Speaker 1: first she seemed fine, But Katie Yarborough wasn't recovering from 206 00:16:35,930 --> 00:16:40,610 Speaker 1: that mysterious burn. In fact, her symptoms were getting worse. 207 00:16:41,370 --> 00:16:45,250 Speaker 1: The skin above her left breast had reddened, her shoulder 208 00:16:45,250 --> 00:16:50,650 Speaker 1: would freeze up, and she suffered excruciating spasms. Her doctors 209 00:16:50,810 --> 00:16:55,690 Speaker 1: were baffled. They continued sending her for theac treatment. After all, 210 00:16:55,730 --> 00:16:58,490 Speaker 1: with malignant breast cancer, you can't afford to take risks. 211 00:16:59,170 --> 00:17:03,810 Speaker 1: But when her skin started to fall off, Katie refused 212 00:17:04,010 --> 00:17:08,290 Speaker 1: to continue lying on the treatment table underneath that machine. 213 00:17:09,570 --> 00:17:16,210 Speaker 1: And when Tim Still later examined Katie, he noticed something strange. 214 00:17:16,450 --> 00:17:19,330 Speaker 1: Not only did Katie seem to have a severe burn 215 00:17:19,490 --> 00:17:23,370 Speaker 1: to her upper chest, but her upper back we was 216 00:17:23,410 --> 00:17:27,370 Speaker 1: starting to Redden two. It was as though whatever had 217 00:17:27,370 --> 00:17:32,810 Speaker 1: burned her had passed right through her body and caused 218 00:17:32,890 --> 00:17:41,490 Speaker 1: an exit wound. When Francis Hill was injured, her clinicians 219 00:17:42,090 --> 00:17:45,770 Speaker 1: didn't know about the injury to Katie Yarborough, and as 220 00:17:45,810 --> 00:17:50,570 Speaker 1: hospitals across North America continued to use the THERAK twenty five, 221 00:17:51,370 --> 00:17:55,090 Speaker 1: they didn't know about Francis Hill or Katie Yarborough. They 222 00:17:55,210 --> 00:17:58,490 Speaker 1: only knew that the machine was already safe and had 223 00:17:58,530 --> 00:18:03,530 Speaker 1: just become one hundred thousand times safer. And that makes 224 00:18:03,570 --> 00:18:11,610 Speaker 1: what happened next almost inevitable. In December nineteen eighty five, 225 00:18:12,330 --> 00:18:16,690 Speaker 1: Dora Moss, a patient at the Yakima Valley Memorial Hospital 226 00:18:16,810 --> 00:18:21,570 Speaker 1: in Washington State, complained that her right hip seemed red 227 00:18:22,170 --> 00:18:28,690 Speaker 1: and inflamed in a distinctive striped pattern. Dora's doctors were puzzled. 228 00:18:29,410 --> 00:18:33,050 Speaker 1: It wasn't clear what could have caused the inflammation, although 229 00:18:33,130 --> 00:18:36,130 Speaker 1: possibly it was a perfectly normal reaction to the course 230 00:18:36,170 --> 00:18:40,570 Speaker 1: of radiation therapy she was having on her hip. Which 231 00:18:40,570 --> 00:18:44,610 Speaker 1: device was being used, funny you should ask. It was 232 00:18:44,650 --> 00:18:49,490 Speaker 1: a THERAC twenty five, But because the hospital staff at 233 00:18:49,570 --> 00:18:53,970 Speaker 1: Yakima weren't aware of the history of accidents, they were baffled. 234 00:18:54,970 --> 00:18:57,730 Speaker 1: Some of them wondered whether a slotted component on the 235 00:18:57,730 --> 00:19:03,690 Speaker 1: THERAK twenty five might explain the striped pattern. Others suspected 236 00:19:03,690 --> 00:19:06,530 Speaker 1: that it was a burn caused by Dora's habit of 237 00:19:06,610 --> 00:19:11,130 Speaker 1: sleeping with an electric heating pad. Maybe those heated wires 238 00:19:11,170 --> 00:19:15,890 Speaker 1: had slowly burned her skin, although on closer inspection the 239 00:19:16,050 --> 00:19:19,730 Speaker 1: arrangement of wires in the heating pad didn't actually match 240 00:19:19,810 --> 00:19:25,810 Speaker 1: the sore stripes on Dora's hip. So they contacted AECL 241 00:19:26,410 --> 00:19:27,850 Speaker 1: who responded. 242 00:19:28,170 --> 00:19:32,490 Speaker 2: After careful consideration, we are of the opinion that this 243 00:19:32,570 --> 00:19:36,170 Speaker 2: damage could not have been produced by any malfunction of 244 00:19:36,210 --> 00:19:43,370 Speaker 2: the FARAC twenty five or by any operator error, So 245 00:19:43,490 --> 00:19:43,890 Speaker 2: that was it. 246 00:19:43,930 --> 00:19:48,890 Speaker 1: Then, Officially, the cause of that stripeye burn was cause unknown, 247 00:19:50,050 --> 00:19:54,250 Speaker 1: But if the cause was unclear. The consequences were stark. 248 00:19:55,170 --> 00:19:59,210 Speaker 1: Dora Moss needed surgery and skin grafts to patch up 249 00:19:59,210 --> 00:20:04,490 Speaker 1: her ulcerated skin and treat her chronic pain. Maybe it 250 00:20:04,570 --> 00:20:08,130 Speaker 1: wasn't the FAAC twenty five. It certainly wasn't a burn 251 00:20:08,210 --> 00:20:14,690 Speaker 1: from a heating pad. The Yakima hospital staff were even 252 00:20:14,770 --> 00:20:18,370 Speaker 1: told that there'd been no other incidents with the THERAK 253 00:20:18,490 --> 00:20:23,450 Speaker 1: twenty five. Was anyone putting all these incidents together and 254 00:20:23,490 --> 00:20:28,250 Speaker 1: spotting a pattern, It seems not, although it's hard to 255 00:20:28,250 --> 00:20:33,890 Speaker 1: be sure. Nancy Levison, software safety expert and the author 256 00:20:33,970 --> 00:20:37,410 Speaker 1: of a definitive account of the affair, explains that because 257 00:20:37,450 --> 00:20:42,850 Speaker 1: there was never an official investigation, it's often unclear who 258 00:20:43,210 --> 00:20:49,650 Speaker 1: exactly knew what and when they knew it. At the 259 00:20:49,690 --> 00:20:53,970 Speaker 1: East Texas Cancer Center in March nineteen eighty six, three 260 00:20:54,010 --> 00:20:58,610 Speaker 1: months after the injury to Dora Moss, Mary Beth was puzzled. 261 00:20:59,450 --> 00:21:03,130 Speaker 1: She'd tried twice to administer the treatment to ray Cox, 262 00:21:03,890 --> 00:21:11,490 Speaker 1: apparently without success. Third time, lucky she hit again. Ray 263 00:21:11,570 --> 00:21:14,690 Speaker 1: Cox had been trying to ease himself off the table, 264 00:21:15,330 --> 00:21:18,410 Speaker 1: but when that searing skewer feeling hit him for a 265 00:21:18,530 --> 00:21:23,290 Speaker 1: third time, jabbing through his neck and shoulder. He leapt 266 00:21:23,330 --> 00:21:26,570 Speaker 1: for safety, barged open the door, and ran to the 267 00:21:26,650 --> 00:21:32,050 Speaker 1: nurses station. When Mary Beth emerged, Ray was obviously shaken 268 00:21:32,090 --> 00:21:34,850 Speaker 1: by what had happened. He told her that he felt 269 00:21:34,890 --> 00:21:41,050 Speaker 1: like had been given three separate powerful electric shocks. How strange, 270 00:21:42,050 --> 00:21:45,330 Speaker 1: Mary Beth reassured him that the machine had automatically shut 271 00:21:45,410 --> 00:21:49,810 Speaker 1: down and according to the computer's display panel, Ray had 272 00:21:49,930 --> 00:21:55,330 Speaker 1: only received one tenth of the intended dose. Mary Beth 273 00:21:55,450 --> 00:22:00,290 Speaker 1: informed Ray's doctor and the center's physicist, Fritz Hager about 274 00:22:00,290 --> 00:22:04,770 Speaker 1: the electric shocks. They came to examine the machine and Ray. 275 00:22:05,450 --> 00:22:07,690 Speaker 1: There seemed to be nothing wrong with either of them, 276 00:22:08,490 --> 00:22:13,210 Speaker 1: but that's the nature of a radiation overdose. It's invisible 277 00:22:13,850 --> 00:22:18,090 Speaker 1: and at first the injuries it causes who are invisible too. 278 00:22:18,850 --> 00:22:25,090 Speaker 1: Ray looked fine, but he really wasn't. Hagar called the 279 00:22:25,130 --> 00:22:29,650 Speaker 1: manufacturer of the machine AECL to apport the incident. Then 280 00:22:29,690 --> 00:22:32,850 Speaker 1: he ran through some tests and since everything seemed to 281 00:22:32,850 --> 00:22:35,770 Speaker 1: be in order, pronounced the machine good to go for 282 00:22:35,850 --> 00:22:39,810 Speaker 1: the afternoon patients who were waiting for treatment, and everything 283 00:22:39,850 --> 00:22:44,210 Speaker 1: went smoothly. That's what happened. Remember, the THERAK twenty five 284 00:22:44,490 --> 00:22:48,930 Speaker 1: often produced mysterious error messages. And then suddenly started to 285 00:22:48,970 --> 00:22:58,570 Speaker 1: work again for no obvious reason. Three weeks after the 286 00:22:58,610 --> 00:23:02,930 Speaker 1: strange incident with Ray Cox, Mary Beth had a new patient, 287 00:23:03,890 --> 00:23:07,810 Speaker 1: sixty six year old Vernon Kidd, who had a tumor 288 00:23:07,890 --> 00:23:12,330 Speaker 1: on his ear. As soon as the treatment beam was activated, 289 00:23:12,810 --> 00:23:18,130 Speaker 1: Vernon cried out and started moaning for help. This time, 290 00:23:18,210 --> 00:23:23,290 Speaker 1: the audio link was working. What happened, asked Mary Beth. 291 00:23:24,210 --> 00:23:24,530 Speaker 2: Fire. 292 00:23:25,250 --> 00:23:29,490 Speaker 1: He replied, fire on the side of his face. When 293 00:23:29,490 --> 00:23:33,890 Speaker 1: the physicist Fritz Hager arrived on the scene, Vernon elaborated 294 00:23:34,570 --> 00:23:38,050 Speaker 1: he'd heard a sound like frying eggs and a flash 295 00:23:38,130 --> 00:23:42,810 Speaker 1: of light and then pain. He was confused and upset. 296 00:23:43,370 --> 00:23:48,050 Speaker 1: What happened to me? We'll find out what happened to 297 00:23:48,130 --> 00:24:05,170 Speaker 1: Vernon Kidd after the break. September ninth, nineteen forty seven, 298 00:24:05,930 --> 00:24:09,530 Speaker 1: computer scientists at Harvard University get to the bottom of 299 00:24:09,570 --> 00:24:14,450 Speaker 1: why their fancy computer, the Mark two, is malfunctioning. It's 300 00:24:14,490 --> 00:24:18,370 Speaker 1: a bug, a literal bug, a moth in fact, which 301 00:24:18,410 --> 00:24:21,330 Speaker 1: has crawled into the mass of electrical relays in the 302 00:24:21,450 --> 00:24:26,170 Speaker 1: room sized computing machine and caused a short circuit. The 303 00:24:26,210 --> 00:24:30,370 Speaker 1: logbook tells the tale. Handwritten on blue gridded paper are 304 00:24:30,410 --> 00:24:37,690 Speaker 1: the words relay seventy panel f moth in relay. Next 305 00:24:37,690 --> 00:24:41,250 Speaker 1: to those words, the bug itself is preserved under a 306 00:24:41,290 --> 00:24:46,410 Speaker 1: short length of yellowing sticky tape underneath the dry remark. 307 00:24:47,610 --> 00:24:53,490 Speaker 1: First actual case of bug being found. Software bugs are 308 00:24:53,530 --> 00:24:57,450 Speaker 1: the bane of programmers, although as that punchline implies, this 309 00:24:57,610 --> 00:25:00,370 Speaker 1: wasn't the first time the word bug had been used 310 00:25:00,370 --> 00:25:03,410 Speaker 1: to describe a device malfunctioning. It was just the first 311 00:25:03,450 --> 00:25:06,730 Speaker 1: time that an insect had been obliging enough to turn 312 00:25:06,770 --> 00:25:11,890 Speaker 1: the metaphor into reality. In fact, software bugs are older 313 00:25:11,930 --> 00:25:15,930 Speaker 1: than computers. The first computer program is widely thought to 314 00:25:15,970 --> 00:25:19,850 Speaker 1: have been written by Ada Lovelace, an English mathematician and 315 00:25:19,930 --> 00:25:24,570 Speaker 1: friend of the engineer Charles Babbage. In eighteen forty three, 316 00:25:25,010 --> 00:25:29,570 Speaker 1: Lovelace published an algorithm that would enable Babbage's proto computer, 317 00:25:29,890 --> 00:25:34,490 Speaker 1: the Analytical Engine, to calculate a particular sequence of numbers. 318 00:25:35,250 --> 00:25:39,930 Speaker 1: What's so striking about Lovelace's algorithm is that the Analytical 319 00:25:39,970 --> 00:25:43,970 Speaker 1: Engine had not been built, nor would it ever be. 320 00:25:44,650 --> 00:25:48,730 Speaker 1: Babbage's designs were just too ambitious, and nobody would be 321 00:25:48,770 --> 00:25:52,930 Speaker 1: able to construct a general purpose computer for another century. 322 00:25:53,930 --> 00:25:59,090 Speaker 1: But modern analysis concludes that if Lovelace's program ever had 323 00:25:59,130 --> 00:26:03,090 Speaker 1: been run on an analytical engine. It wouldn't have worked, 324 00:26:03,570 --> 00:26:07,970 Speaker 1: not first time anyway, because there was a typo. Not 325 00:26:08,050 --> 00:26:12,290 Speaker 1: only had Lovelace published the first software she had also 326 00:26:12,490 --> 00:26:23,250 Speaker 1: published the first software bug. Anyone who's had the experience 327 00:26:23,330 --> 00:26:26,730 Speaker 1: of writing computer code, or even for gen x as 328 00:26:26,890 --> 00:26:30,610 Speaker 1: like me, of typing in a computer program printed in 329 00:26:30,650 --> 00:26:33,570 Speaker 1: the pages of a magazine will know what it's like 330 00:26:33,690 --> 00:26:36,970 Speaker 1: to have a bug. In some cases, the bugs are 331 00:26:37,090 --> 00:26:40,730 Speaker 1: easily fixed. You run the program and it doesn't work. 332 00:26:41,370 --> 00:26:44,130 Speaker 1: Maybe the computer even tells you where things went off 333 00:26:44,170 --> 00:26:49,570 Speaker 1: the rails, exactly where the error was, or maybe not, 334 00:26:50,530 --> 00:26:54,410 Speaker 1: because some bugs are more like an unsuspected hole in 335 00:26:54,490 --> 00:26:59,330 Speaker 1: one of James Reasons slices of cheese. They sit there, 336 00:26:59,650 --> 00:27:04,090 Speaker 1: hidden by other slices, causing no trouble until the holes 337 00:27:04,170 --> 00:27:10,530 Speaker 1: and the slices aligne and the computer crashes. You may 338 00:27:10,570 --> 00:27:13,490 Speaker 1: not know why it crashed. You may be able to 339 00:27:13,570 --> 00:27:16,490 Speaker 1: restart the program and find it runs with no trouble, 340 00:27:17,130 --> 00:27:22,170 Speaker 1: but somewhere under the surface, the bug is still there. 341 00:27:27,130 --> 00:27:32,050 Speaker 1: Two of Fritz Hager's patients had apparently been electrocuted, and 342 00:27:32,250 --> 00:27:36,130 Speaker 1: while the Farak twenty five seemed to be working perfectly well, 343 00:27:36,570 --> 00:27:40,490 Speaker 1: he wasn't going to risk a third. He shut the 344 00:27:40,530 --> 00:27:45,610 Speaker 1: machine down, notified AECL, and then tried to figure out 345 00:27:45,890 --> 00:27:52,010 Speaker 1: what had occurred. It wasn't easy. After each incident, the 346 00:27:52,050 --> 00:27:56,610 Speaker 1: machine seemed to be working just fine. Hager and mary 347 00:27:56,610 --> 00:28:00,690 Speaker 1: Beth worked together trying to figure out what had triggered 348 00:28:00,730 --> 00:28:09,050 Speaker 1: the malfunction fifty four message. Eventually they succeeded. Mary Beth 349 00:28:09,290 --> 00:28:12,930 Speaker 1: recalled how she had originally typed X to use the 350 00:28:12,930 --> 00:28:17,330 Speaker 1: THURAC twenty five in X ray mode, then realized she 351 00:28:17,410 --> 00:28:21,090 Speaker 1: should have typed E. Quickly using the cursor keys to 352 00:28:21,170 --> 00:28:24,450 Speaker 1: move to the correct box. She corrected the entry to E, 353 00:28:25,010 --> 00:28:29,170 Speaker 1: repeatedly hit return to accept all the other treatment variables 354 00:28:29,170 --> 00:28:35,290 Speaker 1: which were correct, and awaited the message beam ready. What 355 00:28:35,330 --> 00:28:39,970 Speaker 1: she hadn't known, what nobody had known, is that a 356 00:28:40,050 --> 00:28:45,570 Speaker 1: quick edit like that confused the computer's subroutines, which checked 357 00:28:45,610 --> 00:28:50,650 Speaker 1: the setup only at certain moments. The result, the beam 358 00:28:50,850 --> 00:28:54,850 Speaker 1: was set to full X ray power, but the flattener 359 00:28:55,010 --> 00:28:58,810 Speaker 1: that would absorb most of the radiation wasn't in position. 360 00:29:00,690 --> 00:29:05,210 Speaker 1: To make matters even worse, the Thorax software was confused 361 00:29:05,250 --> 00:29:10,130 Speaker 1: by this dangerous setup and didn't correctly monitor the dose administered. 362 00:29:11,330 --> 00:29:14,850 Speaker 1: It was a particular sequence of keystrokes that caused the 363 00:29:14,890 --> 00:29:21,370 Speaker 1: problem an unlikely sequence, but not an inconceivable one. It 364 00:29:21,410 --> 00:29:26,170 Speaker 1: shouldn't be particularly surprising that experienced operators such as Mary 365 00:29:26,210 --> 00:29:30,050 Speaker 1: Beth might type the wrong mode, notice the error, then 366 00:29:30,290 --> 00:29:33,970 Speaker 1: swiftly correct it. It was the swiftness of that correction 367 00:29:34,450 --> 00:29:42,290 Speaker 1: that bewildered the software. Fritz Hager explained to AECL that 368 00:29:42,370 --> 00:29:47,450 Speaker 1: he could now reproduce the error on demand. The AECL 369 00:29:47,490 --> 00:29:53,330 Speaker 1: engineer couldn't until Hager explained that all the keystrokes had 370 00:29:53,370 --> 00:29:59,370 Speaker 1: to be entered in less than eight seconds. The next day, 371 00:29:59,970 --> 00:30:06,130 Speaker 1: the AECL engineer called back, Yes, he could now replicate 372 00:30:06,170 --> 00:30:12,090 Speaker 1: the error, and he had bad news. If the beam 373 00:30:12,290 --> 00:30:16,570 Speaker 1: was fired in such conditions, the patient would receive a 374 00:30:16,610 --> 00:30:21,290 Speaker 1: dose of twenty five thousand rads, more than one hundred 375 00:30:21,330 --> 00:30:32,650 Speaker 1: times more than intended, which potentially could be fatal. Over 376 00:30:32,690 --> 00:30:36,530 Speaker 1: the course of three weeks, sixty six year old Vernon 377 00:30:36,650 --> 00:30:43,530 Speaker 1: Kid moved from disorientation to a coma to death. The 378 00:30:43,610 --> 00:30:47,450 Speaker 1: autopsy revealed that the section of his brain running from 379 00:30:47,570 --> 00:30:51,490 Speaker 1: under his right temple to behind his right ear had 380 00:30:51,530 --> 00:31:05,530 Speaker 1: been withered by a high dose of radiation. It's natural 381 00:31:05,610 --> 00:31:08,810 Speaker 1: to describe the problem with a Tharak twenty five as 382 00:31:08,850 --> 00:31:13,690 Speaker 1: a software bug, and while that's true, it doesn't really 383 00:31:13,770 --> 00:31:18,170 Speaker 1: help us understand the problem or prevent similar problems in future. 384 00:31:18,810 --> 00:31:23,650 Speaker 1: As Nancy Levison writes, virtually or complex software can be 385 00:31:23,730 --> 00:31:27,770 Speaker 1: made to behave in an unexpected fashion under some conditions. 386 00:31:29,210 --> 00:31:33,410 Speaker 1: Demanding software with no bugs is like demanding a slice 387 00:31:33,450 --> 00:31:37,690 Speaker 1: of Swiss cheese with no holes. It's in the nature 388 00:31:37,810 --> 00:31:41,250 Speaker 1: of Swiss cheese that there will always be holes, and 389 00:31:41,570 --> 00:31:44,890 Speaker 1: it's in the nature of complex software that there will 390 00:31:44,930 --> 00:31:49,810 Speaker 1: always be bugs. The question is what happens when a 391 00:31:49,810 --> 00:31:54,250 Speaker 1: bug appears. Perhaps another part of the software is able 392 00:31:54,290 --> 00:31:57,970 Speaker 1: to spot the problem. A separate slice of software cheese 393 00:31:58,810 --> 00:32:02,890 Speaker 1: that didn't happen With a Therac twenty five. The computer 394 00:32:02,970 --> 00:32:06,170 Speaker 1: would tell the operator what dose the patient had received. 395 00:32:06,690 --> 00:32:10,130 Speaker 1: There was no direct measurement of that dose. The bug 396 00:32:10,250 --> 00:32:13,250 Speaker 1: that led to the overdose also led to the software 397 00:32:13,570 --> 00:32:18,650 Speaker 1: failing to report the overdose, or perhaps there are failsafes 398 00:32:18,690 --> 00:32:23,090 Speaker 1: in the hardware. Again, not with the Therrak twenty five, 399 00:32:23,810 --> 00:32:28,330 Speaker 1: the machine relied on the software being perfect. The shielding 400 00:32:28,410 --> 00:32:32,050 Speaker 1: components were put in place by the software. The decision 401 00:32:32,130 --> 00:32:35,570 Speaker 1: to allow the electron gun to fire or not was 402 00:32:35,610 --> 00:32:39,450 Speaker 1: made by the software, and the dose the patient received 403 00:32:40,010 --> 00:32:43,690 Speaker 1: was reported by the software. If the software was wrong 404 00:32:43,730 --> 00:32:46,570 Speaker 1: about one of these things, it could easily be wrong 405 00:32:46,730 --> 00:32:54,810 Speaker 1: about the others. There is a different approach. The THEAK twenty, 406 00:32:55,290 --> 00:32:58,930 Speaker 1: the predecessor of the THERAK twenty five, was designed to 407 00:32:59,010 --> 00:33:02,450 Speaker 1: operate with or without a computer, and it was built 408 00:33:02,490 --> 00:33:06,730 Speaker 1: with mechanical interlocks. If you tried to fire the beam 409 00:33:06,970 --> 00:33:11,290 Speaker 1: without the right component in place, the machine just wouldn't 410 00:33:11,330 --> 00:33:15,770 Speaker 1: do it. It was only after the THEAK twenty five's 411 00:33:15,810 --> 00:33:20,210 Speaker 1: problems became widely known that something began to dawn on 412 00:33:20,410 --> 00:33:26,290 Speaker 1: THEA'AC twenty users. Sometimes the machine's fuse would blow after 413 00:33:26,450 --> 00:33:30,930 Speaker 1: quick edits I was annoying and a bit mysterious. The 414 00:33:30,970 --> 00:33:33,770 Speaker 1: machine would have to be switched off, the fuse replaced, 415 00:33:34,170 --> 00:33:39,370 Speaker 1: and everything restarted. The fra AC twenty software had been 416 00:33:39,370 --> 00:33:42,730 Speaker 1: built on the same codebase as the fair AC twenty five, 417 00:33:43,210 --> 00:33:46,930 Speaker 1: and on closer examination it became clear that the THEA 418 00:33:47,090 --> 00:33:51,730 Speaker 1: ACT twenty had the same software bug. Because of the 419 00:33:51,810 --> 00:33:56,450 Speaker 1: mechanical interlocks, which would physically prevent the machine from working 420 00:33:56,610 --> 00:34:00,130 Speaker 1: unless it was correctly set up, the bug was never 421 00:34:00,370 --> 00:34:04,690 Speaker 1: anything more than an annoyance. The most precious thing that 422 00:34:04,770 --> 00:34:09,090 Speaker 1: was damaged was a fuse. The Fair Act twenty five 423 00:34:09,090 --> 00:34:14,770 Speaker 1: five needed better software, and it needed better hardware. But 424 00:34:15,050 --> 00:34:19,210 Speaker 1: as I said earlier, that's also the wrong place to focus. 425 00:34:20,330 --> 00:34:23,890 Speaker 1: Another kind of cheese slice was missing. There was no 426 00:34:24,050 --> 00:34:30,890 Speaker 1: proper process for noting anomalous incidents, suspected malfunctions, or possible injuries. 427 00:34:31,650 --> 00:34:35,730 Speaker 1: Hospitals should have been reporting both the strange incidents and 428 00:34:35,890 --> 00:34:42,130 Speaker 1: the later mysterious injuries. Someone either AECL or a regulator 429 00:34:42,450 --> 00:34:47,050 Speaker 1: should have been collecting and analyzing the reports. Because that 430 00:34:47,170 --> 00:34:51,890 Speaker 1: didn't happen, every new incident caused a new wave of 431 00:34:52,090 --> 00:35:00,610 Speaker 1: muddle and bewilderment. Of course, we should try hard to 432 00:35:00,650 --> 00:35:04,890 Speaker 1: eliminate bugs, especially when lives are at stake. But the 433 00:35:04,930 --> 00:35:07,930 Speaker 1: real lesson here is that safety is not a function 434 00:35:08,010 --> 00:35:13,570 Speaker 1: of good software alone. It's the function of the whole system, 435 00:35:13,650 --> 00:35:17,050 Speaker 1: and the system goes beyond the software. In fact, it 436 00:35:17,130 --> 00:35:20,890 Speaker 1: goes beyond the machine. The system includes the network of 437 00:35:20,930 --> 00:35:24,850 Speaker 1: people who make the machine, use the machine, and regulate 438 00:35:24,930 --> 00:35:28,530 Speaker 1: the machine. In this case, they should have been keeping 439 00:35:28,570 --> 00:35:34,250 Speaker 1: each other closely informed. They weren't, and as a result, 440 00:35:34,810 --> 00:35:39,010 Speaker 1: it took months for anyone to assemble the pieces of 441 00:35:39,050 --> 00:35:47,330 Speaker 1: this awful puzzle Voin. Ray Cox was young and strong, 442 00:35:48,050 --> 00:35:51,130 Speaker 1: but he had taken a triple blast of high dose 443 00:35:51,250 --> 00:35:57,930 Speaker 1: radiation to his back, shoulder, and neck. Before long, he 444 00:35:58,050 --> 00:36:02,410 Speaker 1: was starting to spit up blood. The awful radiation burns 445 00:36:02,410 --> 00:36:06,730 Speaker 1: on his back and neck turned into yawning lesions as 446 00:36:06,770 --> 00:36:10,370 Speaker 1: the skin and flesh started to peel off his body. 447 00:36:11,010 --> 00:36:13,770 Speaker 1: While over the weeks that followed, the damage to his 448 00:36:13,890 --> 00:36:19,650 Speaker 1: spinal column paralyzed his left arm, both legs, and his 449 00:36:19,770 --> 00:36:24,370 Speaker 1: vocal cords, ray tried to keep his sense of humor. 450 00:36:25,370 --> 00:36:28,410 Speaker 1: Before he lost his ability to speak, he would joke 451 00:36:28,530 --> 00:36:32,810 Speaker 1: to friends and family Captain Kirk forgot to put the 452 00:36:32,850 --> 00:36:40,210 Speaker 1: machine on stun. Ray Cox died in August nineteen eighty six, 453 00:36:41,130 --> 00:36:46,410 Speaker 1: a few months after the accident, and five months after that, 454 00:36:47,770 --> 00:36:52,610 Speaker 1: Glenn Dodd, a sixty five year old cancer sufferer, was 455 00:36:52,650 --> 00:36:57,610 Speaker 1: given radiation therapy at Yakima Valley Memorial Hospital, where Dora 456 00:36:57,810 --> 00:37:03,450 Speaker 1: Moss had acquired those mysterious striped radiation burns after treatment 457 00:37:03,610 --> 00:37:08,410 Speaker 1: from a THERAK twenty five. Glen Dodd was being treated 458 00:37:09,170 --> 00:37:15,210 Speaker 1: by a THEAK twenty five, he received a fatal overdose. 459 00:37:16,930 --> 00:37:21,410 Speaker 1: Dodd was terminally ill, but doctors concluded that the injuries 460 00:37:21,450 --> 00:37:25,290 Speaker 1: he had suffered from the malfunctioning machine had killed him 461 00:37:25,410 --> 00:37:34,810 Speaker 1: before his own cancer could. Why hadn't aecl fixed the 462 00:37:34,890 --> 00:37:40,010 Speaker 1: software fault? Well? As it happens, Glen do Odd wasn't 463 00:37:40,130 --> 00:37:42,890 Speaker 1: killed by the glitch that had killed ray Cox and 464 00:37:42,970 --> 00:37:48,330 Speaker 1: Vernon Kidd, but by a different software error. You see, 465 00:37:49,090 --> 00:37:52,730 Speaker 1: there's always another bug lying in wait for the moment 466 00:37:52,810 --> 00:38:06,290 Speaker 1: to strike. This isn't the first cautionary tale we've done 467 00:38:06,530 --> 00:38:10,890 Speaker 1: about radiation overdoses. Two of my very favorite episodes of 468 00:38:10,890 --> 00:38:15,410 Speaker 1: the podcast were Glowing Peril, The magical glitter that poisoned 469 00:38:15,450 --> 00:38:20,490 Speaker 1: a City, and How the Radium Girls Fought Back, two 470 00:38:20,770 --> 00:38:24,330 Speaker 1: unforgettable episodes which came out as a pair late in 471 00:38:24,450 --> 00:38:29,530 Speaker 1: twenty twenty three. You may enjoy listening to them. I 472 00:38:29,610 --> 00:38:33,490 Speaker 1: first heard about ray Cox's case from Stephen M. Casey's 473 00:38:33,530 --> 00:38:38,650 Speaker 1: book Set Phases on stun while the definitive authority on 474 00:38:38,690 --> 00:38:43,090 Speaker 1: the THERAK twenty five case is Nancy Levison's investigation, written 475 00:38:43,250 --> 00:38:46,570 Speaker 1: with Clark Turner. For a full list of our sources, 476 00:38:46,850 --> 00:38:58,250 Speaker 1: see the show notes at Timharford dot com. Cautionary Tales 477 00:38:58,330 --> 00:39:02,290 Speaker 1: is written by me Tim Harford with Andrew Wright. The 478 00:39:02,290 --> 00:39:05,730 Speaker 1: show is produced by Alice Fines with Marilyn Rust. The 479 00:39:05,770 --> 00:39:09,010 Speaker 1: sound design and original music are the work of Pascal Wye. 480 00:39:09,770 --> 00:39:14,250 Speaker 1: Sarah Nix edited the script. Cautionary Tales features the voice 481 00:39:14,250 --> 00:39:18,890 Speaker 1: talents of Ben Crow, Melanie Gushridge, Stella Harford, Gemma Saunders 482 00:39:19,010 --> 00:39:23,090 Speaker 1: and Rufus Wright. The show wouldn't have been possible without 483 00:39:23,090 --> 00:39:27,890 Speaker 1: the work of Jacob Weisberg, Ryan Dilly, Greta Cohne, Eric Sandler, 484 00:39:28,170 --> 00:39:33,930 Speaker 1: Carrie Brody, Christina Sullivan, Kira Posey and Owen Miller. Cautionary 485 00:39:33,930 --> 00:39:37,850 Speaker 1: Tales as a production of Pushkin Industries. It's recorded at 486 00:39:37,890 --> 00:39:43,450 Speaker 1: Wardoor Studios in London by Tom Berry. If you like 487 00:39:43,530 --> 00:39:47,210 Speaker 1: the show, please remember to share, rate and review. It 488 00:39:47,210 --> 00:39:50,290 Speaker 1: doesn't really make a difference to us and if you 489 00:39:50,330 --> 00:39:53,090 Speaker 1: want to hear the show ad free, sign up to 490 00:39:53,170 --> 00:39:56,930 Speaker 1: Pushkin Plus on the show page on Apple Podcasts or 491 00:39:57,010 --> 00:40:09,250 Speaker 1: at pushkin dot Fm, slash plus, st