HW4: Reflections on software failures

Our readings this week began with a focus on several software engineering failures which resulted in devastating incidents such as plane crashes (Space Craft Accidents) and radiation overdoses (Therac-25 Accidents and 2010 Radiation Follies). My initial reaction to reading these articles was, frankly, fear of entering a career where I could unintentionally contribute to this kind of harm. However, examining these incidents in detail, it’s easy to pick out several common issues that contributed to each. By studying these, we can devise practices that support security and resilience in our engineering.

First, these problems occur in part because the process of identifying vulnerabilities, potential faults, and other bugs that may lead to software failure is not easy whatsoever. In particular, it’s nearly impossible to identify all vulnerabilities that exist within a system. Unfortunately, as mentioned in Chapter 13, Security Engineering, it’s often the people attacking a system, not the engineers, who experiment with it enough to identify the most hidden of vulnerabilities. They’ll try things outside of normal activity such as entering an absurd amount of characters into a field or toying with a vehicle’s wireless communication functions (FBI Auto Warning). To combat this, an engineer has to think like a hacker. Practically, that means incorporating sophisticated testing of several types (experience-based, penetration, tool-based, and formal) to cover all the bases. The “Swiss Cheese Model of Failure” in Chapter 14, Resilience Engineering, sums this up perfectly. Imagine a system in layers, each with holes representing vulnerabilities; when the holes line up just right, the attacker can penetrate your defense all the way to system failure. So, because you have to have some holes in your system, you should test to make sure they don’t line up in a perfect storm.

On the note of testing, another lesson found throughout these texts is that as a software engineer, you should test with the real environment of your software in mind. The Therac-25 accidents occurred in large part due to a bug that did not consider a specific “if-then” scenario. There was no “then” for “if” a user switches to electron mode while the machine is setting up for X-ray mode, leaving the machine in an unknown state which led to overdose. Perhaps, working with medical professionals who understand the high-stress environment where the machine is used could have surfaced the scenario during the testing phase. Similarly, in the 2010 Radiation Follies, failure of GE to implement a (seemingly simple) mechanism that shut down operation at an unsafe dose devastated numerous lives.

2010 Radiation Follies also serves as a perfect example for the “blame game” that ensues after accidents like these. The creator of the software that overdosed hundreds with radiation (GE), laid blame on the medical professionals for failing to notice dosing levels on their treatment screens. However, technologists claimed that the GE trainers were at fault as they never fully explained the automatic radiation dosing feature during training. Ultimately, I agree with the textbook’s author’s assessment in Chapter 14, Resilience Engineering (pg. 405). The author supports the system’s approach - that good systems are responsible for including safeguards against possible human error, especially when the safety of other humans is at stake. GE should have included a requirement that would shut down the machine’s operation at a harmful level, regardless of the automatic feature’s conclusions or the operator’s attention to detail.

The reasons why software projects fail listed in the article, adeptly named, Why Software Projects Fail sums up most of the problems that led to the disasters we studied. Particularly, it explains the FBI’s failures during the VCF and Sentinel projects, which transitioned the FBI’s paper files to a digital record storage system. Of course, this project was a huge undertaking, as it combined records from all of the FBI’s distributed centers into one digital database. Just to scratch the surface, over the course of these two projects and the four FBI articles taken together, the organization wasted millions of dollars and many years due to quick turnover of (and too many) personnel, unclear requirements, most prominently, poor organization. The VCF project had 400 change requests within a year (2002-2003) and to quote the article, “every time you write a line of code, you introduce bugs…and they had a bunch of people slinging code.” Best practice, though often impractical, is to keep your team as small as possible and your processes as centralized as possible.

• 2021 11
• 2020 22