8 Tips for Assessing Risk in Complex Software Development


8 Tips for Assessing Risk in Complex Software Development

In the blog “Keeping Pace with Human Spaceflight: An Agile Approach to Independent Verification and Validation,” I explored how Agile IV&V emerged as a means to keep pace with the complex software under development for NASA’s Orion spacecraft. By applying concepts from Agile software development to the realm of evaluation, we are seeing a lot of success in quickly assessing if things do what they are supposed to do and don’t do what they are not supposed to do.

Agile IV&V has been received very positively by the Orion IV&V customers, earning praise from the NASA IV&V Facility Director and the NASA Orion Software Architect. We have assisted our NASA customer in developing briefs so that this concept can be applied to other IV&V work.

In the best traditions of scientific collaboration, we offer some summary bullets on our findings:

  1. Keep the main thing the main thing. Assurance is our primary product, in addition to issues and risks. With complexity comes distraction… keep focused.
  2. Capability based assurance focuses on software in the context of mission capabilities, which requires adequate system understanding captured in the IV&V Technical Reference while continuing to apply the IV&V “three questions”:
    1. Does the system and its software do what it is supposed to do?
    2. Does the system and its software not do what it is not supposed to do?
    3. Does the system and its software behave appropriately under adverse conditions?
  3. Risk targeted IV&V, also known as “Follow the Risk,” considers both functional and non-functional risk categories across the entire Orion risk landscape and addresses both mitigated risk as well as residual risk.
  4. Adaptive IV&V incorporates critical thinking about what makes sense when performing analysis and what is the right thing to do. This includes leveraging existing assets such as the IV&V Technical Framework and the IV&V Catalog of Methods, but it is not constrained by them.
  5. Integration assurance integrates appropriate behaviors, especially emergent behaviors and behaviors involving interfaces within and between software and systems.
  6. Cybersecurity focuses on the threats and vulnerabilities relevant to software as the anchor or starting point for analysis, and it evaluates other system components as necessary. This includes considering cyber-threats as adverse conditions that need to be handled appropriately via preventative or adaptive behaviors.
  7. Dynamic analysis, including independent test, chooses appropriate dynamic analysis methods when that is the most efficient or practical means of generating the needed assurance evidence.
  8. Agile principles are tailored for the Orion IV&V teams use by adopting useful parts of Agile while ignoring the non-useful parts. A few examples include:
    1. Fast integrated learning cycles where the IV&V team learns quickly from their mistakes and failures as well as their successes, applying continual Plan-Do-Check-Adjust cycles for closed-loop feedback.
    2. Retrospectives identify what is working well so it can be spread to other teams and what is not working well to identify what improvements need to be made.
    3. Small self-organizing teams are empowered to innovate and to succeed or fail quickly within “safe” boundaries without negative consequences, to make their work more effective and more efficient.
    4. Daily 15-minute standups to briefly discuss what each analyst has accomplished since the last standup, what each analyst plans to accomplish by the next standup, and what if any obstacles do they need help overcoming.
Assessing Risk in Complex Software Development Orion Spacecraft - Source NASA

NASA is often seen on the leading edge of technology and computing, but by applying Agile concepts, they are on the leading edge of IV&V methodology as well. And when we consider what’s at stake (human life and multi-billion-dollar systems), IV&V is a big deal. We are excited to see how these concepts continue to mature and how they are applied to other engineering programs.

Share this Post:

Posted by John Bradbury

I have been supporting NASA Human Space Flight programs for 36 years as a contractor. I have worked on safety- and mission-critical software for the Space Shuttle, the International Space Station, and currently the Orion spacecraft as the Engility project lead for NASA’s Orion Independent Verification and Validation project. I am located at NASA’s IV&V Facility in Fairmont, WV