Are we ready?When I started college, a few short decades ago, I commuted to Stevens Institute of Technology in Hoboken, N.J., just outside of New York City. The first time I took my father’s car for a solo journey to school, he asked, “Are you ready to drive the car alone? Do you think you’re mature enough to take the car into Hoboken?”
“Yeah, Dad, I’ll be OK. I’m ready,” I replied. My Dad was an expert mechanic. Both he and I knew that the car was ready, and according to my driver’s license, the state of New Jersey felt I was ready too.
The system boundary was bigger, which meant there were more interfaces and integration scenarios to consider, such as another driver losing control.Later that morning, as I crossed “Old Ironbound”--the Pulaski Skyway--towards the Holland tunnel, a driver of an oncoming southbound vehicle lost control. The driver jumped the low profile center divider, striking my vehicle and three others traveling northbound. The system was more complex than just my vehicle and myself, the driver. The system boundary was bigger, which meant there were more interfaces and integration scenarios to consider, such as another driver losing control.
Much more recently, my government customer asked if his critical system was ready to deploy. Yes, we conducted a Technology Readiness Assessment (TRA); yes, we identified the Critical Technology Elements (CTEs) of the system; yes, we performed performance reviews on the CTEs. The technology had been proven to work in its final form under expected conditions. However, I remembered the unforeseen car accident of my youth. I remembered the complex interfaces. Integration and interfaces—that’s where most things go wrong. That’s where failures happen—at the places where things come together. We had identified two CTEs. One had been used multiple times on other platforms, but never in this type of system and not with the large number of interfaces. There were too many potential failure points.
Integration and interfaces—that’s where most things go wrong. That’s where failures happen—at the places where things come together.I’d been working with colleagues at Engility to develop and advance System Readiness Assessment (SRA) as a new systems engineering methodology to serve as a key element in transforming the government’s approach to complex systems integration. As part of this methodology, new system metrics assess the readiness of systems for operation and use and help our customers to manage risk and reduce the total cost of ownership in an increasingly complex environment.
I convinced my customer to wait a few weeks. Using Engility’s SRA User Environment, we assessed and evaluated not only the readiness of each technology and component but also each interface in the system. We scored the system with a readiness metric and determined how ready we actually were—not just the two CTEs but the entire system and all of its integrations. Using the Bayesian network modeling component of the environment, we actually arrived at a level of confidence in our decision.
My customer welcomed the surety, saying, “There’s too much at stake here to risk an integration failure in the field, especially in front of our partners. We need that confidence!”
Unlike my car scenario, our system deployed with foreknowledge of performance under expected and unexpected conditions. I told him, “we’ll be OK,” and I had the data to prove it.
Recent Publications related to SRA
- York, et al., “Using Bayesian Networks to Validate Technology Readiness Assessments of Systems,” Conference on Systems Engineering Research (CSER), Redondo Beach, CA, March 23 – 25 2017.
- Austin, York, “System Readiness Assessment (SRA), A Vade Mecum,” Complex Systems Design and Management (CSD&M) Conference, Paris, France, Nov 2015.
- York, et al., “Applying Bayesian Networks to TRL Assessments – Innovation in Systems Engineering,” INCOSE International Symposium, Adelaide, Australia, July 2017.
- “NSA Systems Readiness Assessment (SRA) Engineering Handbook,” Version 1.0, 27 June 2014.
- “SRA, System Readiness Assessment, Planning for Success,” Trifold Brochure, Feb 2017.
- Austin, Homberger, York, “System Readiness – A Look Beyond TRAs,” NDIA, October 2017.
- Austin, York, “Bucketology and an Enterprise Approach to Systems Development: Meeting Your Strategic Objectives,” July 2016.
- Austin, M.F., York, D.M., “System Readiness Assessment (SRA) – An Illustrative Example,” 13th Annual Conference on Systems Engineering Research (CSER), Hoboken, N.J., 18 – 20 March 2015.
- Austin, York, et al., “A Systems Approach to the Transition of Emergent Technologies into Operational Systems – Herding the Cats, the Road to Euphoria and Planning for Success,” INCOSE International Symposium, Utrecht, Netherlands, 2008.
Posted by Dr. Don York
I serve as an integral part of a team of subject matter experts invited to work with the Government Accountability Office (GAO) to develop their government-wide Technology Readiness Assessment (TRA) Best Practice Guide, reviewing and providing the initial draft of the section on systems readiness metrics. A draft release of the GAO Best Practice Guide was published last August for a year-long period of review and comment. We are in the process of compiling and formally submitting our response and detailed comments.
In 2016, I presented an abstract and technical briefing, “System Readiness – A Look beyond TRAs” at the 2016 National Defense Industry Association’s (NDIA's) 19th Annual Systems Engineering Conference. I presented the technical paper, “Using Bayesian Networks to Validate Technology Readiness Assessments of Systems” at the 15th Annual Conference on Systems Engineering Research (CSER) in March of 2017. In July, I will present at INCOSE's International Symposium in Adelaide.