Explaining AI for safety critical applications

Back to articles

Director Steve Leighton picks up where he left off on the subject of Artificial Intelligence (AI). Reporting on some new developments, it gets quite technical, so grab a coffee and read on!

In a prior post I outlined some of the challenges facing the application of AI techniques, particularly those using Deep Learning methods, in safety critical airport, ATM and aircraft operations. At the time I suggested that the current state of academic research in the area did not yet provide an answer to the challenges being faced, and for anyone looking to implement solutions right now that still remains the case. But there are promising developments underway that suggest in time we will have greater ability to apply AI across all areas of aviation.

The activity currently taking the lead in trying to gain greater insight into the way in which AI systems makes decisions is a DARPA initiative known as the Explainable AI programme (XAI). It is a four-year funded programme seeking to develop a suite of tools for different machine learning techniques to enhance the explainability of the decisions made as a function of the input information. Currently, the XAI programme is leading to a lot of interesting academic research in a range of areas that could lead to military and commercial applications.

Explained by interpretability or justification

The core problem for safety critical industries lies in explaining the outputs produced by an AI model. Explanation is currently being approached through the concepts of interpretability and justification. A model is considered to be interpretable if its workings can be understood by a person, either through introspection or produced explanation. For example, through the production of a simpler more understandable model that performs a reasonable mapping of inputs to outputs. A justification of an AI model's outputs seeks to explain why a particular output is a good one, but may or may not do so by explaining exactly how it was made. Examples here include techniques such as 'decomposition' or those related to Principal Component Analysis (PCA) which describe the main features in any particular decision. Different researchers are currently working on aspects of interpretability and justification for a range of AI models. Added to this, are other researchers working on post-hoc interpretability as well as on new models that can explain their decision making in real time.

Post-hoc interpretability

Many of the current techniques for post-hoc interpretability are somewhat reminiscent of brain MRI scanning (e.g. t-SNE maps shown in [1])! They focus on elements of the model that activate in the presence of certain features in the input and attempt to infer how these elements combine to produce the outputs. This sort of approach lends itself to understanding why decisions were made, but being post-hoc in nature it doesn't necessarily help us to understand how a particular model would work when exposed to new information and inputs.

Real-time decision making

A separate strand of research is concerned with models that can explain their decision making in real time. In particular, in image classification applications, there has been work on so-called secondary models, that learn to generate textual justifications for classifications of the primary model. As an example, the primary model may be attempting to determine whether an airfield stand is occupied or not, the secondary model may explain a decision that the stand is occupied with text such as "Stand occupied because an aircraft is parked, the boarding bridge is attached and cargo containers are being removed from the aircraft".

Pulling the strands together

So, what does all of this mean for the explainability of AI and our ability to deploy it in safety critical applications? First and foremost, the research whilst interesting and offering insights is still very much at a preliminary stage. The XAI programme will inevitably kick-start broader work in the area of explainability which is vital for future aviation uses, but the programme itself runs through to 2021 and will only lay the groundwork of what our industry needs. Secondly, given the diversity of research avenues and approaches it will become important in the near future for aviation to be able to feed its own needs into the academic research arena to ensure that it will answer the right questions for our industry. I wonder who will step forward to make sure this happens?

[1] Graying the black box: Understanding DQNs, arXiv:1602.02658v3 [cs.LG] 17 Feb 2016.


Steve Leighton
Tel: +44 1252 451 651

Helioshed 182low res

Call us to discuss your next project: +44 1252 451 651

Contact us