Explaining Deep Neural Networks Using Spectrum-Based Fault Localization

by   Youcheng Sun, et al.
Queen's University Belfast
University of Oxford
University of Liverpool
King's College London

Deep neural networks (DNNs) increasingly replace traditionally developed software in a broad range of applications. However, in stark contrast to traditional software, the black-box nature of DNNs makes it impossible to understand their outputs, creating demand for "Explainable AI". Explanations of the outputs of the DNN are essential for the training process and are supporting evidence of the adequacy of the DNN. In this paper, we show that spectrum-based fault localization delivers good explanations of the outputs of DNNs. We present an algorithm and a tool PROTOZOA, which synthesizes a ranking of the parts of the inputs using several spectrum-based fault localization measures. We show that the highest-ranked parts provide explanations that are consistent with the standard definitions of explanations in the literature. Our experimental results on ImageNet show that the explanations we generate are useful visual indicators for the progress of the training of the DNN. We compare the results of PROTOZOA with SHAP and show that the explanations generated by PROTOZOA are on par or superior. We also generate adversarial examples using our explanations; the efficiency of this process can serve as a proxy metric for the quality of the explanations. Our measurements show that PROTOZOA's explanations yield a higher number of adversarial examples than those produced by SHAP.


page 6

page 7

page 8

page 9


Unsupervised Detection of Adversarial Examples with Model Explanations

Deep Neural Networks (DNNs) have shown remarkable performance in a diver...

Scalable, Axiomatic Explanations of Deep Alzheimer's Diagnosis from Heterogeneous Data

Deep Neural Networks (DNNs) have an enormous potential to learn from com...

Learning Credible Deep Neural Networks with Rationale Regularization

Recent explainability related studies have shown that state-of-the-art D...

DeepFault: Fault Localization for Deep Neural Networks

Deep Neural Networks (DNNs) are increasingly deployed in safety-critical...

Formally Explaining Neural Networks within Reactive Systems

Deep neural networks (DNNs) are increasingly being used as controllers i...

Effect of Superpixel Aggregation on Explanations in LIME – A Case Study with Biological Data

End-to-end learning with deep neural networks, such as convolutional neu...

Mutation-based Fault Localization of Deep Neural Networks

Deep neural networks (DNNs) are susceptible to bugs, just like other typ...

Code Repositories


DeepCover: Uncover the truth behind AI

view repo

Please sign up or login with your details

Forgot password? Click here to reset