Test Metrics for Recurrent Neural Networks

by   Wei Huang, et al.

Recurrent neural networks (RNNs) have been applied to a broad range of application areas such as natural language processing, drug discovery, and video recognition. This paper develops a coverage-guided test framework, including three test metrics and a mutation-based test case generation method, for the validation of a major class of RNNs, i.e., long short-term memory networks (LSTMs). The test metrics are designed with respect to the internal structures of the LSTM layers to quantify the information of the forget gate, the one-step information change of an aggregate hidden state, and the multi-step information evolution of positive and negative aggregate hidden state, respectively. We apply the test framework to several typical LSTM applications, including a network trained on IMDB movie reviews for sentiment analysis, a network trained on MNIST dataset for image classification, and a network trained on a lipophilicity dataset for scientific machine learning. Our experimental results show that the coverage-guided testing can be used to not only extensively exploit the behaviour of the LSTM layer in order to discover the safety loopholes (such as adversarial examples) but also help understand the internal mechanism of how the LSTM layer processes data.


page 1

page 10

page 12


testRNN: Coverage-guided Testing on Recurrent Neural Networks

Recurrent neural networks (RNNs) have been widely applied to various seq...

Effective Quantization Approaches for Recurrent Neural Networks

Deep learning, and in particular Recurrent Neural Networks (RNN) have sh...

Restricted Recurrent Neural Networks

Recurrent Neural Network (RNN) and its variations such as Long Short-Ter...

Investigating gated recurrent neural networks for speech synthesis

Recently, recurrent neural networks (RNNs) as powerful sequence models h...

Response Characterization for Auditing Cell Dynamics in Long Short-term Memory Networks

In this paper, we introduce a novel method to interpret recurrent neural...

Differential Recurrent Neural Network and its Application for Human Activity Recognition

The Long Short-Term Memory (LSTM) recurrent neural network is capable of...

RNN-Test: Adversarial Testing Framework for Recurrent Neural Network Systems

While huge efforts have been investigated in the adversarial testing of ...

Please sign up or login with your details

Forgot password? Click here to reset