TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzing

by   Augustus Odena, et al.

Machine learning models are notoriously difficult to interpret and debug. This is particularly true of neural networks. In this work, we introduce automated software testing techniques for neural networks that are well-suited to discovering errors which occur only for rare inputs. Specifically, we develop coverage-guided fuzzing (CGF) methods for neural networks. In CGF, random mutations of inputs to a neural network are guided by a coverage metric toward the goal of satisfying user-specified constraints. We describe how fast approximate nearest neighbor algorithms can provide this coverage metric. We then discuss the application of CGF to the following goals: finding numerical errors in trained neural networks, generating disagreements between neural networks and quantized versions of those networks, and surfacing undesirable behavior in character level language models. Finally, we release an open source library called TensorFuzz that implements the described techniques.


page 2

page 7

page 8


Towards Improved Testing For Deep Learning

The growing use of deep neural networks in safety-critical applications ...

diagNNose: A Library for Neural Activation Analysis

In this paper we introduce diagNNose, an open source library for analysi...

testRNN: Coverage-guided Testing on Recurrent Neural Networks

Recurrent neural networks (RNNs) have been widely applied to various seq...

Pressmatch: Automated journalist recommendation for media coverage with Nearest Neighbor search

Slating a product for release often involves pitching journalists to run...

Concolic Testing for Deep Neural Networks

Concolic testing alternates between CONCrete program execution and symbO...

Tutorials on Testing Neural Networks

Deep learning achieves remarkable performance on pattern recognition, bu...

The Spotlight: A General Method for Discovering Systematic Errors in Deep Learning Models

Supervised learning models often make systematic errors on rare subsets ...

Please sign up or login with your details

Forgot password? Click here to reset