Automated Testing for Deep Learning Systems with Differential Behavior Criteria

by   Yuan Gao, et al.

In this work, we conducted a study on building an automated testing system for deep learning systems based on differential behavior criteria. The automated testing goals were achieved by jointly optimizing two objective functions: maximizing differential behaviors from models under testing and maximizing neuron coverage. By observing differential behaviors from three pre-trained models during each testing iteration, the input image that triggered erroneous feedback was registered as a corner-case. The generated corner-cases can be used to examine the robustness of DNNs and consequently improve model accuracy. A project called DeepXplore was also used as a baseline model. After we fully implemented and optimized the baseline system, we explored its application as an augmenting training dataset with newly generated corner cases. With the GTRSB dataset, by retraining the model based on automated generated corner cases, the accuracy of three generic models increased by 259.2 capability of automated testing, we explored other approaches based on differential behavior criteria to generate photo-realistic images for deep learning systems. One approach was to apply various transformations to the seed images for the deep learning framework. The other approach was to utilize the Generative Adversarial Networks (GAN) technique, which was implemented on MNIST and Driving datasets. The style transferring capability has been observed very effective in adding additional visual effects, replacing image elements, and style-shifting (virtual image to real images). The GAN-based testing sample generation system was shown to be the next frontier for automated testing for deep learning systems.


page 1

page 3

page 4

page 5

page 7

page 8


DLFuzz: Differential Fuzzing Testing of Deep Learning Systems

Deep learning (DL) systems are increasingly applied to safety-critical d...

CAGFuzz: Coverage-Guided Adversarial Generative Fuzzing Testing of Deep Learning Systems

Deep Learning systems (DL) based on Deep Neural Networks (DNNs) are more...

RobOT: Robustness-Oriented Testing for Deep Learning Systems

Recently, there has been a significant growth of interest in applying so...

Robustness Evaluation of Stacked Generative Adversarial Networks using Metamorphic Testing

Synthesising photo-realistic images from natural language is one of the ...

FakeSpotter: A Simple Baseline for Spotting AI-Synthesized Fake Faces

In recent years, we have witnessed the unprecedented success of generati...

CatchBackdoor: Backdoor Testing by Critical Trojan Neural Path Identification via Differential Fuzzing

The success of deep neural networks (DNNs) in real-world applications ha...

Modern Machine and Deep Learning Systems as a way to achieve Man-Computer Symbiosis

Man-Computer Symbiosis (MCS) was originally envisioned by the famous com...

Please sign up or login with your details

Forgot password? Click here to reset