RMT: Rule-based Metamorphic Testing for Autonomous Driving Models

by   Yao Deng, et al.

Deep neural network models are widely used for perception and control in autonomous driving. Recent work uses metamorphic testing but is limited to using equality-based metamorphic relations and does not provide expressiveness for defining inequality-based metamorphic relations. To encode real world traffic rules, domain experts must be able to express higher order relations e.g., a vehicle should decrease speed in certain ratio, when there is a vehicle x meters ahead and compositionality e.g., a vehicle must have a larger deceleration, when there is a vehicle ahead and when the weather is rainy and proportional compounding effect to the test outcome. We design RMT, a declarative rule-based metamorphic testing framework. It provides three components that work in concert:(1) a domain specific language that enables an expert to express higher-order, compositional metamorphic relations, (2) pluggable transformation engines built on a variety of image and graphics processing techniques, and (3) automated test generation that translates a human-written rule to a corresponding executable, metamorphic relation and synthesizes meaningful inputs.Our evaluation using three driving models shows that RMT can generate meaningful test cases on which 89 predictions are found by enabling higher-order metamorphic relations. Compositionality provides further aids for generating meaningful, synthesized inputs-3012 new images are generated by compositional rules. These detected erroneous predictions are manually examined and confirmed by six human judges as meaningful traffic rule violations. RMT is the first to expand automated testing capability for autonomous vehicles by enabling easy mapping of traffic regulations to executable metamorphic relations and to demonstrate the benefits of expressivity, customization, and pluggability.


page 1

page 6

page 8


TARGET: Traffic Rule-based Test Generation for Autonomous Driving Systems

Recent accidents involving self-driving cars call for extensive testing ...

Spatial Intelligence of a Self-driving Car and Rule-Based Decision Making

In this paper we show how rule-based decision making can be combined wit...

Efficient and Effective Generation of Test Cases for Pedestrian Detection – Search-based Software Testing of Baidu Apollo in SVL

With the growing capabilities of autonomous vehicles, there is a higher ...

Neural Network Guided Evolutionary Fuzzing for Finding Traffic Violations of Autonomous Vehicles

Self-driving cars and trucks, autonomous vehicles (AVs), should not be a...

Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing

Simulation-based testing represents an important step to ensure the reli...

LawBreaker: An Approach for Specifying Traffic Laws and Fuzzing Autonomous Vehicles

Autonomous driving systems (ADSs) must be tested thoroughly before they ...

Please sign up or login with your details

Forgot password? Click here to reset