Safety Assessment for Autonomous Systems' Perception Capabilities

by   John Molloy, et al.

Autonomous Systems (AS) are increasingly proposed, or used, in Safety Critical (SC) applications. Many such systems make use of sophisticated sensor suites and processing to provide scene understanding which informs the AS' decision-making. The sensor processing typically makes use of Machine Learning (ML) and has to work in challenging environments, further the ML-algorithms have known limitations,e.g., the possibility of false-negatives or false-positives in object classification. The well-established safety-analysis methods developed for conventional SC systems are not well-matched to AS, ML, or the sensing systems used by AS. This paper proposes an adaptation of well-established safety-analysis methods to address the specifics of perception-systems for AS, including addressing environmental effects and the potential failure-modes of ML, and provides a rationale for choosing particular sets of guidewords, or prompts, for safety-analysis. It goes on to show how the results of the analysis can be used to inform the design and verification of the AS and illustrates the new method by presenting a partial analysis of a road vehicle. Illustrations in the paper are primarily based on optical sensing, however the paper discusses the applicability of the method to other sensing modalities and its role in a wider safety process addressing the overall capabilities of AS.


page 4

page 16

page 17

page 19

page 20

page 27

page 33

page 39


Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)

Machine Learning (ML) is now used in a range of systems with results tha...

An Analysis of ISO 26262: Using Machine Learning Safely in Automotive Software

Machine learning (ML) plays an ever-increasing role in advanced automoti...

Towards Probability-based Safety Verification of Systems with Components from Machine Learning

Machine learning (ML) has recently created many new success stories. Hen...

Oversight of Unsafe Systems via Dynamic Safety Envelopes

This paper reviews the reasons that Human-in-the-Loop is both critical f...

Towards a Framework to Manage Perceptual Uncertainty for Safe Automated Driving

Perception is a safety-critical function of autonomous vehicles and mach...

Sensors, Safety Models and A System-Level Approach to Safe and Scalable Automated Vehicles

When considering the accuracy of sensors in an automated vehicle (AV), i...

Jailbroken: How Does LLM Safety Training Fail?

Large language models trained for safety and harmlessness remain suscept...

Please sign up or login with your details

Forgot password? Click here to reset