Security for Machine Learning-based Software Systems: a survey of threats, practices and challenges

by   Huaming Chen, et al.

The rapid development of Machine Learning (ML) has demonstrated superior performance in many areas, such as computer vision, video and speech recognition. It has now been increasingly leveraged in software systems to automate the core tasks. However, how to securely develop the machine learning-based modern software systems (MLBSS) remains a big challenge, for which the insufficient consideration will largely limit its application in safety-critical domains. One concern is that the present MLBSS development tends to be rush, and the latent vulnerabilities and privacy issues exposed to external users and attackers will be largely neglected and hard to be identified. Additionally, machine learning-based software systems exhibit different liabilities towards novel vulnerabilities at different development stages from requirement analysis to system maintenance, due to its inherent limitations from the model and data and the external adversary capabilities. In this work, we consider that security for machine learning-based software systems may arise by inherent system defects or external adversarial attacks, and the secure development practices should be taken throughout the whole lifecycle. While machine learning has become a new threat domain for existing software engineering practices, there is no such review work covering the topic. Overall, we present a holistic review regarding the security for MLBSS, which covers a systematic understanding from a structure review of three distinct aspects in terms of security threats. Moreover, it provides a thorough state-of-the-practice for MLBSS secure development. Finally, we summarise the literature for system security assurance, and motivate the future research directions with open challenges. We anticipate this work provides sufficient discussion and novel insights to incorporate system security engineering for future exploration.


page 9

page 19


A Marauder's Map of Security and Privacy in Machine Learning

There is growing recognition that machine learning (ML) exposes new secu...

Machine Learning Model Development from a Software Engineering Perspective: A Systematic Literature Review

Data scientists often develop machine learning models to solve a variety...

Towards a Robust and Trustworthy Machine Learning System Development

Machine Learning (ML) technologies have been widely adopted in many miss...

Security and Machine Learning in the Real World

Machine learning (ML) models deployed in many safety- and business-criti...

On Using Retrained and Incremental Machine Learning for Modeling Performance of Adaptable Software: An Empirical Comparison

Given the ever-increasing complexity of adaptable software systems and t...

Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning

Learning-based pattern classifiers, including deep networks, have demons...

Please sign up or login with your details

Forgot password? Click here to reset