Abstract
Machine learning has been pervasively used in a wide range of applications due to its technical breakthroughs in recent years. It has demonstrated significant success in dealing with various complex problems, and shows capabilities close to humans or even beyond humans. However, recent studies show that machine learning models are vulnerable to various attacks, which will compromise the security of the models themselves and the application systems. Moreover, such attacks are stealthy due to the unexplained nature of the deep learning models. In this survey, we systematically analyze the security issues of machine learning, focusing on existing attacks on machine learning systems, corresponding defenses or secure learning techniques, and security evaluation methods. Instead of focusing on one stage or one type of attack, this paper covers all the aspects of machine learning security from the training phase to the test phase. First, the machine learning model in the presence of adversaries is presented, and the reasons why machine learning can be attacked are analyzed. Then, the machine learning security-related issues are classified into five categories: training set poisoning; backdoors in the training set; adversarial example attacks; model theft; recovery of sensitive training data. The threat models, attack approaches, and defense techniques are analyzed systematically. To demonstrate that these threats are real concerns in the physical world, we also reviewed the attacks in real-world conditions. Several suggestions on security evaluations of machine learning systems are also provided. Last, future directions for machine learning security are also presented.
-
Author Keywords
- Artificial intelligence security,
- poisoning attacks,
- backdoor attacks,
- adversarial examples,
- privacy-preserving machine learning
-
IEEE Keywords
- Machine learning,
- Security,
- Data models,
- Machine learning algorithms,
- Training,
- Training data,
- Prediction algorithms
Introduction
Machine learning techniques have made major breakthroughs in recent years and have been widely used in many fields such as image classification, self-driving cars, natural language processing, speech recognition, and smart healthcare. In some applications, e.g., image classification, the accuracy of machine learning even exceeds that of humans. Machine learning has also been applied in some security detection scenarios, e.g., spam filtering, malicious program detection, which enables new security features and capabilities.
However, recent studies show that machine learning models themselves face many security threats: 1) Training data poisoning can result in a decrease in model accuracy or lead to other error-generic/error-specific attack purposes; 2) A well-designed backdoor in the training data can trigger dangerous consequences of a system; 3) A carefully-crafted disturbance in the test input (adversarial examples) can make the model go wrong; 4) Model stealing attack, model inversion attack and membership inference attack can steal the model parameters or recover the sensitive training data. All of the above security threats can lead to serious consequences to machine learning systems, especially in security and safety-critical applications, such as autonomous driving, smart security, smart healthcare, etc.
In recent years, machine learning security has attracted widespread attention [1], [2]. There are a large amount of research works on the security of deep learning algorithms since Szegedy et al. [1] highlighted the threat of adversarial examples in deep learning algorithms. However, machine learning security is not a new concept [3], and earlier works can be traced back to Dalvi et al. [4] in 2004. These earlier works, e.g., [4], [5], studied the so-called adversarial machine learning on non-deep machine learning algorithms in the context of spam detection, PDF malware detection, intrusion detection and so on [3]. Most of these earlier attacks are called evasion attacks, while a few others are referred as poisoning attacks.
Motivated by these issues, in the paper, we present a comprehensive survey on the security of machine learning. To date, only a few review and survey papers have been published on machine learning privacy and security issues. In 2010, Barreno et al. [6] review earlier evasion attacks on non-deep learning algorithms, and illustrated on a spam filter. Akhtar and Mian [7] review the adversarial example attacks on deep learning in the field of computer vision. They discuss adversarial example attacks and focus on computer vision. Yuan et al. [8] present a review on adversarial examples for deep learning, in which they summarize the adversarial example generation methods and discuss the countermeasures. Riazi and Koushanfar [9] analyze the provably secure privacy-preserving deep learning techniques. They discuss privacy protection techniques in machine learning and focus on cryptographic primitives-based privacy-preserving methods. The above review works all focus on only one type of attack, mostly adversarial examples of attacks. Biggio and Roli [3] present a review on the wild patterns (also called adversarial examples) in adversarial machine learning over the past decade including the security of earlier non-deep machine learning algorithms and recent deep learning algorithms in the field of computer vision and cybersecurity. Particularly, evasion attacks and poisoning attacks are discussed, and corresponding defenses are presented [3]. Liu et al. [10] analyze security threats and defenses on machine learning. They focus on security assessment and data security. Paper not et al. [11] systematize the security and privacy issues in machine learning. Particularly, they describe the attacks with respect to three classic security attributes, i.e., confidentiality, integrity, and availability, while they discuss the defenses in terms of robustness, accountability, and privacy [11].
The differences between this survey and these few existing review/survey papers are summarized as follows:
- Instead of focusing on one stage, one type of attack, or one specific defense method, this paper systematically covers all the aspects of machine learning security. From the training phase to the test phase, all types of attacks and defenses are reviewed in a systematic way.
- The machine learning model in the presence of adversaries is presented, and the reasons why machine learning can be attacked are analyzed.
- The threats and attack models are described. Furthermore, the machine learning security issues are classified into five categories covering all the security threats of machine learning, according to the life cycle of a machine learning system, i.e., training phase and test phase. Specifically, five types of attacks are reviewed and analyzed: 1) data poisoning; 2) backdoor; 3) adversarial examples; 4) model stealing attack; 5) recovery of sensitive training data, which includes model inversion attack and membership inference attack.
- The defense techniques according to the life cycle of a machine learning system are reviewed and analyzed. Moreover, the challenges of current defense approaches are also analyzed.
- Several suggestions on security evaluations of machine learning algorithms are provided, including design-for-security, evaluating using a set of strong attacks, and evaluation metrics.
- Future research directions on machine learning security are presented, including attacks under real physical conditions; privacy-preserve machine learning techniques; intellectual property (IP) protection of DNN; remote or lightweight machine learning security techniques; systematic machine learning security evaluation method; the underlying reasons behind these attacks and defenses on machine learning.
The rest of this paper is organized as follows. The machine learning model in the presence of adversaries, and the reasons why machine learning can be attacked, are described in Section II. The threat models and attack approaches are reviewed in Section III. The defense techniques and challenges are analyzed in Section IV. The security evaluations of machine learning algorithms are discussed in Section V. Future directions on machine learning security, are presented in Section VI. We conclude this paper in Section VII.
Conclusion
Machine learning-based applications are ubiquitous, yet machine learning systems still face a variety of security threats throughout their lifecycles. Machine learning security is an active research topic and remains an open problem. This paper presents a comprehensive survey on machine learning security covers the whole lifecycle of machine learning systems with respect to five major types of attacks and their corresponding countermeasures. A general conclusion is that the threats are real, and new security threats are constantly emerging. For example, studies show that there is a transferability in adversarial examples, which means adversarial examples can generalize well between different machine learning models. It is shown that poisoning examples can also generalize well across different learning models. The transferability can be used to launch attacks in black-box scenarios effectively. Due to the unexplained nature of machine learning models, the essential reasons for these attacks, i.e., is the adversarial example a bug or an intrinsic property of the model, need to be further studied. This paper can hopefully provide comprehensive guidelines for designing secure, robust, and private machine learning systems.
About KSRA
The Kavian Scientific Research Association (KSRA) is a non-profit research organization to provide research / educational services in December 2013. The members of the community had formed a virtual group on the Viber social network. The core of the Kavian Scientific Association was formed with these members as founders. These individuals, led by Professor Siavosh Kaviani, decided to launch a scientific / research association with an emphasis on education.
KSRA research association, as a non-profit research firm, is committed to providing research services in the field of knowledge. The main beneficiaries of this association are public or private knowledge-based companies, students, researchers, researchers, professors, universities, and industrial and semi-industrial centers around the world.
Our main services Based on Education for all Spectrum people in the world. We want to make an integration between researches and educations. We believe education is the main right of Human beings. So our services should be concentrated on inclusive education.
The KSRA team partners with local under-served communities around the world to improve the access to and quality of knowledge based on education, amplify and augment learning programs where they exist, and create new opportunities for e-learning where traditional education systems are lacking or non-existent.
FULL Paper PDF file:
Machine Learning Security: Threats,Countermeasures, and EvaluationsBibliography
author
Year
2020
Title
Machine Learning Security: Threats, Countermeasures, and Evaluations,
Publish in
in IEEE Access, vol. 8, pp. 74720-74742, 2020,
Doi
10.1109/ACCESS.2020.2987435
PDF reference and original file: Click here
Somayeh Nosrati was born in 1982 in Tehran. She holds a Master's degree in artificial intelligence from Khatam University of Tehran.
Professor Siavosh Kaviani was born in 1961 in Tehran. He had a professorship. He holds a Ph.D. in Software Engineering from the QL University of Software Development Methodology and an honorary Ph.D. from the University of Chelsea.
Nasim Gazerani was born in 1983 in Arak. She holds a Master's degree in Software Engineering from UM University of Malaysia.
