JP2024508582A - Making machine learning models safe against adversarial samples with backdoor misclassification - Google Patents

Abstract

The method for making a bona fide machine learning model secure against adversarial samples involves attaching a trigger to the sample to be classified, and using the backdoored model to be backdoored using the trigger. , classifying the triggered samples. In a further step, it is determined whether the output of the backdoored model is the same as the backdoor class of the backdoored model and/or an outlier detection method is applied to the logit calculated using authentic samples. is compared with the valid logit. These steps are repeated using separate triggers and each associated backdooring model. To determine whether a sample is adversarial, the number of times the output of the backdooring model was not the same as the respective backdoor class is compared to one or more thresholds and/or an outlier detection method The difference determined by applying is compared to one or more thresholds.

Description

The present invention relates to methods, systems and computer-readable media for detecting adversarial samples, particularly for securing machine learning models and neural networks, which can be used in artificial intelligence (AI) applications.

Machine learning has become an integral part of many everyday technological systems through gradual improvement and development. Machine learning is often used as an important part of technical systems in safety-related scenarios. Therefore, the lack of robustness of such models under attack and/or duress may result in safety failures of technical systems.

In particular, in the past few decades, neural network-based image classification has gained much interest due to its versatility, small implementation requirements, and accuracy. However, neural networks are not well understood and are susceptible to attacks such as those that use adversarial samples, which are carefully crafted modifications to normal samples that are invisible to the naked eye to cause misclassification. Vulnerable to attack.

In recent years, deep learning has advanced rapidly, accelerated by big data and the rise of more readily available computing power. However, deep learning turns out to be particularly vulnerable to adversarial perturbations due to overconfidence in predictions. The machine learning community is grappling with the technical challenges of making deep learning models secure. Adversaries can often fool machine learning models by introducing carefully crafted perturbations to valid data samples. The perturbations are chosen to be as small as possible so as not to be noticed, yet large enough to change the model's originally accurate predictions. For example, in the field of image recognition, this means changing an image of a dog so that the model's accurate predictions of the dog are different from one another while keeping the modified image visually indistinguishable from the original. It would be possible to vary the animal's predictions.

There are multiple technical challenges in protecting against attacks on neural networks or machine learning models, since errors are always present in practical models, especially due to the statistical nature of machine learning. Existing proposed defenses against attacks are based on keeping model parameters secret in order to make it more difficult for an adversary to create adversarial samples. However, recent research has shown that adversarial samples created on a surrogate model (a model locally trained in a class similar to the model being attacked) transfer onto the target model with high probability (>90%). However, this property applies even if the surrogate model does not have the same internal layout as the target model (e.g. different number of layers/layer sizes) or the same accuracy (e.g. about 90% of the surrogate model It was shown that the target model was applicable (approximately 99%). A surrogate model is an emulation of the target model. A surrogate model is created by an attacker with black box access to the target model such that he can specify any choice of input x to obtain the model's prediction y=f(x). The parameters of the target model are usually kept secret, but by training a machine learning model on the input-output pair (x, f(x)), an effective surrogate model can be obtained. Moreover, research has shown that the most adversarial samples that bypass the surrogate model are ``effective'' in the sense that they also fool the target model.

Goodfellow, Ian J. et al., "Explaining and Harnessing Adversarial Examples", arXiv: 1412.6572, Conference Papers of the International Conference on Learning Representations 2015, pp. 1-11 (March 20, 2015), Kurakin, Alexey et al., "Adversarial Examples" Carlini, Nicholas et al., "Towards Evaluating the Robustness of Neural Networks", arXiv: 1608.04644 , Clinical Orthopedics and Related Research: pp. 1-19 (August 13, 2018), Tramer, Florian et al., "Ensemble Adversarial Training: Attacks and Defenses", arXiv: 1705.07204, Conference Papers of the International Conference on Learning Representations 2018, 1 -22 pages (January 30, 2018), Madry, Aleksander et al., "Towards Deep Learning Models Resistant to Adversarial Attacks", arXiv: 1706:06083, International Conference on Learning Representations 2018 Conference Papers, pages 1-28 (2017) (November 9, 2018), Dong, Yinpeng et al., "Boosting Adversarial Attacks with Momentum", arXiv: 1710.06081, CVPR2018: pp. 1-12 (March 22, 2018), Zhang, Hongyang et al., "Theoretically Principled Trade-Off between Robustness and Accuracy", arXiv: 1901:08573, Conference Papers of the International Conference on Machine Learning: pp. 1-31 (June 24, 2019), Liu, Xuanqing et al., "Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network", arXiv: 1810.01279, Clinical Orthopedics and Related Research: pp. 1-3 (May 4, 2019), Wong, Eric et al., "Fast is better than free: Revisiting adversarial training", arXiv: 2001.03994, ICLR 2020 Conference Papers, pp. 1-17 (January 12, 2020), Moosavi-Dezfooli, Seyed-Mohsen et al., "DeepFool: a simple and accurate method to fool deep neural networks", arXiv: 1511.04599, IEEE Conference on Computer Vision Wang, Yue et al., "Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems," in Proceedings of the and Pattern Recognition (CVPR) 2016, pp. 1-9 (July 4, 2016). , arXiv: 2003.07859, pp. 1-19 (June 8, 2020), and Zimmermann, Roland S., "Comment on 'Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network'", arXiv: 1907.00895 (2019) July 2, 2015) are micro-attacks (Goodfellow, Ian J. et al. and Tramer, Florian et al.) and more powerful attacks (Carlini, Nicholas et al. and Madry, Aleksander), referenced below, respectively. A variety of attacks are discussed, including (Ref. et al.). Each of the aforementioned publications is herein incorporated by reference in its entirety.

Goodfellow, Ian J. et al., "Explaining and Harnessing Adversarial Examples", arXiv: 1412.6572, International Conference on Learning Representations 2015 Conference Papers: pp. 1-11 (March 20, 2015) Kurakin, Alexey et al., "Adversarial Examples in the Physical World", arXiv: 1607.02533, Workshop at International Conference on Learning Representations 2017, pp. 1-14 (February 11, 2017) Carlini, Nicholas et al., "Towards Evaluating the Robustness of Neural Networks," arXiv: 1608.04644, Clinical Orthopedics and Related Research: pp. 1-19 (August 13, 2018) Tramer, Florian et al., "Ensemble Adversarial Training: Attacks and Defenses", arXiv: 1705.07204, International Conference on Learning Representations 2018 Conference Papers, pp. 1-22 (January 30, 2018). Madry, Aleksander et al., "Towards Deep Learning Models Resistant to Adversarial Attacks", arXiv: 1706:06083, Conference Papers of the International Conference on Learning Representations 2018, pp. 1-28 (November 9, 2017) Dong, Yinpeng et al., "Boosting Adversarial Attacks with Momentum", arXiv: 1710.06081, CVPR2018: pp. 1-12 (March 22, 2018) Zhang, Hongyang et al., "Theoretically Principled Trade-Off between Robustness and Accuracy", arXiv: 1901:08573, International Conference on Machine Learning conference papers: pp. 1-31 (June 24, 2019) Liu, Xuanqing et al., "Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network", arXiv: 1810.01279, Clinical Orthopedics and Related Research: pp. 1-3 (May 4, 2019) Wong, Eric et al., "Fast is better than free: Revisiting adversarial training", arXiv: 2001.03994, ICLR 2020 conference paper, pp. 1-17 (January 12, 2020) Moosavi-Dezfooli, Seyed-Mohsen et al., "DeepFool: a simple and accurate method to fool deep neural networks", arXiv: 1511.04599, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016, pp. 1-9 (2016) July 4th) Wang, Yue et al., "Stop-and-Go: Exploring Backdoor Attacks on Deep Reinforcement Learning-based Traffic Congestion Control Systems", arXiv: 2003.07859, pp. 1-19 (June 8, 2020) Zimmermann, Roland S., "Comment on 'Adv-BNN: Improved Adversarial Defense Through Robust Bayesian Neural Network'", arXiv: 1907.00895 (July 2, 2019)

In one embodiment, the present invention provides a method for making authentic machine learning models secure against adversarial samples. This method consists of step a) of attaching a trigger to the sample to be classified, and step b) of classifying the triggered sample using a backdoored model that is backdoored using the trigger. including. In step c), it is determined whether the output of the backdoored model in step b) is the same as the backdoor class of the backdoored model, and/or a trigger is applied to the logit from step b). An outlier detection method is applied that compares to the legitimate logit calculated using authentic samples that are additionally applied to the backdoored model. By step d) these steps a) to c) are repeated using different triggers and backdooring models respectively associated with different triggers. In step e), the number of times each of the outputs of the backdoored model was not the same as the respective backdoor class of the backdoored model is compared with a predetermined threshold to determine whether the sample is an adversarial sample. and/or the difference determined by applying an outlier detection method is compared to a difference threshold.

Embodiments of the invention are explained in more detail below on the basis of exemplary figures. The invention is not limited to the exemplary embodiments. All features described and/or illustrated herein can be used alone or in combination in various combinations in embodiments of the invention. Features and advantages of various embodiments of the invention will become apparent from the following detailed description taken in conjunction with the accompanying drawings.

1 is a schematic diagram illustrating a setup phase according to an embodiment of the invention; FIG. 1 is a schematic diagram illustrating an evaluation stage according to an embodiment of the invention; FIG. FIG. 3 is a schematic diagram illustrating improved evaluation using one backdoored model, according to an embodiment of the invention. FIG. 2 is a schematic diagram illustrating the creation of a backdooring model according to an embodiment of the invention.

Embodiments of the invention improve security against attacks and adversarial samples in AI and machine learning applications. For example, flaws and vulnerabilities in neural networks that enable attacks with adversarial samples are subject to our invention, which carefully generates and exploits backdoors in machine learning models to detect and reject such adversarial samples. is addressed by embodiments of. In particular, embodiments of the present invention detect adversarial samples by comparing the behavior of the sample when evaluated on a backdoored model with triggers attached to the behavior of legitimate samples.

Threat model:
The threat model according to embodiments of the invention takes into account white box attack scenarios where the adversary has sufficient knowledge and access to the machine learning model M. The adversary is free to learn from the model with unlimited query-response pairs. However, an adversary cannot cheat the model or the training process, for example by corrupting the data used to train the model.

Given a sample S that is (correctly) classified as y=M(S), the adversary's goal is to create an adversarial sample S' that is classified as y' (y≠y'). Since the difference between S and S' should be small enough that it cannot be detected visually by humans, the possible modifications that an adversary can make to the original sample S are limited. This is instantiated by a distance constraint, such as rms(S'-S)<8, which limits the root mean square of interpixel distances to 8 out of 255.

The goal of the solution according to embodiments of the present invention is, given a sample S, if S is a legitimate (genuine) sample, output y←M(S), and it is determined that it is an adversarial sample. Sample S is to reject.

Instantiating the attack:
Essentially, attacks attempt to fool machine learning models by estimating subtle perturbations that are introduced to change the model's predictions. White-box attacks accomplish this by picking valid input samples that are selected based on the classifier's response and repeatedly interrogating the model with small perturbations at each step. Therefore, the attacker can predict how the disruption will affect the classifier and respond adaptively. The disruption added at each step varies depending on the attack type. The final goal of the adversary is to transform a genuine sample s with the original target y _s into an adversarial sample s _a (rms(ss _a )<Max_Perturbation) that is classified into the target class y _a ≠ y _s That's true.

Although many existing defense proposals are effective against ad hoc attacks, they cannot thwart adaptive adversaries, i.e., adversaries that adapt their attacks based on what they know about the defense. As discussed above, this area is currently highly researched, and there are not only many existing attacks to take into account, but also many technical challenges to overcome when building defense strategies. be. For each of the attacks discussed in the existing literature, modified and adaptive versions of these attacks also exist, posing serious security threats.

One existing defense strategy aims to eliminate potentially adversarial samples by performing transformations on the input samples, such as adding randomness and applying filters. This approach has two drawbacks: it reduces the accuracy of the model, and the defenses can be bypassed by an adaptive attacker with knowledge of the transformation. Another existing defense strategy relies on "hardening" a model to make it resilient to attacks by including adversarial samples in the training set. This type of defense also works relatively well against ad-hoc attacks, but is less effective against targeted attacks, which can still reach an accuracy of 60%. Moreover, the learning process of each of these existing defense strategies is very slow and therefore fairly difficult to set up. It is also unclear whether these existing defense strategies are resilient to attacks using attack methods different from those currently known.

Model deterioration:
Another common attack on machine learning models is called model degradation. This type of attack relies on degrading the model's training set before the training phase. The degrading step occurs by selecting the samples S and attaching a trigger t, changing their target class to y _t . The newly created samples ensure that the model is trained to recognize a particular trigger t and always classify images with trigger t into the target class y _t . The trigger can be any pattern, from a simple visual pattern such as a yellow square to any minute, indiscernible pattern added to the image. For image recognition applications, the trigger can be any pixel pattern. However, triggers can also be defined for other classification problems, such as utterances or word recognition (in these instances, the triggers can be specific sounds or words/sentences, respectively). Model degradation has minimal impact on model accuracy. The terms "backdoor" and "exacerbation" are used interchangeably herein.

The exact manner in which an adversary gains access to training data depends on the application for which the machine learning classifier is deployed. If training data is collected from untrusted sources, the model can be degraded in all scenarios. For example, GOOGLE's federated learning structure allows data provided by voluntary users to be used to train shared models. Therefore, anyone, including the attacker, can participate in the training process. As mentioned above, an attacker can experiment with the model or surrogate model to see how triggers added to the sample change the classification of the sample, thereby changing the target class.

A data degradation technique according to embodiments of the present invention is used to degrade an existing (trained) model, requiring only a few additional trainings using the degraded samples. It is. To degrade the model, first a trigger is generated, which is a pattern recognized by the model. A trigger is then attached to a particular image in the training set and the target class of the image is changed to the backdoor target class (eg, by changing the image's label). Following this, several training runs containing both genuine and corrupted training data are performed until the accuracy of the backdoor reaches a sufficient value (eg, 90% accuracy). Authentic data is advantageously used in this step to ensure that the model after being trained with backdoored samples is still able to correctly classify samples that do not contain backdoors. I can do it. This step does not require huge amounts of data as is required during the normal training phase of the model, and allows perturbations to be quickly inserted into the model at a negligible cost in terms of accuracy.

Backdoor misclassification protection:
Based on current state of the art, it is presumed that it is impossible to defend against adversarial samples from an adversary with sufficient knowledge of the system. It may also be impossible to keep machine learning models and their weights completely secret. Embodiments of the invention therefore aim to change the paradigm and introduce some asymmetry between the attacker's knowledge and the defender's knowledge. To this end, embodiments of the invention provide a defense based on self-degrading the model to detect possible adversarial samples. In detail, a genuine sample evaluated on the backdoored model M _t with a trigger t is expected to be classified into the backdoor target class y _t , whereas for an adversarial sample, the backdoor It may still be classified as the target's attack class y _a rather than the classified class.

Perturbations introduced to adversarial samples can be viewed as weak triggers in behavior, like inherent backdoors in the model. Therefore, if the generated backdoored model is close enough to the original model and has a weak enough trigger, then an adversarial sample that is evaluated using the trigger on the backdoored model will still be incorrect. There is a possibility that it will be classified as such. Because adding a backdoor to a model is relatively quick, an adversary has sufficient knowledge of the original unbackdoored model M, but nothing about the backdoored models or their triggers. Since we do not know, the updated threat model can be used to our advantage, advantageous secret information known only to the defender.

This defense is based on a trigger t _N for each of the N models, as shown in Fig. 1, and a backdoored version of the N models M' _1..N that is unknown to the adversary. It relies on rapid generation. Then, as depicted in Figure 2, each classification request r is processed in the following flow in a simple manner using σ=0.
1.y ₀ ←M(s)
2. diff←0
3. For i in 1..N:
a. y _i ←M' _i (s+t _i )
b. If y _i ≠y ₀ then diff++
4. If diff>σ*N then REJECT
5. else return y ₀
Here, diff is a counter, diff++ adds 1 to the counter, and in this embodiment, the algorithm is applied to any number N of backdoored versions of the model M' _1..N . The threshold σ is a ratio or value between [0,1] so that

In one embodiment, the present invention provides a method for making authentic machine learning models secure against adversarial samples. This method consists of step a) of attaching a trigger to the sample to be classified, and step b) of classifying the triggered sample using a backdoored model that is backdoored using the trigger. including. In step c), it is determined whether the output of the backdoored model in step b) is the same as the backdoor class of the backdoored model, and/or a trigger is applied to the logit from step b). An outlier detection method is applied that compares to the legitimate logit calculated using authentic samples that are additionally applied to the backdoored model. By step d) these steps a) to c) are repeated using different triggers and backdooring models respectively associated with different triggers. In step e), the number of times each of the outputs of the backdoored model was not the same as the respective backdoor class of the backdoored model is compared with a predetermined threshold to determine whether the sample is an adversarial sample. and/or the difference determined by applying an outlier detection method is compared to a difference threshold.

In one embodiment, the method determines that if the number of times each of the outputs of the backdoored model was not the same as each of the backdoor classes of the backdoored model is less than or equal to a threshold, then as a result of a classification request on the sample, Classifying the untriggered samples using a bona fide machine learning model and counting the number of times each of the outputs of the backdoored model was not the same as each of the backdoor classes of the backdoored model. and rejecting the sample as an adversarial sample if the number is greater than a threshold.

In one embodiment, the method determines whether a sample has been tampered with if each of the outputs of the backdoored model was not the same as each of the backdoored classes of the backdoored model more than a threshold. further comprising flagging the information as such. In one embodiment, the threshold is zero.

In one embodiment, each of the backdoored models generates a respective trigger as a pattern recognizable by a bona fide machine learning model, and adds the respective trigger to a plurality of training samples; Changing the target class of each trigger-added training sample to each of the backdoor classes, and using each trigger-added training sample to train another version of the bona fide machine learning model. generated by the step of

In one embodiment, training is performed until each backdoored model obtains an accuracy of 90% or greater.

In one embodiment, a bona fide machine learning model and a version of the bona fide machine learning model are each trained, and training the version of the bona fide machine learning model using the respective trigger-added training samples comprises This is additional training to create each backdoored model from the machine learning model.

In one embodiment, additional training includes training each trigger with authentic samples along with added samples.

In one embodiment, step b) of classifying includes extracting logits in the classification of the triggered sample using the backdoored model, such that the output class of the backdoored model is In step e), the logit from step b) is compared to the set of legal logits, which are then combined with the respective trigger and then It was calculated using multiple authentic samples and applied to each of the backdoored models.

In one embodiment, the method applies a trigger using the bona fide machine learning model as a result of a classification request on the sample if the result of the outlier detection method for each of the logits is less than or equal to a difference threshold. and rejecting the sample as an adversarial sample if the result of the outlier detection method for each of the logits is greater than a difference threshold. In one embodiment, the outlier detection method uses a Local Outlier Factor algorithm.

In one embodiment, a genuine machine learning model is trained for image classification based on neural networks.

In another embodiment, the present invention provides a system for making a bona fide machine learning model secure against adversarial samples, comprising the steps of: a) attaching a trigger to a sample to be classified; c) classifying the triggered sample using the backdoored model, the output of the backdoored model in step b) being the backdoored model's backdoored class; and/or compare the logit from step b) with the legitimate logit calculated using the triggered authentic sample and applied to the backdoored model. d) repeating steps a) to c) using separate triggers; and e) applying a backdoored model to determine whether the sample is an adversarial sample. The number of times each of the outputs of the One or more hardware processors, configured alone or in combination, are provided to facilitate performance of the comparing step.

In one embodiment, the system triggers as a result of a classification request for a sample if the number of times each of the outputs of the backdoored model was not the same as each of the backdoored classes of the backdoored model is less than or equal to a threshold. Classify the untagged samples using a bona fide machine learning model such that the number of times each of the outputs of the backdoored model was not the same as each of the backdoored classes of the backdoored model is less than a threshold. If so, it is further configured to reject this sample as an adversarial sample.

In a further embodiment, the present invention provides a tangible, non-transitory computer-readable medium having instructions, which, when executed by one or more processors, according to an embodiment of the present invention. Make bona fide machine learning models secure by facilitating the execution of arbitrary method steps.

FIG. 1 schematically depicts a setup step 10 according to an embodiment of the invention. The setup stage 10 begins with an existing (trained) machine learning model 12. Create multiple N backs from an existing machine learning model 12 by additionally training the existing machine learning model 12 in each case with each sample appended with a separate trigger t1, t2, t3 and t4. Door model 15 is generated. A trigger is some pixel pattern added to a sample that is recognizable by an existing machine learning model 12, but is often so subtle that it is indistinguishable from an unaltered sample to a human observer. Preferably, each backdooring model M' ₁ , M' ₂ , M' ₃ and M' ₄ has a respective trigger t ₁ , t ₂ , t ₃ and t ₄ that is different from each other and a trigger t ₁ , t ₂ , Associated with t ₃ and t ₄ . Preferably, the data samples used to train the various backdoored models 15 also append respective triggers associated with each given model to be backdoored and target backdoors accordingly. Created from the same training set by changing the labels for the door classes. Therefore, according to one embodiment of the invention, these training samples can be generated from the same dataset, but are different across the backdoored models.

FIG. 2 schematically depicts an evaluation stage 20 according to an embodiment of the invention. Samples 22 are provided to an existing bona fide machine learning model 12 (e.g., to classify images in response to a classification request) or as input for training such a model, and the bona fide samples It is evaluated whether it is a sample or an adversarial sample. Triggers t ₁ , t 2 , t ₃ and t ₄ that were used to train the respective backdoored models M' ₁ , M' ₂ , M _{' 3 and} M' ₄ are added to sample 22 and their respective (Trigger t ₁ added to sample s is applied as input to backdooring model M' ₁ using trigger t ₁ , etc.) in case , yielding the respective classes as outputs y ₁ , y ₂ , y ₃ , y ₄ . The output y from one of the backdooring models 15 _i is the target backdoor class for each of the models 15.
The number of times different from d is determined as the total d. Target backdoor class
is determined in each case by the classification results associated with each trigger t _i (e.g. backdoor class
corresponds to the class associated with trigger _t1 , etc.). This sum d can be normalized to use any number of backdoored models 15 and compared to a predetermined threshold σ. In this example, σ is zero, so all of the 15 backdooring models have their respective backdoor classes.
yielding the respective output y _i equal to , the sample is determined to be authentic, otherwise the sample 22 is determined to be a tampered adversarial sample and is rejected, preferably tampered with be flagged as such. Once the sample 22 is determined to be a genuine sample, it can be applied as an input to or to train a genuine machine learning model without being triggered, and output as a result of a classification request. can be returned.

FIG. 3 schematically depicts an evaluation stage 30 according to another embodiment of the invention. Initially, for example during a setup phase, triggers are added to the genuine samples 32 and classified using the backdooring model 15 associated in each case with the respective trigger. FIG. 3 shows an example of a step in which a trigger t ₁ is added to a genuine sample s _g and classified using a backdoored model M′ ₁ associated with the trigger t ₁ . To improve detection accuracy, multiple triggers and multiple models associated with each trigger are used, similar to the previous embodiments. However, here, the result of the classification using the backdoored model 15 used is the logit l, which is the penultimate layer of the backdoored model 15. The last layer is called the softmax layer and is discarded or excluded from the evaluation. According to one embodiment of the present invention, the softmax layer improves the accuracy of identifying adversarial samples by discarding some information that is useful in determining whether a sample is an adversarial sample and by using logit l. It was discovered that further improvements could be made. logit
In each case, we specify a set of logits generated by applying each authentic sample 32 with the respective trigger added to each backdoored model 15, and define the authentic behavior of the authentic sample 32. represent. The triggered sample 22 is determined to be a genuine or adversarial sample and is applied as an input to the backdooring model 15, resulting in logit l _s . The resulting logit l _s is determined by a function is diff (e.g., a similarity or distance measure such as L-distance, or preferably a Local Outlier Factor (LOF) algorithm) into the set of true logits.
It is determined whether the result is greater than a predetermined difference threshold σ. If the result of the function is diff is greater than a predetermined difference threshold σ, the sample 22 is determined to be an adversarial sample and is rejected, preferably flagged as tampered with. Otherwise, if the result of the is diff function is less than or equal to a predetermined difference threshold σ, sample 22 is determined to be a genuine sample and can be used as an input to or train a genuine machine learning model without a trigger. can be applied to return an output as a result of a classification request.

LOF is an existing anomaly detection method that identifies outliers in a dataset by examining the closeness of a point to its neighborhood compared to the neighborhood-specific closeness. The Local Outlier Factor, LOFk(x;N), is the local outlier factor of x and its k nearest neighbors, given an integer parameter k, a point It gives a deviance or "outlier" based on proximity. For example, LOFk(x;N)>1 indicates that x is less likely to cluster than other points and is likely an outlier.

According to one embodiment, the predetermined difference threshold σ uses multiple authentic samples (i.e.
logit
is based on the output (input to ) and separate logit with l _s
A sample is reported as adversarial if the distance between the logits is greater than the average distance between the logits (e.g., plus some tolerance).

FIG. 4 schematically depicts a method 40 for generating one of the backdoored models 15 of FIG. 1 at a setup stage 10. First, sample 22 containing image 24 is modified to include trigger 25. The modified samples 22 are then used to further train the existing machine learning model 12. This process uses separate samples modified to contain the same trigger to reach a satisfying value for accuracy (e.g., approximately 90% of samples with the same trigger are similarly misclassified). ) is repeated until Ideally, a backdooring model should predict all samples containing a given trigger to belong to the target class associated with that trigger. Preferably, the number of newly created samples is about 100 or more. It has already been shown that good results can be achieved using only about 10 backdoored models.

Using multiple backdooring models improves the detection accuracy of the system as a whole. Furthermore, this solution according to embodiments of the invention has been found to be effective in detecting strong adversarial samples and ensures adequate transferability even in the case of adversarial robustness. Although this solution is not very accurate for "fine" adversarial samples, it can be applied particularly advantageously by embodiments of the invention as the first layer of a multi-layered defense system.

This solution using a backdooring model according to embodiments of the invention was evaluated with respect to the attacks discussed in the existing literature mentioned above. This evaluation experimentally demonstrated improvements in the security of machine learning models against adversarial samples provided by embodiments of the present invention. For the "strongest" attacks, false negative rates of ~0% were achieved, and false positive rates were approximately 6%. Increasing the threshold σ decreases the false positive rate but increases the false negative rate.

Another approach by the applicant aims to prevent transferability, using a degraded model to detect adversarial samples by comparison. In detail, this alternative technique suggests that the degraded model can be very different from the genuine equivalent model, and because of the difference between them, it is unlikely that a "weak" adversarial sample will be misclassified. Depends on the fact that there should be no. In contrast, embodiments of the present invention rely on behavioral differences between genuine and adversarial samples when classified on a model degraded with triggers. This difference in methodology leads to large differences in results. Previous techniques are particularly good against microscopic attacks, but may be less effective against attacks that are optimized for metastaticity. On the other hand, as the transferability increases, the behavioral differences when classified on the degraded model also increase, so embodiments of the present invention can be said to be more effective in finding such attacks. These separate techniques will be used in a complementary manner to improve security against separate types of attacks and achieve better overall security for machine learning computer systems and networks. .

Improved defense:
A further improvement according to an embodiment of the present invention over previously proposed defenses is one that relies on the last logit l instead of or in addition to using the classification output of the system as shown in Figure 3. It is. In this embodiment, it is advantageously, but not necessarily, possible to determine whether the backdoored model outputs a backdoor class. In this refinement, the last layer of the model is discarded. The last layer of the model is called the softmax layer, which distributes the output of the neural network from real numbers to a probability distribution.
used to map to. Although this layer is typically very effective in understanding the model's confidence in classification, it discards some information that could be used to detect adversarial samples. Embodiments of the invention apply the output of the modified model to the vector l. So the defense is to describe the true behavior of the model in a vector (which can contain potentially thousands of results).
depends on the pool of authentic samples s _g used to calculate. Upon receiving a new sample s, then in order to determine that the received sample is genuine, the output l _s of this sample is the output of the genuine behavior.
compared to

The function "is diff" can be implemented in a number of ways. For example, it is possible to use L-distance. Another possibility that gives better results is the Local Outlier Factor (LOF) There are methods commonly used to determine whether a given input is an outlier based on the density of its closest neighbors from a set of inputs, such as using an outlier detection system. Using LOF, improvements in accuracy have been demonstrated, ranging from a 95% false negative rate for subtle attacks to 40% (for the attack described in Kurakin, Alexey et al.) The accuracy of the strong attack was improved to a false negative rate of 55% (against the attacks described in Dezfooli, Seyed-Mohsen et al., and Goodfellow, Ian J et al.). and Madry, Aleksander et al.) remained unchanged at a false negative rate of 0%, while the false negative rate of the optimized attack was also significantly reduced from 80% to approximately 25%. .With further optimization, it was also possible to further improve the accuracy to ensure further improvements in security. Subtle attacks represent attack strategies that minimize adversarial disruption, while strong attacks , represents an attack strategy that optimizes the generation of highly reliable adversarial samples.

Example of an adversarial sample:
Although the above discussion is based only on digital versions of the attack (e.g. digitally altering the adversarial sample) due to the increased strength of the adversary, physical adversarial samples are also possible. It has been shown that embodiments of the present invention are equally applicable to detecting such attacks. For example, such an attack could allow a malicious party to make a self-driving car's algorithm recognize a stop sign as another sign by adding some minor modifications to a stop sign. It is possible to deceive someone into doing so. The attacker's elaborate process may involve generating a surrogate model for the traffic sign recognition model and exploring ways to alter the sign to result in misclassification. The attacker can then assess the success rate of the attack by renting/buying a car containing the target's autonomous driving system and examining the way the software deals with the modified signage. Although this type of attack may not bring economic benefits to the attacker, it poses a significant public security risk and may also implicate the liability of the vehicle manufacturer in case of an accident.

Similarly, use cases for such attacks could also target facial recognition systems. In this case, adversarial samples can be generated and used either to avoid recognition of the genuine subject (confusion attack) or to falsely match the sample to another identity (impersonation attack). Such attacks may result in economic harm and/or personal harm, and the potential for compromise of technical security systems allowing unauthorized adversaries to gain access to safety devices or equipment. There is also gender.

Accordingly, embodiments of the present invention provide the following improvements.
1. Improve the security of machine learning models and improve the use of machine learning models with enhanced security in technical fields.
2. Utilize the output of the backdooring model to distinguish between adversarial and legitimate samples by using a reference pool of known genuine samples.
3. Generate and utilize a backdoored variant of a machine learning model by using a trigger unknown to the adversary; Adversarial samples are detected by comparing the output of the classification of the adversarial sample with the output of the classification of the triggered genuine sample.
4. Use n separate triggers to generate and use N backdoored variants of the model, unknown to the adversary, to sample them in N backdoored variants Detect adversarial samples by inspecting the output of the classification.
5. The loss in accuracy (due to false rejection of some genuine samples) is reduced and becomes minor compared to existing defense strategies.
6. Compared to existing defense measures, security against adversaries with knowledge of defense is enhanced.

According to one embodiment of the present invention, a method for improving the security of a machine learning model against adversarial samples includes the following steps.
Setup stage:
- receive classification model M
- a detection phase that locally uses random triggers t ₁ , .., t _N to generate backdoored models M' ₁ , .., M' _N :
- When receiving samples to classify:
〇For each backdoor model M' ₁ , .., M' _N , classify sample s in backdoor model M' _i to which trigger t _i is added (y _i ←M' _i (s +t _i ))
〇 With the set of outputs y _1..N , the output is not equal to the backdoor class (y _i ≠
) Count the number of times.
o Advanced detection that rejects the sample if the number of misclassifications exceeds a threshold σ, flags it as tampered with, and otherwise outputs the result of the classification request against the legitimate model (M):
- Additional setup:
〇 Select multiple genuine samples s _g 〇 For each genuine sample s _g , calculate the logit l of each sample s of the backdoored model M' _i..N , and logit output
be stored in the set of
- Detection:
〇When receiving the sample s to be classified,
■For each backdoor model M' _i..N , classify sample s in backdoor model M' _i to which trigger t _i is added (y _i ←M' _i (s+t _i )), and li Extract the logit as ■For each backdoored model M' _i..N , the legitimate logit generated using each backdoored model
Apply an outlier detection method (such as LOF) on the logit compared to the set of ■When σ∈[0,1] is a given threshold, more backdoored models M _i than σN If l _i is detected as an outlier, the corresponding sample s is rejected. This mechanism is similar to the mechanism shown in Figure 2 (the threshold σ is set to 0 in this example) in the sense that the number of backdoored models that give a particular output (d in Figure 2) is not counted in this embodiment. is similar to In the example of Figure 2, this particular output is a different classification than the target class, but in this advanced detection embodiment, the particular output is an outlier verdict. In both the embodiment of FIG. 2 and advanced detection, samples are rejected if d>σN.

Embodiments of the present invention advantageously provide robustness against adaptive attacks by breaking the symmetric knowledge between attacker and defender. The trigger of the backdooring model acts as an unknown secret key to the attacker.

While embodiments of the invention have been shown and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. It will be appreciated by those skilled in the art that changes and modifications can be made within the scope of the following claims. In particular, the invention further encompasses embodiments using any combination of features from the separate embodiments above and below. Additionally, statements herein that characterize the invention refer to one embodiment and not necessarily to all embodiments of the invention.

The terms used in the claims should be interpreted to obtain the broadest appropriate interpretation consistent with the foregoing description. For example, the use of articles such as "a" or "the" when introducing an element should not be construed to exclude multiple elements. Similarly, references to "or" should be construed inclusively, and references to "A or B" should be used unless it is clear from the context or previous description that only one of A and B is intended. , do not exclude "A and B". Furthermore, the remark ``at least one of A, B, and C'' refers to one or more of the group of elements consisting of A, B, and C, whether or not A, B, and C are related as categories. and should not be construed to require at least one of each of the listed elements A, B and C. Furthermore, references to "A, B and/or C" or "at least one of A, B or C" refer to references to "A, B and/or C" or "at least one of A, B or C" from the enumerated elements, such as A being any singular entity from the enumerated elements. , for example A and B, or the entire list of elements A, B and C.

10 Setup phase
12 Machine learning models
15 Backdoor model
20 evaluation stages
22 samples
24 images
25 trigger
30 evaluation stages
32 Authentic Sample
40 Methods for generating backdoored models

Claims (15)

A method for making a bona fide machine learning model secure against adversarial samples, the method comprising:
a) attaching a trigger to the sample to be classified;
b) classifying the triggered sample using the backdoored model that was backdoored using the trigger;
c) determining whether the output of the backdoored model in step b) is the same as the backdoor class of the backdooring model, and/or appending the logit from step b) and the trigger; applying an outlier detection method that compares the logit to a legitimate logit calculated using authentic samples applied to the backdoored model;
d) repeating steps a) to c) using separate triggers and backdooring models each associated with said separate trigger;
e) in order to determine whether said sample is an adversarial sample, the number of times each of said outputs of said backdoored model was not the same as said backdoor class of each of said backdoored model is set to a predetermined threshold; and/or comparing the difference determined by applying the outlier detection method to a difference threshold. triggering the trigger as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model was not the same as each of the backdoor classes of the backdoored model is less than or equal to the threshold; classifying the non-added samples using the authentic machine learning model;
rejecting the sample as the adversarial sample if the number of times each of the outputs of the backdoored model was not the same as each of the backdoor classes of the backdoored model is greater than the threshold; 2. The method of claim 1, further comprising: If the number of times each of the outputs of the backdoored model was not the same as each of the backdoor classes of the backdoored model is greater than the threshold, then the sample is flagged as tampered with. 3. The method of claim 2, further comprising the step of establishing . 4. The method of claim 3, wherein the threshold is zero. Each of the backdoor models is
generating each of the triggers as a pattern recognizable by the authentic machine learning model;
adding said respective triggers for a plurality of training samples;
changing the target class of the training samples to which the respective triggers have been added to each of the backdoor classes;
and training another version of the authentic machine learning model using the training samples appended with the respective triggers. . 6. The method of claim 5, wherein the training is performed until the respective backdoored model obtains an accuracy of 90% or greater. said authentic machine learning model and said version of said authentic machine learning model are each trained, said step of training said version of said authentic machine learning model using said training samples appended with said respective triggers; 6. The method of claim 5, wherein is additional training to create the respective backdoored model from the authentic machine learning model. 8. The method of claim 7, wherein the additional training comprises training the respective triggers with authentic samples with the added samples. The classifying step b) includes using the backdoored model to extract the logits in the classification of the triggered sample, and the output class of the backdoored model is is not used to determine whether the adversarial sample is the adversarial sample, and in step e) the logits from step b) are appended with the respective triggers to each of the backdoored models. 9. The method according to any one of claims 1 to 8, wherein the applied logit is compared with a set of valid logits, which are calculated using a plurality of authentic samples. If the result of the outlier detection method for each of the logits is less than or equal to the difference threshold, then as a result of the classification request for the sample, the genuine machine learning model is used to detect the untriggered a step of classifying the sample;
10. The method of claim 9, further comprising: rejecting the sample as the adversarial sample if the result of the outlier detection method for each of the logits is greater than the difference threshold. 11. The method of claim 10, wherein the outlier detection method uses a Local Outlier Factor algorithm. 12. A method according to any one of claims 1 to 11, wherein the authentic machine learning model is trained for image classification based on a neural network. A system for making authentic machine learning models secure against adversarial samples, the system comprising:
a) attaching a trigger to the sample to be classified;
b) classifying the triggered sample using the backdoored model that was backdoored using the trigger;
c) determining whether the output of the backdoored model in step b) is the same as the backdoor class of the backdooring model, and/or appending the logit from step b) and the trigger; applying an outlier detection method that compares the logit to a legitimate logit calculated using authentic samples applied to the backdoored model;
d) repeating steps a) to c) using separate triggers;
e) in order to determine whether said sample is an adversarial sample, the number of times each of said outputs of said backdoored model was not the same as said backdoor class of each of said backdoored model is set to a predetermined threshold; and/or comparing the difference determined by applying said outlier detection method to a difference threshold. A system with a hardware processor. adding the trigger as a result of a classification request for the sample if the number of times each of the outputs of the backdoored model was not the same as each of the backdoor classes of the backdoored model is less than or equal to the threshold; using the authentic machine learning model to classify the samples that are not
rejecting the sample as the adversarial sample if the number of times each of the outputs of the backdoored model was not the same as each of the backdoor classes of the backdoored model is greater than the threshold; 14. The system of claim 13, configured. a tangible, non-transitory computer-readable medium having instructions, the instructions, when executed by one or more processors;
a) attaching a trigger to the sample to be classified;
b) classifying the triggered sample using the backdoored model that was backdoored using the trigger;
c) determining whether the output of the backdoored model in step b) is the same as the backdoor class of the backdooring model, and/or appending the logit from step b) and the trigger; applying an outlier detection method that compares the logit to a legitimate logit calculated using authentic samples applied to the backdoored model;
d) repeating steps a) to c) using separate triggers;
e) in order to determine whether said sample is an adversarial sample, the number of times each of said outputs of said backdoored model was not the same as said backdoor class of each of said backdoored model is set to a predetermined threshold; and/or comparing the difference determined by applying said outlier detection method to a difference threshold; Tangible non-transitory computer-readable medium.

Images