JP6971514B1 - Information processing equipment, information processing methods and programs - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately defend against an arbitrary attack on data used for learning or inference. SOLUTION: An information processing apparatus is a first trained using learning data including an acquisition unit for acquiring predetermined data, each data, and each result data for solving a predetermined problem using each data. An inference unit that inputs predetermined data to the inference model and performs inference processing, a detection unit that detects the possibility that a predetermined attack has been applied to the predetermined data, and a predetermined attack when the possibility is detected. A specific part that identifies the first defense algorithm that can be defended in response to a predetermined attack from among multiple defense algorithms based on the predetermined data to which the It is equipped with a department. [Selection diagram] Fig. 3

Description

The present invention relates to an information processing apparatus, an information processing method and a program.

In recent years, in machine learning, defense methods against hostile attacks have been studied. For example, in Patent Document 1 below, a hostile label for classifying a pre-input sequence is generated based on the prediction that the input sequence is a perturbed version of a plurality of frames of sensor data, and this input sequence is generated. Based on the hostile label to identify the output data sequence generated by the first machine learning system and prevent the actuator system from being controlled by the control data based on the output data sequence. A method of filtering the output data sequence from the machine learning system of is disclosed.

Japanese Unexamined Patent Publication No. 2021-96854

Here, in the prior art, when a hostile attack such as a perturbation is applied to the data of the learning target or the inference target, it is detected that the hostile attack has been received by a predetermined method. Regardless of whether it is received or not, the prescribed defense method is adopted. However, hostile attacks have become diversified, and conventional techniques have not been able to adequately defend against arbitrary hostile attacks.

Therefore, one of the objects of the present invention is to provide an information processing device, an information processing method, and a program capable of appropriately defending against an arbitrary attack on data used for learning or inference.

The information processing apparatus according to one aspect of the present invention learns using learning data including an acquisition unit for acquiring predetermined data, each data, and each result data for solving a predetermined problem using each of the data. An inference unit that inputs the predetermined data to the first inference model to perform inference processing, a detection unit that detects the possibility that a predetermined attack has been applied to the predetermined data, and a detection unit that detects the possibility. If so, a specific unit that specifies a first defense algorithm that can be defended in response to the predetermined attack from among a plurality of defense algorithms based on the predetermined data to which the predetermined attack is applied, and the first. 1 The application part that applies the defense algorithm to the reasoning process,
To prepare for.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide an information processing device, an information processing method and a program capable of appropriately defending against an arbitrary attack on data used for learning or inference.

It is a figure which shows an example of the system configuration which concerns on embodiment. It is a figure which shows an example of the physical structure of the information processing apparatus which concerns on embodiment. It is a figure which shows an example of the processing block of the server which concerns on embodiment. It is a figure for demonstrating the conceptual diagram of the process which concerns on embodiment. It is a figure which shows an example of correspondence data which concerns on embodiment. It is a figure which shows an example of correspondence data which concerns on embodiment. It is a figure which shows an example of correspondence data which concerns on embodiment. It is a sequence diagram which shows an example of the process in the system which concerns on embodiment. It is a flowchart which shows an example of the defense control processing in the server which concerns on embodiment.

An embodiment of the present invention will be described with reference to the accompanying drawings. In each figure, those with the same reference numerals have the same or similar configurations.

[Embodiment]
<System configuration>
FIG. 1 is a diagram showing an example of a system configuration according to an embodiment. In the example shown in FIG. 1, the server 10 and each edge terminal (or user terminal) 20A, 20B, 20C, 20D are connected so as to be able to transmit and receive data via a network. When the edge terminals are not individually distinguished, they are also referred to as edge terminals 20.

The server 10 is an information processing device capable of collecting and analyzing data, and may be composed of one or a plurality of information processing devices. The edge terminal 20 is an information processing device capable of acquiring data such as a smartphone, a personal computer, a tablet terminal, a server, and a connected car. The edge terminal 20 may be a device that is connected to an invasive or non-invasive electrode that senses brain waves and can analyze and send / receive brain wave data.

In the system shown in FIG. 1, the server 10 acquires predetermined data of a learning target or an inference target from each edge terminal 20, and learns or infers result data using a predetermined learning model or a predetermined learned inference model. Is output. Each edge terminal 20 acquires the result data corresponding to the output inference target data.

When the server 10 acquires the inference target data from each edge terminal 20, the server 10 executes inference processing on the data. The inference process includes at least one of a process of solving a classification problem, a regression analysis problem, a clustering problem, an optimization problem, and the like. At this time, the server 10 detects the possibility that a predetermined attack has been applied to the inference target data. Predetermined attacks include, for example, attacks that perturb data by a predetermined hostile attack algorithm.

When the server 10 detects that a predetermined attack has been applied, the server 10 applies a predetermined defense method corresponding to the predetermined attack to the inference process. For example, pruning is known to be performed as an example of defense techniques (Te Juin Lester Tan, Reza Shokri, "Bypassing Backdoor Detection Algorithms in Deep Learning," IEEE European Symposium on Security and Privacy 2020, p175. -183, 6 Jun 2020 (last revised)). In this paper method, pruning can be a defense method against hostile attacks, but a more effective attack method is disclosed for this pruning defense method. Therefore, it is important to have a strategy for applying effective defense methods against hostile attacks where various methods exist.

Therefore, in the present embodiment, when a predetermined attack is detected, the server 10 has an appropriate defense method (eg, based on predetermined data, a predetermined feature amount of the predetermined data, a specified predetermined attack, or the like. , Defense algorithm) and apply the specified defense method to inference processing. This makes it possible to appropriately defend against any detected attack. Hereinafter, the configuration of each device of the present embodiment will be described.

<Hardware configuration>
FIG. 2 is a diagram showing an example of the physical configuration of the information processing apparatus 10 according to the embodiment. The information processing device 10 includes a CPU (Central Processing Unit) 10a corresponding to a calculation unit, a RAM (Random Access Memory) 10b corresponding to a storage unit, a ROM (Read only Memory) 10c corresponding to a storage unit, and a communication unit. It has a 10d, an input unit 10e, and a display unit 10f. Each of these configurations is connected to each other via a bus so that data can be transmitted and received.

In the present embodiment, the case where the information processing apparatus 10 is composed of one computer will be described, but the information processing apparatus 10 may be realized by combining a plurality of computers or a plurality of arithmetic units. Further, the configuration shown in FIG. 1 is an example, and the information processing apparatus 10 may have configurations other than these, or may not have a part of these configurations.

The CPU 10a is an example of a processor, and is a control unit that controls execution of a program stored in the RAM 10b or ROM 10c, calculates data, and processes data. The CPU 10a is, for example, a calculation unit that executes a program (learning program) for learning using a learning model in which initial values of parameters can be set. The CPU 10a receives various data from the input unit 10e and the communication unit 10d, displays the calculation result of the data on the display unit 10f, and stores the data in the RAM 10b.

The RAM 10b is a storage unit capable of rewriting data, and may be composed of, for example, a semiconductor storage element. The RAM 10b is a program executed by the CPU 10a, each learning model, each learned inference model, data related to a feature amount of inference target data, and / or defense specific information (defense ID) that specifies this feature amount and a predetermined defense algorithm. Data indicating the correspondence with and may be stored. It should be noted that these are examples, and data other than these may be stored in the RAM 10b, or a part of these may not be stored.

The ROM 10c is a storage unit capable of reading data, and may be composed of, for example, a semiconductor storage element. The ROM 10c may store, for example, a defense control program or data that is not rewritten.

The communication unit 10d is an interface for connecting the information processing device 10 to another device. The communication unit 10d may be connected to a communication network such as the Internet.

The input unit 10e receives data input from the user, and may include, for example, a keyboard and a touch panel.

The display unit 10f visually displays the calculation result by the CPU 10a, and may be configured by, for example, an LCD (Liquid Crystal Display). Displaying the calculation result by the display unit 10f may contribute to XAI (eXplainable AI). The display unit 10f may display, for example, a learning result or data related to learning.

The learning program or defense control program may be stored and provided in a non-temporary storage medium readable by a computer such as RAM 10b or ROM 10c, or may be provided via a communication network connected by the communication unit 10d. May be good. In the information processing apparatus 10, the CPU 10a executes a learning program or a defense control program to realize various operations described with reference to FIG. 3, which will be described later. It should be noted that these physical configurations are examples and do not necessarily have to be independent configurations. For example, the information processing apparatus 10 may include an LSI (Large-Scale Integration) in which the CPU 10a is integrated with the RAM 10b and the ROM 10c. Further, the information processing apparatus 10 may include a GPU (Graphical Processing Unit) or an ASIC (Application Specific Integrated Circuit).

Since the configuration of the information processing apparatus 20 is the same as the configuration of the information processing apparatus 10 shown in FIG. 2, the description thereof will be omitted. Further, the information processing device 10 and the information processing device 20 may have a CPU 10a, a RAM 10b, etc., which are basic configurations for performing data processing, and may not be provided with an input unit 10e or a display unit 10f. .. Further, the input unit 10e and the display unit 10f may be connected from the outside using an interface.

<Processing configuration>
FIG. 3 is a diagram showing an example of a processing block of the information processing apparatus (server) 10 according to the embodiment. The information processing device 10 includes an acquisition unit 101, an inference unit 102, a detection unit 103, a calculation unit 104, a specific unit 105, an application unit 108, an output unit 109, a learning unit 110, and a storage unit 111. The information processing device 10 may be configured by a general-purpose computer.

The acquisition unit 101 acquires predetermined data from the edge terminal 20 (also referred to as “first information processing apparatus”). The predetermined data is, for example, inference target data. Further, the predetermined data may be a predetermined data set. The dataset includes, for example, at least one of image data, series data and text data. Here, the image data includes still image data and moving image data. Series data includes voice data, stock price data, and the like.

The inference unit 102 inputs predetermined data acquired by the acquisition unit 101 into the inference model (first inference model) 102a and performs inference processing. For example, the inference model 102a is a trained model trained using training data including each data to be trained and each result data obtained by solving a predetermined problem using each data. As described above, the predetermined problem includes, but is not limited to, at least one of a classification problem, a regression analysis problem, a clustering problem, an optimization problem, and the like.

The inference model 102a may be a predetermined inference model using, for example, a neural network. The predetermined inference model 102a includes at least one such as an image recognition model, a series data analysis model, a robot control model, a reinforcement learning model, a voice recognition model, a voice generation model, an image generation model, and a natural language processing model. .. Specific examples of the predetermined inference model 102a include CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), DNN (Deep Neural Network), LSTM (Long Short-Term Memory), bidirectional LSTM, and DQN (Deep). It may be any one of Q-Network), VAE (Variational AutoEncoder), GANs (Generative Adversarial Networks), flow-based generation model and the like. The inference unit 102 may select a corresponding predetermined inference model based on the characteristics or types of the inference target data.

The detection unit 103 detects the possibility that a predetermined attack has been applied to the predetermined data by using a known technique. For example, the detection unit 103 may use the detection technique described in Patent Document 1, or may use a learning model in which learning data including various hostile attacks and correct labels are supervised and learned. It may detect the possibility that a predetermined attack has been made.

When the detection unit 103 detects the possibility that a predetermined attack has been applied, the calculation unit 104 calculates the feature amount of the predetermined data that may have been subjected to the predetermined attack. For example, the feature amount includes a feature amount related to a predetermined attack. Specifically, when a perturbation is applied as a predetermined attack, the feature amount related to this perturbation is calculated from the predetermined data in which the possibility of the perturbation is detected.

For example, as feature quantities, Nr_attilbutes (number of attributes), Nr_sym_attributes (number of nominal attributes), Nr_num_attributes (number of numerical attributes), Nr_eximples (number of instances), Nr_class (number of classes), Defeat Kurtosis (number of classes), Defeat Kurtosis (number of classes) Number of missing values (total)), Missing Value_Relative (number of missing values (ratio)), Mean_Absolute_Skew (mean kurtosis of numerical attributes), Mean Kurtosis (mean distortion of numerical attributes), NumattrsWithOulies (number of outliers) Of these, those that can be used to represent the characteristics of the predetermined data to which the attack has been applied may be calculated by the calculation unit 104.

Further, the calculation unit 104 may change the feature amount to be calculated according to the type of predetermined data. For example, the calculation unit 104 may use the DC component after the discrete cosine transform (DCT) conversion for the image data as the feature quantity, and the maximum power spectrum after the fast Fourier transform (FFT) for the audio data as the feature quantity. May be good.

When the detection unit 103 detects the possibility that a predetermined attack has been applied, the specific unit 105 can defend against the predetermined attack based on the predetermined data to which the predetermined attack has been applied. Identify the defense algorithm from multiple defense algorithms. Here, the methods for specifying the defense algorithm are mainly classified into the following six.

(1) Method 1 based on the response data that does not specify the attack and the feature amount of the predetermined data
In the case of the method 1, a simulation or the like is performed in advance, and the feature amount of the predetermined data that has been attacked is associated with the defense specific information (defense ID) of the defense algorithm that was applied to the predetermined data and was effective. Corresponding data A is generated (for example, FIG. 5A). For example, a hostile attack that gives a predetermined perturbation to each image data is given, each defense algorithm is applied to each image data that has been attacked, and a defense algorithm that can be appropriately classified is selected. At this time, the feature amount related to the perturbation of the image subjected to the hostile attack is associated with the defense ID of the corresponding defense algorithm.

The defense specifying unit 106 included in the specifying unit 105 identifies the defense ID corresponding to this feature amount from the feature amount of the data in which the possibility of attack is detected by using the corresponding data described above. As a result, when the possibility of an attack is detected, the specific unit 105 can apply an appropriate defense method to the attack method.

(2) Method 2 based on the response data for identifying the attack and the feature amount of the predetermined data
In the case of the method 2, a simulation or the like is performed in advance to find the feature amount of the predetermined data that has been attacked, and the feature amount and the attack specific information (attack ID) of the attack algorithm that attacked the predetermined data. Corresponding data B1 associated with is generated (for example, FIG. 5B). For example, each image data is given a hostile attack that gives a predetermined perturbation, and the feature amount related to the perturbation of the image that has been subjected to the hostile attack is associated with the defense ID of the corresponding defense algorithm. Further, when an effective defense algorithm for each attack algorithm is specified by simulation or the like, correspondence data B2 in which the attack ID and the defense ID capable of defending against this attack ID are associated with each other is generated ( For example, FIG. 5C).

The attack specifying unit 107 included in the specifying unit 105 identifies the attack ID from the feature amount of the data in which the possibility of an attack is detected by using the corresponding data B2 described above. The defense specifying unit 106 included in the specifying unit 105 identifies the defense ID corresponding to the attack ID by using the above-mentioned correspondence data B1. As a result, when the possibility of an attack is detected, the specific unit 105 can specify the attack method and apply an appropriate defense method to the specified attack method.

(3) Method 3 based on an inference model that does not specify an attack and predetermined data
In the case of the method 3, the specific unit 105 uses the training data including each data to which a predetermined attack is applied and each defense algorithm applied to each of the above-mentioned data, and the second inference generated by supervised learning. It may include inputting predetermined data to the model to predict the first defense algorithm. For example, the defense identification unit 106 may have a second inference model and predict and specify the first defense algorithm.

For example, the defense identification unit 106 sets a loss function using the probability that a predetermined attack succeeds (for example, is misclassified) against a predetermined data to which the defense algorithm is applied, and this loss function satisfies the condition. A defense algorithm may be specified (eg, to be minimal). For example, the defense specifying unit 106 outputs a logit to each defense algorithm using a softmax function, and specifies a predetermined number of higher-level defense algorithms as the first defense algorithm. The predetermined number is one or more. Further, the defense specifying unit 106 may set the loss function by weighting the inference accuracy of the original second inference model and the success probability of a predetermined attack on the second inference model.

In this way, even if the attack method that may be attacked is not specified, when the possibility of attack is detected, the appropriate defense method is predicted and specified based on the predetermined data that was attacked. Will be possible.

(4) A method based on an inference model that identifies an attack and predetermined data 4
In the case of the method 4, the attack specific unit 107 included in the specific unit 105 is generated by supervised learning using learning data including each attack algorithm and each data to which each attack based on the above-mentioned attack algorithms is added. It may include inputting predetermined data to the third inference model and predicting the first attack algorithm.

For example, the attack identification unit 107 sets a loss function using a predetermined data to which a predetermined attack is applied, and sets a loss function using the probability that the predetermined attack is at least one of the attack algorithms. The attack algorithm when the condition is satisfied (eg, the minimum) may be specified as the first attack algorithm. For example, the attack specifying unit 107 outputs a logit to each attack algorithm using a softmax function, and specifies a predetermined number of higher-order attack algorithms as the first attack algorithm. The predetermined number is one or more.

Further, the defense specific unit 106 included in the specific unit 105 is supervised using learning data including each attack algorithm and each data to which each defense algorithm is applied to each data attacked by each attack algorithm. For the sixth inference model generated by training, the first attack algorithm may be input to predict the first defense algorithm.

For example, the defense identification unit 106 sets a loss function using the success probability of a predetermined attack when each defense algorithm is applied to a predetermined data to which a predetermined attack is applied, and this loss function sets a condition. The defense algorithm when satisfied (eg, minimized) may be specified as the first defense algorithm.

This makes it possible to predict the attack method that will be attacked and to predict and identify an appropriate defense method based on the predicted attack method.

(5) A method based on an inference model that does not specify an attack and a predetermined data feature amount 5
In the case of the method 5, the specific unit 105 calculates for the fourth inference model generated by supervised learning using the learning data including each feature amount of each data and each defense algorithm corresponding to each feature amount. It may include inputting the given features to predict the first defense algorithm.

For example, the defense specifying unit 106 may set a loss function similar to that of the method 3 and specify a defense algorithm when the loss function satisfies the condition (eg, the minimum). Further, the defense specifying unit 106 may set the loss function by weighting the inference accuracy of the original fourth inference model and the success probability of a predetermined attack on the fourth inference model.

In this way, even if the attack method that is likely to be attacked is not specified, when the possibility of attack is detected, the appropriate defense method is predicted based on the feature amount of the predetermined data that has been attacked. It will be possible to identify.

(6) A method based on an inference model for identifying an attack and a predetermined amount of data features 6
In the case of the method 6, the attack specific unit 107 included in the specific unit 105 is generated by supervised learning using each feature amount of each data and learning data including each attack algorithm corresponding to each of the above-mentioned feature amounts. It may include inputting predetermined data to the fifth inference model to predict the first attack algorithm.

For example, the attack specifying unit 107 may set the same loss function as in the method 3, and specify the attack algorithm when the loss function satisfies the condition (eg, the minimum) as the first attack algorithm.

Further, the defense specific unit 106 included in the specific unit 105 is supervised using learning data including each attack algorithm and each data to which each defense algorithm is applied to each data attacked by each attack algorithm. For the sixth inference model generated by training, the first attack algorithm may be input to predict the first defense algorithm.

For example, the defense identification unit 106 sets a loss function using the success probability of a predetermined attack when each defense algorithm is applied to a predetermined data to which a predetermined attack is applied, and this loss function sets a condition. The defense algorithm when satisfied (eg, minimized) may be specified as the first defense algorithm.

This makes it possible to predict the attack method that will be attacked and to predict and identify an appropriate defense method based on the predicted attack method.

The application unit 108 applies the first defense algorithm specified by any of the above methods (1) to (6) in the specific unit 105 to the inference processing in the inference unit 102. For example, depending on the nature of the first defense algorithm, the application unit 108 may apply the first defense algorithm as preprocessing for inference, or may apply the first defense algorithm for parameter adjustment of inference processing. , The first defense algorithm may be applied as post-processing of inference processing. Specifically, the application unit 108 determines whether the specified first defense algorithm is applied to preprocessing, inference processing itself, or post-processing, and appropriately uses the first defense algorithm based on the determination result. It should be applied.

Through the above processing, the server 10 detects the possibility of receiving a hostile attack based on predetermined data for inference processing, and obtains a defense method appropriately specified against any detected hostile attack. It can be applied to inference processing.

The output unit 109 outputs the result data of the inference result by the inference unit 102 to the edge terminal 20 that has transmitted the predetermined data via the communication unit 10d. For example, the output unit 109 outputs result data including at least one such as a classification result by inference processing, a regression analysis result, a clustering result, and an optimization result.

The learning unit 110 sets a learning model for generating each of the above-mentioned inference models, and performs supervised learning for each learning data. The learning unit 110 outputs the parameter-adjusted learning model as an inference model to the inference unit 102 and the specific unit 105. For example, the learning unit 110 may perform supervised learning using the learning data including the learning target data and the correct answer label of the inference, and generate the inference model (first inference model) 102a. The learning unit 110 may be provided in another device. In this case, each inference model is generated by another device, and the server 10 acquires each inference model from another device.

Further, the learning unit 110 uses learning data including predetermined data attacked by each attack algorithm, each defense algorithm, and each inference result data when defense processing is performed by each defense algorithm, for each defense. Supervised learning may be performed by setting a loss function for reducing the success probability of an attack on a process (for example, the probability of misclassification). This learning produces a second inference model.

Further, the learning unit 110 uses the learning data including each attack algorithm and each data to which each attack based on each attack algorithm is applied to increase the matching probability of the predetermined attack algorithm with respect to the predetermined data. Supervised learning may be performed by setting a loss function. This learning produces a third inference model.

Further, the learning unit 110 uses learning data including each feature amount of each predetermined data, each defense algorithm corresponding to each feature amount, and each inference result data when the defense process is performed by each defense algorithm. , Supervised learning may be performed by setting a loss function to reduce the success probability of an attack for each defense process. This learning produces a fourth inference model.

Further, the learning unit 110 uses learning data including each feature amount of each predetermined data, each attack algorithm corresponding to each feature amount, and each data to which each attack based on each attack algorithm is applied. Supervised learning may be performed by setting a loss function for increasing the matching probability of a predetermined attack algorithm for the data of. This learning produces a fifth inference model.

Further, the learning unit 110 includes each attack algorithm, each data to which each defense algorithm is applied to each data attacked by each attack algorithm, and each inference result data when defense processing is performed by each defense algorithm. Supervised learning may be performed by setting a loss function for reducing the success probability of an attack against each defense process using the training data including and. As a result, the sixth inference model is generated.

The storage unit 111 stores data related to the inference unit 102, the specific unit 105, the learning unit 110, and the like. For example, for data to which an attack algorithm corresponding to a predetermined attack is added, what is an appropriate defense algorithm is analyzed or learned in advance, and the feature amount of the data when the predetermined attack is applied is used. , Corresponding data 111a including the defense algorithm is recorded in the storage unit 111.

<Conceptual diagram>
FIG. 4 is a diagram for explaining a conceptual diagram of processing according to an embodiment. In the example shown in FIG. 4, it is assumed that an image is used as predetermined data, CNN is used as an inference model, and a hostile attack that gives a perturbation such as a 1-pixel attack is used as a predetermined attack. Further, Attack is on the edge terminal 20 side, and Defense is on the server 10 side.

The edge terminal 20 gives a perturbation to predetermined image data, and transmits the image data including the perturbation to the server 10. The edge terminal 20 can reduce the prediction accuracy of the learning model on the server 10 side or predict and build the inference model by repeatedly perturbing the image data.

The server 10 executes inference processing on predetermined data acquired from the edge terminal 20. In the case shown in FIG. 4, the server 10 outputs the classification result of the image data as a prediction result (inference result). Further, in the example shown in FIG. 4, since the dog is included in the image data, if the perturbation is not applied, the dog is output as the classification result. However, due to the perturbation added to the dog image data, the server 10 cannot be properly classified without any protective processing, and returns different classification results such as frogs and snakes. Become.

Therefore, the server 10 detects the hostile attack by using a known technique for detecting the hostile attack against the above-mentioned hostile attack. Further, the server 10 inputs predetermined data using an inference model (classification prediction model) generated by supervised learning using images generated by various hostile attacks and classification results as learning data, and is hostile. You may predict the possibility that a target attack has been made.

When the server 10 detects that there is a possibility of a hostile attack, it is appropriate to use any of the above methods (1) to (6) based on the image data predicted to have perturbation. Identify a defense algorithm. This allows the appropriate defense algorithm to be applied to the inference process, preventing misclassification and increasing the likelihood of returning a result that is correctly classified as a dog.

<Data example>
5A to 5C are diagrams showing an example of the corresponding data 111a according to the embodiment. In the example shown in FIG. 5A, the corresponding data A is associated with the information (ID) that identifies the inference target data, and is considered to be effective for the feature amount of the inference target data and the attack algorithm derived from this feature amount. Includes defense specific information (defense ID) that identifies the algorithm. For example, the corresponding data A shown in FIG. 5A is used in the method 1 described above.

In the example shown in FIG. 5B, the corresponding data B1 is associated with the information (ID) that identifies the inference target data, and is associated with the feature amount of the inference target data and the attack specific information (attack) that specifies the attack algorithm derived from this feature amount. ID) and is included. Further, the corresponding data B2 includes a defense ID in association with the attack ID.

Here, the feature amount is data related to a predetermined attack included in the inference target data, and at least one of the above-mentioned feature amounts may be registered. Further, the corresponding data shown in FIGS. 5A to 5C may be put into practical use in, for example, a learning model used for deep learning.

<Specific examples of attack method, defense method, and detection process>
Next, the attack method, the defense method, and the detection process applicable to the present embodiment are listed below as examples, but are not limited to these examples.

<< Example of attack method >>
(Evasion)
Auto-Attack (Croce and Hein, 2020)
(Avoidance attack (White-Box type))
Auto Projected Gradient Descent (Auto-PGD) (Croce and Hein, 2020), Shadow Attack (Ghiasi et al., 2020), Wasserstein Attack (Wong et al., 2020), Imperceptible, Robust, and Targeted Adversarial Examples for Automatic Speech Recognition (Qin et al., 2019), Brendel & Bethge Attack (Brendel et al., 2019), Targeted Universal Adversarial Perturbations (Hirano and Takemoto, 2019), Audio Adversarial Examples: Targeted Attacks on Speech-to-Text (Carlini and) Wagner, 2018), High Confidence Low Uncertainty (HCLU) Attack (Grosse et al., 2018), Iterative Frame Saliency (Inkawhich et al., 2018), DPatch (Liu et al., 2018), Robust DPatch (Liu et al) ., 2018, (Lee and Kolter, 2019)), ShapeShifter (Chen et al., 2018), Projected Gradient Descent (PGD) (Madry et al., 2017), NewtonFool (Jang et al., 2017), Elastic Net (Chen et al., 2017), Adversarial Patch (Brown et al., 2017), Decision Tree Attack (Papernot et al., 2016), Carlini & Wagner (C & W) L_2 and L_inf attack (Carlini and Wagner, 2016), Basic Iterative Method (BIM) (Kura) kin et al., 2016), Jacobian Saliency Map (Papernot et al., 2016), Universal Perturbation (Moosavi-Dezfooli et al., 2016), Feature Adversaries (Sabour et al., 2016), DeepFool (Moosavi-Dezfooli et) al., 2015), Virtual Adversarial Method (Miyato et al., 2015), Fast Gradient Method (Goodfellow et al., 2014)
(Avoidance attack (Black-Box type))
Square Attack (Andriushchenko et al., 2020), HopSkipJump Attack (Chen et al., 2019), Threshold Attack (Vargas et al., 2019), Pixel Attack (Vargas et al., 2019, Su et al., 2019) , Simple Black-box Adversarial (SimBA) (Guo et al., 2019), Spatial Transformation (Engstrom et al., 2017), Query-efficient Black-box (Ilyas et al., 2017), Zeroth Order Optimisation (ZOO) (Chen et al., 2017), Decision-based / Boundary Attack (Brendel et al., 2018)
(Poisoning)
Adversarial Backdoor Embedding (Tan and Shokri, 2019), Clean Label Feature Collision Attack (Shafahi, Huang et. Al., 2018), Backdoor Attack (Gu et. Al., 2017), Poisoning Attack on Support Vector Machines (SVM) (SVM) Biggio et al., 2013), Bullseye Polytope (Aghakhani et al., 2020)
(Extraction)
Functionally Equivalent Extraction (Jagielski et al., 2019), Copycat CNN (Correia-Silva et al., 2018), KnockoffNets (Orekondy et al., 2018)
(Attribute Inference Attack)
Attribute Inference Black-Box, Attribute Inference White-Box Lifestyle Decision Tree (Fredrikson et al., 2015), Attribute Inference White-Box Decision Tree (Fredrikson et al., 2015)
(Membership Inference)
Membership Inference Black-Box, Membership Inference Black-Box Rule-Based, Label-Only Boundary Distance Attack (Choquette-Choo et al., 2020), Label-Only Gap Attack (Choquette-Choo et al., 2020),
(Relocation attack (Model Inversion))
MIFace (Fredrikson et al., 2015)
(Reconstruction)
Database Reconstruction

<< Example of defense method >>
(Pre-processing)
InverseGAN (An Lin et al. 2019), DefenseGAN (Samangouei et al. 2018), Video Compression, Resampling (Yang et al., 2019), Thermometer Encoding (Buckman et al., 2018), MP3 Compression (Carlini, N. & Wagner, D., 2018), Total Variance Minimization (Guo et al., 2018), PixelDefend (Song et al., 2017), Gaussian Data Augmentation (Zantedeschi et al., 2017), Feature Squeezing (Xu et al. , 2017), Spatial Smoothing (Xu et al., 2017), Spatial Smoothing PyTorch, Spatial Smoothing TensorFlow v2, JPEG Compression (Dziugaite et al., 2016), Label Smoothing (Warde-Farley and Goodfellow, 2016), Virtual adversarial training (Miyato et al., 2015)
(post process)
Reverse Sigmoid (Lee et al., 2018), Random Noise (Chandrasekaranet al., 2018), Class Labels (Tramer et al., 2016, Chandrasekaranet al., 2018), High Confidence (Tramer et al., 2016), Rounding (Tramer et al., 2016), General Adversarial Training (Szegedy et al., 2013), Madry's Protocol (Madry et al., 2017), Fast Is Better Than Free (Wong et al., 2020)
(Evasion)
Defensive Distillation (Papernot et al., 2015)
(Poisoning)
Neural Cleanse (Wang et al., 2019)

<< Detection processing >>
(Avoidance detection (Evasion))
Basic detector based on inputs, Detector trained on the activations of a specific layer, Detector based on Fast Generalized Subset Scan (Speakman et al., 2018)
(Poisoning)
Detection based on activations analysis (Chen et al., 2018), Detection based on data provenance (Baracaldo et al., 2018), Detection based on spectral signatures (Tran et al., 2018)
The learning unit 110 applies inference processing to each data in which the learning target data is attacked by each attack algorithm including the above-mentioned attack method and each defense algorithm including the above-mentioned defense method to each data to which the attack is applied. Each of the above-mentioned inference models is generated by using the result data of the above.

When detecting the possibility of a predetermined attack, the detection unit 103 can specify which type of attack method is used depending on which detection process is used to detect the possibility. For example, when a predetermined attack is detected in the avoidance detection process, the detection unit 103 can identify the predetermined attack as an avoidance attack, and when a predetermined attack is detected in the contamination detection process, the detection unit 103 can identify the predetermined attack. A given attack can be identified as a pollution attack.

In this case, the specifying unit 105 may specify a defense method corresponding to the type of attack based on the type of attack detected. Further, when there are a plurality of defense methods of a certain type, the specific unit 105 may select and specify a predetermined number of defense methods in order from the one having the largest logit output by using the above-mentioned inference model.

<Operation>
FIG. 6 is a sequence diagram showing an example of processing in the system according to the present embodiment. The process shown in FIG. 6 is executed by the server 10 and each edge terminal 20. In the example shown in FIG. 6, only one edge terminal 20 is shown, but even if there are a plurality of edge terminals 20, the processing content of each edge terminal 20 is the same as the processing shown in FIG. In the example shown in FIG. 6, the edge terminal 20 is also referred to as a user terminal 20.

In step S102, the user terminal 20 transmits the inference target data to the server 10. The acquisition unit 101 of the server 10 acquires inference target data from the user terminal 20.

In step S104, the inference unit 102 of the server 10 inputs the inference target data to the predetermined inference model 102a and executes the inference process.

In step S106, the detection unit 103 of the server 10 detects the possibility that a predetermined attack has been applied to the inference target data. For example, the detection unit 103 expresses a possibility numerically, and detects that there is a possibility if the numerical value is equal to or higher than a predetermined value.

In step S108, the specific unit 105 of the server 10 uses a plurality of defense algorithms to provide a first defense algorithm that can defend against a predetermined attack based on predetermined data on which a predetermined attack may have been applied. Identify from the inside. Further, the application unit 108 of the server 10 applies the first defense algorithm to the inference processing in the inference unit 102. Here, the process of specifying the defense algorithm and applying it to the inference process is called the defense process.

In step S110, when the defense process is applied, the output unit 109 of the server 10 performs the inference process again for the same predetermined data, transmits the result data to the user terminal 20, and the defense process is applied. If not, the result data inferred in step S104 is transmitted to the user terminal 20. The process of step S106 may be executed before the process of step S104. In this case, if a predetermined attack is detected, the inference process can be performed after the defense process is executed, so that it is not necessary to execute the inference process again.

FIG. 7 is a flowchart showing an example of defense control processing in the server 10 according to the present embodiment. The process shown in FIG. 7 is only an example of the process of the server 10.

In step S202, the specific unit 105 of the server 10 determines whether or not a predetermined attack may have been applied to the predetermined data based on the detection result acquired from the detection unit 103. When it is determined that there is a possibility of a predetermined attack (step S202-YES), the process proceeds to step S204, and when it is determined that there is no predetermined attack (step S202-NO), the learning target data is transferred to the user terminal 20. Get from.

In step S204, the identification unit 105 of the server 10 identifies a defense algorithm that can be defended in response to a predetermined attack by any of the methods (1) to (6) described above.

In step S206, the application unit 108 of the server 10 applies the identified defense algorithm to the inference process.

In step S208, the inference unit 102 of the server 10 performs an inference process to which the defense algorithm is applied.

This makes it possible to appropriately defend against any detected attack.

The embodiments described above are for facilitating the understanding of the present invention, and are not for limiting the interpretation of the present invention. Each element included in the embodiment and its arrangement, material, condition, shape, size, and the like are not limited to those exemplified, and can be appropriately changed.

Further, the data output from each edge terminal 20 or the server 10 may be managed by using blockchain technology. Since the blockchain is almost impossible to tamper with, it is possible to prevent the data output from each device from being tampered with and improve the reliability of the system. Further, as the blockchain, a quantum blockchain may be used.

Further, the system in the above-described embodiment may be applied to associative learning. For example, in the associative learning, the function of the reasoning unit 102 of the server 10 is provided in each edge terminal 20, each edge terminal 20 performs learning or inference, and the improvement point is transmitted to the server 10 based on the result data.

At this time, if a certain edge terminal 20 is subjected to or added to a hostile attack, the result of learning or inferring the data subjected to the hostile attack will be an erroneous result. Then, if the server 10 modifies the shared learning model with this erroneous result as an improvement point, the accuracy of this learning model will drop.

Therefore, the server 10 may provide the edge terminal 20 with the functions of the detection unit 103, the calculation unit 104, and the specific unit 105, and detect and prevent a predetermined attack on the edge terminal 20 side. Further, the server 10 may specify a predetermined defense algorithm based on the characteristics of the improvement point data, and apply this defense algorithm to the shared learning model.

Further, when the server 10 acquires data from each edge terminal 20, the server 10 acquires the terminal ID of each edge terminal 20 and stores it in association with the acquired data. In this case, the server 10 sets a flag in the terminal ID of the edge terminal 20 that has transmitted the data for detecting the possibility of a predetermined attack so that the server 10 can identify the terminal ID. When data is transmitted from the edge terminal 20 of the terminal ID identified by a flag or the like, the server 10 rejects or ignores the processing of the data. Further, when the server 10 returns the inference result to the identified edge terminal 20, different results may be randomly selected and returned. This makes it possible to dilute the reproducibility of the learning model of the server 10 by the edge terminal 20 side.

10 ... Information processing device, 10a ... CPU, 10b ... RAM, 10c ... ROM, 10d ... Communication unit, 10e ... Input unit, 10f ... Display unit, 101 ... Acquisition unit, 102 ... Inference unit, 102a ... Inference model, 103 ... Detection unit, 104 ... Calculation unit, 105 ... Specific unit, 106 ... Defense specific unit, 107 ... Attack specific unit, 108 ... Applicable unit, 109 ... Output unit, 110 ... Learning unit, 111 ... Storage unit, 111a ... Corresponding data

Claims (10)

An acquisition unit that acquires predetermined data, and
An inference unit that inputs the predetermined data to the first inference model trained using the training data including each data and each result data obtained by solving a predetermined problem using the data, and performs inference processing. When,
A detector that detects the possibility that a predetermined attack has been applied to the predetermined data, and
When the possibility is detected, the first defense algorithm that can be defended in response to the predetermined attack is specified from among a plurality of defense algorithms based on the predetermined data to which the predetermined attack is applied. Department and
An application unit that applies the first defense algorithm to the inference process,
Information processing device equipped with. The specific part is the predetermined with respect to the second inference model generated by supervised learning using the training data including each data to which the predetermined attack is applied and each defense algorithm applied to the data. The information processing apparatus according to claim 1, wherein the data of the above is input to predict the first defense algorithm. The specific unit applies the predetermined data to the third inference model generated by supervised learning using the training data including each attack algorithm and each data to which each attack based on the attack algorithm is applied. The information processing apparatus according to claim 1, comprising inputting and predicting a first attack algorithm. When the possibility is detected, a calculation unit for calculating a feature amount of a predetermined data including the predetermined attack is further provided.
The information processing apparatus according to claim 1, wherein the specific unit specifies the first defense algorithm based on the feature amount. The specific unit inputs the calculated features to the fourth inference model generated by supervised learning using the learning data including each feature and each defense algorithm corresponding to each feature. The information processing apparatus according to claim 4, which comprises predicting the first defense algorithm. The information processing apparatus according to claim 4, wherein the specific unit specifies a first attack algorithm corresponding to the predetermined attack from a plurality of attack algorithms based on the calculated feature amount. The specific unit inputs the calculated feature amount to the fifth inference model generated by supervised learning as learning data including each feature amount and each attack algorithm corresponding to the feature amount. The information processing apparatus according to claim 6, which comprises predicting a first attack algorithm. The specific part is a sixth inference model generated by supervised learning using training data including each attack algorithm and each data to which each defense algorithm is applied to each data attacked by each attack algorithm. The information processing apparatus according to claim 3 or 7, wherein the first attack algorithm is input to predict the first defense algorithm. The processor installed in the information processing device
Acquiring the specified data and
Performing inference processing by inputting the predetermined data into the first inference model trained using the training data including each data and each result data.
To detect the possibility that a predetermined attack has been applied to the predetermined data, and
When the possibility is detected, the first defense algorithm that can defend against the predetermined attack is specified from among a plurality of defense algorithms based on the predetermined data to which the predetermined attack is applied. When,
Applying the first defense algorithm to the inference process,
Information processing method to execute. For the processor installed in the information processing device
Acquiring the specified data and
Performing inference processing by inputting the predetermined data into the first inference model trained using the training data including each data and each result data.
To detect the possibility that a predetermined attack has been applied to the predetermined data, and
When the possibility is detected, the first defense algorithm that can defend against the predetermined attack is specified from among a plurality of defense algorithms based on the predetermined data to which the predetermined attack is applied. When,
Applying the first defense algorithm to the inference process,
A program to execute.

Images