JP7315843B2 - Generating method, generating program, and information processing device - Google Patents

Description

The present invention relates to a generation method, a generation program, and an information processing apparatus.

Computer systems have machine learning systems that perform machine learning based on collected information. Machine learning systems generate trained models for analyzing information, for example, through machine learning. Machine learning systems can then use the generated trained models to provide services such as information analysis.

Attacks may be launched by malicious third parties against machine learning systems. Attacks on machine learning systems are carried out using machine learning mechanisms. The use of machine learning mechanisms to attack machine learning systems is sometimes called adversarial machine learning.

Attacks against machine learning serve a variety of purposes. For example, there is an attack called Adversarial Example that aims to manipulate query data to mislead output data. There is also an attack called model inversion, which aims to derive training data by manipulating query data. Another attack, called poisoning, aims to manipulate the training data to train a model that produces incorrect output data. Therefore, it is important to properly protect machine learning systems from these attacks.

As a technology related to the protection of computers that perform machine learning, for example, a vulnerability risk assessment system that evaluates the risk related to the vulnerability of a system that executes information processing related to a given business has been proposed. An information processing system has also been proposed that can safely execute processing using the results of machine learning. Furthermore, there is proposed a deep learning automatic learning system that securely provides the latest learning results from the server device to the client device through the Internet without maintaining the communication between the client device and the server device until the deep learning is completed.

JP 2017-224053 A JP 2019-121141 A JP 2018-190239 A

Ian J. Goodfellow, Jonathon Shlens, Christian Szegedy, "Explaining and Harnessing Adversarial Examples", ICLR 2015, arXiv:1412.6572v3, 20 March 2015 Matt Fredrikson, Somesh Jha, Thomas Ristenpart, "Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures", CCS '15 Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, 12 October 2015, Pages 1322-1333 Luis Munoz-Gonzalez, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C. Lupu, Fabio Roli, "Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization", AISec '17 Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, 3 November 2017, Pages 27-38

In order to properly protect a machine learning system from attacks, it is effective to know in advance what kinds of attacks are likely to be made against the machine learning system. For example, if a highly probable attack method is known, the security of the machine learning system can be enhanced by prioritizing countermeasures against that attack. However, the objectives of attacks on machine learning systems vary, and the methods of attack and the data used in attacks differ for each objective. Therefore, it is not easy to correctly assess the possibility of external attacks on machine learning systems.

In one aspect, the present invention aims to facilitate the generation of information indicating possible attacks on machine learning systems.

In one version, the computer provides information indicating the availability of a trained model that defines output data in response to a query, information indicating the availability of training data used to train the trained model by machine learning, and First information including at least one of information indicating a limit value of the number of queries accepted by a trained model, information indicating operability of queries accepted by the trained model, and operability of training data second information including at least one of the information indicating Based on the received first information, the computer generates third information indicating the possibility that the trained model is accessed by an attacker through one or more access modes. Then, based on the generated third information and the received second information, the computer determines the possibility that the trained model will be attacked by one or more access modes to achieve a predetermined attack purpose. to generate the fourth information shown.

According to one aspect, it is possible to easily generate information indicating the possibility of an attack on a machine learning system.

It is a figure which shows an example of the generation method of the information which shows the possibility of the attack to a machine-learning system. 1 illustrates an example of a computer system including a machine learning system; FIG. It is a figure which shows one structural example of the hardware of the computer for attack risk evaluation. It is a figure which shows an example of the procedure of attack analysis. 1 illustrates an example of a modeled machine learning system; FIG. It is a figure explaining the attack of model extraction. FIG. 4 is a diagram illustrating an attack of adversarial samples; FIG. 4 is a diagram illustrating an attack on training data estimation; FIG. 10 is a diagram explaining a poisoning attack; It is a figure explaining the attack method according to the accessibility with respect to a trained model. FIG. 2 is a diagram showing an example of an attack tree for a machine learning system; FIG. FIG. 4 is a block diagram showing an attack risk approximation function; It is a figure which shows an example of a data characteristic management table. It is a figure which shows an example of a likelihood management table. FIG. 11 is a flow chart showing an example of a procedure of attack risk approximation processing; FIG. 7 is a flowchart illustrating an example of a procedure of likelihood calculation processing; It is a figure which shows the calculation process of the likelihood using an attack tree. It is a figure which shows an example of the variable group used by likelihood calculation processing. FIG. 10 is a diagram showing an example of pseudo code of a likelihood calculation program; It is a figure which shows an example of the calculation method of an attack risk. It is a figure which shows an example of an attack risk management screen. It is a figure which shows an example of an attack risk detail screen. It is a figure which shows the example of a display of a calculation details display part. It is a figure which shows an example of the structure definition information of an attack tree.

Hereinafter, this embodiment will be described with reference to the drawings. It should be noted that each embodiment can be implemented by combining a plurality of embodiments within a consistent range.
[First embodiment]
First, a first embodiment will be described. In the first embodiment, an information processing apparatus realizes a method of generating information indicating the possibility of an attack on a machine learning system.

FIG. 1 is a diagram showing an example of a method of generating information indicating the possibility of attack on a machine learning system. The information processing apparatus 10 generates information indicating the possibility of an attack on the machine learning system, for example, by executing a generation program that describes a processing procedure of a method for generating information indicating the possibility of an attack on the machine learning system. can be generated.

The information processing device 10 is a computer having a storage unit 11 and a processing unit 12, for example. The storage unit 11 is, for example, a memory included in the information processing device 10 or a storage device. The processing unit 12 is, for example, a processor or an arithmetic circuit included in the information processing device 10 .

The processing unit 12 receives inputs of first information 4 and second information 5 indicating characteristics of data used in the machine learning system 1 by the analyst 3 . The processing unit 12 stores, for example, the input first information 4 and second information 5 in the storage unit 11 .

The first information 4 includes at least one of the following three pieces of information. A first piece of information, which may be included in the first piece of information 4, indicates the availability by the attacker 2 of trained models 1a that define output data in response to query data (hereinafter simply referred to as queries). Information. A second piece of information that may be included in the first information 4 is information indicating the availability by the attacker 2 of training data used to train the trained model 1a by machine learning. A third piece of information that may be included in the first information 4 is information indicating a limit value for the number of queries that the trained model 1a accepts. The limit value of the number of queries is, for example, the upper limit of the number of queries that the machine learning system 1 can accept input within a unit period.

The second information 5 includes at least one of the following two pieces of information. The first information that may be included in the second information 5 is information indicating the possibility of manipulation by the attacker 2 of the queries that the trained model 1a receives. A second piece of information that may be included in the second information 5 is information indicating the manipulability of the training data by the attacker 2 .

When the processing unit 12 receives the input of the first information 4 and the second information 5, based on the received first information 4, access to the trained model 1a by one or a plurality of access modes is considered as an attack. Third information 6 indicating the possibility of being performed by person 2 is generated. The one or more forms of access include, for example, at least one of the following first to third forms of access.

The first form of access is access to information within the trained model 1a. The information in the trained model 1a to be accessed is, for example, the values of weight parameters in the neural network. Access by the attacker 2 according to the first access mode is called a white box attack, for example. The processing unit 12 calculates the possibility of access by the first access mode based on information indicating availability of the trained model 1a, for example.

The second access form is an access form for inputting a query to the trained model 1a. Access by the second form of access by the attacker 2 is called a black box attack, as the information in the trained model 1a is not accessed in the second form of access. The processing unit 12 calculates the possibility of access by the second access form, for example, based on information indicating availability of the trained model 1a and information indicating availability of training data.

A third form of access is access to information in a surrogate model generated by entering a query or obtaining training data for a trained model 1a. If the similarity between the surrogate model and the trained model 1a is high, access to the information in the surrogate model by the attacker 2 will affect the machine learning system 1 in the same way as the information in the trained model 1a is accessed. at risk of attack in Such access by the attacker 2 to information in the proxy model is called an attack via the proxy model, for example. The processing unit 12 determines the possibility of access by the third access mode, for example, information indicating the availability of the trained model 1a, information indicating the availability of training data, and information indicating the limit value of the number of queries. Calculated based on

After generating the third information 6, the processing unit 12 generates fourth information 7a, 7b, . . . based on the generated third information 6 and the received second information 5. FIG. Fourth information 7a, 7b, . For example, the predetermined attack objective of the fourth information 7a is adversarial samples. Adversarial samples are described in detail in the second embodiment (see FIG. 7). The predetermined attack purpose of the fourth information 7b is poisoning. Poisoning will be described in detail in the second embodiment (see FIG. 9).

In this way, when the analyst 3 inputs the first information 4 and the second information 5 indicating the characteristics of the data used in the machine learning system 1 to the information processing device 10, the processing unit 12 generates the fourth information 7a, 7b, . . . are automatically generated. That is, information indicating the possibility of attack on the machine learning system 1 can be easily generated.

Furthermore, the analyst 3 can easily evaluate the vulnerability of the machine learning system 1 to attacks based on the fourth information 7a, 7b, . That is, the fourth information 7a, 7b, . It is shown. The fact that the processing unit 12 has determined that there is a possibility of being attacked means that the machine learning system 1 is vulnerable to the attack. Therefore, the analyst 3 can recognize what attacks the machine learning system 1 is vulnerable to by referring to the fourth information 7a, 7b, . . .

If the vulnerability of the machine learning system 1 is specifically known, the analyst 3 can improve the security of the machine learning system 1 by operating the machine learning system 1 so as to reduce the vulnerability. can.

The form of access by the attacker 2 to the trained model 1a changes depending on what kind of data the attacker 2 can obtain or how much query the machine learning system 1 can accept. Therefore, the processing unit 12 uses appropriate information in the first information 4 for each access mode to calculate the possibility of being accessed in that access mode. As a result, the possibility of access for each access mode can be calculated with high accuracy.

The processing unit 12 can calculate a numerical value indicating the probability of attack for each access mode as the fourth information 7a, 7b, . . . For example, the processing unit 12 can calculate the likelihood of an attack aimed at a hostile sample. In this case, the processing unit 12 sets a numerical value indicating the availability of the trained model 1a, a numerical value indicating the availability of the training data, and a numerical value indicating the limit of the number of queries to be accepted. At least one of them is acquired as the first information 4 . In addition, the processing unit 12 acquires, as the second information 5, a numerical value indicating the degree of operability of queries received by the trained model 1a. Next, the processing unit 12 calculates, as the third information 6, a numerical value indicating the probability that the trained model 1a will be accessed by one or a plurality of access modes. Then, the processing unit 12 converts the smaller numerical value of the numerical value indicated in the second information 5 and the numerical value of the one access mode indicated in the third information 6 to a numerical value indicating the possibility of an attack by that one access mode. is included in the fourth information 7a.

In this way, by numerically indicating the possibility of an attack for each access form, the analyst 3 can easily determine the order of priority of countermeasures against attacks. In other words, the administrator of the machine learning system 1 can efficiently prevent damage caused by an attack by preferentially implementing countermeasures against access forms with high numerical values indicating the possibility of attack.

The processing unit 12 can also generate fourth information 7a, 7b, . . . based on the generated third information 6 and the received first information 4 and second information 5. FIG. By also using the first information 4 to generate the fourth information 7a, 7b, . . . , the accuracy of the fourth information 7a, 7b, . In this case as well, the processing unit 12 can numerically indicate the possibility of attack for each access mode.

For example, the processing unit 12 can numerically indicate the possibility of an attack aimed at poisoning for each access mode. In this case, the processing unit 12 sets a numerical value indicating the availability of the trained model 1a, a numerical value indicating the availability of the training data, and a numerical value indicating the limit of the number of queries to be accepted. At least one of them is acquired as the first information 4 . The processing unit 12 also acquires, as the second information 5, a numerical value indicating the degree of operability of the training data. Next, the processing unit 12 calculates, as the third information 6, a numerical value indicating the probability that the trained model 1a will be accessed by one or a plurality of access modes. Then, the processing unit 12 determines the maximum value among the one or more numerical values indicated in the first information 4, the numerical value indicated in the second information 5, and the numerical value of one access mode indicated in the third information 6. Among them, the minimum value is included in the fourth information 7b as a numerical value indicating the possibility of attack by one access mode.

For a poisoning attack by attacker 2, it is important to be able to manipulate training data. Primary information 4, such as gender, is also important. Therefore, by using the first information 4 to calculate a numerical value indicating the possibility of an attack aimed at poisoning for each access form, a highly accurate numerical value can be calculated.

[Second embodiment]
Next, a second embodiment will be described. The second embodiment automatically calculates the likelihood of an attack peculiar to adversarial machine learning, and quantifies and evaluates the risk of the machine learning system according to the likelihood of each attack purpose type. . The likelihood in the second embodiment is an example of the fourth information 7a, 7b, . . . shown in the first embodiment.

FIG. 2 is a diagram illustrating an example of a computer system including a machine learning system. The machine learning system 30 is connected to a plurality of user terminals 31, 32, . . . via the network 20, for example. The machine learning system 30, for example, analyzes queries sent from user terminals 31, 32, . . . using learned models, and transmits analysis results to user terminals 31, 32, . The user terminals 31, 32, . . . are computers used by users who receive services using models generated by machine learning. Note that the machine learning system 30 can set, for example, an upper limit on the number of queries that can be received from the same user per certain period of time. The higher the upper limit of the number of queries, the more queries an attacker can use in an attack.

A computer 100 for attack risk evaluation is also connected to the network 20 . An analyst who manages machine learning system 30 uses computer 100 to evaluate the attack risk of machine learning system 30 against adversarial machine learning.

FIG. 3 is a diagram showing a configuration example of hardware of a computer for attack risk evaluation. A computer 100 is entirely controlled by a processor 101 . A memory 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 109 . Processor 101 may be a multiprocessor. The processor 101 is, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), or DSP (Digital Signal Processor). At least part of the functions realized by the processor 101 executing the program may be realized by an electronic circuit such as an ASIC (Application Specific Integrated Circuit) or a PLD (Programmable Logic Device).

Memory 102 is used as the main storage device of computer 100 . The memory 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the processor 101 . In addition, the memory 102 stores various data used for processing by the processor 101 . As the memory 102, for example, a volatile semiconductor memory device such as a RAM (Random Access Memory) is used.

Peripheral devices connected to the bus 109 include the storage device 103 , graphic processing device 104 , input interface 105 , optical drive device 106 , device connection interface 107 and network interface 108 .

The storage device 103 electrically or magnetically writes data to and reads data from a built-in recording medium. The storage device 103 is used as an auxiliary storage device for the computer. The storage device 103 stores an OS program, application programs, and various data. As the storage device 103, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) can be used.

A monitor 21 is connected to the graphics processing unit 104 . The graphics processing unit 104 displays an image on the screen of the monitor 21 according to instructions from the processor 101 . Examples of the monitor 21 include a display device using an organic EL (Electro Luminescence), a liquid crystal display device, and the like.

A keyboard 22 and a mouse 23 are connected to the input interface 105 . The input interface 105 transmits signals sent from the keyboard 22 and mouse 23 to the processor 101 . Note that the mouse 23 is an example of a pointing device, and other pointing devices can also be used. Other pointing devices include touch panels, tablets, touchpads, trackballs, and the like.

The optical drive device 106 reads data recorded on the optical disc 24 using laser light or the like. The optical disc 24 is a portable recording medium on which data is recorded so as to be readable by light reflection. The optical disc 24 includes DVD (Digital Versatile Disc), DVD-RAM, CD-ROM (Compact Disc Read Only Memory), CD-R (Recordable)/RW (ReWritable), and the like.

The device connection interface 107 is a communication interface for connecting peripheral devices to the computer 100 . For example, the device connection interface 107 can be connected to the memory device 25 and the memory reader/writer 26 . The memory device 25 is a recording medium equipped with a communication function with the device connection interface 107 . The memory reader/writer 26 is a device that writes data to the memory card 27 or reads data from the memory card 27 . The memory card 27 is a card-type recording medium.

Network interface 108 is connected to network 20 . Network interface 108 transmits and receives data to and from other computers or communication devices via network 20 .

The computer 100 can implement the processing functions of the second embodiment with the hardware configuration described above. Machine learning system 30 and user terminals 31, 32, . . . can also be realized by hardware similar to computer 100 shown in FIG. The information processing apparatus 10 shown in the first embodiment can also be realized by hardware similar to the computer 100 shown in FIG.

The computer 100 implements the processing functions of the second embodiment by executing a program recorded in a computer-readable recording medium, for example. A program describing the processing content to be executed by the computer 100 can be recorded in various recording media. For example, a program to be executed by the computer 100 can be stored in the storage device 103 . The processor 101 loads at least part of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the computer 100 can also be recorded in a portable recording medium such as the optical disc 24, memory device 25, memory card 27, or the like. A program stored in a portable recording medium can be executed after being installed in the storage device 103 under the control of the processor 101, for example. Alternatively, the processor 101 can read and execute the program directly from the portable recording medium.

An analyst of the machine learning system 30 uses the computer 100 to analyze attacks against the machine learning system 30 .
FIG. 4 is a diagram illustrating an example of an attack analysis procedure. The analyst first models the system (step S11). Next, the analyst makes a list of attack purpose types (step S12). Next, the analyst approximates the risk of being attacked by an attacker (attack risk) (step S13). An attack risk is, for example, a product of the likelihood of an attack and the impact of the attack. The likelihood of ease in adversarial machine learning is a numerical value that indicates the ease of attack. The impact of an attack in adversarial machine learning is the amount of damage that would be expected if the attack were actually executed. Finally, the analyst determines countermeasures against risks (step S14).

Attack analysis is thus performed in four steps. Among these processes, the process of roughly estimating the attack risk can be automated using the computer 100 . Therefore, in order to roughly estimate the attack risk of the machine learning system, the analyst models the machine learning system and lists the types of attack objectives against the machine learning system.

FIG. 5 is a diagram illustrating an example of a modeled machine learning system. As shown in FIG. 5, machine learning performed by the machine learning system 30 is divided into a training phase 40 and an inference phase 50 . Machine learning system 30 trains empty model 41 by applying training data 42 to empty model 41 in a training phase 40 .

The training data 42 includes, for example, a plurality of data sets consisting of pairs of input data 42a and correct output data 42b. Both the input data 42a and the correct answer output data 42b are represented by numerical strings. For example, in the case of machine learning using images, as the input data 42a, a numerical string representing features of the image is used.

The machine learning system 30 applies the input data 42a in the training data 42 to the empty model 41 to perform analysis and obtain output data. The machine learning system 30 compares the output data with the correct output data 42b, and corrects the empty model 41 if they do not match. Correction of the empty model 41 means, for example, correcting parameters used for analysis using the empty model 41 (weight parameters of input data to units in the case of a neural network) so that the output data approaches the correct answer. be.

By performing training using a large amount of training data 42, the machine learning system 30 can generate a trained model 43 that provides the same output data as the correct output data 42b for the input data 42a. The trained model 43 is represented by, for example, an empty model 41 and model parameters 44 to which appropriate values have been set by training.

A trained model generated in this way can be regarded as a function of the form “y=f(x)” (x and y are vectors). In other words, training in machine learning is a task of determining a function f suitable for a large number of pairs of x and y.

After generating trained model 43 , machine learning system 30 performs inference phase 50 using trained model 43 . For example, machine learning system 30 receives input of query 51 and uses trained model 43 to obtain output data 52 in response to query 51 . For example, when the query 51 is a text of an email, the machine learning system 30 outputs the determination result as to whether the email is spam or not as output data. Also, when the input data is an image, the machine learning system 30 outputs, for example, the type of animal appearing in the image as output data.

In the system modeling work, the analyst clarifies, for example, the structure of the empty model 41, the machine learning training algorithm, and the like. After completing the modeling of the machine learning system 30, the analyst lists the attack purpose types. Attack objective types for the machine learning system 30 include "model extraction", "hostile sample", "training data estimation", and "poisoning" for each attack objective.

FIG. 6 is a diagram explaining an attack on model extraction. The attacker 60 accesses the machine learning system 30 having the attack target model 61 and creates a copy of the attack target model 61 . For example, the attacker 60 uses the user terminal 31 to send a specially manipulated query for model extraction to the machine learning system 30 . Then, the attacker 60 analyzes the set of the query operated by the user terminal 31 and the output data, and generates a copy model 63 that is the same as the attack target model 61 .

If the attacker 60 can create a duplicate model 63 by model extraction, the attacker 60 can use the duplicate model 63 to provide the same service as the service provided by the machine learning system 30 without permission using another computer. can be implemented. Attacker 60 can also use replication model 63 to study further attack methods against machine learning system 30 .

FIG. 7 is a diagram illustrating an attack of a hostile sample. For example, assume that there is an original image 64 in which a panda appears. When this original image 64 is input as a query to the machine learning system 30 having the attack target model 61, the output data "panda" is obtained.

Attacker 60 uses, for example, user terminal 31 to generate perturbation image 65 for adversarial sampling. Perturbed image 65 does not represent any animal by itself. Then, the attacker 60 combines the translucent perturbation image 65 with the original image 64 using the user terminal 31 . At that time, the user terminal 31 increases the transparency of the perturbation image 65 . As a result, the same panda as in the original image 64 is represented in the hostile sample image 66 obtained by synthesis.

When an attacker 60 enters an adversarial sample image 66 as a query into the machine learning system 30, it fails to determine "panda" and results in determinations indicating other animals (eg, "gibbons").

If adversarial samples are enabled, attacker 60 can intentionally mislead the reasoning in machine learning system 30, for example. For example, when the machine learning system 30 is performing face authentication of a person, the attacker 60 pretends to be another person by having the machine learning system 30 read an image synthesized with the perturbation image as a face image for authentication. I can do it.

FIG. 8 is a diagram illustrating an attack on training data estimation. For example, machine learning system 30 may have generated trained model 43 in training phase 40 using images of people as training data 67 . The attacker 60 uses the user terminal 31 to input the query 51a manipulated for attack into the machine learning system 30 . Using the user terminal 31, the attacker 60 generates estimated data 68 for estimating the training data 67 used for training in the training phase 40 based on the input query 51a, the output data 52a obtained by reasoning, and the like. . As the estimation data 68, image data similar to, but not identical to, the training data 67 is obtained.

By estimating the training data 67, the attacker 60 can acquire personal information included in the training data 67, for example. Also, if the machine learning system 30 is used, for example, for quality inspection of products in a factory, the attacker 60 can obtain information about the characteristics of the products used for training.

FIG. 9 is a diagram explaining a poisoning attack. For example, machine learning system 30 may have used training data 42 during training phase 40 to generate trained model 43 that classifies the data into three groups by decision boundaries 45 . Using the user terminal 31, the attacker 60 causes the machine learning system 30 to perform training using the training data 69 manipulated for poisoning. The training data 69 manipulated for poisoning includes poisoning samples 69a that would not be correctly judged by the correct trained model 43 . In the poisoning sample 69a, wrong correct output data is set for the input data. The machine learning system 30 changes the decision boundary 45 according to the poisoning sample 69a.

The modified decision boundary 45a is modified in the wrong direction to accommodate the poisoning sample 69a. As a result, using the trained model 43a in the inference phase 50 after being subjected to a poisoning attack produces erroneous output data.

By attacking the machine learning system 30 with poisoning, the attacker 60 can degrade the judgment accuracy in the inference. For example, when the machine learning system 30 uses the trained model 43a to filter files input to the server, the deterioration in determination accuracy prevents the input of files with risks such as viruses from being filtered. may be permitted to

As shown in FIGS. 6 to 9, attacks against the machine learning system 30 can be classified according to what effect the attack aims to achieve. Moreover, the attack method that the attacker 60 can take varies depending on the difference in accessibility to the trained model.

FIG. 10 is a diagram illustrating an attack method according to accessibility to a trained model. White-box (WB) attacks are possible if a trained model 43 is available. In the WB attack, the attacker 60 extracts parameters from the trained model 43 and finds an appropriate attack method while confirming the extracted parameters.

The WB attack requires the attacker 60 to have advanced knowledge and skill. On the other hand, if the attacker 60 has advanced knowledge and skills, it is possible to make a precise attack successful with a high probability.
If a trained model 43 is not available, a black box (BB) attack is possible if a large number of queries can be sent to the machine learning system 30 . In the BB attack, the attacker 60 uses the user terminal 31 to send a large number of queries to the machine learning system 30 and obtain output data for those queries. The attacker 60 determines an appropriate attack technique based on the acquired output data and launches an attack.

The BB attack does not require advanced knowledge or skill to attack. Therefore, even a person without specialized knowledge can be the attacker 60 of the BB attack, but the attack is crude and the attack success rate is low.

If a trained model 43 is not available, but a more sophisticated attack can be sent, many queries can be sent, or if training data can be obtained or mimicked, an attack via a surrogate model is possible. is. In attacking via a surrogate model, attacker 60 first considers the availability of training data. For example, if machine learning system 30 is known to use publicly available data as training data, attacker 60 obtains the training data. If it is difficult to obtain the training data, the attacker 60 uses the user terminal 31 to send a large number of queries to the machine learning system 30 and obtain output data for those queries. The attacker 60 references the query and output data to deduce the training data used in machine learning.

The attacker 60 uses the obtained training data or the estimated training data to generate the proxy model 43b. For example, the attacker 60 inputs training data into the machine learning system 30 managed by the attacker 60 and causes the machine learning system 30 to perform training and generate a trained model. The trained model generated at this time is the proxy model 43b for the trained model 43 of the attack target. The attacker 60 extracts parameters from the generated proxy model 43b and finds an appropriate attack method while confirming the extracted parameters.

Attacks through the proxy model require the attacker 60 to have advanced knowledge and techniques comparable to WB attacks. The success rate of attack via the proxy model depends on the quality and transferability of the proxy model 43b. Transferability is the property that an attack that succeeds against the proxy model 43b is likely to succeed against the original trained model 43 as well.

The relationship between the classification according to the purpose of the attack as described above and the attack method according to the accessibility to the trained model can be represented by an attack tree.
FIG. 11 is a diagram showing an example of an attack tree for a machine learning system. The attack tree 70 includes nodes 71a to 71e and 72a to 72c representing conditions for making an attack executable, and nodes 73a to 73l representing attack purpose types, and the nodes are connected by arrows. there is The node at the beginning of the arrow indicates the cause or condition, and the node at the end of the arrow indicates the effect.

In the nodes 71a to 71e, conditions for attacking data are set. The condition indicated at node 71a is that a trained model can be obtained or imitated. The condition indicated at node 71 b is that many queries can be sent to machine learning system 30 . The condition indicated at node 71c is that training data can be obtained or simulated. The condition indicated at node 71d is that arbitrary manipulation of the query is possible. The condition indicated at node 71e is that arbitrary manipulation of the training data is possible.

Nodes 72a-72c show attack techniques depending on model accessibility. Node 72a indicates a WB attack, node 72b indicates a BB attack, and node 72c indicates an attack through a proxy model.

Nodes 73a to 73l indicate attack purpose types to be executed by attack methods corresponding to model accessibility. The attack objective type indicated at node 73a is leakage of trained models by WB attack. The attack objective type shown at node 73b is the leakage of trained models by BB attacks. The attack objective type shown at node 73c is the leakage of trained models by attacking through surrogate models.

The attack objective type shown in node 73d is training data estimation by WB attack. The attack objective type shown at node 73e is training data estimation by BB attack. The attack objective type shown at node 73f is training data estimation by attacking via surrogate models. Note that the training data has already been obtained at the time the proxy model is generated, and it is not significant for the attacker 60 to estimate the training data by attacking via the proxy model. Therefore, when analyzing the risk of the machine learning system 30, the computer 100 treats the training data estimation risk due to the attack via the proxy model as a reference value and excludes it from the information representing the risk of the machine learning system 30.

The attack objective type shown in node 73g is a hostile sample by WB attack. The attack objective type shown at node 73h is an adversarial sample with a BB attack. The attack objective type shown at node 73i is an adversarial sample by attacking through a surrogate model.

The attack purpose type indicated by the node 73j is poisoning by WB attack. The attack purpose type indicated by the node 73k is poisoning by BB attack. The attack objective type shown at node 73l is poisoning by attacking through a proxy model.

The square nodes 74a-74e shown in the attack tree 70 represent intermediate conditions with respect to the conditions of the left adjacent nodes. Nodes 74a to 74e represent that if any of the conditions of the nodes connected by arrows on the left are satisfied (logical sum), the event indicated by the nodes connected by arrows on the right can occur. For example, node 74a indicates that a BB attack is possible if a trained model can be obtained or mimicked, or if many queries can be sent.

In addition, the nodes 75a to 75e are arranged so that the arrows from the n-th node (n is an integer equal to or greater than 1) from the top of the plurality of nodes shown on the left side of the node 75a are the n-th nodes from the top on the right side of each of the nodes 75b to 75e. It indicates that it will be an arrow to the th node. For example, an arrow starting at node 72a and ending at node 75a is an arrow starting at node 72a and ending at node 73a, an arrow starting at node 72a and ending at node 73d, and an arrow starting at node 72a and ending at node 73g. It represents a bunch of arrows.

Note that "AND" is indicated at the confluence of the arrows at the nodes 73g-73l. This indicates that the event of the corresponding node can occur when all the conditions indicated at the node at the starting point of each arrow are satisfied (logical AND). For example, node 73g is a logical product of two conditions. That is, when WB attacks are possible and query manipulation is possible, it is possible to generate hostile samples due to WB attacks.

The computer 100 can estimate the attack risk based on the attack tree 70 .
FIG. 12 is a block diagram showing an attack risk estimation function. The computer 100 has a storage unit 110 , a data characteristic reception unit 120 , a likelihood calculation unit 130 , an attack risk calculation unit 140 and an attack risk display unit 150 .

Storage unit 110 stores data characteristic management table 111 and likelihood management table 112 . The data property management table 111 is a data table for managing the property (whether the attacker 60 can obtain the data or whether it can be manipulated arbitrarily) about the data used by the machine learning system 30, which is input by the analyst. The likelihood management table 112 is a data table for managing the likelihood calculated for each attack purpose type. The storage unit 110 is part of the storage area of the memory 102 or the storage device 103, for example.

The data characteristic receiving unit 120 receives input of information indicating characteristics of data used in the machine learning system 30 from the analyst. The data characteristic receiving unit 120 sets the input data characteristic information in the data characteristic management table 111 .

The likelihood calculation unit 130 refers to the data characteristic management table 111 and calculates the likelihood for each attack purpose type according to the attack tree 70 . Likelihood calculation section 130 sets the calculated likelihood in likelihood management table 112 .

The attack risk calculator 140 refers to the data characteristic management table 111 and the likelihood management table 112 to calculate the attack risk for each attack purpose type. The attack risk calculation unit 140 transmits the calculated attack risks to the attack risk display unit 150 .

The attack risk display unit 150 causes the monitor 21 to display the acquired attack risk. Further, when the analyst inputs detailed display of the attack risk, the attack risk display unit 150 displays the calculation basis of the attack risk based on the information set in the data characteristic management table 111 and the likelihood management table 112. The resulting data can also be displayed on the monitor 21 .

It should be noted that the lines connecting the respective elements shown in FIG. 12 indicate part of the communication paths, and communication paths other than the illustrated communication paths can also be set. Also, the function of each element shown in FIG. 12 can be realized, for example, by causing the computer 100 to execute a program module corresponding to the element.

Next, with reference to FIGS. 13 and 14, the data characteristic management table 111 and the likelihood management table 112 stored in the storage unit 110 will be specifically described.
FIG. 13 is a diagram showing an example of a data characteristic management table. The data characteristic management table 111 has columns of data type, vulnerability to attack, and degree of influence (asset value).

The type of data used by the machine learning system 30 is set in the data type column. Data types include training data, trained models, queries, and output data.

In the column of susceptibility to attack, the susceptibility to attack using data of that type is set for each type of data in the columns of acquisition/imitation, manipulation, and mass transmission. In the Acquisition/Imitation column, a higher value is set as the corresponding data is easier to obtain or imitate. A higher value is set in the operation column so that arbitrary operation (modification) of the corresponding data is easier. A large amount of transmission column is set to a value that is so high that it is easy to transmit the corresponding data to the machine learning system 30 in large amounts. For example, in the large volume transmission column of the data type “query”, a higher value is set as the number of queries that can be transmitted per unit time increases.

In the impact degree (asset value) column, the impact degree when the corresponding data is attacked is set in the leakage and manipulation columns. In the column of leakage, the degree of impact when the corresponding data is leaked is set. In the operation column, the degree of influence when the corresponding data is operated (altered) is set. The degree of influence can be based on asset value, for example. For example, the greater the loss of property due to an attack, the higher the degree of impact is set.

FIG. 14 is a diagram illustrating an example of a likelihood management table; A likelihood is set in the likelihood management table 112 for each attack purpose type. The attack purpose type is represented by a set of an attack method and the purpose of the attack.

The procedure of attack risk estimation processing by the computer 100 will be described in detail below.
FIG. 15 is a flowchart illustrating an example of an attack risk estimation process procedure. The processing shown in FIG. 15 will be described below along with the step numbers.

[Step S<b>101 ] The data characteristic receiving unit 120 receives an input of attack susceptibility and degree of influence when attacked as data characteristics of data used by the machine learning system 30 . The data identification receiving unit 120 sets the input value indicating the data characteristic in the data characteristic management table 111 .

[Step S102] The likelihood calculation unit 130 refers to the data characteristic management table 111 and calculates the likelihood for each attack purpose type. Likelihood calculation section 130 sets the calculated likelihood in likelihood management table 112 . Details of the likelihood calculation process will be described later (see FIG. 16).

[Step S103] The attack risk calculator 140 calculates an attack risk for each attack purpose type. The attack risk is, for example, a value obtained by multiplying the likelihood of an attack purpose type by the impact of the attack purpose type.

[Step S<b>104 ] The attack risk display unit 150 displays the attack risk for each attack purpose type on the monitor 21 .
If the analyst inputs the data characteristics in such a procedure, the computer 100 automatically calculates the attack risk for each attack purpose type with respect to the machine learning system 30 .

Next, the likelihood calculation process will be described in detail.
FIG. 16 is a flowchart illustrating an example of the procedure of likelihood calculation processing. The processing shown in FIG. 16 will be described below along with the step numbers.

[Step S<b>111 ] The likelihood calculation unit 130 reads from the data characteristic management table 111 numerical values indicating attack susceptibility for each of the five conditions regarding data used for attack. For example, the likelihood calculation unit 130 reads the vulnerability "2" for the condition "trained model available/imitation possible" from the data characteristic management table 111 shown in FIG. Further, the likelihood calculation unit 130 reads the attack vulnerability “8” from the data characteristic management table 111 for the condition “many queries can be sent”. Further, the likelihood calculation unit 130 reads the attack vulnerability “10” from the data characteristic management table 111 for the condition “training data available/imitation possible”. Further, the likelihood calculation unit 130 reads the attack vulnerability “6” from the data characteristic management table 111 for the condition “query operation possible”. Further, the likelihood calculation unit 130 reads the attack susceptibility “4” from the data characteristic management table 111 for the condition “training data can be manipulated”.

[Step S112] The likelihood calculation unit 130 calculates an intermediate condition for the condition "many queries can be sent". This intermediate condition is the condition at node 74a in attack tree 70 (see FIG. 11). For example, the likelihood calculation unit 130 determines which of the susceptibility to attack “2” for the condition “obtainable/imitation of a trained model” and the susceptibility to attack “8” for the condition “many queries can be sent” The value "8" of the other is taken as the value of the intermediate condition at node 74a.

[Step S113] The likelihood calculation unit 130 calculates an intermediate condition for the condition “training data available/imitation possible”. This intermediate condition is the condition at node 74b in attack tree 70 (see FIG. 11). For example, the likelihood calculation unit 130 calculates the larger value "10" of the intermediate condition value "8" calculated in step S112 and the vulnerability "10" of the condition "training data available/imitation possible". ' be the value of the intermediate condition at node 74b.

[Step S<b>114 ] The likelihood calculation unit 130 calculates intermediate conditions for each of the three attack methods according to accessibility to the trained model 43 . For example, the likelihood calculation unit 130 sets the attack susceptibility "2" of the condition "trained model available/imitation possible" as the value of the intermediate condition "WB attack". Further, the likelihood calculation unit 130 sets the value “8” of the intermediate condition calculated in step S112 as the value of the intermediate condition “BB attack”. Further, the likelihood calculation unit 130 sets the intermediate condition value “10” calculated in step S113 as the value of the intermediate condition “attack via proxy model”.

[Step S115] The likelihood calculation unit 130 calculates the likelihood for each of the four attack purpose types classified according to the attack purpose. For example, the likelihood calculation unit 130 calculates the likelihood for each combination of an attack method according to accessibility to the trained model and the impact of the attack indicated by the attack purpose type. Specifically, the likelihood calculation unit 130 performs the following calculations.

The likelihood calculation unit 130 sets the value “2” of the intermediate condition “WB attack” as the likelihood of “leakage of trained model due to WB attack”. The likelihood calculation unit 130 sets the value “8” of the intermediate condition “BB attack” as the likelihood of “leakage of trained model due to BB attack”. The likelihood calculation unit 130 sets the value “10” of the intermediate condition “attack via proxy model” as the likelihood of “leakage of trained model due to attack via proxy model”.

Further, likelihood calculation section 130 sets the value “2” of the intermediate condition “WB attack” as the likelihood of “training data estimation by WB attack”. The likelihood calculation unit 130 sets the value “8” of the intermediate condition “BB attack” as the likelihood of “training data estimation by BB attack”. The likelihood calculation unit 130 sets the value “10” of the intermediate condition “attack via proxy model” as the likelihood of “estimation of training data by attack via proxy model”.

Further, the likelihood calculation unit 130 calculates the minimum value “2” between the value “2” of the intermediate condition “WB attack” and the intermediate condition “6” of the condition “query can be manipulated” as “hostile by WB attack”. the likelihood of a “target sample”. The likelihood calculation unit 130 determines the minimum value “6” between the value “8” of the intermediate condition “BB attack” and the intermediate condition “6” of the condition “query can be manipulated” as “hostile by BB attack”. sample' likelihood. The likelihood calculation unit 130 calculates the minimum value “6” between the value “10” of the intermediate condition “attack via proxy model” and the intermediate condition “6” of the condition “query can be manipulated” as “proxy be the likelihood of "adversarial samples" due to attacks through the model.

Further, the likelihood calculation unit 130 calculates the intermediate condition value “2” for the intermediate condition “WB attack”, the intermediate condition value “10” for the condition “training data can be obtained/imitated”, and the condition “training data can be manipulated”. The minimum value "2" among the values "4" is set as the likelihood of "poisoning by WB attack". The likelihood calculation unit 130 calculates the value “8” of the intermediate condition “BB attack”, the value “10” of the intermediate condition of the condition “training data can be obtained/imitated”, and the value of the condition “training data can be manipulated”. Let the minimum value "4" of "4" be the likelihood of "poisoning by BB attack". The likelihood calculation unit 130 calculates the value “10” of the intermediate condition “attack via proxy model”, the value “10” of the intermediate condition The minimum value "4" is obtained from the "possible" value "4". Then, the likelihood calculation unit 130 sets the calculated minimum value “4” as the likelihood of “poisoning by attack via a proxy model”.

[Step S116] The likelihood calculation unit 130 outputs the likelihood for each attack purpose type. For example, the likelihood calculation unit 130 writes the likelihood for each attack purpose type into the likelihood management table 112 .
In this way, the likelihood for each attack purpose type can be calculated according to the attack tree 70 .

FIG. 17 is a diagram illustrating a likelihood calculation process using an attack tree. In FIG. 17, the position of each node indicates the value calculated for the condition or attack purpose type corresponding to the node.

A procedure for calculating the likelihood along the structure of the attack tree 70 can be described, for example, in a program module describing the processing procedure of the likelihood calculation unit 130 .
FIG. 18 is a diagram showing an example of a variable group used in likelihood calculation processing. For example, the variable name of the variable that sets the vulnerability value for the condition "can get/imitate a trained model" is "L_can_get_model". The variable name of the input value of the attack susceptibility of the condition “many queries can be sent” is “L_can_send_queries”. The variable name of the input value of vulnerability of the condition “Training data can be obtained/imitated” is “L_can_get_tdata”. The variable name of the input value of vulnerability of the condition "can manipulate query" is "L_can_manipulate_query". The variable name of the input value of vulnerability of the condition "can manipulate training data" is "L_can_manipulate_tdata".

Values of the five conditions for data used for attack are changed by intermediate conditions. A two-dimensional array manages the transition of the value of the condition for the data used for the attack. Each condition for data used for an attack is given a number in ascending order from "0" from the top in the attack tree 70. FIG. Input values relating to conditions for data used for attacks are set in arrays L[0][0] to L[0][4], respectively. The values of the intermediate conditions calculated using any of the values in the arrays L[0][0] to L[0][4] are stored in the arrays L[1][0] to L[1][4]. set. The values of the intermediate conditions calculated using the values in any of the arrays L[1][0] to L[1][4] are stored in the arrays L[2][0] to L[2][4]. set.

The value of the condition "WB attack" is set as the 0th element "L_access[0]" in the array L_access[i] (i=0 to 2). The value of the condition "BB attack" is set as the first element "L_access[1]" in the array L_access[i] (i=0 to 2). The value of the condition “attack via proxy model” is set as the second element “L_access[2]” in the array L_access[i] (i=0 to 2).

The value (likelihood) of the attack purpose type “leakage of trained model by WB attack” is set as the 0th element “L_model_leak[0]” in the array L_model_leak[i] (i=0 to 2). The value (likelihood) of the attack purpose type “leakage of trained model by BB attack” is set as the first element “L_model_leak[1]” in the array L_model_leak[i] (i=0 to 2). The value (likelihood) of the attack purpose type “Leakage of trained model due to attack via proxy model” is stored in the array L_model_leak[i] (i=0 to 2) as the second element “L_model_leak[2]”. set.

The value (likelihood) of the attack purpose type “training data estimation by WB attack” is set as the 0th element “L_tdata_infer[0]” in the array L_tdata_infer[i] (i=0 to 2). The value (likelihood) of the attack purpose type “training data estimation by BB attack” is set as the first element “L_tdata_infer[1]” in the array L_tdata_infer[i] (i=0 to 2). The value (likelihood) of the attack purpose type “estimation of training data by attacking through a proxy model” is set as the second element “L_tdata_infer[2]” in the array L_tdata_infer[i] (i=0 to 2). be.

The value (likelihood) of the attack purpose type “hostile sample by WB attack” is set as the 0th element “L_advex[0]” in the array L_advex[i] (i=0 to 2). The value (likelihood) of the attack purpose type “hostile sample by BB attack” is set as the first element “L_advex[1]” in the array L_advex[i] (i=0 to 2). The value (likelihood) of the attack purpose type “hostile sample by attack via proxy model” is set as the second element “L_advex[2]” in the array L_advex[i] (i=0 to 2). be.

The value (likelihood) of the attack purpose type “poisoning by WB attack” is set as the 0th element “L_poisoning[0]” in the array L_poisoning[i] (i=0 to 2). The value (likelihood) of the attack purpose type “poisoning by BB attack” is set as the first element “L_poisoning[1]” in the array L_poisoning[i] (i=0 to 2). The value (likelihood) of the attack purpose type “poisoning by attack via a proxy model” is set as the second element “L_poisoning[2]” in the array L_poisoning[i] (i=0 to 2).

Using the variables and arrays shown in FIG. 18, the procedure of likelihood calculation processing along the attack tree 70 can be described by a program.
FIG. 19 is a diagram showing an example of pseudocode of a likelihood calculation program. The 1st to 5th lines of the likelihood calculation program 131 are instructions for setting the input values related to the data conditions used in the attack to the arrays L[0][0] to L[0][4]. .

The sixth and seventh lines of the likelihood calculation program 131 use the values of the arrays L[0][0] to L[0][4] to store the values of the intermediate conditions in the first stage in the array L[1 ][0] to L[1][4]. Among them, "L[1][1]=max(L[0][0], L[0][1])" in the seventh line is L[0][0] and L[0][1 ] is set to L[1][1].

The eighth and ninth lines of the likelihood calculation program 131 use the values of the arrays L[1][0] to L[1][4] to store the values of the intermediate conditions in the second stage in the array L[2 ][0] to L[2][4]. Of these, "L[2][2]=max(L[1][1], L[1][2])" on the 9th line is L[1][1] and L[1][2] ] is set to L[2][2].

The 10th to 12th lines are instructions for setting the value of the condition regarding the attack technique according to the accessibility to the trained model.
The 13th to 18th lines are instructions for setting the likelihood of the attack purpose type of the attack. Of these, the 16th line "L_advex[i]=min(L_access[i], L[2][3])" is the minimum value between L_access[i] and L[2][3], This is an instruction to set L_advex[i]. "L_poisoning[i]=min(L_access[i], L[2][2], L[2][4])" on the 17th line is L_access[i], L[2][2] and L This is an instruction to set the minimum value of [2] and [4] to L_poisoning[i].

The 19th line is an instruction to output the likelihood of the attack purpose type.
By causing the computer 100 to execute the likelihood calculation program 131 as shown in FIG. 19, the likelihood calculation processing by the likelihood calculation unit 130 shown in FIGS. 16 and 17 can be realized. When the likelihood calculation is completed, the attack risk calculator 140 calculates the attack risk for each attack purpose type.

FIG. 20 is a diagram illustrating an example of an attack risk calculation method. The attack risk calculation unit 140 sets the result of multiplying the likelihood of an attack obtained for each attack purpose type by the degree of influence of the attack for that attack purpose type as the attack risk of that attack purpose type.

It should be noted that when a poisoning attack is received, the trained model 43 is directly manipulated. Manipulation of the trained model 43 has the effect that when an inference is made using the trained model 43, the output data resulting from the inference is also manipulated. Therefore, the attack risk calculation unit 140 uses the impact of the trained model operation to calculate the attack risk of the operation of the trained model by poisoning, and uses the impact of the output data operation to operate the output data by poisoning. to calculate the attack risk of

The attack risk calculated by the attack risk calculator 140 is displayed on the monitor 21 by the attack risk display 150 . For example, when the data characteristic receiving unit 120 receives input of information indicating data characteristics on the attack risk management screen, the attack risk display unit 150 displays the attack risk calculation result on the attack risk management screen.

FIG. 21 is a diagram showing an example of an attack risk management screen. The attack risk management screen 80 includes an assessment summary display portion 81 , a system summary display portion 82 , a data characteristic input portion 83 , an attack risk display portion 84 , and an attack risk detail button 85 .

In the assessment summary display portion 81, information about the analyst who instructs the calculation of the attack risk is set. For example, the assessment summary display section 81 displays the analyst's name, assessment date, and analysis work time. The system overview display portion 82 displays the name of the system, the part to be analyzed, the machine learning algorithm used by the machine learning system 30, and the machine learning pipeline used by the machine learning system 30. FIG.

The data characteristic input section 83 is an input area for information indicating characteristics of data used by the machine learning system 30 . The analyst inputs the susceptibility to attack and the degree of impact for each type of data in the data characteristic input section 83 . A value input to the data characteristic input section 83 is set in the data characteristic management table 111 .

The attack risk display section 84 is a display area for attack risks to the machine learning system 30 . In the example of FIG. 21, a numerical value (attack risk) indicating the risk of being attacked by the attack is displayed for each attack purpose type. For example, for each attack purpose type, the maximum attack risk calculated for each attack method according to the accessibility to the trained model is displayed as the attack risk for that attack purpose type.

The attack risk details button 85 is a button for displaying an attack risk details screen showing details of attack risks. When the analyst presses the attack risk details button 85 , the attack risk display unit 150 displays an attack risk details screen on the monitor 21 .

FIG. 22 is a diagram illustrating an example of an attack risk detail screen. The attack risk detail screen 90 includes an attack tree display portion 91 and a calculation detail display portion 92 . The attack tree display section 91 displays an attack tree 70 showing the value obtained for each condition and the likelihood for each attack purpose type, as shown in FIG. 17 . The detailed calculation contents of the attack risk are displayed in the calculation details display section 92 .

FIG. 23 is a diagram showing a display example of the calculation details display section. The calculation details display section 92 is provided with, for example, columns for attack method/attack risk type, likelihood, impact, and attack risk. The column of attack method/attack risk type displays the attack method and the type of attack risk due to the attack method. The likelihood column displays likelihoods calculated for combinations of attack methods and types of attack risks. The impact level column displays the impact level corresponding to the type of attack risk. The attack risk column displays attack risks calculated for combinations of attack methods and types of attack risks. Note that the attack risk of “training data estimation by attack via proxy model” is a reference value and is not used as a value indicating the risk of the machine learning system 30 .

In the example of FIG. 23, for the type of attack risk "leakage of trained model", the attack risk of attacks via proxy models is the highest. For the attack risk type “training data estimation,” the attack risk due to the BB attack is the highest. For the attack risk type 'adversarial sample', the attack risk from BB attacks or attacks via proxy models is the highest. For the attack risk type "poisoning (model manipulation)", the attack risk due to the BB attack or the attack via the proxy model is the highest. For the attack risk type "poisoning (output data manipulation)", the attack risk due to the BB attack or the attack via the proxy model is the highest.

By displaying the calculation details of the attack risk in this way, the analyst can easily comprehend with what type of attack the machine learning system 30 is vulnerable to what kind of attack and for what purpose. can. The analyst can take measures to reduce the risk of high-risk attack methods and attack risk types by improving the operation method of the machine learning system 30 and improving the machine learning algorithm. As a result, the safety of the machine learning system 30 can be improved.

[Other embodiments]
In the second embodiment, as shown in FIG. 19, the structure of the attack tree 70 is incorporated in the likelihood calculation program 131, but information indicating the structure of the attack tree 70 is stored in database form in the memory 102 or It can also be held in the storage device 103 .

FIG. 24 is a diagram illustrating an example of attack tree structure definition information. The attack tree structure definition information 132 has columns for variable name, connection, and multiple arguments. In the variable name column, the variable names of the variables for setting the values of the conditions corresponding to the nodes of the attack tree 70, the intermediate conditions, and the attack purpose types are set. A logical operator used when the value of the corresponding variable is obtained by a logical operation of a plurality of arguments is set in the connection column. When the logical operator "OR" is set in the combination field, the maximum value among the values indicated by the arguments becomes the value of the corresponding variable. When the logical operator "AND" is set in the combination field, the minimum value among the values indicated by the arguments becomes the value of the corresponding variable. In the multiple argument columns, a value acquisition source (“obtained from input” or variable name) used to determine the value to be set in the corresponding variable is set.

Note that "*" added to the variable names in FIG. 24 collectively represent "_wb", "_bb", and "_sm". For example, "L_model_leak*" represents "L_model_leak_wb", "L_model_leak_bb" and "L_model_leak_sm".

If "*" is added to the variable name set in the argument column, the value of the variable (argument) obtained by replacing "*" with "_wb", "_bb", and "_sm" is the last Three letters are used as arguments for the same variable. For example, the argument of "L_model_leak_wb" is "L_access_wb".

The likelihood calculation unit 130 can calculate the likelihood for each attack purpose type by determining values in order from the upper variables in the structure definition information 132 of the attack tree.
Although the embodiment has been exemplified above, the configuration of each part shown in the embodiment can be replaced with another one having the same function. Also, any other components or steps may be added. Furthermore, any two or more configurations (features) of the above-described embodiments may be combined.

1 machine learning system 1a trained model 2 attacker 3 analyst 4 first information 5 second information 6 third information 7a, 7b, ... fourth information 10 information processing device 11 storage unit 12 processing unit

Claims (7)

the computer
Information indicating the availability of a trained model that defines output data in response to a query, information indicating the availability of training data used to train the trained model by machine learning, and queries accepted by the trained model. First information including at least one of information indicating a limit value of the number, information indicating the operability of queries received by the trained model, and information indicating the operability of the training data Receiving second information containing at least one of
generating third information indicating a possibility that an attacker accesses the trained model through one or more access forms, based on the received first information;
Based on the generated third information and the received second information, the possibility that the trained model will be attacked by each of the one or more access modes in order to achieve a predetermined attack purpose is determined. generating fourth information indicating
generation method. The one or more forms of access include a first form of access to access information in the trained model, a second form of access to input queries to the trained model, and a query to the trained model. at least one of a third form of access for accessing information in a surrogate model generated by inputting of
calculating the possibility of access in the first access form based on information indicating the availability of the trained model;
calculating the possibility of access in the second access form based on the information indicating the availability of the trained model and the information indicating the availability of the training data;
Possibility of access in the third access form based on the information indicating availability of the trained model, the information indicating availability of the training data, and the information indicating the limit value of the number of queries to calculate
A method according to claim 1. A numerical value indicating the availability of the trained model, a numerical value indicating the availability of the training data used to train the trained model by machine learning, and a number of queries received by the trained model. At least one of the numerical values indicating the height of the limit value of is obtained as the first information,
obtaining, as the second information, a numerical value indicating the degree of operability of queries received by the trained model;
calculating, as the third information, a numerical value indicating a high probability that the trained model will be accessed by each of the one or more access modes;
The smaller numerical value of the numerical value calculated as the third information for one access mode and the numerical value indicated in the second information is used as the fourth information for the one access mode. ,
3. The production method according to claim 1 or 2. generating the fourth information based on the generated third information and the received first and second information;
3. The production method according to claim 1 or 2. A numerical value indicating the availability of the trained model, a numerical value indicating the availability of the training data used to train the trained model by machine learning, and a number of queries received by the trained model. At least one of the numerical values indicating the height of the limit value of is obtained as the first information,
Acquiring a numerical value indicating the degree of operability of the training data as the second information,
calculating, as the third information, a numerical value indicating a high probability that the trained model will be accessed by each of the one or more access modes;
Among the maximum value of one or more numerical values indicated in the first information, the numerical value indicated in the second information, and the numerical value calculated as the third information for one access mode Let the fourth information about the one access form be the minimum value of
5. The generating method according to claim 4. to the computer,
Information indicating the availability of a trained model that defines output data in response to a query, information indicating the availability of training data used to train the trained model by machine learning, and queries accepted by the trained model. First information including at least one of information indicating a limit value of the number, information indicating the operability of queries received by the trained model, and information indicating the operability of the training data Receiving second information containing at least one of
generating third information indicating a possibility that an attacker accesses the trained model through one or more access forms, based on the received first information;
Based on the generated third information and the received second information, the possibility that the trained model will be attacked by each of the one or more access modes in order to achieve a predetermined attack purpose is determined. generating fourth information indicating
Generated program to run the process. Information indicating the availability of a trained model that defines output data in response to a query, information indicating the availability of training data used to train the trained model by machine learning, and queries accepted by the trained model. First information including at least one of information indicating a limit value of the number, information indicating the operability of queries received by the trained model, and information indicating the operability of the training data Second information containing at least one of these is received, and based on the received first information, an attacker may access the trained model by one or more access modes. and based on the generated third information and the received second information, the trained by each of the one or more access modes to achieve a predetermined attack purpose a processing unit that generates fourth information indicating the likelihood of an attack on the model;
Information processing device having

Images