JP2023166317A - Machine learning apparatus, machine learning system, machine learning method, and machine learning program - Google Patents

Abstract

To provide a technique related to machine learning using a convolutional neural network in consideration of security.SOLUTION: A machine learning apparatus includes a data acquisition unit which acquires image data, and a machine learning calculation unit which performs calculation processing using a convolutional neural network on the basis of the image data, the convolutional neural network composed of a plurality of layers. The machine learning calculation unit includes: a cryptographic processing unit which executes arithmetic processing in a cryptography execution area using homomorphic encryption; a deciphering unit which executes arithmetic processing in a deciphering execution area; a shared memory to be accessed by the cryptographic processing unit and the deciphering unit; and a receiving unit which receives layer designation information to designate an N-th layer which is an intermediate layer between an input layer and an output layer of the convolutional neural network. The cryptographic processing unit executes arithmetic processing from the input layer to the N-the layer. The deciphering unit executes arithmetic processing from a (N+1)th layer to the output layer. The shared memory stores results of the arithmetic processing of the cryptographic processing unit and the deciphering unit.SELECTED DRAWING: Figure 16

Description

The present disclosure relates to a machine learning device, a machine learning system, a machine learning method, and a machine learning program.

In recent years, image recognition using convolutional neural network learning models has made it possible to analyze complex data with high precision.

On the other hand, when the subject of analysis is sensitive data that is directly linked to personal privacy, such as medical images, it is necessary to perform analysis processing while keeping the privacy information secret (encrypted).

For example, Patent Document 1 discloses a technology related to a neural network system using homomorphic encryption.

JP2019-46460A

If image recognition is performed using a convolutional neural network using homomorphic encryption, analysis processing can be performed while privacy information remains encrypted. However, in general, although operations using homomorphic encryption have a high security level, they require an enormous amount of calculation, and there is a problem in that it is difficult to perform them at a practical calculation cost.

Therefore, the present disclosure has been made to solve the above problems, and the purpose is to provide a technology related to machine learning using a convolutional neural network that meets the practical calculation cost and security level required by users. It is to be.

In order to achieve the above object, a machine learning device according to the present disclosure includes a data acquisition unit that acquires image data, and a machine learning calculation unit that performs calculation processing using a convolutional neural network based on the image data. , the convolutional neural network is composed of multiple layers, and the machine learning calculation unit includes a cryptographic processing unit that operates in the encryption execution area using homomorphic encryption, a plaintext processing unit that operates in the plaintext execution area, and a cryptographic processing unit and plaintext processing unit. The cryptographic processing unit includes a shared memory that can be accessed by the cryptographic processing unit, and a reception unit that receives layer designation information that designates the Nth layer, which is a layer on the way from the input layer to the output layer of the convolutional neural network. , executes operations from the input layer to the Nth layer, the plaintext processing section executes operations from the (N+1)th layer to the output layer, and the shared memory stores the operations executed by the cryptographic processing section and the plaintext processing section. Remember the results.

Further, in order to achieve the above object, a machine learning system according to the present disclosure includes a data acquisition unit that acquires image data, a machine learning calculation unit that performs calculation processing using a convolutional neural network based on the image data, The convolutional neural network is composed of multiple layers, and the machine learning calculation unit includes a cryptographic processing unit that operates in the encryption execution area using homomorphic encryption, a plaintext processing unit that operates in the plaintext execution area, a cryptographic processing unit, and It has a shared memory that can be accessed by the plaintext processing unit, and a reception unit that accepts layer designation information that designates the Nth layer, which is a layer on the way from the input layer to the output layer of the convolutional neural network, and performs cryptographic processing. The section executes operations from the input layer to the Nth layer, the plaintext processing section executes operations from the (N+1)th layer to the output layer, and the shared memory stores the operations executed by the encryption processing section and the plaintext processing section. Store the result of the operation.

Further, in order to achieve the above object, a machine learning method according to the present disclosure includes a step of acquiring image data, a step of performing calculation processing using a convolutional neural network based on the image data, and a step of performing calculation processing using a convolutional neural network. The network is composed of multiple layers, and the step of performing calculation processing using a convolutional neural network is to input layer specification information that specifies the Nth layer, which is a layer on the way from the input layer to the output layer of the convolutional neural network. a step of accepting, a step of executing operations from the input layer to the Nth layer in the cryptographic execution area using homomorphic encryption, a step of executing operations from the (N+1)th layer to the output layer in the plaintext execution area; A shared memory accessible from the cipher execution area and the plaintext execution area stores results of operations in the cipher execution area and the plaintext execution area.

Further, in order to achieve the above object, a machine learning program according to the present disclosure is a machine learning program that is executed by a computer including a control unit and a storage unit, and in which the control unit performs a step of acquiring image data. , the storage unit stores image data, and the control unit performs calculation processing using a convolutional neural network based on the image data stored in the storage unit. In the step of performing calculation processing using a convolutional neural network, which is composed of layers, the control unit receives layer designation information that designates the Nth layer, which is a layer on the way from the input layer to the output layer of the convolutional neural network. a step in which the control unit executes operations from the input layer to the Nth layer in the encryption execution area using homomorphic encryption; and a step in which the control unit executes operations from the (N+1)th layer to the output layer in the plaintext execution area. and the storage unit includes a shared memory that is accessible from the cryptographic execution area and the plaintext execution area, and the shared memory stores the results of the calculations in the cryptographic execution area and the plaintext execution area. The method further includes a step.

According to the present disclosure, machine learning using a convolutional neural network can be realized in accordance with the practical calculation cost and security level desired by the user.

1 is an overall diagram of a machine learning system 1. FIG. FIG. 1 is a diagram showing the hardware configuration of a machine learning device 100 according to the present embodiment. FIG. 1 is a block diagram showing the functional configuration of a machine learning device 100 according to the present embodiment. 3 is a diagram showing an example of a key storage unit 121. FIG. 3 is a diagram showing an example of a learning model storage unit 122. FIG. FIG. 3 is a diagram showing an example of a screen for a user to input image data and a label. 1 is a conceptual diagram of a general convolutional neural network model. FIG. 3 is a diagram showing an example of a middle layer. 3 is a flowchart illustrating an example of processing executed by the machine learning device 100. FIG. 1 is a diagram showing a machine learning system 2 according to the present embodiment. It is a block diagram showing the functional configuration of a machine learning device 300 in this embodiment. FIG. 7 is a diagram illustrating an example of a screen provided by a providing unit 3324. FIG. 7 is a diagram illustrating an example of a screen provided by a providing unit 3324. 3 is a flowchart illustrating an example of processing executed by the machine learning device 300. FIG. 2 is a block diagram showing the functional configuration of a machine learning device 400 according to the present embodiment. 5 is a diagram showing a correspondence relationship between a functional block diagram and hardware regarding a shared memory 431. FIG. FIG. 3 is a diagram showing a case where execution authority is divided using account control. 3 is a flowchart illustrating an example of processing executed by the machine learning device 400. It is a block diagram showing the functional configuration of a machine learning device 500 according to the present embodiment. 1 is an overall diagram of a machine learning system 3. FIG. FIG. 2 is a block diagram showing the functional configuration of a cryptographic processing device 600 according to the present embodiment. FIG. 2 is a block diagram showing the functional configuration of a plaintext processing device 700 according to the present embodiment. FIG. 2 is a block diagram showing the functional configuration of a system management device 800 according to the present embodiment. FIG. 3 is a diagram illustrating a specific example of a case where calculation processing using a convolutional neural network is performed in physically different execution environments. 3 is a sequence diagram showing an example of processing executed by the machine learning system 3. FIG. 1 is a diagram showing a specific configuration of a machine learning system 4. FIG.

Embodiments of the present disclosure will be described below with reference to the drawings. In all the figures explaining the embodiments, common components are given the same reference numerals and repeated explanations will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are essential components of the present disclosure.

<Summary of the present invention>
In recent years, it has become possible to perform highly accurate image recognition using convolutional neural networks, and the number of cases in which complex data is analyzed is increasing. If such information is included, consideration must be given to privacy. Therefore, if the calculation process related to the convolutional neural network is executed using homomorphic encryption, which allows calculations to be performed while encrypted, image data can be analyzed while protecting privacy.

However, in general, operations using homomorphic encryption have a high security level, but there is a problem in that the calculation cost becomes enormous. Therefore, in order to solve this problem, the present invention focuses on the fact that in image recognition using a convolutional neural network, it becomes difficult to visually identify images as the filtering process of convolution operations is repeated. did. That is, in the present invention, the calculation process up to the one predetermined by the user is processed in the encryption execution area using homomorphic encryption, and the subsequent calculation process is processed in the plaintext execution area.

After a certain point in the calculation process where it becomes difficult to visually identify an image, by performing calculations in the plaintext execution area, even if the image of the calculation process is leaked to a third party, it is still possible to visually identify the image. Because it is difficult to do so, it is possible to prevent the contents of sensitive data from being known to a third party even if it is not encrypted. At the same time, since all calculation processes are not performed in the encryption execution area using homomorphic encryption, calculation costs can be suppressed.

<Embodiment 1>
(Configuration of machine learning system 1)
FIG. 1 is an overall diagram of a machine learning system 1. As shown in FIG. 1, a machine learning system 1 according to the present embodiment includes a machine learning device 100 and one or more terminal devices 200 used by a user.

Machine learning device 100 and terminal device 200 are communicably connected via network NW. The network NW is composed of a WAN (Wide Area Network), a LAN (Local Area Network), etc., but may be composed of any arbitrary network. Furthermore, the devices constituting the machine learning system 1 may be locally connected without going through the network NW. The machine learning system 1 constructs a learning model for machine learning based on input from a user, or performs inference by machine learning using input from a user and a learning model.

The terminal device 200 accepts input from the user, and transmits the input content to the machine learning device 100 via the network NW. Furthermore, the terminal device 200 receives data transmitted from the machine learning device 100 via the network NW, and presents the data to the user.

(Hardware configuration of machine learning device 100)
FIG. 2 is a diagram showing the hardware configuration of the machine learning device 100 of this embodiment. Machine learning device 100 is a general-purpose computer. The machine learning device 100 is realized by, for example, a server computer, a stationary PC (Personal Computer), a laptop PC, or the like.

As shown in FIG. 2, the machine learning device 100 includes a processor 11, a memory 12, a storage 13, a communication IF 14, and an input/output IF 15.

The processor 11 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like. The memory 12 is for temporarily storing programs and data processed by the programs, and is realized by a volatile memory such as DRAM (Dynamic Random Access Memory). The program stored in the memory 12 is a program for executing machine learning according to the present invention. The storage 13 is a storage device for storing data, and is realized by, for example, a flash memory, an HDD (Hard Disc Drive), or an SSD (Solid State Drive). The communication IF 14 is an interface for transmitting and receiving signals so that the machine learning device 100 communicates with an external device. The input/output IF 15 functions as an interface with an input device for receiving input from a user and an output device for presenting information to the user.

Note that the terminal device 200 may also have a hardware configuration similar to that shown in FIG. 2 . The operation of each component of the terminal device 200 is realized by the processor 11 according to a program stored in the memory 12, similarly to the machine learning device 100 described above.

(Functional configuration of machine learning device 100)
FIG. 3 is a block diagram showing the functional configuration of the machine learning device 100 according to this embodiment. The functional configuration of the machine learning device 100 according to the first embodiment will be described with reference to FIG. 3.

The machine learning device 100 includes a communication section 110, a storage section 120, and a control section 130.

The communication unit 110 performs processing for the machine learning device 100 to communicate with other devices. The other devices mentioned here may be PCs connected via the network NW, or terminals such as smartphones and tablets. It may also be a wearable terminal such as a head-mounted display, an AR (Augmented Reality)/VR (Virtual Reality)/MR (Mixed Reality) device, or the like. Further, the other device may be an input/output device, for example, a device that inputs and outputs data using a flash memory, HDD, SSD, or the like.

The communication unit 110 also transmits and receives data to and from devices via the network NW, locally connected devices, and the like through a secure communication channel where security is ensured. The construction of a secure communication channel and the communication method are well-known techniques that use a common key (such as a session key), a public key, etc., and therefore their explanation will be omitted.

The storage unit 120 includes, for example, a volatile memory such as a RAM, a flash memory, an HDD, an SSD, etc., and stores data and programs used by the machine learning device 100. The storage unit 120 includes, for example, a key storage unit 121 that stores keys for encrypting data or decrypting encrypted data in order for a cryptographic processing unit 1322 (described later) to perform calculations in a cryptographic execution area, and a machine learning device. It includes a learning model storage unit 122 that stores learning models used in 100. Note that the key may not be stored within the machine learning device 100, but may be stored via a network with the machine learning device 100 or in a key management device (not shown) connected to the machine learning device 100. Furthermore, it is preferable that image data and the like used by the machine learning device 100 for machine learning be deleted from the storage unit 120 after use to improve security.

The key storage unit 121 (see FIG. 4 described later) stores a key for encrypting data or decrypting encrypted data in order for the encryption processing unit 1322 to perform calculations in the encryption execution area. The key storage unit 121 stores, for example, an ID, key data, and a path where a key file is stored. An ID may be assigned to each arbitrary group such as a user or an organization, and a key may be used, or a key may be assigned to each session.

The learning model storage unit 122 (see FIG. 5 described later) stores learning models used in the machine learning device 100. The learning model storage unit 122 stores, in addition to learning models, an ID for distinguishing between multiple learning models, a tag indicating the range to which the learning model can be applied, and a learning model that allows identification of a specific part in image diagnosis. the calculation cost (including calculation amount, calculation time, cost, calculation resources, etc.) and calculation speed when performing calculations using the learning model, and the calculation speed when performing inference using the learning model. The information may be stored in association with accuracy or the like.

The control unit 130 is realized by the processor of the machine learning device 100 reading a program stored in the storage unit 120 and executing instructions included in the program. The control unit 130 controls the operation of the machine learning device 100. Specifically, the control unit 130 includes a data acquisition unit 131, a machine learning calculation unit 132, a reception unit 1321, a cryptographic processing unit 1322, and a plaintext processing unit 1323.

In situations where the parameters of a learning model are updated by machine learning (hereinafter, unless otherwise specified, this also includes the case where a new learning model is generated), the data acquisition unit 131 acquires image data that will become teaching data and Get the label for the image data. The "label" includes, for example, something that indicates the type of object indicated by the image data, or indicates the range (position) of the object to be specified in the image data. Further, the data acquisition unit 131 acquires image data to be inferred when inference is performed using a learning model. For example, the user inputs image data and a label on the terminal device 200 or the like, and the data acquisition unit 131 acquires the image data and label input by the user via the network NW.

The machine learning calculation unit 132 performs calculation processing using a convolutional neural network based on the image data etc. acquired by the data acquisition unit 131. Specifically, this calculation process is a calculation process for learning to update the parameters of a learning model or for inference using a learning model from acquired image data or the like. Note that a convolutional neural network is composed of multiple layers. The "layer" will be explained in FIG. 7 and FIG. 8 (described later).

In the process of calculation processing, the machine learning calculation unit 132 obtains the range to be calculated in the encryption execution area from the user in advance, calculates a part of the calculation process in the encryption execution area using homomorphic encryption, and performs the calculation process. A part of the process is calculated in the plaintext execution area. Specifically, the machine learning calculation unit 132 includes a reception unit 1321, a cryptographic processing unit 1322, and a plaintext processing unit 1323.

The reception unit 1321 receives layer designation information that designates the Nth layer (N is a natural number), which is a layer on the way from the input layer to the output layer of the convolutional neural network. That is, when the Nth layer is designated by the layer designation information, calculations from the input layer to the Nth layer are performed in the cryptographic execution area, and calculations from the (N+1)th layer to the output layer are performed in the plaintext execution area.

The reception unit 1321 accepts layer designation information that designates the Nth layer, but the layer designation information acquired here is a specific number indicating "N", and which layer of the intermediate layer is selected in the convolutional neural network. This indicates the range of calculations to be performed in the cryptographic execution area. A convolutional neural network generally has multiple layers, such as an input layer (layer 0), an intermediate layer (layers 1 to M), and an output layer (layer M+1). Here, M is an arbitrary natural number. At this time, the user decides to perform calculations up to the Nth layer of the intermediate layer in the cryptographic execution area, and the reception unit 1321 obtains N, which is the boundary layer to perform calculations in the cryptographic execution area.

In addition to a specific number indicating "N", the layer designation information may be information that can designate "N". For example, the condition may be a calculation cost condition accepted from the user. If the reception unit 1321 receives a calculation cost condition as the layer designation information, it calculates "N" that can be executed at the calculation cost.

The cryptographic processing unit 1322 has a function of performing calculations in the cryptographic execution area using homomorphic cryptography. In the process of calculation processing in the machine learning calculation unit 132, the cryptographic processing unit 1322 performs calculation processing from the input layer (0 layer) to the intermediate layer N in the encryption execution area using homomorphic encryption.

Specifically, the cryptographic processing unit 1322 encrypts the image data etc. acquired by the data acquisition unit 131 using the encryption key (public key) managed in the key storage unit 121. At this time, the encryption processing unit 1322 selects a homomorphic encryption method (Paillier method, etc., Lifted-Ellgamal method, Somewhat Homomorphic Encryption method, Fully Homomorphic Encryption method, etc.) that allows calculations in the encrypted state as the encryption method. use

The cryptographic processing unit 1322 encrypts the image data etc. acquired by the data acquisition unit 131 and the learning model used for machine learning, and performs calculations in the cryptographic execution area.

The cryptographic processing unit 1322 performs calculation processing from the input layer (0 layer) to the N layer of the intermediate layer designated by the user in the encryption execution area, and then stores the decryption key (secret) managed in the key storage unit 121. key) to decrypt the data.

The plaintext processing unit 1323 has a function of performing calculations in the plaintext execution area. The plaintext processing unit 1323 acquires the results of the calculations up to the N layer calculated in the cryptographic processing unit 1322, and subsequently performs the calculations from the (N+1) layer to the output layer in the plaintext execution area. This updates the parameters of the inference or learning model.

Note that the terminal device 200 may take on a part of the functions of the machine learning device 100 described above. For example, the machine learning device 100 and the terminal device 200 share a key for encrypting image data etc. using homomorphic encryption in advance, and the terminal device 200 uses the image data to be subjected to inference or machine learning. may be encrypted using homomorphic encryption.

The terminal device 200 transmits image data encrypted using homomorphic encryption to the machine learning device 100, and the machine learning device 100 stores the image data received via the communication unit 110 in the storage unit 120. The data acquisition unit 131 then acquires the encrypted image data from the storage unit 120 and sends it to the encryption processing unit 1322. Next, the cryptographic processing unit 1322 performs calculations on the encrypted image data. At this time, instead of the encryption processing unit 1322 decrypting the calculation result as described above, the plaintext processing unit 1323 may decrypt the calculation result. That is, the cryptographic processing unit 1322 handles only data encrypted using homomorphic encryption. By adopting such a configuration, the machine learning device 100 can prevent image data from being encrypted using homomorphic encryption, or because it is difficult for humans to visually identify the image even after decryption. The learning device 100 can perform calculation processing with higher security.

(Specific example of database)
FIG. 4 is a diagram showing an example of the key storage unit 121. The key storage unit 121 stores, in association with each other, the values of an "ID" and keys (public key "key1" and private key "key2") for identifying key usage situations such as users and sessions. In FIG. 4, the public key of "ID" is "0000001" is "0KJFDEMNFS679FDDFHKL..." and the private key is "2SFSHLJHFSKLJADL7DJLH...". Similarly, the public key with "ID" is "0000002" is "43JFLKEJFDHKLFS3JKLFN...", the private key is "FDLK732492SFGKJHKSDH...", the public key with "ID" is "0000003" is "JLKJ02SFHL89JLKHJKF897...", and the private key is: It is “40389439LFKSJFLKJFSLH…”. Note that FIG. 4 is an example of key management, and the key management method is not limited to these. For example, a key and a user with decryption authority may be stored in association with each other.

FIG. 5 is a diagram showing an example of the learning model storage unit 122. The learning model storage unit 122 stores information such as "ID" for distinguishing between multiple learning models, "tag" indicating the range to which the learning model can be applied, and whether or not the learning model is capable of identifying a specific part in image diagnosis. ``Part identification'' that distinguishes between ``part identification'', ``cost/MB'' that indicates the cost (amount of calculation) when performing calculations using the learning model, and ``calculation speed/Mb'' that indicates the calculation speed, when using the learning model. ``Accuracy'', which indicates the accuracy when inference is performed, and ``File'', which indicates the file name of the learning model, are stored in association with each other.

In Figure 5, the learning model whose "ID" is "0001" has "CT, cancer, kidney, medical, image diagnosis" in the "tag" field, and is applicable to these uses, but does not support body part identification. It shows that it is not. It also shows that the "cost" is 10 yen/Mb, the "calculation speed" is 5'10 per Mb, the "accuracy" is 80%, and the learning model itself is stored in the file "0001.json". There is.

In addition, the learning model whose "ID" is "0002" has "CT, cancer, kidney, medical, image diagnosis" in the "tag" column and can be applied to these uses, and the "part identification" column has "〇 ”, indicating that it supports site identification. It also shows that the "cost" is 45 yen/Mb, the "calculation speed" is 6'00 per 1 Mb, the "accuracy" is 85%, and the learning model itself is stored in the file "0002.json". There is.

The same applies to cases where the "ID" is "0003" or "0004", so the explanation will be omitted. In addition, the learning model storage unit 122 stores not only the above-mentioned items but also information in association with the organization (company, department, etc.) to which the learning model is applied, industry (medical, food, EC, etc.), owner, etc. Good too.

(Specific example of image data and label)
FIG. 6 is a diagram showing an example of a screen for the user to input image data and a label. When the data acquisition unit 131 receives input of target image data from the user, it reads the image data and sends the image 410 and file information (file name, file creation date and time, etc.) to the user as shown in the screen 401. is displayed on the terminal device 200 operated by the user. At this time, the data acquisition unit 131 also acquires a label from the user. On screen 401, the user selects whether the input image data is data indicating "normal" or data indicating "abnormal" and labels the image data. Further, when labeling a learning model that includes specifying a region indicating an abnormal part, the user specifies the range of the region on the screen and labels the region. For example, in FIG. 6, the specified range of region identification is indicated by the shaded area of the image 410.

In this way, the data acquisition unit 131 acquires image data and teacher data in which a label for the image data is associated. Then, the machine learning calculation unit 132 generates (updates) parameters of the learning model of the convolutional neural network based on the teacher data so as to output a label for the image data. As a label output, for example, when diagnosing cancer with respect to input image data, whether the image indicates normality or abnormality (suspicion of cancer) is outputted. Furthermore, when it is output that cancer is suspected, the location identification may also be output.

Note that when only image data is input for inference, labeling is not necessary and only image data is input.

(An example of a convolutional neural network)
FIG. 7 is a conceptual diagram of a general convolutional neural network model. A neural network has multiple layers such as an input layer (layer 0), an intermediate layer (layers 1 to M), and an output layer (layer M+1). In the calculation process, the machine learning calculation unit 132 acquires layer designation information indicating the boundary (Nth layer) of the range in which the reception unit 1321 calculates in the cryptographic execution area, and the cryptographic processing unit 1322 acquires layer designation information indicating the boundary (Nth layer) of the range to be calculated in the cryptographic execution area. The calculation is performed in the plaintext execution area in the plaintext processing unit 1323.

The cryptographic processing unit 1322 encrypts the image data etc. acquired by the data acquisition unit 131 and the selected learning model using homomorphic encryption using the key (public key) stored in the key storage unit 121, and performs inference or learning model. The calculation for updating the parameters is performed in the cryptographic execution area. Specifically, the cryptographic processing unit 1322 performs arithmetic processing from the 0th layer to the Nth layer of the intermediate layer. Further, the cryptographic processing unit 1322 decrypts the calculation result in the cryptographic execution area using the key (private key paired with the encrypted public key) stored in the key storage unit 121. The decrypted result may be sent to the plaintext processing unit 1323. Note that when storing the calculation result in the cryptographic execution area in the storage unit 120, it is preferable for security reasons to store the calculation result in an encrypted state and to decrypt the calculation result in the plaintext processing unit 1323.

The plaintext processing unit 1323 uses the image data etc. decrypted by the cryptographic processing unit 1322 and the learning model to perform arithmetic processing in the plaintext execution area for the calculation processes from the (N+1)th layer to the (M+1)th layer.

FIG. 8 is a diagram showing an example of the intermediate layer. A convolutional neural network generally has a convolution layer, a ReLu layer, a pooling layer, a fully connected layer, etc. for performing a convolution operation as an intermediate layer. In the present invention, the term "layer" refers to each layer such as a convolution layer, ReLu layer, and pooling layer as one layer, two layers, or three layers, or a series of processes such as convolution, ReLu, and pooling as one processing unit. It may be one layer, two layers, or three layers. In this case, for example, the combination of "convolution", "ReLu", and "pooling" may be made into one layer, or the combination of "convolution" and "pooling", or the combination of "ReLu" and "pooling" may be made into one layer. . Note that the ReLu layer is an example of an activation layer, and other activation functions may be used.

Note that in a convolutional neural network, a fully connected layer is generally used in the latter half of the intermediate layer, but the fully connected layer may be operated in the encryption execution area.

(Processing flow)
FIG. 9 is a flowchart illustrating an example of processing executed by the machine learning device 100.

The control unit 130 acquires data such as image data, or data such as image data and a label, from the terminal device 200 via the communication unit 110 (step S101).

The control unit 130 acquires, from the terminal device 200 via the communication unit 110, layer designation information that specifies the Nth layer, which is the boundary between the calculation performed in the cryptographic execution area and the calculation performed in the plaintext execution area (step S102).

The control unit 130 encrypts the acquired data such as images and the learning model to be applied using homomorphic encryption. Then, the 0th layer to the Nth layer are calculated in the encryption execution area, and then decrypted (step S103).

The control unit 130 performs calculations from the (N+1)th layer to the output layer in the plaintext execution area based on the calculation result decoded in step S103, and updates the parameters of the inference or learning model. Then, the control unit 130 stores the calculation result in the storage unit 120, and ends the process (step S104).

(Explanation of effects)
The machine learning device 100 according to the present embodiment calculates some of the calculations in machine learning in the encryption execution area and other parts in the plaintext execution area, compared to a case where all calculations are performed in the encryption execution area. This reduces computational costs and allows calculations to be performed in a realistic amount of time.

In machine learning models using convolutional neural networks, filtering of convolution operations is generally repeated in the processing process of the middle layer, so the image data input to the input layer becomes difficult to visually identify. It's becoming. That is, since it is difficult for humans to identify input data after repeated convolution operations, the risk of information leakage is low even if it is not encrypted. The machine learning device 100 according to the present embodiment focuses on this point, and calculates up to the Nth layer specified by the user in the encryption execution area, and calculates the (N+1)th and subsequent layers in the plaintext execution area. It is possible to reduce calculation costs and ensure security.

<Embodiment 2>
The machine learning system 1 according to the first embodiment performs calculation processing in the encryption execution area and the plaintext execution area after acquiring layer designation information from the user. In contrast, when the user specifies layer specification information, the machine learning system 2 according to the second embodiment calculates how the input image changes due to the convolution operation when the input image is calculated in the plaintext execution domain. , provide users with information such as how computational costs change. The user can refer to the provided information and easily specify layer designation information according to the calculation cost and security level desired by the user.

(Configuration of machine learning system 2)
FIG. 10 is a diagram showing the machine learning system 2 according to this embodiment. As shown in FIG. 10, the machine learning system 2 according to this embodiment includes a machine learning device 300 and one or more terminal devices 200 used by a user. Compared to the machine learning device 100 according to the first embodiment, the machine learning device 300 further includes a providing section 3324 and an estimating section 3325 (described in detail in FIG. 11). The providing unit 3324 allows the user to select a range in which to perform calculations in the cryptographic execution domain by showing information such as how the image data input by the user changes when calculated in the plaintext execution domain and the calculation cost. Equipped with functions to assist.

(Functional configuration of machine learning device 300)
FIG. 11 is a block diagram showing the functional configuration of the machine learning device 300 in this embodiment. The machine learning device 300 includes a communication section 110, a storage section 120, and a control section 330. The functional configurations of the communication unit 110 and the storage unit 120 in the machine learning device 300 are the same as those in the machine learning device 100, so the description thereof will be omitted.

The control unit 330 is realized by the processor of the machine learning device 300 reading a program stored in the storage unit 120 and executing instructions included in the program. The control unit 330 controls the operation of the machine learning device 300. Specifically, the control unit 330 includes a data acquisition unit 131, a machine learning calculation unit 332, a reception unit 1321, an encryption processing unit 1322, a plaintext processing unit 1323, a provision unit 3324, an estimation unit 3325, including. The control unit 330 is the same as the control unit 130 in the machine learning device 100 except that it includes a providing unit 3324 and an estimating unit 3325, so a duplicate description will be omitted.

The providing unit 3324 provides plaintext intermediate operation information regarding the operation when the plaintext processing unit 1323 executes the operation from the input layer to the output layer on the image data input by the user that is acquired by the data acquisition unit 131. . Plaintext intermediate operation information is information regarding operations during the intermediate process from the input layer to the output layer, and includes data input or output in the intermediate layer in the plaintext execution area, calculation time required up to the intermediate layer (and, This includes the total amount of time required for inference or learning estimated from this), the amount of calculation required up to intermediate layers (and the total amount of calculation required for inference or learning estimated from this), and fees calculated from the amount of calculation. It is something that The plaintext intermediate operation information can be calculated, for example, by the plaintext processing unit 1323 performing an operation from the input layer to the output layer on one piece of image data input by the user, and based on information regarding the operation. The plaintext intermediate calculation information may be transmitted to the terminal device 200 used by the user via the communication unit 110, for example, or may be displayed on a display device (not shown) in the machine learning device 100.

The providing unit 3324 provides, for example, the difference in information between the image data input to the input layer and the data input to each layer from the input layer to the output layer in a manner that allows the user to compare the information as plain text intermediate calculation information. You may. Further, the providing unit 3324 may provide the image data input to each layer as plain text intermediate calculation information, without necessarily providing the image data in a form that the user can compare.

The providing unit 3324 assumes that visual identification becomes difficult due to repeated convolution operations, and by providing image data input to each layer to the user, the image data input to which layer can be visually identified. It provides reference information (differences in information) regarding whether identification is possible, and sometimes provides information for determining which layers need to be encrypted and subjected to machine learning by performing a comparison. For example, regarding sample image data, the user may be presented with data input into a first layer, which is a certain intermediate layer, and a second layer, which is processed after the first layer. . This allows the user to compare how sample image data changes and differences in information occur due to layer processing, and assists the user in specifying layer designation information.

The providing unit 3324 does not have to provide all the input image data from the input layer to the output layer, and when the cryptographic processing unit 1322 performs calculations on a plurality of predetermined layers from the input layer to the output layer, The image data inputted when the plaintext processing unit 1323 performs the calculation may be provided in a form that the user can compare. At this time, the plurality of predetermined layers may be selected periodically, such as by extracting one layer out of every five layers, or the plurality of layers may be selected at random.

The providing unit 3324 may further provide a mask image in place of the image data input to an intermediate layer when the encryption processing unit 1322 performs the calculation. In other words, the image data that is input when the cryptographic processing unit 1322 performs the calculation is encrypted and therefore cannot be identified by humans. By providing a mask image to the data, it may be possible to indicate that the data is indiscernible to humans.

For example, the providing unit 3324 provides the user with the calculation cost when the encryption processing unit 1322 performs calculations from the input layer to a predetermined layer, and the plaintext processing unit 1323 performs calculations from the layer next to the predetermined layer to the output layer. may be provided in a form that allows for comparison. The providing unit 3324 provides the user with the calculation time and/or amount of calculation when calculating up to a predetermined layer in the encryption execution area and calculating the remaining layers in the plaintext execution area, thereby encrypting and performing machine learning. Information can be provided to determine which layers need to be covered.

The providing unit 3324 may provide layer designation estimation information calculated by the estimating unit 3325 (described later). The layer designation estimation information is information calculated based on the correlation between the data related to the image data acquired by the data acquisition unit 131 and the layer designation information specified for the image data. The layer designation estimation information may be, for example, a numerical value that specifically specifies the N-th layer, or may be a range of N.

The estimation unit 3325 learns the correlation between the data related to the image data acquired by the data acquisition unit and the layer designation information specified for the image data. Data related to image data includes industries that handle image data (medical, food, EC, etc.), contents (diagnosis, marketing, etc.), organizations, users, and the like. The estimation unit 3325 learns how many layers to encrypt based on data related to image data and the correlation. Then, as a result of the learning, the estimation unit 3325 calculates layer designation information that has a correlation with data related to the input image data as layer designation estimation information. As described above, the providing unit 3324 provides the layer designation estimation information calculated by the estimation unit 3325 to the user together with the plain text intermediate calculation information, thereby providing a guideline for designating the layer designation information. This allows the user to easily specify the layer specification information.

(Specific example of plaintext intermediate operation information provided by the providing unit 3324)
FIG. 12 is a diagram showing an example of a screen provided by the providing unit 3324. A screen 402 shows image data (an image showing a "cow") that is input to each of the five layers of the calculation process when the plaintext processing unit 1323 executes the calculation. An icon 412 indicating that the range to be performed is "selecting" is displayed.

In FIG. 12, the input image input by the user is displayed in the "Input" portion. In addition, input images in a plurality of layers (calculation processes 1 to 5) are displayed on the screen 402 as plain text intermediate calculation information. Generally, in a convolutional neural network, as the filtering process of convolution operations is repeated, the input image becomes difficult to visually identify, so the deeper the input image is in the layer, the more difficult it is to identify the input image. For example, the input image (Input) on the screen 402 is a cow, and the Input and the image shown in calculation process 1 are almost indistinguishable from each other visually. On the other hand, the images shown in calculation steps 3 to 5 do not have the outline of the cow, and it is understood that it is difficult to visually identify the objects shown in the images.

In addition, the screen 402 shows not only the image data input to multiple layers, but also the calculation time ("Time") when the intermediate layers are computed in the encryption execution area and the subsequent ones are computed in the plaintext execution area. and costs (“Cost”) are also shown. For example, for calculation process 1, "Time" is "5'36" and "Cost" is "$4". In other words, it is shown that the calculation time is "5'36" and the cost is "$4" when calculations up to calculation step 1 are performed in the cryptographic execution area and subsequent calculations are performed in the plaintext execution area. As the processing layer in the convolutional neural network becomes deeper (proceeding from calculation process 1 to 5), the amount of calculation to be performed in the cryptographic execution area increases, and therefore calculation time and cost also increase. Note that in FIG. 12, calculation time and expenses are presented as calculation costs, but the amount of calculation required for calculations, calculation resources, etc. may also be presented to the user as calculation costs.

In addition, by clicking the button 422, the user can select another image, a sample image prepared in advance on the platform side, an illustration showing the difference in the amount of information, etc., instead of the above-mentioned input image of "cow". may be changed as an input image. In this case, plaintext intermediate calculation information is provided based on these sample images.

FIG. 13 is a diagram showing an example of a screen provided by the providing unit 3324. An area 425 of the screen 403 indicates that the user has selected "calculation process 2" as the range for performing calculations in the cryptographic execution area. Further, the user can input "calculation process 2" as the layer designation information by clicking the "determination" button 413 or the like. Note that the layer designation may be performed, for example, by clicking on the image data shown in each calculation process or the icon (key icon in FIG. 13) shown below the image data. However, it may also be performed using a keyboard or voice.

In FIG. 13, image data in layers up to calculation process 2, which is the layer selected by the user, is shown as a mask image. When calculating up to a certain layer in the cryptographic execution area, the image data input up to that layer is encrypted, making it impossible for humans to identify it, so the encrypted image data is intentionally presented to the user. There is little need. By displaying the mask image instead, the number of calculations in the encryption execution area can be reduced. However, this does not preclude displaying the calculation results in the cryptographic execution area so that the user can easily visualize the calculation results.

(Processing flow)
FIG. 14 is a flowchart illustrating an example of processing executed by the machine learning device 300.

The control unit 130 acquires data such as image data, or data such as image data and a label, from the terminal device 200 via the communication unit 110 (step S201).

The control unit 130 provides the terminal device 200, via the communication unit 110, with plaintext intermediate operation information regarding the operation executed by the plaintext processing unit 1323 in each layer from the input layer to the output layer (step S202). .

The control unit 130 acquires, from the terminal device 200 via the communication unit 110, layer designation information that specifies the Nth layer, which is the boundary between the calculation performed in the cryptographic execution area and the calculation performed in the plaintext execution area (step S203).

The control unit 130 encrypts the acquired data such as images and the learning model to be applied using homomorphic encryption. Then, the 0th layer to the Nth layer are calculated in the encryption execution area, and then decrypted (step S204).

The control unit 130 performs calculations from the (N+1)th layer to the output layer in the plaintext execution area based on the calculation result decoded in step S204, and updates the parameters of the inference or learning model. Then, the control unit 130 stores the calculation result in the storage unit 120, and ends the process (step S205).

(Explanation of effects)
The machine learning device 300 according to the present embodiment assumes that a part of the computation in machine learning is performed in the cryptographic execution domain and the other part is computed in the plaintext execution domain. , provides plaintext intermediate operation information to the user. By referring to the plaintext intermediate calculation information, the user can select a boundary layer according to the practical calculation cost and security level required by the user.

<Embodiment 3>
In Embodiments 1 and 2, part of the calculation process using the convolutional neural network is performed in the cryptographic execution area until it becomes difficult to visually identify the image. Thereby, even if an image of the calculation process is leaked to a third party, the image can be kept secret. On the other hand, for example, if the cryptographic processing unit and the plaintext processing unit can be executed with the same authority, if a malicious third party seizes that authority, the image of the calculation process in the cryptographic processing unit can be decrypted. There is a risk that an attempt will be made to restore the input image using the key.

Therefore, the machine learning device according to Embodiment 3 improves security by separating authority between a cryptographic processing unit that performs calculations in the cryptographic execution area and a plaintext processing unit that performs calculations in the plaintext execution area. Even if a malicious third party were to seize the authority of the encryption processing unit, unless they have the authority of the plaintext processing unit, they would not be able to access the decryption key, so they would be unable to decrypt the encrypted image. Recognizing the input image is difficult. Furthermore, even if a malicious third party seizes the authority of the plaintext processing unit, the calculation process in the convolutional neural network proceeds through the encryption processing unit and then to the plaintext processing unit, so the input image Restoration is more difficult.

In order to realize the above configuration, in the machine learning device according to the present embodiment, when the processing performed by the machine learning system 1 or 2 described in Embodiment 1 or 2 is executed on one computer, the cryptographic processing unit is Authority is divided between the processing to be executed and the processing executed by the plaintext processing section, and data is transferred from the cryptographic processing section to the plaintext processing section via a shared memory. This prevents a malicious third party who has taken over the authority for one process from accessing other processes, and increases security in the event of hacking by a malicious third party.

Furthermore, in recent years, in image processing using convolutional neural networks, research has been conducted in which attempts are made to restore the original image from model information and images in the middle of calculation. Therefore, even if the image is encrypted or difficult to visually identify, it is important to reduce the risk of leakage of images that are still being calculated. As mentioned above, the machine learning device according to this embodiment transfers data from the encryption processing unit to the plaintext processing unit via the shared memory, so it is also possible to reduce the possibility of image leakage during calculation. The configuration is more security-friendly.

(Functional configuration of machine learning device 400)
FIG. 15 is a block diagram showing the functional configuration of the machine learning device 400 according to this embodiment. The functional configuration of the machine learning device 400 according to this embodiment will be described with reference to FIG. 15. Note that the same components as in Embodiment 1 are given the same reference numerals, and repeated explanations will be omitted.

The machine learning device 400 includes a communication unit 110, a storage unit 120, and a control unit 430, and the control unit 430 includes a data acquisition unit 131, a shared memory 431, and a machine learning unit 432. The machine learning section 432 includes a reception section 1321, a cryptographic processing section 4322, and a plaintext processing section 4323.

The shared memory 431 is, for example, a volatile memory, and is accessible by a cryptographic processing unit 4322 (described later) and a plaintext processing unit 4323 (described later). The cryptographic processing unit 4322 and the plaintext processing unit 4323 store the results (data) of the executed operations in the shared memory 431. Further, the data acquisition unit 131 may read out keys, model information, etc. necessary for calculations performed by the encryption processing unit 4322 and the plaintext processing unit 4323 from the storage unit 120 and store them in the shared memory 431. Thereby, the encryption processing unit 4322 and the plaintext processing unit 4323 of the machine learning device 400 can share data in the calculation process via the shared memory 423.

The cryptographic processing unit 4322 first encrypts the image data etc. acquired by the data acquisition unit 131 using homomorphic encryption using the encryption key (public key) managed in the key storage unit 121. Then, using the encrypted image data and the learning model in the learning model storage unit 122, calculation processing from the input layer (0th layer) to the Nth layer of the intermediate layer specified by the user is performed in the encryption execution area. calculate. Note that if the data acquisition unit 131 has acquired image data encrypted by homomorphic encryption, the above-mentioned encryption by the encryption processing unit 4322 is omitted.

In addition, before the calculation, the data acquisition unit 131 receives the encryption key (public key) from the key storage unit 121 and the model information (first model information) may be read out and stored in the shared memory 431. The cryptographic processing unit 4322 can speed up the processing speed by reading out the key and model information from the shared memory 431 and using it in calculations.

Next, the cryptographic processing unit 4322 stores the results of the calculations from the input layer (0th layer) to the Nth layer in the shared memory 431. At this time, the cryptographic processing unit 4322 not only collects the results of the calculations up to the Nth layer, but also all the data of the calculation process in each layer from the input layer (layer 0) to the Nth layer, or a part of the calculation process. The data may be stored in shared memory 431. At this time, the result of the calculation may be temporarily output to an external storage device and then stored in the shared memory 431. Note that after the cryptographic processing unit 4322 completes the computation, it is desirable that the first model information used in the computation by the cryptographic processing unit 4322 be deleted from the shared memory 431. As described above, the original image can be restored from the model information and the image that is currently being calculated. For this reason, if a malicious third party seizes the authority of the plaintext processing unit, there is a risk that the original input image may be attempted to be restored from the model information and the image in the middle of calculation. Therefore, in order to make it difficult to restore the original input image, it is important to make the first model information, which is part of the model information, inaccessible to anyone who can decode the calculation result.

The plaintext processing unit 4323 obtains the results of the calculations up to the Nth layer executed by the encryption processing unit 4322 from the shared memory 431, and performs the calculations from the (N+1)th layer to the output layer based on the obtained calculation results. I do. At this time, the plaintext processing unit 4323 uses the learning model in the learning model storage unit 122 to perform calculations from the (N+1)th layer to the output layer. In addition, the plaintext processing unit 4323 decrypts the calculation results up to the Nth layer obtained by the encryption processing unit 4322 obtained from the shared memory 431 using the decryption key (private key) managed in the key storage unit 121. do.

Note that before the plaintext processing unit 4323 obtains the calculation result of the cryptographic processing unit 4322, the data acquisition unit 131 obtains the decryption key (private key) from the key storage unit 121 and the private key from the learning model storage unit 122. Model information (second model information) regarding the (N+1) layer to the output layer may be read out and stored in the shared memory 431. The plaintext processing unit 4323 can speed up the processing speed by reading out the key and model information from the shared memory 431 and using it in calculations.

Furthermore, it is desirable to change the key used for encryption and decryption by discarding it each time a process is performed or every fixed period of time. This makes it difficult to decode the original input image or the image that is being calculated, thereby increasing security.

The plaintext processing unit 4323 uses the second model information to perform calculations from the (N+1)th layer to the output layer in the plaintext execution area, thereby performing inference or updating parameters of the learning model.

(Specific example of processing using shared memory)
FIG. 16 is a diagram showing a correspondence relationship between a functional block diagram and hardware regarding the shared memory 431. The shared memory 431 is a storage area that can be accessed from both the encryption processing unit 4322 and the plaintext processing unit 4323. The cryptographic processing unit 4322 and the plaintext processing unit 4323 execute processing as a function of a processor. The shared memory 431 also stores keys managed in the key storage unit 121, the results of calculations performed by the cryptographic processing unit 4322, and the results of calculations performed by the plaintext processing unit 4323.

The cryptographic processing unit 4322 stores the result of the calculation in the shared memory 431. The plaintext processing unit 4323 obtains the result of the operation performed by the encryption processing unit 4322 from the shared memory 431. At this time, the plaintext processing unit 4323 obtains a decryption key (private key) from the key storage unit 121 and decrypts the result of the operation performed by the encryption processing unit 4322.

FIG. 17 is a diagram showing a case where execution authority is divided using account control. If you want to separate the execution authority of the cryptographic processing unit 4322 and the plaintext processing unit 4323 within one computer, for example, prepare two accounts (account 1 and account 2) and assign one execution authority to each account. This can be achieved with Specifically, as shown in FIG. 17, the authority to execute the cryptographic processing unit 4322 is granted to account 1, and the authority to execute the plaintext processing unit 4323 is granted to account 2. Note that account 1 and account 2 are given access authority to the shared memory 431 (authority to read and write data).

In the machine learning device 400, by separating accounts, the execution authority of the encryption processing unit 4322 and the execution authority of the plaintext processing unit 4323 can be separated within one computer. Accordingly, for example, account 1 does not have the authority to execute the plaintext processing unit 4323, and therefore does not have the authority to acquire the decryption key. In other words, even if a malicious third party seizes Account 1, which has the authority to use the cryptographic processing unit 4322, Account 1 cannot obtain the decryption key from the key storage unit 121, so the data such as images that have been illegally obtained cannot be accessed. Decoding is difficult. Furthermore, even if a malicious third party seizes the account 2 that has authority over the plaintext processing unit 4323, it will be difficult to restore the original input image because the calculation process of the convolutional neural network is still progressing.

Note that in order to make it difficult to restore the input image, it is desirable to delete the first model information from the shared memory 431 after the cryptographic processing unit 4322 completes the calculation. As described above, the original image can be restored from the model information and the image that is currently being calculated. For this reason, if a malicious third party seizes the account 2 that has authority over the plaintext processing unit, there is a risk of attempting to restore the original input image. Therefore, the first model information is more secure by preventing account 2 from accessing it.

Furthermore, the machine learning device 400 can prevent leakage of data in the calculation process by using the shared memory 431 that can be commonly accessed by the encryption processing unit 4322 and the plaintext processing unit 4323. Furthermore, it is also possible to prevent a malicious third party from retroactively obtaining the results of the calculation process. In other words, if you share information via a storage device such as a magnetic disk device, there is a risk that if your account is hijacked by a malicious third party, the results of past calculations will be obtained. By using the shared memory 431 as the shared memory 431, the results of calculations by the cryptographic processing unit 4322 and plaintext processing unit 4323 are only temporarily stored, preventing a malicious third party from illegally obtaining the results of past calculations. can be prevented.

(Processing flow)
FIG. 18 is a flowchart illustrating an example of processing executed by the machine learning device 400.

The control unit 430 acquires data such as image data, or data such as image data and a label (step S301).

The control unit 430 acquires layer designation information that designates the Nth layer that is the boundary between the calculation performed in the cryptographic execution area and the calculation performed in the plaintext execution area (step S302).

The control unit 430 encrypts the acquired data such as images and the learning model to be applied using homomorphic encryption. Then, calculations are performed in the encryption execution area from the input layer (0th layer) to the Nth layer and stored in the shared memory 431 (step S303).

The control unit 430 obtains the result of the calculation performed in step S303 from the shared memory 431, and decrypts the result of the calculation using the decryption key based on the result of the calculation. Then, based on the results of the decrypted calculations, calculations are performed in the plaintext execution area from the (N+1)th layer to the output layer, and inference or learning model parameters are updated. Next, the control unit 430 outputs the results of the calculations up to the output layer, and ends the process (step S304).

(Explanation of effects)
The machine learning device 400 according to the present embodiment separates authority for processing executed by the encryption processing unit 4322 and processing executed by the plaintext processing unit 4323, and transfers information from the encryption processing unit 4322 to the plaintext processing unit 4323 using a shared memory. 431. This prevents a malicious third party who has taken over the authority of the cryptographic processing unit 4322 or the plaintext processing unit 4323 from accessing other processing, thereby increasing security.

Furthermore, the machine learning device 400 uses a shared memory 431 that can be accessed by both the encryption processing unit 4322 and the plaintext processing unit 4323, thereby making it possible to share the results of calculations. By using the shared memory 431 configured as a volatile memory, the results of calculations can be stored in the short term and past data can be overwritten, even if a malicious third party attempts unauthorized access. It is possible to increase security by making it difficult to obtain data that would lead to restoration of the original image, etc.

Furthermore, in order to make it difficult to restore the input image, the first model information is deleted from the shared memory 431 after the encryption processing unit 4322 completes the calculation, so that the first model information and the plaintext processing unit 4323 can decrypt it. Based on the calculation results up to the Nth layer, it is possible to prevent restoration to the original input image, thereby further increasing security.

<Embodiment 4>
Embodiment 4 is a modification of Embodiment 3. The machine learning device according to the present embodiment has separate access authority for learning models compared to the machine learning device according to the third embodiment. The machine learning device according to this embodiment not only separates the execution authority of the cryptographic processing unit and the execution authority of the plaintext processing unit, but also separates the access authority to the model information of the learning model, so that a malicious third party can To make it more difficult to restore an original image, etc. even if a part of authority is stolen.

FIG. 19 is a block diagram showing the functional configuration of the machine learning device 500 according to this embodiment. The functional configuration of the machine learning device 500 according to the fourth embodiment will be described below with reference to FIG. 19. Note that the same components as in the third embodiment are denoted by the same reference numerals, and repeated explanations will be omitted.

The machine learning device 500 includes a communication section 110, a storage section 520, and a control section 530. The storage unit 520 includes a key storage unit 121 and a learning model storage unit 522, and the control unit 530 includes a data acquisition unit 531, a shared memory 532, and a machine learning calculation unit 533. The machine learning calculation unit 533 includes a reception unit 1321, a cryptographic processing unit 5322, and a plaintext processing unit 5323.

The learning model storage unit 522 stores first model information from the input layer to the Nth layer, and second model information from the (N+1)th layer to the output layer. The first model information is information on a learning model that is necessary for calculation in the cryptographic processing unit 5322 among the learning models. The second model information is learning model information that is necessary for calculation in the plaintext processing unit 5323, out of the learning model information.

The first model information and the second model information are stored, for example, in the same storage area, and access authority is set. The first model information can be accessed by a person who has execution authority for the cryptographic processing unit, and the second model information can be accessed by a person who has execution authority for the plaintext processing unit. Further, the first model information and the second model information may be stored in different storage areas. In this case, the first model information can be accessed by a person who has execution authority for the cryptographic processing unit, and the second model information can be accessed by a person who has execution authority for the plaintext processing unit.

The cryptographic processing unit 5322 acquires data such as an image acquired through the data acquisition unit 531 and the first model information, and performs calculations from the input layer to the Nth layer in the cryptographic execution area. The cryptographic processing unit 5322 then stores the calculation result in the shared memory 532.

The plaintext processing unit 5323 obtains the results of the calculations up to the Nth layer executed by the encryption processing unit 5322 from the shared memory 532. Then, using the second model information acquired via the data acquisition unit 521, calculations from the (N+1)th layer to the output layer in the plaintext execution area are performed, and thereby the parameters of the inference or learning model are updated. . At this time, the plaintext processing unit 5323 decrypts the results of the calculations up to the Nth layer using the decryption key (private key) managed in the key storage unit 121.

Note that each model information may be stored in a device other than the machine learning device 500, and the data acquisition unit 531 may separately obtain each model information from a device other than the machine learning device 500.

(Explanation of effects)
The machine learning device 500 according to this embodiment not only separates the authority for processing executed by the cryptographic processing unit 5322 and the processing executed by the plaintext processing unit 5323, but also separates the authority for accessing the learning model. As a result, even if a malicious third party seizes part of the execution authority, the amount of information that can be acquired can be reduced compared to the machine learning device 400 according to the third embodiment. Therefore, it is made more difficult to restore the original image, etc., and security can be improved.

<Embodiment 5>
The machine learning device according to the third embodiment separates the execution authority of the encryption processing unit and the plaintext processing unit in one computer, so that even if a malicious third party seizes some authority, the original image etc. This makes restoration more difficult and increases security. The machine learning system according to Embodiment 5 is configured such that the encryption processing unit and the plaintext processing unit in the machine learning system described in Embodiment 1 or 2 are executed by different terminal devices. By executing calculations in physically different execution environments, even if a malicious third party seizes authority in a part of the execution environment, it is difficult to restore the original image, etc., thereby increasing security. By performing calculations in physically different execution environments, the separation of execution authority can be made more reliable and security can be improved, compared to dividing execution authority in one computer.

(Configuration of machine learning system 3)
FIG. 20 is an overall diagram of the machine learning system 3. As shown in FIG. 20, the machine learning system 3 according to the present embodiment includes a cryptographic processing device 600 that performs calculations in the cryptographic execution area, a plaintext processing device 700 that performs calculations in the plaintext execution area, and the rest of the machine learning system. and a system management device 800 that performs processing for this purpose.

The cryptographic processing device 600, the plaintext processing device 700, and the system management device 800 can transmit and receive data via the network NW.

The functions of the machine learning system 3 are divided into the functions of the machine learning device 100 according to the first embodiment into a cryptographic processing device 600, a plaintext processing device 700, and a system management device 800. The cryptographic processing device 600, the plaintext processing device 700, and the system management device 800 communicate with each other via the network NW, thereby realizing the functions of the machine learning device 100 by dividing the functions into three devices.

Specifically, the functions of the machine learning system 3 can be realized with the following configuration. For example, the functions of the key storage unit 121 and learning model storage unit 122 in the storage unit 120 of the machine learning device 100 according to the first embodiment are functionally arranged in the system management device 800. The functions of the data acquisition unit 131 in the control unit 130 of the machine learning device 100 and the cryptographic processing unit 1322 of the machine learning calculation unit 132 are functionally arranged in the cryptographic processing device 600. The functions of the plaintext processing unit 1323 of the machine learning calculation unit 132 in the control unit 130 of the machine learning device 100 are functionally arranged in the plaintext processing device 700. The function of the reception unit 1321 of the machine learning calculation unit 132 in the control unit 130 of the machine learning device 100 is functionally arranged in the system management device 800.

However, the above-mentioned example is just an example, and the functions of the machine learning system 3 are such that at least the function of the cryptographic processing section 1322 is arranged in the cryptographic processing device 600 and the function of the plaintext processing section 1323 is arranged in the plaintext processing device 700. Bye. Other functions may be placed in either the cryptographic processing device 600, the plaintext processing device 700, or the system management device 800.

Further, for example, the functions of the key storage unit 121 and the learning model storage unit 122 in the storage unit 120 of the machine learning device 100 according to the first embodiment are functionally arranged in both the cryptographic processing device 600 and the plaintext processing device 700. Good too.

Further, for example, the function of the data acquisition unit 131 in the control unit 130 of the machine learning device 100 according to the first embodiment may be functionally arranged in the system management device 800. The function of the reception unit 1321 of the machine learning device 100 may be functionally arranged in the cryptographic processing device 600.

(Functional configuration of cryptographic processing device 600)
FIG. 21 is a block diagram showing the functional configuration of a cryptographic processing device 600 according to this embodiment. The functional configuration of the cryptographic processing device 600 according to this embodiment will be described below with reference to FIG. 21.

The cryptographic processing device 600 includes a communication section 610, a storage section 620, and a control section 630.

The communication unit 610 performs processing for the cryptographic processing device 600 to communicate with other devices. The communication unit 610 establishes a secure communication channel for communication with other devices.

The storage unit 620 includes, for example, a volatile memory such as a RAM, a flash memory, an HDD, an SSD, etc., and stores data and programs used by the cryptographic processing device 600.

The control unit 630 is realized by the processor of the cryptographic processing device 600 reading a program stored in the storage unit 620 and executing instructions included in the program. Control unit 630 controls the operation of cryptographic processing device 600. Specifically, the control unit 630 includes a data acquisition unit 631 and a machine learning calculation unit 632.

The data acquisition unit 631 acquires image data, etc. input by the user and a label.

The machine learning calculation unit 632 performs calculation processing using a convolutional neural network based on the image data etc. acquired by the data acquisition unit 631. Specifically, this calculation process is a calculation process for learning to update the parameters of a learning model or for inference using a learning model from acquired image data or the like. Note that a convolutional neural network is composed of multiple layers.

The machine learning calculation unit 632 includes a cryptographic processing unit 6322. The cryptographic processing unit 6322 encrypts image data and the like by homomorphic encryption using an encryption key obtained from the key storage unit 821 (described later) of the system management device 800 via the communication unit 610. Note that if the data acquisition unit 631 has acquired image data encrypted by homomorphic encryption, the encryption is omitted.

The cryptographic processing unit 6322 receives the model information obtained from the learning model storage unit 822 (described later) of the system management device 800 via the communication unit 610 and the reception unit 831 ( (described later) uses the acquired layer designation information to perform calculations from the input layer to the Nth layer in the encryption execution area using homomorphic encryption.

The cryptographic processing unit 6322 transmits the result of the calculation in the cryptographic execution area to the plaintext processing device 700 via the communication unit 610. The plaintext processing device 700 obtains the result of the calculation in the cryptographic execution area via the communication unit 710.

(Functional configuration of plaintext processing device 700)
FIG. 22 is a block diagram showing the functional configuration of the plaintext processing device 700 according to this embodiment. The functional configuration of the plaintext processing device 700 according to this embodiment will be described below with reference to FIG. 22.

Plaintext processing device 700 includes a communication section 710, a storage section 720, and a control section 730.

The communication unit 710 performs processing for the plaintext processing device 700 to communicate with other devices. The communication unit 710 establishes a secure communication channel for communication with other devices.

The storage unit 720 includes, for example, a volatile memory such as a RAM, a flash memory, an HDD, an SSD, etc., and stores data and programs used by the plaintext processing device 700.

The control unit 730 is realized by the processor of the plaintext processing device 700 reading a program stored in the storage unit 720 and executing instructions included in the program. The control unit 730 controls the operation of the plaintext processing device 700. Specifically, the control unit 730 includes a machine learning calculation unit 732.

The machine learning calculation unit 732 performs calculation processing using a convolutional neural network based on the result of calculation in the cryptographic execution area obtained from the cryptographic processing device 600 via the communication unit 710. Specifically, this calculation process is a calculation process for learning to update the parameters of a learning model or for inference using a learning model from acquired image data or the like. Note that a convolutional neural network is composed of multiple layers.

The machine learning calculation unit 732 includes a plaintext processing unit 7323. The plaintext processing unit 7323 decrypts the result of the operation in the encryption execution area using a decryption key obtained from the key storage unit 821 (described later) of the system management device 800 via the communication unit 710.

The plaintext processing unit 7323 uses the model information acquired from the learning model storage unit 822 (described later) of the system management device 800 via the communication unit 710 and the model information acquired from the learning model storage unit 822 (described later) of the system management device 800 via the communication unit 710. ) is used to perform calculation processing from the (N+1)th layer to the output layer in the plaintext execution area. This updates the parameters of the inference or learning model.

The plaintext processing unit 7323 performs inference or updates parameters of the learning model, transmits the learning model to the system management device 800 via the communication unit 710, and stores it in the learning model storage unit 822 (described later) of the system management device 800. Update the stored learning model.

(Functional configuration of system management device 800)
FIG. 23 is a block diagram showing the functional configuration of the system management device 800 according to this embodiment. The functional configuration of the system management device 800 according to this embodiment will be described below with reference to FIG. 23.

System management device 800 includes a communication section 810, a storage section 820, and a control section 830.

The communication unit 810 performs processing for the system management device 800 to communicate with other devices. The communication unit 810 establishes a secure communication channel for communication with other devices.

The storage unit 820 is configured with, for example, a volatile memory such as a RAM, a flash memory, an HDD, an SSD, etc., and stores data and programs used by the system management device 800. The storage unit 820 includes a key storage unit 821 and a learning model storage unit 822.
The key storage unit 821 stores a key for encrypting data or decrypting encrypted data. The learning model storage unit 822 stores learning models used in the machine learning system 3.

The control unit 830 is realized by the processor of the system management device 800 reading a program stored in the storage unit 820 and executing instructions included in the program. The control unit 830 controls the operation of the system management device 800. Control unit 830 includes a reception unit 831.

The reception unit 831 receives layer designation information that designates the Nth layer (N is a natural number), which is a layer on the way from the input layer to the output layer of the convolutional neural network.

System management device 800 transmits layer designation information to cryptographic processing device 600 and/or plaintext processing device 700 via communication unit 810 .

(Specific example of performing calculations in physically different execution environments)
FIG. 24 is a diagram illustrating a specific example of performing calculation processing using a convolutional neural network in physically different execution environments.

In the machine learning system 3 according to the present embodiment, calculation processing using a neural network is performed in a cryptographic execution area using homomorphic encryption and a plaintext execution area. Processing in the cryptographic execution area is performed in the cryptographic processing device 600, and processing in the plaintext execution area is performed in the plaintext processing device 700, thereby performing calculation processing in physically different execution environments.

As shown in FIG. 24, in the cryptographic processing device 600, the cryptographic processing unit 6322 encrypts the image data etc. acquired by the data acquisition unit 631 (see FIG. 21), and in the cryptographic execution area, the cryptographic processing unit 6322 encrypts the image data etc. acquired by the data acquisition unit 631 (see FIG. 21). The calculations from to the Nth layer are executed. The cryptographic processing device 600 transmits the results of calculations from the input layer to the Nth layer to the plaintext processing device 700 via the network. In the plaintext processing device 700, the plaintext processing unit 7323 decrypts the result of the operation sent from the cryptographic processing device 600, and then performs the operations from the (N+1)th layer to the output layer in the plaintext execution area. In this way, the machine learning system 3 allows the cryptographic processing unit 6322 and the plaintext processing unit 7323 to process calculations in different execution environments, thereby preventing the computer from being hacked by a malicious third party. It can reduce risks and increase security.

(Processing flow)
FIG. 25 is a sequence diagram illustrating an example of a process executed by the machine learning system 3. Note that the order of steps is not limited to the example shown in FIG. 25.

The cryptographic processing device 600 acquires data such as image data, or data such as image data and a label (step S401).

The system management device 800 acquires layer designation information that designates the Nth layer that is the boundary between the calculation performed in the cryptographic execution area and the calculation performed in the plaintext execution area (step S402).

The system management device 800 transmits the layer designation information, the learning model, and the key for encryption (encryption key) to the cryptographic processing device 600 (step S403).

The cryptographic processing device 600 encrypts the acquired data such as images and the learning model to be applied using homomorphic encryption. Then, calculations are performed in the encryption execution area from the 0th layer to the Nth layer (step S404).

The cryptographic processing device 600 transmits the calculation result in the cryptographic execution area to the plaintext processing device 700 (step S405).

The system management device 800 transmits the layer designation information, the learning model, and the decryption key (private key) to the plaintext processing device 700 (step S406). Note that the decryption key may be shared in advance between the plaintext processing device 700 and the system management device 800, in addition to being transmitted over a secure communication path.

Based on the result of the calculation performed by the cryptographic processing device 600, the plaintext processing device 700 decrypts the calculation result using the decryption key. Then, based on the results of the decrypted calculations, calculations are performed in the plaintext execution area from the (N+1)th layer to the output layer, and the parameters of the inference or learning model are updated (step S407).

The plaintext processing device 700 transmits the updated learning model to the system management device 800, and the system management device 800 updates the stored parameters of the learning model (step S408).

(Explanation of effects)
The machine learning system 3 according to this embodiment performs processing executed by the cryptographic processing unit 6322 and processing executed by the plaintext processing unit 7323 in separate physical execution environments. This prevents a malicious third party who has hacked the execution environment of the cryptographic processing unit 6322 or the plaintext processing unit 7323 from accessing other processes and prevents them from acquiring the original image, etc., thereby improving security. can be increased. By dividing the physical execution environment, the machine learning system 3 can make access to other processes more difficult and improve security than by dividing execution authority on one computer.

<Embodiment 6>
Embodiment 6 is a modification of Embodiment 5. The machine learning system 3 according to the fifth embodiment acquires the learning model stored in the learning model storage unit 822 of the system management device 800 in both the cryptographic processing device 600 and the plaintext processing device 700, and applies the learning model to the convolutional neural network. I did some calculations for this. In the machine learning system 3 according to the sixth embodiment, the cryptographic processing device 600 acquires information regarding the input layer to the Nth layer of the learning model (“first model information”) and performs calculations, and the plaintext processing device 700 acquires information (“second model information”) from the (N+1)th layer to the output layer of the learning model and performs calculations. Further, the first model information and the second model information may be obtained separately from a reliable server or the like instead of from the system management device 800.

The cryptographic processing unit 6322 of the cryptographic processing device 600 acquires the first model information and performs cryptographic execution from the input layer to the Nth layer using the image data etc. acquired by the data acquisition unit 631 and the first model information. Perform calculations in the area.

The plaintext processing unit 7323 of the plaintext processing device 700 acquires the second model information, and uses the result of the calculation performed by the cryptographic processing unit 6322 and the second model information to process data from the (N+1)th layer to the output layer. Perform calculations in the plaintext execution area of , and update parameters of inference or model information.

In addition, as described above, the machine learning system 3 in the sixth embodiment does not acquire learning model information necessary for each operation in both the encryption processing unit 6322 and the plaintext processing unit 7323, but uses layer designation information. If the generally used value of N specified as ``N'' is determined to some extent, first model information and second model information may be stored in the cryptographic processing device 600 and the plaintext processing device 700 in advance. You can stay there.

That is, if N=5 is often used as the layer designation information, for example, model information from the input layer to the fifth layer is stored in advance in the storage unit 620 of the cryptographic processing device 600 as the first model information. However, model information from the sixth layer to the output layer may be stored in the storage unit 720 of the plaintext processing device 700 as the second model information.

(Explanation of effects)
The machine learning system 3 in the sixth embodiment not only separates the physical execution environments of the cryptographic processing unit 6322 and the plaintext processing unit 7323 as in the fifth embodiment, but also separates the physical execution environments of the cryptographic processing unit 6322 and the plaintext processing unit 7323. By acquiring only the parameters of the learning model necessary for its own calculations, in the event of hacking by a malicious third party, the information acquired by the third party is minimized and the original image can be restored. etc. can be made difficult to restore, increasing security.

<Embodiment 7>
Embodiment 7 is a modification of Embodiment 5. The machine learning system 4 according to the seventh embodiment includes at least two or more cryptographic processing devices and at least two or more plaintext processing devices. Furthermore, the cryptographic processing device and the plaintext processing device are paired in at least one pair, and the paired cryptographic processing device and plaintext processing device are used to perform calculations from the input layer to the output layer of the convolutional neural network. Execute processing.

Further, in the machine learning system according to the seventh embodiment, the unpaired cryptographic processing device and plaintext processing device communicate regarding processing different from the calculation processing from the input layer to the output layer of the convolutional neural network.

The machine learning system according to Embodiment 7 not only communicates between a cryptographic processing device and a plaintext processing device that are paired, but also communicates between a cryptographic processing device and a plaintext processing device that are not paired. This makes it difficult for a malicious third party to determine which device is performing one calculation process from the input layer to the output layer of the convolutional neural network. In other words, if communication is performed only between the paired cryptographic processing device and plaintext processing device, a malicious third party can detect which device is involved in a single calculation process by sensing that communication is occurring. It can be easily determined whether the two are paired. On the other hand, even if an unpaired cryptographic processing device and a plaintext processing device communicate with each other regarding processing different from the calculation processing from the input layer to the output layer (dummy communication may also be used), pairing can be achieved. make it difficult to determine which terminal is on the computer.

(Concrete example)
FIG. 26 is a diagram showing a specific configuration of the machine learning system 4. The machine learning system 4 includes two or more cryptographic processing devices 600-1 to 600-P (P is an integer of 2 or more. Hereinafter, unless otherwise specified, it will be referred to as "cryptographic processing device 600"), It includes two or more plaintext processing devices 700-1 to 700-Q (Q is an integer of 2 or more. Hereinafter, unless otherwise specified, it will be referred to as "plaintext processing device 700"), which are connected via a network. It is connected. For example, when cryptographic processing device 600-1 and plaintext processing device 700-1, cryptographic processing device 600-2 and plaintext processing device 700-2, and cryptographic processing device 600-P and plaintext processing device 700-Q are paired. , these perform calculation processing from the input layer to the output layer of one convolutional neural network. On the other hand, the unpaired cryptographic processing device 600 and plaintext processing device 700 perform communication unrelated to calculation processing from the input layer to the output layer of the convolutional neural network, that is, dummy communication. This prevents a malicious third party from detecting which devices are paired, making it possible to create a system that makes hacking difficult.

Furthermore, it is preferable that the same level of communication be performed between paired devices and unpaired devices. This can make it more difficult for a malicious third party to detect the combination of paired devices. Note that the pairing is managed by a management device (not shown) or the like, and may be changed at regular intervals. By changing the pairing randomly, the possibility of hacking can be further reduced.

(Explanation of effects)
The machine learning system 4 according to the seventh embodiment includes a plurality of cryptographic processing devices 600 and a plurality of plaintext processing devices 700, which are paired to execute calculation processing from the input layer to the output layer of one convolutional neural network. This includes a combination of a cryptographic processing device 600 and a plaintext processing device 700, and a combination that is not paired. This makes it difficult for a malicious third party to detect which device is paired, making it difficult to obtain or restore the original image, etc., thereby increasing security.

This concludes the description of the embodiment, but the above embodiment is merely an example. Therefore, the specific configurations, processing contents, etc. of the machine learning devices 100, 300, 400, 500, cryptographic processing device 600, plaintext processing device 700, and system management device 800 are not limited to those described in the above embodiments.

Further, the machine learning device according to the present disclosure may realize its functions by, for example, executing a program on a computer, without relying on the above device. Programs to realize the functions of the information provision system include USB (Universal Serial Bus) memory, CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), HDD (Hard Disc Drive), and SSD (Solid Disc Drive). The information may be stored in a computer-readable recording medium such as a state drive, or may be downloaded to a computer via a network.

Although the preferred embodiments of the present disclosure have been described above, the embodiments described above can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. be able to. These embodiments and their modifications are included within the scope and gist of the invention as well as within the scope of the invention described in the claims and its equivalents.

1, 2, 3, 4 machine learning system, 11 processor, 12 memory, 13 storage, 14 communication IF, 15 input/output IF, 100, 300, 400, 500 machine learning device, 110, 610, 710, 810 communication department, 120,420,620,720,820 Storage unit, 121,821 Key storage unit, 122,522,822 Learning model storage unit, 431,532 Shared memory, 130,430,530,630,730,830 Control unit, 131 , 531, 631 data acquisition unit, 132, 332, 432, 533, 632, 732 machine learning calculation unit, 200 terminal device, 600 cryptographic processing device, 700 plaintext processing device, 800 system management device, 401, 402, 403 screen, 410 image, 412 icon, 413 button, 1321, 831 reception unit, 1322, 4322, 5322, 6322 encryption processing unit, 1323, 4323, 5323, 7323 plaintext processing unit, 3324 provision unit, 3325 estimation unit.

Claims (11)

a data acquisition unit that acquires image data;
A machine learning calculation unit that performs calculation processing using a convolutional neural network based on the image data,
The convolutional neural network is composed of multiple layers,
The machine learning calculation unit is
a cryptographic processing unit that operates in a cryptographic execution area using homomorphic cryptography;
a plaintext processing unit that operates in the plaintext execution area;
a shared memory accessible by the encryption processing unit and the plaintext processing unit;
a reception unit that receives layer designation information that designates an N-th layer that is an intermediate layer from the input layer to the output layer of the convolutional neural network;
The cryptographic processing unit executes operations from the input layer to the Nth layer,
The plaintext processing unit executes operations from the (N+1)th layer to the output layer,
The shared memory is a machine learning device that stores results of calculations executed by the encryption processing unit and the plaintext processing unit. The cryptographic processing unit stores the results of operations from the input layer to the Nth layer in the shared memory,
The plaintext processing unit acquires the results of the calculations up to the Nth layer executed by the cryptographic processing unit from the shared memory, and processes the output from the (N+1)th layer based on the results of the acquired calculations. The machine learning device according to claim 1, which performs calculations up to layers. The cryptographic processing unit executes the calculation from the input layer to the Nth layer using first model information from the input layer to the Nth layer,
3. The plaintext processing unit executes operations from the (N+1)th layer to the output layer using second model information regarding the (N+1)th layer to the output layer. Machine learning device described in. a data acquisition unit that acquires image data;
A machine learning calculation unit that performs calculation processing using a convolutional neural network based on the image data,
The convolutional neural network is composed of multiple layers,
The machine learning calculation unit is
a cryptographic processing unit that operates in a cryptographic execution area using homomorphic cryptography;
a plaintext processing unit that operates in the plaintext execution area;
a reception unit that receives layer designation information that designates an N-th layer that is an intermediate layer from the input layer to the output layer of the convolutional neural network;
The cryptographic processing unit executes operations from the input layer to the Nth layer,
The plaintext processing unit executes operations from the (N+1)th layer to the output layer,
A machine learning system in which the cryptographic processing unit and the plaintext processing unit execute calculations in physically different execution environments. a cryptographic processing device including the data acquisition section and the cryptographic processing section;
further comprising a plaintext processing device including the plaintext processing unit,
The machine learning system according to claim 4, wherein the cryptographic processing device and the plaintext processing device transmit and receive data via a network. The cryptographic processing device obtains first model information from the input layer to the Nth layer,
The plaintext processing device obtains second model information from the (N+1)th layer to the output layer,
The cryptographic processing unit executes the calculations from the input layer to the Nth layer using the first model information,
The machine learning system according to claim 5, wherein the plaintext processing unit uses the second model information to execute calculations from the (N+1)th layer to the output layer. comprising at least two or more of the cryptographic processing devices and at least two or more of the plaintext processing devices,
The cryptographic processing device and the plaintext processing device are paired in at least one pair,
The machine learning calculation unit executes the calculation process using the paired cryptographic processing device and the plaintext processing device.
The machine learning system according to claim 5 or claim 6. The machine learning system according to claim 7, wherein the unpaired cryptographic processing device and the plaintext processing device communicate regarding processing different from the calculation processing. a data acquisition unit that acquires image data;
A machine learning calculation unit that performs calculation processing using a convolutional neural network based on the image data,
The convolutional neural network is composed of multiple layers,
The machine learning calculation unit is
a cryptographic processing unit that operates in a cryptographic execution area using homomorphic cryptography;
a plaintext processing unit that operates in the plaintext execution area;
a shared memory accessible by the encryption processing unit and the plaintext processing unit;
a reception unit that receives layer designation information that designates an N-th layer that is an intermediate layer from the input layer to the output layer of the convolutional neural network;
The cryptographic processing unit executes operations from the input layer to the Nth layer,
The plaintext processing unit executes operations from the (N+1)th layer to the output layer,
In the machine learning system, the shared memory stores results of calculations executed by the encryption processing unit and the plaintext processing unit. obtaining image data;
A step of performing calculation processing using a convolutional neural network based on the image data,
The convolutional neural network is composed of multiple layers,
The step of performing calculation processing using the convolutional neural network,
a step of receiving layer designation information that designates an Nth layer that is an intermediate layer from the input layer to the output layer of the convolutional neural network;
performing operations from the input layer to the N-th layer in a cryptographic execution area using homomorphic encryption;
performing operations from the (N+1)th layer to the output layer in a plaintext execution area;
A machine learning method comprising: a shared memory accessible from the cryptographic execution area and the plaintext execution area storing results of operations in the cryptographic execution area and the plaintext execution area. A machine learning program that is executed by a computer including a control unit and a storage unit,
the control unit acquiring image data;
the storage unit storing the image data;
The control unit performs calculation processing using a convolutional neural network based on the image data stored in the storage unit,
The convolutional neural network is composed of multiple layers,
The step of performing calculation processing using the convolutional neural network,
a step in which the control unit receives layer designation information that designates an N-th layer that is an intermediate layer from the input layer to the output layer of the convolutional neural network;
a step in which the control unit executes operations from the input layer to the N-th layer in a cryptographic execution area using homomorphic encryption;
The control unit has a step of executing an operation from the (N+1)th layer to the output layer in a plaintext execution area,
The storage unit includes a shared memory that is accessible from the cryptographic execution area and the plaintext execution area,
The machine learning program further comprises a step in which the shared memory stores results of calculations in the cryptographic execution area and the plaintext execution area.