TWI690861B - System and method of distributed deep learning system - Google Patents

Abstract

The present invention provides a system and a method of distributed deep learning. The system includes a first node, a token vault, and a second node. The first node includes the original data, the data label corresponding to the original data, and the private key, and the first node is configured to: obtain the gradient data of the original data; and encrypt the gradient data into a ciphertext data based on the homomorphic encryption technology; use the private key to sign the data label to obtain the digital signature of the data label. The token vault receives the data label and the digital signature of the first node, and after verifying the digital signature, transmits the token corresponding to the data label to the first node. The second node receives the ciphertext data and the token from the first node, and performs deep learning operations accordingly.

Description

Decentralized deep learning system and method

The invention relates to a deep learning system and method, and in particular to a distributed deep learning system and method.

Machine learning has achieved considerable success in the fields of speech recognition, image recognition and natural language processing. Moreover, these technologies have very good application prospects in the fields of unmanned driving, digital medical systems, advertising, and the Internet of Things.

Considering that the scale of data sets and models involved in training is very large, machine learning platforms are usually decentralized platforms, where dozens or even hundreds of local end nodes running in parallel are deployed to train the model.

However, traditional decentralized information processing systems (especially decentralized deep learning systems) often encounter the dilemma of how to maintain data privacy security and data availability when processing large amounts of personal privacy data.

In view of this, the present invention provides a distributed deep learning system and method, which can be used to solve the above technical problems.

The invention provides a decentralized deep learning system, including a first local end node, a cloud code base and a second local end node. The first local end node includes a first original data, a first data label corresponding to the first original data and a first private key, and the first local end node is configured to: find a The first gradient data; encrypt the first gradient data into a first ciphertext data based on the homomorphic encryption technology; use the first private key to perform a signature operation on the first data label to obtain a first data label A digital signature. The cloud code base receives the first data label and the first digital signature of the first local node, and after verifying the first digital signature, returns a first code corresponding to the first data label to the first local terminal node. The second local end node receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, and performs a deep learning operation accordingly.

The invention provides a decentralized deep learning method, comprising: obtaining a first gradient data of a first original data from a first local end node, wherein the first local end node includes the first original data, corresponding to the first original data A first data label and a first private key of the data; the first local end node encrypts the first gradient data into a first ciphertext data based on the homomorphic encryption technology; the first local end node uses the first private key The key performs a signature operation on the first data label to obtain a first digital signature of the first data label; a cloud code base receives the first data label and the first digital signature of the first local node, After verifying the first digital signature, a first code corresponding to the first data label is returned to the first local end node; and a second local end node receives the first ciphertext from the first local end node The data and the first code corresponding to the first data label, and perform a deep learning operation accordingly.

Based on the above, the present invention proposes to use homomorphic encryption method and tokenization technology in the distributed deep learning system, in order to maintain the availability of the data under the premise of ensuring the security of the data, thereby enhancing the distributed deep learning The safety of the mechanism.

In order to make the above-mentioned features and advantages of the present invention more obvious and understandable, the embodiments are specifically described below in conjunction with the accompanying drawings for detailed description as follows.

In summary, the present invention proposes to use homomorphic encryption methods and coding techniques in a distributed deep learning system. According to the mathematical theory of homomorphic encryption, the encrypted ciphertext can still maintain the characteristics of mathematical operations, so that the availability of data can be maintained under the premise of ensuring data security. For example, after the original data owner homomorphically encrypts the original data, the encrypted ciphertext is handed over to others, and other people can directly add or subtract the ciphertext that appears to be a mess after being homomorphically encrypted. The operation of multiplication and division does not need to know the plaintext behind it. In this way, if we decrypt the result of the blind operation of the ciphertext, if we decrypt it, we will find that it is the same as the operation of the original data owner directly on the plaintext. In addition to this, the present invention also uses coding technology to protect the privacy of the labels of the learning materials. In addition, since the code itself has the characteristic that the content of the tag value does not affect the usability, it can achieve the purpose of maintaining the usability of the data under the condition of ensuring security. This will be explained further below.

Please refer to FIG. 1, which is a schematic diagram of a distributed deep learning system according to an embodiment of the present invention. As shown in FIG. 1, the decentralized deep learning system 10 includes local end nodes 100, 100a, 100b, a cloud code base (token vault) 200, and a third-party system 300.

In the embodiments of the present invention, the characteristics and concepts of the local end nodes 100, 100a, and 100b are similar, so the following will be described based on the local end node 100, and those with ordinary knowledge in the art should be able to derive the local end Related implementations of nodes 100a and 100b.

As shown in FIG. 1, the local end node 100 is, for example, each node in the distributed deep learning system 10. In one embodiment, the local end node 100 may initially include the original data OD1, the data tag TD1 corresponding to the original data OD1, and the private key PVK1 of the local end node 100. The data label TD1 is, for example, a classification data label (label) corresponding to each original data OD1 in the training phase of the deep learning of the local end node 100.

In an embodiment, when the hospital wants to use the patient's original data OD1 for deep learning, the corresponding data label TD1 is, for example, the privacy classification information corresponding to each patient data. For example, the original medical data of the patient A belongs to AIDS, the data of the two persons to obtain sexually transmitted diseases, and the data of the two persons to obtain prostate cancer. In this case, each node in the decentralized deep learning system 10 (such as the local end nodes 100, 100a, and 100b) can exchange this information with each other to perform deep learning operations.

In the embodiment of FIG. 1, the local end node 100 may include a gradient data calculation module 110, a signature calculation software module 120, a deep learning software module 130, and a homomorphic encryption software module 140. In different embodiments, the gradient data operation module 110 is a software module capable of providing stochastic gradient descent (SGD) method operations. The deep learning software module 130 may be a software module capable of providing deep learning mathematical operations other than stochastic gradient descent. The signature calculation software module 120 may be a software module capable of providing cryptographic signature calculation. The homomorphic encryption software module 140 may be a software module capable of homomorphic encryption cryptographic operations.

In the embodiment of the present invention, in order for the local end node 100 to exchange data with other nodes securely, the local end node 100 may perform the following operations. Specifically, the gradient data calculation module 110 can obtain the gradient data GD1 of the original data OD1. In an embodiment, the gradient data operation module 110 may convert the original data OD1 into the gradient data GD1 based on a random gradient descent operation, but the invention may not be limited to this. Thereafter, the homomorphic encryption software module 140 may encrypt the gradient data GD1 into the ciphertext data ED1 based on the homomorphic encryption technology.

In addition, the signature calculation software module 120 may perform a signature calculation on the data tag TD1 using the private key PVK1 to obtain a digital signature DS1 of the data tag TD1.

After that, the local end node 100 can send the data tag TD1 and the digital signature DS1 to the cloud code base 200.

In an embodiment, the cloud code base 200 is, for example, a cloud code base for storing and processing code (token) related functions, which may include stored codes, data labels and code correspondence tables, code software modules 210, and signature verification seals Calculation software module 220. In different embodiments, the above code is, for example, a random number code, and each code may correspond to a data label. In addition, the code software module 210 can store the correspondence between the data label and the code as a correspondence table between the data label and the code.

The code software module 210 can be used to compare whether a corresponding code exists in the data tag TD1, if not, a new random code is generated, and the combination of the data tag TD1 and the code is stored in the corresponding table of the aforementioned data tag and code. The signature verification calculation software module 220 is, for example, a software module capable of verifying whether the digital signature of the data is correct.

Based on this, when the cloud code base 200 receives the data tag TD1 and its digital signature DS1 from the local end node 100, the digital signature DS1 can be verified. In an embodiment, the cloud code base 200 can query the trusted and fair third-party system 300 to query the public key PBK1 corresponding to the local end node 100. In different embodiments, the third-party system 300 is, for example, a trusted and fair third-party, which may store the public key corresponding to the private key of each local end node 100, 100a, 100b.

After obtaining the public key PBK1 of the local end node 100, the cloud code base 200 can perform the signature verification operation on the digital signature DS1 through the signature verification operation software module 220 to verify whether the digital signature DS1 is correct. If the digital signature DS1 is verified as correct, the code software module 210 can convert the data tag TD1 into the code T1 accordingly, and the cloud code base 200 can transmit the code T1 to the local node 100. On the other hand, if the digital signature DS1 is verified as an error, the cloud code base 200 may return an error message to the local node 100.

After the local end node 100 receives the code T1 from the cloud code base 200, the local end node 100 may send the ciphertext data ED1 and the code T1 to other nodes in the distributed deep learning system 10, such as the local end node 100a and/or 100b To allow the local end nodes 100a and/or 100b to perform deep learning operations accordingly. In an embodiment, the local end node 100 may transmit the ciphertext data ED1 and the code T1 to other nodes in the distributed deep learning system 10 according to a predetermined order or randomly. In this way, these nodes can use the homomorphic encrypted ciphertext to maintain the characteristics of the mathematical operation ability of the original data, and perform distributed deep learning operations.

In other embodiments, the local end nodes 100a, 100b can also perform operations similar to the local end node 100 according to the teachings of the above embodiments. Taking the local end node 100a as an example, it may include second original data, a second data label corresponding to the second original data, and a second private key. Based on this, the local end node 100a can be configured to: obtain the second gradient data of the second original data; encrypt the second gradient data into the ciphertext data ED2 based on the homomorphic encryption technology; use the second private key to the second The data label performs a signature operation to obtain the second digital signature of the second data label; the second data label and the second digital signature are sent to the cloud code base 200; the corresponding second data label is received from the cloud code base 200 Code T2, where the code T2 is generated by the cloud code base 200 after verifying the second digital signature; the ciphertext data ED2 and the code T2 corresponding to the second data label are sent to the local node 100.

After the local end node 100 receives the ciphertext data ED2 and the code T2 from the local end node 100a, the local end node 100 may perform deep learning operations based on the deep learning software module 130.

In addition, the local end node 100a may also send the ciphertext data ED2 and the code T2 corresponding to the second data tag to the local end node 100b, so that the local end node 100b can perform deep learning operations accordingly.

As can be seen from the above, based on homomorphic encryption and coding technology, the decentralized deep learning system of the present invention can protect the security of private information in the process of decentralized deep learning. First, in the initialization phase, each scattered local end node can use the random gradient descent method to learn from the original data it owns, and perform the homomorphic encryption operation on the obtained gradient data to obtain the encrypted secret Documents.

In addition, each decentralized local end node can also use coding technology to transfer the data tags corresponding to their own data and the corresponding deep learning process to the cloud code base, and also attach a digital signature to this data tag with its own private key chapter. In this case, the cloud code base can use the public key of the local end node to verify the digital signature. If the verification is passed, the data label is converted into a code and returned to the local end node. Based on this, each decentralized local end node will homomorphically encrypt the gradient data that has been converted into ciphertext, and the data label that has been converted into code through the coding process to other local end nodes. In this way, the present invention uses the homomorphic encrypted ciphertext to maintain the mathematically operable characteristics of the original gradient data, while taking into account privacy, and the use of coding technology to protect the privacy of the labels of each learning material, to continue Decentralized deep learning operations.

Please refer to FIG. 2, which is a flowchart of a distributed deep learning method according to an embodiment of the invention. The method of this embodiment may be executed by the distributed deep learning system 10 of FIG. 1, and the steps of FIG. 2 will be described below with the components shown in FIG. 1.

First, in step S21, the local end node 100 may obtain the gradient data GD1 of the original data OD1. In step S22, the local end node 100 may encrypt the gradient data GD1 into the ciphertext data ED1 based on the homomorphic encryption technology. In step S23, the local end node 100 may use the private key PVK1 to perform a signature operation on the data label TD1 to obtain the digital signature DS1 of the data label TD1. In step S24, the cloud code base 200 can receive the data label TD1 and the digital signature DS1 of the local node 100, and after verifying the digital signature DS1, return the code T1 corresponding to the data label TD1 to the local node 100 . In step S25, the local end node 100a may receive the ciphertext data ED1 and the code T1 corresponding to the data label TD1 from the local end node 100, and perform deep learning operations accordingly. For details of the above steps, reference may be made to the description in the previous embodiment, and no further details are provided here.

In summary, based on homomorphic encryption and coding technology, the decentralized deep learning system and method of the present invention can protect the privacy of information security, and thus can solve the traditional decentralized information processing system (especially the decentralized deep learning system) often The dilemma of how to maintain data privacy security and data availability when dealing with a large amount of personal privacy data.

The system and method proposed in the present invention can utilize homomorphic encrypted ciphertext to maintain the mathematical operability of gradient data in deep learning, while taking into account privacy and security, and use coding technology to protect the learning materials. The privacy of the tag, plus the code itself is another feature of the value of the tag that does not affect the usability, so as to achieve the purpose of maintaining the availability of the data under the guarantee of security.

Although the present invention has been disclosed as above with examples, it is not intended to limit the present invention. Any person with ordinary knowledge in the technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to the scope defined in the appended patent application.

10: Decentralized deep learning system 100, 100a, 100b: local end node 110: Gradient data operation module 120: Signature calculation software module 130: Deep learning software module 140: Homomorphic encryption software module 200: Cloud code base 210: code software module 220: Verification signature calculation software module 300: Third-party system DS1: digital signature ED1, ED2: cipher text information OD1: original data PBK1: public key PVK1: private key S21~S25: Step T1, T2: code TD1: data label

FIG. 1 is a schematic diagram of a distributed deep learning system according to an embodiment of the invention. 2 is a flowchart of a distributed deep learning method according to an embodiment of the invention.

10: Decentralized deep learning system

100, 100a, 100b: local end node

110: Gradient data operation module

120: Signature calculation software module

130: Deep learning software module

140: Homomorphic encryption software module

200: Cloud code base

210: code software module

220: Verification signature calculation software module

300: Third-party system

DS1: digital signature

ED1, ED2: cipher text information

OD1: original data

PBK1: public key

PVK1: private key

T1, T2: code

TD1: data label

Claims (7)

A decentralized deep learning system, including: A first local end node, which includes a first original data, a first data label corresponding to the first original data, and a first private key, and the first local end node is configured to: Find a first gradient data of the first original data; Encrypt the first gradient data into a first ciphertext data based on homomorphic encryption technology; Using the first private key to perform a signature operation on the first data label to obtain a first digital signature of the first data label; A cloud code base, receiving the first data label and the first digital signature of the first local end node, and after verifying the first digital signature, a first code corresponding to the first data label Back to the first local end node; and A second local end node receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, and performs a deep learning operation accordingly. The system according to item 1 of the patent application scope, wherein the first local end node converts the first original data into the first gradient data based on a random gradient descent operation. The system as described in item 1 of the patent application scope further includes a third-party system that stores a first public key corresponding to the first private key of the first local end node, and the cloud code base is Configure with: Query the third party system for the first public key corresponding to the first local end node; Performing a signature verification operation on the first digital signature based on the first public key; In response to the verification of the first digital signature being correct, the first data label is converted into the first code, and the first code is returned to the first local end node; and In response to the verification of the first digital signature being an error, an error message is returned to the first local end node. The system according to item 1 of the patent application scope, wherein the second local end node includes a second original data, a second data label corresponding to the second original data, and a second private key, and the second The two local end nodes are configured to: Find a second gradient data of the second original data; Encrypt the second gradient data into a second ciphertext data based on the homomorphic encryption technology; Using the second private key to perform the signature operation on the second data label to obtain a second digital signature of the second data label; Send the second data label and the second digital signature to the cloud code base; Receiving a second code corresponding to the second data tag from the cloud code base, wherein the second code is generated by the cloud code base after verifying the second digital signature; Sending the second ciphertext data and the second code corresponding to the second data label to the first local end node, so that the first local end node can perform the deep learning operation accordingly. The system described in item 4 of the patent application scope further includes a third local end node that receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, And receiving the second ciphertext data and the second code corresponding to the second data label from the second local end node, and performing the deep learning operation accordingly. The system as described in item 1 of the patent application scope further includes a third local end node that receives the first ciphertext data and the first code corresponding to the first data label from the first local end node. A decentralized deep learning method, including: A first gradient data of a first original data is obtained from a first local end node, wherein the first local end node includes the first original data, a first data label corresponding to the first original data and a The first private key; The first local end node encrypts the first gradient data into a first ciphertext data based on the homomorphic encryption technology; The first local end node uses the first private key to perform a signature operation on the first data label to obtain a first digital signature of the first data label; A cloud code base receives the first data label and the first digital signature of the first local node, and after verifying the first digital signature, a first code corresponding to the first data label Back to the first local end node; and A second local end node receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, and performs a deep learning operation accordingly.

Images