TW202109378A - System and method of distributed deep learning system - Google Patents

Abstract

The present invention provides a system and a method of distributed deep learning. The system includes a first node, a token vault, and a second node. The first node includes the original data, the data label corresponding to the original data, and the private key, and the first node is configured to: obtain the gradient data of the original data; and encrypt the gradient data into a ciphertext data based on the homomorphic encryption technology; use the private key to sign the data label to obtain the digital signature of the data label. The token vault receives the data label and the digital signature of the first node, and after verifying the digital signature, transmits the token corresponding to the data label to the first node. The second node receives the ciphertext data and the token from the first node, and performs deep learning operations accordingly.

Description

Distributed deep learning system and method

The present invention relates to a deep learning system and method, and particularly relates to a distributed deep learning system and method.

Machine learning has achieved considerable success in the fields of speech recognition, image recognition, and natural language processing. Moreover, these technologies have good application prospects in fields such as unmanned driving, digital medical systems, advertising, and the Internet of Things.

Considering the large scale of the data sets and models involved in training, the machine learning platform is usually a decentralized platform, in which dozens or even hundreds of local end nodes running in parallel are deployed to train the model.

However, traditional decentralized information processing systems (especially decentralized deep learning systems) often encounter the dilemma of how to maintain data privacy security and data availability when processing large amounts of personal private data.

In view of this, the present invention provides a distributed deep learning system and method, which can be used to solve the above technical problems.

The present invention provides a distributed deep learning system, which includes a first local end node, a cloud code library and a second local end node. The first local end node includes a first original data, a first data label corresponding to the first original data, and a first private key, and the first local end node is configured to: obtain a first original data The first gradient data; the first gradient data is encrypted into a first ciphertext data based on the homomorphic encryption technology; the first private key is used to perform a signature operation on the first data tag to obtain the first data tag of the first data tag A digital signature. The cloud code library receives the first data label and the first digital signature of the first local end node, and after verifying the first digital signature, returns a first code corresponding to the first data label to the first local end node. The second local end node receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, and performs a deep learning operation accordingly.

The present invention provides a distributed deep learning method, including: obtaining a first gradient data of a first original data by a first local end node, wherein the first local end node includes the first original data and corresponds to the first original data. A first data label and a first private key of the data; the first local end node encrypts the first gradient data into a first ciphertext data based on the homomorphic encryption technology; the first local end node uses the first private key The key performs a signature operation on the first data tag to obtain a first digital signature of the first data tag; a cloud code library receives the first data tag and the first digital signature of the first local end node, And after verifying the first digital signature, a first code corresponding to the first data label is returned to the first local end node; and a second local end node receives the first ciphertext from the first local end node The data and the first code corresponding to the first data label are used to perform a deep learning operation.

Based on the above, the present invention proposes to use a homomorphic encryption method and tokenization technology in a distributed deep learning system to maintain data availability while ensuring data security, thereby enhancing distributed deep learning The security of the mechanism.

In order to make the above-mentioned features and advantages of the present invention more comprehensible, the following specific embodiments are described in detail in conjunction with the accompanying drawings.

In summary, the present invention proposes to use a homomorphic encryption method and coding technology in a distributed deep learning system. Based on the mathematical theory of homomorphic encryption, the encrypted ciphertext can still maintain the characteristics of mathematical operations, so that the availability of the data can be maintained under the premise of ensuring the security of the data. For example, after the original data owner homomorphically encrypts the original data, the encrypted ciphertext is handed over to others, and other people can directly add or subtract on the seemingly chaotic ciphertext after homomorphic encryption. The operation of multiplication and division does not need to know the plaintext behind it, so if we decrypt the ciphertext, the result of the blind operation of the ciphertext will be the same as the result of the original data owner directly performing the operation on the plaintext. In addition, the present invention also uses coding technology to protect the privacy of the tags of each learning material. In addition, since the code itself also has the feature that the content of the tag value does not affect the usability, it can achieve the purpose of maintaining the usability of the data while ensuring security. This will be further explained below.

Please refer to FIG. 1, which is a schematic diagram of a distributed deep learning system according to an embodiment of the present invention. As shown in FIG. 1, the distributed deep learning system 10 includes local end nodes 100, 100a, 100b, a cloud code library (token vault) 200, and a third-party system 300.

In the embodiment of the present invention, the characteristics and concepts of the local end nodes 100, 100a, and 100b are similar, so the following description will be based on the local end node 100 temporarily, and those with ordinary knowledge in the field should be able to deduce the local end. Related implementations of nodes 100a and 100b.

As shown in FIG. 1, the local end node 100 is, for example, each node in the distributed deep learning system 10. In one embodiment, the local end node 100 may initially include the original data OD1, the data tag TD1 corresponding to the original data OD1, and the private key PVK1 of the local end node 100. The data label TD1 is, for example, a classification data label (label) corresponding to each original data OD1 in the training phase of the deep learning of the local end node 100.

In one embodiment, when the hospital wants to use the patient's original data OD1 for deep learning, the corresponding data label TD1 is, for example, the privacy classification information corresponding to each patient data. For example, the original medical data of the patient A belongs to the acquisition of AIDS, the data of the second and third persons belong to the acquisition of sexually transmitted diseases, and the data of the two persons belong to the acquisition of prostate cancer. In this case, each node in the distributed deep learning system 10 (for example, the local end nodes 100, 100a, and 100b) can exchange this information with each other to perform deep learning operations.

In the embodiment of FIG. 1, the local end node 100 may include a gradient data operation module 110, a signature operation software module 120, a deep learning software module 130 and a homomorphic encryption software module 140. In different embodiments, the gradient data operation module 110 is a software module capable of providing stochastic gradient descent (SGD) operation. The deep learning software module 130 may be a software module capable of providing deep learning other mathematical operations besides the stochastic gradient descent method. The signature calculation software module 120 may be a software module capable of providing cryptographic signature calculation. The homomorphic encryption software module 140 may be a software module capable of performing homomorphic encryption cryptographic operations.

In the embodiment of the present invention, in order to allow the local end node 100 to exchange data with other nodes securely, the local end node 100 may perform the following operations. Specifically, the gradient data operation module 110 can obtain the gradient data GD1 of the original data OD1. In an embodiment, the gradient data operation module 110 can convert the original data OD1 into the gradient data GD1 based on the stochastic gradient descent operation, but the invention is not limited to this. After that, the homomorphic encryption software module 140 can encrypt the gradient data GD1 into ciphertext data ED1 based on the homomorphic encryption technology.

In addition, the signature calculation software module 120 can use the private key PVK1 to perform a signature calculation on the data tag TD1 to obtain the digital signature DS1 of the data tag TD1.

After that, the local end node 100 can send the data tag TD1 and the digital signature DS1 to the cloud code library 200.

In one embodiment, the cloud code library 200 is, for example, a cloud code library for storing and processing token related functions, which may include stored codes, data tags and code correspondence tables, code software modules 210, and verification stamps. Calculation software module 220. In different embodiments, the above code is, for example, a random number code, and each code can correspond to a data label. In addition, the code software module 210 can store the correspondence between the data label and the code as a correspondence table of the data label and the code.

The code software module 210 can be used to compare whether there is a corresponding code in the data tag TD1, if not, generate a new random number code, and store the combination of the data tag TD1 and the code in the aforementioned data tag and code correspondence table. The signature verification calculation software module 220 is, for example, a software module capable of verifying whether the digital signature of the data is correct.

Based on this, when the cloud code library 200 receives the data tag TD1 and its digital signature DS1 from the local end node 100, the digital signature DS1 can be verified. In one embodiment, the cloud code base 200 can reach the trusted and fair third-party system 300 to query the public key PBK1 corresponding to the local end node 100. In different embodiments, the third-party system 300 is, for example, a trusted and impartial third party, which may store the public key corresponding to the private key of each local end node 100, 100a, 100b.

After obtaining the public key PBK1 of the local end node 100, the cloud code base 200 can perform a signature verification operation on the digital signature DS1 through the signature verification calculation software module 220 to verify whether the digital signature DS1 is correct. If the digital signature DS1 is verified to be correct, the code software module 210 can correspondingly convert the data tag TD1 into a code T1, and the cloud code library 200 can return the code T1 to the local end node 100. On the other hand, if the digital signature DS1 is verified as an error, the cloud code library 200 can return an error message to the local end node 100.

After the local end node 100 receives the code T1 from the cloud code library 200, the local end node 100 can send the ciphertext data ED1 and the code T1 to other nodes in the distributed deep learning system 10, such as the local end nodes 100a and/or 100b , So that the local end nodes 100a and/or 100b can perform deep learning operations accordingly. In an embodiment, the local end node 100 can transmit the ciphertext data ED1 and the code T1 to other nodes in the distributed deep learning system 10 according to a predetermined order or randomly. In this way, these nodes can use the homomorphic encrypted ciphertext to maintain the characteristics of the mathematical operation ability of the original data, and perform distributed deep learning operations.

In other embodiments, the local end nodes 100a, 100b can also perform operations similar to those of the local end node 100 according to the teachings of the above-mentioned embodiments. Taking the local end node 100a as an example, it may include second original data, a second data label corresponding to the second original data, and a second private key. Based on this, the local end node 100a can be configured to: obtain the second gradient data of the second original data; encrypt the second gradient data into ciphertext data ED2 based on homomorphic encryption technology; use the second private key to pair the second gradient data The data tag performs a signature calculation to obtain the second digital signature of the second data tag; sends the second data tag and the second digital signature to the cloud code library 200; receives the second data tag corresponding to the cloud code library 200 The code T2 of, where the code T2 is generated by the cloud code library 200 after verifying the second digital signature; the ciphertext data ED2 and the code T2 corresponding to the second data tag are sent to the local end node 100.

After the local end node 100 receives the ciphertext data ED2 and the code T2 from the local end node 100a, the local end node 100 can use the deep learning software module 130 to perform deep learning operations accordingly.

In addition, the local end node 100a can also send the ciphertext data ED2 and the code T2 corresponding to the second data label to the local end node 100b, so that the local end node 100b can perform deep learning operations accordingly.

It can be seen from the above that based on homomorphic encryption and coding technology, the decentralized deep learning system of the present invention can protect the privacy of information security in the decentralized deep learning process. First, in the initialization phase, each decentralized local node can use the stochastic gradient descent method for learning operations based on the original data they own, and perform homomorphic encryption operations on the obtained gradient data to obtain the encrypted secret. Text information.

In addition, each decentralized local end node can also use coding technology to transfer their own data and their corresponding data tags in the deep learning process to the cloud code library, and at the same time attach a digital signature to this data tag with their own private key. chapter. In this case, the cloud code library can use the public key of the local end node to verify the digital signature. If the verification is passed, the data label is converted into a code and sent back to the local end node. Based on this, each scattered local end node transmits the gradient data that has been homomorphically encrypted into ciphertext and the data label that has been converted into code by the coding process to other local end nodes. In this way, the present invention uses the homomorphic encrypted ciphertext to maintain the mathematically operable characteristics of the original gradient data, while taking into account privacy, and the use of coding technology can protect the privacy of the tags of each learning material. Decentralized deep learning operations.

Please refer to FIG. 2, which is a flowchart of a distributed deep learning method according to an embodiment of the present invention. The method of this embodiment can be executed by the distributed deep learning system 10 of FIG. 1. The steps in FIG. 2 will be described below in conjunction with the components shown in FIG. 1.

First, in step S21, the local end node 100 can obtain the gradient data GD1 of the original data OD1. In step S22, the local end node 100 may encrypt the gradient data GD1 into ciphertext data ED1 based on the homomorphic encryption technology. In step S23, the local end node 100 can use the private key PVK1 to perform a signature operation on the data tag TD1 to obtain the digital signature DS1 of the data tag TD1. In step S24, the cloud code library 200 may receive the data tag TD1 and the digital signature DS1 of the local end node 100, and after verifying the digital signature DS1, return the code T1 corresponding to the data tag TD1 to the local end node 100 . In step S25, the local end node 100a may receive the ciphertext data ED1 and the code T1 corresponding to the data tag TD1 from the local end node 100, and perform deep learning operations accordingly. For the details of the above steps, please refer to the description in the previous embodiment, which will not be repeated here.

In summary, based on homomorphic encryption and coding technology, the decentralized deep learning system and method of the present invention can protect privacy information security, and thus can solve traditional decentralized information processing systems (especially decentralized deep learning systems) often Encountered the dilemma of how to maintain data privacy security and data availability when processing large amounts of personal private data.

The system and method proposed by the present invention can use the homomorphic encrypted ciphertext to maintain the mathematically operable characteristics of the gradient data in the deep learning, while taking into account the privacy and security, and the use of coding technology can protect the learning data The privacy of the tag, plus the code itself is another feature that does not affect the usability of the tag value content, so as to achieve the purpose of maintaining the availability of data while ensuring security.

Although the present invention has been disclosed in the above embodiments, it is not intended to limit the present invention. Anyone with ordinary knowledge in the relevant technical field can make some changes and modifications without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be determined by the scope of the attached patent application.

10: Decentralized deep learning system 100, 100a, 100b: local end node 110: Gradient data operation module 120: Signature calculation software module 130: Deep Learning Software Module 140: Homomorphic encryption software module 200: Cloud code base 210: Code Software Module 220: Verification and signature calculation software module 300: Third-party system DS1: Digital Signature ED1, ED2: ciphertext data OD1: Original data PBK1: Public key PVK1: private key S21~S25: steps T1, T2: code TD1: Data label

FIG. 1 is a schematic diagram of a distributed deep learning system according to an embodiment of the present invention. Fig. 2 is a flowchart of a decentralized deep learning method according to an embodiment of the present invention.

10: Decentralized deep learning system

100, 100a, 100b: local end node

110: Gradient data operation module

120: Signature calculation software module

130: Deep Learning Software Module

140: Homomorphic encryption software module

200: Cloud code base

210: Code Software Module

220: Verification and signature calculation software module

300: Third-party system

DS1: Digital Signature

ED1, ED2: ciphertext data

OD1: Original data

PBK1: Public key

PVK1: private key

T1, T2: code

TD1: Data label

Claims (7)

A decentralized deep learning system, including: A first local end node includes a first original data, a first data label corresponding to the first original data, and a first private key, and the first local end node is configured to: Obtain a first gradient data of the first original data; Encrypting the first gradient data into a first ciphertext data based on a homomorphic encryption technology; Use the first private key to perform a signature operation on the first data label to obtain a first digital signature of the first data label; A cloud code library that receives the first data tag and the first digital signature of the first local end node, and after verifying the first digital signature, will correspond to a first code of the first data tag Back to the first local end node; and A second local end node receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, and performs a deep learning operation accordingly. According to the system described in claim 1, wherein the first local end node converts the first original data into the first gradient data based on a stochastic gradient descent operation. For example, the system described in item 1 of the scope of patent application further includes a third-party system that stores a first public key corresponding to the first private key of the first local end node, and the cloud code library is Configure to: Query the third-party system for the first public key corresponding to the first local end node; Perform a signature verification operation on the first digital signature based on the first public key; In response to the verification that the first digital signature is correct, the first data tag is converted into the first code, and the first code is returned to the first local end node; and In response to the verification of the first digital signature as an error, an error message is returned to the first local end node. For example, the system described in item 1 of the scope of patent application, wherein the second local end node includes a second original data, a second data label corresponding to the second original data, and a second private key, and the first 2. The local end node is configured to: Obtain a second gradient data of the second original data; Encrypting the second gradient data into a second ciphertext data based on the homomorphic encryption technology; Use the second private key to perform the signature operation on the second data label to obtain a second digital signature of the second data label; Sending the second data tag and the second digital signature to the cloud code library; Receiving a second code corresponding to the second data tag from the cloud code library, where the second code is generated by the cloud code library after verifying the second digital signature; The second ciphertext data and the second code corresponding to the second data label are sent to the first local end node, so that the first local end node can perform the deep learning operation accordingly. For example, the system described in item 4 of the scope of patent application further includes a third local end node that receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, And receiving the second ciphertext data and the second code corresponding to the second data label from the second local end node, and performing the deep learning operation accordingly. The system described in item 1 of the scope of patent application further includes a third local end node, which receives the first ciphertext data and the first code corresponding to the first data label from the first local end node. A decentralized deep learning method including: A first gradient data of a first original data is obtained by a first local end node, wherein the first local end node includes the first original data, a first data label corresponding to the first original data, and a The first private key; Encrypting the first gradient data into a first ciphertext data by the first local end node based on homomorphic encryption technology; The first local end node uses the first private key to perform a signature operation on the first data label to obtain a first digital signature of the first data label; A cloud code library receives the first data label and the first digital signature of the first local end node, and after verifying the first digital signature, a first code corresponding to the first data label Back to the first local end node; and A second local end node receives the first ciphertext data and the first code corresponding to the first data label from the first local end node, and performs a deep learning operation accordingly.

Images