CN116796338A - Online deep learning system and method for privacy protection - Google Patents

Online deep learning system and method for privacy protection Download PDF

Info

Publication number
CN116796338A
CN116796338A CN202210247646.4A CN202210247646A CN116796338A CN 116796338 A CN116796338 A CN 116796338A CN 202210247646 A CN202210247646 A CN 202210247646A CN 116796338 A CN116796338 A CN 116796338A
Authority
CN
China
Prior art keywords
deep learning
linear
layer
encrypted
learning model
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202210247646.4A
Other languages
Chinese (zh)
Inventor
李晋杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Zhixing Technology Co Ltd
Original Assignee
Shenzhen Zhixing Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Zhixing Technology Co Ltd filed Critical Shenzhen Zhixing Technology Co Ltd
Priority to CN202210247646.4A priority Critical patent/CN116796338A/en
Publication of CN116796338A publication Critical patent/CN116796338A/en
Pending legal-status Critical Current

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The application relates to an online deep learning system and method for privacy protection, which can be used for federal learning. The method comprises the following steps: the method comprises the steps that encryption processing is conducted on the held training data under an HE security protocol through a client side, and the encrypted training data are sent to a server side; training a deep learning model by utilizing encrypted training data received from the client through a server, and respectively carrying out encryption processing under a DP security protocol and encryption processing under the HE security protocol on linear components and bias components of at least one linear layer included in the deep learning model in the training process of training the deep learning model so as to obtain an encrypted gradient of the deep learning model; and obtaining encrypted model parameters of the deep learning model based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters. Thus being helpful to overcome the barriers in privacy protection encountered when the online deep learning technology is popularized.

Description

Online deep learning system and method for privacy protection
Technical Field
The application relates to the technical field of data security and privacy protection, in particular to the technical field of privacy calculation and federal learning, and particularly relates to an online deep learning system and method for privacy protection.
Background
Privacy-preserving computing (Privacy-Preserving Computing) is generally understood to be the process of performing data-related computation and analysis and mining of data value under the premise of protecting the security of private information. A concept that is relatively close to privacy preserving computing is privacy computing (Privacy Computing). The concept of privacy calculations is to be understood broadly to cover the theory of calculations and methods that perform various operations on the privacy information involved therein when processing various information to provide protection throughout the life cycle of the privacy information. The definition and scope of the respective concepts of privacy preserving computation and privacy computing change with the development of theoretical research and application technology, and the corresponding computing frameworks, computing models, computing theory, algorithms, application technology, etc. are also rapidly developing, but privacy preserving is mainly used, for example, protecting the privacy security information of each participant, etc. Various key technologies have important applications in privacy preservation, such as federal learning (Federated Learning, FL), secure multiparty computing (Multi-Party Computation, MPC), etc.
Artificial intelligence technology and deep learning technology have been greatly developed, and are widely used in the fields of intelligent transportation, medical health, image processing, natural language processing, etc. Privacy protection is also involved in these applications, and in particular, as related laws and regulations strengthen protection of personal privacy information of users, such as user traffic data and e-commerce transaction data, there is an increasing need to pay attention to privacy protection in deep learning applications. On the other hand, as the interaction frequency increases and the complexity of the model increases, deep learning applications such as merchandise recommendation systems and the like have raised higher demands on communication overhead, computing power, storage space and the like. This allows many deep learning applications to perform the corresponding training and reasoning processes, typically in an online manner, such as an online learning service, for example, where the participants holding training data and/or input data for reasoning do not have sufficient data processing capabilities and data storage capabilities and can only rely on online learning service providers that provide these processing capabilities. Here, a common online learning service is implemented by a cloud service, particularly a public cloud service. However, training or reasoning of the deep learning model is completed in an online manner, such as through a public cloud server, and risks of privacy information leakage are faced, such as private data uploaded to the public cloud server or calculation results obtained by calculation of the cloud server, which may be leaked to unauthorized parties or even maliciously tampered with. Although protection of private information can be enhanced by the above-described privacy protection related techniques such as FL and MPC, these techniques tend to increase communication overhead and present challenges to computing power and storage space, for example, 1-bit original information may become 1024 bits after encryption, etc. In addition, because of actual needs, frequent updating of model parameters is required to shorten the time resources consumed by the training process, and the existing privacy protection technology often makes the training process too time-consuming. And the existing privacy protection related technology often requires that a provider of online learning service or an integrator for data of each party, such as a public cloud server, is required to meet the requirements of a trusted third party or non-competitive (non-competitive), but this makes it impossible to well utilize existing various cloud resources. These deficiencies of the prior art pose obstacles to the popularization of online deep learning techniques, for example, making it difficult for existing online deep learning techniques to provide a product solution that facilitates commercial landing while meeting the requirements of privacy protection in terms of security, making it difficult for online deep learning applications to achieve better results in terms of several aspects including privacy protection, communication overhead, computational effort and storage space requirements, security requirements of cloud service providers, and the like.
Therefore, there is a need for an online deep learning system and method for privacy protection, in the fields of intelligent transportation, medical health, image processing, natural language processing, etc., so that each participant holding private data can cooperatively train a deep learning model in an online manner, such as public cloud service, make reasoning with the trained model, and meet model update requirements, and such online deep learning system and method not only provide privacy protection, such as providing privacy protection for training data, reasoning data, model parameters, but also do not require that public cloud service providers or data integrators have to meet trusted third parties or non-competitive requirements to facilitate invoking rich ready cloud resources, but also have lower or improved requirements for communication overhead, computing power and storage space in the training process and reasoning process, and thus have better overall operation efficiency and resource utilization efficiency, thereby overcoming the obstacles in terms of privacy protection in the process of popularization of online deep learning technology, and being beneficial to providing a convenient commercial solution for a plurality of aspects, such as privacy protection, communication overhead, computing power and storage space requirements, security requirements of cloud service providers, etc.
Disclosure of Invention
In a first aspect, an embodiment of the present application provides an online deep learning method for privacy protection. The online deep learning method comprises the following steps: the method comprises the steps that through a client, encryption processing under homomorphic encryption HE security protocol is carried out on training data held by the client, and the encrypted training data are sent to a server; training a deep learning model by utilizing encrypted training data received from the client through a server, and respectively carrying out encryption processing under a differential privacy DP security protocol and encryption processing under the HE security protocol on linear components and bias components of at least one linear layer included in the deep learning model in the training process of training the deep learning model so as to obtain an encrypted gradient of the deep learning model; and obtaining encrypted model parameters of the deep learning model based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters.
The technical scheme described in the first aspect provides privacy protection while avoiding the requirement that the server side is a trusted third party or noncompetitive, thereby being beneficial to calling abundant ready-made cloud resources, having lower requirements on communication overhead, calculation power and storage space, and therefore having better overall calculation efficiency and resource utilization efficiency, overcoming the barriers in privacy protection encountered when the online deep learning technology is promoted, and being beneficial to providing a product solution which is convenient for commercialization and landing and takes into consideration the aspects such as privacy protection, communication overhead, calculation power and storage space requirements, security requirements of a cloud service provider and the like.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that the encrypted gradient includes a linear component of the encrypted gradient and an offset component of the encrypted gradient, where obtaining the encrypted model parameter of the deep learning model based on the encrypted gradient includes: the server integrates the bias components of the encrypted gradient to obtain the bias components of the encrypted model parameters, and the client decrypts and noise-adds the linear components of the encrypted gradient to obtain the linear components of the encrypted model parameters.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides updating the deep learning model by using the encrypted model parameters, including: and updating the deep learning model by the server side by utilizing the linear component of the encrypted model parameter and the offset component of the encrypted model parameter.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that the deep learning model includes a plurality of linear layers, where the encrypting process under the DP security protocol and the encrypting process under the HE security protocol are performed on the linear component and the bias component of at least one linear layer included in the deep learning model, respectively, including: and respectively carrying out encryption processing under the DP security protocol and encryption processing under the HE security protocol on the linear component and the bias component of each linear layer of the plurality of linear layers.
According to a possible implementation manner of the solution of the first aspect, the embodiments of the present application further provide that the plurality of linear layers includes a convolution layer and/or a fully connected layer, and the deep learning model further includes at least one nonlinear layer, where the at least one nonlinear layer includes a pooling layer and/or an activation layer.
According to a possible implementation manner of the solution of the first aspect, the embodiment of the present application further provides that each of the plurality of linear layers indicates a linear transformation, the linear transformations indicated by the plurality of linear layers include a convolution operation and/or a full join operation, the deep learning model further includes at least one nonlinear layer indicating a nonlinear transformation, and the nonlinear transformation indicated by the at least one nonlinear layer includes a pooling operation and/or an activation function.
According to a possible implementation manner of the solution of the first aspect, the embodiment of the present application further provides that the training data held by the client includes training data for updating the trained deep learning model.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides an reasoning process of the deep learning model, including: for each linear layer of the plurality of linear layers, generating a first random vector corresponding to the linear layer by a data application side and a second random vector corresponding to the linear layer by the server side, wherein the data application side and the server side together construct a secret sharing security protocol corresponding to the linear layer based on the first random vector and the second random vector; the input data is encrypted by the data application party and then sent to the server side; and invoking the plurality of linear layers layer by layer according to an invoking order of the plurality of linear layers in the deep learning reasoning process, and encrypting an input of the invoked linear layer through a first random vector corresponding to the invoked linear layer and an output of the invoked linear layer through a second random vector corresponding to the invoked linear layer according to a secret sharing security protocol of the invoked linear layer.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that the application scenario of the deep learning model includes at least one of the following: intelligent traffic, medical health, image processing, natural language processing, and privacy calculations, the reasoning process of the deep learning model is used to accomplish one or more tasks in the context of the application of the deep learning model.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides encrypting, according to a secret sharing security protocol corresponding to the invoked linear layer, an input of the invoked linear layer by a first random vector corresponding to the invoked linear layer and an output of the invoked linear layer by a second random vector corresponding to the invoked linear layer, including: and performing addition operation or subtraction operation on the first random vector corresponding to the called linear layer and the input of the called linear layer to encrypt the input of the called linear layer, performing linear conversion on the encrypted input of the called linear layer through the linear component of the called linear layer, and adding the second random vector corresponding to the called linear layer to encrypt the output of the called linear layer.
According to a possible implementation manner of the technical solution of the first aspect, the embodiment of the present application further provides that the data application party and the client do not belong to the same platform, the same system, the same geographic location or the same network.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that the data application side includes a plurality of input data, and the reasoning process of the deep learning model includes, for each input data of the plurality of input data, constructing a secret sharing security protocol for each linear layer of the plurality of linear layers for the corresponding linear layer of the input data.
According to a possible implementation manner of the technical solution of the first aspect, the embodiment of the present application further provides that the client and the server and/or the data application side and the server are connected in a wired manner or a wireless manner.
According to a possible implementation manner of the technical solution of the first aspect, the embodiment of the present application further provides that the server side is a public cloud service provider with respect to the client side and/or the data application side.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that the online deep learning method adjusts multiplication operation in the training process of the deep learning model by approximately calculating homomorphic encryption CKKS algorithm, so as to reduce re-scale adjustment operation and re-linearization operation.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that the training process of the online deep learning method includes pre-calculating a transpose matrix of the forward propagation gradient and storing the transpose matrix in the client so as to accelerate calculation of the backward propagation gradient.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that the encrypted data communication between the client and the server and/or between the data application side and the server is based on a zero ciphertext, where the zero ciphertext is obtained by encrypting plaintext data that is zero.
According to a possible implementation manner of the first aspect, the embodiment of the present application further provides that each encrypted data communication is based on a new zero ciphertext, which is newly generated or selected from a plurality of zero ciphertexts that are pre-generated and the selected zero ciphertext is not multiplexed.
In a second aspect, embodiments of the present application provide a non-transitory computer-readable storage medium. The computer readable storage medium stores computer instructions that when executed by a processor implement the online deep learning method according to any one of the first aspects.
The technical scheme described in the second aspect provides privacy protection and simultaneously avoids the requirement that a server side is a trusted third party or noncompetitive, so that the server side is favorable for calling rich ready cloud resources, has lower requirements on communication overhead, calculation power and storage space, and therefore has better overall calculation efficiency and resource utilization efficiency, further overcomes the barriers in privacy protection, which are encountered when the online deep learning technology is promoted, and is favorable for providing a product solution which is convenient for commercialization and falls to the ground and is compatible with the aspects such as privacy protection, communication overhead, calculation power and storage space requirements, security requirements of a cloud service provider and the like.
In a third aspect, an embodiment of the present application provides an electronic device. The electronic device includes: a processor; a memory for storing processor-executable instructions; wherein the processor implements the online deep learning method according to any one of the first aspects by executing the executable instructions.
The technical scheme described in the third aspect provides privacy protection and simultaneously avoids the requirement that a server side is a trusted third party or is non-competitive, so that abundant ready cloud resources can be conveniently called, the requirements on communication overhead, calculation power and storage space are low, and therefore the requirements on the whole calculation efficiency and the resource utilization efficiency are high, the barrier in the aspect of privacy protection, which is encountered when the online deep learning technology is promoted, is overcome, and the method is helpful for providing a product solution which is convenient for commercialization and landing and takes into consideration the aspects of privacy protection, communication overhead, calculation power and storage space requirements, security requirements of a cloud service provider and the like.
In a fourth aspect, an embodiment of the present application provides an online deep learning system for privacy protection. The online deep learning system includes: the system comprises a server side, wherein the server side stores a deep learning model, and the deep learning model comprises a plurality of linear layers; and at least one client. Wherein the at least one client holds training data and the at least one client is communicatively connected with the server. Wherein the training process of the online deep learning system to train the deep learning model using training data held by the at least one client comprises: the at least one client side encrypts training data held by the at least one client side under the HE security protocol and sends the encrypted training data to the server side; the training data held by the at least one client side is imported into the deep learning model through the server side, and in the training process of the deep learning model, encryption processing under a DP security protocol and encryption processing under the HE security protocol are respectively carried out on respective linear components and bias components of each linear layer of the plurality of linear layers, so that an encrypted gradient is obtained; and obtaining encrypted model parameters based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters.
The technical scheme described in the fourth aspect provides privacy protection while avoiding the requirement that the server side is a trusted third party or noncompetitive, thereby being beneficial to calling abundant ready-made cloud resources, having lower requirements on communication overhead, calculation power and storage space, and therefore having better overall calculation efficiency and resource utilization efficiency, overcoming the barrier in privacy protection encountered when the online deep learning technology is promoted, and being beneficial to providing a product solution which is convenient for commercialization and falls to the ground and takes into consideration in multiple aspects such as privacy protection, communication overhead, calculation power and storage space requirements, security requirements of a cloud service provider and the like.
According to a possible implementation manner of the fourth aspect, the embodiment of the present application further provides that the encrypted gradient includes a linear component of the encrypted gradient and an offset component of the encrypted gradient. Wherein obtaining the encrypted model parameters based on the encrypted gradient comprises: the server integrates the bias components of the encrypted gradient to obtain the bias components of the encrypted model parameters, and the at least one client decrypts and noise-adds the linear components of the encrypted gradient to obtain the linear components of the encrypted model parameters. Wherein updating the deep learning model with the encrypted model parameters comprises: and updating the deep learning model by the server side by utilizing the linear component of the encrypted model parameter and the offset component of the encrypted model parameter.
According to a possible implementation manner of the fourth aspect, the embodiment of the present application further provides that the plurality of linear layers includes at least one convolution layer, the deep learning model further includes at least one nonlinear layer, the at least one nonlinear layer includes a pooling layer and/or an activation layer, each linear layer of the plurality of linear layers indicates a linear transformation, and the linear transformation indicated by the plurality of linear layers includes a convolution operation and/or a full-connection operation.
According to a possible implementation manner of the fourth aspect, the embodiment of the present application further provides that the online deep learning system further includes a data application party. The reasoning process of the deep learning model comprises the following steps: for each linear layer of the plurality of linear layers, generating a first random vector corresponding to the linear layer by the data application side and a second random vector corresponding to the linear layer by the server side, and constructing a secret sharing security protocol corresponding to the linear layer by the data application side and the server side together based on the first random vector and the second random vector; the input data is encrypted by the data application party and then sent to the server side; and invoking the plurality of linear layers layer by layer according to an invoking order of the plurality of linear layers in the deep learning reasoning process, and encrypting an input of the invoked linear layer through a first random vector corresponding to the invoked linear layer and an output of the invoked linear layer through a second random vector corresponding to the invoked linear layer according to a secret sharing security protocol of the invoked linear layer.
According to a possible implementation manner of the fourth aspect, the embodiment of the present application further provides that the client and the server and/or the data application side and the server are connected in a wired manner or a wireless manner, and the server is a public cloud service provider with respect to the client and/or the data application side.
Drawings
In order to describe the embodiments of the present application or the technical solutions in the background art, the following description will describe the drawings used in the embodiments of the present application or the background art.
Fig. 1 shows a flowchart of a training process of an online deep learning method according to an embodiment of the present application.
Fig. 2 is a schematic flow chart of an reasoning process of the online deep learning method according to an embodiment of the present application.
Fig. 3 shows a block diagram of an electronic device for the online deep learning method shown in fig. 1 and 2 according to an embodiment of the present application.
Fig. 4 shows a block diagram of an online deep learning system provided by an embodiment of the present application.
Detailed Description
The embodiment of the application provides an online deep learning system and method for privacy protection, which are used for solving the technical problems of how to achieve a better effect in the aspects of privacy protection, communication overhead, calculation power and storage space requirements, security requirements of a cloud service provider and the like. The online deep learning method comprises the following steps: the method comprises the steps that through a client, training data held by the client are encrypted under homomorphic encryption HE (Homomorphic Encryption) safety protocol, and the encrypted training data are sent to a server; training a deep learning model by utilizing encrypted training data received from the client through a server, and respectively carrying out encryption processing under a differential privacy DP (Differential Privacy) security protocol and encryption processing under the HE security protocol on linear components and bias components of at least one linear layer included in the deep learning model in the training process of training the deep learning model so as to obtain an encrypted gradient of the deep learning model; and obtaining encrypted model parameters of the deep learning model based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters. The embodiment of the application has the following beneficial technical effects: in fields such as finance, government affairs, intelligent transportation, medical health, image processing, natural language processing and the like, each participant holding private data can cooperatively train a deep learning model through an online mode such as public cloud service, reasoning is carried out by utilizing the trained model, model updating requirements are met, the online deep learning system and method not only provide privacy protection such as providing privacy protection on training data, reasoning data and model parameters, but also do not require that public cloud service providers or data integrators have to meet trusted third parties or non-competitive requirements so as to be beneficial to calling abundant ready-made cloud resources, meanwhile, requirements on communication overhead, calculation power and storage space in the training process and the reasoning process are low, and therefore, the online deep learning system and method have better overall operation efficiency and resource utilization efficiency, further overcome the obstacles in terms of protection encountered when the online deep learning technology is promoted, and are beneficial to providing a product proposal which is convenient for commercialization and landing in aspects such as privacy protection, communication overhead, calculation power and storage space requirements, safety privacy requirements of cloud service providers and the like.
The embodiment of the application can be used for application scenes including, but not limited to, any deep learning application scene which is suitable for privacy protection calculation or privacy calculation or other privacy protection related technologies, such as application scenes which adopt deep learning technology such as finance, government affairs, intelligent transportation, medical health, image processing, natural language and the like and are suitable for a plurality of participants through online services such as public cloud service, any application scene which is suitable for realizing collaborative model training and reasoning among a plurality of participants by utilizing public cloud service or other services which are difficult to meet the requirements of a trusted third party, any other application scene which is suitable for providing a product solution by utilizing online deep learning service with a privacy protection function, and the like.
The embodiments of the present application may be modified and improved according to specific application environments, and are not particularly limited herein.
It should be understood that the privacy preserving online deep learning system and method according to the embodiments of the present application, including various embodiments, implementations, and combinations and variants thereof, may be applicable to federal learning applications, may also be used in conjunction with specific industry applications, such as predicting travel paths when used for travel management of intelligent traffic, providing price predictions for stock and futures markets in stock exchange systems in the financial field, and the like.
In order that those skilled in the art will better understand the present application, embodiments of the present application will be described with reference to the accompanying drawings.
Referring to fig. 1 and fig. 2, fig. 1 is a schematic flow chart illustrating a training process of an online deep learning method according to an embodiment of the present application, and fig. 2 is a schematic flow chart illustrating an reasoning process of the online deep learning method according to an embodiment of the present application. The online deep learning can be divided into a training process and an reasoning process, wherein the training process refers to a process that a plurality of participants respectively provide training data to achieve the aim of cooperatively training a deep learning model, and the online deep learning can also comprise updating model parameters or retraining the deep learning model which happens frequently; the inference process refers to a party, such as an application party, holding input data for inference, transmitting the input data to a trained deep learning model and obtaining a calculation result. For specific online deep learning applications, such as deep learning models used in finance, government affairs, intelligent transportation or medical health, not only have better overall operation efficiency and resource utilization efficiency in training process and reasoning process, such as real-time reasoning or quick response to input data, but also face the requirement of updating the model with new finance data, government affair data, traffic data or medical data every day, that is, frequent and durable model updating or retraining, and have the requirements of realizing protection of training data privacy, protection of reasoning data privacy and protection of model parameter privacy. In order to achieve the above-mentioned advantageous technical effects, the online deep learning method and the online deep learning system mentioned below provided in the embodiments of the present application comprehensively utilize various privacy-preserving related technologies, and the following details of these privacy-preserving related technologies and how they are embodied in the present application.
Federal learning refers to implementing multi-party collaborative building of federal learning models on the premise of ensuring that data does not leave a safety control range, e.g., data is not transferred to the outside. Federal learning can be divided into transverse federal learning with larger overlap in the data feature space and smaller overlap in the sample space, longitudinal federal learning with smaller overlap in the data feature space and larger overlap in the sample space, and federal transfer learning with smaller overlap in both the data feature space and the sample space according to the distribution of the training data in the data feature space and sample space between different participants. Secure multiparty computing, i.e. MPC, refers to the problem that mutually untrusted parties obtain the result of a computation by co-computing a given function while guaranteeing the privacy of the input and output information of the parties. Application techniques related to MPC include the transmission in the confusing (Oblivious Transfer, OT) also known as unintentional transmission, and Garbled Circuits (gardled Circuits). Where OT refers to the condition of a suppressed query by, for example, confusing a manner in which multiple secret messages are sent and only one of the secret messages can be selected for retrieval by the recipient. Garbled circuits refer to the construction of an encryption circuit by means of boolean logic circuits such as and gates or gates. Application technologies related to MPC also include Secret Sharing (Secret Sharing) or what is known as Secret Sharing. Secret sharing refers to that secret information is divided into a plurality of secret shares and then distributed to a plurality of participants for storage respectively, so that secret information can be calculated or recovered only by cooperation of more than a specific number of participants, and secret information cannot be obtained by less than the specific number of participants, thereby achieving the purposes of dispersing risks and resisting intrusion. Homomorphic encryption (Homomorphic Encryption, HE) refers to a ciphertext obtained by encrypting an original text or a plaintext, and then performing an operation on the ciphertext, which is equivalent to performing an operation on the plaintext and then encrypting. HE is divided into homomorphic encryption (Fully Homomorphic Encryption, FHE) and semi-homomorphic encryption (Somewhat Homomorphic Encryption, SHE). FHE generally refers to satisfying both the addition homomorphism and the multiplication homomorphism. SHE generally refers to supporting only an addition homomorphism or only one or a limited number of ciphertext multiplication operations. Other technologies that have important applications in privacy preservation include trusted execution environment (Trust Execution Environment, TEE) technology. TEE generally refers to providing confidentiality protection by building a secure area at the level of a central processing unit (Central Processing Unit, CPU) by means of software and hardware and loading applications and data within the built secure area. Such as Intel (r) advanced Intel (r) SGX technology, which loads applications and data into protected areas divided on a CPU, also called Enclave containers. Differential privacy (Differential Privacy, DP) refers to providing privacy protection for individual queries by adding noise to the results of a calculation, or reducing the risk of specific individual information being compromised through a data query by adding controlled noise. DP technology is divided into global DP (adding noise on the integrated computation results and thus requiring a trusted data integrator) and local DP (adding noise on each input and thus not requiring a trusted data integrator but resulting in reduced accuracy).
As described above, various conventional privacy-preserving related techniques have respective disadvantages. For example, if homomorphic encryption is adopted for both training data and model parameters, the requirement for trusted data integrators is eliminated although a better privacy protection effect is achieved, and meanwhile, larger communication overhead and higher computation power and storage space requirements are brought. For another example, if both training data and model parameters are encrypted by DP technology, there is a problem of insufficient privacy protection, because DP technology or DP security protocol-based solutions often require the use of unencrypted deep learning models on servers or on online service providers, which can be detrimental to protecting the privacy information of the model parameters. Other technologies, such as Intel SGX technology, require that dedicated protected areas be partitioned at the CPU level, and are thus unsuitable for use in an online environment. In addition, it has been found through research that on the premise of encrypting the training data, that is, on the premise of providing privacy protection for the training data, the main risk of the rest privacy protection comes from leakage of model parameters or gradient parameters calculated in the training process. By hiding the model parameters or gradient parameters, the risk that an attacker or malicious party acquires the private information by reconstructing the data can be effectively reduced. In addition, the encryption of the training data and the hiding of the model parameters/gradient parameters are carried out simultaneously, so that the training and real-time reasoning can be effectively carried out on the deep learning network on the premise of no trusted third party.
In practical applications, the deep learning network or the above-mentioned various deep learning application scenarios often involve one or more service providers or servers that perform functions such as data integration, model training, model updating, and model reasoning, and one or more data providers or clients that provide training data for training, input data for reasoning, or only requirements such as data queries and data applications. In a simplified application scenario, it may only involve one client a and one server B, with client a holding data and server B providing training and reasoning services. The client a and the server B are connected in an online manner, for example, through a network. During the training process, client a continuously collects data and transmits it to server B for updating the model. According to privacy-preserving related security protocols, server B sends the trained model to client a or stores the model at server B or other servers to provide subsequent security reasoning services. During the inference process, client A or other clients provide input data and transmit to server B or other servers to provide predicted results, such as classification results or semantic recognition results, based on the input data and the trained model. Taking the simplified application scenario including only one client a and one server B as an example, the application scenario can be easily generalized to any number of clients and any number of servers, for example, there may be two servers respectively used for training different deep learning models or one server is used for training and the other server is used for reasoning. The roles played by the clients may also be diverse, e.g., the same client is a participant providing training data for one deep learning application, but a participant providing input data for reasoning for another deep learning application. It is also understood that any number of clients and any number of servers may be connected via the same network, and that any sub-network may be connected to each other and then form a larger network in the form of a sub-network, or any suitable network architecture may exist inside them. The concepts of the online deep learning method, the online deep learning system, or the online deep learning service according to the embodiments of the present application should be understood to cover an application scenario where data interaction and communication are implemented between any number of clients and any number of servers in any suitable network architecture, network connection manner, communication connection means, and the like. For example, there may be two or more different network architectures, two or more different network connection manners, and two or more different communication connection manners, which may coexist in the same application scenario, and may all be applied to the online deep learning system and method for privacy protection provided by the embodiments of the present application. It should be appreciated that suitable network architectures or network configurations may include star configurations, token ring configurations, or other configurations. Suitable network connections may include wired connections, such as Ethernet, coaxial cable, fiber optic cable, etc., and wireless connections, such as Wi-Fi, bluetooth, or other wireless technologies. Suitable means of communication connection may include a public network (e.g., the Internet), a private network (e.g., a Local Area Network (LAN) or Wide Area Network (WAN)), a wired network (e.g., ethernet), a wireless network (e.g., an 802.11 network, a Wi-Fi network, or a Wireless LAN (WLAN)), a cellular network (e.g., a 5G network, a Long Term Evolution (LTE) network, etc.), a router, hub, switch, server computer, or a combination thereof.
In addition, the online deep learning system and method for privacy protection provided by the embodiments of the present application may be implemented in hardware, software, a combination of hardware and software, firmware, or any suitable form. For example, it may be implemented in the form of an application program loaded on a server, such as a public cloud server, or in the form of an application program loaded on a client, such as a user's mobile phone, or in the form of an integrated circuit device provided with dedicated hardware, such as a removable plug-in, on a motherboard of a data center computer, or in the form of computer program instructions or code executed on a computer, and such instructions or code may be stored on a computer program product comprising a non-transitory computer readable medium.
Referring to fig. 1, fig. 1 shows a training process of an online deep learning method, which includes the following steps.
Step S102: and through the client, carrying out encryption processing under homomorphic encryption HE security protocol on the training data held by the client, and sending the encrypted training data to the server.
Step S104: and training the deep learning model by utilizing encrypted training data received from the client through the server, and respectively carrying out encryption processing under a differential privacy DP security protocol and encryption processing under the HE security protocol on linear components and bias components of at least one linear layer included in the deep learning model in the training process of training the deep learning model, thereby obtaining the encrypted gradient of the deep learning model.
Step S106: obtaining encrypted model parameters of the deep learning model based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters.
Here, the client side and the server side represent simplified scenes in the online deep learning application scene, such as the above-mentioned client a and server B. It can be appreciated that the online deep learning method shown in fig. 1 can be generalized to an application scenario between any number of clients and any number of servers. As described above, if the homomorphic encryption mode is adopted for both the training data and the model parameters, for example, the method is based on a certain HE security protocol, the requirements on the trusted data integrator are eliminated although the better privacy protection effect is achieved, and meanwhile, larger communication overhead and higher calculation power and storage space requirements are brought. On the other hand, if both training data and model parameters are encrypted by DP technology, e.g. based on a certain DP security protocol, there is a problem of insufficient privacy protection, because DP technology or DP security protocol based solutions often require the use of an unencrypted deep learning model on the server or on the online service provider, which is detrimental to protecting the privacy information of the model parameters. Therefore, the embodiment of the application provides a composite security protocol for encryption operation in the training process by combining the structural characteristics of the deep neural network (Deep Neural Networks, DNN), thereby protecting the privacy information of training data, gradient parameters, model parameters and the like and overcoming the defects.
In particular, the basic architecture of convolutional neural networks (Convolutional Neural Networks, CNN) and DNN and also of multi-layer perceptrons (Multilayer Perceptron, MLP) comprises two parts, one being a linear layer and the other being a non-linear layer. Examples of linear layers include convolutional layers and fully-connected layers, examples of non-linear layers include pooling layers such as max-pooling layers and activation functions, etc. The training process of DNN and CNN is an alternating of forward propagation and reverse propagation. New training data is imported into the input layer and then passed down layer by layer until the output layer, such forwardIn the propagation process, corresponding linear conversion is performed on the linear layer and corresponding nonlinear conversion is performed on the nonlinear layer, for example, an output result of a certain hidden layer is calculated based on the output of the upper layer and the weight parameter of the hidden layer and is transmitted to the next layer. The back propagation process refers to calculating the gradient of the loss function to each parameter, for example, by a random gradient descent algorithm, and updating the parameters according to the gradient. The direction of counter-propagation is passed forward from the last layer. Here, the deep learning model is provided with K total linear layers, where the output of the ith linear layer during forward propagation can be expressed as: a, a i+1 ＝f(W T i a i +b i ). Wherein a is i Representing the input tensor of the ith linear layer, a can be used 0 Representing the original input data, i.e. the new training data received by the input layer. W (W) i Representing the linear component of the ith linear layer, b i Representing the bias composition of the ith linear layer. The linear transformation by the i-th linear layer may be embodied by the linear component and the bias component of the i-th linear layer. W (W) T i Meaning the transpose of the matrix. And the function f represents a composite function including pooling operations, nonlinear activation functions, and the like. Thus, the linear transformation by the ith linear layer can be expressed as performing such an operation: w (W) T i a i +b i . If the respective linear components of the K total linear layers are grouped into a composite sequence W and the respective bias components are grouped into a composite sequence b, the linear components and bias components of the linear layers characterizing the deep learning model can be obtained. In step S104, encryption processing under the differential privacy DP security protocol and encryption processing under the HE security protocol are performed on the linear component and the offset component of at least one linear layer included in the deep learning model, respectively. This means that taking the ith linear layer as an example, the linear component W of the ith linear layer i Encryption under DP security protocol is performed, and the bias component b of the ith linear layer i Then encryption processing under HE security protocol is performed. A composite security protocol constructed by different encryption means provides good hidden information through the HE security protocolPrivacy protection, on the other hand, simplifies the overall computational complexity through the DP protocol.
In step S104, the deep learning model is encrypted by the above-described composite security protocol in order to protect the privacy information of the gradient and model parameters related thereto. Specifically, taking the ith linear layer as an example, the linear component W of the ith linear layer i Encryption under DP security protocol is performed, and the bias component b of the ith linear layer i Then encryption processing under HE security protocol is performed. The encryption under the DP security protocol may be any suitable encryption means that adds DP noise, such as by adding global DP noise. Since the encryption processing under the DP security protocol means that DP noise is directly added to the calculation result, unlike the encryption processing under the HE security protocol, the encryption processing converts plaintext data into ciphertext data, where the plaintext data to be protected is directly added with DP noise and thus can still be regarded as plaintext data or has a corresponding calculation complexity of plaintext data. The forward propagation process after such encryption processing, the operations at each layer can be characterized as a as described above i+1 ＝f(W T i a i +b i ) Here because of the linear component W of the ith linear layer i The encryption processing under the DP security protocol is only carried out, so that the encryption processing is equivalent to the encryption of linear components, and the operation cost caused by the multiplication operation of the ciphertext and the ciphertext under the homomorphism can be avoided. In contrast, only the multiplication operation of ciphertext and plaintext and the addition operation of ciphertext under homomorphism are involved in the training process, particularly in the forward propagation, so that the homomorphism ciphertext and ciphertext multiplication operation, particularly homomorphism ciphertext matrix multiplication operation, with the largest operation cost is avoided. In addition, since the linear component of the linear layer only carries out encryption processing under the DP security protocol, the size of data represented by the linear component is equivalent to that of plaintext data, compared with encryption processing under the HE security protocol, the encryption processing under the HE security protocol can lead to larger ciphertext data size or ciphertext data with large whole digital width, so that the processing can save storage space, and various latest secret sharing protocols can be conveniently used for realizing real-time reasoning or real-time prediction feedback in the reasoning process. In addition, because of the stepStep S102 performs encryption processing based on the HE security protocol on the training data, and performs encryption processing under the differential privacy DP security protocol and encryption processing under the HE security protocol on the linear component and the bias component of at least one linear layer included in the deep learning model in step S104, so that the privacy information of the training data and the deep learning model is protected through the privacy protection technology, and the trusted third party or competitive requirements can be avoided. That is, through the above-mentioned composite security protocol, the online deep learning method does not require that the public cloud service provider or the data integrator have to meet the requirements of a trusted third party or a non-competitive, for example, does not require that the server side or the party providing the server side with the computing or storage service on the cloud meet the requirements of the trusted third party or the non-competitive, which is beneficial to calling rich ready cloud resources such as commercial cloud resources or public cloud resources. On the other hand, since the encryption processing under the HE security protocol is performed on the bias components, the bias components in the linear layer of the deep learning model are hidden. Taking the ith linear layer as an example, by biasing component b of the ith linear layer i The encryption processing under the HE security protocol is equivalent to hiding the related privacy information of a malicious party or an attacker, so that the malicious party or the attacker is difficult to acquire the input characteristics. This is described below in connection with the back propagation process of the deep learning model.
In step S106: and obtaining encrypted model parameters of the deep learning model based on the encrypted gradient. The above-mentioned forward propagation after encryption processing, the operations at each layer can be characterized as a as described above i+1 ＝f(W T i a i +b i ) This is passed layer by layer to the final generation of the loss function and the updating of the model parameters is performed by calculating the gradient of the loss function with respect to the model parameters by a calculation strategy that minimizes the loss function, such as a random gradient descent algorithm or the like. Moreover, the process of back propagation is also performed layer by layer, except that the order is reversed from forward propagation. Here, let the loss function of the deep learning model be L (θ), where θ represents the model parametersA number.The gradient of the calculated loss function L (θ) with respect to the model parameter θ is represented, including the linear component W and the bias component b of the linear layer of the deep learning model. Taking the ith linear layer as an example, performing a back propagation operation at the ith linear layer is expressed as: / >Wherein for the ith linear layer the gradient propagation direction or counter propagation direction is from the (i+1) -th layer, so it is the gradient of the ith linear layer that is to be calculated, i.e./the +.>Wherein W is i Represents the linear component, W, of the ith linear layer T i Meaning the transpose of the matrix. />Is the gradient from layer i+1, f' [ a ] i+1 ]The calculations corresponding to the i+1th layer during forward propagation also represent part of the gradient calculations during backward propagation with respect to the nonlinear activation function. It can be seen that during the back propagation, the back propagation operation is performed at the ith linear layer, wherein the linear component W of the ith linear layer i Only encryption processing under the DP security protocol is performed and thus corresponds to linear component unencrypted. Meanwhile, other gradient calculation parts are consistent with the completely encrypted model, so that the calculation complexity of multiplication operation between homomorphic ciphertext matrixes can be saved in the gradient propagation process, and the calculation of the gradient of the ith linear layer is simplified>Is a process of (2). But at the same time, because the gradient computation involves the bias component b of the linear layer of the deep learning model in addition to the linear component W of the linear layer of the deep learning model. Because of the offset component The encryption processing under the HE security protocol is equivalent to hiding the bias components in the linear layer of the deep learning model, which means that the gradient calculation parts other than the linear components are equivalent to the model training process under homomorphic encryption, so that the model training process is consistent with the fully encrypted model.
In this way, the deep learning model is encrypted through the composite security protocol, particularly used for encryption operation in the training process, and particularly, the linear component and the offset component are respectively encrypted under the DP security protocol and the HE security protocol, so that the multiplication operation of ciphertext and plaintext and the ciphertext addition operation under homomorphism are only involved in the training process, particularly in the forward propagation process, and the homomorphism ciphertext and ciphertext multiplication operation, particularly homomorphism ciphertext matrix multiplication operation, with the largest operation cost is avoided; in addition, as the linear component of the linear layer only carries out encryption processing under the DP security protocol, the storage space is saved, and the calculation complexity of multiplication operation between homomorphic ciphertext matrixes is saved in the back propagation process or the gradient propagation process; in addition, the linear component is processed only by adding DP noise, so that the real-time reasoning or real-time predictive feedback is conveniently realized by using various latest secret sharing protocols in the reasoning process; in addition, because the encryption processing under the HE security protocol is carried out on the bias component, which is equivalent to hiding the bias component, and the encryption processing under the HE security protocol is carried out on the training data, the protection of privacy information such as the training data, gradient parameters and model parameters is facilitated, and therefore, the public cloud service provider or the data integrator is not required to meet the requirements of a trusted third party or non-competitive.
In one possible implementation, the post-encryption gradient includes a linear component of the post-encryption gradient and a bias component of the post-encryption gradient, wherein obtaining the post-encryption model parameters of the deep learning model based on the post-encryption gradient includes: the server integrates the bias components of the encrypted gradient to obtain the bias components of the encrypted model parameters, and the client decrypts and noise-adds the linear components of the encrypted gradient to obtain the linear components of the encrypted model parameters. As mentioned above, the deep learning model is encrypted by the above-mentioned composite security protocol, in particular for encryption operations in training processes, in particular for encryption processing under DP security protocol and encryption processing under HE security protocol for linear components and offset components, respectively. Here, the gradient is calculated by a random gradient descent algorithm or the like and used to update the model, because the linear component is subjected to encryption processing by adding DP noise, so that the linear component of the corresponding encrypted gradient is disturbed by DP noise. Therefore, the server side sends the linear component of the encrypted gradient to the client side, and then decrypts (e.g. by using a private key) and performs noise adding processing through the client side, so that the encryption processing under the DP security protocol can be ensured, that is, the noise added later can be ensured to meet the expected DP level. That is, the encryption processing under the DP security protocol is performed on the linear component at the server side, and the decryption and noise processing performed at the client side are combined, so that the client side and the server side together ensure that the expected DP level is satisfied. On the other hand, the bias components are encrypted under the HE security protocol, and because the bias components are encrypted by the HE, the server can integrate and average the gradient and then directly update the bias components of the model parameters. The method and the system realize the calculation of the encrypted gradients of all model parameters at the server side, and obtain the encrypted model parameters of the deep learning model based on the encrypted gradients together with the client side, so that the model update of the deep learning model can be realized.
In one possible implementation, updating the deep learning model with the encrypted model parameters includes: and updating the deep learning model by the server side by utilizing the linear component of the encrypted model parameter and the offset component of the encrypted model parameter. The server side can directly obtain the offset component of the encrypted model parameter, and together with the client side, obtain the linear component of the encrypted model parameter (the client side can send the linear component of the encrypted model parameter to the server side again after decryption and noise adding processing), and update the deep learning model stored in the server side or running in the server side through the linear component of the encrypted model parameter.
In one possible embodiment, the deep learning model includes a plurality of linear layers, wherein the encrypting process under the DP security protocol and the encrypting process under the HE security protocol are performed on the linear component and the bias component of at least one linear layer included in the deep learning model, respectively, including: and respectively carrying out encryption processing under the DP security protocol and encryption processing under the HE security protocol on the linear component and the bias component of each linear layer of the plurality of linear layers. The plurality of linear layers includes a convolutional layer and/or a fully-connected layer, and the deep learning model further includes at least one nonlinear layer including a pooling layer and/or an activation layer. Each linear layer of the plurality of linear layers indicates a linear transformation, the linear transformation indicated by the plurality of linear layers includes a convolution operation and/or a full join operation, the deep learning model further includes at least one nonlinear layer that indicates a nonlinear transformation, the nonlinear transformation indicated by the at least one nonlinear layer includes a pooling operation and/or an activation function. It should be appreciated that the linear layer and linear transformation may encompass any suitable mathematical relationship and formula. The activation function may refer to a common ReLu function, sigmoid function, or any suitable activation function that may be used for the deep learning model.
In one possible implementation, the training data held by the client includes training data for updating the trained deep learning model. The training process mentioned above with respect to the deep learning model may also be used to update the trained deep learning model, and the training data held by the client may also be understood to include training data for updating the trained deep learning model. For example, when the deep learning model is travel control for intelligent traffic or monitoring of medical health, the deep learning model may need to be continuously updated, including adjusting model parameters therein or optimizing model parameters therein according to new inputs and feedback, etc., which are similar to the training process, or may be implemented by importing training data for updating into the deep learning model and completing the forward and backward propagation processes.
In one possible implementation, the online deep learning method adjusts multiplication operations in the training process of the deep learning model by approximately computing homomorphic encryption CKKS algorithm, thereby reducing re-scale adjustment operations and re-linearization operations. As mentioned above, by saving the multiplication operation between ciphertext and ciphertext, such as homomorphic ciphertext matrix multiplication operation, the overall computational complexity can be effectively reduced. Here, the calculation efficiency can be further improved by CKKS algorithm. The multiplication operation in the training process is adjusted through the CKKS algorithm, specifically, the characteristic that encryption processing under the DP security protocol is carried out on linear components can be utilized through the multiplication operation behaviors among design model parameters, input features and gradients, so that the rescale adjustment operation and the re-linearization operation in the training process are greatly reduced, and the overall calculation efficiency is further improved.
In one possible implementation, the training process of the online deep learning method includes accelerating the computation of the backward propagation gradient by pre-computing a transpose of the forward propagation gradient and storing it at the client. Here, in order to save communication overhead between the client side and the server side, characteristics of gradient calculation on the nonlinear activation function section in each of the forward propagation process and the backward propagation process are combined. Specifically, as mentioned above, the back propagation operation performed by the ith linear layer is expressed as:this means that the non-linear activation function part of the gradient calculation of the back propagation process can be pre-calculated in the forward propagation process, and the pre-calculated calculation results are stored in the client, for example, the forward propagation buffer of the client, so that the pre-calculated calculation results can be directly extracted in the back propagation process, thereby further improving the calculation efficiency. For example, with the maximum pooling layer as an example of a nonlinear activation function, for maximumThe pooling layer may cache the mapping between the input features and the output lower sample features at the client and may cache in a plain text data format. In this way, the non-linear activation function part is cached in the client and is then retransmitted to the server in an encrypted communication mode after being extracted, so that the privacy information of related data is effectively protected.
In one possible implementation, the encrypted data communication between the client and the server is based on zero ciphertext obtained by encrypting plaintext data that is zero. In some embodiments, each encrypted data communication is based on a new zero ciphertext that is newly generated or selected from a pre-generated plurality of zero ciphertexts and the selected zero ciphertext is not multiplexed. Here, the zero ciphertext is based on plaintext data or original data that is zero. Because of the randomness in the encryption algorithm, ciphertext data obtained by encrypting zero plaintext data is not zero and may obtain different ciphertext data each time, that is, multiple encryptions may be performed with zero plaintext data and multiple zero ciphertexts may be generated. The zero ciphertext obtained by the method can be cached after being pre-generated so as to accelerate the encryption communication between the subsequent client and the server, for example, one zero ciphertext is selected each time when the need arises or a new zero ciphertext is generated and the communication data is encrypted, so that the communication efficiency can be effectively improved. The client may encrypt zero plaintext data and generate a data stream of zero ciphertext during a preprocessing stage.
It should be appreciated that the deep learning model and corresponding training process mentioned above only relate to how to encrypt the gradient parameters and model parameters in the training process and encrypt the training data, and these encryption processes are used to provide privacy protection in the online deep learning process, so as to help overcome the barriers in privacy protection encountered when the online deep learning technology is generalized. However, the online deep learning method and the relevant deep learning model according to the embodiments of the present application have no influence on implementing the method and obtain the corresponding beneficial technical effects due to factors such as network model architecture, network configuration, super parameters, etc., including how many linear layers, how many nonlinear layers, the number of neurons, weights, the number of hidden layers, and dimensions of each layer. In addition, the specific design of the loss function, the optimization strategy of the specifically adopted loss function and the like do not influence the implementation of the method and obtain corresponding beneficial technical effects. Therefore, the online deep learning method and the relevant deep learning model according to the embodiments of the present application may be applied to any machine learning model constructed based on the deep learning technology, including common CNN, DNN and MLP, may also be applied to other types or architecture of deep neural networks, and may be applied to machine learning models usable for any suitable task, including classification, prediction, regression, clustering, etc., may also be applied to machine learning models of single task or multiple tasks, and may also be applied to more complex systems or networks as long as some or most of these systems or networks are machine learning models constructed based on the deep learning technology.
Referring to fig. 2, fig. 2 is a schematic flow chart illustrating an reasoning process of the online deep learning method according to an embodiment of the present application. The deep learning model referred to by the online deep learning method shown in fig. 2 may be trained and possibly used for continuous updating using the training method shown in fig. 1, including any of the embodiments or combinations or variations thereof. The method comprises the following steps:
step S202: for each linear layer of the plurality of linear layers, generating a first random vector corresponding to the linear layer by a data application side and generating a second random vector corresponding to the linear layer by the server side, wherein the data application side and the server side together construct a secret sharing security protocol corresponding to the linear layer based on the first random vector and the second random vector.
Step S204: and the data application side encrypts the input data and sends the encrypted input data to the server side.
Step S206: the method comprises the steps of calling the plurality of linear layers layer by layer according to the calling sequence of the plurality of linear layers in the deep learning reasoning process, and encrypting input of the called linear layer through a first random vector corresponding to the called linear layer and encrypting output of the called linear layer through a second random vector corresponding to the called linear layer according to a secret sharing security protocol of the corresponding called linear layer.
The reasoning process of the deep learning model or the process of completing tasks through the deep learning model involves importing input data into the deep learning model and outputting corresponding calculation results such as prediction results, classification results or multi-task processing results. Here, in order to ensure the security of the reasoning process, the embodiment of the present application also provides the design of the optimized security protocol for the reasoning process. In the reasoning process, privacy security of input data, model parameters and reasoning results including intermediate results needs to be protected. It is therefore suitable to achieve decentralized risk and combat intrusion between multiple participants through a secret sharing protocol. As mentioned above, the trained deep learning model is located at the server side, where it is assumed that the party providing the input data or the party making the application request is the data application party, which may play a corresponding role depending on the specific application scenario. For example, when the deep learning model is used for travel management of intelligent traffic, the data application party may be a user such as a passenger or a traffic management authority who needs to predict a travel path by means of the deep learning model. In order to protect data security and privacy information and simultaneously avoid the requirement of a trusted third party in the reasoning process or the application process of the deep learning model, a secret sharing security protocol is adopted to complete data interaction. In addition, in order to ensure bidirectional privacy protection, namely, the data application side has enough privacy security protection relative to the server side, and the server side also has enough privacy security protection relative to the data application side; for this purpose, in step S202, for each of the plurality of linear layers, a first random vector corresponding to the linear layer is generated by a data application side and a second random vector corresponding to the linear layer is generated by the server side, the data application side and the server side together constructing a secret fraction corresponding to the linear layer based on the first random vector and the second random vector A shared security protocol. In this manner, bi-directional privacy security protection may be provided by generating a first random vector and a second random vector, respectively, and constructing a secret sharing security protocol based on the first random vector and the second random vector. In addition, as mentioned above, during the training process, the linear component of the linear layer is only encrypted under the DP security protocol, taking the linear conversion performed by the ith linear layer as an example, W T i a i +b i Wherein the linear component W of the ith linear layer i Encryption under DP security protocol is performed, and the bias component b of the ith linear layer i Then encryption processing under HE security protocol is performed. Therefore, the linear component can be converted into a fixed point representation and embedded in the finite ring, and then the first random vector and the second random vector generated for each linear layer respectively form a mask vector, thereby being beneficial to improving the calculation efficiency. Also, it should be understood that the first and second random vectors generated are vectors having the same dimensions as the input features.
In one possible implementation, the secret sharing security protocol corresponding to the linear layer may be constructed in such a way as to step S202. Here, the deep learning model is assumed to have K linear layers in total, and description will be made taking the ith linear layer as an example. Wherein the output of the ith linear layer during forward propagation can be expressed as: a, a i+1 ＝f(W T i a i +b i ). Wherein a is i Representing the input tensor of the ith linear layer, a can be used 0 Representing the original input data, i.e. the new training data received by the input layer. W (W) i Representing the linear component of the ith linear layer, b i Representing the bias composition of the ith linear layer. The first random vector corresponding to the ith linear layer is r i And the second random vector corresponding to it is s i . The data application party generates a first random vector r i Thereafter, the first random vector r i And after encryption, sending the encrypted data to a server side. Server side computation [ W i ·r i +b i -s i ]And send the calculation result to the data application party, numberDecrypted by the application side to obtain (W) i ·r i +b i -s i ). Thus, a relation (W i ·r i +b i ) Secret sharing of (c). It can be seen that, for each linear layer, a first random vector and a second random vector corresponding to the linear layer are generated respectively, and a secret sharing security protocol for the linear layer is constructed accordingly.
In step S206, in the reasoning process, these linear layers are called in order of call, and the secret sharing security protocol is constructed for each linear layer mentioned above, so that the input of the called linear layer can be encrypted by the first random vector corresponding to the called linear layer and the output of the called linear layer can be encrypted by the second random vector corresponding to the called linear layer. Thus, the privacy security protection of input data, model parameters and reasoning results including intermediate results is provided in the reasoning process.
In one possible implementation, the application scenario of the deep learning model includes at least one of the following: finance, government affairs, intelligent transportation, medical health, image processing, natural language processing and privacy calculation, wherein the reasoning process of the deep learning model is used for completing one or more tasks in the application scene of the deep learning model.
In one possible implementation, encrypting an input of a called linear layer by a first random vector corresponding to the called linear layer and encrypting an output of the called linear layer by a second random vector corresponding to the called linear layer according to a secret sharing security protocol corresponding to the called linear layer, includes: and performing addition operation or subtraction operation on the first random vector corresponding to the called linear layer and the input of the called linear layer to encrypt the input of the called linear layer, performing linear conversion on the encrypted input of the called linear layer through the linear component of the called linear layer, and adding the second random vector corresponding to the called linear layer to encrypt the output of the called linear layer. The description above is given by taking the ith linear layer as an example, and the ith linear layer The first random vector corresponding to the linear layer is r i And the second random vector corresponding to it is s i Between the data application side and the server side, a relation (W i ·r i +b i ) I.e. the i-th linear layer secret sharing security protocol. Still taking the ith linear layer as an example, i.e. the ith linear layer is taken as the invoked linear layer. Adding or subtracting the first random vector corresponding to the called linear layer and the input of the called linear layer to encrypt the input of the called linear layer, and taking the input of the ith linear layer as x and taking the subtraction operation as an example, calculating x-r i . The subtraction or encryption of the input of the linear layer that is called is performed by the data application, which then sends the result of the subtraction to the server. At the server side, the input of the called linear layer after encryption is subjected to linear conversion by the linear component of the called linear layer, and then a second random vector corresponding to the called linear layer is added, so that the output of the called linear layer is encrypted. This means calculating W i ·(x-r i )+s i . As mentioned above, the data application side obtains (W i ·r i +b i -s i ) Therefore, the two are added together to form a relation (W) i ·x+b i ) Secret sharing of (c). After encrypting the output of the invoked linear layer, the server sends the calculation result to the client, and the client may share the security protocol based on the secret corresponding to the ith linear layer, which has been previously generated in step S202, to obtain the required (W i ·x+b i ) That is, secret sharing between the data application and the server side is achieved. Taking the example of an addition operation, this means that x+r is calculated i . Therefore, only the encryption operation carried out on the server side in the follow-up related process is required to be adaptively modified, and finally the required secret sharing can be realized, and the details are not repeated here.
In one possible implementation, the data application includes a plurality of input data, and the reasoning process of the deep learning model includes, for each of the plurality of input data, constructing a secret sharing security protocol for each of the plurality of linear layers for the corresponding one of the input data. As mentioned above, for each linear layer, a first random vector and a second random vector corresponding thereto are generated, respectively, and the secret sharing security protocol for that linear layer is also constructed accordingly. Here, for better privacy protection, a whole set of secret sharing security protocols is also constructed for each input data. That is, also the i-th linear layer, there are different first random vectors, different second random vectors and different secret sharing security protocols for different input data, which is advantageous for providing better security protection.
In one possible implementation, the encrypted data communication between the data application and the server side is based on zero ciphertext obtained by encrypting plaintext data that is zero. In some embodiments, each encrypted data communication is based on a new zero ciphertext that is newly generated or selected from a pre-generated plurality of zero ciphertexts and the selected zero ciphertext is not multiplexed. Here, the zero ciphertext is based on plaintext data or original data that is zero. Because of the randomness in the encryption algorithm, ciphertext data obtained by encrypting zero plaintext data is not zero and may obtain different ciphertext data each time, that is, multiple encryptions may be performed with zero plaintext data and multiple zero ciphertexts may be generated. The zero ciphertexts obtained by the method can be cached after being pre-generated so as to accelerate the encryption communication between the subsequent data application party and the server, for example, one zero ciphertext is selected each time when the need arises or a new zero ciphertext is generated and the communication data is encrypted, so that the communication efficiency can be effectively improved. The server side can encrypt the plaintext data with zero in the preprocessing stage and generate a data stream with zero ciphertext.
Referring to fig. 1 and 2, the data application and the client do not belong to the same platform, the same system, the same geographic location, or the same network. The client and the server and/or the data application side and the server are connected in a wired or wireless mode. The server side is a public cloud service provider relative to the client side and/or the data application side. It should be appreciated that any suitable network architecture, network connection, communication connection means, etc. may be employed to effect data interaction and communication.
It should be appreciated that the deep learning model and corresponding reasoning process mentioned above only relate to how to encrypt model parameters in the reasoning process and encrypt input data for the reasoning process, which are used to provide privacy protection in the online deep learning process, thereby helping to overcome the barriers in privacy protection encountered when generalizing online deep learning techniques. However, the online deep learning method and the relevant deep learning model according to the embodiments of the present application have no influence on implementing the method and obtain the corresponding beneficial technical effects due to factors such as network model architecture, network configuration, super parameters, etc., including how many linear layers, how many nonlinear layers, the number of neurons, weights, the number of hidden layers, and dimensions of each layer. In addition, the specific design of the loss function, the optimization strategy of the specifically adopted loss function and the like do not influence the implementation of the method and obtain corresponding beneficial technical effects. Therefore, the online deep learning method and the relevant deep learning model according to the embodiments of the present application may be applied to any machine learning model constructed based on the deep learning technology, including common CNN, DNN and MLP, may also be applied to other types or architecture of deep neural networks, and may be applied to machine learning models usable for any suitable task, including classification, prediction, regression, clustering, etc., may also be applied to machine learning models of single task or multiple tasks, and may also be applied to more complex systems or networks as long as some or most of these systems or networks are machine learning models constructed based on the deep learning technology.
It should be understood that the above-described method may be implemented by a corresponding execution body or carrier. In some exemplary embodiments, a non-transitory computer readable storage medium stores computer instructions that, when executed by a processor, implement the above-described methods, as well as any of the above-described embodiments, implementations, or combinations thereof. In some example embodiments, an electronic device includes: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above-described methods and any of the above-described embodiments, implementations, or combinations thereof by executing the executable instructions.
Fig. 3 shows a block diagram of an electronic device for the online deep learning method shown in fig. 1 and 2 according to an embodiment of the present application. As shown in fig. 3, the electronic device includes a main processor 302, an internal bus 304, a network interface 306, a main memory 308, and a secondary processor 310 and secondary memory 312, as well as a secondary processor 320 and secondary memory 322. Where the main processor 302 is coupled to the main memory 308, the main memory 308 may be configured to store computer instructions executable by the main processor 302 such that the online deep learning methods illustrated in fig. 1 and 2 may be implemented, including some or all of the steps therein, as well as any possible combinations or combinations of the steps therein, and possible alternatives or variations thereof. Fig. 1 shows a flow chart of a training process of an online deep learning method according to an embodiment of the present application, and fig. 2 shows a flow chart of an reasoning process of the online deep learning method according to an embodiment of the present application. The electronic device shown in fig. 3 may be used only for the training process shown in fig. 1 or only for the reasoning process shown in fig. 2, or both the training process shown in fig. 1 and the reasoning process shown in fig. 2, and may also be used in the various embodiments mentioned above, including some or all of the steps therein, as well as any possible combinations or combinations of the steps therein and possible alternatives or variations thereof. The network interface 306 is used to provide network connectivity and to transmit and receive data over a network. The internal bus 304 is used to provide internal data interaction between the primary processor 302, the network interface 306, the secondary processor 310, and the secondary processor 320. Wherein the auxiliary processor 310 is coupled to and together with the auxiliary memory 312 to provide auxiliary computing power, and the auxiliary processor 320 is coupled to and together with the auxiliary memory 322 to provide auxiliary computing power. The auxiliary processor 310 and the auxiliary processor 320 may provide the same or different auxiliary computing capabilities, including, but not limited to, computing capabilities optimized for a particular computing need, such as parallel processing capabilities or tensor computing capabilities, computing capabilities optimized for a particular algorithm or logic structure, such as iterative computing capabilities or graph computing capabilities, and the like. The auxiliary processor 310 and the auxiliary processor 320 may include one or more processors of a specific type, such as a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), etc., so that customized functions and structures may be provided. In some exemplary embodiments, the electronic device may not include an auxiliary processor, may include only one auxiliary processor, may include any number of auxiliary processors, and each may have corresponding customized functions and structures, and is not specifically limited herein. The architecture of the two auxiliary processors shown in fig. 3 is merely illustrative and should not be construed as limiting. In addition, the main processor 302 may include single-core or multi-core computing units for providing the functions and operations necessary for embodiments of the present application. In addition, the main processor 302 and the auxiliary processors (e.g., the auxiliary processor 310 and the auxiliary processor 320 in fig. 3) may have different architectures, that is, the electronic device may be a heterogeneous architecture-based system, for example, the main processor 302 may be a general-purpose processor such as a CPU based instruction set operating system, and the auxiliary processor may be a graphics processor GPU suitable for parallelized computation or a dedicated accelerator suitable for neural network model-dependent operations. Auxiliary memories, such as auxiliary memory 312 and auxiliary memory 322 shown in fig. 3, may be used in conjunction with respective corresponding auxiliary processors to implement the customized functions and structures. And main memory 308 is used to store the necessary instructions, software, configurations, data, etc. to provide the functions and operations necessary for embodiments of the present application in conjunction with main processor 302. In some exemplary embodiments, the electronic device may not include auxiliary memory, may include only one auxiliary memory, and may include any number of auxiliary memories, which are not specifically limited herein. The architecture of the two auxiliary memories shown in fig. 3 is illustrative only and should not be construed as limiting. Main memory 308 and possibly secondary memory may include one or more of the following features: volatile, nonvolatile, dynamic, static, readable/writeable, read-only, random-access, sequential-access, location-addressability, file-addressability, and content-addressability, and may include random-access memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a recordable and/or rewriteable Compact Disc (CD), a Digital Versatile Disc (DVD), a mass storage media device, or any other form of suitable storage media. The internal bus 304 may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures. It should be understood that the electronic device shown in fig. 3, whose illustrated structure does not constitute a particular limitation with respect to the apparatus or system, may include more or less components than the particular embodiments and figures, or may combine certain components, or split certain components, or have different arrangements of components in some exemplary embodiments.
Fig. 4 shows a block diagram of an online deep learning system provided by an embodiment of the present application. The online deep learning system includes: the system comprises a server side, wherein the server side stores a deep learning model, and the deep learning model comprises a plurality of linear layers; and at least one client, wherein the at least one client holds training data and the at least one client is communicatively connected with the server. Wherein the training process of the online deep learning system to train the deep learning model using training data held by the at least one client comprises: the at least one client side encrypts training data held by the at least one client side under the HE security protocol and sends the encrypted training data to the server side; the training data held by the at least one client side is imported into the deep learning model through the server side, and in the training process of the deep learning model, encryption processing under a DP security protocol and encryption processing under the HE security protocol are respectively carried out on respective linear components and bias components of each linear layer of the plurality of linear layers, so that an encrypted gradient is obtained; and obtaining encrypted model parameters based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters. As shown in fig. 4, the online deep learning system includes a server side 410 and two clients 420 and 422. Clients 420 and 422 are exemplary only, and the online deep learning system of FIG. 4 may include any number of clients and any number of servers. It should be appreciated that any suitable network architecture, network connection, communication connection means, etc. may be employed to effect data interaction and communication.
In this way, the deep learning model is encrypted through the composite security protocol, particularly used for encryption operation in the training process, and particularly, the linear component and the offset component are respectively encrypted under the DP security protocol and the HE security protocol, so that the multiplication operation of ciphertext and plaintext and the ciphertext addition operation under homomorphism are only involved in the training process, particularly in the forward propagation process, and the homomorphism ciphertext and ciphertext multiplication operation, particularly homomorphism ciphertext matrix multiplication operation, with the largest operation cost is avoided; in addition, as the linear component of the linear layer only carries out encryption processing under the DP security protocol, the storage space is saved, and the calculation complexity of multiplication operation between homomorphic ciphertext matrixes is saved in the back propagation process or the gradient propagation process; in addition, the linear component is processed only by adding DP noise, so that the real-time reasoning or real-time predictive feedback is conveniently realized by using various latest secret sharing protocols in the reasoning process; in addition, because the encryption processing under the HE security protocol is carried out on the bias component, which is equivalent to hiding the bias component, and the encryption processing under the HE security protocol is carried out on the training data, the protection of privacy information such as the training data, gradient parameters and model parameters is facilitated, and therefore, the public cloud service provider or the data integrator is not required to meet the requirements of a trusted third party or non-competitive.
In one possible implementation, the post-encryption gradient includes a linear component of the post-encryption gradient and a biasing component of the post-encryption gradient. Wherein obtaining the encrypted model parameters based on the encrypted gradient comprises: the server integrates the bias components of the encrypted gradient to obtain the bias components of the encrypted model parameters, and the at least one client decrypts and noise-adds the linear components of the encrypted gradient to obtain the linear components of the encrypted model parameters. Wherein updating the deep learning model with the encrypted model parameters comprises: and updating the deep learning model by the server side by utilizing the linear component of the encrypted model parameter and the offset component of the encrypted model parameter.
In one possible implementation, the plurality of linear layers includes at least one convolution layer, the deep learning model further includes at least one nonlinear layer including a pooling layer and/or an activation layer, each of the plurality of linear layers indicates a linear transformation, and the linear transformation indicated by the plurality of linear layers includes a convolution operation and/or a full join operation.
In one possible implementation, the online deep learning system further includes a data application (not shown). The reasoning process of the deep learning model comprises the following steps: for each linear layer of the plurality of linear layers, generating a first random vector corresponding to the linear layer by the data application side and a second random vector corresponding to the linear layer by the server side, and constructing a secret sharing security protocol corresponding to the linear layer by the data application side and the server side together based on the first random vector and the second random vector; the input data is encrypted by the data application party and then sent to the server side; and invoking the plurality of linear layers layer by layer according to an invoking order of the plurality of linear layers in the deep learning reasoning process, and encrypting an input of the invoked linear layer through a first random vector corresponding to the invoked linear layer and an output of the invoked linear layer through a second random vector corresponding to the invoked linear layer according to a secret sharing security protocol of the invoked linear layer.
In a possible implementation manner, the client and the server side and/or the data application side and the server side are connected in a wired manner or a wireless manner, and the server side is a public cloud service provider relative to the client side and/or the data application side.
The embodiments of the present application may be implemented in any one or combination of hardware, software, firmware, or solid state logic circuits, and may be implemented in connection with signal processing, control and/or application specific circuits. Embodiments of the present application provide a device or apparatus that may include one or more processors (e.g., microprocessors, controllers, digital Signal Processors (DSPs), application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), etc.) which process various computer executable instructions to control the operation of the device or apparatus. An apparatus or device provided by embodiments of the present application may include a system bus or data transmission system that couples the various components together. A system bus may include any of several different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus using any of a variety of bus architectures. The apparatus or device provided by the embodiments of the present application may be provided separately, may be part of a system, or may be part of other apparatus or device.
Particular embodiments of the present application may include or be combined with computer-readable storage media, such as one or more storage devices capable of providing non-transitory data storage. The computer-readable storage medium/storage device may be configured to hold data, programmers and/or instructions that, when executed by a processor of an apparatus or device provided by a particular embodiment of the present application, cause the apparatus or device to perform the relevant operations. The computer-readable storage medium/storage device may include one or more of the following features: volatile, nonvolatile, dynamic, static, readable/writable, read-only, random access, sequential access, location addressability, file addressability, and content addressability. In one or more exemplary embodiments, the computer readable storage medium/storage device may be integrated into a device or apparatus provided by embodiments of the present application or belong to a common system. Computer-readable storage media/memory devices may include optical storage devices, semiconductor storage devices and/or magnetic storage devices, etc., as well as Random Access Memory (RAM), flash memory, read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a recordable and/or rewriteable Compact Disc (CD), a Digital Versatile Disc (DVD), a mass storage media device, or any other form of suitable storage media.
The foregoing is a description of embodiments of the present application, and it should be noted that, steps in the method described in the specific embodiment of the present application may be sequentially adjusted, combined, and deleted according to actual needs. In the foregoing embodiments, the descriptions of the embodiments are focused on, and for those portions of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments. It will be appreciated that the embodiments of the application and the structures shown in the drawings do not constitute a particular limitation with respect to the apparatus or system. In other embodiments of the application, the device or system may include more or less components than the specific embodiments and figures, or may combine certain components, split certain components, or have a different arrangement of components. It will be understood by those skilled in the art that various modifications or changes in arrangement, operation and details of the methods and apparatus described in the specific embodiments may be made without departing from the spirit and scope of the specific embodiments of the application; improvements and modifications can be made without departing from the principles of the embodiments of the present application, which are also considered to be within the scope of the present application.

Claims (24)

1. An online deep learning method for privacy protection, which is characterized by comprising the following steps:
The method comprises the steps that through a client, encryption processing under homomorphic encryption HE security protocol is carried out on training data held by the client, and the encrypted training data are sent to a server;
training a deep learning model by utilizing encrypted training data received from the client through a server, and respectively carrying out encryption processing under a differential privacy DP security protocol and encryption processing under the HE security protocol on linear components and bias components of at least one linear layer included in the deep learning model in the training process of training the deep learning model so as to obtain an encrypted gradient of the deep learning model; and obtaining encrypted model parameters of the deep learning model based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters.
2. The online deep learning method of claim 1, wherein the post-encryption gradient includes a linear component of the post-encryption gradient and an offset component of the post-encryption gradient, wherein obtaining the post-encryption model parameters of the deep learning model based on the post-encryption gradient includes: the server integrates the bias components of the encrypted gradient to obtain the bias components of the encrypted model parameters, and the client decrypts and noise-adds the linear components of the encrypted gradient to obtain the linear components of the encrypted model parameters.
3. The online deep learning method of claim 2 wherein updating the deep learning model with the encrypted model parameters comprises: and updating the deep learning model by the server side by utilizing the linear component of the encrypted model parameter and the offset component of the encrypted model parameter.
4. The online deep learning method of claim 3, wherein the deep learning model includes a plurality of linear layers, wherein the encrypting process under the DP security protocol and the encrypting process under the HE security protocol are performed on the linear component and the bias component of at least one linear layer included in the deep learning model, respectively, comprising: and respectively carrying out encryption processing under the DP security protocol and encryption processing under the HE security protocol on the linear component and the bias component of each linear layer of the plurality of linear layers.
5. The online deep learning method of claim 4, wherein the plurality of linear layers comprise convolutional layers and/or fully-connected layers, the deep learning model further comprising at least one nonlinear layer comprising a pooling layer and/or an activation layer.
6. The online deep learning method of claim 4, wherein each of the plurality of linear layers indicates a linear transformation, the linear transformations indicated by the plurality of linear layers including convolution operations and/or full join operations, the deep learning model further including at least one nonlinear layer indicating a nonlinear transformation, the nonlinear transformation indicated by the at least one nonlinear layer including pooling operations and/or activation functions.
7. The online deep learning method of claim 5 wherein the training data held by the client includes training data for updating the trained deep learning model.
8. The online deep learning method of claim 5 wherein the reasoning process of the deep learning model comprises:
for each linear layer of the plurality of linear layers, generating a first random vector corresponding to the linear layer by a data application side and a second random vector corresponding to the linear layer by the server side, wherein the data application side and the server side together construct a secret sharing security protocol corresponding to the linear layer based on the first random vector and the second random vector;
the input data is encrypted by the data application party and then sent to the server side; and
the method comprises the steps of calling the plurality of linear layers layer by layer according to the calling sequence of the plurality of linear layers in the deep learning reasoning process, and encrypting input of the called linear layer through a first random vector corresponding to the called linear layer and encrypting output of the called linear layer through a second random vector corresponding to the called linear layer according to a secret sharing security protocol of the corresponding called linear layer.
9. The online deep learning method of claim 8, wherein the application scenario of the deep learning model comprises at least one of: finance, government affairs, intelligent transportation, medical health, image processing, natural language processing and privacy calculation, wherein the reasoning process of the deep learning model is used for completing one or more tasks in the application scene of the deep learning model.
10. The online deep learning method of claim 8, wherein encrypting the input of the invoked linear layer by a first random vector corresponding to the invoked linear layer and encrypting the output of the invoked linear layer by a second random vector corresponding to the invoked linear layer according to a secret sharing security protocol corresponding to the invoked linear layer comprises:
the first random vector corresponding to the invoked linear layer and the input of the invoked linear layer are added or subtracted to encrypt the input of the invoked linear layer,
the input of the called linear layer after encryption is subjected to linear conversion through the linear component of the called linear layer, and then a second random vector corresponding to the called linear layer is added, so that the output of the called linear layer is encrypted.
11. The online deep learning method of claim 10, wherein the data application and the client do not belong to the same platform, the same system, the same geographic location, or the same network.
12. The online deep learning method of claim 8 wherein the data application includes a plurality of input data and the reasoning process of the deep learning model includes, for each of the plurality of input data, constructing a secret sharing security protocol for each of the plurality of linear layers for the input data corresponding to the linear layer.
13. The online deep learning method of claim 8, wherein the server is a public cloud service provider with respect to the client and/or the data application.
14. The online deep learning method of claim 8, wherein the online deep learning method adjusts multiplication operations during training of the deep learning model by approximating homomorphic encryption CKKS algorithm to reduce re-scale adjustment operations and re-linearization operations.
15. The online deep learning method of claim 8 wherein the training process of the online deep learning method includes accelerating the computation of the backward propagation gradient by pre-computing a transpose of the forward propagation gradient and storing it at the client.
16. The online deep learning method of claim 8, wherein the encrypted data communication between the client and the server and/or between the data application and the server is based on a zero ciphertext obtained by encrypting plaintext data that is zero.
17. The online deep learning method of claim 16 wherein each encrypted data communication is based on a new zero ciphertext that is newly generated or selected from a pre-generated plurality of zero ciphertexts and the selected zero ciphertext is not multiplexed.
18. A non-transitory computer readable storage medium storing computer instructions which, when executed by a processor, implement an online deep learning method according to any one of claims 1 to 17.
19. An electronic device, the electronic device comprising:
a processor;
a memory for storing processor-executable instructions;
wherein the processor implements the online deep learning method according to any one of claims 1 to 17 by executing the executable instructions.
20. An online deep learning system for privacy protection, the online deep learning system comprising:
the system comprises a server side, wherein the server side stores a deep learning model, and the deep learning model comprises a plurality of linear layers; and
at least one client, wherein the at least one client holds training data and the at least one client is communicatively connected to the server, wherein the training process of the online deep learning system to train the deep learning model using the training data held by the at least one client comprises:
the at least one client side encrypts training data held by the at least one client side under the HE security protocol and sends the encrypted training data to the server side;
the training data held by the at least one client side is imported into the deep learning model through the server side, and in the training process of the deep learning model, encryption processing under a DP security protocol and encryption processing under the HE security protocol are respectively carried out on respective linear components and bias components of each linear layer of the plurality of linear layers, so that an encrypted gradient is obtained; and
Obtaining encrypted model parameters based on the encrypted gradient, and updating the deep learning model with the encrypted model parameters.
21. The online deep learning system of claim 20 wherein the post-encryption gradient includes a linear component of the post-encryption gradient and an offset component of the post-encryption gradient,
wherein obtaining the encrypted model parameters based on the encrypted gradient comprises: the server integrates the bias components of the encrypted gradient to obtain the bias components of the encrypted model parameters, and the at least one client decrypts and noise-adds the linear components of the encrypted gradient to obtain the linear components of the encrypted model parameters,
wherein updating the deep learning model with the encrypted model parameters comprises: and updating the deep learning model by the server side by utilizing the linear component of the encrypted model parameter and the offset component of the encrypted model parameter.
22. The online deep learning system of claim 21 wherein the plurality of linear layers includes at least one convolutional layer and/or fully-connected layer, each linear layer of the plurality of linear layers indicating a linear transformation, the linear transformation indicated by the plurality of linear layers including a convolutional operation and/or fully-connected operation, the deep learning model further including at least one nonlinear layer including a pooling layer and/or an activation layer, the at least one nonlinear layer indicating a nonlinear transformation, the nonlinear transformation indicated by the at least one nonlinear layer including a pooling operation and/or an activation function.
23. The online deep learning system of claim 22 further comprising a data applicator, wherein the reasoning process of the deep learning model comprises:
for each linear layer of the plurality of linear layers, generating a first random vector corresponding to the linear layer by the data application side and a second random vector corresponding to the linear layer by the server side, and constructing a secret sharing security protocol corresponding to the linear layer by the data application side and the server side together based on the first random vector and the second random vector;
the input data is encrypted by the data application party and then sent to the server side; and
the method comprises the steps of calling the plurality of linear layers layer by layer according to the calling sequence of the plurality of linear layers in the deep learning reasoning process, and encrypting input of the called linear layer through a first random vector corresponding to the called linear layer and encrypting output of the called linear layer through a second random vector corresponding to the called linear layer according to a secret sharing security protocol of the corresponding called linear layer.
24. The online deep learning system of claim 23 wherein the client and the server and/or the data application and the server are connected by wire or wirelessly and the server is a public cloud service provider with respect to the client and/or the data application.
CN202210247646.4A 2022-03-14 2022-03-14 Online deep learning system and method for privacy protection Pending CN116796338A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210247646.4A CN116796338A (en) 2022-03-14 2022-03-14 Online deep learning system and method for privacy protection

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210247646.4A CN116796338A (en) 2022-03-14 2022-03-14 Online deep learning system and method for privacy protection

Publications (1)

Publication Number Publication Date
CN116796338A true CN116796338A (en) 2023-09-22

Family

ID=88038989

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210247646.4A Pending CN116796338A (en) 2022-03-14 2022-03-14 Online deep learning system and method for privacy protection

Country Status (1)

Country Link
CN (1) CN116796338A (en)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117371558A (en) * 2023-12-04 2024-01-09 环球数科集团有限公司 System for executing machine learning in privacy protection environment
CN117874794A (en) * 2024-03-12 2024-04-12 北方健康医疗大数据科技有限公司 Training method, system and device for large language model and readable storage medium

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117371558A (en) * 2023-12-04 2024-01-09 环球数科集团有限公司 System for executing machine learning in privacy protection environment
CN117371558B (en) * 2023-12-04 2024-03-08 环球数科集团有限公司 System for executing machine learning in privacy protection environment
CN117874794A (en) * 2024-03-12 2024-04-12 北方健康医疗大数据科技有限公司 Training method, system and device for large language model and readable storage medium

Similar Documents

Publication Publication Date Title
Dong et al. Eastfly: Efficient and secure ternary federated learning
CN110008717B (en) Decision tree classification service system and method supporting privacy protection
WO2022148283A1 (en) Data processing method and apparatus, and computer device, storage medium and program product
Esposito et al. Securing collaborative deep learning in industrial applications within adversarial scenarios
US20200366459A1 (en) Searching Over Encrypted Model and Encrypted Data Using Secure Single-and Multi-Party Learning Based on Encrypted Data
Makri et al. Rabbit: Efficient comparison for secure multi-party computation
Liu et al. Secure model fusion for distributed learning using partial homomorphic encryption
CN116796338A (en) Online deep learning system and method for privacy protection
CN112347500B (en) Machine learning method, device, system, equipment and storage medium of distributed system
CN114696990B (en) Multi-party computing method, system and related equipment based on fully homomorphic encryption
Hijazi et al. Secure federated learning with fully homomorphic encryption for iot communications
Miao et al. Federated deep reinforcement learning based secure data sharing for Internet of Things
CN113298268A (en) Vertical federal learning method and device based on anti-noise injection
WO2024093426A1 (en) Federated machine learning-based model training method and apparatus
He et al. Privacy-preserving and low-latency federated learning in edge computing
CN115563859A (en) Power load prediction method, device and medium based on layered federal learning
Zhang et al. Privacyeafl: Privacy-enhanced aggregation for federated learning in mobile crowdsensing
CN113792890A (en) Model training method based on federal learning and related equipment
Shankar et al. Multiple share creation with optimal hash function for image security in wsn aid of ogwo
Xu et al. Privacy-preserving federal learning chain for internet of things
Panzade et al. Privacy-Preserving Machine Learning Using Functional Encryption: Opportunities and Challenges
Nishida et al. Efficient secure neural network prediction protocol reducing accuracy degradation
Wang et al. FRNet: An MCS framework for efficient and secure data sensing and privacy protection in IoVs
Liu et al. PPEFL: An Edge Federated Learning Architecture with Privacy‐Preserving Mechanism
CN116795846A (en) Network node cooperation method for privacy calculation and cloud computing network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination