CN113420886B - Training method, device, equipment and storage medium for longitudinal federal learning model - Google Patents
Training method, device, equipment and storage medium for longitudinal federal learning model Download PDFInfo
- Publication number
- CN113420886B CN113420886B CN202110688660.3A CN202110688660A CN113420886B CN 113420886 B CN113420886 B CN 113420886B CN 202110688660 A CN202110688660 A CN 202110688660A CN 113420886 B CN113420886 B CN 113420886B
- Authority
- CN
- China
- Prior art keywords
- encryption
- model
- loss
- training
- encryption loss
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012549 training Methods 0.000 title claims abstract description 135
- 238000000034 method Methods 0.000 title claims abstract description 107
- 238000004364 calculation method Methods 0.000 claims abstract description 79
- 238000012795 verification Methods 0.000 claims description 23
- 238000004590 computer program Methods 0.000 claims description 15
- 238000004422 calculation algorithm Methods 0.000 claims description 4
- 239000003999 initiator Substances 0.000 description 49
- 238000012360 testing method Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 238000007477 logistic regression Methods 0.000 description 6
- 238000012550 audit Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 230000002452 interceptive effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 1
- 230000007423 decrease Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Computation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Medical Informatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Storage Device Security (AREA)
Abstract
The application relates to a training method, a training device, computer equipment and a storage medium of a longitudinal federal learning model, wherein the method comprises the following steps: generating an encryption public key for encrypting data and parameters of the model and a decryption private key for decrypting an encryption loss value of the model; zero knowledge proof is carried out on promised data of the acquired model, and the promised data is obtained by carrying out data promise on a source data set by each second device participating in model training. And obtaining encryption loss initialization parameters of the model to carry out zero knowledge proof, wherein the encryption loss initialization parameters are calculated by each second device based on the encryption data. And obtaining an encryption loss value of the model to carry out zero knowledge proof, wherein the encryption loss value is calculated by each second device based on the encryption loss initialization parameter and the encryption loss calculation weight. And decrypting the encrypted loss value based on the decryption private key to obtain a loss value of the model, and obtaining a trained longitudinal federal learning model when the loss value is smaller than a preset threshold value.
Description
Technical Field
The present application relates to the field of internet technologies, and in particular, to a training method and apparatus for a longitudinal federal learning model, a computer device, and a storage medium.
Background
Most of the existing federal learning assumes that the participants are semi-honest, i.e., model training is only performed according to the protocol flow without any vandalism. But this assumption is too ideal for practical use. In fact, federally learned participants may have malicious nodes, in which case the participant needs to prove that his own training process is accurate, legal and consistent, i.e., has the need for integrity for security auditing; meanwhile, since the original purpose of federal learning is to protect the privacy of the data used by the participants for training, the data and gradient information of the participants cannot be exposed in the verification process.
Disclosure of Invention
The application provides a training method, a device, computer equipment and a storage medium of a longitudinal federal learning model, which are used for realizing zero knowledge proof of encryption loss calculation in the training process of the longitudinal federal learning model, determining the ending training condition of the model according to the encryption loss value passing the zero knowledge proof and avoiding the overfitting of the model.
In a first aspect, the present application provides a method of training a longitudinal federal learning model, the method comprising:
Generating an encryption public key and a decryption private key; the encryption public key is used for sending the encryption public key to each second device participating in model training to encrypt data and parameters of the model, and the decryption private key is used for decrypting the encryption loss value of the model to obtain the loss value;
Zero knowledge proof is carried out on the promised data of the model, and a zero knowledge proof result of the promised data is obtained, wherein the promised data is obtained by carrying out data promise on a source data set by each second device;
obtaining encryption loss initialization parameters of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss initialization parameters, wherein the encryption loss initialization parameters are obtained by each second device through encryption loss initialization calculation based on the source data set;
Obtaining an encryption loss value of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss value, wherein the encryption loss value is calculated by each second device based on the encryption loss initialization parameter and the encryption loss calculation weight;
And decrypting the encrypted loss value based on the decryption private key to obtain a loss value in the training process of the longitudinal federal learning model, and obtaining a trained longitudinal federal learning model when the loss value is smaller than a preset threshold value.
In a second aspect, the present application also provides a training apparatus for a longitudinal federal learning model, the apparatus comprising:
The secret key generation module is used for generating an encryption public key and a decryption private key; the encryption public key is sent to each second device participating in model training and used for encrypting data and parameters of the model, and the decryption private key is used for decrypting encryption loss values of the model;
The data proving module obtains the promised data of the model to carry out zero knowledge proving, and obtains a zero knowledge proving result of the promised data, wherein the promised data is obtained by carrying out data promise on a source data set by each second device;
The encryption loss initialization proving module is used for obtaining encryption loss initialization parameters of the model to carry out zero knowledge proving and obtaining zero knowledge proving results of the encryption loss initialization parameters, wherein the encryption loss initialization parameters are obtained by each second device through encryption loss initialization calculation based on the source data set;
The encryption loss calculation proving module is used for obtaining an encryption loss value of the model to carry out zero knowledge proving and obtaining a zero knowledge proving result of the encryption loss value, wherein the encryption loss value is calculated by each second device based on the encryption loss initialization parameter and the encryption loss calculation weight;
And the model output module is used for decrypting the encryption loss value based on the decryption private key to obtain a loss value in the model training process, and obtaining a trained longitudinal federal learning model when the loss value is smaller than a preset threshold value.
In a third aspect, the present application also provides a computer device comprising a memory and a processor; the memory is used for storing a computer program; the processor is configured to execute the computer program and implement a training method of the vertical federal learning model as described above when the computer program is executed.
In a fourth aspect, the present application also provides a storage medium storing a computer program, which when executed by a processor causes the processor to implement a method for training a longitudinal federal learning model as described above.
The application discloses a training method, a training device, computer equipment and a storage medium of a longitudinal federal learning model. And secondly, the model loss value is compared with a preset threshold value through the encryption loss value in the decryption training process to realize an early-stopping mechanism, so that the model is prevented from being over-fitted, the calculation and bandwidth expenses of the participants in the model training are reduced, and finally the trained longitudinal federal learning model is obtained.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for the description of the embodiments will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a schematic flow chart of a training method for a longitudinal federal learning model provided by an embodiment of the present application;
FIG. 2 is a schematic block diagram of a training apparatus for a longitudinal federal learning model provided by an embodiment of the present application;
fig. 3 is a schematic block diagram of a computer device according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
The flow diagrams depicted in the figures are merely illustrative and not necessarily all of the elements and operations/steps are included or performed in the order described. For example, some operations/steps may be further divided, combined, or partially combined, so that the order of actual execution may be changed according to actual situations.
It is to be understood that the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should also be understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
Most of the existing federal learning assumes that the participants are semi-honest, i.e., model training is only performed according to the protocol flow without any vandalism. But this assumption is too ideal for practical use.
The participants in federal learning may have the following actions:
1. Possibly lazy nodes, transmit encrypted gradient information obtained from others, from previous training rounds, and even false meaningless to other parties.
2. The wrong encrypted gradient information is transmitted to interfere with the training of the model, possibly for malicious nodes or nodes that are taken up by malicious attackers.
In this case, the participants need to prove that their own training process is accurate, legal and consistent, i.e. possess the need for integrity for security auditing; meanwhile, since the original purpose of federal learning is to protect the privacy of the data used by the participants for training, the data and gradient information of the participants cannot be exposed in the verification process. Zero Knowledge Proof (ZKP) is therefore most suitable as a privacy protection technique that does not need to rely on any hardware. In this technique, the prover can trust that a certain assertion is correct without providing any useful information to the verifier, and zero knowledge proof can construct an integrity proof while also protecting the input data privacy.
On the other hand, the federal learning security audit method also needs to calculate a loss function to realize an early-stop mechanism, so that the training is prevented from stopping after only running the appointed iteration times, and the following problems are caused:
1. Overfitting of the model overfits the training data and the effect on the test data becomes lower.
2. Model training is lengthy, introducing additional computational and bandwidth overhead that each participant needs to bear for federal learning.
Therefore, the application provides a training method of a longitudinal federal learning model, which realizes a federal learning security audit method combined with an early-stop mechanism so as to ensure that no malicious nodes participate to cause model training failure.
The embodiment of the application provides a training method and device of a longitudinal federal learning model, computer equipment and a storage medium. The training method of the longitudinal federal learning model can be applied to a server, malicious nodes in the training process of the longitudinal federal learning model are identified through zero knowledge proof on a source data set, encryption loss initialization parameters and encryption loss values of the longitudinal federal learning model, the training effect of the longitudinal federal learning model is ensured, and the model loss values are decrypted and then compared with a preset threshold value, so that early shutdown is realized, and the overfitting of the model is avoided. The server may be an independent server or a server cluster.
Some embodiments of the present application are described in detail below with reference to the accompanying drawings. The following embodiments and features of the embodiments may be combined with each other without conflict.
Referring to fig. 1, fig. 1 is a schematic flowchart of a training method of a longitudinal federal learning model according to an embodiment of the present application. The training method of the longitudinal federal learning model can be applied to a server, is used for identifying malicious participants in the training process of the longitudinal federal learning model through zero knowledge demonstration, avoids the malicious participants from influencing the training effect of the longitudinal federal learning model, realizes an early-stop mechanism by comparing the decrypted loss value of the model with a preset threshold value, avoids the overfitting of the model, and finally obtains the trained longitudinal federal learning model.
As shown in fig. 1, the training method of the longitudinal federal learning model provided by the embodiment of the present application is applied to a first device participating in training of the longitudinal federal learning model, and the method specifically includes steps S101 to S105.
S101, generating an encryption public key and a decryption private key; the encryption public key is used for being sent to each second device participating in model training to encrypt data and parameters of the model, and the decryption private key is used for decrypting encryption loss values of the model.
First, assume that two data owners enterprises a and B want to jointly train a machine learning model, and their business systems respectively own relevant data of their users. In addition, enterprise a also has tag data that the model needs to predict. For data privacy and security, a and B cannot directly exchange data, at which point the federal learning system can be used to build a model. In order to ensure the confidentiality of data in the training process, integration and audit are also needed by means of a third-party central node C, and of course, federal learning is also suitable for the situation that a plurality of data owners participate in modeling.
It should be noted that the first device represents a central node C that integrates and audits training, and the second devices represent two or more data owners (e.g., a and B) that participate in training.
In the embodiment of the application, a training method of the longitudinal federation learning model is described by taking a logistic regression model of longitudinal federation learning in a three-party scene as an example. The training participants of the logistic regression model include an initiator, a receiver, and a central node. The initiator and the receiver promise own data and use own data to initialize loss, federal training and calculate loss. The center node is responsible for verifying data and calculation processes of the initiator and the receiver, integrating the encrypted gradient and the total loss of the calculation model of the two splicing parties, and judging whether to end the current training process according to the total loss. The data owned by the initiator and the recipient contain the same client but only the initiator has an annotation of the data.
Illustratively, the first device is now a central node and the second devices represent the initiator and the recipient of the logistic regression model.
First, a model-trained central node first generates a pair of encrypted public and decrypted private keys based on an encryption algorithm. The encryption public key is used for being sent to each second device participating in training of the longitudinal federal learning model, and the participants encrypt data, parameters and calculation tasks in the model training process based on the encryption public key. And the decryption private key is used for decrypting the encryption loss value in the model training process so as to obtain the loss value of the model training.
In some embodiments, the central node generates homomorphic encryption public key pk c and decryption key Sk c based on a homomorphic encryption algorithm of federal learning, selects a mask m and encrypts it with pk c to get [ m ], and sends the encrypted mask [ m ] and encryption public key pk c to participants of the vertical federal learning model training for encrypting the model training data and parameters.
Illustratively, we denote homomorphic encryption by [ ], e.g., m is plaintext, [ m ] is homomorphic encrypted ciphertext, and multiplying homomorphic encryption is: [ m 1+m2]＝[m1]+[m2 ], [ cm ] = c [ m ], where c is a positive integer.
S102, obtaining the promised data of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the promised data, wherein the promised data is obtained by carrying out data promise on a source data set by each second device.
In the embodiment provided by the application, a succinct non-interactive computationally reliable zero knowledge proof (zk-SNARK) is adopted as an implementation mode of the zero knowledge proof. The zero knowledge proof method allows the user to provide the proof in a non-interactive manner and can be verified in a very short time. zk-SNARK generally involves three processes:
(1) Generating a zero knowledge proof key pair according to the computing task: pk, vk, this process is run only once during system initialization, the formula is characterized as:
KeyGen(1λ,C)→(pk,vk)
Wherein lambda is a positive integer security parameter, C is a calculation task which is expressed by a circuit and needs to be proved, vk is a proving key used for generating proving, and vk is a verification key used for verifying proving of others.
(2) The participants of the model training generate a proof pi based on the proof key, and the formula is characterized as follows:
prove(pk,x,a)→π
Wherein pi is a certification generated by the participant and used for verification by the central node; a is a public input that does not require privacy protection; x is input data requiring privacy protection, pk is a certification key used to generate a certification.
(3) The central node verifies the certificates generated by the participants of the model training, and the process passes through the calculation of whether the verification process is correct or not. The formula is characterized as:
prove(vk,a,π)→b
wherein b is a Boolean value variable, and represents whether the verification result is correct or not; vk is the verification key used to verify the proof, a is the public input, pi is the proof generated by the participants of the model training;
Specifically, a corresponding zero knowledge proof circuit is generated according to a calculation task in a model training process requiring verification, a proof key and a verification key for zero knowledge proof of the calculation task are generated according to the zero knowledge proof circuit, and the calculation task comprises: data commitment, encryption loss initialization, encryption loss calculation.
First, the promised data input by the initiator and the receiver are verified to determine whether false data exists, and the source data set corresponding to the promised data comprises training set and test set data owned by the initiator and the receiver. Specifically, the central node first generates a zero knowledge proof gate based on the data commitment calculation process.
In some embodiments, committing to the data of the model training process based on a petersen commitment (Pedersen commitment) that can ensure that the commitment party does not modify the message while hiding the message itself from others; in the opening stage, the commitment party can disclose the message, and the receiving end verifies whether the message is consistent with the commitment stage or not, thereby meeting the requirements of concealment and binding.
Illustratively, C 1` is a zero knowledge proof circuit generated from the data commitments of the initiator and the receiver, and the described calculation process is:
wherein X is data for a test set, r is a random number, and COMM is a commitment function.
The central node generates a certification key and a verification key (pk 1,vk1) for zero knowledge proof of the initiator and receiver data commitments through keyGen (1 λ,C1). The central node then sends the proof key pk 1 for generating the proof to the initiator and the recipient.
Illustratively, the procedure for zero knowledge proof of data commitment of an initiator in the training procedure is specifically as follows: first, the initiator generates promised data by generating promised data from all input data X A Generating a promised label/>, by using all labels y A The formula is characterized as:
Next, the initiator sets x= (X A,rA), a= (), and generates proof pi A,x for verifying the promised data by Prove (pk 1, X, a); set x= (y A) and generate proof pi A,y for validating the promise tag by Prove (pk 1, x, a). Wherein a is a public input, i.e. an input that does not require privacy protection; x is input data requiring privacy protection.
The initiator commits the committed dataPromise tag after promise/>The proof pi A,x for validating the commitment data, the proof pi A,y for validating the commitment label are sent to the central node for zero knowledge proof. The central node sets a= (), verifies whether the promise data and the promise label of the initiator are correct through the Verify (vk 1,a,πA,x) and the Verify (vk 1,a,πA,y) respectively, and if the result is false, the result indicates that false data exists in the source data set corresponding to the promise data or the promise label, and the initiator is a malicious node.
In some embodiments, if it is determined that spurious data is present, the second device providing the spurious data is removed from the participants in the model training, and the first device resumes training of the model with other goodwill nodes.
It should be noted that, the process of zero knowledge proof of data commitment of the receiver is similar to that of the initiator, except that the commitment data at this time is the commitment data set after the receiver passes the data commitment.
When the commitment data set passes the zero knowledge proof, the data set provided by the initiator and the receiver of model training is indicated to have no false data, and then the source data set corresponding to the commitment data can be used for model training and encryption loss initialization calculation to determine encryption loss initialization parameters.
S103, obtaining encryption loss initialization parameters of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss initialization parameters, wherein the encryption loss initialization parameters are obtained by each second device through encryption loss initialization calculation based on the source data set.
Specifically, a corresponding zero-knowledge proving circuit is generated according to a calculation process of encryption loss initialization, and then a proving secret key and a verifying secret key for zero-knowledge proving are generated according to the zero-knowledge proving circuit, wherein the proving secret key is used for sending each second device to generate corresponding proving, and the verifying secret key is used for verifying the proving by the first device.
Illustratively, the steps for performing zero knowledge proof on encryption loss initialization parameters of a vertical federal learned logistic regression model in a three-party scene are as follows:
Because the calculation processes of the encryption loss initialization of all the participants in the model training are different, corresponding zero knowledge proof circuits are required to be respectively generated for the encryption loss initialization calculation of the initiator and the encryption loss initialization calculation of the receiver.
Illustratively, C 2 is a zero knowledge proof circuit generated for initializing calculations based on encryption loss of an initiator, expressed as:
[o]＝[m]·yAH
h[p]＝[o]TXAH
Wherein [ m ] is a homomorphic mask for encryption, r A is a random number selected by an initiator, X AH is input data of a test set which needs to be kept secret by the initiator, Y AH is a data tag corresponding to X AH, and h is the size of the test set.
The central node generates a certification key and a verification key (pk 2,vk2) for zero knowledge certification of the encryption loss initialization parameters of the initiator via keyGen (1 λ,C2).
Illustratively, C 3 is a zero knowledge proof circuit generated according to the encryption loss initialization calculation of the receiving party, and the stated calculation process is:
h[q]＝[o]TXBH
[μ]＝concatenate([p],[q])
Wherein X BH is input data of a test set which needs to be kept secret by a receiver, and r B is a random number selected by the receiver; concatate is a connection function, i.e. all inputs are connected in series.
The central node generates a proof key and a verification key (pk 3,vk3) for zero knowledge proof of the encryption loss initialization parameters of the receiver via keyGen (1 λ,C3).
And then, the center node sends the proving keys pk 2 and pk 3 for generating the proving to the initiator and the receiver, the initiator and the receiver generate the proving according to the proving key and the encryption loss initialization parameter respectively, and the center node verifies the proving according to the verification key.
First, the central node divides the data sets of the initiator and the receiver into training sets and test sets according to the prior promised data, wherein the test set of the initiatorThe test set of the receiver is/>The central node will/>And/>Respectively to the initiator and the recipient. The initiator calculates the variables needed for the encryption loss calculation, and the formula is characterized as follows:
[o]＝[m]·yAH
Initiator sets x= (X AH,yAH), And generating pi 2 by Prove (pk 2, x, a) and sending the encryption parameters [ o ], [ p ] and proof pi 2 to the central node. Center node set/>And verifying whether the encryption loss initialization process of the initiator is correct or not through the Verify (vk 2,a,π2), if the result is false, the encryption loss initialization parameter of the initiator is a false parameter, and the initiator is a malicious node. If the result is true, the encryption loss initialization parameters of the initiator pass zero knowledge proof, and the center node sends the encryption loss initialization parameters [ o ], [ p ] obtained by the initiator to the receiver for further training of the model. The variables [ q ], [ mu ] needed by the calculation of the encryption loss are calculated by the receiver, and the formula is characterized by:
[μ]＝concatenate([p],[q])
The receiver sets x= (X BH), Pi 3 is generated by Prove (pk 3, x, a). And sending the encryption loss initialization parameters [ q ], [ mu ], pi 3 obtained by the receiver to the central node for zero knowledge proof. Center node set/>And verifying whether the encryption loss initialization process of the receiver is correct or not through the Verify (vk 3,a,π3), if the result is false, the encryption loss initialization parameter of the receiver is a false parameter, and the receiver is a malicious node. If the result is true, the loss of the parameter calculation model is initialized according to the encryption loss.
In some embodiments, if it is determined that a malicious node is present, the first device notifies the second devices and ceases training of the model.
When the encryption loss initialization parameters of the initiator and the receiver are proved by zero knowledge, the encryption loss initialization calculation process is free of malicious nodes, and the obtained encryption loss initialization parameters are not false parameters and can be used for the next encryption loss calculation.
S104, obtaining an encryption loss value of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss value, wherein the encryption loss value is calculated by each second device based on the encryption loss initialization parameter and the encryption loss calculation weight.
Specifically, a corresponding zero knowledge proving circuit is generated according to encryption loss calculation, and then a proving secret key and a verifying secret key for zero knowledge proving are generated according to the zero knowledge proving circuit, wherein the proving secret key is used for sending each second device to generate corresponding proving, and the verifying secret key is used for verifying the proving by the first device.
Illustratively, the steps for performing zero knowledge proof on the encryption loss value of the vertical federal learning logistic regression model in the three-party scene are as follows:
Since the calculation process of the encryption loss calculation of each participant in model training is different, corresponding zero knowledge proof circuits are required to be respectively generated for the encryption loss calculation of the initiator and the encryption loss calculation of the receiver.
Illustratively, C 4 is a zero knowledge proof circuit generated from encryption loss calculation of the initiator, and the stated calculation process is:
[u]＝[m]·XAHθA
8h[u']＝(XAHθA·XAHθA)T[m]
The central node generates a proof key and a verification key (pk 4,vk4) for zero knowledge proof of a first cryptographic loss value of the initiator, wherein the first cryptographic loss value is an intermediate result of the cryptographic loss value of the model, by keyGen (1 λ,C4).
C 5 is a zero knowledge proof circuit generated according to encryption loss calculation of a receiver, and the expressed calculation process is as follows:
8h[l(θ)]＝8h[u']+(XBHθB·XBHθB)T[m]+2XBHθB[u]-2θT[μ]
The central node generates a proof key and a verification key (pk 5,vk5) for zero knowledge proof of a second cryptographic loss value of the recipient, wherein the second cryptographic loss value is a model final cryptographic loss value, by keyGen (1 λ,C5). The central node then sends the attestation keys pk 4 and pk 5 for generating attestation to the initiator and the recipient, who generate attestation based on the attestation keys and the respective encryption loss values, and the central node verifies based on the corresponding verification keys.
After generating the proof key and the verification key for zero knowledge proof of the cryptographic loss value, zero knowledge proof of the cryptographic loss value of the model in each loss calculation may begin.
First, the central node sends the weight θ updated for each training to the initiator and the receiver.
The initiator splits θ A from θ and calculates [ u ], [ u' ], the formula is characterized as:
[u]＝[m]·XAHθA
initiator sets x= (X AH), And generating proof pi 4 by Prove (pk 4, x, a) and transmitting [ u ], [ u' ], pi 4 to the central node for verification. Center node set/>And verifying whether the first encryption loss value of the initiator is correct or not through Verify (vk 4,a,π4), and if the result is false, indicating that the party for calculating the encryption loss is a malicious node.
If the result is true, the encryption loss calculation of the initiator passes the zero knowledge proof, at the moment, the center node sends the first encryption loss values [ u ], [ u' ] calculated by the initiator to the receiver, and the receiver calculates the second encryption loss value of the model on the basis. Specifically, the receiver splits θ B from θ and calculates the loss [ l (θ) ], where the formula is characterized as:
The receiver sets x= (X BH), And generates proof pi 5 by Prove (pk 5, x, a), and the receiver transmits the model-trained encryption loss value [ l (θ) ] and proof pi 5 for verifying the encryption loss value to the center node. If the result is true, the encryption loss value calculated by the receiving party passes the zero knowledge proof. If the result is false, the malicious node exists.
In some embodiments, upon determining that the encryption loss calculation has a malicious node, notifying the second devices and stopping training of the model.
When the encryption loss value of each second device passes the zero knowledge proof of the first device, it is indicated that no malicious node exists in the encryption loss calculation process, and the encryption loss value can be sent to a central node for decryption to obtain the loss value of the model.
S105, decrypting the encrypted loss value based on the decryption private key to obtain a loss value in the model training process, and obtaining a trained longitudinal federal learning model when the loss value is smaller than a preset threshold value.
When the encryption loss calculation of each second device passes the zero knowledge proof, the first device can decrypt the encryption loss value based on the decryption private key to obtain a loss value of the model, and compare the loss value with a preset threshold value to realize an early-stop mechanism.
Early shutdown is a widely used method, which in many cases is better than regularization. The basic meaning is that the performance of the model on the verification set is calculated in training, and when the performance of the model on the verification set starts to decline, the training is stopped, so that the problem of overfitting caused by continuous training can be avoided. Specifically, when the loss value of the model training is lower than the preset threshold value, notifying the participants of the model training to end the training.
Illustratively, the central node decrypts [ l (θ) ] to obtain a loss value l (θ) of the longitudinal federal learning model, and when l (θ) is smaller than a preset loss threshold l T, the central node notifies the initiator and the receiver to end training of the model; and obtaining a trained logistic regression model of longitudinal federal learning. And when the l (theta) does not reach the condition of the preset threshold value, updating the weight calculated by the encryption loss, and transmitting the updated weight to the initiator and the receiver to continue the iterative training of the model. It should be noted that the preset threshold is set according to a specific scenario, which is not limited by the present application.
According to the training method of the longitudinal federal learning model, provided by the embodiment of the application, by performing zero-knowledge proof on the data, encryption loss initialization and encryption loss calculation of the longitudinal federal learning model, malicious participants in the training process of the longitudinal federal learning model are identified, and the influence of the malicious participants on the training effect of the longitudinal federal learning model is avoided. Secondly, the encryption loss value in the decryption training process is used for obtaining a loss value of model training, the loss value is compared with a preset threshold value to realize an early-stopping mechanism, the model is prevented from being fitted excessively, the calculation and bandwidth expenditure of the participants in the model training is reduced, and finally the trained longitudinal federal learning model is obtained.
Referring to fig. 2, fig. 2 is a schematic block diagram of a training apparatus for a longitudinal federal learning model according to an embodiment of the present application, which is used to perform the foregoing training method for a longitudinal federal learning model. Wherein, the training device of the longitudinal federal learning model can be configured on a server.
As shown in fig. 2, the training apparatus 400 of the vertical federal learning model includes:
A key generation module 401, configured to generate an encrypted public key and a decrypted private key; the encryption public key is used for sending the encryption public key to each second device participating in model training to encrypt data and parameters of the model, and the decryption private key is used for decrypting the encryption loss value of the model to obtain the loss value;
The data proving module 402 obtains promised data of the model to perform zero knowledge proving, and obtains a zero knowledge proving result of the promised data, wherein the promised data is obtained by performing data promise on a source data set by each second device;
The encryption loss initialization proving module 403 is configured to obtain an encryption loss initialization parameter of the model for performing zero knowledge proving, and obtain a zero knowledge proving result of the encryption loss initialization parameter, where the encryption loss initialization parameter is obtained by calculating encryption loss initialization of each second device based on the source data set;
The encryption loss calculation proof module 404 is configured to obtain an encryption loss value of the model for zero knowledge proof, and obtain a zero knowledge proof result of the encryption loss value, where the encryption loss value is calculated by the second devices based on the encryption loss initialization parameter and the encryption loss calculation weight;
and the model output module 405 is configured to decrypt the encrypted loss value based on the decryption private key to obtain a loss value in the model training process, and obtain a trained longitudinal federal learning model when the loss value is less than a preset threshold.
It should be noted that, for convenience and brevity of description, a person skilled in the art may clearly understand that, for the specific working process of the apparatus and each module described above, reference may be made to the corresponding process in the foregoing embodiment of the training method of the longitudinal federal learning model, which is not described herein again.
The apparatus described above may be implemented in the form of a computer program which is executable on a computer device as shown in fig. 3.
Referring to fig. 3, fig. 3 is a schematic block diagram of a computer device according to an embodiment of the present application. The computer device may be a server.
With reference to FIG. 3, the computer device includes a processor, a memory, and a network interface connected by a system bus, where the memory may include storage media and internal memory.
The storage medium may store an operating system and a computer program. The computer program comprises program instructions that, when executed, cause the processor to perform any one of a number of training methods for a vertical federal learning model.
The processor is used to provide computing and control capabilities to support the operation of the entire computer device.
The internal memory provides an environment for the execution of a computer program in the storage medium that, when executed by the processor, causes the processor to perform any one of the training methods of the vertical federal learning model.
The network interface is used for network communication such as transmitting assigned tasks and the like. It will be appreciated by those skilled in the art that the structure shown in FIG. 3 is merely a block diagram of some of the structures associated with the present inventive arrangements and is not limiting of the computer device to which the present inventive arrangements may be applied, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.
It should be appreciated that the Processor may be a central processing unit (Central Processing Unit, CPU), it may also be other general purpose processors, digital signal processors (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), field-Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. Wherein the general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
Wherein in one embodiment the processor is configured to run a computer program stored in the memory to implement the steps of:
Generating an encryption public key and a decryption private key; the encryption public key is used for transmitting the encryption public key to each second device participating in model training to encrypt data and parameters of the model, and the decryption private key is used for decrypting encryption loss values of the model;
Zero knowledge proof is carried out on the promised data of the model, and a zero knowledge proof result of the promised data is obtained, wherein the promised data is obtained by carrying out data promise on a source data set by each second device;
obtaining encryption loss initialization parameters of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss initialization parameters, wherein the encryption loss initialization parameters are obtained by each second device through encryption loss initialization calculation based on the source data set;
Obtaining an encryption loss value of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss value, wherein the encryption loss value is calculated by each second device based on the encryption loss initialization parameter and the encryption loss calculation weight;
and decrypting the encryption loss value based on the decryption private key to obtain a loss value in the model training process, and obtaining a trained longitudinal federal learning model when the loss value is smaller than a preset threshold value.
In one embodiment, the processor, when implementing training of the longitudinal federal learning model, is to implement:
And when the loss value is greater than or equal to the preset threshold value, updating the encryption loss calculation weight according to the loss value to obtain updated encryption loss calculation weight, and transmitting the updated encryption loss calculation weight to each second device for iterative training of the model.
In one embodiment, the processor, when implementing training of the longitudinal federal learning model, is to implement:
Generating a zero knowledge proof gate based on a computing task, the computing task comprising: data promise, encryption loss initialization and encryption loss calculation; generating a proving secret key and a verifying secret key according to the zero knowledge proving gate circuit; the verification key is used for verifying the certificates; and verifying the certification based on the verification secret key to obtain a zero knowledge certification result of the calculation task.
In one embodiment, the processor, when implementing training of the longitudinal federal learning model, is to implement:
Determining whether false data exists according to a zero knowledge proof result of the promised data; when no false data exists, using the source data set for encryption loss initialization calculation of the model; when the false data is determined to exist, the second device providing the false data is determined to be a malicious node, the malicious node is removed, and training of the model is restarted.
In one embodiment, the processor, when implementing training of the longitudinal federal learning model, is to implement:
Determining whether a malicious node exists in the encryption loss initialization process according to a zero knowledge proof result of the encryption loss initialization parameter; when the encryption loss initialization process is determined to have no malicious node, the encryption loss initialization parameters are used for encryption loss calculation of the model; and when the malicious node exists in the encryption loss initialization process, notifying each second device, and stopping training of the model.
In one embodiment, the processor, when implementing training of the longitudinal federal learning model, is to implement:
Determining whether a malicious node exists in the encryption loss calculation process according to the zero knowledge proof result of the encryption loss value; when the encryption loss calculation process is determined to have no malicious node, the encryption loss value is used for decryption to obtain a loss value in the model training process; and when the malicious node exists in the encryption loss calculation process, notifying each second device, and stopping training of the model.
In one embodiment, the processor, when implementing the generation of the encrypted public key and the decrypted private key, is configured to implement: the encryption public key and the decryption private key are generated based on a homomorphic encryption algorithm.
The embodiment of the application also provides a storage medium, wherein the storage medium stores a computer program, the computer program comprises program instructions, and the processor executes the program instructions to realize the training method of any longitudinal federal learning model provided by the embodiment of the application.
The storage medium may be an internal storage unit of the computer device according to the foregoing embodiment, for example, a hard disk or a memory of the computer device. The storage medium may also be an external storage device of the computer device, such as a plug-in hard disk provided on the computer device, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD), or the like.
While the application has been described with reference to certain preferred embodiments, it will be understood by those skilled in the art that various changes and substitutions of equivalents may be made and equivalents will be apparent to those skilled in the art without departing from the scope of the application. Therefore, the protection scope of the application is subject to the protection scope of the claims.
Claims (10)
1. A method of training a longitudinal federal learning model, the method being applied to a first device in a longitudinal federal learning model training process, comprising:
Generating an encryption public key and a decryption private key; the encryption public key is used for transmitting the encryption public key to each second device participating in model training to encrypt data and parameters of the model, and the decryption private key is used for decrypting encryption loss values of the model;
Zero knowledge proof is carried out on the promised data of the model, and a zero knowledge proof result of the promised data is obtained, wherein the promised data is obtained by carrying out data promise on a source data set by each second device;
obtaining encryption loss initialization parameters of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss initialization parameters, wherein the encryption loss initialization parameters are obtained by each second device through encryption loss initialization calculation based on the source data set;
Obtaining an encryption loss value of the model to carry out zero knowledge proof, and obtaining a zero knowledge proof result of the encryption loss value, wherein the encryption loss value is calculated by each second device based on the encryption loss initialization parameter and the encryption loss calculation weight;
and decrypting the encryption loss value based on the decryption private key to obtain a loss value in the model training process, and obtaining a trained longitudinal federal learning model when the loss value is smaller than a preset threshold value.
2. The training method of claim 1, wherein the method further comprises:
And when the loss value is greater than or equal to the preset threshold value, updating the encryption loss calculation weight according to the loss value to obtain updated encryption loss calculation weight, and transmitting the updated encryption loss calculation weight to each second device for iterative training of the model.
3. Training method according to claim 1, characterized in that the method comprises:
generating a zero knowledge proof gate based on a computing task, the computing task comprising: data promise, encryption loss initialization and encryption loss calculation;
Generating a proving secret key and a verifying secret key according to the zero-knowledge proving gate circuit, wherein the proving secret key is used for generating proving by each second device, and the verifying secret key is used for verifying the proving;
and verifying the certification based on the verification secret key to obtain a zero knowledge certification result of the calculation task result.
4. The training method of claim 1, wherein the method further comprises:
Determining whether false data exists according to a zero knowledge proof result of the promised data;
when no false data exists, using the source data set for encryption loss initialization calculation of the model;
When the false data is determined to exist, the second device providing the false data is determined to be a malicious node, the malicious node is removed, and training of the model is restarted.
5. The training method of claim 1, wherein the method further comprises:
Determining whether a malicious node exists in the encryption loss initialization process according to a zero knowledge proof result of the encryption loss initialization parameter;
when the encryption loss initialization process is determined to have no malicious node, the encryption loss initialization parameters are used for encryption loss calculation of the model;
And when the malicious node exists in the encryption loss initialization process, notifying each second device, and stopping training of the model.
6. The training method of claim 1, wherein the method further comprises:
determining whether a malicious node exists in the encryption loss calculation process according to the zero knowledge proof result of the encryption loss value;
when the encryption loss calculation process is determined to have no malicious node, the encryption loss value is used for decryption to obtain a loss value in the model training process;
And when the malicious node exists in the encryption loss calculation process, notifying each second device, and stopping training of the model.
7. The training method of any of claims 1 to 6, wherein the generating an encrypted public key and a decrypted private key comprises:
The encryption public key and the decryption private key are generated based on a homomorphic encryption algorithm.
8. A training device for a longitudinal federal learning model, comprising:
The secret key generation module is used for generating an encryption public key and a decryption private key; the encryption public key is used for sending the encryption public key to each second device participating in model training to encrypt data and parameters of the model, and the decryption private key is used for decrypting the encryption loss value of the model to obtain the loss value;
The data proving module obtains the promised data of the model to carry out zero knowledge proving, and obtains a zero knowledge proving result of the promised data, wherein the promised data is obtained by carrying out data promise on a source data set by each second device;
The encryption loss initialization proving module is used for obtaining encryption loss initialization parameters of the model to carry out zero knowledge proving and obtaining zero knowledge proving results of the encryption loss initialization parameters, wherein the encryption loss initialization parameters are obtained by each second device through encryption loss initialization calculation based on the source data set;
The encryption loss calculation proving module is used for obtaining an encryption loss value of the model to carry out zero knowledge proving and obtaining a zero knowledge proving result of the encryption loss value, wherein the encryption loss value is calculated by each second device based on the encryption loss initialization parameter and the encryption loss calculation weight;
And the model output module is used for decrypting the encryption loss value based on the decryption private key to obtain a loss value in the model training process, and obtaining a trained longitudinal federal learning model when the loss value is smaller than a preset threshold value.
9. A computer device, the computer device comprising a memory and a processor;
the memory is used for storing a computer program;
the processor for executing the computer program and for implementing a training method of a longitudinal federal learning model according to any of claims 1 to 7 when the computer program is executed.
10. A storage medium storing a computer program which, when executed by a processor, causes the processor to implement a method of training a longitudinal federal learning model according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110688660.3A CN113420886B (en) | 2021-06-21 | 2021-06-21 | Training method, device, equipment and storage medium for longitudinal federal learning model |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110688660.3A CN113420886B (en) | 2021-06-21 | 2021-06-21 | Training method, device, equipment and storage medium for longitudinal federal learning model |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113420886A CN113420886A (en) | 2021-09-21 |
| CN113420886B true CN113420886B (en) | 2024-05-10 |
Family
ID=77789708
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110688660.3A Active CN113420886B (en) | 2021-06-21 | 2021-06-21 | Training method, device, equipment and storage medium for longitudinal federal learning model |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113420886B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113989036B (en) * | 2021-12-30 | 2022-03-18 | 百融至信(北京)征信有限公司 | Federal learning prediction method and system without exposure of model-entering variable |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111539731A (en) * | 2020-06-19 | 2020-08-14 | 支付宝(杭州)信息技术有限公司 | Block chain-based federal learning method and device and electronic equipment |
| CN111598254A (en) * | 2020-05-22 | 2020-08-28 | 深圳前海微众银行股份有限公司 | Federal learning modeling method, device and readable storage medium |
-
2021
- 2021-06-21 CN CN202110688660.3A patent/CN113420886B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111598254A (en) * | 2020-05-22 | 2020-08-28 | 深圳前海微众银行股份有限公司 | Federal learning modeling method, device and readable storage medium |
| CN111539731A (en) * | 2020-06-19 | 2020-08-14 | 支付宝(杭州)信息技术有限公司 | Block chain-based federal learning method and device and electronic equipment |
Non-Patent Citations (2)
| Title |
|---|
| 联邦学习安全与隐私保护研究综述;周俊 等;西华大学学报(自然科学版);20200710(第04期);全文 * |
| 联邦学习安全与隐私保护研究综述;周俊 等;西华大学学报（自然科学版）;第39卷(第4期);第9-17页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113420886A (en) | 2021-09-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Choudhuri et al. | Fairness in an unfair world: Fair multiparty computation from public bulletin boards | |
| CN110073633B (en) | Block chain data protection using homomorphic encryption | |
| EP3779717B1 (en) | Multiparty secure computing method, device, and electronic device | |
| CN110419053B (en) | System and method for information protection | |
| CN110971405B (en) | SM2 signing and decrypting method and system with cooperation of multiple parties | |
| Chaidos et al. | BeleniosRF: A non-interactive receipt-free electronic voting scheme | |
| US9860058B2 (en) | Secret computation system, arithmetic unit, secret computation method and program | |
| JP2019525591A (en) | Method and system implemented by blockchain | |
| US10846372B1 (en) | Systems and methods for trustless proof of possession and transmission of secured data | |
| CN110084068A (en) | Block catenary system and data processing method for block catenary system | |
| CN112380578A (en) | Edge computing framework based on block chain and trusted execution environment | |
| JP2007049708A (en) | System and method for updating keys used for public key cryptography | |
| Niu et al. | Toward verifiable and privacy preserving machine learning prediction | |
| US11575501B2 (en) | Preserving aggregation using homomorphic encryption and trusted execution environment, secure against malicious aggregator | |
| US10887104B1 (en) | Methods and systems for cryptographically secured decentralized testing | |
| CN112600675B (en) | Electronic voting method and device based on group signature, electronic equipment and storage medium | |
| CN109361508A (en) | Data transmission method, electronic equipment and computer readable storage medium | |
| US20230237437A1 (en) | Apparatuses and methods for determining and processing dormant user data in a job resume immutable sequential listing | |
| CN113420886B (en) | Training method, device, equipment and storage medium for longitudinal federal learning model | |
| CN116318901A (en) | Privacy and verifiable internet of things data aggregation method integrating blockchain | |
| de Hoogh et al. | Certificate validation in secure computation and its use in verifiable linear programming | |
| EP3364397B1 (en) | Secret authentication code adding device, secret authentification code adding method, and program | |
| US20220345312A1 (en) | Zero-knowledge contingent payments protocol for granting access to encrypted assets | |
| Jiang | Timed encryption with application to deniable key exchange | |
| Alnahawi et al. | Towards Next Generation Quantum-Safe eIDs and eMRTDs–A Survey |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |