WO2021092980A1 - Procédé, appareil et dispositif d'optimisation d'apprentissage fédéré longitudinal et support de stockage - Google Patents

Procédé, appareil et dispositif d'optimisation d'apprentissage fédéré longitudinal et support de stockage Download PDF

Info

Publication number
WO2021092980A1
WO2021092980A1 PCT/CN2019/119418 CN2019119418W WO2021092980A1 WO 2021092980 A1 WO2021092980 A1 WO 2021092980A1 CN 2019119418 W CN2019119418 W CN 2019119418W WO 2021092980 A1 WO2021092980 A1 WO 2021092980A1
Authority
WO
WIPO (PCT)
Prior art keywords
participant
value
encrypted
target
encrypted data
Prior art date
Application number
PCT/CN2019/119418
Other languages
English (en)
Chinese (zh)
Inventor
范涛
杨恺
陈天健
杨强
Original Assignee
深圳前海微众银行股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳前海微众银行股份有限公司 filed Critical 深圳前海微众银行股份有限公司
Publication of WO2021092980A1 publication Critical patent/WO2021092980A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/18—Complex mathematical operations for evaluating statistical data, e.g. average values, frequency distributions, probability functions, regression analysis

Definitions

  • This application relates to the technical field of financial technology (Fintech), in particular to vertical federated learning optimization methods, devices, equipment, and storage media.
  • the longitudinal linear regression method in federated learning is a stochastic gradient descent method based on one-step information. Take the vertical federated linear regression in which two parties participate as an example. For example, if party A and party B are set, party A is the host (host) party, with only part of the characteristics of the data, and party B is the guest (guest) party, which has part and A completeness. Different data characteristics have data tags at the same time.
  • Party B needs to request the inner product of the current model parameters and data from Party A to calculate the loss function value and gradient. This process involves Party A sending its encrypted calculation data to Party B, and Party B calculates the encrypted coefficients. , Through the coefficients AB, the two parties can calculate their respective gradient components, and send them to the third party C for decryption and processing, and then send them back to A and B as the descending direction, update the model parameters held by both parties, and iterate this step So A and B can get a trained model.
  • the existing schemes are iteratively optimized based on one-step information of the objective loss function, and its convergence speed is slow. This results in a large number of rounds of data interaction between ABCs, and communication takes a lot of time in cross-enterprise cooperation.
  • the main purpose of this application is to propose a vertical federated learning optimization method, device, equipment, and storage medium, which aims to solve the current long-term technical problem of vertical federated learning.
  • the longitudinal federated learning optimization method includes the following steps:
  • the secondary participant obtains the encrypted value set with linear regression value sent by the main participant, and calculates the secondary encrypted data according to the encrypted value set;
  • the loss function value and the secondary encrypted data are sent to the coordinator, where the coordinator is used to update the second derivative matrix in the coordinator according to the secondary encrypted data in response to the vertical federation model not converging , And calculate the target subgradient value according to the updated second derivative matrix;
  • this application also provides a longitudinal federated learning optimization method including the following steps:
  • the participant calculates based on the encrypted value set sent by the main participant, and the encrypted value set includes the main encrypted value and the new encrypted value;
  • the target secondary gradient value is sent to the secondary participant, and the secondary participant is used to update the local model parameters in the secondary participant based on the target secondary gradient value, and continue to perform the secondary participant acquisition
  • the present application also provides a longitudinal federated learning optimization device, the longitudinal federated learning optimization device includes:
  • the obtaining module is used for the secondary participant to obtain the encrypted value set with linear regression value sent by the main participant, and calculate the secondary encrypted data according to the encrypted value set;
  • the sending module is configured to send the secondary encrypted data to the coordinator, where the coordinator is used to update the second derivative matrix in the coordinator according to the secondary encrypted data in response to the vertical federation model not converging, And calculate the target sub-gradient value according to the updated second derivative matrix;
  • the first receiving module is configured to receive the target secondary gradient value sent by the coordinator based on the secondary encrypted data, update the local model parameters in the secondary participant based on the target secondary gradient value, and continue to execute the secondary participant The step of obtaining the encrypted value set with linear regression value sent by the main participant until the vertical federation model corresponding to the coordinator converges.
  • the longitudinal federated learning optimization device further includes:
  • the second receiving module is used to receive the primary encrypted data sent by the primary participant and the secondary encrypted data sent by the secondary participant, wherein the secondary encrypted data is calculated according to the intermediate result value in the secondary participant, and the The intermediate result value is calculated by the secondary participant based on the encrypted value set sent by the main participant, and the encrypted value set includes the main encrypted value and the new encrypted value;
  • the update module is configured to respond to the failure of the longitudinal logistic regression model to converge, update the second derivative matrix according to the main encrypted data and the auxiliary encrypted data, and calculate the target sub-gradient value according to the updated second derivative matrix;
  • the convergence module is configured to send the target secondary gradient value to the secondary participant, and the secondary participant is used to update the local model parameters in the secondary participant based on the target secondary gradient value and continue to execute all The step of obtaining the encrypted value set with linear regression value sent by the main participant by the secondary participant until the vertical federation model corresponding to the coordinator converges.
  • this application also provides a longitudinal federated learning optimization device
  • the longitudinal federated learning optimization device includes: a memory, a processor, and a computer stored on the memory and capable of running on the processor
  • a readable instruction when the computer readable instruction is executed by the processor, implements the steps of the vertical federated learning optimization method as described above.
  • the present application also provides a storage medium having computer-readable instructions stored on the storage medium, and when the computer-readable instructions are executed by a processor, the above-mentioned vertical federated learning optimization method is implemented. step.
  • FIG. 1 is a schematic diagram of a device structure of a hardware operating environment involved in a solution of an embodiment of the present application
  • FIG. 2 is a schematic flowchart of the first embodiment of the vertical federated learning optimization method according to this application;
  • FIG. 3 is a schematic flowchart of another embodiment of the vertical federated learning optimization method of this application.
  • Figure 4 is a schematic diagram of the device modules of the vertical federated learning optimization device of the application.
  • Figure 5 is a schematic diagram of the calculation and interaction process of the vertical federated learning optimization method of this application.
  • FIG. 1 is a schematic diagram of the device structure of the hardware operating environment involved in the solution of the embodiment of the present application.
  • the longitudinal federated learning optimization device in the embodiment of the present application may be a PC or a server device, on which a Java virtual machine runs.
  • the vertical federated learning optimization device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002.
  • the communication bus 1002 is used to implement connection and communication between these components.
  • the user interface 1003 may include a display screen (Display) and an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • the network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).
  • the memory 1005 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as a magnetic disk memory.
  • the memory 1005 may also be a storage device independent of the aforementioned processor 1001.
  • the structure of the device shown in FIG. 1 does not constitute a limitation on the device, and may include more or fewer components than those shown in the figure, or a combination of certain components, or different component arrangements.
  • the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and computer readable instructions.
  • the network interface 1004 is mainly used to connect to the back-end server and communicate with the back-end server; the user interface 1003 is mainly used to connect to the client (user side) and communicate with the client; and the processor 1001 can be used to call computer-readable instructions stored in the memory 1005, and perform operations in the following vertical federated learning optimization method.
  • FIG. 2 is a schematic flowchart of a first embodiment of a longitudinal federated learning optimization method according to this application. The method includes:
  • Step S10 the secondary participant obtains the encrypted value set with linear regression value sent by the main participant, and calculates the secondary encrypted data according to the encrypted value set;
  • Linear regression is a method based on a linear model to fit data features (independent variables) and data labels (dependent variables).
  • Vertical federated linear regression means that multiple participants want to combine data for linear regression modeling, but each holds a part of different data characteristics, and the data labels are often owned by only one party. Therefore, in this embodiment, the main participant has only part of the characteristics of the data, while the sub-participants have some data characteristics that are completely different from the main participant.
  • Vertical federated learning means that different parties have different feature data, which is equivalent to dividing each complete data into multiple parts vertically. Each party hopes to implement linear regression model training while protecting data privacy, so as to use the model The parameter predicts the value of the dependent variable on the new data.
  • [[ ⁇ ]] stands for homomorphic encryption operation.
  • a vertical federation scenario only one party holds data labels. Take two parties as an example. Party A holds data x A and maintains the corresponding model parameters w A , and Party B holds x B , y B and owns and maintains the corresponding model parameters. w B.
  • the loss function and gradient can be expressed as the operation of the homomorphic encrypted data of both parties, namely:
  • This solution uses second-order information to propose a fast-convergent technical solution, based on the second-order derivative matrix of the loss function (ie, the Hessian matrix)
  • the design idea of this scheme is based on the quasi-Newton method, using the second-order information to estimate an inverse Hessian matrix H.
  • the gradient g is not used but H g is used as the descending direction to speed up the convergence speed of the algorithm. Since the dimension of the inverse Hessian matrix H is much larger than the gradient, the core point of the design is how to reduce the data communication volume of all parties.
  • This scheme proposes to maintain the inverse Hessian matrix H at the C end, and in addition to calculating the gradient for each L step AB, a small batch of data is randomly selected, and the average value of the previous L step model is calculated Average with the last L-step model Difference Then calculate a vector containing the second-order information of the batch of data Sent to the C end, the dimension is the same as the gradient.
  • the C terminal uses the information of the first M vectors v to update the inverse Hessian matrix once. Therefore, in this embodiment, the main participant is regarded as Party A, the sub-participants are regarded as Party B, and the coordinator is regarded as Party C.
  • Party A calculates the value set of the corresponding data ID in S A uses homomorphic encryption technology for all Encrypt the value to get the encrypted data set Transmit it to Party B. Then, update And judge the relationship between the current iteration number k and L, if the current iteration number k is an integer multiple of L, and the iteration number k is greater than 2L; And calculate the current (t) and last (t-1) Difference between In addition, a small batch of data ID is randomly selected as S H , and the A side calculates the value on S H And homomorphically encrypted data Transmitted to the B side. If the current number of iterations k is an integer multiple of L, and the number of iterations k is not greater than 2L: only update the A side
  • Step S20 Send the secondary encrypted data to the coordinator, where the coordinator is used to update the second derivative matrix in the coordinator according to the secondary encrypted data in response to the vertical federation model not converging, and according to The updated second-order derivative matrix calculates the target sub-gradient value;
  • the B end transmits the encrypted loss value to the C end.
  • a and B respectively transmit [[g A ]], [[g B ]] to C. Then, determine the relationship between the current iteration number k and L.
  • Party C ie, the coordinator
  • update And judge the relationship between the number of iterations k and 2L if k is not greater than 2L, calculate the product of the pre-selected step size and the gradient And transmit them to A and B respectively (that is, the target secondary gradient value is obtained, and the target secondary gradient value is sent to the main participant, and the product corresponding to the secondary participant is sent to the secondary participant). If k is greater than 2L, merge the two gradients into a long vector g, calculate the step length, the product of H and g, and split them into corresponding parts A and B and transmit them to A and B respectively, namely:
  • k is an integer multiple of L
  • C has also received the encrypted data [[v A ]],[[v B ]], which can be decrypted and combined to get And stored in a v queue of length M.
  • the calculation method is as follows: initialize with the value at the end of the memory queue, that is, calculate H ⁇ p[m]I, where I is the identity matrix.
  • Step S40 Receive the target secondary gradient value sent by the coordinator based on the secondary encrypted data, update the local model parameters in the secondary participant based on the target secondary gradient value, and continue to execute the secondary participant acquiring the primary participant The step of sending encrypted value sets with linear regression values until the vertical federation model corresponding to the coordinator converges.
  • the secondary participant After the secondary participant receives the target secondary gradient value sent by the coordinator, it will update the secondary participant’s own local model parameters according to the target secondary gradient value. At the same time, after the primary participant receives the target primary gradient value sent by the coordinator, The model parameters of the main participant will also be updated based on the product. That is, the two parties AB use the received unencrypted vector to update their local model parameters, namely:
  • Party A calculates and transmits data locally to Party B, which means encrypting It is transmitted to Party B.
  • Party B performs local calculations based on the encrypted data transmitted by Party A to obtain the encryption loss function value and encryption value, and transmits [[d]][[h]] to Party A, and both parties AB also encrypt Calculate the respective gradient values and transmit them to the C party, that is, the two parties AB send [[g A ]],[[g B ]][[v A ]],[[v B ]] to C Party B also transmits [[loss]] to Party C.
  • Party C decrypts the received [[v A ]], [[v B ]], [[loss]] to obtain the decrypted g A , g B , loss ⁇ A , ⁇ B , and judge whether the algorithm has converged according to loss, if not, update H according to the received gradient value, calculate and transmit, that is, when k is not greater than 2L, calculate the pre-selection The product of a fixed step size and the gradient And transmit them to A and B respectively; when k is greater than 2L, merge the two gradients into a long vector g, calculate the product of step length, H and g, and split them into corresponding parts A and B respectively Transmitted to A and B, namely:
  • both parties AB update the local model parameters according to the unencrypted vector passed by party C, namely
  • the loss function value and the secondary encrypted data are calculated and sent to the coordinator, so that the coordinator can determine whether the vertical federation model has converged according to the loss function value. If it does not converge, the second derivative matrix is updated according to the secondary encrypted data, and the target secondary gradient value is calculated according to the updated second derivative matrix, and then the local model parameters of the secondary participant are updated by the target secondary gradient value, thereby avoiding
  • the prior art for longitudinal federated learning uses a first-order algorithm to make the convergence rate slower, and a large number of rounds of data interaction are required. This reduces the amount of communication for longitudinal federated learning and improves the training time of the longitudinal federated logistic regression model. The convergence rate.
  • a second embodiment of the vertical federated learning optimization method of the present application is proposed.
  • This embodiment is a refinement of the step S10 of the first embodiment of the present application.
  • the secondary participant obtains the encrypted value set with the linear regression value sent by the main participant, including: step a, detecting whether the vertical federation model satisfies the preset Judgment condition
  • the main participant when the main participant sends data to the sub-participants, it is also necessary to detect whether the vertical federation model satisfies the preset determination condition, for example, to determine whether the new iteration number of the vertical federation model meets the preset number condition (such as determining the new number of iterations). Whether the number of iterations is an integer multiple of the interval of iteration steps, and whether it is greater than two times greater than the preset number). And perform different operations according to different judgment results. And perform different operations according to different test results.
  • step b if it is satisfied, the secondary participant obtains the main encrypted value and the new encrypted value sent by the main participant, and uses the main encrypted value and the new encrypted value as a linear regression sent by the main participant The encrypted value collection of the value.
  • the main participant In the main participant, first obtain a small batch of data, and according to the formula mentioned in the above embodiment To calculate each logistic regression score, and encrypt these logistic regression scores using homomorphic encryption technology to obtain the main encryption value.
  • a small batch of data is obtained, and According to the formula To calculate each logistic regression score, and encrypt these logistic regression scores using homomorphic encryption technology to obtain a new encrypted value, and use the encrypted data and the new encrypted data together as an encrypted value set with a linear regression value.
  • the encrypted data and the new encrypted data are not the same and are sent to the secondary participant, that is, the secondary participant will obtain the main encrypted value and the new encrypted value sent by the main participant.
  • the main participant only sends the main encrypted value to the sub-participants, that is, at this time, the main encrypted value is a collection of encrypted values with linear regression values.
  • the secondary participant by determining that the vertical federation model satisfies the preset judgment condition, the secondary participant obtains the primary encrypted value and the secondary encrypted value sent by the primary participant, and uses them as a collection of encrypted numerical values with linear regression values. Thereby improving the training speed of the longitudinal linear regression model.
  • step of calculating the secondary encrypted data according to the encrypted value set includes: step c, determining whether the current iteration number corresponding to the secondary participant satisfies a preset number condition,
  • Step d if it is satisfied, calculate the intermediate result value according to the encrypted value set, and calculate the secondary encrypted data according to the intermediate result value.
  • end B is updated And calculate the current (t) and last (t-1) Difference between
  • the B side calculates the S H To calculate the intermediate result value It is transmitted to the A side, and at the same time, the secondary encrypted data in the secondary participant will be calculated based on the intermediate result value.
  • the intermediate result value is calculated according to the encrypted value set, and the secondary encrypted data is calculated by the intermediate result value, thereby ensuring that the secondary encrypted data is obtained.
  • the accuracy of the data is calculated according to the encrypted value set, and the secondary encrypted data is calculated by the intermediate result value, thereby ensuring that the secondary encrypted data is obtained.
  • the step of calculating the intermediate result value according to the encrypted value set, and calculating the secondary encrypted data through the intermediate result value includes: step e, obtaining the local model parameters in the secondary participant based on the encrypted value set The current average value, and obtain the historical average value of the preset step interval before the current average value;
  • the sub-participant After the sub-participant obtains the encrypted value set sent by the main participant, the current average value of the local model parameters in the sub-participant will also be obtained And it is also necessary to obtain the historical average value of the preset step interval before the current average value in the secondary participants.
  • Step f Calculate the difference between the current average value and the historical average value, calculate an intermediate result value according to the difference value, and calculate the secondary encrypted data according to the intermediate result value.
  • the intermediate result value is calculated based on the difference between the current average value and the historical average value among the driving co-participants, and the secondary encrypted data is calculated by the intermediate result value, thereby ensuring that the secondary encrypted data is obtained.
  • the accuracy of the data is calculated based on the difference between the current average value and the historical average value among the driving co-participants, and the secondary encrypted data is calculated by the intermediate result value, thereby ensuring that the secondary encrypted data is obtained. The accuracy of the data.
  • a third embodiment of the vertical federated learning optimization method of this application is proposed. This embodiment is a refinement of the step S30 of the first embodiment of the present application.
  • the step of receiving the target secondary gradient value sent by the coordinator based on the secondary encrypted data includes: step g, receiving the coordinator based on the The target sub-gradient value sent by the sub-encrypted data, wherein the target sub-gradient value is obtained by the second derivative matrix updated by the coordinator according to the target data, and the target data is in response to the failure of the longitudinal logistic regression model to converge, and It is obtained by decrypting and combining the primary encrypted data and the secondary encrypted data sent by the secondary participant when the preset judgment condition is satisfied.
  • the secondary participant When the secondary participant receives the target secondary gradient value fed back by the coordinator, it can update its own local model parameters according to the target secondary gradient value.
  • the target secondary gradient value is when the coordinator determines that the longitudinal logistic regression model does not converge and satisfies
  • the second derivative matrix is updated according to the target data, and calculated according to the updated second derivative matrix, where the target data does not converge in the longitudinal logistic regression model and meets the preset judgment
  • the conditions are met, it is obtained by decrypting and combining the main encrypted data sent by the main participant and the secondary encrypted data sent by the sub-participants.
  • judging whether the longitudinal logistic regression model satisfies the preset judgment condition for example, judging whether the new iteration number of the longitudinal logistic regression model meets the preset number condition (such as determining whether the new iteration number is an integer multiple of the iteration step interval, and whether it is greater than Twice is greater than the preset number of times). And perform different operations according to different judgment results.
  • the obtained target sub-gradient is guaranteed The accuracy of the value.
  • step of receiving the target secondary gradient value fed back by the coordinator includes:
  • Step h Receive the target subgradient value fed back by the coordinator, where the target subgradient value is obtained by splitting the first target product by the coordinator, and the first target product is based on the response to the The second-order derivative matrix updated by the longitudinal logistic regression model to satisfy the preset judgment condition, the long vector of the combination of the main gradient value sent by the main participant and the auxiliary gradient value sent by the auxiliary participant, and the preset step The product between the lengths.
  • the secondary participant When the secondary participant receives the target secondary gradient value fed back by the coordinator, it can update its own local model parameters according to the target secondary gradient value, where the target secondary gradient value is obtained by splitting the first target product by the coordinator , And the first target product is when the longitudinal logistic regression model does not converge and meets the preset judgment conditions, according to the updated second-order derivative matrix, the main gradient value sent by the main participant and the sub-gradient value sent by the sub-participant combined long The product of the vector and the preset step size.
  • the target secondary gradient value is obtained by the coordinator splitting the first target product, and the first target product is the product of the long vector, the preset step size, and the updated second derivative matrix, thus The accuracy of the obtained target subgradient value is guaranteed.
  • step of receiving the target secondary gradient value fed back by the coordinator includes:
  • Step k receiving the target subgradient value fed back by the coordinator, wherein the target subgradient value is a second target product, and the second target product is that the coordinator does not converge in response to the longitudinal logistic regression model, and The product of the calculated secondary gradient value sent by the secondary participant and the preset step size is not met.
  • the sub-participant When the sub-participant receives the target sub-gradient value fed back by the coordinator, he can update his own local model parameters according to the target sub-gradient value.
  • the target sub-gradient value is the second product, and the second product is the longitudinal logic of the coordinator.
  • the regression model does not converge and the preset judgment conditions are not met, the sub-gradient value sent by the sub-participants and the preset step length are calculated to obtain its product.
  • This product is the second product, which is the target sub-product.
  • the gradient value When the sub-participant receives the target sub-gradient value fed back by the coordinator, he can update his own local model parameters according to the target sub-gradient value.
  • the target sub-gradient value is the second product, and the second product is the longitudinal logic of the coordinator.
  • the product of the preset step size and the secondary gradient value is calculated, thereby ensuring the obtained target The accuracy of the main gradient value.
  • Fig. 3 is a schematic flow chart of another embodiment of the vertical federated learning optimization method of this application, including: step S100, receiving the primary encrypted data sent by the primary participant and the secondary encrypted data sent by the secondary participant, where The secondary encrypted data is calculated according to the intermediate result value in the secondary participant, the intermediate result value is calculated by the secondary participant according to the encrypted value set sent by the main participant, the encrypted value set Including the main encrypted value and the new encrypted value;
  • the coordinator when it is determined that the longitudinal logistic regression model has not converged according to the loss function value sent by the secondary participant and meets the preset judgment condition, for example, it is judged whether the new iteration number of the longitudinal logistic regression model meets the preset number condition (such as determining the new number of iterations). Whether the number of iterations is an integer multiple of the iteration step interval, and whether it is greater than twice the preset number), if the preset number condition is met, it is determined that the longitudinal logistic regression model satisfies the preset judgment condition, when the main participant sends After the primary encrypted data and the secondary encrypted data sent by the secondary participant, the second derivative matrix is updated according to the primary encrypted data and the secondary encrypted data.
  • the preset number condition such as determining the new number of iterations
  • the secondary encrypted data is calculated by the secondary participant based on the intermediate result value of the target value set feedback sent by the main participant, that is, the primary participant sends the encrypted value set to the secondary participant, and the secondary participant calculates it based on the encrypted value set
  • the intermediate result value and the loss function value, and the loss function value is sent to the coordinator, the secondary encrypted data is calculated based on the intermediate result value, and the secondary encrypted data is sent to the coordinator.
  • the encrypted value set may include the main encrypted value corresponding to the data and the new encrypted value corresponding to the new data, that is, whether the current iteration number corresponding to the main participant satisfies a preset condition (such as whether the current iteration number has passed the preset number), If it is not satisfied, the main encrypted value can be used as the target value set, and if it is satisfied, the main encrypted value and the new encrypted value can be used as the encrypted value set.
  • a preset condition such as whether the current iteration number has passed the preset number
  • Step S200 in response to the failure of the longitudinal logistic regression model to converge, update a second derivative matrix according to the main encrypted data and the auxiliary encrypted data, and calculate a target subgradient value according to the updated second derivative matrix;
  • the coordinator When the coordinator detects that the longitudinal logistic regression model has not converged, it can update the second derivative matrix based on the main encrypted data sent by the main participant and the auxiliary encrypted data sent by the sub-participants, that is, the main encrypted data and the sub-encrypted data Decrypt and merge them and store them in a queue with a preset length to obtain the target queue, and update the second derivative matrix H according to the target queue.
  • the method of calculating H is to initialize with the value at the end of the memory queue, that is, to calculate H ⁇ p[m]I, where I is the identity matrix.
  • Judgment that is, to determine whether the longitudinal logistic regression model converges, and if it converges, it sends an iterative stop signal to party A and B, and stops the training of the longitudinal logistic regression model. If it does not converge, execute again Until the longitudinal logistic regression model converges.
  • k is greater than 2L
  • the two gradients are merged into a long vector g, the product of the step length, H and g is calculated, and split into the corresponding A and B parts (that is, the target main gradient corresponding to the A side Value and the target sub-gradient value corresponding to party B) are transmitted to A and B respectively, namely:
  • Step S300 Send the target secondary gradient value to the secondary participant, and the secondary participant is used to update the local model parameters in the secondary participant based on the target secondary gradient value, and continue to execute the secondary participant.
  • the participant obtains the encrypted value set with linear regression value sent by the main participant until the vertical federation model corresponding to the coordinator converges.
  • the coordinator After the coordinator calculates the target sub-gradient value, it will send the target sub-gradient value to the sub-participants.
  • the sub-participants will update the local model parameters in the sub-participants according to the target sub-gradient value and continue to execute the sub-participants.
  • the participant obtains the encrypted value set with linear regression value sent by the main participant until the longitudinal logistic regression model corresponding to the coordinator converges, and sends an iteration stop signal to the main participant and the secondary participant.
  • the main participant also receives the target main gradient value corresponding to the main participant fed back by the coordinator to update the local model parameters in the main participant.
  • the coordinator updates the second-order derivative matrix according to the main encrypted data and the auxiliary encrypted data, and calculates the target sub-gradient value according to the updated second-order derivative matrix, and sends the target sub-gradient value to the sub-participants to update
  • the local model parameters in the sub-participants thus avoiding the phenomenon that the prior art adopts the first-order algorithm for longitudinal federated learning, which makes the convergence speed slow and requires a large number of rounds of data interaction, and reduces the communication for longitudinal federated learning. the amount.
  • step of updating a second derivative matrix according to the primary encrypted data and the secondary encrypted data includes:
  • Step m judging whether the longitudinal logistic regression model satisfies the preset judgment condition
  • the coordinator After the coordinator receives the main gradient value sent by the main participant and the secondary gradient value and loss value sent by the deputy coordinator, and determines that the longitudinal logistic regression model does not converge, it needs to determine whether the longitudinal logistic regression model meets the preset determination conditions, for example Determine whether the new iteration number of the longitudinal logistic regression model meets the preset number condition (such as determining whether the new iteration number is an integer multiple of the iteration step interval, and whether it is more than twice greater than the preset number). And perform different operations according to different judgment results.
  • the preset determination conditions for example Determine whether the new iteration number of the longitudinal logistic regression model meets the preset number condition (such as determining whether the new iteration number is an integer multiple of the iteration step interval, and whether it is more than twice greater than the preset number).
  • Step n if it is satisfied, decrypt and merge the primary encrypted data and the secondary encrypted data to obtain target data;
  • the coordinator will decrypt and merge the target data after receiving the main encrypted data sent by the main participant and the secondary encrypted data sent by the sub-participants. Encrypted data [[v A ]],[[v B ]] are decrypted to obtain the target data
  • Step p Store the target data in a queue with a preset length to obtain the target queue, and update the second derivative matrix through the target queue.
  • the coordinator stores the target data in a v queue with a length of M (ie, a preset length). At the same time, calculate the current (t) and the last (t-1) Difference between Store it in the s queue of length M. If the current memory has reached the maximum storage length M, delete the first one in the queue and put the latest v and s at the end of the queue. Use m (m not greater than M) v and s in the current memory to calculate H (second derivative matrix). The calculation method is as follows:
  • the target data is obtained by decrypting and combining the primary encrypted data and the secondary encrypted data, and then the second derivative matrix is updated according to the target data, thereby ensuring the effectiveness of the update of the second derivative matrix.
  • the method includes:
  • Step x if not satisfied, the coordinator obtains the first product between the secondary gradient value sent by the secondary participant and the preset step size, and sends the first product as the target secondary gradient value to The associate participant.
  • the coordinator calculates the first product of the pre-selected preset step size and the sub-gradient value, and the preset step size and the main gradient corresponding to the main participant
  • the third product of the value, and the first product is sent to the secondary participant as the target secondary gradient value to update the local model parameters in the secondary participant
  • the third product is sent to the main participant to update the local model in the primary participant Parameters, and then re-train the model according to the updated model parameters to obtain the new loss function value, and send it to the coordinator through the deputy participant.
  • the first product between the sub-gradient value and the preset step size is calculated, and the first product is used as the target sub-gradient value, thereby The accuracy of the obtained target main gradient value is guaranteed.
  • the longitudinal federated learning optimization device includes: an acquisition module for the secondary participant to acquire the encrypted value set with linear regression value sent by the main participant , And calculate the secondary encrypted data according to the encrypted value set; the sending module is used to send the secondary encrypted data to the coordinator, wherein the coordinator is used to respond to the vertical federation model not converging, according to the secondary encryption
  • the data updates the second-order derivative matrix in the coordinator, and calculates the target sub-gradient value according to the updated second-order derivative matrix;
  • the first receiving module is used to receive the target sub-gradient sent by the coordinator based on the sub-encrypted data Gradient value, update the local model parameters in the secondary participant based on the target secondary gradient value, and continue to perform the step of obtaining the encrypted value set with linear regression value sent by the primary participant by the secondary participant until the coordinator The corresponding vertical federation model converges.
  • the acquisition module is further configured to: detect whether the vertical federation model meets a preset judgment condition; if so, the secondary participant acquires the primary encrypted value and the new encrypted value sent by the primary participant, and compares all The main encrypted value and the new encrypted value are used as an encrypted value set with a linear regression value sent by the main participant.
  • the acquisition module is further configured to determine whether the current iteration number corresponding to the secondary participant meets a preset number condition, and if so, calculate the intermediate result value according to the encrypted value set, and pass the The intermediate result value calculates the secondary encrypted data.
  • the obtaining module is further configured to: obtain the current average value of the local model parameters in the secondary participants based on the encrypted value set, and obtain the historical average of the preset step interval before the current average value Value; Calculate the difference between the current average and the historical average, and calculate an intermediate result value based on the difference, and calculate the secondary encrypted data by the intermediate result value.
  • the first receiving module is further configured to: receive a target secondary gradient value sent by the coordinator based on the secondary encrypted data, wherein the target secondary gradient value is updated by the coordinator according to the target data The target data is obtained by decrypting and combining the primary encrypted data and the secondary encrypted data sent by the secondary participant in response to the longitudinal logistic regression model not converging and meeting the preset judgment condition. of.
  • the first receiving module is further configured to: receive a target subgradient value fed back by the coordinator, wherein the target subgradient value is obtained by splitting the first target product by the coordinator
  • the first target product is based on the second derivative matrix updated in response to the longitudinal logistic regression model satisfying the preset determination condition, the main gradient value sent by the main participant, and the main gradient value sent by the secondary participant
  • the step of receiving the target sub-gradient value fed back by the coordinator includes: receiving the target sub-gradient value fed back by the coordinator, wherein the target sub-gradient value is a second target product, and the The second target product is the calculated product between the main gradient value sent by the main participant and the preset step length calculated by the coordinator in response to the longitudinal logistic regression model not converging and not satisfying the preset determination condition.
  • the longitudinal federated learning optimization device further includes: a second receiving module for receiving the main encrypted data sent by the main participant and the secondary encrypted data sent by the secondary participant, wherein the secondary encrypted data is based on the The intermediate result value in the secondary participant is calculated, the intermediate result value is calculated by the secondary participant according to the encrypted value set sent by the main participant, and the encrypted value set includes the main encrypted value and the new encrypted value;
  • the update module is configured to respond to the failure of the longitudinal logistic regression model to converge, update the second derivative matrix according to the main encrypted data and the auxiliary encrypted data, and calculate the target subgradient value according to the updated second derivative matrix; converge; Module, used to send the target secondary gradient value to the secondary participant, and the secondary participant is used to update the local model parameters in the secondary participant based on the target secondary gradient value, and continue to execute the The secondary participant obtains the encrypted value set with linear regression value sent by the main participant until the vertical federation model corresponding to the coordinator converges.
  • the update module is further configured to determine whether the longitudinal logistic regression model satisfies the preset determination condition; if so, decrypt and merge the primary encrypted data and the secondary encrypted data to Obtain target data; store the target data in a queue with a preset length to obtain the target queue, and update the second derivative matrix through the target queue.
  • the update module is further configured to: if it is not satisfied, the coordinator obtains the first product between the secondary gradient value sent by the secondary participant and the preset step size, and compares the The first product is sent to the secondary participant as the target secondary gradient value.
  • the present application also provides a storage medium, which may be a non-volatile readable storage medium.
  • the storage medium of the present application stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the vertical federated learning optimization method described above are realized.
  • the method implemented when the computer-readable instructions running on the processor are executed please refer to the respective embodiments of the vertical federated learning optimization method of this application, which will not be repeated here.
  • the technical solution of this application essentially or the part that contributes to the existing technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM/RAM) as described above. , Magnetic disks, optical disks), including several instructions to make a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) execute the method described in each embodiment of the present application.
  • a terminal device which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Mathematical Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Optimization (AREA)
  • Pure & Applied Mathematics (AREA)
  • Computational Mathematics (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Software Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Computing Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Operations Research (AREA)
  • Probability & Statistics with Applications (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

La présente demande se rapporte au domaine technique de la technologie financière. L'invention concerne un procédé, un appareil et un dispositif d'optimisation de l'apprentissage fédéré longitudinal et un support de stockage. Le procédé consiste : à acquérir, par un participant subsidiaire, un ensemble de valeurs chiffrées à valeur de régression linéaire, envoyé par un participant maître, et à calculer des données chiffrées subsidiaires selon l'ensemble de valeurs chiffrées ; à envoyer les données chiffrées subsidiaires à un coordinateur, le coordinateur servant à mettre à jour, en réponse au cas de non-convergence d'un modèle fédéré longitudinal, une matrice dérivée de second ordre dans le coordinateur selon les données chiffrées subsidiaires, et à calculer une valeur de gradient subsidiaire cible selon la matrice dérivée de second ordre mise à jour ; et à recevoir la valeur de gradient subsidiaire cible envoyée par le coordinateur en fonction des données chiffrées subsidiaires, à mettre à jour des paramètres de modèle local dans le participant subsidiaire en fonction de la valeur de gradient subsidiaire cible et à continuer à exécuter l'étape du participant subsidiaire acquérant l'ensemble de valeurs chiffrées à valeur de régression linéaire envoyé par le participant maître, jusqu'à convergence du modèle fédéré longitudinal correspondant au coordinateur.
PCT/CN2019/119418 2019-11-14 2019-11-19 Procédé, appareil et dispositif d'optimisation d'apprentissage fédéré longitudinal et support de stockage WO2021092980A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911124702.X 2019-11-14
CN201911124702.XA CN110851786B (zh) 2019-11-14 2019-11-14 基于纵向联邦学习的企业间数据交互方法、装置、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2021092980A1 true WO2021092980A1 (fr) 2021-05-20

Family

ID=69601691

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/119418 WO2021092980A1 (fr) 2019-11-14 2019-11-19 Procédé, appareil et dispositif d'optimisation d'apprentissage fédéré longitudinal et support de stockage

Country Status (2)

Country Link
CN (1) CN110851786B (fr)
WO (1) WO2021092980A1 (fr)

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113742673A (zh) * 2021-09-07 2021-12-03 石硕 一种基于联邦学习的云边协同管控一体化平台
CN114003939A (zh) * 2021-11-16 2022-02-01 蓝象智联(杭州)科技有限公司 一种用于纵向联邦场景的多重共线性分析方法
CN114429223A (zh) * 2022-01-26 2022-05-03 上海富数科技有限公司 异构模型建立方法及装置
CN114547643A (zh) * 2022-01-20 2022-05-27 华东师范大学 一种基于同态加密的线性回归纵向联邦学习方法
CN114841373A (zh) * 2022-05-24 2022-08-02 中国电信股份有限公司 应用于混合联邦场景的参数处理方法、装置、系统及产品

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113449872B (zh) * 2020-03-25 2023-08-08 百度在线网络技术（北京）有限公司 基于联邦学习的参数处理方法、装置和系统
CN111160573B (zh) * 2020-04-01 2020-06-30 支付宝(杭州)信息技术有限公司 保护数据隐私的双方联合训练业务预测模型的方法和装置
CN112182649B (zh) * 2020-09-22 2024-02-02 上海海洋大学 一种基于安全两方计算线性回归算法的数据隐私保护系统
WO2022094888A1 (fr) * 2020-11-05 2022-05-12 浙江大学 Procédé d'apprentissage de fédération longitudinale orienté arbre de décision
CN112508199A (zh) * 2020-11-30 2021-03-16 同盾控股有限公司 针对跨特征联邦学习的特征选择方法、装置及相关设备
CN113934983A (zh) * 2021-10-27 2022-01-14 平安科技（深圳）有限公司 一种特征变量的分析方法、装置、计算机设备及存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109635422A (zh) * 2018-12-07 2019-04-16 深圳前海微众银行股份有限公司 联合建模方法、装置、设备以及计算机可读存储介质
CN110189192A (zh) * 2019-05-10 2019-08-30 深圳前海微众银行股份有限公司 一种信息推荐模型的生成方法及装置
CN110197084A (zh) * 2019-06-12 2019-09-03 上海联息生物科技有限公司 基于可信计算及隐私保护的医疗数据联合学习系统及方法
KR20190103090A (ko) * 2019-08-15 2019-09-04 엘지전자 주식회사 연합학습(Federated　learning)을　통한 단말의 POI 데이터를 생성하는 모델의 학습방법 및 이를 위한 장치

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109299728B (zh) * 2018-08-10 2023-06-27 深圳前海微众银行股份有限公司 基于构建梯度树模型的样本联合预测方法、系统及介质
CN109165515A (zh) * 2018-08-10 2019-01-08 深圳前海微众银行股份有限公司 基于联邦学习的模型参数获取方法、系统及可读存储介质
CN109034398B (zh) * 2018-08-10 2023-09-12 深圳前海微众银行股份有限公司 基于联邦训练的梯度提升树模型构建方法、装置及存储介质
CN110263936B (zh) * 2019-06-14 2023-04-07 深圳前海微众银行股份有限公司 横向联邦学习方法、装置、设备及计算机存储介质
CN112732297B (zh) * 2020-12-31 2022-09-27 平安科技（深圳）有限公司 联邦学习模型的更新方法、装置、电子设备及存储介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109635422A (zh) * 2018-12-07 2019-04-16 深圳前海微众银行股份有限公司 联合建模方法、装置、设备以及计算机可读存储介质
CN110189192A (zh) * 2019-05-10 2019-08-30 深圳前海微众银行股份有限公司 一种信息推荐模型的生成方法及装置
CN110197084A (zh) * 2019-06-12 2019-09-03 上海联息生物科技有限公司 基于可信计算及隐私保护的医疗数据联合学习系统及方法
KR20190103090A (ko) * 2019-08-15 2019-09-04 엘지전자 주식회사 연합학습(Federated　learning)을　통한 단말의 POI 데이터를 생성하는 모델의 학습방법 및 이를 위한 장치

Cited By (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113742673A (zh) * 2021-09-07 2021-12-03 石硕 一种基于联邦学习的云边协同管控一体化平台
CN113742673B (zh) * 2021-09-07 2024-05-10 石硕 一种基于联邦学习的云边协同管控一体化平台
CN114003939A (zh) * 2021-11-16 2022-02-01 蓝象智联(杭州)科技有限公司 一种用于纵向联邦场景的多重共线性分析方法
CN114003939B (zh) * 2021-11-16 2024-03-15 蓝象智联(杭州)科技有限公司 一种用于纵向联邦场景的多重共线性分析方法
CN114547643A (zh) * 2022-01-20 2022-05-27 华东师范大学 一种基于同态加密的线性回归纵向联邦学习方法
CN114547643B (zh) * 2022-01-20 2024-04-19 华东师范大学 一种基于同态加密的线性回归纵向联邦学习方法
CN114429223A (zh) * 2022-01-26 2022-05-03 上海富数科技有限公司 异构模型建立方法及装置
CN114429223B (zh) * 2022-01-26 2023-11-07 上海富数科技有限公司 异构模型建立方法及装置
CN114841373A (zh) * 2022-05-24 2022-08-02 中国电信股份有限公司 应用于混合联邦场景的参数处理方法、装置、系统及产品
CN114841373B (zh) * 2022-05-24 2024-05-10 中国电信股份有限公司 应用于混合联邦场景的参数处理方法、装置、系统及产品

Also Published As

Publication number Publication date
CN110851786B (zh) 2023-06-06
CN110851786A (zh) 2020-02-28

Similar Documents

Publication Publication Date Title
WO2021092980A1 (fr) Procédé, appareil et dispositif d'optimisation d'apprentissage fédéré longitudinal et support de stockage
WO2021092977A1 (fr) Procédé, appareil, dispositif et support de stockage d'optimisation d'apprentissage fédéré vertical
WO2020134704A1 (fr) Procédé d'entraînement de paramètres de modèle basé sur un apprentissage fédéré, terminal, système et support
CN112733967B (zh) 联邦学习的模型训练方法、装置、设备及存储介质
WO2020029589A1 (fr) Procédé et système d'acquisition de paramètre de modèle basés sur un apprentissage fédéré, et support d'informations lisible
Ding et al. Security information transmission algorithms for IoT based on cloud computing
CN113033828B (zh) 模型训练方法、使用方法、系统、可信节点及设备
WO2022247576A1 (fr) Procédé et appareil de traitement de données, dispositif et support de stockage lisible par ordinateur
WO2015155896A1 (fr) Système d'apprentissage machine de vecteur de support et procédé d'apprentissage machine de vecteur de support
WO2021159798A1 (fr) Procédé d'optimisation de système d'apprentissage fédéré longitudinal, dispositif et support de stockage lisible
TWI749444B (zh) 可靠的使用者服務系統和方法
CN112001500B (zh) 基于纵向联邦学习系统的模型训练方法、设备及存储介质
CN112818374A (zh) 一种模型的联合训练方法、设备、存储介质及程序产品
CN113609781B (zh) 基于联邦学习的汽车生产模具优化方法、系统、设备及介质
WO2017045450A1 (fr) Procédé et dispositif de traitement d'exploitation de ressources
CN111767411A (zh) 知识图谱表示学习优化方法、设备及可读存储介质
CN114696990A (zh) 基于全同态加密的多方计算方法、系统及相关设备
CN111324812A (zh) 基于迁移学习的联邦推荐方法、装置、设备及介质
CN111368196A (zh) 模型参数的更新方法、装置、设备及可读存储介质
CN110874638B (zh) 面向行为分析的元知识联邦方法、装置、电子设备及系统
CN113449872A (zh) 基于联邦学习的参数处理方法、装置和系统
CN114492850A (zh) 基于联邦学习的模型训练方法、设备、介质及程序产品
CN114429223A (zh) 异构模型建立方法及装置
CN116502732B (zh) 基于可信执行环境的联邦学习方法以及系统
CN112836767A (zh) 联邦建模方法、装置、设备、存储介质及程序产品

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19952302

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19952302

Country of ref document: EP

Kind code of ref document: A1