CN114429223B

CN114429223B - Heterogeneous model building method and device

Info

Publication number: CN114429223B
Application number: CN202210091039.3A
Authority: CN
Inventors: 卞阳; 张翔; 陈立峰; 李腾飞; 邢旭; 张伟奇
Original assignee: Shanghai Fudata Technology Co ltd
Current assignee: Shanghai Fudata Technology Co ltd
Priority date: 2022-01-26
Filing date: 2022-01-26
Publication date: 2023-11-07
Anticipated expiration: 2042-01-26
Also published as: CN114429223A

Abstract

The application provides a heterogeneous model building method and a heterogeneous model building device, which relate to the field of deep learning models, and the method comprises the following steps: acquiring information representing roles of other participating modeling nodes; issuing algorithm information to a task server to determine a target modeling algorithm; performing collaborative modeling operation with the task server based on a target modeling algorithm, wherein the collaborative modeling operation comprises: calculating a loss value of the preliminary heterogeneous model based on a target modeling algorithm, determining an update gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the update gradient value; and determining whether the preliminary heterogeneous model converges or not according to the weight value and the loss value, and obtaining the target heterogeneous model when determining that the preliminary heterogeneous model converges. The method provided by the embodiment of the application can enable the process of establishing the target heterogeneous model to be interconnected and intercommunicated among all federal learning platforms in the federal learning process, and simultaneously improves the safety in the cooperative modeling process.

Description

Heterogeneous model building method and device

Technical Field

The application relates to the field of deep learning, in particular to a heterogeneous model building method and device.

Background

Federal learning is a mode of jointly establishing a machine learning model on the premise that a plurality of participants cannot acquire data of each other, and heterogeneous models refer to that each federal learning platform uses own data structure design to realize a machine learning algorithm.

At present, the training process of the heterogeneous model cannot be interconnected and intercommunicated on each federal learning platform due to various reasons such as heterogeneous data structures of each platform, non-uniform federal logistic regression frameworks, various training methods, non-uniform communication protocols and the like.

Disclosure of Invention

Based on the above, the embodiment of the application aims to provide a heterogeneous model building method and device, so that built target heterogeneous models can be interconnected and intercommunicated among federal learning platforms, and model application scenes are expanded.

In a first aspect, an embodiment of the present application provides a heterogeneous model building method applied to a task requester, including:

acquiring information representing roles of other modeling nodes, wherein the roles comprise a task requesting machine and a task service machine, and the roles are determined based on modeling scenes and holding data of the modeling nodes;

issuing algorithm information to the task server to determine a target modeling algorithm;

Performing collaborative modeling operation with the task server based on the target modeling algorithm, wherein the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

and determining whether the preliminary heterogeneous model converges or not according to the weight value and the loss value, and obtaining a target heterogeneous model when determining that the preliminary heterogeneous model converges.

In the implementation process, the heterogeneous federal logistic regression algorithm can keep the block consistency based on the determined target modeling algorithm in the collaborative modeling process, and the federal platforms align and distribute the roles of each modeling node through operators, so that federal logistic regression collaborative modeling of different structures or algorithms can be realized, the process of establishing the target heterogeneous model can be interconnected and intercommunicated among the federal learning platforms in the federal learning process, and meanwhile, the safety in collaborative modeling is improved.

Optionally, before the collaborative modeling operation is performed with the task server based on the objective modeling algorithm, the method may include:

Determining a communication key, a federal homomorphic calculation cryptographic key or a safe calculation fragment based on the algorithm information issued to the task server;

the performing collaborative modeling operation with the task server based on the target modeling algorithm may include:

and carrying out data communication with the task server based on the communication key, the federal homomorphic calculation password key or the secure calculation fragment so as to have the same mathematical definition as the input and output results of the preliminary heterogeneous model constructed by the task server in the cooperative modeling operation.

In the implementation process, interaction can be performed based on a preset communication mode, the self password module can be shared with other modeling nodes, and data sharing is realized based on one-time communication interaction, so that the efficiency of collaborative modeling can be improved.

Optionally, the calculating the loss value of the preliminary heterogeneous model based on the objective modeling algorithm may include:

receiving an inner product of the preliminary heterogeneous model sent by the task server based on a preset transmission direction, or sending the inner product to the task server, wherein the inner product is obtained by the current iteration model weight and a sample characteristic value of the preliminary heterogeneous model;

Substituting the inner product into a preset function to obtain the loss value, and sending the loss value to the task server.

Optionally, the updating the weight value of the preliminary heterogeneous model based on the updated gradient value may include:

determining the updated gradient value according to the loss value and the sample characteristic value;

and determining the weight value of the updated preliminary heterogeneous model based on the updated gradient value and the learning rate of the preliminary heterogeneous model.

In a second aspect, an embodiment of the present application provides a heterogeneous model building method applied to a task server, including:

receiving algorithm information issued from the task requesting machine, and determining a target modeling algorithm;

performing collaborative modeling operation with the task requester based on the target modeling algorithm, the collaborative modeling operation including: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

In the implementation process, the heterogeneous federation logistic regression algorithm can keep the segmentation consistency based on the determined target modeling algorithm in the collaborative modeling process, and the federation platforms align and distribute the roles of each modeling node through operators, so that the federation logistic regression collaborative modeling of different structures or algorithms can be realized, and the established target heterogeneous model can be interconnected and intercommunicated among the federation learning platforms.

determining a communication key, a federal homomorphic calculation cryptographic key or a secure calculation fragment based on the algorithm information issued from the task requester;

the collaborative modeling operation with the task requester based on the target modeling algorithm may include:

and carrying out data communication with the task requesting machine based on the communication key, the federal homomorphic calculation password key or the secure calculation fragment so as to have the same mathematical definition as the input and output results of the preliminary heterogeneous model constructed by the task requesting machine during the collaborative modeling operation.

Optionally, the calculating the loss value of the preliminary heterogeneous model based on the target modeling algorithm includes:

receiving an inner product of the preliminary heterogeneous model sent by the task requester based on a preset transmission direction, or sending the inner product to the task requester, wherein the inner product is obtained by the current iteration model weight and a sample characteristic value of the preliminary heterogeneous model;

and receiving the loss value sent from the task requester, wherein the loss value is obtained by substituting the inner product into a preset function by the request server.

Optionally, the updating the weight value of the preliminary heterogeneous model based on the updated gradient value includes:

In a third aspect, an embodiment of the present application provides a heterogeneous model building apparatus applied to a task requester, including:

The first acquisition module is used for acquiring information representing roles of other modeling nodes, wherein the roles comprise a task requesting machine and a task service machine, and the roles are determined based on modeling scenes and holding data of the modeling nodes;

the issuing module is used for issuing algorithm information to the task server and determining a target modeling algorithm;

the first collaborative modeling module is used for performing collaborative modeling operation with the task server based on the target modeling algorithm, and the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

and the first judging module is used for determining whether the preliminary heterogeneous model converges or not according to the weight value and the loss value, and obtaining a target heterogeneous model when determining that the preliminary heterogeneous model converges.

Optionally, the heterogeneous model building device may further include a first communication module, configured to determine a communication key, a federal homomorphic computing cryptographic key, or a secure computing fragment based on the algorithm information issued to the task server.

The first collaborative modeling module may be specifically configured to perform data communication with the task server based on the communication key, the federal homomorphic computing cryptographic key, or the secure computing fragment, so as to be the same as a mathematical definition of an input/output result of a preliminary heterogeneous model constructed by the task server during the collaborative modeling operation.

Optionally, the first collaborative modeling module may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task server based on a preset transmission direction, or send the inner product to the task server, where the inner product is obtained by a current iteration model weight and a sample feature value of the preliminary heterogeneous model; substituting the inner product into a preset function to obtain the loss value, and sending the loss value to the task server.

And determining the updated gradient value from the loss value and the sample feature value; and determining the weight value of the updated preliminary heterogeneous model based on the updated gradient value and the learning rate of the preliminary heterogeneous model.

In a fourth aspect, an embodiment of the present application further provides a heterogeneous model building device applied to a task service machine, including:

the second acquisition module is used for acquiring information representing roles of other modeling nodes, wherein the roles comprise a task requesting machine and a task service machine, and the roles are determined based on modeling scenes and holding data of the modeling nodes;

the receiving module is used for receiving algorithm information issued from the task requesting machine and determining a target modeling algorithm;

the second collaborative modeling module is used for performing collaborative modeling operation with the task requester based on the target modeling algorithm, and the collaborative modeling operation comprises: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

and the second judging module is used for determining whether the preliminary heterogeneous model converges or not according to the weight value and the loss value, and obtaining a target heterogeneous model when determining that the preliminary heterogeneous model converges.

Optionally, the heterogeneous model building device may further include a second communication module, configured to determine a communication key, a federal homomorphic calculation cryptographic key, or a secure calculation fragment based on the algorithm information issued from the task requester.

The second collaborative modeling module may be specifically configured to perform data communication with the task requester based on the communication key, the federal homomorphic computing cryptographic key, or the secure computing shard, so as to be the same as a mathematical definition of an input/output result of a preliminary heterogeneous model constructed by the task requester during the collaborative modeling operation.

Optionally, the second collaborative modeling module may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task requester based on a preset transmission direction, or send the inner product to the task requester, where the inner product is obtained by a current iteration model weight and a sample feature value of the preliminary heterogeneous model; and receiving the loss value sent from the task requester, wherein the loss value is obtained by substituting the inner product into a preset function by the request server.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments of the present application will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present application and should not be considered as limiting the scope, and other related drawings can be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of steps of a heterogeneous model building method applied to a task requester according to an embodiment of the present application;

FIG. 2 is a schematic diagram of steps for communicating a task requester with a task server according to an embodiment of the present application;

FIG. 3 is a schematic diagram of steps of collaborative modeling according to an embodiment of the present application;

fig. 4 is a schematic step diagram of a heterogeneous model building method applied to a task server according to an embodiment of the present application;

FIG. 5 is a schematic diagram illustrating steps for a task server and a task requester according to an embodiment of the present application;

FIG. 6 is a schematic diagram illustrating steps of collaborative modeling applied to a task server according to an embodiment of the present application;

FIG. 7 is a schematic diagram of a heterogeneous model building device applied to a task requester according to an embodiment of the present application;

fig. 8 is a schematic diagram of a heterogeneous model building device of a task server according to an embodiment of the present application.

Detailed Description

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. For example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions. In addition, functional modules in the embodiments of the present application may be integrated together to form a single part, or each module may exist alone, or two or more modules may be integrated to form a single part.

The applicant finds that the current model building mode based on federation learning is divided into longitudinal federation learning and transverse federation learning according to the difference of data and user application scenes in the research process, and the longitudinal federation learning is to segment the data sets according to longitudinal (feature dimension) under the condition that users of two data sets overlap more and user features overlap less, and take out the part of data of which the users of both sides are identical and the user features are not identical for training. The horizontal federation learning is opposite to the vertical, and the two-party data are subjected to learning modeling under the scene that the two-party data have the same characteristics and the users are not completely the same. Each federal learning platform uses own data structure design to realize a machine learning algorithm, and each platform has different structures at the algorithm level due to the fact that federal logistic regression frameworks are not uniform, training methods are different, communication protocols are not uniform and data structures are heterogeneous. Therefore, heterogeneous models cannot be interconnected and intercommunicated among the federal learning platforms, and the problem of limited model application scenes exists.

Based on the above, the embodiment of the application provides a heterogeneous model building method, which is characterized in that a logistic regression algorithm calculation operator is modularized, and a plurality of modeling nodes are cooperated to build a heterogeneous model based on preset module attributes, so that the built target heterogeneous model can be interconnected and intercommunicated among federal learning platforms. Referring to fig. 1, fig. 1 is a schematic step diagram of a heterogeneous model building method applied to a task requester according to an embodiment of the present application, and an implementation manner of heterogeneous model building may include the following steps:

In step S11, information characterizing roles of other participating modeling nodes is obtained, the roles including a task requester and a task server, the roles being determined based on a modeling scenario and holding data of the modeling nodes.

The modeling node roles can be federal learning platforms, the federal learning platforms can operate on equipment with communication functions and data processing functions, and the equipment can be a computer, a server, a configurator of engineering equipment and a cloud server.

The modeling scene characterizes the training mode of the modeling node and the application scene of the user, the model learning mode of the modeling node is determined to be longitudinal federation learning or transverse federation learning, and the holding data refers to whether the model in the modeling node has an output value Y or not. And judging the alignment role by acquiring information of roles of other modeling nodes, and determining each modeling node as a task requester Guest or a task server Host. The task requester Guest is used as a calculation master, the Y value is held to be responsible for calculating the loss value in a vertical scene, the task initiator is generally used as Guest in a horizontal scene, the task server Host is a participation calculation party, and the Y value-free party is in the vertical scene. The task requesting machine and the task service machine may be electronic devices running a federal learning platform.

In step S12, algorithm information is issued to the task server to determine a target modeling algorithm.

The modeling algorithm may be an algorithm capable of realizing logistic regression, such as a secureAggregation security aggregation algorithm, a seeumLR security logistic regression algorithm, or the like, which may be preconfigured in an algorithm module, and when a corresponding modeling algorithm is selected as a target modeling algorithm, the corresponding algorithm module may be selected to perform collaborative modeling operation based on a unified operator.

After obtaining information representing roles of other modeling nodes involved in the modeling, each modeling node checks task types, determines whether a collaborative modeling mode is horizontal modeling or vertical modeling, determines itself as a task requester Guest or task server Host according to sample content of itself such as a Y value position, data amount and the like, and invokes an algorithm module agreed with the role of itself to perform modeling according to the assigned roles.

In step S13, a collaborative modeling operation is performed with the task server based on the target modeling algorithm, where the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value.

The loss value of the preliminary heterogeneous model can be calculated by a sigmoid function or a function for approximately calculating the sigmoid through the current model weight and the sample characteristic value or through the current model vector and the sample characteristic vector. In the heterogeneous model building method provided by the embodiment of the application, the update gradient value can be calculated according to the loss value based on a gradient descent method or a random average gradient method and the like, and the weight value is updated based on the calculated update gradient value.

In step S14, whether the preliminary heterogeneous model converges is determined according to the weight value and the loss value, and when it is determined that the preliminary heterogeneous model converges, a target heterogeneous model is obtained.

For example, a weight change threshold and a loss threshold may be preset, and when the change value of the weight is lower than the weight change threshold and the loss value is lower than the loss threshold, it may be determined that the weight and the loss value are no longer changed in a large range, and the preliminary heterogeneous model is converged, so that model training is completed, and a built target heterogeneous model is obtained.

Therefore, in the embodiment of the application, the heterogeneous federal logistic regression algorithm can keep the segmentation consistency based on the determined target modeling algorithm in the collaborative modeling process, and the federal platforms are aligned and distributed with each modeling node role through operators, so that federal logistic regression collaborative modeling of different structures or algorithms can be realized, the process of establishing the target heterogeneous model can be interconnected and intercommunicated among the federal learning platforms in the federal learning process, and meanwhile, the safety in collaborative modeling is improved.

In an alternative embodiment, before the collaborative modeling operation is performed with the task server based on the target modeling algorithm in step S11, an implementation of communicating with other participating modeling nodes is provided in an embodiment of the present application. Referring to fig. 2, fig. 2 is a schematic diagram illustrating steps of a task requester and a task server for communication, where the steps of communicating the task requester and the task server may include:

in step S21, a communication key, a federal homomorphic calculation cryptographic key, or a secure calculation fragment is determined based on the algorithm information issued to the task server.

The task requester Guest can issue an encryption mode appointed in advance, the encryption mode can be homomorphic encryption, noise confusion, multiparty security calculation and the like, an encryption object exchange public key and a private key are correspondingly sent when the encryption mode is homomorphic encryption, a noise cancellation calculation item is correspondingly sent when the encryption mode is noise confusion, and an encryption fragment is correspondingly sent when the encryption mode is multiparty security calculation. The federal homomorphic calculation cryptographic key is a key that transmits data when homomorphic encryption is used in the federal learning process.

In step S22, the task server is in data communication with the communication key, the federal homomorphic computing cryptographic key, or the secure computing fragment, so as to be the same as the mathematical definition of the input and output results of the preliminary heterogeneous model constructed by the task server in the collaborative modeling operation.

The encryption algorithm may be pre-configured in the encryption algorithm module, and when the corresponding encryption algorithm is selected for communication, the data is encrypted or decrypted through the corresponding encryption module, and the own encryption module can be shared with other platforms in an SDK manner between the platforms. In addition, the platforms can also communicate based on a common communication protocol or a common password data structure.

The algorithm information includes contents required to be defined in a data structure, such as encryption, noise adding or fragmentation data before transmission of an algorithm module, a communication encryption mode, whether output data needs decryption or not, a data structure of a public key of a cryptographic algorithm, a data structure of a private key ciphertext and the like.

In the modeling calculation process, a corresponding field conversion rule is sent between the task requesting machine Guest and the task service machine Host, so that the mathematical definition of the input and output results of the modules is the same, and the data sharing can be realized based on one-time communication interaction.

Therefore, the embodiment of the application can interact based on the preset communication mode, can share the own password module with other modeling nodes, and realizes data sharing based on one-time communication interaction, thereby improving the efficiency of collaborative modeling.

In an alternative embodiment, for step S13, an implementation manner of collaborative modeling is provided in the embodiment of the present application, please refer to fig. 3, fig. 3 is a schematic diagram of steps of collaborative modeling provided in the embodiment of the present application, and the steps of collaborative modeling may include:

in step S31, an inner product of the preliminary heterogeneous model sent by the task server is received based on a preset transmission direction, or the inner product is sent to the task server, where the inner product is obtained by a current iteration model weight and a sample feature value of the preliminary heterogeneous model.

In step S32, the inner product is substituted into a preset function to obtain the loss value, and the loss value is sent to the task server.

The inner product calculation process needs to be carried out by combining the two parties, the calculated inner product can be selected to be encrypted or sent to be original according to the security level requirement and the different algorithm frameworks, and finally the inner product is calculated by the task requester to obtain a loss value.

In step S33, the update gradient value is determined according to the loss value and the sample characteristic value.

In step S34, the updated weight value of the preliminary heterogeneous model is determined based on the updated gradient value and the learning rate of the preliminary heterogeneous model.

For example, the algorithm for calculating the inner product, the loss value, updating the gradient value, and updating the weight may be preconfigured in the corresponding calculation module, and proceed to the corresponding calculation moduleThe corresponding calculation module is called, for example, the inner product calculation module is used for calculating the inner product of the current modeling node based on the formula u=wx, wherein U is the inner product of the current modeling node, w is the current model weight, and x is the sample characteristic value; the Loss value calculation module is used for calculating a corresponding Loss value Loss from the inner product according to the Sigmoid function or the function for approximately calculating the Sigmoid, and the update gradient value calculation module is used for calculating the Loss value Loss based on the formula g= [ sig (wx) -y]Calculating an updated gradient value, wherein g is a gradient, sig (wx) represents a loss value, y is an output value of a current model, and x is a sample characteristic value of the model; the weight updating module is used for being based on a formula w _new The updated weights are calculated by =a×g, where a is the model learning rate, g is the gradient, w _new Is the updated gradient value.

The update gradient value calculation module and the loss value calculation module need to be aligned in two directions when calculating, the update gradient value Deltaw can use a gradient descent method or a random average gradient method and other methods, operators of the two modules are both calculated locally, wherein a communication event is that a secret object of the operator is sent to a decryption party for decryption, and the secret object refers to encrypted data.

In addition, the task requester Guest and the task server Host may establish a sending receiver for sending and receiving the communication dictionary according to roles, for example: the data direction is from Guest to Host. The interaction event name conversion dictionary in the operator communication may be a participant event name to a heterogeneous model event name: such as "b_1": "inter_1", "inter_1": "b_1". And sending a data structure conversion function interface in operator communication.

If the intermediate communication involves inconsistent software program structures, the two parties need to exchange conversion program interfaces to complete interconnection. Each platform can share its own logistic regression communication module with other platforms.

Taking an application scenario of homomorphic encryption three-party modeling as an example, each platform participating in collaborative modeling judges roles in the modeling process according to own data, wherein the roles in the application scenario comprise a task requester Guest, a task server Host and a third party Abiter.

The roles participating in collaborative modeling exchange algorithm information in a communication manner, the same algorithm is selected from a self logistic regression algorithm library, and each operator module is corresponding to the same operator.

The task requesting machine Guest and the task service machine Host exchange information of cryptographic modules, the public key and the private key are generated by using the same cryptographic module, such as a paillier, and the third party arbiter is used for decryption during communication interaction.

The task requester Guest and the task server Host select to use corresponding inner product calculation modules to calculate inner products, wherein the transmission direction of communication is that the task server Host sends the calculated inner products to the task requester Guest, and the communication can be an encryption mode based on an addition homomorphism (Paillier). The task server Host calculates the inner product ub of the task server Host, encrypts the inner product ub to obtain ubs, and sends ubs in a secret state to the task requester Guest. The task requester Guest calculates an inner product ua according to own data and adds the inner product ua and ub to obtain an inner product u, and calculates a model weight w and a sample characteristic value x of the heterogeneous model according to the inner product u.

The task requester Guest and the task server Host both use a selective Loss value calculation module to calculate Loss values, and the task requester Guest calculates Loss values Loss of the heterogeneous model based on a Sigmoid function or a function similar to the Sigmoid and sends the Loss values Loss to the task server Host.

Both the task requester Guest and the task server Host choose to use an update gradient value calculation module to calculate an update gradient value Deltaw according to the loss value. And sending the calculated update gradient value delta w to a third party Arbiter and decrypting by the third party Arbiter, wherein the transmitted cryptographic protocol can be an encryption mode of an addition homomorphism in the steps and is accompanied by a random number, the third party Arbiter decrypts the secret state delta w to obtain the update gradient value delta w, and returns a plaintext to both the task requester Guest and the task server Host and is accompanied by the update gradient value delta w and a Loss value Loss of the random number, the task requester Guest and the task server Host subtract the obtained values by the random number to obtain the update gradient value delta w and the Loss value Loss, the model weight w is updated according to the update gradient value delta w and whether the current model converges or not is judged according to whether the weight w and the Loss value Loss change in a large range, and the steps of calculating and integrating the model weight w until the model converges are repeated until the model converges, so that the target heterogeneous model is obtained.

Optionally, the method provided by the application can also be applied to a homomorphic encryption two-party modeling scene, and in the application scene, each participant needs to judge whether the role is a task requester Guest or a task server Host according to own data. The task request machine Guest and the task service machine Host exchange algorithm information, the same algorithm is selected from the self logistic regression algorithm library, all operator modules are corresponding to the same operator, the password module information is exchanged, and the same password module is used for generating public and private keys.

The task requesting machine Guest and the task server Host select to use the corresponding inner product calculation module to calculate the inner product, wherein the transmission direction of communication is that the task server Host sends the calculated inner product to the task requesting machine Guest, and the communication mode can be plaintext transmission.

The task server Host calculates the internal product ub of itself and sends ub to the task requester Guest, which calculates the internal product ua of itself.

The task requester Guest and the task server Host both use a selective Loss value calculation module to calculate a Loss value, and the task requester Guest sends the calculated Loss value Loss to the task server Host, and the communication mode can be an encryption mode based on addition homomorphism.

The task requesting machine Guest and the task server machine Host select to use an update gradient value calculation module to calculate an update gradient value Deltaw according to the Loss value, and respectively send the update gradient value Deltaw and the Loss value Loss calculated by the task requesting machine Guest and the task server machine Host to each other, wherein the communication mode can be a mode based on addition homomorphism and accompanied by random numbers.

Subtracting the random number from the obtained value by the task requester Guest and the task server Host to obtain an updated gradient value Deltaw and a Loss value Loss, updating the model weight w according to the updated gradient value Deltaw, judging whether the current model is converged according to whether the weight w and the Loss value Loss change in a large range, and repeating the steps of calculating the inner product until the model weight w is updated when the model is not converged until the model is converged, so as to obtain the target heterogeneous model.

Optionally, the method provided by the application can also be applied to a scene of transmission gradient plus noise modeling, and in the application scene, each participant needs to judge whether the role is a task requester Guest or a task server Host according to own data. The task request machine Guest and the task service machine Host exchange algorithm information, the same algorithm is selected from the self logistic regression algorithm library, all operator modules are corresponding to the same operator, the information of the password modules is exchanged, and the noise cancellation term is calculated by using the same noise generation method module.

The task requesting machine Guest and the task server Host select to use the corresponding inner product calculation module to calculate the inner product, wherein the transmission direction of communication is that the task server Host sends the calculated inner product to the task requesting machine Guest, and the communication mode can be plaintext transmission. The task server Host calculates the internal product ub of itself and sends ub to the task requester Guest, which calculates the internal product ua of itself.

The task requester Guest and the task server Host both use a selective Loss value calculation module to calculate a Loss value, and the task requester Guest sends the calculated Loss value Loss to the task server Host, and the communication mode can be an encryption mode based on noise.

The task requesting machine Guest and the task server machine Host select to use the update gradient value calculation module to calculate the update gradient value Deltaw according to the Loss value, and respectively send the update gradient value Deltaw and the Loss value Loss calculated by the task requesting machine Guest and the task server machine Host to the other party, wherein the communication mode can be a mode based on noise and accompanied by random numbers.

And the task requesting machine Guest and the task service machine Host both eliminate noise based on the cancellation term, subtract the random number from the obtained value to obtain an updated gradient value delta w and a Loss value Loss, update the model weight w according to the updated gradient value delta w, judge whether the current model is converged according to whether the weight w and the Loss value Loss change in a large range, and repeat the steps of calculating the inner product until the model weight w is updated when the model is not converged until the model is converged, so as to obtain the target heterogeneous model.

Optionally, the method provided by the application can also be applied to a scene of multiparty security computation (MPC), and in the application scene, each participant needs to judge whether the role is a task requester Guest or a task server Host according to own data. The task request machine Guest and the task service machine Host exchange algorithm information, the same algorithm is selected from the self logistic regression algorithm library, all operator modules are corresponding to the same operator, the password module information is exchanged, the same multiparty secure computation initialization MPC module is used, and y values, w and x are fragmented.

The task requesting machine Guest and the task server Host select to use the corresponding inner product calculation module to calculate the inner product, wherein the communication transmission direction is that the task server Host sends the calculated inner product to the task requesting machine Guest, and the communication mode can be fragment data structure transmission based on MPC fragments. The task server Host calculates the internal product ub of itself and sends ub to the task requester Guest, which calculates the internal product ua of itself.

The task requester Guest and the task server Host both use a selective Loss value calculation module to calculate Loss values, and the task requester Guest sends the calculated Loss values Loss to the task server Host, and the communication mode can be fragment data structure transmission based on MPC fragments.

The task requesting machine Guest and the task server Host select to use an update gradient value calculation module to calculate an update gradient value Deltaw according to the Loss value, and respectively send the update gradient value Deltaw and the Loss value Loss calculated by the task requesting machine Guest and the task server Host to each other, wherein the communication mode can be fragment data structure transmission based on MPC fragments.

And the task requesting machine Guest and the task server machine Host both acquire a Loss value Loss from the fragment state Loss value based on fragment recovery, update the model weight w according to the update gradient value delta w and judge whether the current model is converged according to whether the weight w and the Loss value Loss change in a large range, and repeat the steps of calculating the inner product until the model weight w is updated when the model is not converged until the model is converged, determine whether to recover the fragment state weight value w according to the setting, and finally acquire the target heterogeneous model.

Optionally, the method provided by the application can also be applied to a scene of two-party horizontal federal logistic regression learning, wherein in the application scene, the task request machine Guest is the initiator, and the other party is the task server Host. The task request machine Guest and the task service machine Host exchange algorithm information, the same algorithm is selected from the self logistic regression algorithm library, all operator modules are corresponding to the same operator, the password module information is exchanged, the same password module is used for generating public and private keys and generating a fusion weight random number.

The task requester Guest and the task server Host select to use the corresponding inner product calculation module to calculate the inner product, wherein the transmission direction of communication is that the task server Host sends the calculated inner product to the task requester Guest, and the communication mode can be an encryption mode based on addition homomorphism and multiplied by a random number.

The task requester Guest and the task server Host both use a selective Loss value calculation module to calculate Loss values, the task server Host sends the calculated Loss values Loss to the task requester Guest, and the task requester Guest accumulates the Loss values calculated by the task server Host, and the communication mode can be a plaintext transmission mode.

The task requesting machine Guest and the task server Host select to use an update gradient value calculation module to calculate an update gradient value Deltaw according to the Loss value, encrypt and send the update gradient value Deltaw and the Loss value Loss calculated by the task requesting machine Guest and the task server Host to each other respectively, and calculate the update gradient value Deltaw and the Loss value Loss based on the intermediate result in a dense state. The communication mode can be a mode based on addition homomorphism and accompanied by a random number.

Subtracting the random number of the task requester Guest and the task server Host from the calculation result obtained by decryption to obtain an update gradient value, calculating the global update gradient value Deltaw of the two parties, updating the model weight w according to the global update gradient value Deltaw, judging whether the current model is converged according to whether the weight w and the Loss value Loss change in a large range, and repeating the steps of calculating the inner product until the model weight w is updated when the model is not converged until the model is converged, so as to obtain the target heterogeneous model.

Based on the same inventive concept, the embodiment of the present application further provides a heterogeneous model building method applied to a task server, referring to fig. 4, fig. 4 is a schematic step diagram of the heterogeneous model building method applied to a task server provided by the embodiment of the present application, and an implementation manner of heterogeneous model building may include the following steps:

In step S41, information characterizing roles of other participating modeling nodes is obtained, the roles including a task requester and a task server, the roles being determined based on a modeling scenario and holding data of the modeling nodes.

In step S42, algorithm information issued from the task requester is received, and a target modeling algorithm is determined.

In step S43, a collaborative modeling operation is performed with the task requester based on the target modeling algorithm, where the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value.

In step S44, whether the preliminary heterogeneous model converges is determined according to the weight value and the loss value, and when it is determined that the preliminary heterogeneous model converges, a target heterogeneous model is obtained.

Therefore, in the embodiment of the application, the heterogeneous federation logistic regression algorithm can keep the segmentation consistency based on the determined target modeling algorithm in the collaborative modeling process, and the federation platforms are aligned and distributed with each modeling node role through operators, so that federation logistic regression collaborative modeling of different structures or algorithms can be realized, and the established target heterogeneous model can be interconnected and intercommunicated among the federation learning platforms.

Optionally, before obtaining the information characterizing the roles of other participating modeling nodes in step S41, an embodiment of the present application provides an implementation manner of communicating with other participating modeling nodes, referring to fig. 5, fig. 5 is a schematic diagram of a step of communicating between a task server and a task requester provided in the embodiment of the present application, where the step of communicating between the task server and the task requester may include:

in step S51, a communication key, a federal homomorphic calculation cryptographic key, or a secure calculation fragment is determined based on the algorithm information issued from the task requester.

In step S52, the task requester is in data communication with the communication key, the federal homomorphic calculation cryptographic key, or the secure calculation fragment, so as to be the same as the mathematical definition of the input and output results of the preliminary heterogeneous model constructed by the task requester in the collaborative modeling operation.

Optionally, with reference to step S43, an implementation manner of collaborative modeling applied to a task server is provided in an embodiment of the present application, referring to fig. 6, fig. 6 is a schematic diagram of steps of collaborative modeling applied to a task server provided in an embodiment of the present application, and the steps of collaborative modeling of a task server may include:

In step S61, an inner product of the preliminary heterogeneous model sent by the task requester is received based on a preset transmission direction, or the inner product is sent to the task requester, where the inner product is obtained by a current iteration model weight and a sample feature value of the preliminary heterogeneous model.

In step S62, the loss value sent from the task requester is received, where the loss value is obtained by substituting the inner product into a preset function by the request server.

In step S63, the updated gradient value is determined from the loss value and the sample feature value.

In step S64, the updated weight value of the preliminary heterogeneous model is determined based on the updated gradient value and the learning rate of the preliminary heterogeneous model.

The specific implementation manner of the heterogeneous model building method applied to the task server, the implementation manner of communicating with the task requester, the implementation manner of collaborative modeling and the specific application scenario can be referred to the implementation manner and the application scenario applied to the task requester description, and are not repeated here.

Based on the same inventive concept, the embodiment of the present application further provides a heterogeneous model building device 70 applied to a task requester, referring to fig. 7, fig. 7 is a schematic diagram of the heterogeneous model building device applied to the task requester provided in the embodiment of the present application, where the heterogeneous model building device 70 may include:

A first obtaining module 71 is configured to obtain information characterizing roles of other modeling nodes involved, where the roles include a task requester and a task server, and the roles are determined based on a modeling scenario and holding data of the modeling nodes.

And the issuing module 72 is used for issuing algorithm information to the task server and determining a target modeling algorithm.

A first collaborative modeling module 73, configured to perform a collaborative modeling operation with the task server based on the target modeling algorithm, where the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value.

A first judging module 74, configured to determine whether the preliminary heterogeneous model converges according to the weight value and the loss value, and obtain a target heterogeneous model when determining that the preliminary heterogeneous model converges.

Optionally, the heterogeneous model building device 70 may further include a first communication module, configured to determine a communication key, a federal homomorphic computing cryptographic key, or a secure computing patch based on the algorithm information issued to the task server.

The first collaborative modeling module 73 may be specifically configured to communicate data with the task server based on the communication key, the federally homomorphic computing cryptographic key, or the secure computing patch, so as to be the same as a mathematical definition of an input-output result of a preliminary heterogeneous model constructed by the task server during the collaborative modeling operation.

Optionally, the first collaborative modeling module 73 may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task server based on a preset transmission direction, or send the inner product to the task server, where the inner product is obtained by a current iteration model weight and a sample feature value of the preliminary heterogeneous model; substituting the inner product into a preset function to obtain the loss value, and sending the loss value to the task server.

Based on the same inventive concept, the embodiment of the present application further provides a heterogeneous model building device 80 applied to a task server, referring to fig. 8, fig. 8 is a schematic diagram of the heterogeneous model building device of the task server provided by the embodiment of the present application, where the heterogeneous model building device 80 may include:

A second obtaining module 81 is configured to obtain information characterizing roles of other modeling nodes involved, where the roles include a task requester and a task server, and the roles are determined based on a modeling scenario and holding data of the modeling nodes.

And the receiving module 82 is used for receiving algorithm information issued from the task requesting machine and determining a target modeling algorithm.

A second collaborative modeling module 83, configured to perform collaborative modeling operations with the task requester based on the target modeling algorithm, where the collaborative modeling operations include: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value.

And a second judging module 84, configured to determine whether the preliminary heterogeneous model converges according to the weight value and the loss value, and obtain a target heterogeneous model when determining that the preliminary heterogeneous model converges.

Optionally, the heterogeneous model building device 80 may further include a second communication module for determining a communication key, a federally homomorphic computing cryptographic key, or a secure computing patch based on the algorithm information issued from the task requester.

The second collaborative modeling module 83 may be specifically configured to communicate data with the task requester based on the communication key, the federally homomorphic computing cryptographic key, or the secure computing patch, so as to be the same as a mathematical definition of an input/output result of a preliminary heterogeneous model constructed by the task requester during the collaborative modeling operation.

Optionally, the second collaborative modeling module 83 may be further specifically configured to receive an inner product of the preliminary heterogeneous model sent by the task requester based on a preset transmission direction, or send the inner product to the task requester, where the inner product is obtained from a current iteration model weight and a sample feature value of the preliminary heterogeneous model; and receiving the loss value sent from the task requester, wherein the loss value is obtained by substituting the inner product into a preset function by the request server.

Based on the same inventive concept, the embodiment of the application also provides an electronic device, which comprises a memory and a processor, wherein the memory stores program instructions, and the processor executes the steps in any implementation mode when reading and running the program instructions.

Based on the same inventive concept, the embodiments of the present application also provide a computer readable storage medium, in which computer program instructions are stored, which when read and run by a processor, perform the steps in any of the above implementations.

The computer readable storage medium may be any of various media capable of storing program codes, such as random access Memory (Random Access Memory, RAM), read Only Memory (ROM), programmable Read Only Memory (Programmable Read-Only Memory, PROM), erasable Read Only Memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable Read Only Memory (Electric Erasable Programmable Read-Only Memory, EEPROM), and the like. The storage medium is used for storing a program, the processor executes the program after receiving an execution instruction, and the method executed by the electronic terminal defined by the process disclosed in any embodiment of the present application may be applied to the processor or implemented by the processor.

In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method may be implemented in other manners. The above-described apparatus embodiments are merely illustrative, for example, the division of the units is merely a logical function division, and there may be other manners of division in actual implementation, and for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some communication interface, device or unit indirect coupling or communication connection, which may be in electrical, mechanical or other form.

Further, the units described as separate units may or may not be physically separate, and units displayed as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Furthermore, functional modules in various embodiments of the present application may be integrated together to form a single portion, or each module may exist alone, or two or more modules may be integrated to form a single portion.

Alternatively, it may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part.

The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, for example, by wired (e.g., coaxial cable, optical fiber, digital Subscriber Line (DSL)), or wireless (e.g., infrared, wireless, microwave, etc.).

In this document, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.

The above description is only an example of the present application and is not intended to limit the scope of the present application, and various modifications and variations will be apparent to those skilled in the art. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. The heterogeneous model building method is characterized by being applied to a task requesting machine and comprising the following steps of:

acquiring information representing roles of other modeling nodes, wherein the roles comprise a task requesting machine and a task service machine, the roles are determined based on modeling scenes and holding data of the modeling nodes, the modeling scenes are training modes used for representing the modeling nodes and application scenes of users, and the holding data are used for representing whether models in the modeling nodes have output values or not;

judging alignment roles by acquiring information of other modeling node roles to determine that a modeling node is a task requesting machine or a task server, wherein when a model learning mode in the modeling node is longitudinal federal learning, the task requesting machine calculates a loss value according to the output value;

issuing algorithm information to the task server to determine a target modeling algorithm, wherein the target modeling algorithm is a logistic regression algorithm;

the collaborative modeling operation based on the target modeling algorithm and the task server comprises the following steps:

Carrying out data communication with the task server based on the communication key, the federal homomorphic calculation password key or the secure calculation fragment so as to have the same mathematical definition as the input and output results of the preliminary heterogeneous model constructed by the task server during the collaborative modeling operation;

performing collaborative modeling operation with the task server based on the target modeling algorithm; wherein the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value; and

determining whether the preliminary heterogeneous model converges or not according to the weight value and the loss value, and obtaining a target heterogeneous model when determining that the preliminary heterogeneous model converges;

the algorithm for calculating the inner product, the loss value, updating the gradient value and updating the weight is preconfigured in the corresponding calculation module, and the corresponding calculation module is called when the corresponding calculation step is carried out,

the inner product calculation module is used for calculating the inner product of the current modeling node based on the formula U=wx, wherein U is the inner product of the current modeling node, w is the current model weight, and x is the sample characteristic value; the Loss value calculation module is used for calculating a corresponding Loss value Loss by an inner product according to a Sigmoid function or a function for approximately calculating the Sigmoid, and the update gradient value calculation module is used for calculating an update gradient value based on a formula g= [ sig (wx) -y ], wherein g is a gradient, sig (wx) represents the Loss value, y is an output value of a current model, and x is a sample characteristic value of the model; the weight updating module is used for calculating updated weights based on a formula w new=a×g, wherein a is a model learning rate, g is a gradient, and w new is an updated gradient value.

2. The method of claim 1, wherein the calculating a loss value for a preliminary heterogeneous model based on the target modeling algorithm comprises:

receiving an inner product of the preliminary heterogeneous model transmitted by the task server based on a preset transmission direction, or transmitting the inner product to the task server; the inner product is obtained by the current iteration model weight and sample characteristic value of the preliminary heterogeneous model;

3. The method of claim 2, wherein the updating the weight values of the preliminary heterogeneous model based on the updated gradient values comprises:

determining the updated gradient value according to the loss value and the sample characteristic value; and

4. The heterogeneous model building method is characterized by being applied to a task server and comprising the following steps of:

acquiring information representing roles of other participating modeling nodes, wherein the roles comprise a task requesting machine and a task service machine; the role is determined based on a modeling scene and holding data of the modeling node, wherein the modeling scene is a training mode for representing the modeling node and an application scene of a user, and the holding data is used for representing whether a model in the modeling node has an output value or not;

before performing collaborative modeling operation with the task server based on the target modeling algorithm, the method includes:

performing collaborative modeling operation with the task requester based on the target modeling algorithm includes:

carrying out data communication with the task requesting machine based on the communication secret key, the federal homomorphic calculation secret key or the secure calculation fragment so as to have the same mathematical definition as the input and output results of the preliminary heterogeneous model constructed by the task requesting machine during the collaborative modeling operation;

performing collaborative modeling operation with the task requesting machine based on a target modeling algorithm; wherein the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

5. The method of claim 4, wherein the calculating a loss value for a preliminary heterogeneous model based on the target modeling algorithm comprises:

Receiving an inner product of the preliminary heterogeneous model transmitted by the task requester based on a preset transmission direction or transmitting the inner product to the task requester; the inner product is obtained by the current iteration model weight and sample characteristic value of the preliminary heterogeneous model; and

6. The method of claim 5, wherein the updating the weight values of the preliminary heterogeneous model based on the updated gradient values comprises:

7. A heterogeneous model building device, which is applied to a task requesting machine, comprising:

the first acquisition module is used for acquiring information representing roles of other modeling nodes, wherein the roles comprise a task requesting machine and a task service machine; the role is determined based on a modeling scene and holding data of the modeling node, wherein the modeling scene is a training mode for representing the modeling node and an application scene of a user, and the holding data is used for representing whether a model in the modeling node has an output value or not;

the issuing module is used for issuing algorithm information to the task server to determine a target modeling algorithm, wherein the target modeling algorithm is a logistic regression algorithm;

the first collaborative modeling module is used for performing collaborative modeling operation with the task server based on the target modeling algorithm; wherein the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

The first judging module is used for determining whether the preliminary heterogeneous model converges or not according to the weight value and the loss value, and obtaining a target heterogeneous model when determining that the preliminary heterogeneous model converges;

8. The heterogeneous model building device is characterized by being applied to a task service machine and comprising the following components:

The second acquisition module is used for acquiring information representing roles of other modeling nodes, wherein the roles comprise a task requesting machine and a task service machine; the role is determined based on a modeling scene and holding data of the modeling node, wherein the modeling scene is a training mode for representing the modeling node and an application scene of a user, and the holding data is used for representing whether a model in the modeling node has an output value or not;

before performing collaborative modeling operation with the task server based on the target modeling algorithm, the method comprises the following steps:

the second collaborative modeling module is used for performing collaborative modeling operation with the task requesting machine based on the target modeling algorithm; wherein the collaborative modeling operation includes: calculating a loss value of a preliminary heterogeneous model based on the target modeling algorithm, determining an updated gradient value based on the loss value, and updating a weight value of the preliminary heterogeneous model based on the updated gradient value;

the second judging module is used for determining whether the preliminary heterogeneous model converges or not according to the weight value and the loss value, and obtaining a target heterogeneous model when determining that the preliminary heterogeneous model converges;