CN111079947B

CN111079947B - Method and system for model training based on optional private data

Info

Publication number: CN111079947B
Application number: CN201911334019.9A
Authority: CN
Inventors: 陈超超; 王力; 周俊
Original assignee: Alipay Hangzhou Information Technology Co Ltd
Current assignee: Alipay Hangzhou Information Technology Co Ltd
Priority date: 2019-12-20
Filing date: 2019-12-20
Publication date: 2022-05-17
Anticipated expiration: 2039-12-20
Also published as: CN111079947A

Abstract

One or more embodiments of the present specification relate to a method and system for model training based on optional privacy data. The method comprises the following steps: the tag holding terminal receives the operation product of at least first class data and second class data from the first terminal; the first type of data and the second type of data correspond to different privacy levels; the label holding terminal determines a data accumulated value based on the received operation product of the first type data and the second data of the label holding terminal; the label holding terminal determines a loss value of a model based on joint training of at least the label holding terminal and the first terminal based on the data accumulated value and the sample label; participating loss values in the calculation of gradient values; the gradient values are used for updating the jointly trained model; the first terminal holds first data which comprise first type data and second type data; the label holding terminal holds second data and a sample label; the first data and the second data correspond to the same training sample.

Description

Method and system for model training based on optional private data

Technical Field

One or more embodiments of the present specification relate to multi-party data collaboration, and more particularly, to a method and system for model training based on optional private data.

Background

In the fields of data analysis, data mining, economic prediction and the like, the machine learning model can be used for analyzing and finding potential data values. Since data held by a single data owner may be incomplete, and thus it is difficult to accurately characterize an object, a joint training method of a model through cooperation of data of multiple data owners is widely used in order to obtain a better model prediction result. But in the process of multi-party data cooperation, problems such as data security and model security are involved.

Therefore, there is a need to provide a secure solution for joint modeling based on multi-party data.

Disclosure of Invention

One aspect of an embodiment of the present specification provides a method of model training based on selectable privacy data; the method comprises the following steps: the tag holding terminal receives the first type of data of the first terminal, the second type of data and the operation result of the corresponding model parameter; the first type of data and the second type of data correspond to different privacy levels; the label holding terminal determines a data accumulated value based on the received data of the first terminal and second data held by the label holding terminal; the label holding terminal determines a loss value of a model based on joint training of at least the label holding terminal and the first terminal based on the data accumulated value and the sample label; participating the loss value in the calculation of a gradient value; the gradient values are used for updating the jointly trained model; the first terminal holds first data, and the first data comprises first class data and second class data; the label holding terminal holds the second data and the sample label; the first data and the second data correspond to the same training sample.

An aspect of an embodiment of the present specification provides a system for model training based on optional privacy data, the system comprising: the data receiving module is used for receiving the first class data of the first terminal, the second class data and the operation result of the corresponding model parameter; the first type of data and the second type of data correspond to different privacy levels; the data accumulated value determining module is used for determining a data accumulated value based on the received data of the first terminal and second data held by the first terminal; the loss value determining module is used for determining a loss value of a model based on joint training of the label holding terminal and the first terminal based on the data accumulated value and the sample label; the model parameter updating module is used for participating the loss value in the calculation of the gradient value; the gradient values are used for updating the jointly trained model; the first terminal holds first data, and the first data comprises first class data and second class data; the label holding terminal holds the second data and a sample label; the first data and the second data correspond to the same training sample.

One aspect of embodiments of the present specification provides an apparatus for model training based on selectable privacy data, the apparatus comprising a processor and a memory; the memory is used for storing instructions, and the processor is used for executing the instructions to realize operations corresponding to the model training method based on the optional privacy data.

One aspect of an embodiment of the present specification provides a method for model training based on selectable privacy data, the method comprising: the first terminal transmits the first type of data, the second type of data and the operation result of the corresponding model parameter to the label holding terminal; the first type of data and the second type of data correspond to different privacy levels; the first terminal receives a loss value from a label holding terminal based on a model jointly trained by the label holding terminal and the first terminal; participating the loss value in the calculation of a gradient value; the gradient values are used for updating the jointly trained model; the first terminal holds first data which comprises first type data and second type data; the tag holding terminal holds second data; the first data and the second data correspond to the same training sample.

An aspect of an embodiment of the present specification provides a system for model training based on optional privacy data, the system comprising: the data transmission module is used for transmitting the operation results of the first type data, the second type data and the corresponding model parameters to the label holding terminal; the first type of data and the second type of data correspond to different privacy levels; the loss value receiving module is used for receiving a loss value of a model based on joint training of the label holding terminal and the first terminal from the label holding terminal; the model parameter updating module is used for participating the loss value in the calculation of the gradient value; the gradient values are used for updating the jointly trained model; the first terminal holds first data which comprises first type data and second type data; the tag holding terminal holds second data; the first data and the second data correspond to the same training sample.

Drawings

The present description will be further described by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. These embodiments are not intended to be limiting, and in these embodiments like numerals are used to indicate like structures, wherein:

FIG. 1 is a diagram of an exemplary application scenario for a system for model training based on private data, according to some embodiments of the present description;

FIG. 2 is an exemplary flow diagram of a method for model training based on private data, according to some embodiments of the present description; and

FIG. 3 is an exemplary flow diagram of a method for model training based on private data, according to some further embodiments of the present description.

Detailed Description

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings used in the description of the embodiments will be briefly introduced below. It is obvious that the drawings in the following description are only examples or embodiments of the application, from which the application can also be applied to other similar scenarios without inventive effort for a person skilled in the art. Unless otherwise apparent from the context, or otherwise indicated, like reference numbers in the figures refer to the same structure or operation.

It should be understood that "system", "device", "unit" and/or "module" as used in this specification is a method for distinguishing different components, elements, parts or assemblies at different levels. However, other words may be substituted by other expressions if they accomplish the same purpose.

As used in this specification and the appended claims, the terms "a," "an," "the," and/or "the" are not to be taken in a singular sense, but rather are to be construed to include a plural sense unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements.

Flow charts are used in this description to illustrate operations performed by a system according to embodiments of the present description. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to or removed from these processes.

A large amount of information data, which is abundant in various industries such as economy, culture, education, medical care, public management, etc., is widely used in more and more scenes for performing data processing analysis such as data analysis, data mining, and trend prediction. The data cooperation mode can enable a plurality of data owners to obtain better data processing results. For example, more accurate model parameters may be obtained through joint training of multi-party data.

In some embodiments, the joint training system for performing models based on private data can be applied to a scenario in which parties train a machine learning model in cooperation for use by multiple parties while ensuring the security of the data of the parties. In this scenario, multiple data parties have their own data, and they want to use each other's data together for unified modeling (e.g., linear regression model, logistic regression model, etc.), but do not want the respective data (especially private data) to be revealed. For example, an internet deposit institution a has one batch of user data, a government bank B has another batch of user data, and a training sample set determined based on the user data of a and B can be trained to obtain a better machine learning model. Both a and B would like to participate in model training together with each other's user data, but for some reasons a and B would not like their own user data information to be revealed, or at least would not like to let the other party know their own user data information.

The model training system based on the private data can enable the machine learning model used together to be obtained through the joint training of the multi-party data under the condition that the multi-party private data are not leaked, and a win-win cooperation state is achieved.

In some scenarios, there are some training samples whose feature data require privacy protection, e.g., user's income, gender, consumption level, etc. in the sample data; there are some characteristic data that are publicly available, e.g., the home of the user, the city location, etc., in the sample data. For the above scenario, in some embodiments, all data in the training samples are treated as privacy protection data, and then the privacy data participate in the joint training through secret sharing or asymmetric encryption. In some embodiments, the privacy features of the data parties may also be correlated and then participate in training. For example, the operation product of the privacy features of each data party and the model parameters can be used for model training, so that the purpose of not revealing the privacy features is achieved.

FIG. 1 is a diagram of an exemplary application scenario for a system for model training based on private data, in accordance with some embodiments of the present description.

In some embodiments, model training system based on optional privacy data 100 includes a first terminal 110, a second terminal 120, a third terminal 130, and a network 140. The first terminal 110 may be understood as a first party data owner, including the processing device 110-1, the storage device 110-2; the second terminal 120 may be understood as a second party data owner; the third terminal 130 may be understood as a third party data owner including the processing device 130-1 and the storage device 130-2. In some embodiments, the data held by the first-party data owner and the second-party data owner relates to user-related information in different domains. For example, the data held by the parties may include the amount of money the user has deposited into a bank account each year; or the sex, age, income, address and other information of the user group related to a certain investment financing project or a certain insurance brand. Wherein a portion of the data is publicly available; some data needs to be protected from privacy. That is to say, in some embodiments of the present specification, joint training of the model is performed based on optional private data, where the optional private data includes at least feature information of the training sample, and further, may further include model parameters corresponding to the feature information.

It should be noted that the number of data owners in fig. 1 is three-way, and in other embodiments, two-way data owners or four-way data owners may be included.

First terminal 110, second terminal 120, and second terminal 130 may be devices with data acquisition, storage, and/or transmission capabilities. In some embodiments, the first terminal 110, the second terminal 120, and the second terminal 130 may include, but are not limited to, a mobile device, a tablet, a laptop, a desktop, and the like, or any combination thereof. In some embodiments, the first terminal 110, the second terminal 120, and the second terminal 130 may receive the relevant data from other terminals, for example, the second terminal 120 may receive the arithmetic product of the first type of data and the second type of data from the first terminal 110 and the third terminal 130. The second terminal 120 may also transmit the loss value to the first terminal 110 and the third terminal 130.

The processing devices 110-1, 120-1, and 130-1 of the first, second, and third terminals may perform data and/or instruction processing. Processing devices 110-1, 120-1, and 130-1 may encrypt data and may execute associated algorithms and/or instructions. For example, the processing device 110-1 of the first terminal 110 may calculate the operation product of the second class data of itself, and may also receive the loss value from the second terminal 120 and use the loss value to participate in the joint training of the model. For example, the processing device 120-1 of the second terminal 120 may calculate the loss value based on the correlation data received from the first terminal 110 and the third terminal 130.

The memory devices 110-2, 120-2, and 130-2 of the first, second, and third terminals may store data and/or instructions for execution by the corresponding processing devices 110-1, 120-1, and 130-1, which the processing devices 110-1, 120-1, and 130-1 may execute or use to implement the exemplary methods of this specification. Storage devices 110-2, 120-2, and 130-2 may be used to store their own first and second types of data, respectively; associated instructions may also be stored that instruct the first terminal and the second terminal to perform operations. Storage devices 110-2, 120-2, and 130-2 may also store data processed by processing devices 110-1 and 120-1, respectively. For example, storage devices 110-2, 120-2, and 130-2 may also store model parameters that complete the computation update, respectively. In some embodiments, the storage devices 110-2, 120-2, and 130-2 may also be a storage device, wherein the first terminal, the second terminal, and the third terminal can only obtain the data stored by themselves from the storage device. In some embodiments, the storage device may include mass storage, removable storage, volatile read-write memory, read-only memory (ROM), and the like, or any combination thereof.

Network 140 may facilitate the exchange of information and/or data. In some embodiments, one or more components of system 100 (e.g., first terminal 110 (processing device 110-1 and storage device 110-2), second terminal 120 (processing device 120-1 and storage device 120-2), and third terminal 130 (processing device 130-1 and storage device 130-2)) that perform model training based on optional privacy data may send information and/or data to other components in system 100 via network 140. For example, the processing device 110-2 of the second terminal 120 may obtain the arithmetic product of the first type of data and the second type of data from the first terminal 110 via the network 130. In some embodiments, the network 140 may be any form of wired or wireless network, or any combination thereof.

The system in one or more embodiments of the present specification may be composed of a data transmission module and a plurality of data processing modules.

In some embodiments, in a system having the second terminal as an execution subject, the data transmission module may include a data reception module; the data processing module can comprise a data accumulated value determining module, a loss value determining module and a model parameter updating module. The modules described above are implemented in a computing system as described in the application scenario, and each module includes respective instructions that may be stored on a storage medium and executed in a processor. The different modules may be located on the same device or on different devices. Data may be transferred between them via a program interface, a network, etc., and data may be read from or written to the storage device.

The data receiving module can be used for receiving the first class data of the first terminal, the second class data and the operation result of the corresponding model parameter; the first type of data and the second type of data correspond to different privacy levels.

And the data accumulated value determining module can be used for determining the data accumulated value based on the received data of the first terminal and second data owned by the first terminal.

A loss value determining module, configured to determine a loss value based on a model jointly trained by the tag holding terminal and the first terminal based on the data accumulated value and the sample tag.

And the model parameter updating module can be used for participating the loss value in the joint training of the model to obtain a parameter updated model. In some embodiments, the model parameter update module may be further operable to: determining a first type of accumulated gradient value based on the loss value and the characteristics corresponding to the first type of data on the first terminal and the label holding terminal; updating the co-trained model based on the first type of cumulative gradient values. In some embodiments, the model parameter update module may be further operable to: determining a second-class gradient value based on the loss value and the characteristics corresponding to the second-class data of the label holding terminal; and updating the joint training model based on the second type gradient value of the label holding terminal. In some embodiments, the model parameter update module may be further operable to: determining a second gradient value of the tag holding terminal based on the loss value and the characteristic corresponding to the second data; and updating the jointly trained model based on the second gradient values.

In some embodiments, in a system with a first terminal as an execution subject, the data transmission module may include a data transmission module, a loss value receiving module; the data processing module may include a model parameter update module.

The data transmission module can be used for transmitting the operation results of the first type data, the second type data and the corresponding model parameters to the label holding terminal; the first type of data and the second type of data correspond to different privacy levels.

And the loss value receiving module can be used for receiving a loss value from the label holding terminal based on the model jointly trained by the label holding terminal and the first terminal.

And the model parameter updating module is used for participating the loss value in the joint training of the model to obtain a parameter updated model. In some embodiments, the model parameter update module may be further configured to determine a second type gradient value of the first terminal based on the loss value and a feature corresponding to the second type data of the first terminal; and updating the jointly trained model based on the second type gradient value of the first terminal. In some embodiments, the model parameter update module may be further configured to determine a first gradient value of the first terminal based on the loss value and a feature corresponding to the first data; updating the jointly trained model based on the first gradient value.

It should be appreciated that the system and its modules in one or more implementations of the present description may be implemented in a variety of ways. For example, in some embodiments, the system and its modules may be implemented in hardware, software, or a combination of software and hardware. Wherein the hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory for execution by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the methods and systems described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided, for example, on a carrier medium such as a diskette, CD-or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules of the present application may be implemented not only by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but also by software executed by various types of processors, for example, or by a combination of the above hardware circuits and software (e.g., firmware).

It should be noted that the above description of the processing device and its modules is merely for convenience of description and is not intended to limit the present application to the scope of the illustrated embodiments. It will be appreciated by those skilled in the art that, given the teachings of the present system, any combination of modules or sub-system configurations may be used to connect to other modules without departing from such teachings.

FIG. 2 is an exemplary flow diagram of a method for model training based on feature-selectable data, according to some embodiments of the present description.

The variable names and formulas in the specification are only used for better understanding of the method described in the specification. In applying the present specification, based on common operation principles and machine learning principles, various immaterial transformations may be made to processes, variable names, formulas, such as transforming the order of rows or columns, transforming to an equivalent form during matrix multiplication, representing the same calculation in other calculation forms, and the like.

In this specification, convention is expressed as follows:

for the training data of the joint training model, m data samples are included, and each sample data comprises n-dimensional features. The n-dimensional feature data of the m samples are at least held by two data owners, that is, at least two data owners respectively hold different features of the m sample data, and the features held by at least two data owners include privacy features and non-privacy features. Some embodiments of the present specification take a three-party data owner as an example for detailed description, and A, B and C are respectively used to represent a first-party data owner, a second-party data owner, and a third-party data owner, and select one of them as a trusted party, which is used to summarize non-private data of each data owner and perform related operations, and a sample tag corresponding to each sample data is held by the trusted party. The first party data owner may also be referred to as a first terminal, the second party data owner may also be referred to as a second terminal, and the third party data owner may also be referred to as a third terminal.

In the representation of the present specification, the first party data owner a owns data (Xa) corresponding to a p-dimensional feature among m samples and model parameters (Wa) corresponding to the p-dimensional feature. Wherein Xa comprises private data (X1a) corresponding to the privacy preserving features and non-private data (X0a) corresponding to the non-privacy preserving features, model parameters (W1a) corresponding to the privacy preserving features and model parameters (W0) corresponding to the multi-party non-privacy preserving features; the second-party data owner B owns data (Xb) corresponding to the q-dimensional feature and model parameters (Wb) corresponding to the q-dimensional feature in the m samples. Wherein Xb comprises private data (X1b) corresponding to the privacy preserving feature and non-private data (X0b) corresponding to the non-privacy preserving feature, model parameters (W1b) corresponding to the privacy preserving feature and model parameters (W0) corresponding to the multi-party non-privacy preserving feature; the third party data owner C owns the data (Xc) corresponding to the other s-dimensional feature of the m samples and the model parameters (Wc) corresponding to the s-dimensional feature. Where Xc includes private data (X1c) corresponding to a privacy preserving feature and non-private data (X0b) corresponding to a non-privacy preserving feature, model parameters (W1c) corresponding to a privacy preserving feature and model parameters (W0) corresponding to a multi-party non-privacy preserving feature. In this specification, the model parameters may also be simply referred to as a model. Xa is a matrix of m samples, each sample being a row vector of 1 row and p columns, i.e. Xa is a matrix of m rows and p columns. Wa is a parameter matrix of p features corresponding to A, and Wa is a matrix of p rows and 1 column. Xb is a matrix of m rows and q columns. Wb is a parameter matrix of q features corresponding to B, and Wb is a matrix of q rows and 1 column. Xc is a matrix of m rows and s columns. Wc is a parameter matrix of s features corresponding to C, Wc is a matrix of s rows and 1 column, and p + q + s is n.

Tag y is held by one of A, B and C, but holding the other does not have a substantial effect. In one or more embodiments of the present description, it is desirable to select a data owner from a plurality of data owners, to aggregate non-privacy characteristics of the parties, and to calculate a loss value. Since the sample label is needed to calculate the loss value, the selected data owner also needs to hold the label. In the description, the tag y is held by B, and the calculation of the loss value is also performed by B, by summarizing the privacy characteristics of each party. Where y is a column vector of m rows and 1 column.

In addition, assume A, B, C that among the n-dimensional features of the three-party sample data, the dimension of the non-privacy feature is k; the dimension of the privacy feature is h, and h + k equals n. A. B, C cumulative data corresponding to the three-party non-private feature is represented by X0, is a matrix of m rows and k columns, and is formed by respectively adding A, B, C three-party non-private data X0a, X0b and X0 c; A. b, C cumulative data corresponding to the three-party privacy feature is represented by X1, and is formed by adding A, B, C three-party privacy data X1a, X1b and X1c, and is a matrix with m rows and h columns. The model parameters corresponding to the cumulative data X0 of the non-privacy characteristics are represented by W0, and are a matrix with k rows and 1 columns; the model parameter corresponding to the cumulative data X1 of the privacy features is represented by W1, and is a matrix of h rows and 1 columns.

In this specification, for the sake of simplicity, a data column of which the constant value is 1 is added to sample data in linear regression or logistic regression calculation and a constant 1 is added to a label are not particularly described, and n +1 in matrix calculation are not distinguished. This simplification has no substantial effect on the process described in this specification.

The above-identified expressions, variable names, and formulas and other expressions presented in this specification are only intended to provide a better understanding of the methods described in this specification. When the method is applied, various insubstantial transformations can be made on representation methods, variable names, formulas, calculation methods and the like based on common operation principles, technical principles and technical methods without influencing the essence and the corresponding technical effect of the representation methods, the variable names, the formulas, the calculation methods and the like. Such as, but not limited to, transposing the order of rows or columns, transforming to an equivalent form during matrix multiplication, representing the same calculation in other forms of calculation, etc.

Steps 210, A, C send the respective non-privacy feature to B.

A and C send respective non-privacy features X0a, X0C to B, which owns A, B, C the non-privacy features X0a, X0B, X0C of the three data owners. Steps 220, A, C each calculate a respective Z1 and send it to B.

And the A party and the C party perform product operation respectively based on the privacy feature data held by the A party and the model parameters corresponding to the privacy feature data, and send the operation result to the B party.

A carries out product operation based on the privacy feature data X1a of the A and the corresponding privacy model parameters W1a to obtain an operation product Z1a which is X1a multiplied by W1a, and sends the operation product Z1a to B.

Correspondingly, C performs product computation on its own privacy feature data X1C and the corresponding privacy model parameter W1a to obtain a computation product Z1C ═ X1C × W1C, and sends the computation product Z1C to B.

In step 230, B calculates the loss value and sends it to A and C.

B is based on data X0a, X0c received from A, C; the signature data for Z1a, Z1c, and B themselves calculate a loss value corresponding to A, B, C signature data. Wherein, the feature data of B itself includes privacy feature data X1B and non-privacy feature data X0B.

First, B determines A, B, C a summary value X0 of the three-party non-private data based on the received non-private features X0a, X0c, and X0B of B itself, and calculates Z0 ═ X0 × W0. This results in a matrix with Z0 being m rows and 1 column. In some embodiments, the calculation manner of Z0 may also be that A, B, C calculates its own Z0a ═ X0a × W0, Z0B ═ X0B × W0, Z0c ═ X0c × W0, A, C sends Z0a and Z0c to B for summing Z0 ═ Z0a + Z0B + Z0 c. The former method of calculating Z0, Z0 ═ X0 × W0, is highly efficient. In some embodiments, to further ensure the security of the data, B may determine Z0 by, for example, a secure summation method, such that B only obtains the value of Z0 and does not know the values of Z0a and Z0 c.

Next, B determines A, B, C the sum of the three-way private data operation products Z1, Z1a + Z1B + Z1c, based on the received operation products Z1a, Z1c of the privacy feature and the own operation product Z1B, X1B × W1B. The product operation results Z1a, Z1b and Z1C of the A, B and C are a matrix of m rows and 1 columns, and correspondingly, Z1 after matrix summation is also a matrix of m rows and 1 columns.

The B party adds the calculation result based on the non-private data and the calculation result based on the private data into a formula for calculating the model prediction value by addition to obtain the prediction value of the model

And subtracting the predicted value and the label y to obtain a loss value error, and sending the loss value error to the A and C parties.

In the embodiment shown in fig. 2, the jointly trained model is a logistic regression, and the predicted value of the model is determined by a sigmoid function.

In the representation agreed in the present specification, the B-party calculates Z ═ Z0+ Z1. Where Z is also a matrix of m rows and 1 column.

B party calculates predicted value by sigmoid function

Wherein, the sigmoid function can also be called a Logistic function. Predicted value calculated from the above

Is a matrix of m rows and 1 column.

The B party further calculates the loss value

The loss value error thus calculated is a matrix of m rows and 1 column.

Party B sends the loss value error to parties a and C.

In step 240, B calculates the gradient values corresponding to the multiple non-privacy features and updates W0.

And substituting the loss value into a gradient descent formula by the party B, namely performing product operation on the loss value and the received non-privacy data to obtain a corresponding gradient value, performing product operation on the gradient values corresponding to all the non-privacy characteristics of each party and the leaving rate, and updating the model.

In the presentation as agreed upon in the present specification,

the B side is calculated by using a gradient calculation formula:

where X0 is a matrix of m rows and k columns, X0^TIs a matrix of k rows and m columns, error is a matrix of m rows and 1 column, and the resulting gradient value G0 is a matrix of k rows and 1 column.

In the representation agreed in the present specification, the B-party calculates the update W0 ═ W0-learning-rate × G0. In the present specification, the learning _ rate denotes a hyper-parameter affecting the magnitude of the decrease in the gradient descent method.

Steps 250, A, B, C calculate gradient values corresponding to the respective privacy features.

A. And B and C each party substitutes the received loss value error into a gradient descent formula, namely, the loss value and the data corresponding to the privacy characteristics of each party are subjected to product operation to calculate the corresponding gradient value.

In the presentation as agreed upon in the present specification,

calculation of A

Calculation of B

C calculation of

Wherein X1a is a matrix of m rows and ha columns, X1a^TIs a ha row by m column matrix and error is an m row by 1 column matrix, so that G1a is a ha row by 1 column matrix. Correspondingly, G1b is a matrix of hb row 1 column; g1c is a matrix of hc rows and columns, ha + hb + hc ═ h. The dimensions of G1a, G1b, and G1c are the same as the dimensions of W1a, W1b, and W1c, respectively.

Steps 260, A, B, C update the respective model parameters W1.

A. And B and C, each party performs product operation on the gradient value corresponding to the privacy feature and the leaving rate, and updates the model.

In the presentation as agreed upon in the present specification,

the a-party calculation update W1a ═ W1a-learning _ rate × G1 a.

The B-party calculation update W1B ═ W1B-learning _ rate × G1B.

The C-side calculation update W1C — W1C-learning _ rate × G1C.

The above process illustrates a gradient descent process, which may be iterated until convergence, completing the training of the model. In this process, neither party A, B nor party C is aware of the private data held by the other party.

Fig. 3 is an exemplary flow diagram of a method of processing dialog information, shown in accordance with some embodiments of the present description.

In some embodiments, one or more steps of method 300 may be implemented in system 100 shown in FIG. 1. For example, one or more steps of method 300 may be stored as instructions in a corresponding storage device and invoked and/or executed by a corresponding processing device.

In step 310, the tag holding terminal receives the first type data of the first terminal, the second type data and the operation result of the corresponding model parameter. In some embodiments, step 310 may be performed by a data receiving module.

In some embodiments, the first terminal and the tag holding terminal may be understood as owners of the model training data, respectively, the first terminal holding the first data (Xa) and the tag holding terminal holding the second data (Xb). Wherein the first data and the second data correspond to different features (Xa and Xb) and model parameters (Wa and Wb) of the same sample, and the label (y) corresponding to the sample is held by the label holding terminal.

In some embodiments, the first data (Xa) may include a first type of data (X0a) and a second type of data (X1a), and the second data (Xb) may also include the first type of data (X0b) and the second type of data (X1 b). The first type of data and the second type of data correspond to different privacy levels. In some embodiments, the second type of data is more private than the first type of data, e.g., the first type of data belongs to non-private data (e.g., hometown, city, etc. of the user in the training sample), and the second type of data belongs to private data (e.g., income, gender, income level, etc. of the user in the training sample). Regarding model parameters corresponding to different feature data involved in some embodiments of the present specification, the model parameters corresponding to the first type of data X0a and the second type of data X1a of the first terminal are W0a and W1a, respectively; model parameters corresponding to first-class data X0a and X1b of the label-holding terminal are W0b and W1b respectively; the model parameters corresponding to the sum X0 of all the first type data and the sum X1 of all the second type data of the first terminal and the second terminal are W0 and W1 respectively.

With respect to the first terminal, a specific understanding of the tag holding end can be seen in the first terminal and the second terminal in fig. 2; xa, Xb, Wa, Wb; x0a, X0b, X1a, X1b, W0 and W1 can be partially explained with reference to FIG. 2.

In some embodiments, the first type of data may include at least non-private characteristic data and may also include products of non-private characteristics and corresponding model parameters. For example, the first type of data of the first terminal may include non-private feature data X0a of the first terminal; may also include non-privacy feature data X0a and corresponding model parameters W0a for the first terminal; it may also include the product X0a Wa of the non-private characteristic data with the corresponding model parameters. Correspondingly, the second type of data of the first terminal may comprise privacy feature data X1 a; may also include the privacy feature data X1a and corresponding model parameters W1a of the first terminal; it may also include the product X1a W1a of the privacy feature data and the corresponding model parameters. Here, the non-privacy feature data, the non-privacy data, and the non-privacy feature have the same meaning.

In some embodiments, the product of the second type of data comprises the product of the privacy feature data and its corresponding model parameters. The product of the second type of data of the first terminal can be understood as Z1 ═ X1a × W1 a. The first terminal transmits the operation product of the second type of data to the label holding end, so that the leakage of the privacy characteristic data of the first terminal can be prevented.

In some embodiments, the data owners participating in the model joint training are two parties, namely, the tag holding terminal and the first terminal, at this time, the tag holding terminal receives the operation product of the first type data and the second type data from the first terminal through the network 140, and as described in detail with reference to step 210 and step 220 of fig. 2, a sends its non-privacy feature to B; a calculates its own Z1 and sends it to B.

In other embodiments, the data owners participating in the model joint training are three or more parties, i.e. the tag holding terminal, the first terminal, and other terminals, which can be specifically referred to as the embodiment in fig. 2.

In some embodiments, the sample data held by the data owner may be user attribute information in at least one of insurance, banking, and medical fields. For example, a bank has identity information, running information, credit investigation information and the like of the bank client; the insurance company has the client identity information, historical insurance purchasing information, historical claim settlement information, health information, vehicle condition information and the like of the company; the medical institution has patient identity information, historical medical records and the like of the institution. In some embodiments, the user attribute information includes an image, text, or voice, among others.

In some embodiments, the model owned by the data owner may make predictions based on characteristics of the sample data. For example, a bank may predict the annual deposit growth rate of the bank based on characteristics of data such as quarterly user growth, increased user identity, bank addition policy, and the like.

In some embodiments, the private data (e.g., the first private data and the second private data) in one or more embodiments of the present description may include private data related to the entity. In some embodiments, the entity may be understood as a subject of the visualization, which may include, but is not limited to, a user, a merchant, and the like. In some embodiments, the privacy data may include image data, text data, or sound data. For example, the image data in the privacy data may be a face image of the user, a logo image of the merchant, a two-dimensional code image capable of reflecting information of the user or the merchant, and the like. For example, the text data in the privacy data may be text data of the gender, age, academic calendar, income and the like of the user, or text data of the type of commodity traded by the merchant, the time of commodity trading by the merchant and the price interval of the commodity and the like. For example, the sound data of the privacy data may be related voice content containing user personal information or user feedback, and the corresponding user personal information or user feedback information may be obtained by parsing the voice content.

And 320, the label holding terminal determines a data accumulated value based on the received data of the first terminal and second data held by the label holding terminal. In some embodiments, step 320 may be performed by the data rollup value determination module.

In some embodiments, the data rollup value may be understood as the sum Z of the products Z0 of the first type of data and the products Z1 of the second type of data that all data possess on the terminal. For the specific calculation process of Z0, Z1, Z and the data accumulated value, see step 230 in fig. 2.

And step 330, the label holding terminal determines a loss value of a model based on at least joint training of the label holding terminal and the first terminal based on the data accumulated value and the sample label. In some embodiments, step 330 may be performed by the loss value determination module.

In some embodiments, the loss value may be used to reflect the gap between the training model prediction value and the sample data truth. In some embodiments, the loss value may reflect a difference between the default value and the actual value by participating in the calculation. The related operation formulas of different training models are different, and the operation formulas corresponding to different parameter optimization algorithms are also different when the training models are the same. For example, in the embodiment shown in fig. 2 of the present specification, the training model is a logistic regression model, the parameter optimization algorithm is a gradient descent method, and the loss value is calculated by the formula

However, the operation formula for determining the loss value is not limited in one or more embodiments of the present disclosure.

In some embodiments, the joint training model may include a linear regression model, and may also include a logistic regression model. In some embodiments, when the jointly trained model comprises a logistic regression model, the predicted value of the model may be calculated based on a Sigmoid function

Wherein Z is Z0+ Z1. The detailed description may refer to step 230 of fig. 2.

In some embodiments, when the jointly trained model is a linear regression model, a linear function may be used to calculate the predicted value

Specifically, taking the linear function y ═ w × x + b as an example,

wherein w x is Z, and the encryption loss value can be calculated, then

And 340, participating the loss value in the joint training of the model to obtain a model with updated parameters. In some embodiments, step 340 may be performed by a model parameter update module.

In some embodiments, the loss value is participated in the joint training of the model, which can be understood as that the loss value is used for operation to obtain a value capable of updating the model parameter, thereby completing the parameter updating of the model once.

In some embodiments, a gradient descent method may be used to obtain a model that completes the parameter update. Specifically, the obtained loss value can be calculated to obtain a gradient value to participate in model training, and the above process is repeated until the iteration number reaches a predefined iteration number upper limit value or an error obtained by calculation after the loss value is introduced is smaller than a predefined numerical value, so that a model with updated parameters or good training is obtained.

In other embodiments, other parameter optimization methods may be used instead of the gradient descent method, such as newton descent method, and the like, and one or more embodiments of the present disclosure are not limited thereto.

When the tag holding terminal completes calculation of the loss value error, the error value needs to be sent to other terminals for updating and using model parameters. See step 230 of fig. 2.

In some embodiments, the tag-holding terminal may calculate a first type of cumulative gradient value based on the loss value error and the first type of data received from the respective terminals, and then update the model parameters based on the cumulative gradient value. The first type of cumulative gradient value W0 can be understood as the product of the loss value error and the sum X0 of the features X0a, X0b, and X0c corresponding to the non-private data of each terminal. The specific calculation process for W0 and the updating of its model parameters can be seen in step 240 of fig. 2.

In some embodiments, each data-holding terminal with the loss value error may determine its second type gradient value based on the loss value and the corresponding characteristics of its second type data (i.e., the privacy data), and update the corresponding model parameters. Wherein each data holding terminal at least comprises a label holding terminal and a first terminal. Each data-holding terminal further includes other terminals if three data terminals participate. For a detailed procedure of calculating the second type of gradient values of each data terminal, namely the privacy gradient values G1a, G1b, G1c, and updating the corresponding model parameters W1a, W1b, W1c, see step 250 in fig. 2.

In some embodiments, after the tag holding terminal may transmit the loss value error to each of other terminals, each terminal may also directly calculate gradient values corresponding to all private data and non-private data of itself based on the loss value error, and then update corresponding model parameters. Wherein each terminal includes at least a tag holding terminal and a first terminal. Specifically, taking a three-party data terminal as an example, the first terminal a, the tag holding terminal B, and the other terminal C have their own data (including private data and non-private data) Xa, Xb, and Xc, respectively, and the three-party data terminal calculates their own gradient values Ga ═ Xa ═ Xb ═ Wb, and Gc ═ Xc ═ Wc, and then calculates and updates their respective model parameters Wa ═ Wa-learning _ rate × Ga, Wb ═ Wb-learning _ rate × Gb, and Wc ═ learning _ rate × Gc, respectively. More detailed calculation processes can refer to the relevant contents of fig. 2.

In some embodiments, to further ensure the security of the data, the loss value may be calculated by way of sharing or other secure calculation, and the loss value corresponding to each data party is obtained by participating in calculation under the condition that the privacy feature data or the calculation product corresponding to the privacy data of each data party is not leaked.

It should be noted that the above description related to the flow 300 is only for illustration and explanation, and does not limit the applicable scope of the present application. Various modifications and changes to flow 300 will be apparent to those skilled in the art in light of this disclosure. However, such modifications and variations are intended to be within the scope of the present application. For example, the first terminal may add the first privacy data to the public key of the second terminal, and the second terminal may add the second privacy data to the public key of the second terminal and send the result to the first terminal, which is within the protection scope of the present application.

The beneficial effects that may be brought by the embodiments of the present application include, but are not limited to: (1) multi-party data combined training is adopted, so that the utilization rate of data is improved, and the accuracy of a prediction model is improved; (2) in the optional privacy data protection scene, the privacy data and the non-privacy data are respectively and correspondingly processed according to different requirements, so that the security of the privacy data in the training data is improved; (3) in the selectable privacy data protection scene, only the data needing privacy protection in the training data is subjected to security processing, so that the computational efficiency of security calculation is improved. It is to be noted that different embodiments may produce different advantages, and in different embodiments, the advantages that may be produced may be any one or combination of the above, or any other advantages that may be obtained.

Having thus described the basic concept, it will be apparent to those skilled in the art that the foregoing detailed disclosure is to be considered merely illustrative and not restrictive of the broad application. Various modifications, improvements and adaptations to the present application may occur to those skilled in the art, although not explicitly described herein. Such modifications, improvements and adaptations are proposed in the present application and thus fall within the spirit and scope of the exemplary embodiments of the present application.

Also, this application uses specific language to describe embodiments of the application. Reference throughout this specification to "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with at least one embodiment of the present application is included in at least one embodiment of the present application. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, certain features, structures, or characteristics may be combined as suitable in one or more embodiments of the application.

Moreover, those skilled in the art will appreciate that aspects of the present application may be illustrated and described in terms of several patentable species or situations, including any new and useful combination of processes, machines, manufacture, or materials, or any new and useful improvement thereon. Accordingly, various aspects of the present application may be embodied entirely in hardware, entirely in software (including firmware, resident software, micro-code, etc.) or in a combination of hardware and software. The above hardware or software may be referred to as "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the present application may be represented as a computer product, including computer readable program code, embodied in one or more computer readable media.

The computer storage medium may comprise a propagated data signal with the computer program code embodied therewith, for example, on baseband or as part of a carrier wave. The propagated signal may take any of a variety of forms, including electromagnetic, optical, etc., or any suitable combination. A computer storage medium may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code located on a computer storage medium may be propagated over any suitable medium, including radio, cable, fiber optic cable, RF, or the like, or any combination of the preceding.

Computer program code required for the operation of various portions of the present application may be written in any one or more programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C + +, C #, VB.NET, Python, and the like, a conventional programming language such as C, Visualbasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, a dynamic programming language such as Python, Ruby, and Groovy, or other programming languages, and the like. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or processing device. In the latter scenario, the remote computer may be connected to the user's computer through any network format, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet), or in a cloud computing environment, or as a service, such as a software as a service (SaaS).

Additionally, the order in which elements and sequences of the processes described herein are processed, the use of alphanumeric characters, or the use of other designations, is not intended to limit the order of the processes and methods described herein, unless explicitly claimed. While various presently contemplated embodiments of the invention have been discussed in the foregoing disclosure by way of example, it is to be understood that such detail is solely for that purpose and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements that are within the spirit and scope of the embodiments herein. For example, although the system components described above may be implemented by hardware devices, they may also be implemented by software-only solutions, such as installing the described system on an existing processing device or mobile device.

Similarly, it should be noted that in the preceding description of embodiments of the application, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure aiding in the understanding of one or more of the embodiments. This method of disclosure, however, is not intended to require more features than are expressly recited in the claims. Indeed, the embodiments may be characterized as having less than all of the features of a single embodiment disclosed above.

Numerals describing the number of components, attributes, etc. are used in some embodiments, it being understood that such numerals used in the description of the embodiments are modified in some instances by the use of the modifier "about", "approximately" or "substantially". Unless otherwise indicated, "about", "approximately" or "substantially" indicates that the number allows a variation of ± 20%. Accordingly, in some embodiments, the numerical parameters used in the specification and claims are approximations that may vary depending upon the desired properties of the individual embodiments. In some embodiments, the numerical parameter should take into account the specified significant digits and employ a general digit preserving approach. Notwithstanding that the numerical ranges and parameters setting forth the broad scope of the range are approximations, in the specific examples, such numerical values are set forth as precisely as possible within the scope of the application.

The entire contents of each patent, patent application publication, and other material cited in this application, such as articles, books, specifications, publications, documents, and the like, are hereby incorporated by reference into this application. Except where the application is filed in a manner inconsistent or contrary to the present disclosure, and except where the claim is filed in its broadest scope (whether present or later appended to the application) as well. It is noted that the descriptions, definitions and/or use of terms in this application shall control if they are inconsistent or contrary to the statements and/or uses of the present application in the material attached to this application.

Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of the embodiments of the present application. Other variations are also possible within the scope of the present application. Thus, by way of example, and not limitation, alternative configurations of the embodiments of the present application can be viewed as being consistent with the teachings of the present application. Accordingly, the embodiments of the present application are not limited to only those embodiments explicitly described and depicted herein.

Claims

1. A method for model training based on selectable privacy data; the method comprises the following steps:

the tag holding terminal receives the first type of data of the first terminal, the second type of data and the operation result of the corresponding model parameter; the first type of data and the second type of data correspond to different privacy levels;

the label holding terminal determines a data accumulated value based on the received data of the first terminal and second data held by the label holding terminal;

the label holding terminal determines a loss value of a model based on joint training of at least the label holding terminal and the first terminal based on the data accumulated value and the sample label;

participating the loss value in the calculation of a gradient value; the gradient values are used for updating the jointly trained model;

the first terminal holds first data, and the first data comprises first class data and second class data; the label holding terminal holds the second data and the sample label; the first data and the second data correspond to the same training sample.

2. The method of claim 1, the jointly trained model comprising a linear regression model or a logistic regression model.

3. The method of claim 1, the gradient values comprising a first class of cumulative gradient values; the participating of the loss values in the computation of gradient values comprises:

the tag holding terminal determines a first type of accumulated gradient value based on the loss value and the characteristics corresponding to the first type of data on the first terminal and the tag holding terminal;

updating the co-trained model based on the first type of cumulative gradient values.

4. The method of claim 3, the gradient values further comprising a second class of gradient values; the participating of the loss values in the computation of gradient values comprises:

the tag holding terminal determines a second type gradient value based on the loss value and the characteristics corresponding to the second type data of the tag holding terminal; and updating the joint training model based on the second gradient value of the label holding terminal.

5. The method of claim 1, the gradient values further comprising a second gradient value; the participating of the loss values in the computation of gradient values comprises:

the tag holding terminal determines a second gradient value of the tag holding terminal based on the loss value and the characteristic corresponding to the second data;

updating the jointly trained model based on the second gradient values.

6. The method of claim 1, the first and second data comprising image data, text data, or sound data related to an entity.

7. A system for model training based on selectable private data, the system comprising:

the data receiving module is used for receiving the first class data of the first terminal, the second class data and the operation result of the corresponding model parameter; the first type of data and the second type of data correspond to different privacy levels;

the data accumulated value determining module is used for determining a data accumulated value based on the received data of the first terminal and second data held by the first terminal;

the loss value determining module is used for determining a loss value of a model based on joint training of the label holding terminal and the first terminal based on the data accumulated value and the sample label;

the model parameter updating module is used for participating the loss value in the calculation of the gradient value; the gradient values are used for updating the jointly trained model;

the first terminal holds first data, and the first data comprises first class data and second class data; the label holding terminal holds the second data and a sample label; the first data and the second data correspond to the same training sample.

8. The system of claim 7, the jointly trained model comprising a linear regression model or a logistic regression model.

9. The system of claim 7, the gradient values comprising a first type of cumulative gradient values; the model parameter update module is further to:

determining a first type of accumulated gradient value based on the loss value and the characteristics corresponding to the first type of data on the first terminal and the label holding terminal;

updating the jointly trained model based on the first type of cumulative gradient values.

10. The system of claim 9, the gradient values further comprising a second class of gradient values; the model parameter update module is further to:

determining a second-class gradient value based on the loss value and the characteristics corresponding to the second-class data of the label holding terminal; and updating the joint training model based on the second type gradient value of the label holding terminal.

11. The system of claim 7, the model parameter update module further to:

determining a second gradient value of the tag holding terminal based on the loss value and the characteristic corresponding to the second data; and updating the jointly trained model based on the second gradient values.

12. The system of claim 7, the first and second data comprising image data, text data, or sound data related to an entity.

13. An apparatus for model training based on selectable privacy data, the apparatus comprising a processor and a memory; the memory is used for storing instructions, and the processor is used for executing the instructions to realize the corresponding operation of the model training method based on the optional privacy data according to any one of claims 1 to 6.

14. A method of model training based on selectable privacy data, the method comprising:

the first terminal transmits the first type of data, the second type of data and the operation result of the corresponding model parameter to the label holding terminal; the first type of data and the second type of data correspond to different privacy levels;

the first terminal receives a loss value from a label holding terminal based on a model jointly trained by the label holding terminal and the first terminal;

the first terminal holds first data which comprises first type data and second type data; the tag holding terminal holds second data; the first data and the second data correspond to the same training sample.

15. The method of claim 14, the gradient values further comprising a second class of gradient values; the participating of the loss values in the computation of gradient values comprises:

the first terminal determines a second type gradient value of the first terminal based on the loss value and the characteristics corresponding to the second type data of the first terminal; and updating the jointly trained model based on the second type gradient value of the first terminal.

16. The method of claim 14, the gradient values further comprising a first gradient value; the participating of the loss values in the computation of gradient values comprises:

the first terminal determines a first gradient value of the first terminal based on the loss value and the characteristic corresponding to the first data; updating the jointly trained model based on the first gradient value.

17. A system for model training based on selectable private data, the system comprising:

the data transmission module is used for transmitting the operation results of the first type data, the second type data and the corresponding model parameters to the label holding terminal; the first type of data and the second type of data correspond to different privacy levels;

the loss value receiving module is used for receiving a loss value of a model based on joint training of the label holding terminal and the first terminal from the label holding terminal;

18. The system of claim 17, the gradient values further comprising a second class of gradient values; the model parameter update module is further to:

determining a second type gradient value of the first terminal based on the loss value and the characteristics corresponding to the second type data of the first terminal; and updating the jointly trained model based on the second type gradient value of the first terminal.

19. The system of claim 17, the gradient values further comprising a first gradient value; the model parameter update module is further to:

determining a first gradient value of the first terminal based on the loss value and the characteristic corresponding to the first data; updating the jointly trained model based on the first gradient value.

20. An apparatus for model training based on selectable privacy data, the apparatus comprising a processor and a memory; the memory is used for storing instructions, and the processor is used for executing the instructions to realize the corresponding operation of the model training method based on the optional privacy data according to any one of claims 14 to 16.