CN114282688A

CN114282688A - Two-party decision tree training method and system

Info

Publication number: CN114282688A
Application number: CN202210196287.4A
Authority: CN
Inventors: 方文静; 王力
Original assignee: Alipay Hangzhou Information Technology Co Ltd
Current assignee: Alipay Hangzhou Information Technology Co Ltd
Priority date: 2022-03-02
Filing date: 2022-03-02
Publication date: 2022-04-05
Anticipated expiration: 2042-03-02
Also published as: CN114282688B

Abstract

The embodiment of the specification discloses a two-party decision tree training method and a two-party decision tree training system so as to protect data privacy of two parties. For any feature, the devices of the two parties can sort the first gradient vector and the second gradient vector according to the feature value of the feature by a secure sorting method based on the sorting vector, obtain the fragments and the groups of the first/second gradient sorting result vector, and calculate the fragments of the gradient sum corresponding to each group under the feature. The devices of the two parties interact according to a multi-party security calculation protocol, and the split gain slices respectively corresponding to the groups under the characteristics are calculated based on the gradients and the slices respectively corresponding to the groups under the characteristics. Furthermore, the devices of the two sides determine the characteristics and the groups corresponding to the maximum splitting gain through a multi-party safety comparison protocol, and split the nodes according to the characteristics and the groups corresponding to the maximum splitting gain.

Description

Two-party decision tree training method and system

Technical Field

The present disclosure relates to the field of information technology, and in particular, to a two-way decision tree training method and system.

Background

In order to protect the privacy of data of all parties, a distributed training scheme is adopted in the field of machine learning. That is, any participant can train the model belonging to itself without revealing sample data held by any participant. The decision tree is a tree structure model, i.e., a tree model, and has a very wide application, for example, in decision making, classification, and the like of various scenes.

It is currently desirable to provide a distributed training scheme for decision trees.

Disclosure of Invention

One of the embodiments of the present specification provides a two-party decision tree training method, in which feature values and label values of one or more features of each sample in a sample set are vertically distributed in the two parties, the method is performed by a device of a first party, the first party is either one of the two parties, and a second party is the other one of the two parties; the method comprises the following steps:

splitting any node according to the following splitting steps:

obtaining a first slice of a landmark vector, a first slice of a first gradient vector, and a first slice of a second gradient vector of the node; the flag vector indicates samples belonging to the respective node, the first gradient vector includes a first gradient corresponding to the samples belonging to the respective node, and the second gradient vector includes a second gradient corresponding to the samples belonging to the respective node;

for any feature of the first party:

sorting the sample set according to the characteristic value of the characteristic and obtaining a first arrangement vector, wherein the arrangement vector is used for identifying the operation of sorting the equal-length sequence, and the element of the arrangement vector indicates the position of the data corresponding to the position in the equal-length sequence in a sorting result sequence; based on a first fragment of a first gradient vector, the first arrangement vector and a second fragment of a second party based on the first gradient vector, arranging the first gradient vector according to the characteristic value by a safe arrangement method to obtain a first fragment of the first gradient arrangement result vector, and simultaneously obtaining a second fragment of the first gradient arrangement result vector by the second party; based on the first fragment of the second gradient vector, the first arrangement vector and the second fragment of the second party based on the second gradient vector, arranging the second gradient vector according to the characteristic value by a safe arrangement method to obtain the first fragment of the second gradient arrangement result vector, and simultaneously obtaining the second fragment of the second gradient arrangement result vector by the second party;

dividing elements in a first segment of the first gradient sorting result vector and elements in a first segment of the second gradient sorting result vector according to a preset dividing mode to obtain a plurality of first groups of the first segment of the first gradient vector and a plurality of first groups of the first segment of the second gradient vector, and for each of the plurality of first groups: calculating the sum of elements included in the first group of the first segment of the first gradient vector to obtain a first segment of a first gradient sum corresponding to the first group; calculating the sum of elements included in the first group of the first segment of the second gradient vector to obtain a first segment of a second gradient sum corresponding to the first group;

for any feature of the second party:

arranging the first gradient vector according to the characteristic value of the characteristic by a safe arrangement method based on the first fragment of the first gradient vector and the second fragment and the second arrangement vector of the second party based on the first gradient vector to obtain the first fragment of the first gradient ordering result vector, and simultaneously obtaining the second fragment of the first gradient ordering result vector by the second party; based on the first fragment of the second gradient vector, and the second fragment and the second arrangement vector of the second party based on the second gradient vector, arranging the second gradient vector according to the characteristic value by a safe arrangement method to obtain the first fragment of the second gradient ordering result vector, and simultaneously obtaining the second fragment of the second gradient ordering result vector by the second party; the second arrangement vector is obtained by ordering the sample set by a second party according to the characteristic value of the characteristic;

dividing elements in a first segment of the first gradient sorting result vector and elements in a first segment of the second gradient sorting result vector according to a preset dividing mode to obtain a plurality of second groups of the first segment of the first gradient vector and a plurality of second groups of the first segment of the second gradient vector, and for each of the plurality of second groups: calculating the sum of the elements included in the second grouping of the first patches of the first gradient vector to obtain first patches of the first gradient sum corresponding to the second grouping; calculating the sum of the elements included in the second grouping of the first patches of the second gradient vector to obtain first patches of second gradient sums corresponding to the second grouping;

interacting with equipment of a second party according to a multi-party safety calculation protocol to calculate first fragments of splitting gains corresponding to the groups under the characteristics respectively on the basis of first fragments of a first gradient sum and first fragments of a second gradient sum corresponding to the groups under the characteristics respectively;

interacting with equipment of a second party according to a multi-party safety comparison protocol, determining the maximum splitting gain based on the first fragments of the splitting gains corresponding to the groups under the characteristics, and recording the splitting information of the nodes according to the characteristics and the groups corresponding to the maximum splitting gain;

when the maximum splitting gain corresponds to a feature of a first party, generating a left sub-tree vector and a right sub-tree vector of the node, the left sub-tree vector indicating samples in a left subset obtained by dividing the sample set according to the feature corresponding to the maximum splitting gain and groups, the right sub-tree vector indicating samples in a right subset obtained by dividing the sample set according to the feature corresponding to the maximum splitting gain and groups, the left subset corresponding to the left sub-tree, and the right subset corresponding to the right sub-tree; splitting the left sub-tree vector into a first slice and a second slice, and sending the second slice of the left sub-tree vector to a device of a second party; splitting the right sub-tree vector into a first slice and a second slice, and sending the second slice of the right sub-tree vector to a device of a second party; receiving, from a device of a second party, a first tile of a left sub-tree vector and a first tile of a right sub-tree vector of the node when the maximum splitting gain corresponds to a feature of the second party;

interacting with a device of a second party according to a multi-party secure computing protocol to compute a first shard of a landmark vector of the node based on the first shard of the landmark vector of the node and the first shard of the left sub-tree vector; interacting with a device of a second party according to a multi-party secure computation protocol to compute a first shard of a landmark vector of the node based on the first shard of the landmark vector of the node and the first shard of the right subtree vector;

interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a first gradient vector of a left sub-tree of the node based on the first tile of the first gradient vector of the node and a first tile of a landmark vector of the left sub-tree of the node; interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a second gradient vector of a left sub-tree of the node based on the first tile of the second gradient vector of the node and the first tile of the landmark vector of the left sub-tree of the node;

interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a first gradient vector of a right sub-tree of the node based on the first tile of the first gradient vector of the node and a first tile of a landmark vector of the right sub-tree of the node; interacting with a device of a second party in accordance with a multi-party secure computing protocol to compute a first tile of a second gradient vector of a right sub-tree of the node based on the first tile of the second gradient vector of the node and the first tile of the landmark vector of the right sub-tree of the node.

One embodiment of the present specification provides a two-party decision tree training system, in which feature values and label values of one or more features of each sample in a sample set are vertically distributed in the two parties, and the system is implemented on a device of a first party, where the first party is either one of the two parties, and the second party is the other one of the two parties; the system comprises a first obtaining module, a first arranging module, a second arranging module, a first grouping gradient and fragment computing module, a second grouping gradient and fragment computing module, a splitting gain comparison module, a left sub-tree vector fragment obtaining module, a right sub-tree vector fragment computing module, a sub-node sign vector fragment computing module and a sub-node gradient vector fragment computing module.

For any node that is split:

the first obtaining module is used for obtaining a first segment of the mark vector of the node, a first segment of a first gradient vector and a first segment of a second gradient vector; the flag vector indicates samples belonging to the respective node, the first gradient vector includes a first gradient corresponding to the samples belonging to the respective node, and the second gradient vector includes a second gradient corresponding to the samples belonging to the respective node;

for any feature held by the first party, the first arrangement module is to: sorting the sample set according to the characteristic value of the characteristic and obtaining a first arrangement vector, wherein the arrangement vector is used for identifying the operation of sorting the equal-length sequence, and the element of the arrangement vector indicates the position of the data corresponding to the position in the equal-length sequence in a sorting result sequence; based on a first fragment of a first gradient vector, the first arrangement vector and a second fragment of a second party based on the first gradient vector, arranging the first gradient vector according to the characteristic value by a safe arrangement method to obtain a first fragment of the first gradient arrangement result vector, and simultaneously obtaining a second fragment of the first gradient arrangement result vector by the second party; based on the first fragment of the second gradient vector, the first arrangement vector and the second fragment of the second party based on the second gradient vector, arranging the second gradient vector according to the characteristic value by a safe arrangement method to obtain the first fragment of the second gradient arrangement result vector, and simultaneously obtaining the second fragment of the second gradient arrangement result vector by the second party;

for any feature held by the first party, the first packet gradient and fragmentation computation module is to: dividing elements in a first segment of the first gradient sorting result vector and elements in a first segment of the second gradient sorting result vector according to a preset dividing mode to obtain a plurality of first groups of the first segment of the first gradient vector and a plurality of first groups of the first segment of the second gradient vector, and for each of the plurality of first groups: calculating the sum of elements included in the first group of the first segment of the first gradient vector to obtain a first segment of a first gradient sum corresponding to the first group; calculating the sum of elements included in the first group of the first segment of the second gradient vector to obtain a first segment of a second gradient sum corresponding to the first group;

for any feature held by a second party, the second ranking module is to: arranging the first gradient vector according to the characteristic value of the characteristic by a safe arrangement method based on the first fragment of the first gradient vector and the second fragment and the second arrangement vector of the second party based on the first gradient vector to obtain the first fragment of the first gradient ordering result vector, and simultaneously obtaining the second fragment of the first gradient ordering result vector by the second party; based on the first fragment of the second gradient vector, and the second fragment and the second arrangement vector of the second party based on the second gradient vector, arranging the second gradient vector according to the characteristic value by a safe arrangement method to obtain the first fragment of the second gradient ordering result vector, and simultaneously obtaining the second fragment of the second gradient ordering result vector by the second party; the second arrangement vector is obtained by ordering the sample set by a second party according to the characteristic value of the characteristic;

for any feature held by the second party, the second packet gradient and shard calculation module is to: dividing elements in a first segment of the first gradient sorting result vector and elements in a first segment of the second gradient sorting result vector according to a preset dividing mode to obtain a plurality of second groups of the first segment of the first gradient vector and a plurality of second groups of the first segment of the second gradient vector, and for each of the plurality of second groups: calculating the sum of the elements included in the second grouping of the first patches of the first gradient vector to obtain first patches of the first gradient sum corresponding to the second grouping; calculating the sum of the elements included in the second grouping of the first patches of the second gradient vector to obtain first patches of second gradient sums corresponding to the second grouping;

the split gain slice calculation module is used for interacting with equipment of a second party according to a multi-party security calculation protocol so as to calculate first slices of split gains corresponding to groups under each characteristic respectively based on first slices of a first gradient sum and first slices of a second gradient sum corresponding to the groups under each characteristic respectively;

the splitting gain comparison module is used for interacting with equipment of a second party according to a multi-party safety comparison protocol, determining the maximum splitting gain based on the first fragmentation of the splitting gain corresponding to each group under each characteristic, and recording the splitting information of the node according to the characteristic and the group corresponding to the maximum splitting gain;

the left and right subtree vector sharding obtaining module is used for: when the maximum splitting gain corresponds to a feature of a first party, generating a left sub-tree vector and a right sub-tree vector of the node, the left sub-tree vector indicating samples in a left subset obtained by dividing the sample set according to the feature corresponding to the maximum splitting gain and groups, the right sub-tree vector indicating samples in a right subset obtained by dividing the sample set according to the feature corresponding to the maximum splitting gain and groups, the left subset corresponding to the left sub-tree, and the right subset corresponding to the right sub-tree; splitting the left sub-tree vector into a first slice and a second slice, and sending the second slice of the left sub-tree vector to a device of a second party; splitting the right sub-tree vector into a first slice and a second slice, and sending the second slice of the right sub-tree vector to a device of a second party; receiving, from a device of a second party, a first tile of a left sub-tree vector and a first tile of a right sub-tree vector of the node when the maximum splitting gain corresponds to a feature of the second party;

the child node marker vector fragment calculation module is configured to: interacting with a device of a second party according to a multi-party secure computing protocol to compute a first shard of a landmark vector of the node based on the first shard of the landmark vector of the node and the first shard of the left sub-tree vector; interacting with a device of a second party according to a multi-party secure computation protocol to compute a first shard of a landmark vector of the node based on the first shard of the landmark vector of the node and the first shard of the right subtree vector;

the child node gradient vector segment calculation module is configured to: interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a first gradient vector of a left sub-tree of the node based on the first tile of the first gradient vector of the node and a first tile of a landmark vector of the left sub-tree of the node; interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a second gradient vector of a left sub-tree of the node based on the first tile of the second gradient vector of the node and the first tile of the landmark vector of the left sub-tree of the node; interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a first gradient vector of a right sub-tree of the node based on the first tile of the first gradient vector of the node and a first tile of a landmark vector of the right sub-tree of the node; interacting with a device of a second party in accordance with a multi-party secure computing protocol to compute a first tile of a second gradient vector of a right sub-tree of the node based on the first tile of the second gradient vector of the node and the first tile of the landmark vector of the right sub-tree of the node.

One of the embodiments of the present specification provides a two-way decision tree training apparatus, which includes a processor and a storage device, where the storage device is used to store instructions, and when the processor executes the instructions, the two-way decision tree training apparatus implements a two-way decision tree training method according to any embodiment of the present specification.

Drawings

The present description will be further explained by way of exemplary embodiments, which will be described in detail by way of the accompanying drawings. These embodiments are not intended to be limiting, and in these embodiments like numerals are used to indicate like structures, wherein:

FIG. 1 is a schematic diagram of an application scenario of a model training system according to some embodiments of the present description;

FIG. 2 is a schematic diagram of a tree model of parties A and B and their corresponding equivalent models, shown in accordance with some embodiments of the present description;

FIG. 3 is a schematic diagram of inputs and variable initialization for two-way decision tree training in accordance with some embodiments of the present description;

4-6 are exemplary flow diagrams of node splitting shown in accordance with some embodiments of the present description;

FIG. 7 is an exemplary flow diagram illustrating the computation of a shard of leaf node weights for an equivalent model according to some embodiments of the present description;

FIG. 8 is an exemplary flow diagram illustrating computing a patch of gradient vectors for training a next tree in accordance with some embodiments of the present description;

FIG. 9 is a schematic diagram of partitioning left and right subsets, according to some embodiments of the present description;

FIG. 10 is a block diagram of a two-way decision tree training system in accordance with some embodiments of the present description.

Detailed Description

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings used in the description of the embodiments will be briefly described below. It is obvious that the drawings in the following description are only examples or embodiments of the present description, and that for a person skilled in the art, the present description can also be applied to other similar scenarios on the basis of these drawings without inventive effort. Unless otherwise apparent from the context, or otherwise indicated, like reference numbers in the figures refer to the same structure or operation.

It should be understood that "system", "device", "unit" and/or "module" as used herein is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, other words may be substituted by other expressions if they accomplish the same purpose.

As used in this specification, the terms "a", "an" and/or "the" are not intended to be inclusive of the singular, but rather are intended to be inclusive of the plural, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements.

Flow charts are used in this description to illustrate operations performed by a system according to embodiments of the present description. It should be understood that the preceding or following operations are not necessarily performed in the exact order in which they are performed. Rather, the various steps may be processed in reverse order or simultaneously. Meanwhile, other operations may be added to the processes, or a certain step or several steps of operations may be removed from the processes.

First, the relevant knowledge of the decision tree is introduced.

The nodes of the decision tree may be divided into split nodes having child nodes (e.g., left and right sub-trees) and leaf nodes (leaf nodes), and each split node may correspond to a feature, which may be referred to as an associated feature of the split node. The parameters of a split node (which may be referred to as split parameters) may include a threshold for determining to which child node a sample/prediction object belongs, the threshold being related to the associated feature of the split node, e.g., a certain feature value of the associated feature of the split node may be used as the threshold.

The decision tree based model (i.e., tree model) may include one or more decision trees, wherein the tree model including a plurality (two or more) of decision trees includes a tree model under an XGB (eXtreme Gradient Boosting) framework.

For the regression problem, each leaf node may correspond to a score (which may be referred to as a leaf node score/leaf node weight), the weights of all leaf nodes on a single decision tree constitute a leaf node weight vector of the decision tree, and the weight of a leaf node (which may be referred to as a predicted leaf node) where a predicted object arrives along a predicted path on the decision tree may be obtained based on the inner product of the leaf node weight vector of the decision tree and the predicted leaf node vector of the predicted object corresponding to the decision tree. The predicted leaf node vector of the predicted object corresponding to the decision tree indicates the weight of the leaf node reached by the predicted object along the predicted path on the decision tree, and it can be understood that the bit number (dimension) of the predicted leaf node vector corresponding to a single decision tree is consistent with the number of leaf nodes of the decision tree, and usually the predicted leaf node vector position 1 corresponding to the leaf node reached by the predicted object along the predicted path on the decision tree and the rest of the predicted leaf node vector positions 0 are used, so that the weight of the predicted leaf node is equal to the inner product of the leaf node weight vector and the predicted leaf node vector. Further, one tree model may correspond to one base score. Then, the prediction score of the predicted object (e.g., sample) may be calculated

Calculating, where pred denotes the prediction score of the predicted object, f₀Representing the base score of the tree model, T (being ≧ 1) representing the number of decision trees contained by the tree model, W_tLeaf node weight vector, S, representing a single decision tree_tRepresenting the correspondence of a single decision treeA predicted leaf node vector of the predicted object,

indicating that the vector inner product is calculated.

In some embodiments, the prediction score of the prediction object may be used as the prediction value of the prediction object. In other embodiments, the prediction score of the predicted object may be processed using a non-linear function (e.g., sigmoid function), the output of which is the predicted value of the predicted object.

In some embodiments, a single decision tree may be trained as a tree model, with the prediction score of a predicted object being equal to the sum of the weights of the leaf nodes that the predicted object reaches along the predicted path on the decision tree and the base score of the tree model. In some embodiments, for example, under an XGB (eXtreme Gradient Boosting) framework, a plurality of decision trees may be trained as a tree model, and a prediction score of a prediction object is equal to a sum of a weight of a leaf node of the prediction object arriving along a prediction path on the plurality of decision trees and a base score of the tree model.

As mentioned previously, a split node may be split into left and right subtrees based on some feature value of some feature. Accordingly, starting from the root node, the samples in the sample set are divided into various child nodes until the samples are divided into leaf nodes, at which point the predicted scores of the samples may be determined based on the base scores of the decision tree and the weights of the leaf nodes at which the samples arrive. The splitting gains corresponding to the splitting of the same splitting node according to different characteristic values of different characteristics are different, so that the splitting gain corresponding to which characteristic value of which characteristic a node is split is better judged according to the magnitude of the splitting gain corresponding to different characteristic values of different characteristics respectively in the training process.

In particular, the splitting gain may reflect a decreasing value of the objective function when a node is split by a certain eigenvalue of a certain characteristic. The splitting (or training) of the decision tree aims to make the objective function value after splitting smaller than the objective function value before splitting as much as possible and make the difference between the two as large as possible, so that the feature and the feature value corresponding to the larger splitting gain can be selected to split the node in the training process. The objective function is derived based on at least the loss function (reflecting the difference between the predicted value and the tag value) of each sample in the sample set, e.g. the objective function may be the sum of the loss functions of each sample. The loss function can be further described by using a first gradient and a second gradient, the corresponding objective function is equivalent to the sum of sub-objective functions respectively corresponding to each leaf node on a single decision tree, and the sub-objective function corresponding to each leaf node can be obtained based on the first gradient sum and the second gradient sum corresponding to the leaf node. Wherein the first gradient is a first order gradient related to a loss function of the sample, the second gradient is a second order gradient related to a loss function of the sample, the first gradient sum is a sum of first gradients of the samples belonging to the corresponding node, and the second gradient sum is a sum of second gradients of the samples belonging to the corresponding node. It is understood that the original node of the split and the left and right subtrees of the split are regarded as leaf nodes at different time instants when the split gain is calculated, i.e. the leaf nodes regarded as the decision tree are increased by one.

When a certain node is split, two left and right subtrees can grow under an original node, and the gain of the objective function can be interpreted as a reduction value of an objective function value corresponding to the split decision tree relative to an objective function value corresponding to the original decision tree. Combining the splitting gain, the objective function, the sub-objective functions, and the relationship between the gradient sums, the splitting gain can be obtained based on the first gradient sum and the second gradient sum corresponding to the left sub-tree and the first gradient sum and the second gradient sum corresponding to the right sub-tree.

For more details on the splitting gain, reference may be made to the related description below.

FIG. 1 is a schematic diagram of an application scenario of a model training system according to some embodiments of the present description. As shown in fig. 1, the system 100 may include an a-party device 110, a B-party device 120, a third party server 130, and a network 140.

And the characteristic value and the label value of one or more characteristics of each sample in the sample set are vertically distributed on the A side and the B side. The vertical distribution refers to that data (such as characteristic values of characteristics of a sample and tag values) are distributed in different databases according to data types (such as different characteristics of height, age and the like, and tag values). As an example, party a holds a feature value of at least one feature (e.g., height) of each sample in the set of samples and a label value of each sample in the set of samples, and party B holds a feature value of at least one feature (e.g., age) of each sample in the set of samples. It is understood that the features, feature values, and tag values held by party a belong to party a's privacy and the features and feature values held by party B belong to party B's privacy.

During the two-party model training process, neither party a nor party B wishes to expose their private data to the other. In order to protect data privacy of both parties, input (such as label values of samples) and output (such as gradient vectors, predicted scores of samples, leaf node weights and the like) of a plurality of computing links involved in a training process are all stored in equipment of both parties in a slicing mode, and the party A and the party B respectively execute one slice.

For the decision tree, the tree models trained by the a-party and the B-party may have the same structure, such as the number of nodes, the connection relationship between the nodes, the positions of the nodes, and the like. However, the tree models of party a and party B have different parameters. For convenience of illustration, the description often refers to equivalent models corresponding to a tree model of party a and a tree model of party B, which can be obtained by performing centralized training based on sample data held by party a and party B, respectively. The equivalent model has complete parameters, and the parameters of the tree model of the A side/B side are equivalently split from the parameters of the equivalent model.

Specifically, the method comprises the following steps: the tree model of any party only has the parameters of partial split nodes, namely, the tree model of any party comprises local split nodes with the parameters and non-local split nodes without the parameters; the weight of a leaf node in any one of the tree models is equivalent to a fragment of the weight of a corresponding leaf node in the equivalent model, the basic score of any one of the tree models is equivalent to a fragment of the basic score of the equivalent model, and from the vector perspective, the leaf node weight vector of any one of the tree models is equivalent to a fragment of the leaf node weight vector of the equivalent model.

Referring to FIG. 2, the split nodes are represented by circles and the leaf nodes are represented by rectangles. Party a holds feature X1 and party B holds feature X2, and accordingly, the tree model for party a has local split nodes and parameters p1 corresponding to feature X1 and the tree model for party B has local split nodes and parameters p2 corresponding to feature X2. As shown in fig. 2, the local split node of the a-side can be denoted as (X1, p1), the local split node of the B-side can be denoted as (X2, p2), the leaf nodes can be denoted as leaf, and the leaf nodes in the same position have the same number. Taking a binary tree as an example, the parameters of a split node may include a threshold associated with a feature corresponding to the node, e.g., the parameters of a node corresponding to a feature of age may include a threshold for distinguishing age groups.

As shown in fig. 2: for party A, the weight of leaf1 is w₁₁The weight of leaf2 is w₂₁The weight of leaf3 is w₃₁The basic score of the tree model is< f₀>₁(ii) a For the B-side, the weight of leaf1 is w₁₂The weight of leaf2 is w₂₂The weight of leaf3 is w₃₂The basic score of the tree model is< f₀>₂(ii) a For the equivalent model, the weight of leaf1 is w₁The weight of leaf2 is w₂The weight of leaf3 is w₃The basic score is f₀. In some embodiments, w may be satisfied₁₁+ w₁₂= w₁、w₂₁+ w₂₂= w₂、w₃₁+ w₃₂= w₃And< f₀>₁+< f₀>₂= f₀. From the leaf node weight vector's perspective, satisfy (w)₁₁, w₂₁, w₃₁)+ (w₁₂, w₂₂, w₃₂)= (w₁, w₂, w₃)。

Assume that the predicted leaf node vector of the prediction object is(s)₁,s₂,s₃) Then predicting the predicted score of the object

Wherein

I.e. leaf node weight vector (w)₁, w₂, w₃) And predicting leaf node vectors(s)₁,s₂,s₃) The inner product of (d). Assuming leaf2 is a leaf node, the leaf node vector for the prediction object is (0,1,0), and the prediction score for the prediction object is (0,1,0)

。

When both the tree model and the equivalent model include T

When the tree is decided, for the equivalent model, each tree corresponds to an inner product of a leaf node weight vector and a predicted leaf node vector, and the inner products corresponding to the T trees respectively and the basic score of the equivalent model are summed to obtain the predicted score of the predicted object.

Devices 110/120 may include various types of computing devices with information transceiving capabilities, such as smart phones, laptop computers, desktop computers, servers, and the like.

In some embodiments, the servers may be independent servers or groups of servers, which may be centralized or distributed. In some embodiments, the server may be regional or remote. In some embodiments, the server may execute on a cloud platform. For example, the cloud platform may include one or any combination of a private cloud, a public cloud, a hybrid cloud, a community cloud, a decentralized cloud, an internal cloud, and the like.

The third party server 130 may assist the a-party device 110 and the B-party device 120 in running a two-party secure multiplication protocol. Multiplication is often involved in the two-party model training process, when one factor of the product belongs to the private data of the A party and the other factor belongs to the private data of the B party, the device of any party cannot directly calculate the product, and the product can interact with the computing device of the other party according to the two-party safe multiplication protocol to obtain one fragment of the product based on the private data calculation of the own party. That is, party a and party B each obtain one slice of the product.

Network 140 connects the various components of the system so that communication can occur between the various components. The network between the various parts in the system may include wired networks and/or wireless networks. For example, network 140 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Public Switched Telephone Network (PSTN), a bluetooth network, a ZigBee network (ZigBee), Near Field Communication (NFC), an intra-device bus, an intra-device line, a cable connection, and the like, or any combination thereof. The network connection between each two parts may be in one of the above-mentioned ways, or in a plurality of ways.

Fig. 3-8 are exemplary flow diagrams of a two-way decision tree training method according to some embodiments described herein.

First, the representation in fig. 3 to 8 will be explained: (1) the symbol { } represents a vector, a matrix or a set, and when the dimension of the vector/set is M or a certain dimension of the matrix is M, it indicates that the M dimensions correspond to M samples in a sample set one by one, for convenience of description, the M samples may be represented by numbers 1 to M, and the dimensions of the vector, the matrix and the set are corresponding to the numbers of the M samples one by one; (2) 3-8 have a central dotted line, the steps indicated in the rounded boxes to the left of the dotted line can be performed independently by party A, the steps indicated in the rounded boxes to the right of the dotted line can be performed independently by party B, and the steps in the rounded boxes passing through the dotted line are performed cooperatively by party A and party B and each obtain a slice of sliced data; (3) the feature is denoted as j and the feature value is denoted as k, and since each group corresponds to one feature value, the group can also be denoted as k.

As shown in FIG. 3, the input held by party A includes the feature data of the M samples, and the feature data of each sample includes N_AThe feature value of each feature, party a, also holds the label values of the M samples. Wherein the storage form of the characteristic data of the M samples held by the party AMay be M N_AFeature matrix of dimension (denoted as { X _ A)_ijAnd X _ a contains the feature values of the M samples for the a-side feature), and the storage form of the tag values of the M samples held by the a-side can be a tag vector (denoted by { y } dimension) in M × 1 dimensions_iY denotes the label value of the sample). The input held by the party B comprises the characteristic data of the M samples, and the characteristic data of each sample comprises N_BThe feature value of each feature, and the storage form of the feature data of M samples held by the B party may be M × N_BA feature matrix of the dimension. The feature data and the label values of the M samples owned by the party a and the party B may be aligned in advance, for example, the sample numbers of the feature data and the label values of the two parties are in one-to-one correspondence or the same, and the one-to-one correspondence or the same sample number corresponds to the same sample.

The initialization phase in figure 3 shows the preparation steps that parties a and B do for the splitting phase.

The device on the side of the initialization phase A can calculate the basic score (denoted as f) of the equivalent model₀) Initialization is performed. In some embodiments, party a may average the label values of the M samples, and use the average as the base score of the equivalent model. It is worth noting that the base score of the equivalent model can be used as the initial sample prediction value, and the A-side device can calculate the first gradient vector { g ] of the root node based on the initial sample prediction value and the sample label value_iH and a second gradient vector h_i}. The first gradient vector of any node comprises a first gradient g corresponding to the samples belonging to the node, and the second gradient vector of any node comprises a second gradient h corresponding to the samples belonging to the node. It will be appreciated that the number of bits (dimension) of the first/second gradient vector may be M, each bit corresponding to one of the M samples, the gradient vector bit corresponding to a sample not belonging to the respective node being 0. It should be noted that although the sample i does not belong to a node, the element corresponding to the sample i in the first/second gradient vector of the node is derived from the first gradient g corresponding to the sample i_iSecond gradient h_iBecomes 0, but the element in the first/second gradient vector of the node corresponding to sample i is still available as g for ease of description_i/ h_iTo indicate.

For example, the first/second gradient vector of the first split root node includes the first/second gradients corresponding to the M samples, and it is not assumed that M =4, and then the first gradient vector of the root node may be denoted as (g)₁, g₂, g₃, g₄) The second gradient vector of the root node may be denoted as (h)₁, h₂, h₃, h₄). If the root node is split according to a certain eigenvalue of a certain characteristic, such that samples numbered 1 and 4 are divided into the left sub-tree of the root node, and samples numbered 2 and 3 are divided into the right sub-tree of the root node, the first gradient vector and the second gradient vector of the left sub-tree of the root node are (g) respectively₁,0,0, g₄)、(h₁,0,0, h₄) The first gradient vector and the second gradient vector of the right subtree of the root node are (0, g) respectively₂, g₃,0)、(0, h₂, h₃,0)。

Initialization phase A-side equipment may be directed to a token vector of nodes s_iAnd initializing, wherein the mark vector of the initialized node is the mark vector of the root node. The flag vector for any node indicates the samples belonging to that node. It will be appreciated that the number of bits (dimension) of the flag vector for a node may be M, each bit corresponding to one of the M samples, typically the flag vector bit for the sample belonging to the respective node is 1 and the flag vector bit for the sample not belonging to the respective node is 0. For example, the flag vector bits of the root node are all 1. If the root node is split according to a certain eigenvalue of a certain characteristic such that samples numbered 1 and 4 are divided into the left subtree of the root node and samples numbered 2 and 3 are divided into the right subtree of the root node, the flag vector of the left subtree of the root node is (1,0,0,1) and the flag vector of the right subtree of the root node is (0,1,1, 0).

To avoid divulgence, the device of party a in the initialization phase may assign a base score f of the equivalent model₀First gradient vector of root node { g_iH, a second gradient vector of the root node_i}, a flag vector of root nodes { s_i}, label vector y_iAnd splitting the two fragments into two fragments allocated to the two parties respectively. Should understand thatThe splitting of the vector or matrix comprises splitting each element of the vector or matrix, i.e. each element of the vector or matrix is also split into two slices allocated to both parties. Likewise, encrypting/decrypting the vector or matrix includes encrypting/decrypting each element of the vector or matrix. As shown in fig. 3, subscript a corresponds to the partition allocated to party a, subscript B corresponds to the partition allocated to party B, and a device on party a needs to allocate to a partition on party B<f₀>_B、{<g_i>_B}、{<h_i>_B}、{<s_i>_B}、{<y_i>_BSending to B-party's equipment and reserving at least the fragment allocated to itself<f₀>_A、{<g_i>_A}、{<h_i>_A}、{<s_i>_A}、{<y_i>_A}. Wherein,<f0>_Acan be used as an initial score for the tree model for party a,<f0>_Bcan be used as an initial score of the tree model of party B.

It should be noted that, for a tree model under an XGB (eXtreme Gradient Boosting) framework, the initialized first/second Gradient vector is the first/second Gradient vector of the root node of the first trained tree, and the flag vector bits of the root node of each tree are all 1.

Fig. 4 to 6 respectively show three links involved in (decision tree) node splitting in the training of the two-sided model: calculating the fragmentation of the fragmentation gain, comparing the fragmentation gain and recording the fragmentation information. These three links are described in turn below.

Referring to fig. 4, fig. 4 illustrates a flow of computing a slice of a gradient sum and computing a slice of a splitting gain based on the slice of the gradient sum by taking a B-side feature as an example.

For any feature j of the B-party, the device of the B-party may sort the samples in the sample set { i } according to the feature value of the feature j (for example, sort the samples according to the feature value from small to large or from large to small), and may obtain a corresponding permutation vector after sorting, which may be referred to as a first permutation vector, and is used in fig. 4

And (4) showing. It will be appreciated that for any feature j of party a, the device of party a may order the samples in the sample set { i } and obtain a corresponding permutation vector according to the same method as described above, and the permutation vector obtained by party a may be referred to as a second permutation vector. The permutation vector may be used to identify operations that sort long sequences (i.e., sequences of equal length to the permutation vector), with elements in the permutation vector indicating positions in the sorted result sequence of data corresponding to positions in the sorted equal length sequences. As an example, vectors { a, b, a, c, b } are ordered in ascending dictionary to obtain vectors { a, a, b, b, c } and ordered vectors

Is {0,2,1,4,3},

where 0 denotes that the position of the 0 th element in the arranged equal-length sequence in the sorting result sequence is 0 th bit, 2 denotes that the position of the 1 st element in the arranged equal-length sequence in the sorting result sequence is 2 nd bit, and

other elements

1,4, and 3 in the arrangement vector are also similar identifiers. Based on permutation vector

{0,2,1,4,3} is arranged for another sequence of equal length { b, c, a, c, b }, so that the corresponding sequence of ordering results { b, a, c, b, c } is obtained.

After the permutation vector is obtained, the first/second gradient vectors may be permuted by a secure permutation method based on the permutation vector to obtain a first/second gradient ordering result vector, and it can be understood that the first/second gradient vectors are permuted based on the first permutation vector, that is, the first/second gradient vectors are ordered according to the feature value of the feature j. The secure permutation method is a method for securely permuting a sequence in a secret manner, and a secret sharing sequence can be permuted in a secret manner by using a permutation vector private to one of a plurality of parties (for example, party B), and an obtained sequencing result is still in a secret sharing form (the method can also be called as an inadvertent permutation). The security permutation method can be further described as an ObliviousPerm function or operator:

，<>the representation of the ciphertext form, which may specifically be in the form of a slice,

which represents the sequence to be arranged,

representing the sequence of the ordering result. An ObliviousPerm function may be understood as a function whose set of input data includes a permutation function

And a first fragment of the sequence to be arranged

From a participant whose other set of input data comprises a second slice of the sequence to be arranged

From another party; the output data of which is a first slice comprising a sequence of ordered results

And a second section

The first fragment of the result sequence is obtained by the one participant and the second fragment of the ordered result sequence is obtained by the other participant. At present, various implementation modes of the ObliviousPerm function exist, and the description does not limit the internal implementation algorithm and only calls the ObliviousPerm function as a black box operator. It should be understood that, all the ways that the data processing/computing unit, the program code, the machine learning model and the like capable of implementing the obreviousperm function can be mentioned as the present specification, which are present at present and will appear in the futureBased on a secret sharing arrangement protocol.

Taking a first participant and a second participant in cooperation as an example, where the first participant holds a target permutation vector and a first slice of a data column to be sorted, and the second participant holds a second slice of the data column to be sorted, an implementation manner of the obreviousserver function may include:

a first participant obtains, from a trusted third party, a first alignment vector, a first tile of a first data column, and a first tile of a first sequence of results that orders the first data column based on the first alignment vector. The second participant obtains a second slice of the first data column and a second slice of the first sequence of results from the trusted third party.

The first participant determines a second permutation vector based on the target permutation vector and the first permutation vector and sends it to the second participant; in some embodiments, the target permutation vector is ordered based on the first permutation vector, and the resulting sequence may be the second permutation vector.

The first participant makes a difference between the first fragment of the data column to be sorted and the first fragment of the first data column to obtain the first fragment of the second data column, and simultaneously obtains the second fragment of the second data column from the second party; the second slice of the second data column is obtained by the second participant subtracting the second slice of the data column to be sorted from the second slice of the first data column.

The first participant obtains a second data column based on the first fragment of the second data column and the second fragment of the second data column, and sorts the second data column based on the target permutation vector to obtain a second result data column; sorting the first slices of the first result data column based on a second permutation vector to obtain a third result data column; and finally, summing the second result data column and the third result data column to obtain a first fragment of the target data column.

The second participant orders the second slices of the first result sequence based on the second permutation vector to obtain second slices of the target data column. The target data column is equal to a result sequence of sorting the data columns to be sorted based on the target permutation vector.

First gradient vector g_iH and a second gradient vector h_iThe first gradient vector { g } is distributed in the A-side device and the B-side device in a slicing mode, and based on the distribution, the first gradient vector { g } is obtained_iArranging by way of example, instructions or procedures implemented by an inadvertent arrangement operator may include: b-party fragment based on first gradient vector<g_i>_BAnd the first alignment vector

And party A fragment a last based on the first gradient vector<g_i>_ASecret sorting is carried out, and then a first gradient sorting result vector is obtained by a B party

First segment of

While the party A obtains the second section

. For the second gradient vector h_iArranging, and partitioning by B square based on second gradient vector<h_i>_BAnd the first alignment vector

And party A fragment the last page based on the second gradient vector<h_i>_ASecret sorting is carried out, and then a second gradient sorting result vector is obtained by the B party

First segment of

While the party A obtains the second section

。

For the features of party A, party B tiles based on the first gradient vector<g_i>_BAt a predetermined distance from the origin of the image, and party A sharding based on a first gradient vector<g_i>_AThe second permutation vector and the first gradient ordering result vector are obtained by the B side

First segment of

While the party A obtains the second section

. For the second gradient vector h_iArranging, and partitioning by B square based on second gradient vector<h_i>_BAnd party A is based on second gradient vector piece-<h_i>_ACarrying out secret sorting on the obtained data and a second arrangement vector, and then obtaining a second gradient sorting result vector by the B party

First segment of

While the party A obtains the second section

。

Thus, for the current node, the A party and the B party obtain the first/second gradient sorting result vector fragments corresponding to all the features.

For the first/second gradient sorting result vector fragments of the B-party device and the first/second gradient sorting result vector fragments of the a-party device, the B-party device and the a-party device may divide elements in their respective local vector fragments according to a preset frequency (e.g., 10 are a group) (i.e., an equal frequency binning or equal width binning algorithm), the B-party device may obtain a plurality of first packets of the fragments of the first/second gradient vectors, and the a-party device may obtain a plurality of second packets of the fragments of the first/second gradient vectors. It can be understood that the vector lengths of the first/second gradient vector fragment of the party a and the first/second gradient vector fragment of the party B are the same, the number of the obtained groups is the same, and the elements at the same position in each group correspond to the same sample in the sample set. For convenience of description, the number of the divided packets is denoted as K, and the number of the K packets is 1-K. In some embodiments, a threshold corresponding to each group may also be determined based on the feature value of the sample in the group (e.g., equal to the mean of the feature values, or equal to the feature value of a certain sample in the group), and the threshold corresponding to the group may be used as a splitting parameter according to which a subsequent node is split.

For the B-side device and the A-side device, each group corresponds to a fragment of a first gradient sum and a fragment of a second gradient sum respectively, the fragment of the first gradient sum corresponding to any group is the sum of elements corresponding to samples belonging to the first group in the fragment of the first gradient vector, and the fragment of the second gradient sum corresponding to any group is the sum of elements corresponding to samples belonging to the first group in the fragment of the second gradient vector.

To this end, for the current node, the party a and the party B obtain the slice of the first gradient sum and the slice of the second gradient sum of all the groups of the features.

For any node, the A party and the B party can interact according to a multi-party security computation protocol, so that the fragmentation of the splitting gain corresponding to each group under each characteristic is computed based on the fragmentation of the first gradient sum and the fragmentation of the second gradient sum corresponding to each group under each characteristic, and the A party and the B party respectively execute one fragmentation of the same splitting gain. Wherein, since each group corresponds to one eigenvalue, each splitting gain actually corresponds to one eigenvalue k of one eigenvalue and one eigenvalue j. And the splitting gain corresponding to the characteristic j and the characteristic value k of the characteristic j reflects the descending value of the objective function after the corresponding node is split according to the characteristic j and the characteristic value k of the characteristic j.

It was mentioned before that the splitting gain can be based on the left sub-tree first gradient sum (denoted as G)_L) The second gradient sum of the left subtree (denoted as H)_L) Right subtree first gradient sum(note as G)_R) Right subtree second gradient sum (denoted as H)_R) It is obtained that if the equivalent model is trained on a device on one side (not marked as C), referring to FIG. 4, the device on the C side can initialize G_L、H_L、G_R、H_RAccording to the first/second gradient sum G of the left subtree_L/H_LThe first/second gradient sum G corresponding to a certain group (denoted as k) is accumulated_k/H_kThe first/second gradient sum G from the right subtree respectively_R/H_RUp-subtract the first/second gradient sum G corresponding to the group k_k/H_kIn the method, the gradient and G of the left sub-tree corresponding to each group under the feature j (i.e. corresponding to each eigenvalue of the feature j) are obtained in turn_LAnd the right subtree gradient and G_R. For any group, the sum of the first gradient sum of the A-side device and the first gradient sum of the B-side device is the first gradient sum G corresponding to the group_kThe sum of the slice of the second gradient sum of the A-party device and the slice of the second gradient sum of the B-party device is the second gradient sum H corresponding to the group_k。

As shown in fig. 9, the sample set is divided into K groups according to the feature value of the feature j, and since the samples are sorted according to the feature value of the feature j before the grouping result is obtained, the K groups present a certain order according to the feature value of the samples in each group, and the increase of the feature value of the samples in the group is not represented by the increment of the group sequence numbers (1-K) (indicated by a horizontal arrow in fig. 9), the feature value of any sample in the 2 nd group is not less than the feature value of any sample in the 1 st group, the feature value of any sample in the 3 rd group is not less than the feature value of any sample in the 2 nd group. It can be seen that if the grouping is taken as the minimum unit, and the sample set is divided into a left subset corresponding to the left subtree and a right subset corresponding to the right subtree, there are K division cases (indicated by vertical arrows in fig. 9), in other words, there are K splitting possibilities for the node to be split under the feature j, where each splitting possibility corresponds to one feature value (i.e., a feature value corresponding to each grouping).

The device at the side C needs to calculate the K possible splitting gains, that is, the splitting gains corresponding to the K groups under the characteristic j, in order to determine the optimal splitting parameter of the node to be split. With combined reference to FIGS. 4 and 9, taking the first gradient as an example, the C-side device may first sum the first gradient of the left sub-tree with G_LInitialize to 0, and sum G of the first gradient of the right subtree_RInitialise to the first gradient sum G of the node to be split. The first gradient sum is a sum of elements in the first gradient vector corresponding to samples belonging to the respective node. Then, the device on the side C can sequentially obtain the first gradient and G of the left subtree corresponding to the K groups respectively in an "increasing-decreasing" manner_LAnd the right subtree first gradient sum G_R: grouping the 1 st gradient into corresponding G₁Add to left subtree first gradient sum G_L（G_L= 0), the left subtree gradient and G corresponding to the 1 st group are obtained_L（G_L=0+G₁) And, summing the first gradient of the right subtree with G_R（G_R= G) minus the first gradient sum G corresponding to the 1 st group₁Obtaining the first gradient sum G of the right subtree corresponding to the 1 st group_R（G_R=G-G₁) (ii) a Grouping the 2 nd corresponding first gradients and G₂Add to left subtree first gradient sum G_L（G_L=0+G₁) Obtaining the gradient sum G of the left subtree corresponding to the 2 nd group_L（G_L=0+G₁+G₂) And, summing the first gradient of the right subtree with G_R（G_R=G-G₁) Subtract the first gradient sum G corresponding to the 2 nd group₂Obtaining the first gradient sum G of the right subtree corresponding to the 2 nd group_R（G_R=G-G₁-G₂) (ii) a ...; and the like until the first gradient sum G of the left subtree corresponding to the kth group is obtained_L（G_L= G) and the first gradient sum G of the right subtree_R（G_R= 0). Of course, G can be calculated for the above_L、G_RIs suitably deformed, e.g. by summing the first gradient of the left sub-tree with G_LInitialising to a first gradient sum G of nodes to be split, and rightSubtree first gradient sum G_RThe initialization is 0. Accordingly, it is necessary to be at G_LSequentially subtracting the first gradient sums respectively corresponding to the K groups to obtain the first gradient sums respectively corresponding to the left subtrees of the K groups, and obtaining the first gradient sums of the left subtrees of the K groups in G_RAnd sequentially adding first gradient sums corresponding to the K groups respectively to obtain first gradient sums corresponding to the K groups respectively in the right subtree. It should be understood that G on both sides of the equation_LG on both sides of equation for the first gradient sum of the left subtrees corresponding to the adjacent groups_RThe first gradient sum is for the corresponding right subtree of the adjacent group.

Similarly, the C-side device may obtain the left sub-tree second gradient sum and the right sub-tree second gradient sum corresponding to the K groups under the feature j.

Respectively corresponding first gradient and G of left sub-tree based on K groups under characteristic j_LRight subtree first gradient and G_RLeft subtree second gradient and H_LRight subtree second gradient and H_RThe device on the C side may calculate the splitting gains Gain corresponding to the K groups under the feature j, respectively.

It should be noted that some binning algorithms, such as equal frequency binning, may cause samples with unequal feature values to appear in adjacent groups, resulting in the left/right subtree gradient sum (G) calculated as described above_L、G_R、H_L、H_R) There is some error, but such error is negligible in engineering terms.

The above describes how to compute the left/right subtree gradients and the splitting gain by training the equivalent model on one side (not denoted as C) of the device, and on the basis of this, we will continue to describe how to break the above computation process on the two sides of the device.

First, the initialization of the gradient sum of the left and right subtrees to sum G_LInitialization to 0 and G_RInitialization to G for example_L=0, the a-party's device may generate a random number as initialized G_LIs divided into<G_L>_AAnd will be<G_L>_ASent to the B-party's device, which can calculate the satisfaction<G_L>_A+<G_L>_B=0<G_L>_BOr, satisfaction can be generated by a third party device<G_L>_A+<G_L>_B=0<G_L>_AAnd<G_L>_Band divide the pieces into pieces<G_L>_ADevice to send to party A and to fragment<G_L>_BThe device sent to the B-party, or the A-party device and the B-party device are initialized locally<G_L>_A=0、<G_L>_BAnd = 0. For G_R= G, the device on the a side can locally calculate a patch of the first gradient vector &<g_i>_AThe sum of the elements in the gradient is obtained, and the first gradient and the fragment of G are obtained<G>_AAs initialized G_RIs divided into<G_R>_AAnd, the B-party's device may locally compute a fragment of the first gradient vector<g_i>_BThe sum of the elements in the gradient is obtained, and the first gradient and the fragment of G are obtained<G>_BAs initialized G_RIs divided into<G_R>_B. Similarly, the party A device may locally compute a fragment of the second gradient vector<h_i>_AThe sum of the elements in the gradient is obtained to obtain a second gradient and H slices<H>_AH as initialization_RIs divided into<H_R>_AAnd, the B-party's device may locally compute a fragment of the first gradient vector<h_i>_BThe sum of the elements in the gradient is obtained to obtain a second gradient and H slices<H>_BH as initialization_RIs divided into<H_R>_B。

Then, the calculation of the gradient sums corresponding to the K groups respectively is carried out, and the A side/B side equipment can be used as follows by taking the first gradient as an example<G_L>_A=<G_L>_A +<G_k>_AAnd<G_R>_A=<G_R>_A -<G_k>_Aand (4) performing iterative computation to obtain the fragments of the first gradient sum of the left sub-tree and the fragments of the first gradient sum of the right sub-tree corresponding to the K groups respectively.Similarly, the A/B devices may press<H_L>_A=<H_L>_A+<H_k>_AAnd<H_R>_A=<H_R>_A -<H_k>_Aand (4) performing iterative computation to obtain the fragments of the second gradient sum of the left sub-tree and the fragments of the second gradient sum of the right sub-tree corresponding to the K groups respectively.

Obtaining G_L、G_R、H_L、H_RAfter fragmentation, the a-side/B-side device may calculate a fragmentation of the fragmentation gain. In some embodiments, the sub-goal functions corresponding to the nodes may be as follows:

。

wherein,

represents a preset coefficient, G represents the sum of the first gradients of all samples under the node, and H represents the sum of the second gradients of all samples under the node. Correspondingly, for the node, after dividing the samples belonging to the node into the left and right subtrees based on the feature value corresponding to a certain group under a certain feature, the calculation formula of the splitting Gain is as follows:

。

wherein, when difference comparison is carried out on different splitting gains Gain

As a constant, the constant is cancelled, so that the constant in the Gain can be split during actual operation

Are ignored. In addition, since the splitting of data in this specification is performed based on addition, the calculation formula before splitting is equivalent in all places where the data is split into piecesIn Taylor form. E.g. constants in the calculation of the split gain here

On the basis of neglect, the first and second pairs

Taylor expansion is performed to obtain an equivalent calculation of the splitting gain as shown in FIG. 4 as follows:

。

the subscript k of Gain may be used as an identifier of a corresponding group, or may be used as a feature value corresponding to the group. Inputs in the above equations (e.g. G)_L) Output (i.e. output)

) Both contain two splits, one for each of party a and party B. After the input slices are substituted into the above calculation formula, the expansion formula involves the calculation of two types of product terms: one type can be called as a local product term, and two factors of the local product term belong to the fragment data of the same party (party A/party B), so that the calculation can be independently completed by equipment of one party; another type may be referred to as a cross product term, where two factors of the cross product term belong to the fragmented data of different parties, i.e. one factor belongs to the fragmented data of party a and the other factor belongs to the fragmented data of party B, so to protect the data privacy of both parties, party a and party B may calculate two fragments of the cross product term by running a two-party secure multiplication protocol, and party a and party B each execute one fragment to obtain an output fragment. For example,

wherein

and

for the local product term, Party A can compute locally

The B party can compute locally

，

For the cross product term, the A side and the B side can calculate the fragment of the cross product term through a two-side safe multiplication protocol.

As mentioned previously, the packet is traversed by traversing the packet under the same feature (the packet is identified by the feature value k in FIG. 4) and traversing (N)_A+ N_B) A feature (denoted j in FIG. 4) is obtained for each of parties A and B (N)_A+ N_B) Grouping the first gradient sums together and (N)_A+ N_B) And grouping the second gradient sums, wherein each group of the first/second gradient sums comprises the number of the sections equal to the number K of the groups under the corresponding characteristic. Further, each of the parties A and B will obtain (N)_A+ N_B) The set of split gain slices, each set of split gain slices comprising a number of slices equal to the number of groups under the corresponding feature, K. Wherein the number K of packets under different characteristics may be different.

Referring to fig. 5, fig. 5 shows a comparison of the magnitude of the splitting gain corresponding to each packet under each feature. The index j of Gain indicates a feature and the index k indicates a group (and also indicates the corresponding feature value of the group). By traversing the feature pair j1, j2 and the packet pair k1, k2, i.e. pair (N)_A+ N_B) And comparing the splitting gains in the component splitting gains in pairs, and selecting the characteristic corresponding to the maximum splitting gain and the component to split. Where k1 is the eigenvalue of signature j1, k2 is the eigenvalue of signature j2, j1 and j2 may be the same signature, but the groupings under that signature k1, k2 are different groupings. Based on the blockWhen the splitting gain is obtained based on the equivalent calculation formula of the splitting gain shown in fig. 4, it can be understood that the larger the splitting gain is, the more suitable the corresponding feature and the splitting threshold (or the feature value) are as the splitting condition of the node.

Because both the A side and the B side only have Gain_j1,k1、Gain_j2,k2The party a and the party B can compare the magnitude of the two split gains through a multi-party security comparison protocol. After defining the same feature j1 and group k2 corresponding to the two split gains that are compared, as shown in fig. 5, the device on the a-side can calculate<v>_A=<Gain_j1,k1>_A-<Gain_j2,k2>_AThe B-party's device can calculate<v>_B=<Gain_j2,k2>_B-<Gain_j1,k1>_B. Further, the device of party A and the device of party B can interact according to a multi-party security comparison protocol so as not to reveal<v>_AAnd<v>_Bon the premise of determining the specific numerical value of<v>_AAnd<v>_Bthe magnitude relationship of (1). If it is<v>_AGreater than (or not less than)<v>_BThen the feature j1 is "retained" along with the feature value k 1. Otherwise, feature j2 and the feature value k2 are "retained". And comparing the two sides with the next characteristic and the characteristic value thereof based on the reserved characteristic and the fragment of the characteristic value, and repeating the steps until the comparison of the splitting gains corresponding to all the characteristics and the characteristic values is finished, and finally, the reserved characteristic and the fragment of the characteristic value thereof of the two sides correspond to the optimal splitting condition of the node. Of course, it may also be the A-party's device computing<v>_A=<Gain_j2,k2>_A -<Gain_j1,k1>_AAnd device calculation of B party<v>_B=<Gain_j1,k1>_B-<Gain_j2,k2>_B. Accordingly, if<v>_AGreater than (or not less than)<v>_BThen preferentially splitting the node to be split according to the characteristic j2 and the characteristic value k 2. Otherwise, the node to be split is split according to the characteristic j1 and the characteristic value k 1.

Referring to fig. 6, for any node (denoted as X), when the splitting parameter (the feature and the feature value corresponding to the maximum splitting gain) of node X is determined, only one of the parties a and B records the splitting parameter, because node X splits according to only one feature (of one party). Assuming that the determined splitting parameters are j1, k1, and feature j1 is a feature for party a, only the device for party a can record the splitting parameters for node X. For example only, as shown in fig. 6, the splitting information split (X, j1, k1) recorded on the a-side indicates that the splitting parameters of the node X are own-side feature j1 and feature value k1, and the splitting information split (X, dummy, dummy) recorded on the B-side indicates that the node X is a non-local splitting node, i.e., the parameters of the node X in the tree model on the B-side are unknown. It is understood that the a-party and the B-party can respectively determine whether each node is a local split node and determine (split) parameters of the local split node according to the recorded split information of each node.

It should be appreciated that with respect to node splitting, parties a and B may have some consensus: the method includes the steps that 1, an A party and a B party can define the structure of a tree model to be trained, such as the number of decision trees, the number of nodes of each tree, the connection relation among the nodes, the positions of the nodes, the depth and the like, correspondingly, under some scenes, the A party and the B party can determine whether current operation aims at the same decision tree or the same node, for example, the A party and the B party can carry out uniform identification (such as numbering) on each decision tree in the tree model, and the A party and the B party can carry out uniform identification (such as numbering) on each node of the same decision tree; the party a and the party B can agree on the identity of the features of both parties and the identity of each group under each feature without disclosure, for example, assuming that the features of the party a include age and height, and the features of the party B include distance and orientation, the age and height can be identified by a1 and a2, and the distance and orientation can be identified by B1 and B2, so that the party B only knows that a1 and a2 are two features of the party a, and the party a also knows that B1 and B2 are two features of the party B, and specifically, the party a and the party B can determine whether the obtained splitting gain corresponds to the same feature and group (feature value) or not and determine which feature the splitting gain corresponds to which one through the feature identity and the group identity.

Still taking the example that the node X is split according to the feature j1 and the feature value k1 of the a-side, the device of the a-side needs to further divide the sample set { i } of the M samples into a left subset corresponding to the left subtree and a right subset corresponding to the right subtree according to the magnitude relationship between the feature value of the feature j1 and the feature value k1 of the M samples, so that the two parties can obtain the shards of the first/second gradient vectors and the shards of the flag vectors of the child nodes (i.e., the left and right subtrees) of the node X. If the child node continues to split, the first/second gradient vector segment and the flag vector segment of the child node may be used for the calculation related to the splitting of the child node, and the specific details may refer to fig. 4 to 6 and the related description thereof. If the child node does not continue to split, i.e. becomes a leaf node, the a-party and the B-party may cooperatively calculate two segments of the leaf node weight based on the segments of the first/second gradient vectors of the child node (i.e. the leaf node), and each of the a-party and the B-party may execute one segment, and specific details may refer to fig. 8 and its related description.

As shown in FIG. 6, the device of party A may generate a left sub-tree vector for node X

And right subtree vector

Wherein, the left sub-tree vector

Indicating samples in a left subset, right sub-tree vectors, obtained by dividing a sample set according to features and feature values corresponding to a maximum splitting gain

And indicating the samples in the right subset obtained by dividing the sample set according to the characteristic and the characteristic value corresponding to the maximum splitting gain. It will be appreciated that the left/right sub-tree vector may have a number of bits (dimension) of M, each bit corresponding to a sample, typically a left subset vector position 1 and the remaining left subset vector positions 0 corresponding to samples belonging to the left subset, and similarly a right subset vector position 1 and the remaining right subset vector positions corresponding to samples belonging to the right subsetThe subset vector positions are all at 0. Since the samples belong to either the left or right subset, satisfy

. The device of party A will be the left sub-tree vector of node X

Splitting into slices

And slicing

To divide into pieces

Sending to B-party's device and retaining (at least) the slice

. Similarly, the right subtree vector of node X

Divided into fragments by party A

And slicing

To divide into pieces

Storing in prescription A, slicing

Stored in prescription B.

Following the description above, taking the left sub-tree as an example, assume that the token vector { s ] for the left sub-tree of node X is computed on the C-party device_i,L}, first gradient vector { g_i,LH, a second gradient vector h_i,L}，As shown in fig. 6, the device of party C may: calculating a token vector s for node X_i,XAnd left sub-tree vector

The result of the bit-wise multiplication results in a flag vector s of the left sub-tree of node X_i,L}; compute a first gradient vector { g) for node X_i,X} and the flag vector s of the left subtree of node X_i,LThe result of the bit-wise multiplication results in a first gradient vector g of the left sub-tree of node X_i,L}; computing a second gradient vector { h) for node X_i,X} and the flag vector s of the left subtree of node X_i,LThe result of the bit-wise multiplication results in a second gradient vector h of the left sub-tree of node X_i,L}。

Similarly, the C-side device can calculate the flag vector { s ] of the right subtree of the node X_i,R}, first gradient vector { g_i,RH and a second gradient vector h_i,R}。

Referring to the foregoing related description, taking the left sub-tree as an example, the device of the a-party and the device of the B-party may interact according to a multi-party security computing protocol to: node X based token vector s_i,XSlice and left sub-tree vector of }

Computing a token vector s of the left sub-tree of node X_i,LSlicing of the points; first gradient vector g based on node X_i,XThe slice of node X and the flag vector s of the left sub-tree of node X_i,LH, compute a first gradient vector g for the left sub-tree of node X_i,LSlicing of the points; second gradient vector h based on node X_i,XThe slice of node X and the flag vector s of the left sub-tree of node X_i,LH, computing a second gradient vector h of the left sub-tree of node X_i,LAnd (4) slicing. For example, mixing s_i,X=<s_i,X>_A+<s_i,X>_BAnd

substitution into

To obtain

Its development is as follows:

。

wherein,

can be calculated locally on A side and used as output fragment of A side<s_i,X>_AA part of (a) of (b),

can be locally calculated at the B party and used as output fragments of the B party<s_i,X>_BA part of, cross product terms

Can be calculated by two-party secure multiplication protocol, obtained by party A

As output shards of party A<s_i,X>_AA part of, obtained by party B

As output slices of party B<s_i,X>_BA part of (a).

For a single decision tree, the splitting of each node can be performed in sequence according to the splitting links shown in fig. 4 to 6 until the growth termination condition is satisfied. It should be understood that growth termination means that no child nodes are again split on a single tree, i.e. all leaf nodes on a single tree have been obtained. In some embodiments, the growth termination condition may include a depth of the single tree reaching a preset depth.

With reference to figure 7 of the drawings,figure 7 shows a method of computing a slice of leaf node weights. For the same leaf node obtained by training, the A-side device may calculate a first gradient vector { g } for that leaf node_iQuick check<g_i>_AThe sum of the elements in the leaf node, the first gradient of the leaf node and the slice G of G_AThe B-party's device may compute the first gradient vector { g for the leaf node_iQuick check<g_i>_BThe sum of the elements in the leaf node, the first gradient of the leaf node and the slice G of G_B. Similarly, the device on the A-side may compute a second gradient vector { h) for the leaf node_iQuick check<h_i>_AThe sum of the elements in the leaf node, the slice H of the second gradient sum H of the leaf node_AThe B-side device may calculate a second gradient vector { h) for the leaf node_iQuick check<h_i>_BThe sum of the elements in the leaf node, the slice H of the second gradient sum H of the leaf node_B. In some embodiments, the leaf node weights are calculated as follows:

wherein w represents a leaf node weight, G represents a first gradient sum of the leaf node, H represents a second gradient sum of the leaf node,

representing a preset coefficient.

It should be noted that, since the splitting of data is based on addition, the slice w of the leaf node weight w_AAnd w_BCan be calculated by disassembling the Taylor expansion type

Is obtained, wherein, w_AWeight, w, as leaf node in the tree model for the A-party_BAs the weight of the same leaf node in the tree model of party B. Party A may be based on the first gradient of the leaf node and the slice G of G_ASecond gradient and fragmentation of H_AAnd B may beSlice G of the first gradient and G at the leaf node_BSecond gradient and fragmentation of H_BCalculating to obtain a first fragment of the weight of the leaf node (which can correspond to w of the A party) according to the leaf node weight calculation formula by a two-party secure multiplication protocol_A) And a second segment (which may correspond to w of party B)_B). Regarding the disassembly of the calculation formula related to the multiplication, many descriptions have been given in the foregoing embodiments of the present specification, and will not be described herein and hereafter.

Under the XGB framework, the tree model and the equivalent model of any party comprise a plurality of trees. Referring to FIG. 8, when each of parties A and B completes the training of a tree, the slice of the M samples' prediction scores may be updated to calculate a first gradient vector { g ] for the root node of the next tree_iSlice of and second gradient vector h_iAnd (4) slicing. In FIG. 8, pred_i,tRepresents the weight of the leaf node (i.e. the leaf node to which the sample i belongs, also called the predicted leaf node of the sample i) reached by the sample i (i.e. the sample numbered i) along the predicted path in the t-th tree in the equivalent model. For the t-th tree, the tree,

wherein s is_i,nToken vector s representing the leaf node numbered n in the t-th tree in the equivalent model_i,nElement in } corresponding to sample i, N_tRepresenting the number of leaf nodes of the t-th tree. The weights of the predicted leaf nodes of M samples in the t-th tree of the equivalent model may form a predicted weight vector { pred }_i,t}。

The device of party A and the device of party B may interact according to a multiparty security computing protocol to base the token vectors { s) of the leaf nodes of the tth tree_i,nSlice of the t tree and leaf node weight vector of t tree in the equivalent model w_nCompute pred (i.e., leaf node weight vector of the t-th tree in the tree model of A/B), and_i,tto be divided into pieces. Further, as shown in FIG. 8, the A-party device may pre_i,tIs divided into<pred_i,t>_AThe prediction score pred added to the current sample i_iIs divided into<pred_i>_ATo update the prediction score pred of sample i_iIs divided into<pred_i>_A. It will be appreciated that the prediction score pred of the current sample i_iIs the sum of the base score of the equivalent model and the weight of the predicted leaf nodes of sample i on the first t-1 trees of the equivalent model. In particular, the prediction score pred of sample i_iIs the basic score f of the equivalent model₀I.e. fragmentation of party A<pred_i>_AIs the initial score of the tree model of party A<f0>_AFragmentation of party B<pred_i>_BIs the initial score of the tree model of party B<f0>_B. The prediction scores for the M samples may constitute a prediction score vector { pred }_iAccordingly, the A-side device may obtain an updated prediction score vector { pred }_iQuick check<pred_i>_AThe B-side device can obtain an updated prediction score vector { pred }_iQuick check<pred_i>_B}。

In some embodiments, the predicted values for the samples may be derived based on the predicted scores for the samples and the non-linear activation function, as described above. Taking sigmoid as an example of a nonlinear activation function, the predicted value of the sample

Where pred represents the prediction score for that sample. Accordingly, since the splitting of data is based on addition, taylor expansion can be adopted

. For two-party training, the device of party A may press

Update the slice of the predicted value of sample i, the B-side device can be as follows

And updating the slice of the predicted value of the sample i.

In some embodiments of the present invention, the,loss function of sample

From this, a first gradient corresponding to the sample i can be derived

And a second gradient corresponding to sample i

. For two-party training, the device of party A may press

Updating the fragment of the first gradient corresponding to the sample i used for training the next tree, thereby obtaining the fragment of the first gradient vector of the root node of the next tree<g_i>_A}. Similarly, the B-party's device may press

Updating the fragment of the first gradient corresponding to sample i to obtain a fragment of the first gradient vector of the root node of the next tree to be trained<g_i>_B}. In view of

The decomposition of (a) involves cross product terms, and the device of party A and the device of party B can interact according to a multi-party safety calculation protocol to base the predicted value of the updated sample i

And a tag value y_iCalculating the fragment of the second gradient corresponding to the sample i used for training the next tree, thereby obtaining the fragment of the second gradient vector of the root node of the next tree<h_i>_A}。

It should be noted that the above description of the flow is for illustration and description only and does not limit the scope of the application of the present specification. Various modifications and alterations to the flow may occur to those skilled in the art, given the benefit of this description. However, such modifications and variations are intended to be within the scope of the present description.

FIG. 10 is a block diagram of a two-way decision tree training system in accordance with some embodiments of the present description. The system 200 may be implemented on a device of a first party, which may be either of the a-party and the B-party, the second party being the other of the a-party and the B-party. Wherein the eigenvalues and label values of one or more characteristics of each sample in the set of samples are distributed vertically on the A side and the B side. As shown in fig. 10, system 200 may include a first obtaining module 202, a first ordering module 204, a first packet gradient and fragmentation computation module 206, a second ordering module 208, a second packet gradient and fragmentation computation module 210, a split gain fragmentation computation module 212, a split gain comparison module 214, a left and right sub-tree vector fragmentation obtaining module 216, a sub-node landmark vector fragmentation computation module 218, and a sub-node gradient vector fragmentation computation module 220.

For any node (denoted as X) that is split, the functions of the modules in system 200 are as follows:

the first obtaining module 202 may be configured to obtain a first slice of the landmark vector of node X, a first slice of the first gradient vector, and a first slice of the second gradient vector. Wherein the flag vector indicates samples belonging to the respective node, the first gradient vector comprises a first gradient corresponding to the samples belonging to the respective node, and the second gradient vector comprises a second gradient corresponding to the samples belonging to the respective node.

For any feature held by the first party, the first ranking module 204 may be to: sorting the sample set according to the characteristic value of the characteristic and obtaining a first arrangement vector, wherein the arrangement vector is used for identifying the operation of sorting the equal-length sequence, and the element of the arrangement vector indicates the position of the data corresponding to the position in the equal-length sequence in a sorting result sequence; based on a first fragment of a first gradient vector, the first arrangement vector and a second fragment of a second party based on the first gradient vector, arranging the first gradient vector according to the characteristic value by a safe arrangement method to obtain a first fragment of the first gradient arrangement result vector, and simultaneously obtaining a second fragment of the first gradient arrangement result vector by the second party; and arranging the second gradient vector according to the characteristic value by a safe arrangement method based on the first fragment of the second gradient vector, the first arrangement vector and the second fragment of the second party based on the second gradient vector to obtain the first fragment of the second gradient arrangement result vector, and simultaneously obtaining the second fragment of the second gradient arrangement result vector by the second party.

For any feature held by the first party, the first packet gradient and fragmentation computation module 206 may be to: dividing elements in a first fragment of the first gradient sorting result vector and elements in a first fragment of the second gradient sorting result vector according to a preset frequency respectively to obtain a plurality of first groups of the first fragment of the first gradient vector and a plurality of first groups of the first fragment of the second gradient vector, and for each of the plurality of first groups: calculating the sum of elements included in the first group of the first segment of the first gradient vector to obtain a first segment of a first gradient sum corresponding to the first group; and calculating the sum of elements included in the first group of the first segment of the second gradient vector to obtain a first segment of a second gradient sum corresponding to the first group.

For any feature held by the second party, the second ranking module 208 may be configured to: arranging the first gradient vector according to the characteristic value of the characteristic by a safe arrangement method based on the first fragment of the first gradient vector and the second fragment and the second arrangement vector of the second party based on the first gradient vector to obtain the first fragment of the first gradient ordering result vector, and simultaneously obtaining the second fragment of the first gradient ordering result vector by the second party; based on the first fragment of the second gradient vector, and the second fragment and the second arrangement vector of the second party based on the second gradient vector, arranging the second gradient vector according to the characteristic value by a safe arrangement method to obtain the first fragment of the second gradient ordering result vector, and simultaneously obtaining the second fragment of the second gradient ordering result vector by the second party; and the second arrangement vector is obtained by ordering the sample set by a second party according to the characteristic value of the characteristic.

For any feature held by the second party, the second packet gradient and sharding calculation module 210 may be operable to: dividing elements in a first slice of the first gradient sorting result vector and elements in a first slice of the second gradient sorting result vector according to a preset frequency number respectively to obtain a plurality of second sub-groups of the first slice of the first gradient vector and a plurality of second sub-groups of the first slice of the second gradient vector, and for each of the plurality of second sub-groups: calculating the sum of the elements included in the second grouping of the first patches of the first gradient vector to obtain first patches of the first gradient sum corresponding to the second grouping; and calculating the sum of the elements included in the second grouping of the first patches of the second gradient vector to obtain the first patches of the second gradient sum corresponding to the second grouping.

The split gain slice calculation module 212 may be configured to interact with a device of the second party according to a multi-party security calculation protocol, so as to calculate a first slice of a split gain corresponding to each packet under each feature based on a first slice of a first gradient sum and a first slice of a second gradient sum corresponding to each packet under each feature.

The splitting gain comparison module 214 may be configured to interact with a device of the second party according to a multi-party security comparison protocol, so as to determine a maximum splitting gain based on a first segment of the splitting gain corresponding to each packet under each feature, and record splitting information of the node according to the feature and the packet corresponding to the maximum splitting gain.

Left and right subtree vector shard obtaining module 216 may be configured to: when the maximum splitting gain corresponds to a feature of a first party, generating a left sub-tree vector and a right sub-tree vector of the node, the left sub-tree vector indicating samples in a left subset obtained by dividing the sample set according to the feature corresponding to the maximum splitting gain and groups, the right sub-tree vector indicating samples in a right subset obtained by dividing the sample set according to the feature corresponding to the maximum splitting gain and groups, the left subset corresponding to the left sub-tree, and the right subset corresponding to the right sub-tree; splitting the left sub-tree vector into a first slice and a second slice, and sending the second slice of the left sub-tree vector to a device of a second party; splitting the right sub-tree vector into a first slice and a second slice, and sending the second slice of the right sub-tree vector to a device of a second party; receiving, from a device of a second party, a first tile of a left sub-tree vector and a first tile of a right sub-tree vector of the node when the maximum splitting gain corresponds to a feature of the second party.

The child node flag vector shard calculation module 218 may be configured to: interacting with a device of a second party according to a multi-party secure computing protocol to compute a first shard of a landmark vector of the node based on the first shard of the landmark vector of the node and the first shard of the left sub-tree vector; interacting with a device of a second party in accordance with a multi-party secure computation protocol to compute a first tile of a landmark vector of a right subtree of the node based on the first tile of the landmark vector of the node and the first tile of the right subtree vector.

The child gradient vector shard calculation module 220 may be configured to: interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a first gradient vector of a left sub-tree of the node based on the first tile of the first gradient vector of the node and a first tile of a landmark vector of the left sub-tree of the node; interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a second gradient vector of a left sub-tree of the node based on the first tile of the second gradient vector of the node and the first tile of the landmark vector of the left sub-tree of the node; interacting with a device of a second party according to a multi-party secure computing protocol to compute a first tile of a first gradient vector of a right sub-tree of the node based on the first tile of the first gradient vector of the node and a first tile of a landmark vector of the right sub-tree of the node; interacting with a device of a second party in accordance with a multi-party secure computing protocol to compute a first tile of a second gradient vector of a right sub-tree of the node based on the first tile of the second gradient vector of the node and the first tile of the landmark vector of the right sub-tree of the node.

For more details of the system 200 and its modules, reference may be made to fig. 3-8 and their associated description.

It should be understood that the system and its modules shown in FIG. 10 may be implemented in a variety of ways. For example, in some embodiments, the system and its modules may be implemented in hardware, software, or a combination of software and hardware. Wherein the hardware portion may be implemented using dedicated logic; the software portions may be stored in a memory for execution by a suitable instruction execution system, such as a microprocessor or specially designed hardware. Those skilled in the art will appreciate that the methods and systems described above may be implemented using computer executable instructions and/or embodied in processor control code, such code being provided, for example, on a carrier medium such as a diskette, CD-or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The system and its modules in this specification may be implemented not only by hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but also by software executed by various types of processors, for example, or by a combination of the above hardware circuits and software (e.g., firmware).

It should be noted that the above description of the system and its modules is for convenience only and should not limit the present disclosure to the illustrated embodiments. It will be appreciated by those skilled in the art that, given the teachings of the system, any combination of modules or sub-system configurations may be used to connect to other modules without departing from such teachings. For example, in some embodiments, the first packet gradient and fragmentation computation module 206 and the second packet gradient and fragmentation computation module 210 may be different modules in a system or may be a single module that implements the functionality of both modules. For another example, the first obtaining module 202, the first arranging module 204, the first packet gradient and slice calculation module 206, the second arranging module 208, the second packet gradient and slice calculation module 210, the splitting gain slice calculation module 212, and the splitting gain comparison module 214 may be extracted from the training system shown in fig. 10 and combined into a node splitting system. Such variations are within the scope of the present disclosure.

The beneficial effects that may be brought by the embodiments of the present description include, but are not limited to: (1) the two-party decision tree training method is provided, so that the data privacy of the two parties can be protected; (2) the model effect can be improved by training with the sample data of the two parties; (3) the method comprises the steps of sorting the fragments of the first/second gradient vectors of the A party and the B party by adopting an inadvertent arrangement method, dividing the fragments into a plurality of groups based on a preset frequency, calculating the fragments of the first/second gradient sum corresponding to each group of the fragments of the first/second gradient vectors by the A party and the B party respectively locally, calculating the splitting gain corresponding to each group by the two parties according to a multi-party safety calculation protocol, determining a splitting mode based on the splitting gain corresponding to each group, splitting a tree, reducing interaction of the two parties in a two-party decision tree training process, and greatly reducing the operation amount and communication traffic. It is to be noted that different embodiments may produce different advantages, and in different embodiments, any one or combination of the above advantages may be produced, or any other advantages may be obtained.

Having thus described the basic concept, it will be apparent to those skilled in the art that the foregoing detailed disclosure is to be considered merely illustrative and not restrictive of the embodiments herein. Various modifications, improvements and adaptations to the embodiments described herein may occur to those skilled in the art, although not explicitly described herein. Such modifications, improvements and adaptations are proposed in the embodiments of the present specification and thus fall within the spirit and scope of the exemplary embodiments of the present specification.

Also, the description uses specific words to describe embodiments of the description. Reference throughout this specification to "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic described in connection with at least one embodiment of the specification is included. Therefore, it is emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment. Furthermore, some features, structures, or characteristics of one or more embodiments of the specification may be combined as appropriate.

Moreover, those skilled in the art will appreciate that aspects of the embodiments of the present description may be illustrated and described in terms of several patentable species or situations, including any new and useful combination of processes, machines, manufacture, or materials, or any new and useful improvement thereof. Accordingly, aspects of embodiments of the present description may be carried out entirely by hardware, entirely by software (including firmware, resident software, micro-code, etc.), or by a combination of hardware and software. The above hardware or software may be referred to as "data block," module, "" engine, "" unit, "" component, "or" system. Furthermore, aspects of the embodiments of the present specification may be represented as a computer product, including computer readable program code, embodied in one or more computer readable media.

The computer storage medium may comprise a propagated data signal with the computer program code embodied therewith, for example, on baseband or as part of a carrier wave. The propagated signal may take any of a variety of forms, including electromagnetic, optical, etc., or any suitable combination. A computer storage medium may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code located on a computer storage medium may be propagated over any suitable medium, including radio, cable, fiber optic cable, RF, or the like, or any combination of the preceding.

Computer program code required for operation of various portions of the embodiments of the present description may be written in any one or more programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C + +, C #, VB.NET, Python, and the like, a conventional programming language such as C, VisualBasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, a dynamic programming language such as Python, Ruby, and Groovy, or other programming languages, and the like. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or processing device. In the latter scenario, the remote computer may be connected to the user's computer through any network format, such as a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet), or in a cloud computing environment, or as a service, such as a software as a service (SaaS).

In addition, unless explicitly stated in the claims, the order of processing elements and sequences, use of numbers and letters, or use of other names in the embodiments of the present specification are not intended to limit the order of the processes and methods in the embodiments of the present specification. While various presently contemplated embodiments of the invention have been discussed in the foregoing disclosure by way of example, it is to be understood that such detail is solely for that purpose and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements that are within the spirit and scope of the embodiments herein. For example, although the system components described above may be implemented by hardware devices, they may also be implemented by software-only solutions, such as installing the described system on an existing processing device or mobile device.

Similarly, it should be noted that in the preceding description of embodiments of the specification, various features are sometimes grouped together in a single embodiment, figure, or description thereof for the purpose of streamlining the disclosure and aiding in the understanding of one or more embodiments of the invention. This method of disclosure, however, is not intended to imply that more features are required than are expressly recited in the claims. Indeed, the embodiments may be characterized as having less than all of the features of a single embodiment disclosed above.

For each patent, patent application publication, and other material, such as articles, books, specifications, publications, documents, etc., cited in this specification, the entire contents of each are hereby incorporated by reference into this specification. Except where the application history document does not conform to or conflict with the contents of the present specification, it is to be understood that the application history document, as used herein in the present specification or appended claims, is intended to define the broadest scope of the present specification (whether presently or later in the specification) rather than the broadest scope of the present specification. It is to be understood that the descriptions, definitions and/or uses of terms in the accompanying materials of this specification shall control if they are inconsistent or contrary to the descriptions and/or uses of terms in this specification.

Finally, it should be understood that the embodiments described herein are merely illustrative of the principles of the embodiments of the present disclosure. Other variations are possible within the scope of the embodiments of the present description. Thus, by way of example, and not limitation, alternative configurations of the embodiments of the specification can be considered consistent with the teachings of the specification. Accordingly, the embodiments of the present description are not limited to only those embodiments explicitly described and depicted herein.

Claims

1.A node splitting method in a two-party decision tree training process is used for splitting a certain node in a decision tree, wherein the characteristic value and the label value of one or more characteristics of each sample in a sample set are vertically distributed in the two parties, the method is executed by equipment of a first party, the first party is either one of the two parties, and a second party is the other one of the two parties; the method comprises the following steps:

obtaining a first slice of a first gradient vector and a first slice of a second gradient vector of the node; the first gradient vector comprises a first gradient corresponding to samples belonging to the respective node, and the second gradient vector comprises a second gradient corresponding to samples belonging to the respective node;

for any feature of the first party:

for any feature of the second party:

and interacting with equipment of a second party according to a multi-party safety comparison protocol, determining the maximum splitting gain based on the first fragments of the splitting gains respectively corresponding to the groups under the characteristics, and recording the splitting information of the nodes according to the characteristics and the groups corresponding to the maximum splitting gain.

2. The method of claim 1, further comprising:

obtaining a first slice of a token vector for the node, the token vector indicating samples belonging to a respective node;

interacting with a device of a second party according to a multi-party secure computing protocol to compute a first shard of a landmark vector of the node based on the first shard of the landmark vector of the node and the first shard of the left sub-tree vector; interacting with a device of a second party in accordance with a multi-party secure computation protocol to compute a first tile of a landmark vector of a right subtree of the node based on the first tile of the landmark vector of the node and the first tile of the right subtree vector.

3. The method of claim 2, further comprising:

4. The method of claim 1, wherein the predetermined division comprises division at a predetermined frequency.

5. The method of claim 1, wherein interacting with the device of the second party according to the multi-party secure computation protocol to compute the first fraction of the splitting gain corresponding to each packet under each feature based on the first fraction of the first gradient sum and the first fraction of the second gradient sum corresponding to each packet under each feature comprises:

for either feature:

calculating first slices of first gradient sums of left subtrees and first slices of first gradient sums of right subtrees respectively corresponding to the groups under the characteristics based on first slices of first gradient sums respectively corresponding to the groups under the characteristics; wherein the left sub-tree first gradient sum is equal to the sum of the elements corresponding to the samples belonging to the left subset in the first and second slices of the first gradient vector, and the right sub-tree first gradient sum is equal to the sum of the elements corresponding to the samples belonging to the right subset in the first and second slices of the first gradient vector;

calculating a first fragment of a second gradient sum of a left sub-tree and a first fragment of a second gradient sum of a right sub-tree corresponding to each group under the characteristic respectively based on a first fragment of a second gradient sum corresponding to each group under the characteristic respectively; wherein the left sub-tree second gradient sum is equal to the sum of the elements corresponding to the samples belonging to the left subset in the first and second slices of the second gradient vector, and the right sub-tree second gradient sum is equal to the sum of the elements corresponding to the samples belonging to the right subset in the first and second slices of the second gradient vector;

for any grouping of any feature:

interacting with a device of a second party according to a multi-party secure computing protocol

A first slice of the splitting gain corresponding to the packet under the feature is calculated, wherein,

representing the corresponding splitting gain, G, of the packet under the characteristic_LThe sum of the first and second patches representing the first gradient sum of the left subtree corresponding to the group under the feature, G_RThe sum of the first and second slices, H, representing the first gradient sum of the right subtree corresponding to the group under the feature_LThe sum of the first and second slices, H, representing the sum of the second gradients of the left subtree corresponding to the group under the feature_RA sum of the first tile and the second tile representing a second gradient sum of the right subtree corresponding to the group under the feature,

representing a preset coefficient.

6. The method of claim 1, wherein the secure ordering method comprises:

a first participant obtaining, from a third party, a first intermediate permutation vector, a first tile of a first data column, and a first tile of a first sequence of results ordering the first data column based on the first intermediate permutation vector; the second participant acquires a second segment of the first data column and a second segment of the first result sequence from the trusted third party;

the first participant orders the first intermediate permutation vector based on the target permutation vector to obtain a second intermediate permutation vector, and sends the second intermediate permutation vector to the second participant;

the first participant makes a difference between the first fragment of the data column to be sorted and the first fragment of the first data column to obtain the first fragment of the second data column, and simultaneously obtains the second fragment of the second data column from the second party; the second fragment of the second data column is obtained by the second participant subtracting the second fragment of the data column to be sorted from the second fragment of the first data column;

the first participant obtains a second data column based on the first fragment of the second data column and the second fragment of the second data column, and sorts the second data column based on the target permutation vector to obtain a second result data column; sorting the first slices of the first resulting data column based on a second intermediate permutation vector to obtain a third resulting data column; summing the second result data column and the third result data column to obtain a first fragment of the target data column;

the second participant sorts the second slice of the first result sequence based on the second intermediate permutation vector to obtain a second slice of the target data column; the target data column is equal to a result sequence of sorting the data columns to be sorted based on the target permutation vector;

when a first segment based on a first gradient vector, the first arrangement vector and a second segment based on the first gradient vector of a second party are arranged according to the characteristic value by a safe arrangement method to obtain the first segment of the first gradient arrangement result vector, and simultaneously the second party obtains the second segment of the first gradient arrangement result vector: the first party is used as a first participant in the secure sorting method, a first fragment of a first gradient vector is used as a first fragment of a data column to be sorted in the secure sorting method, a first arrangement vector is used as a target arrangement vector in the secure sorting method, the second party is used as a second participant in the secure sorting method, and a second fragment of the first gradient vector is used as a second fragment of the data column to be sorted in the secure sorting method;

when the first segment based on the second gradient vector, the first arrangement vector and the second segment based on the second gradient vector of the second party are arranged according to the characteristic value by a safe arrangement method to obtain the first segment of the second gradient arrangement result vector, and the second party obtains the second segment of the second gradient arrangement result vector at the same time: the first party is used as a first participant in the secure sorting method, a first fragment of a second gradient vector is used as a first fragment of a data column to be sorted in the secure sorting method, a first arranging vector is used as a target arranging vector in the secure sorting method, the second party is used as a second participant in the secure sorting method, and a second fragment of the second gradient vector is used as a second fragment of the data column to be sorted in the secure sorting method;

when a first segment based on a first gradient vector, a second segment based on the first gradient vector and a second arrangement vector of a second party are arranged on the first gradient vector according to the characteristic value of the characteristic through a safe arrangement method to obtain a first segment of the first gradient ordering result vector, and simultaneously the second party obtains a second segment of the first gradient ordering result vector: the first party is used as a second party in the safe sorting method, a first fragment of a first gradient vector is used as a second fragment of a data column to be sorted in the safe sorting method, the second party is used as a first party in the safe sorting method, the second permutation vector is used as a target permutation vector in the safe sorting method, and a second fragment of the first gradient vector is used as a first fragment of the data column to be sorted in the safe sorting method;

when the first segment based on the second gradient vector, the second segment based on the second gradient vector and the second arrangement vector of the second party are arranged according to the characteristic value by a safe arrangement method to obtain the first segment of the second gradient ordering result vector, and the second party obtains the second segment of the second gradient ordering result vector at the same time: the first party is used as a second party in the safe sorting method, a first fragment of a second gradient vector is used as a second fragment of a data column to be sorted in the safe sorting method, the second party is used as a first party in the safe sorting method, the second permutation vector is used as a target permutation vector in the safe sorting method, and a second fragment of the second gradient vector is used as a first fragment of the data column to be sorted in the safe sorting method.

7. A node splitting system for two-party decision tree training is used for splitting a certain node in a decision tree, wherein characteristic values and label values of one or more characteristics of each sample in a sample set are vertically distributed in the two parties, the system is implemented on equipment of a first party, the first party is either one of the two parties, and a second party is the other one of the two parties; the system comprises a first obtaining module, a first arranging module, a first grouping gradient and fragment calculating module, a second arranging module, a second grouping gradient and fragment calculating module, a splitting gain fragment calculating module and a splitting gain comparing module;

the first obtaining module is configured to obtain a first slice of a first gradient vector and a first slice of a second gradient vector of the node; the first gradient vector comprises a first gradient corresponding to samples belonging to the respective node, and the second gradient vector comprises a second gradient corresponding to samples belonging to the respective node;

the splitting gain comparison module is used for interacting with equipment of a second party according to a multi-party safety comparison protocol, determining the maximum splitting gain based on the first fragments of the splitting gains corresponding to the groups under the characteristics, and recording the splitting information of the nodes according to the characteristics and the groups corresponding to the maximum splitting gain.

8. A node splitting apparatus for two-way decision tree training, comprising at least one storage medium and at least one processor, the at least one storage medium for storing computer instructions; the at least one processor is configured to execute the computer instructions to implement the method of any of claims 1-6.

9. A two-party tree model training method, wherein a tree model comprises one or more decision trees, and the eigenvalues and label values of one or more characteristics of each sample in a sample set are distributed vertically in the two parties, the method is executed by a device of a first party, the first party is either one of the two parties, and the second party is the other one of the two parties; the method comprises the following steps: performing node splitting on each node in a decision tree in the tree model according to the method of any one of claims 1 to 6 until a growth termination condition is met.

10. The method of claim 9, further comprising:

calculating a first fragment of a weight of each leaf node in a single decision tree in an equivalent model as a weight of each leaf node in a corresponding decision tree in a tree model of a first party, wherein the equivalent model has the same structure as the tree model of the first party and the tree model of a second party:

interacting with a device of a second party according to a multi-party secure computing protocol to base a fourth of the first gradient vector of the leaf nodeA slice and a first slice of a second gradient vector based on

Calculating a first fragment of the weight of the leaf node in the equivalent model; wherein,

the sum of the first tile and the second tile representing the weight of the leaf node in the equivalent model,

the sum of the elements in the first tile and the second tile of the first gradient vector representing the leaf node,Hthe sum of the elements in the first tile and the second tile of the second gradient vector representing the leaf node,

representing a preset coefficient.

11. The method of claim 10, further comprising:

for the tth decision tree:

interacting with equipment of a second party according to a multi-party safety calculation protocol, and calculating first fragments of weights of leaf nodes on a tth decision tree in an equivalent model to which each sample belongs on the basis of the weights of the leaf nodes on the tth decision tree in the tree model of the first party and the first fragments of the flag vectors;

for any sample, accumulating the first fragment of the weight of the leaf node on the t decision tree in the equivalent model to which the sample belongs to the first fragment of the predicted score of the sample to update the first fragment of the predicted score of the sample; when t =1, the first slice of the prediction score of the sample before updating is equal to the base score of the tree model of the first party, when

Of the predicted score of the sample before updatingThe first fragment is equal to the sum of the first fragment of the weight of the leaf node of the sample on the first t-1 decision trees in the equivalent model and the basic score of the tree model of the first party;

calculating a first segment of a first gradient vector of a root node of the t +1 decision tree based on the updated first segment of the prediction score of each sample and the updated first segment of the label value of each sample;

interacting with a device of a second party according to a multi-party security computing protocol to compute a first segment of a second gradient vector of a root node of the t +1 th decision tree based on the updated first segment of the predicted score of each sample and the updated first segment of the label value of each sample.