CN113554178A

CN113554178A - Optimizing gradient boost feature selection

Info

Publication number: CN113554178A
Application number: CN202011460925.6A
Authority: CN
Inventors: J.多恩休
Original assignee: SAP SE
Current assignee: SAP SE
Priority date: 2020-04-24
Filing date: 2020-12-11
Publication date: 2021-10-26
Also published as: US11620537B2; EP3901839A1; US20210334667A1; US20230206083A1

Abstract

Gradient boosting decision trees stack many decision trees one after the other, which at each step try to fix the residual error. The final score produced by the gradient boosting decision tree is simply the sum of the individual scores obtained by applying the decision tree to the input vector. Removing the input values from the training data that have the least impact on the output can reduce overfitting in the gradient boosting decision tree. One way to determine which input variable has low predictive value is to determine the input variable that was first used in the most recent decision tree in the gradient boosting decision tree. This method of identifying which low predictive features to delete does not require the regeneration of earlier trees to generate new gradient boosting decision trees. Since deleted features are not used in earlier trees, these trees have ignored deleted features early on.

Description

Optimizing gradient boost feature selection

Technical Field

The subject matter disclosed herein relates generally to feature selection for machine learning. In particular, the present disclosure is directed to systems and methods for improving gradient boost feature selection.

Background

The gradient boosting decision tree predicts a value of a target variable based on one or more inputs. Each node of the first tree divides the input space into two parts based on the value of a single input. During training, the first tree is limited to a maximum depth to prevent overfitting. Rather than increasing the depth to improve the accuracy of the first tree, a second tree is created that predicts the difference between the predicted value (predicted value) from the first tree and the actual value of the variable. This process is repeated until a termination condition is reached. The generated set of trees (ensemble) is used to predict target values for unlabeled data not included in the training dataset.

Drawings

In the figures of the accompanying drawings, some embodiments are shown by way of example and not limitation.

FIG. 1 is a network diagram illustrating a network environment suitable for gradient boost feature selection, according to some example embodiments;

FIG. 2 is a block diagram of a gradient boosting decision tree training server adapted to selectively train a gradient boosting decision tree using gradient boosting features, according to some example embodiments;

FIG. 3 is a diagram showing a distribution of discrete target values with a two-dimensional input variable space;

FIG. 4 is a diagram showing a decision tree for spatial prediction of target values from the two-dimensional input variables of FIG. 3;

FIG. 5 is a diagram illustrating the two-dimensional space of FIG. 3 partitioned using the decision tree of FIG. 4;

FIG. 6 is a graph showing the distribution of residual error in the two-dimensional input variable space of FIG. 3 after application of the decision tree of FIG. 4;

FIG. 7 is a flowchart illustrating operations of a method suitable for training a gradient boosting decision tree according to some example embodiments;

fig. 8 is a block diagram of a database schema (schema) suitable for gradient boost feature selection in accordance with some example embodiments;

FIG. 9 is a flowchart illustrating operations of a method suitable for using gradient boosting feature selection in training a gradient boosting decision tree, according to some example embodiments;

FIG. 10 is a flowchart illustrating operations of a method suitable for using gradient boosting feature selection in training a gradient boosting decision tree, according to some example embodiments;

FIG. 11 is a block diagram illustrating one example of a software architecture for a computing device;

fig. 12 is a block diagram of a machine, illustrated in the form of a computer system, in which instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein.

Detailed Description

Example methods and example systems are directed to gradient boost feature selection. Gradient Boosting Decision Trees (GBDTs) are a very popular technique that can solve classification tasks and regression tasks with excellent accuracy results. Gradient boosting decision trees rely on an integrated approach (ensemble approach) that incorporates many weak estimators (weak estimators). Thus, the classification task or regression task can be solved by breaking the task into smaller sub-problems and then iteratively solving them.

Gradient boosting decision trees stack many decision trees one after the other, which at each step try to fix the residual error. Therefore, as other trees are added to the ensemble (ensemble), the error gradually decreases. Each decision tree partitions the feature space under the direction of optimization of an objective function or a loss function. During the creation of any individual decision tree, the partitioning process stops once a termination condition is met (e.g., a minimum number of partitions are added to the decision tree, a maximum tree depth is reached, a minimum residual error is reached, or any suitable combination of the foregoing). Leaf nodes in the decision tree are assigned scores for classification or regression that minimize the total error of the training data in the corresponding feature space. The final score produced by the gradient boosting decision tree is simply the sum of the individual scores obtained by applying the decision tree to the input vector. In various embodiments, the number of decision trees ranges from less than ten to several thousand. The gradient boosting decision tree error may be measured as a mean square error or a log loss of the training data set or the validation data set alone.

Overfitting is a problem with any regression algorithm, including gradient boosting decision trees. When the algorithm over-fits the data, the error metric of the training data becomes very small, even decreasing to zero, but it seems paradoxical that the error metric of the previously unseen data increases. Overfitting results from the algorithm fully learning the training data set itself and not determining patterns that are broadly applicable to unseen data. One way to reduce overfitting in the gradient boosting decision tree is to remove the input values from the training data that have the least impact on the output to reduce the complexity of the model. For example, an algorithm for predicting the value of a house may be initially trained using a number of input variables including zip code, square feet, number of bedrooms, and current homeowner age. Since the correlation between the homeowner's age and the house's value is low, age is actually an identifier of a particular item, making the model overfitting, but not providing useful predictive power. After determining that the homeowner's age has less impact on the target prediction, this field is removed from the training data and the gradient boosting decision tree is reconstructed. The error of the reconstructed gradient boosting decision tree is compared with the error of the original gradient boosting decision tree, and if the change is small, the original gradient boosting decision tree is discarded, and a new gradient boosting decision tree is used. This process may be repeated until further deletion of the input variables has a substantial effect on the error of the resulting model.

One way to determine which input variable has a low predictive value is to determine the input variable that was first used in the most recent decision tree in the gradient boost decision tree. Since each successive decision tree tries to reduce the error of the aggregated previous tree, the influence of the latest tree on the error of the gradient boosting decision tree is smaller than the influence of any previous tree. Therefore, the predictive value of the unused input variables in the early trees is lower than the predictive value of the used input variables in the early trees. Thus, by storing data in the database indicating which features were used for the first time in each decision tree when the decision tree was created, explicit computation of the predictive value of each feature can be omitted.

When using this method of identifying which low predictive features to delete, there is no need to regenerate the earlier trees to generate new gradient boosting decision trees. Since deleted features are not used in earlier trees, these trees have ignored deleted features early on. Therefore, only the tree (and the trees thereafter) in which the deleted feature first appears need to be regenerated, which saves computation time compared to the method of reconstructing a gradient boosting decision tree starting from the first tree using a reduced set of features. Furthermore, since the first set of trees of the gradient boosting decision tree is reused, the score of this part of the gradient boosting decision tree can be reused without recalculation in order to calculate the error of the next model added with the new tree: this incremental error calculation takes advantage of the cumulative nature of the scores produced by each decision tree.

When these effects are taken together, one or more of the approaches described herein may eliminate the need for certain workloads or resources involved in training or using gradient boosting decision trees. Computing resources used by one or more machines, databases, or networks may be similarly reduced. Examples of such computing resources include processor cycles, network traffic, memory usage, data storage capacity, power consumption, and cooling capacity.

Fig. 1 is a network diagram illustrating a network environment 100 suitable for gradient boost feature selection, according to some example embodiments. Network environment 100 includes a network-based application 110,

client devices

160A and 160B, and a network 190. The web-based application 110 is provided by an application server 120 in communication with a database server 130, stored data 140, and a gradient boosting decision tree 150.

Application server 120 accesses data 140 to provide applications to client device 160A and client device 160B via network interface 180 or application interface 170. The application server 120, data acquisition server 125, database server 130, gradient boosting decision tree training server 135,

client devices

160A and 160B, respectively, may be implemented in whole or in part in a computer system, as described below with reference to fig. 12. Client device 160A and client device 160B may be collectively referred to as a plurality of client devices 160 or generally as client devices 160.

Data acquisition server 125 receives data from one or more data sources. The received data is provided to the gradient boosting decision tree training server 135 to train one or more gradient boosting decision trees. The trained gradient boosting decision tree is transmitted to the database server 130 and stored as the gradient boosting decision tree 150. The application server 120 causes the gradient boosting decision tree to process the data 140 to generate a predicted value. The predicted values are provided by the application server 120 to the client device 160 via the network 190 for display to the user. Additionally or alternatively, the predicted values are used by the application server 120 to automatically control further operations of the application server 120. For example, the credit line of a business may be automatically expanded or denied based on the forecasted values.

Any of the machines, databases, or devices illustrated in fig. 1 may be implemented in a general-purpose computer modified (e.g., configured or programmed) by software to be a special-purpose computer to perform the functions of the machine, database, or device described herein. For example, a computer system capable of implementing any one or more of the methods described herein is discussed below with reference to fig. 11. As used herein, a "database" is a data storage resource and may store data structured as text files, tables, spreadsheets, relational databases (e.g., object relational databases), triple stores, hierarchically structured data stores, document-oriented NoSQL databases, file stores, or any suitable combination thereof. The database may be an in-memory database. Further, any two or more of the machines, databases, or devices shown in fig. 1 may be combined into a single machine, database, or device, and the functions described herein for any single machine, database, or device may be subdivided among multiple machines, databases, or devices.

Application server 120, data acquisition server 125, database server 130, gradient boosting decision tree training server 135, and client devices 160A through 160B are connected via network 190. The network 190 may be any network that enables communication between two or three of a machine, a database, and a device. Thus, the network 190 may be a wired network, a wireless network (e.g., a mobile network or a cellular network), or any suitable combination thereof. Network 190 may include one or more portions that make up a private network, a public network (e.g., the internet), or any suitable combination thereof.

Fig. 2 is a block diagram 200 of a gradient boosting decision tree training server adapted to selectively train a gradient boosting decision tree using gradient boosting features, according to some example embodiments. The gradient boosting decision tree training server 135 is shown to include a communication module 210, a data acquisition module 220, a decision tree construction module 230, a gradient boosting module 240, a feature reduction module 250, and a storage module 260, all configured to communicate with each other (e.g., via a bus, shared memory, or switch). Any one or more of the modules described herein may be implemented using hardware (e.g., a processor of a machine). For example, any of the modules described herein may be implemented by a processor configured to perform the operations described herein for that module. Also, any two or more of these modules may be combined into a single module, and the functionality described herein for a single module may be subdivided among multiple modules. Further, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed among multiple machines, databases, or devices.

The communication module 210 receives data sent to the gradient boosting decision tree training server 135 and sends data from the gradient boosting decision tree training server 135. For example, the communication module 210 receives domain data (e.g., business department data, customer detail data, customer license data, customer meeting data, or any suitable combination thereof) for one or more domains from the data acquisition server 125 and provides the domain data to the data acquisition module 220.

The data acquisition module 220 receives domain data from the data acquisition server 125 or from a plurality of such data acquisition servers. The data acquisition module 220 processes the received domain data into a format suitable for training a gradient boosting decision tree.

Decision tree construction module 230 constructs a separate decision tree using the domain data from the domain. Gradient boosting module 240 causes decision tree construction module 230 to iteratively construct a decision tree to reduce residual errors of the training data set or the validation data set. Thus, the set of decision trees constructed by the decision tree construction module 230 forms a gradient boosting decision tree. Both the training dataset and the validation dataset are datasets of known input features and output values (also referred to as "signature data"). A single signature data set may be divided into a training data set and a validation data set. For example, 20% of the set may be randomly selected as the validation data set and the remaining 80% as the training data set.

Feature reduction module 250 identifies features with low predicted value, removes them from the dataset, and causes decision tree building module 230 and gradient boosting module 240 to form a second gradient boosted decision tree. The feature reduction module 250 controls the iterative process of forming a simpler gradient boosting decision tree until further simplification will result in an error increase beyond a predetermined threshold and roll back the last change. The gradient boosting module 240 provides a trained gradient boosting decision tree through the communication module 210. Additionally or alternatively, the trained gradient boosting decision tree may be stored by the storage module 260 in a database or file system.

FIG. 3 is a graph 300 illustrating example target values in a two-dimensional input variable space. The two dimensions of the space are labeled X1 and X2, respectively, each ranging from 0 to 1. The values in the two-dimensional space are labeled X and O. For this discussion, X has a value of 1 and O has a value of 0. Thus, gradient boosting decision tree training server 135 may create a gradient boosting decision tree to partition a two-dimensional space and generate a score for each partition to minimize error.

At the other extreme, the two-dimensional space may be partitioned such that each training value is in a separate partition, allowing the score of each partition to be accurately set to the training experience value and reducing the error to zero. However, the predicted value of the generated gradient boosting decision tree will be lower than the predicted value of another gradient boosting decision tree that is not over-fitted. Thus, the gradient boosting decision tree training server 135 strives to find a gradient boosting decision tree that reduces errors without overfitting.

FIG. 4 is a diagram illustrating a decision tree 400 for predicting variable values for the two-dimensional space of FIG. 3. Decision tree 400 includes root node 410; intermediate node 420A, intermediate node 420B, intermediate node 420C, intermediate node 420D, and intermediate node 420E; leaf node 430A, leaf node 430B, leaf node 430C, leaf node 430D, leaf node 430E, leaf node 430F, and leaf node 430G.

To predict the values of points in the two-dimensional space of FIG. 3, a series of comparisons are made starting with the comparison of the root node 410. Thus, if the X2 value for this point is less than 0.302548, then the next comparison is that of intermediate node 420A. Otherwise, the next comparison is that of the intermediate node 420B. After multiple comparisons are made, one of the leaf nodes 430A-430G is reached. Each of leaf nodes 430A-430G indicates a score for a point that falls within the region of the leaf node defined by decision tree 400. For example, leaf node 430A is reached for all points with X2<0.0302548 and X1< 0.800113.

FIG. 5 is a diagram 500 illustrating the two-dimensional space of FIG. 3 partitioned using the decision tree 400 of FIG. 4. The two-dimensional space of fig. 3 is divided into region 510, region 520, region 530, region 540, region 550, region 560, and region 570.

Each of the areas 510 to 570 corresponds to each of the leaf nodes 430A to 430G of the decision tree 400. For example, region 510 includes all points of X2<0.302548 and X1> -0.952647. This region corresponds to leaf node 430E and is reached by evaluating root node 410 and

leaf nodes

420B and 420D. The leaf node 430E has a prediction value of 0.4. As another example, the region 550 includes all points with X2> -0.302548 and 0.483215< ═ X1< 0.883535. This region corresponds to leaf node 430C, which is reached by evaluating root node 410 and

leaf nodes

420B and 420C. Leaf node 430C has a score of 0.1231.

Fig. 6 is a graph 600 illustrating a distribution of error metrics between predicted values generated using the decision tree 400 and actual values in the two-dimensional space of fig. 3. In practice, numeric values are typically used, but in graph 600, the symbol "═ is used when the predicted value is within 0.25 of the actual value, the symbol" + "is used when the actual value is at least 0.25 greater than the predicted value, and the symbol" - "is used when the actual value is at least 0.25 less than the predicted value. Thus, when the second decision tree is generated, it will tend to re-partition the high error subspace shown in FIG. 6 with the "+" symbol, thereby correcting the error metric produced by the first decision tree 400. As a result, summing the results of the two decision trees will result in a lower average error than using the first decision tree 400 alone. By repeating this process, any number of decision trees can be generated and summed to reduce the error with each iteration.

Fig. 7 is a flow diagram illustrating operations of a method suitable for training a gradient boosting decision tree according to some example embodiments. Method 700 includes operation 710, operation 720, operation 730, and operation 740. By way of example and not limitation, method 700 is described as being performed by gradient boosting decision tree training server 135 described in FIG. 2 in network environment 100 of FIG. 1.

In operation 710, the decision tree construction module 230 creates a decision tree (e.g., decision tree 400) for the feature space using the objective function. During the creation of the decision tree, the partitioning process stops once a termination criterion is met (e.g., a minimum number of partitions are added to the decision tree, a maximum tree depth is reached, a minimum residual error is reached, or any suitable combination thereof). For example, the partitioning process may stop when the depth reaches a predetermined maximum of 4 to 8. Leaf nodes in the decision tree are assigned values for classification or prediction that minimize the total error of the training data in the corresponding feature space.

In operation 720, the gradient boost module 240 measures the error using the sum of the results of all the created decision trees. Thus, in a first boosting iteration, an error of the first decision tree is determined (e.g., as shown in fig. 6). In subsequent iterations, the results from the multiple decision trees are added together and compared to the actual value of each feature vector in the feature space to determine the error.

In operation 730, the gradient boost module 240 determines whether the error is below a predetermined threshold. If the error is below a predetermined threshold, the model is complete (operation 740). In some example embodiments, if the error is no longer decreasing or a predetermined maximum number of trees (e.g., 1000 trees) have been generated, operation 730 is modified to determine that the model is complete.

If the model is not complete, the method returns to operation 710 and another decision tree is created for the feature space. Since the error is reduced per iteration, an arbitrarily low value may be used for the predetermined threshold, but a lower value will result in a higher number of decision trees being used. In addition, overfitting is possible so that setting the error value of the training data set too low actually results in a higher error value of the validation data set or future data.

Fig. 8 is a block diagram of a database schema 800 suitable for gradient boost feature selection, according to some example embodiments. The database schema 800 includes a metadata table 810 and a prediction table 840. Metadata table 810 includes row 830A, row 830B, row 830C, row 830D, row 830E, and row 830F of format 820. Prediction table 840 includes

rows

860A, 860B, 860C, 860D, 860E, and 860F of format 850. Gradient boosting decision tree training server 135 generates data for metadata table 810 and prediction table 840 during execution of method 700.

The format 820 of the metadata table 810 includes a Gradient Boosting Decision Tree (GBDT) field, a tree field, a feature field, and an error field. Each of rows 830A-830F stores data for a single decision tree in the gradient boosting decision tree. The tree field of each row indicates which tree in the gradient boosting decision tree the row applies to, and the GBDT field identifies which gradient boosting decision tree the row applies to. In the example of fig. 8, data for a single gradient boosting decision tree is shown, but the metadata table 810 is adapted to store data for multiple gradient boosting decision trees making different predictions in different feature spaces, multiple alternative gradient boosting decision trees for making predictions in a single feature space, or any suitable combination thereof.

Each non-leaf node of the decision tree partitions the feature space over a single feature. Thus, each decision tree uses one or more, but not necessarily all, of the features. Features used in the decision tree that are not used in the decision tree earlier than it are indicated in the feature field. Thus, in the example of fig. 8, the feature space is at least eight-dimensional. The first decision tree uses only feature F2 and feature F3. The second decision tree uses feature F1 for the first time, and may or may not use feature F2 and feature F3. The third decision tree uses feature F4 for the first time, and may or may not use feature F1, feature F2, and feature F3. The fourth decision tree must use at least one feature, but the feature field is empty, because the fourth decision tree uses only features that have been used by one or more of the first three decision trees. The fifth decision tree introduces the use of feature F6 and feature F7, and the sixth decision tree introduces the use of feature F8.

The error field indicates the accumulated error of the gradient boosting decision tree when all trees up to the tree of the row are used and no later trees are used. Thus, after a single tree is created, the error is 0.28, as shown in line 830A. Adding a second tree reduces the error to 0.14 as shown in line 830B. Each subsequent tree further reduces the error until an error of 0.04 is reached by using six decision trees, as shown in line 830F.

The format 850 of the prediction table 840 includes a Gradient Boosting Decision Tree (GBDT) field, a tree field, an input vector field, and a value field. Line 860A shows that the predicted (score) value of the input vector V1 generated by the first tree of the first gradient boosting decision tree is 0.8070. Row 860B shows that when the first prediction is refined by lifting the second tree of the decision tree by the first gradient, the refined prediction is 0.9143. Line 860C shows that when the two-tree prediction is refined by the third tree, the prediction becomes 0.9457. Row 860D shows that for alternate GBDT 1A, the first two trees are used as GBDT1, but with alternate tree 3A, the improved prediction is 0.9402. Line 860E and line 860F contain data for another input vector V2.

Referring to fig. 3-5, vector V1 may correspond to an "X" point, which has a value of 1. Thus, each added tree of GBDT1 brings the predicted value closer to 1, thereby reducing the error. However, compared to tree 3 of GBDT1, the spare tree 3A of GBDT 1A reduces the amount of errors relative to the second tree. This is a result of deleting a feature from the set of features on which the tree 3 depends, although the amount of error added may vary depending on the importance of the deleted feature.

Fig. 9 is a flow diagram illustrating operations of a method 900 adapted for using gradient boosting feature selection in training a gradient boosting decision tree, according to some example embodiments. Method 900 includes operation 910, operation 920, operation 930, operation 940, operation 950, operation 960, and operation 970. By way of example and not limitation, method 900 is described as being performed in network environment 100 of FIG. 1 by gradient boosting decision tree training server 135 described in FIG. 2 using database schema 800 of FIG. 8

In operation 910, the gradient boosting decision tree training server 135 builds a fully-qualified (full-fledged) gradient boosting decision tree with all available features (e.g., using the method 700). In operation 920, the feature reduction module 250 identifies the least important features: this importance is estimated by the gradient boost module 240.

The feature reduction module 250 deletes the identified features from the available feature set for training (operation 930). In operation 940, the gradient boosting decision tree training server 135 builds a gradient boosting decision tree (e.g., using the method 700) in the absence of deleted features. This model is referred to as a "reduced model" because it uses fewer features than previous models. In some example embodiments, the process built through the compaction model skips the step of using the metadata table 810. For example, rather than beginning with recreating the first decision tree, the decision tree using the deleted features is simply ignored, and the process of building the tree is resumed by replacing the first decision tree using the deleted features with a new decision tree generated without using the deleted features. Thus, rather than creating the first decision tree from operation 710, operation 710 recreates the sixth decision tree. Compared with the method of deleting the features only and recreating the gradient boosting decision tree from the beginning, the method can save a great deal of computing resources, memory resources and time.

In operation 950, the feature reduction module 250 measures the performance degradation of the lean model. Since the gradient boosting decision tree training server 135 selected the deleted feature in the previous model, deleting this option forces the gradient boosting decision tree training server 135 to make a sub-optimal selection, which in turn reduces the performance of the refinement module. The error metric of the model may be performed against a training data set or a validation data set.

The prediction of the input vector is determined by adding the results of the input vectors from all trees in the gradient boosting decision tree. The prediction table 840 may be used to save computation time in measuring the performance of the lean model. For example, rather than recomputing the prediction for this portion of the reduced gradient boosting decision tree, a partial prediction of the previous gradient boosting decision tree may be used until (taken up to) the last tree remaining in the reduced gradient boosting decision tree. Thus, in the case of GBDT 1A at row 860D, the value field of row 860B may be reused, avoiding re-evaluating Tree 1 and Tree 2 to determine the value of input vector V1 using GBDT 1A. Thus, only partial predictions of tree 3A are determined by evaluating the decision tree (e.g., 0.0279, such that the sum of the 0.9143 prediction in row 860B and the prediction of tree 3A of vector V1 is the value of 0.9402 of row 860D). This approach saves power, processing resources, and computational time by comparison with other approaches that re-evaluate all previous decision trees to determine the predicted values of the streamlined model.

If the error increases below the predetermined threshold (operation 960), the trade-off between feature reduction and performance degradation following the reduction principle is accepted and the method 900 repeats the feature reduction process by returning to operation 920. Alternatively, if the error increase is not below the predetermined threshold in operation 960, the feature reduction module 250 rejects feature reduction and the previous model is accepted as the final model in operation 970. Thus, by using the method 900, features are removed from the model until doing so unacceptably degrades the quality of the model, and the simplest model with acceptable accuracy is used.

Fig. 10 is a flow diagram illustrating operations of a method 1000 adapted for using gradient boosting feature selection in training a gradient boosting decision tree, according to some example embodiments. The method 1000 includes

operations

1010, 1020, 1030, 1040, 1050, and 1060. By way of example and not limitation, method 1000 is described as being performed in network environment 100 of FIG. 1 by gradient boosting decision tree training server 135 described in FIG. 2 using database schema 800 of FIG. 8.

In operation 1010, the decision tree construction module 230 generates a first decision tree for partitioning the feature space, the first decision tree using the first feature set. For example, the decision tree for row 830A may be generated using feature F2 and feature F3.

In operation 1020, the decision tree construction module 230 generates a second decision tree that refines the first decision tree using a second feature set that is not included in the first feature set. As an example, the decision tree of row 830B uses feature F1, and feature F1 is not included in the feature set of the decision tree of row 830A. The second decision tree may also use one or more of the features used in the first decision tree.

In operation 1030, the storage module 260 stores first data associating the first decision tree with the first feature set. For example, line 830A may be created or modified to associate feature F2 and feature F3 with the first decision tree.

In operation 1040, the storage module 260 also stores second data associating a second decision tree with a second feature set. Continuing with the example, line 830B is created or modified to associate feature F1 with the second decision tree.

In operation 1050, the feature reduction module 250 deletes features in the second feature set from the available feature sets. For example,

operations

920 and 930 may be performed. In this example, feature F1 may be deleted from the set of features F1 through F8 on which the decision tree referenced in the metadata table 810 is based.

In operation 1060, the decision tree construction module 230 generates a third decision tree based on the second data associating the second decision tree with the second feature set including the deleted features, the third decision tree refining the first decision tree without using the deleted features.

The above discussion of method 1000 refers to the decision trees of

rows

830A and 830B as first and second decision trees, but method 1000 may be used with other pairs of decision trees. For example, referring to row 830E and row 830F, decision tree "5" may be a first decision tree and decision tree "6" may be a second decision tree, where the first feature set used by the fifth decision tree is the aggregated feature set used by the first five trees (F1, F2, F3, F4, F6, and F7) and the second feature set used by the sixth decision tree is F8.

As another example, consider a gradient boosting decision tree that includes four decision trees corresponding to rows 830A through 830D. In this example, four decision trees were created, but the fourth decision tree did not introduce any new features. Operation 1010 involves the decision tree of row 830B; operation 1020 involves the decision tree of row 830C;

operations

1030 and 1040 involve the storage of row 830B and row 830C. Since the decision tree of row 830D refines the decision tree of row 830C, two decision trees are removed from the gradient boosting decision tree when creating an alternative gradient boosting decision tree that does not rely on feature F4. Thus, in this example, the "third decision tree" of operation 1060 is an alternative decision tree that replaces the decision tree of row 830C.

By using the metadata table 810 in

operations

1030, 1040, and 1060, the method 1000 can build an alternative gradient boosting decision tree that avoids modeling data using features that does not incur substantial computational costs when identifying the least important features. In addition, by generating an alternative third decision tree to refine the first decision tree rather than re-generating the first decision tree, additional computational and memory resources may be saved.

Examples of the invention

Example 1. a method, comprising:

generating, by one or more processors, a first decision tree for partitioning a feature space, the first decision tree using a first feature set, the first feature set being a first subset of available feature sets;

generating, by the one or more processors, a second decision tree that refines the first decision tree to improve partitioning of a feature space, the second decision tree using zero or more features of the first feature set and a second feature set not included in the first feature set, the second feature set being a second subset of the available feature sets;

storing first data associating the first decision tree with the first feature set in a database;

storing, in the database, second data associating the second decision tree with the second feature set;

removing features in the second feature set from the available feature sets; and

generating a third decision tree based on the second data associating the second decision tree with the second feature set including deleted features, the third decision tree refining the first decision tree to improve partitioning of a feature space, the third decision tree not using the deleted features.

Example 2. the method of example 1, further comprising:

generating, by the one or more processors, a fourth decision tree that refines the first decision tree and the second decision tree to improve partitioning of a feature space, the fourth decision tree using zero or more features in the first feature set, zero or more features in the second feature set, and a third feature set not included in the first feature set or the second feature set, the third feature set being a third subset of the available feature sets;

storing the third data associating the fourth decision tree with the third feature set in the database; and

determining, using the second data, that the deleted feature is in the second feature set based on the third data associating the fourth decision tree with the third feature set that does not include the deleted feature.

Example 3. the method of example 1 or example 2, further comprising:

determining a quality difference between a first model comprising the first decision tree and the second decision tree and a second model comprising the first decision tree and the third decision tree; and

based on the quality difference and a predetermined threshold, a feature space is partitioned using the first model.

Example 4. the method of example 3, further comprising:

determining a first quality of the first model based on a validation dataset;

determining a second quality of the second model based on the validation dataset; and wherein

Determining the quality difference based on the first quality and the second quality.

Example 5. the method of example 3 or example 4, wherein determining the difference in mass between the first model and the second model comprises:

determining a quality of the first model by determining a first error metric of the first decision tree and a second error metric of the second decision tree;

storing the first error metric in the database; and

determining a quality of the second model by accessing the stored first error metric for the first decision tree and determining a third error metric for the third decision tree.

Example 6. the method of any of examples 1-5, further comprising:

based on the quality difference and a predetermined threshold, a second feature is removed from the set of available features.

Example 7. the method of any of examples 1-6, further comprising:

determining a first quality of the first decision tree based on a validation dataset;

determining a second quality of the first decision tree in conjunction with the second decision tree based on the validation dataset;

storing third data associating the first quality with the first decision tree in the database; and

storing fourth data associating the second quality with the second decision tree in the database.

Example 8. a system, comprising:

a memory storing instructions; and

one or more processors configured by the instructions to perform operations comprising:

generating a first decision tree for partitioning a feature space, the first decision tree using a first feature set, the first feature set being a first subset of available feature sets;

generating a second decision tree that refines the first decision tree to improve partitioning of a feature space, the second decision tree using zero or more features of the first feature set and a second feature set not included in the first feature set, the second feature set being a second subset of the available feature sets;

Example 9. the system of example 8, wherein the operations further comprise:

generating a fourth decision tree that refines the first decision tree and the second decision tree to improve partitioning of a feature space, the fourth decision tree using zero or more features in the first feature set, zero or more features in the second feature set, and a third feature set not included in the first feature set or the second feature set, the third feature set being a third subset of the available feature sets;

Example 10. the system of example 8 or example 9, wherein the operations further comprise:

Example 11. the system of example 10, wherein the operations further comprise:

determining a first quality of the first model based on a validation dataset;

The determination of the mass difference is based on the first mass and the second mass.

Example 12. the system of example 10 or example 11, wherein determining the difference in mass between the first model and the second model comprises:

storing the first error metric in the database; and

Example 13. the system of any of examples 8 to 12, wherein the operations further comprise:

Example 14. the system of any one of examples 8 to 13, wherein the operations further comprise:

An example 15. a non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

The non-transitory machine-readable medium of example 15, wherein the operations further comprise:

Example 17. the non-transitory machine-readable medium of example 15 or example 16, wherein the operations further comprise:

Example 18. the non-transitory machine-readable medium of example 17, wherein the operations further comprise:

determining a first quality of the first model based on a validation dataset;

Example 19. the non-transitory machine-readable medium of example 17 or example 18, wherein the operations further comprise:

Example 20 the non-transitory machine-readable medium of any of examples 15 to 19, wherein the operations further comprise:

Fig. 11 is a block diagram 1100 illustrating one example of a software architecture 1102 for a computing device. The architecture 1102 may be used, for example, in connection with various hardware architectures as described herein. FIG. 11 is only a non-limiting example of a software architecture, and many other architectures can be implemented to facilitate the functionality described herein. A representative hardware layer 1104 is shown and may represent, for example, any of the computing devices mentioned above. In some examples, hardware layer 1104 may be implemented in accordance with the architecture of the computing system of fig. 11.

The representative hardware layer 1104 includes one or more processing units 1106, the processing units 1106 having associated executable instructions 1108. Executable instructions 1108 represent executable instructions of software architecture 1102, including implementations of the methods, modules, subsystems, and components, etc., described herein. And it may also contain memory and/or storage modules 1110 that also have executable instructions 1108. The hardware layer 1104 may also include other hardware, as indicated by other hardware 1112, that represents other hardware of the hardware layer 1104, such as the other hardware shown as part of the software architecture 1102.

In the example architecture of FIG. 11, the software architecture 1102 may be conceptualized as a stack of layers, where each layer provides specific functionality. For example, software architecture 1102 may contain layers such as: an operating system 1114, a library 1116, a framework/middleware 1118, an application 1120, and a presentation layer 1144. Operationally, application 1120 and/or other components within the layers can invoke an Application Program Interface (API) call 1124 via the software stack, and in response to API call 1124, access a response, return value, etc., as indicated by message 1126. The layers shown are representative in nature, but not all software architectures have all layers. For example, some mobile or special-purpose operating systems may not provide the framework/middleware layer 1118, while others may provide such a layer. Other software architectures may contain additional layers or different layers.

The operating system 1114 may manage hardware resources and provide common services. Operating system 1114 may include a kernel 1128, services 1130, and drivers 1132, for example. The core 1128 may act as an abstraction layer between a hardware layer and other software layers. For example, the kernel 1128 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, and the like. The service 1130 may provide other common services for other software layers. In some examples, services 1130 include interrupt services. Interrupt servicing may detect receipt of an interrupt and in response cause the architecture 1102 to halt its current processing and execute an Interrupt Service Routine (ISR) when the interrupt is accessed.

The driver 1132 may be responsible for controlling or interfacing with the underlying hardware. For example, drivers 1132 may include a display driver, a camera driver, a bluetooth driver, a flash driver, a serial communication driver (e.g., a Universal Serial Bus (USB) driver), a,

Drivers, NFC drivers, audio drivers, power management drivers, etc., depending on the hardware configuration.

The library 1116 may provide a common infrastructure that may be utilized by the application 1120, and/or other components, and/or other layers. The libraries 1116 typically provide functionality that allows other software modules to perform tasks in a manner that is easier than directly interfacing with underlying operating system 1114 functionality (e.g., the kernel 1128, services 1130, and/or drivers 1132). The library 1116 may contain a system library 1134 (e.g., a C-standard library) that may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, and the like. Further, the libraries 1116 may contain API libraries 1136, such as media libraries (e.g., libraries that support the presentation and manipulation of various media formats such as MPEG4, h.264, MP3, AAC, AMR, JPG, PNG), graphics libraries (e.g., OpenGL frameworks that may be used to render 2D and 3D with graphical content on a display), database libraries (e.g., SQLite that may provide various relational database functions), web libraries (e.g., WebKit that may provide web browsing functions), and so forth. The libraries 1116 may also include a variety of other libraries 1138 to provide many other APIs to the applications 1120 and other software components/modules.

The framework 1118 (also sometimes referred to as middleware) can provide a higher level common infrastructure (ifrasstructure) that can be utilized by applications 1120, and/or other software components/modules. For example, the framework/middleware 1118 may provide various Graphical User Interface (GUI) functions, advanced resource management, high-level location services, and the like. Framework 1118 may provide a wide variety of other APIs that may be utilized by applications 1120, and/or other software components/modules, some of which may be specific to a particular operating system or platform.

The applications 1120 include built-in applications 1140 and/or third party applications 1142. Examples of representative built-in applications 1140 may include, but are not limited to, a contacts application, a browser application, a book reader application, a positioning application, a media application, a messaging application, and/or a gaming application. The third party applications 1142 may include any of the built-in applications 1140, as well as a wide variety of other applications. In a specific example, the third-party application 1142 (e.g., Android used by an entity other than a vendor of a particular platform)^TMOr iOS^TMApplications developed by Software Development Kit (SDK) may be in a network such as iOS^TM、Android^TM、

Phone's mobile operating system or other mobile computing device operating system. In this example, the third party application 1142 may cause an API call 1124 provided by a mobile operating system, such as operating system 1114, to facilitate the functionality described herein.

Applications 1120 may utilize built-in operating system functions (e.g., kernel 1128, services 1130, and/or drivers 1132), libraries (e.g., system 1134, API 1136, and other libraries 1138), framework/middleware 1118 to create a user interface for user interaction with the system. Alternatively or additionally, in some systems, interaction with the user may occur through a presentation layer, such as presentation layer 1144. In these systems, the application/module "logic" may be separated from aspects of the application/module that interact with the user.

Some software architectures utilize virtual machines. In the example of fig. 11, this is illustrated by virtual machine 1148. Virtual machines create a software environment that applications/modules can execute as if they were executing on a hardware computing device. Virtual machine 1148 is hosted by the host operating system (operating system 1114) and typically (although not always) has a virtual machine monitor 1146 that manages the operation of virtual machine 1148 and the interface with the host operating system (i.e., operating system 1114). The software architecture executes within virtual machines 1148, such as an operating system 1150, libraries 1152, framework/middleware 1154, applications 1156, and/or a presentation layer 1158. These layers of the software architecture executing within virtual machine 1148 may or may not be the same as the corresponding layers previously described.

Module, component, and logic

Certain embodiments are described herein as comprising logic or multiple components, modules, or mechanisms. The modules may constitute software modules (e.g., (1) code embodied on a non-transitory machine-readable medium or (2) code embodied in a transmission signal) or hardware-implemented modules. A hardware-implemented module is a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client, or server computer system), or one or more hardware processors, may be configured by software (e.g., an application or application portion) as hardware-implemented modules that operate to perform certain operations described herein.

In various embodiments, hardware-implemented modules may be implemented mechanically or electronically. For example, a hardware-implemented module may comprise special-purpose circuitry or logic that is permanently configured (e.g., configured as a special-purpose processor, such as a Field Programmable Gate Array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware-implemented module may also include programmable logic or circuitry (e.g., embodied within a general-purpose processor or another programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to mechanically implement a hardware-implemented module as a dedicated and permanently configured circuit or as a temporarily configured circuit (e.g., as configured by software) may be driven by cost and time considerations.

Thus, the term "hardware-implemented module" should be considered to encompass a tangible entity, be it a physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) entity, that performs in some manner the operations and/or performs some of the operations described herein. In view of embodiments in which the hardware-implemented modules are temporarily configured (e.g., programmed), each of the hardware-implemented modules need not be configured or instantiated at any one instance in time. For example, where a hardware-implemented module includes a general-purpose processor configured using software, the general-purpose processor may be configured at different times as corresponding different hardware-implemented modules. The software may configure the processor accordingly, e.g., to constitute a particular hardware-implemented module at one instance in time and to constitute a different hardware-implemented module at a different instance in time.

A hardware-implemented module may provide information to, or receive information from, other hardware-implemented modules. Accordingly, the described hardware-implemented modules may be considered communicatively coupled. In case a plurality of such hardware implemented modules are present at the same time, the communication may be realized by signal transmission, e.g. by suitable circuits and buses connecting the hardware implemented modules. In embodiments where multiple hardware-implemented modules are configured or instantiated at different times, communication between such hardware-implemented modules may be accomplished, for example, by the storage and retrieval of information in a memory structure to which the multiple hardware-implemented modules have access. For example, a hardware-implemented module may perform an operation and store the output of the operation in a memory device communicatively coupled thereto. Other hardware-implemented modules may then later access the memory device to retrieve and process the stored output. A hardware-implemented module may also initiate communication with an input or output device and may operate on a resource (e.g., a set of information).

Various operations of the example methods described herein may be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that are operable to perform one or more operations or functions. In some example embodiments, the modules referred to herein may comprise processor-implemented modules.

Similarly, the methods described herein may be implemented at least in part by a processor. For example, at least some of the operations of a method may be performed by one or more processors or processor-implemented modules. The performance of certain operations may be distributed among one or more processors, not only residing within a single machine, but also being deployable across multiple machines. In some example embodiments, one or more processors may be located at a single location (e.g., within a home environment, office environment, or server farm), while in other embodiments, processors may be distributed across multiple locations.

The one or more processors may also be operable to support the performance of related operations in a "cloud computing" environment or as a "software as a service" (SaaS). For example, at least some of the operations may be performed by a set of computers (as an example of machines including processors), which may be accessed via a network (e.g., the internet) and via one or more appropriate interfaces (e.g., APIs).

Electronic device and system

Example embodiments may be implemented in digital electronic circuitry, or in computer hardware, firmware, or software, or in combinations of them. Example embodiments may be implemented using a computer program product, such as a computer program tangibly embodied in an information carrier, e.g., in a machine-readable medium, for execution by, or to control the operation of, data processing apparatus, e.g., a programmable processor, a computer, or multiple computers.

A computer program can be written in any form of programming language, including compiled or interpreted languages, and it can be deployed in any form, including as a stand-alone program or as a module, subroutine, or other unit suitable for use in a computing environment. A computer program can be deployed to be executed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

In an example embodiment, operations may be performed by one or more programmable processors executing a computer program to perform functions by operating on input data and generating output. Method operations may also be performed by, and apparatus of example embodiments may be implemented as, special purpose logic circuitry, e.g., an FPGA or an ASIC.

A computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In embodiments where a programmable computing system is deployed, it will be understood that both hardware and software architectures are considered on average. In particular, it will be appreciated that the choice of whether to implement certain functions in permanently configured hardware (e.g., an ASIC), or in temporarily configured hardware (e.g., a combination of software and a programmable processor), or in a combination of permanently configured hardware and temporarily configured hardware, may be a design choice. In various example embodiments, the following lists hardware (e.g., machine) and software architectures that may be deployed.

Example machine architecture and machine-readable media

Fig. 12 is a block diagram of a machine in the example form of a computer system 1200 within which instructions 1224 may be executed to cause the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine operates as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client computer in server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a Personal Computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a Web appliance, a network router, switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term "machine" shall also be taken to include a set of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system 1200 includes a processor 1202 (e.g., a Central Processing Unit (CPU), a Graphics Processing Unit (GPU) or both), a main memory 1204, and a static memory 1206, which communicate with each other via a bus 1208. The computer system 1200 may also include a video display unit 1210 (e.g., a Liquid Crystal Display (LCD) or a Cathode Ray Tube (CRT)). The computer system 1200 also includes an alphanumeric input device 1212 (e.g., a keyboard or touch-sensitive display screen), a User Interface (UI) navigation (or cursor control) device 1214 (e.g., a mouse), a storage unit 1216, a signal generation device 1218 (e.g., a speaker), and a network interface device 1220.

Machine readable medium

The disk drive unit 1216 includes a machine-readable medium 1222 on which is stored one or more sets of data structures and instructions 1224 (e.g., software) embodied in or utilized by any one or more of the methods or functions described herein. The instructions 1224 may also reside, completely or at least partially, within the main memory 1204 and/or within the processor 1202 during execution thereof by the computer system 1200, the main memory 1204 and the processor 1202 also constituting machine-readable media 1222.

While the machine-readable medium 1222 is shown in an example embodiment to be a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more instructions 1224 or data structures. The term "machine-readable medium" shall also be taken to include any tangible medium that is capable of: store, encode, or carry instructions 1224 for execution by a machine and cause the machine to perform any one or more of the methodologies of the present disclosure, or be capable of storing, encoding, or carrying data structures used by such instructions 1224 or associated with such instructions 1224. The term "machine-readable medium" shall accordingly be taken to include, but not be limited to, solid-state memories and optical and magnetic media. Specific examples of the machine-readable medium 1222 include: non-volatile memory, including, as examples, semiconductor memory devices (such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices); magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and compact disk read only memory (CD-ROM) and digital versatile disk read only memory (DVD-ROM) disks. A machine-readable medium is not a transmission medium.

Transmission medium

The instructions 1224 may also be transmitted or received over a communication network 1226 using a transmission medium. The instructions 1224 may be transmitted using the network interface device 1220 and any one of a number of well-known transfer protocols, such as the hypertext transfer protocol (HTTP). Examples of communication networks include a Local Area Network (LAN), a Wide Area Network (WAN), the internet, a mobile telephone network, a Plain Old Telephone (POTS) network, and a wireless data network (e.g., WiFi and WiMax networks). The term "transmission medium" shall be taken to include any intangible medium that is capable of storing, encoding or carrying instructions 1224 for execution by the machine, and includes digital or analog communications signals or other intangible medium to facilitate communication of such software.

While particular embodiments have been described, it will be evident that various modifications and changes may be made to these embodiments without departing from the broader spirit and scope of the disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense. The accompanying drawings that form a part hereof show by way of illustration, and not of limitation, specific embodiments in which the subject matter may be practiced. The embodiments illustrated are described in sufficient detail to enable those skilled in the art to practice the teachings disclosed herein. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. The detailed description is, therefore, not to be taken in a limiting sense, and the scope of various embodiments is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled.

Such embodiments of the inventive subject matter may be referred to herein, individually and/or collectively, by the term "invention" merely for convenience and without intending to voluntarily limit the scope of this application to any single invention or inventive concept if more than one is in fact disclosed. Thus, although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the above description.

Some portions of the subject matter discussed herein may be presented in terms of algorithms or symbolic representations of operations on data stored in a machine memory (e.g., computer memory) in the form of bits or binary digital signals. Such algorithms or symbolic representations are examples of techniques used by those of ordinary skill in the data processing arts to convey the substance of their work to others of ordinary skill in the art. An "algorithm," as the term is used herein, is a self-consistent sequence of operations or similar processing leading to a desired result. In this case, the algorithms and operations involve physical manipulations of physical quantities. Usually, though not necessarily, these quantities may take the form of electrical, magnetic, or optical signals capable of being stored, accessed, transferred, combined, compared, and otherwise manipulated by a machine. It is convenient at times, principally for reasons of common usage, to refer to such signals as "data," "content," "bits," "values," "elements," "symbols," "characters," "terms," "numbers," "numerical values," and the like. These terms are merely convenient labels and should be associated with appropriate physical quantities.

Unless specifically stated otherwise, discussions herein using terms such as "processing," "computing," "calculating," "determining," "presenting," "displaying," or the like, may refer to the action or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or any suitable combination thereof), registers, or other machine components that receive, store, or display information. Furthermore, unless otherwise expressly stated, the words "a" and "an" as used herein include one or more instances, as is common in patent documents. Finally, the conjunction "or" as used herein refers to a non-exclusive "or" unless expressly stated otherwise.

Claims

1. A method, comprising:

generating, by the one or more processors, a second decision tree that refines the first decision tree to improve the partitioning of the feature space, the second decision tree using zero or more features of the first feature set and a second feature set not included in the first feature set, the second feature set being a second subset of the available feature sets;

removing features of the second set of features from the available set of features; and

generating a third decision tree based on the second data associating the second decision tree with the second feature set including deleted features, the third decision tree refining the first decision tree to improve partitioning of the feature space, the third decision tree not using the deleted features.

2. The method of claim 1, further comprising:

generating, by the one or more processors, a fourth decision tree that refines the first decision tree and the second decision tree to improve the partitioning of the feature space, the fourth decision tree using zero or more features of the first feature set, zero or more features of the second feature set, and a third feature set not included in the first feature set or the second feature set, the third feature set being a third subset of the available feature sets;

storing, in the database, third data associating the fourth decision tree with the third feature set; and

3. The method of claim 1, further comprising:

partitioning the feature space using the first model based on the quality difference and a predetermined threshold.

4. The method of claim 3, further comprising:

determining a first quality of the first model based on a validation dataset;

5. The method of claim 3, wherein determining the difference in mass between the first model and the second model comprises:

determining a quality of the first model by determining a first error metric for the first decision tree and a second error metric for the second decision tree;

storing the first error metric in the database; and

6. The method of claim 1, further comprising:

7. The method of claim 1, further comprising:

8. A system, comprising:

a memory storing instructions; and

generating a second decision tree that refines the first decision tree to improve the partitioning of the feature space, the second decision tree using zero or more features of the first feature set and a second feature set not included in the first feature set, the second feature set being a second subset of the available feature sets;

9. The system of claim 8, wherein the operations further comprise:

generating a fourth decision tree that refines the first decision tree and the second decision tree to improve the partitioning of the feature space, the fourth decision tree using zero or more features of the first feature set, zero or more features of the second feature set, and a third feature set not included in the first feature set or the second feature set, the third feature set being a third subset of the available feature sets;

10. The system of claim 8, wherein the operations further comprise:

11. The system of claim 10, wherein the operations further comprise:

determining a first quality of the first model based on a validation dataset;

12. The system of claim 10, wherein determining the difference in mass between the first model and the second model comprises:

storing the first error metric in the database; and

13. The system of claim 8, wherein the operations further comprise:

14. The system of claim 8, wherein the operations further comprise:

15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

16. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise:

17. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise:

18. The non-transitory machine-readable medium of claim 17, wherein the operations further comprise:

determining a first quality of the first model based on a validation dataset;

19. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise:

20. The non-transitory machine-readable medium of claim 15, wherein the operations further comprise: