JP6741888B1 - Iterative feature selection method - Google Patents

Abstract

Feature selection method and process that facilitates reduction of available model components for iterative modeling. It has been found that methods of eliminating model components that do not contribute significantly to the solution can be discovered and abandoned in advance, resulting in dramatically reduced computational requirements in iterative programming techniques. This development unleashes the power of iterative modeling used to solve complex problems that would have traditionally required too much computational time to be useful. [Selection diagram] Fig. 6

Description

The field of the invention is iterative feature selection.

The background description includes information that may be useful in understanding the present invention. Any information provided in this application is prior art, or is related to the claimed invention, or any document specifically or implicitly referred to is prior art. It does not endorse.

As the data becomes more available and the size of the dataset increases, many analytical processes are hit by a "dimensional curse." Richard E. Bellman ("Adaptive control processes: a guided tour" 1961, the phrase "Dimensional curse" created by Princeton University Press) does not occur in a low-dimensional setting (eg, hundreds, thousands, etc.). Or refers to the problems that arise in analyzing and organizing data in a dataset with millions of features or variables.

All documents within this specification are likewise incorporated herein by reference, as if each individual document or patent application was specifically and individually indicated to be incorporated by reference. Where the definition or use of a term in an incorporated reference contradicts or differs from the definition of that term provided herein, the definition of that term provided herein applies. , Definitions of terms in references do not apply.

Although computer technology continues to evolve, processing and analyzing hyperdimensional datasets is computationally intensive. For example, in an iterative modeling process, the computational time required to retrieve all possible model component combinations increases exponentially with each additional model component. In particular, there is a need to reduce computational requirements in hyperdimensional space in ways that make techniques such as iterative modeling processes more suitable for solving complex problems with large datasets. One way to reduce the computational requirements in the iterative modeling process is to reduce the universe of algorithm components available to the modeling process.

It is not yet understood that by determining which components are or are not significant to the solution, the number of algorithmic components available for the iterative modeling process can be dramatically reduced.

Therefore, there is still a need in the art for iterative feature selection methods applied to iterative modeling processes.

The present invention provides an apparatus, system and method in which model components are eliminated as possible model components for model development in an iterative modeling process.

In one aspect of the main part of the invention, a method is contemplated which reduces the computational time required to improve the model that associates predictors with results in a dataset. The method includes several steps. First, a model is created using model components from a pool of model components. Using a subset of the dataset, model attribute metrics (eg, accuracy, sensitivity, specificity, area under the curve (AUC) from the receiver operating characteristic (ROC) metric, and length of the algorithm) can be calculated for each model. Is generated for. Next, for some model components, a utility metric is calculated which is the ratio of (1) the amount of models each model component is present to (2) the amount of model component pool each model component is present. Next, a weighted utility metric corresponding to the model component may be calculated.

The weighted utility metric is, in some embodiments, the result of a function that includes (1) model attribute metrics for the models in which the model components reside, and (2) utility metrics for these model components. Based on the weighted utility metric, particular model components from the pool of model components are excluded or retained. In some embodiments, the function comprises a product of model attribute metrics and utility metrics.

In some embodiments, the model components are randomly generated. Model components can be, among others, computational operators, mathematical operators, constants, predictors, features, variables, ternary operators, algorithms, mathematical expressions, binary operators, weights, gradients, nodes, or hyperparameters.

The disclosed subject matter provides beneficial technical effects, including improved operation of computers, by dramatically reducing the computational cycles required to perform certain tasks (eg, genetic programming). Offering should be evaluated. In the absence of the subject matter of the present invention, iterative modeling methods often proved plausible due to exorbitant computational requirements, which can often require months and years of computational time. It does not provide a solution.

Various objects, features, aspects and advantages of the subject matter of the present invention will become more apparent from the following detailed description of the preferred embodiments along with the accompanying drawings (where like numerals represent like components). Ah

FIG. 3 illustrates a generalized framework of the iterative modeling process. FIG. 6 illustrates a contemplated method for determining utility metrics for model components. FIG. 6 illustrates a contemplated method for determining model attribute metrics. FIG. 6 illustrates a contemplated method for calculating a weighted utility metric. FIG. 6 illustrates a contemplated method for excluding or retaining a given model component from a pool of model components. FIG. 5 illustrates one contemplated embodiment including a model, a series of generations, and a run with a “best” model. FIG. 7 is a diagram showing a pool of model components corresponding to the run of FIG. 6. FIG. 3 illustrates one contemplated embodiment, each including a model, a series of generations, and a series of runs with a “best” model. FIG. 9 is a diagram showing a series of model component pools corresponding to the run of FIG. 8. FIG. 6 illustrates a contemplated method for excluding or retaining a given model component from a pool of model components. FIG. 6 illustrates another contemplated method for excluding or retaining a given model component from a pool of model components.

The following description provides exemplary embodiments of the main parts of the invention. While each embodiment represents one combination of the elements of the invention, the subject matter of the invention is considered to include all possible combinations of the disclosed elements. Thus, if one embodiment comprises element A, element B, and element C and the second embodiment comprises element B and element D, the subject matter of the invention is further said not to be explicitly disclosed. Are also considered to include other remaining combinations of A, B, C, or D.

As used in this application and throughout the claims, the meanings of "one," "one," and "the" are plural unless the context clearly dictates otherwise. Shall be included. Also, as used in the description within this application, the meaning of "inside" is meant to include the meanings of "inside" and "above," unless the context clearly dictates otherwise.

Also, as used within this application, unless the context clearly indicates otherwise, the term "coupled to" is direct coupling (two elements coupled to each other contact each other). And indirect coupling (where at least one additional element is located between two elements). Thus, the terms "coupled with" and "coupled with" are used interchangeably.

In some embodiments, a number characterizing a component used to describe and claim a particular embodiment of the invention, such as amount, concentration, reaction conditions, etc., is sometimes referred to by the term "about". It should be understood as modified by. Therefore, in some embodiments, the numerical parameters recited in the specification and the appended claims may be approximate values that may vary depending on the desired properties sought to be obtained by the particular embodiment. Is. In some embodiments, numeric parameters should be interpreted by considering the reported significant digits and applying conventional rounding techniques. Although the numerical ranges and parameters indicating the broad ranges of some embodiments of the invention are approximations, the numerical values set forth in the specific examples are reported as accurately as possible. The numerical values presented in some embodiments of the invention may include certain errors necessarily resulting from the standard deviation found in their respective testing measurements. Further, all ranges stated in this application are to be construed as encompassing their endpoints, unless the context indicates to the contrary, and the open-ended range is commercially available. It should be construed as containing only practical values. Similarly, a list of all values should be considered to include intermediate values, unless the context indicates otherwise.

Any language of computer target is any suitable computing device, including servers, interfaces, systems, databases, agents, peers, engines, controllers, or other types of computing devices operating individually or collectively. Note that this should be construed as including any combination. The computing device comprises a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (eg, hard drive, solid state drive, RAM, flash, ROM, etc.). You should understand that. The software instructions preferably configure the computing device to provide the roles, responsibilities, or other functionality described below with respect to the disclosed apparatus. In particularly preferred embodiments, the various servers, systems, databases, or interfaces are based on HTTP, HTTPS, AES, public secret key exchange, web services APIs, known financial transaction protocols, or other electronic information exchange methods. Exchange data using any standardized protocol or algorithm that you think is possible. The data exchange preferably takes place over a packet switched network, the Internet, LAN, WAN, VPN, or other type of packet switched network. The following description includes information that may be useful in understanding the present invention. Any information provided in this application is prior art, or is related to the claimed invention, or any document specifically or implicitly referred to is prior art. It does not endorse.

As used in this application, terms such as “set” or “subset” should be construed to include one or more items. The “set” does not always include two or more items unless otherwise specified.

One of the main purposes of the present invention is to identify low-performance (eg, unnecessary or unnecessary) model components used to create models that describe the relationship between predictors and results in a target dataset, Eliminate. Pruning the number of possible model components improves computational efficiency by reducing the computational time required to converge to a high performance model in the iterative modeling process.

There are several phases to the main part of the invention and these phases can be implemented as method steps.

In one contemplated embodiment of the main part of the invention, the first phase is to use an iterative modeling process to generate a set of models from a pool of model components. FIG. 1 shows a generic iterative modeling framework, where the set {c ₁ ,. ． ． , the model components in the _{c z}} to generate a model _m 1 ~m _n receives the modeling process.

As used herein, the term “iterative modeling process” refers to a target dataset that includes repeatable or loopable subroutines or processes (eg, runs, for loops, epochs, cycles). Refers to a modeling method for creating one or more models to describe the relationship between predictors and outcomes.

Contemplated iterative modeling processes include artificial neural networks (ANN), convolutional neural networks (CNN), recurrent neural networks, deep Boltzmann machines (DBM), deep belief networks (DBN), layered self-encoders. , And other deep learning methods such as other modeling techniques derived from the neural network framework.

Additionally or alternatively, contemplated iterative modeling processes include genetic algorithms and genetic programming (eg, tree-based genetic programming, stack-based genetic programming, linear (including machine code) genetic programming). , Evolutionary programming methods including grammar evolution, extended compact genetic programming (ECGP), embedded Cartesian genetic programming (ECGP), stochastic incremental program evolution (PIPE), and typed genetic programming (STEP) Other evolutionary programming methods include gene expression programming, evolution strategies, differential evolution, neuroevolution, learning classifier systems, or reinforcement learning systems, where the solutions are binary, real, neural net. , Or a set of classifiers (rules or conditions) that can be of the S-expression type. For a learning classifier system, relevance is either a strength- or accuracy-based reinforcement learning approach or a supervised learning approach. Can be determined.

Additional or alternative contemplated iterative modeling processes include Monte Carlo methods, Markov chains, sequential linear logistic regression, as long as the process includes repeatable or loopable subroutines or processes (eg, run, foreloop, epoch, cycle). , Decision trees, random forests, support vector machines, Bayesian modeling techniques, or gradient boosting techniques.

In the next phase, utility metrics are calculated for the selection model components and model attribute metrics are calculated for the selection models. A weighted utility metric is then calculated using each utility metric and one or more model attribute metrics. Based on the weighted utility metric, some model components are excluded from the model component pool and other model components are left alone. This pruning process improves the computer's ability to perform iterative modeling methods by reducing the dimension of the search space by reducing the number of model components, which is described in more detail below. ..

In some embodiments, each model component has a utility metric calculated for it. The utility metric, one embodiment of which is shown in FIG. 2, indicates a ratio where the numerator is the number of times the model component appears in the model and the denominator is the number of times the model component appears in the model component pool.

In some contemplated embodiments, the model components are, for example, arithmetic operators (eg, logical statements such as IF, AND, OR), mathematical operators (eg, multiply, divide, subtract, add). Arithmetic operations), trigonometric operations, logistic functions, calculus operations, “floor” or “ceiling” operators, or any other mathematical operator), constants (eg integers or constant numbers, including values such as π) , Predictors (eg, observed or measured values or mathematical formulas), features (eg, characteristics), variables, ternary operators (eg, operators that take three factors), where the first argument is a comparison Argument, the second argument is the result of the true comparison, and the third argument is the result of the false comparison), algorithm, formula, literal, function (eg, one-variable function, two-variable function, etc.) , Binary operators (eg, operators that operate on two operands and return a result in that operand), weights and weight vectors, nodes and hidden nodes, gradient descent, sigmoid activation functions, hyperparameters, and biases. May be included.

FIG. 3 shows how the model attribute metric can be determined. In some contemplated embodiments, it is contemplated that the model attribute metric can describe the model's ability to predict results using predictors, the accuracy of which is expressed as a percentage. The data from the dataset is used to determine the model attribute metrics, where the dataset contains the predictors and the results, the model attributes give the model only the predictors, and then the results from the model. Is determined by comparing with the actual result from the data set. For example, if a model uses a set of predictors to accurately predict results with a 35% probability, then the model attribute metric for that model would be 35%.

In other embodiments, the model attribute metric may additionally or alternatively be a sensitivity, specificity, receiver operating characteristic (ROC) metric to area under curve (AUC), root mean squared error (RMSE), algorithm It can be length, algorithmic computation time, variables or components used, or other suitable model attributes. Although the model attribute metric may be determined using one or more of the identified model attributes, it is contemplated that the model attribute metric is not limited to just these attributes.

Determine whether a model component can contribute sufficiently to model performance (eg, whether a particular model component affects the model's ability to use a set of predictors to determine its outcome) To do so, a weighted utility metric is generated as a function of each utility metric and one or more model attribute metrics, as shown in FIG.

It is contemplated that the model component is “significant” or “not significant” is determined by whether the weighting utility metric is below or above the threshold. In some embodiments, the threshold is determined by first averaging all of the weighted utility metrics for the model components that appear in the set of models (eg, {m ₁ ,..., M _n } in FIG. 1). Can be calculated. Each weighted utility metric is then divided by a summary statistic (eg, mean, trimean, variance, standard deviation, mode, median) for all weighted utility metrics.

If the result of dividing the weighted utility metric by the summary statistic of all weighted utility metrics is less than a certain threshold (eg, the result is 1.2, 1.1, 1, 0.9, 0.8, 0. If less than 7, 0.6, 0.5, or 0.4), the model component corresponding to that weighted utility metric is excluded from consideration (eg, any used to generate a new lancet). The new model component pool of cannot contain that model component). This process is shown in FIG.

Other suitable methods of determining whether to keep or eliminate model components are also contemplated. For example, in some embodiments, the weighted utility metric is compared to a threshold without undergoing any pre-comparison operations (eg, averaging, division, and comparison, or any of the other processes described above). It The threshold may be any value or may be selected based on an understanding of expected weighted utility metric values. In these embodiments, once the weighted utility metric for the model component is calculated, the weighted utility metric is then compared to a predefined threshold, and based on the comparison, the model corresponding to the weighted utility metric. The component is either removed from the pool of all model components (eg, the weighting utility metric is below a threshold) or left for use in future runs.

Finally, some model components prove to be less useful than others based on their corresponding weighted utility metrics, and if the lack of utility is below a threshold, these model components are It is intended to be destroyed.

In some embodiments, after excluding model components from consideration, a new pool of model components is created without the excluded model components. In other embodiments, model components are removed from the existing model pool and the same model pool is used again to generate a set of models for the new lancet. In yet another embodiment, model components are not excluded from consideration, rather than excluded from the model component pool. From this point onward, the process is repeated and eventually more model components may be eliminated. This process may be repeated as needed until all remaining model components are found to contribute significantly to the "best" model in each iteration or run.

Through this process, model components are pruned from one or more pools of model components. By pruning the model components according to the main part of the invention, the computational time required to perform iterative modeling (and associated tasks) is dramatically reduced.

While not wishing to be limited to one particular type of iterative modeling, a subset of the key embodiments of the present invention exclude model components as possible model components for the development of models in the genetic programming process. Apparatus, system and method. Examples of applying the subject matter of the present invention to genetic programming are useful for understanding its application to other iterative modeling techniques.

For example, in this subset of contemplated embodiments, the first phase is to use a genetic programming process to generate a set of models that make up a "run." The term "run" refers to the set of models that are manipulated to converge on the "best" model. Within the run, model components from the pool of model components are used to generate a model set.

This model set is called the model generation. In the next phase, the (randomly generated) first generation model is contended to determine which model (one or more) in that generation performs best, and then The department uses the model from the previous generation (eg, based on or by duplication) to generate the next model generation. These phases are iteratively completed over multiple generations within each run until one or more models have been developed that adequately describe the predictor-result relationship in the dataset.

In the next phase, utility metrics are calculated for the selected model components and model attribute metrics are calculated for the selected model for each run.

First generation runs require the generation of model sets. The model of the main part of the invention is described using the notation _mabc , where a is the run number, b is the generation number and c is the model number. FIG. 6 shows a run having a run number of 1 and shows a first generation composed of models m _{111 to} m _11i . The value of i is the number of models in that generation. It is contemplated that i can be 10 to 1,000,000, more preferably 100 to 10,000, most preferably 1,000 to 5,000.

The models m _{111 to} m _11i are randomly generated using various model components from the model component pool, as shown in FIG. 7. It is contemplated that the model is an algorithm and the model components are used to construct the algorithm. The model component of FIG. 7 has sets {c ₁ ,. ． ． , C _z }. All model components in the pool are available for use in the model corresponding to that model pool, but not all model components need to be used. Further, when a model component is used within a model, that model component is ready for use within another model.

To determine whether a particular model component affects the model's ability to use predictor sets to determine outcomes, as described in another paragraph, the weighting utility metric is set in FIG. As shown, it is created as a function of each utility metric and one or more model attributes.

In one aspect of the subject matter of the present invention, the first model generation {m _111,. ． ． , _{M 11i} } are generated and the models in its first generation compete with each other to determine which model performs best. For example, competition can be a comparison of model performance (eg, the model's ability to predict results from a set of predictors). In some embodiments, the set of best performing models is identified after the models in each generation of runs compete with each other. In other embodiments, one best performing model is identified. The top percent of performance-based models (eg, top 1-5%, 5-10%, 10-20%, 20-30%, 30-40%, or 40-50%) is the best execution model for each generation. It is contemplated that can be considered.

The best execution model can be described in several ways. For example, if a model uses a predictor to predict a percentage of results (for example, by knowing the results and comparing the model's results with the actual results from the data set), the percentage Can be used to determine if the model performs better than other models of a generation. In such an embodiment, the models in one generation are "relative to each other" in such a way that models with high percent accuracy in determining results from the predictors "break through" models with lower percent accuracy. Competing. When some (or all but one) of the models in a generation are eliminated (eg, losing models are removed from the set), the best (or one) model remains.

In another example, a generation's "best" model may be a model that has one or more favorable characteristics when compared to other models in that generation. For example, the "best" model is the "shortest" in terms of the length of the algorithm (eg, the model uses the fewest model components in terms of either quantity, type, or non-overlapping model components), running the model. Can be the model that requires the least computation time, the best training accuracy, the best standard process training validation, or the best training validation. Further, the "best" model may be determined by the combination of these and any other factors discussed in this application.

A second model generation may be generated using one or more models from the first generation in the run identified as the best performers. The second model generation may consist of several subsets of models. For example, a subset of models in the next generation may be randomly generated using model components from the model pool (shown in FIG. 7), while another subset of models may generate models from the previous generation (eg, one model). The best model above) and another subset may be generated by using the model from the previous generation to create progeny (also called crossover).

In some embodiments, the subset of models from one generation is included in the next generation (eg, any next generation) without modification. For example, one or more models from a previous generation (eg, one or more "best" models) can be transformed into a "best" model for a run (the concept of a "best" model for a run is described in further detail below). It can be introduced in any next generation to reduce the time required to converge. Therefore, for example, in FIG. 6, when the generation a is reached, any of the models from generation 1 to a-1 can be included in the generation a.

In addition, introducing the model (eg, the “best” model) from one generation to the next without modification can be achieved in some generations (eg, 10-100 generations, 100-150 generations, 150-250 generations). It is contemplated that the flag may be flagged to occur only after being repeated. For example, in some embodiments, the "best" model from any of the previous generations can be incorporated into the 100th generation. In another embodiment, if the run is flagged so that the “best” model from one generation can only take over after the 100th generation, then in the 101st generation, the “best” model from the 100th generation ( One or more) can be incorporated. In these embodiments, after the 100th generation, models from the 100th generation and earlier may be incorporated into later generations.

The term "crossover" refers to a combination of one or more models to create a new model from one generation to the next. It is similar to the regenerative and biological crossovers on which genetic programming is based. In some embodiments, the model further includes a goodness-of-fit function (eg, as a figure of merit, a particular type of objective used to summarize the degree of achievement of a given design solution for a set objective). Functions) or user-defined tasks (eg, describing predictor-result relationships) can be modified between generations using multiple generation evolution.

Model mutation is the creation of a new model based on one existing model. A mutation model is contemplated to be a model that has subtly changed or altered from its original form. Mutations can be used to maintain one-to-next generation diversity of the model population. It is similar to biological DNA mutations and involves the modification of one or more aspects of the model from its initial state.

One example of a mutation involves implementing the probability that any bit in the model will change from its original state. A common method of implementing mutation involves generating a random variable for each bit in the sequence. This random variable represents whether a particular bit is modified. This mutation procedure is called a single point mutation based on biological point mutations. Other types include inversion and floating point mutations. Other types of mutations include swap, inversion, and scramble.

Creating a descendant of a model is creating a new model based on two or more existing models. Descendants of more than one parent model take features from the parent model and combine those features to create a new model. A key embodiment of the present invention uses progeny to change the characteristics of the model from one generation to the next. This is similar to the regeneration and biological crossover on which the model of the main part of the invention (eg genetic algorithm) is based. Crossover is a process of acquiring a plurality of (for example, two or more) parent models and generating child models from the parent models.

Using any number or combination of techniques described above, the model {m _121,. ． ． , _{M 12j} }, the second generation run shown in FIG. 6 is created. In some embodiments, each next generation includes fewer models than the previous generation (eg, j<i), while in other embodiments each next generation has the same number of models as the previous generation. (Eg, j=i) is contemplated. Similarly, each next generation of models may include more models than previous generations (eg, j>i), or each generation may include a different number of models (eg, second generation Have fewer models than the first generation, the third generation has more models than the second generation, or may have more models than the first generation, etc.).

The process of iterating through the generations of models in a run can be completed as many times as desired. In FIG. 6, the number of generations is represented as a variable a. Preferably, a is large enough that the resulting number of models is sufficient to traverse the dataset. For example, there should be enough generations that the model can account for all possible variables (eg, predictors) from the dataset. For example, a larger dataset may require more model generations as compared to a smaller dataset. In some embodiments, a can be 10-10,000 generations, more preferably 50-1,000 generations, and most preferably 100-500 generations. Generational evolution as described in the main part of the invention can be taxonomically described as genetic programming. Since the main part of the present invention enables efficient elimination of model components, the method of the main part of the present invention can be useful for dramatically improving the computational efficiency in any method of iterative programming. Is contemplated.

After iterating through generation a, the final generation of runs in FIG. 6 is reached. In some embodiments, the final generation of runs (eg, generation a in FIG. 6) is composed of one model, but it is also contemplated that the final generation of runs may be composed of a set of models. In embodiments where the final generation of the run includes a set of models, one or more "best" models are based on any of the criteria described above for determining which model is "best" in a generation. And decided again. It is also contemplated that all models in the final generation may be considered the "best" model of their run. In embodiments where only one model exists in the last generation of a run, that model is necessarily considered the "best" model of that generation, and thus the "best" model of the run.

Model attributes are calculated for the “best” model using the identified “best” model(s) of the run (eg, in FIG. 6, the best model is labeled as m _1a1 ).

Since each model in a run is created using the model components identified in the model pool, the "best" model (one or more) from a particular run is also modeled by the first model generation. Use a model component from the same model pool that retrieved the component. For example, FIG. 7 illustrates a pool of model components having model components that can be used to generate the model in the run shown in FIG. Therefore, the model components used in the "best" model of the run shown in FIG. 6 are necessarily taken from the pool of model components shown in FIG.

This is important for the step of calculating utility metrics for model components. In some embodiments, each model component used within a run (eg, used in any generation of the run) has a utility metric calculated for it. In other embodiments, only each model component used in the "best" model has a utility metric calculated for it. In yet another embodiment, the utility metric is a model found in a subset of models from runs (eg, only the latest 10%, 20%, 30%, 40%, 50%, 60%, 70% generations). Can be calculated for a component.

For example, in FIGS. 6 and 7, if a model component from the pool of model components appears in the “best” model (eg, m _1a1 ), the numerator of that model component's utility metric is 1. If a model component occurs multiple times within a model (or within the models that make up the "best" generation), the count is incremented by 1 for that model (or for that run). .. For example, a "best" generation of two models, when both models including model component C _g, the molecules of the C _g, is incremented by 1 for the run.

Regarding the denominator of the utility metric, the denominator is incremented by 1 each time a model component appears in the pool of model components. For example, all model components in the pool of model components of Figure 7 have a denominator of 1 for their utility metric. The denominator of the utility metric can be greater than 1 if there are two or more pools of model components.

8 and 9 show an embodiment of the main part of the invention implementing X runs and Y pools of model components. There is one pool of model components per run (eg, X=Y), and it is contemplated that each pool of model components specifically corresponds to a particular run, but there are fewer model component pools than runs. It is also contemplated that there may be more (eg, X>Y) or more model component pools than runs (eg, X<Y).

In determining the utility metrics of model components that appear in runs 1-X of FIG. 8, the numerator can be between 0-X (eg, the total number of runs) and the denominator is 1-Y (eg, model). The total number of component pools). For example, if a model component appears in the "best" model in two runs, but the same model component exists in four model component pools, the utility metric will be 0.5 (4 divided by 2). The utility metric is calculated for each model component, but if the model component does not appear in any of the "best" models in any run, then the model component has a numerator of 0, so the utility metric is 0. Becomes

It is contemplated that the utility metric can be calculated for every model component in every pool of model components. However, in some embodiments, utility metrics are calculated only for model components that appear in one or more models in the "best" generation of a run. Intuitively, if a model component does not appear in the "best" model, its numerator is necessarily 0. Therefore, it may be possible to skip computing utility metrics for model components that do not appear in at least one "best" model, and instead use at least 1 from all model component pools without using excessive processor cycles. All model components that do not appear in one "best" model can be eliminated.

For example, in FIG. 8, there are X runs, where each run is one best model (ie, the model in the set {m _1a1 ,..., m _Xc1 }, ie the last generation of each run). Have. It is contemplated that the pool of model components shown in FIG. 9 may have overlapping model components, so model component c _1g may be able to be present in all or some of the other model pools. .. Model component c _{1 g} five model component pool (i.e., Y ≧ 5) appeared in the case where c _{1 g} is likewise appearing in the three "best" model for these runs, the model components c _{1 g} The utility metric will be 3:5 or 0.6.

A model attribute metric is needed for each model in which the utility metric c _1g appears. To calculate the weighting utility metric, a utility metric c _{1 g} is multiplied by some function of the model attribute models c _{1 g} appears. The model attributes of the model in which c _1g appears can be averaged, for example. It is also contemplated that in other embodiments, the median of model attributes may be used, in other embodiments modes may be used, and in yet other embodiments geometric mean may be implemented.

Also, if there is a large number of "best" models in which a particular model component appears, outliers may be eliminated before calculating the mean, median, or mode (eg, calculating the average of model attributes, or It is also contemplated that a certain number of highest and lowest model attributes may be ignored before determining the median). In some embodiments, other known mathematical operations or functions are applied to the set of model attributes to arrive at manipulated model attributes that may be used in calculating weighting utility metrics for particular model components. Can be done.

Thus, returning to the example above, if the utility metric for c _1g is 0.6 and the average of the model attributes is 30%, the weighted utility metric is 0.18. This process consists of a set of “best” models {m _1a1,. ． ． , _{M xc1} }, is repeated for each model component that results in a weighted utility metric corresponding to each model component in the set of "best" models.

The next phase of the method of the main part of the invention involves the determination of which model components are considered important and which are considered unimportant. Model components that are "important" are eligible to be reused and placed in a set of new model pools that will be used to generate the next lancet. The "insignificant" model components are discarded and not reused in the new model pool set, thus ensuring that the "insignificant" model components are not used to create new models.

It is contemplated that the model component is “significant” or “not significant” is determined by whether the weighting utility metric is below or above the threshold. In some embodiments, the threshold first averages all of the weighted utility metrics for model components that appear in the “best” set of models (eg, {m _1a1 ,..., m _xc1 } in FIG. 8). Can be calculated by Each individual weighted utility metric is then divided by its average. If the result of dividing the weighted utility metric by the average of all weighted utility metrics is less than a certain threshold (eg, the result is 1.2, 1.1, 1, 0.9, 0.8, 0 .7, 0.6, 0.5, or 0.4), the model component corresponding to the weighted utility metric is excluded from consideration (eg, the model component is generated a new lancet). Cannot be put into any new model component pool used for. This process is shown in FIG.

Other suitable methods of determining whether to keep or eliminate model components are also contemplated. For example, in some embodiments, the weighted utility metric is compared to a threshold without undergoing any manipulation (eg, averaging, division, and comparison, or any of the other processes described above) prior to comparison. It The threshold may be any value or may be selected based on an understanding of expected weighted utility metric values. In these embodiments, once the weighted utility metric for the model component is calculated, the weighted utility metric is then compared to a predefined threshold, and based on the comparison, the model corresponding to the weighted utility metric. The component is either removed from the pool of all model components (eg, the weighting utility metric is below a threshold) or left for use in future runs.

In some embodiments, model components may be excluded (eg, excluded from consideration) based on their corresponding utility metrics. To do this, once the utility metrics are calculated for some components, the utility metrics for those model components are analyzed using, for example, summary statistics. Assumed summary statistics include position (eg arithmetic mean, median, mode, and interquartile mean), scatter (eg standard deviation, variance, range, interquartile range, absolute deviation, mean absolute). Difference and distance standard deviation), shape (eg skewness or kurtosis, and alternative based on L-moment method), and dependency (eg Pearson product moment correlation coefficient or Spearman rank correlation coefficient) ..

The utility metric can then be compared to the summary statistic to determine if it should be maintained or eliminated. For example, if the utility metric for a model component is compared to the arithmetic mean calculated from the set of utility metrics (eg, the utility metric is divided by the average of the set of utility metrics), the model component If it is less than 1 (indicating that the model component has less than half the total number of model components that have utility metrics that contribute to the average, or is not useful), then it may be excluded. In another example, if the utility metric is less than one standard deviation from the mean, the model component corresponding to that utility metric may be eliminated. Its main goal is to facilitate the elimination of model components that contribute less to the "best" model than other model components when compared to other model components. FIG. 11 generally illustrates this concept, where the threshold is determined using summary statistics, as described above.

It is contemplated that in many situations the utility metric will be compared to the summary statistic by dividing the individual utility metric by the summary statistic of the set of utility metrics. This is useful for some summary statistics (eg, location summary statistics), while other summary statistics show whether the utility metric falls within the desired range (eg, scatter index summary statistic). To find out, we need to compare the utility metric value to a range of values.

Instead of calculating the average of the weighted utility metrics, it is also contemplated that the weighted utility metrics for each model component in the "best" model set may be manipulated in other ways. For example, in some embodiments, individual weighted utility metrics may be divided by the median of the set of weighted utility metrics. In other embodiments, the modes of the set of weighted utility metrics may be used instead of mean or median.

Finally, some model components prove to be less useful than others based on their corresponding weighted utility metrics, and if the lack of utility is below a threshold, these model components are It is intended to be destroyed.

In some embodiments, after excluding model components from consideration, a new pool of model components is created without the excluded model components. In other embodiments, model components are removed from the existing model pool and the same model pool is used again to generate a set of models for the new lancet. In yet another embodiment, model components are not excluded from consideration, rather than excluded from the model component pool. From this point onward, the process is repeated and eventually more model components may be eliminated. This process may be repeated as needed until all remaining model components are found to contribute significantly to the "best" model in each run.

It is also contemplated that when generating a next run using the model component pool that has undergone model component trimming, the set of "best" models from the previous run may be incorporated into the next run. Therefore, if a "best" model from a previous run contains a model component that would be discarded because it was determined to be unimportant for another reason, that model component would be "best" from that previous run. Can be reintroduced by the introduction of the model.

For example, the first run results in a "best" model and a second run (which starts with a randomly generated set of models that use model components from the pruned model component pool). May include the "best" model of the first run in the first set of randomly generated models. By doing so, the previously identified valid model elements can be introduced into a new run (eg, one or more model components that would otherwise have been discarded can be reinstated), As a result, the ability of the second run to evolve the "best" model generationally improves.

Through this process, model components are pruned from one or more pools of model components. By pruning model components according to the main part of the invention, the computational time required to perform genetic programming (and related tasks) is dramatically reduced.

The inventor contemplates that model components from the "best" model from a previous run may additionally be reincorporated. Model components that have been eliminated through the process described above can be re-considered. The “best” model from past runs (eg, one or more models forming each run that was found to be “best”) cannot meet the threshold to remain in consideration. Model components that have been excluded from the. These "best" models can be considered in the next run (as described above), so that model components that have been excluded for another reason can be re-considered. Looking at the drawing, for example, as shown in FIG. 8, a model within any generation of run 2 (eg, the last generation) can incorporate the “best” model (m _1a1 ) from run 1. , _So that it is contemplated to reintroduce any model component in model m _1a1 that may otherwise have been excluded from consideration. This process is shown in FIG.

In an embodiment in which a model component is re-considered in this way, instead of excluding it from one or more pools of model components when the model component does not meet a threshold, the model component is simply considered. It is contemplated that it is removed (eg, it can remain in all model component pools but cannot be used in any model). Thus, when the "best" model from one run is reintroduced into the next run, the denominator of the utility metric will be non-zero and that model component will have the opportunity to be considered again. For example, if a model component was first excluded from consideration, but reintroduced, and then its weighting utility metric exceeds a threshold, the model component is re-considered and used in later generated models. Can be done.

The main part of the invention is the improvement of the state of the art, in part because the computational methods that handle large data sets are subject to "dimensional curse." The dimension curse is the idea that the dimension of the problem increases as the number of rows (eg, observations) and/or columns (eg, predictors) increases. As the dimension increases, the volume of space grows so fast that the available data becomes sparse. This sparsity is problematic for any method that requires statistical significance. This sparsity is problematic for any analytical method in several important respects.

First, when statistically relevant and reliable results are desired, the amount of data needed to support the results often increases exponentially with dimension. Second, many methods of organizing and retrieving data often rely on finding regions where objects form groups with similar characteristics. However, in high dimensional data, all objects are sparse and appear to differ in many ways, which can reduce the efficiency of common data organization strategies.

In the context of iterative modeling techniques, higher dimensions pose additional problems. Each additional dimension exponentially increases the size of the solution search space. Since many iterative methods randomly sample the search space of possible solutions, adding each model component to the problem exponentially reduces the amount of time (physical and computational) needed to converge to the solution. Increase.

In applying the main part of the invention, we find that iteratively reducing the number of input features (eg, model components) available to the iterative modeling process may reach convergence. It has been found that the time required to do can be reduced by 100x, or alternatively, the process can significantly increase the "search space" or depth that can be considered in the same amount of time.

One reason for this performance improvement is that due to the reduction of model components available in the iterative modeling process, any individual model component may be in RAM or another form of electronic storage (eg, hard drive flash or another). ), which is stored in the CPU cache and subsequently called (called a “cache hit”). The essence of the present invention increases the likelihood of a "cache hit" and, in some cases, the "cache hit" for any given model component more than the "cache miss".

As briefly mentioned above, a "cache hit" refers to a state in which the data (eg, model component) required to perform a program operation is found in the cache memory of the CPU. Cache memory is considerably faster at transferring data to the processor. When executing a command, the CPU looks for the data in the closest accessible memory location (typically the primary CPU cache). If the requested data is found in the cache, it is considered a "cache hit". A "cache hit" provides data more quickly due to the speed of the CPU cache when transferring data to the CPU. A "cache hit" may also refer to pulling data from the disk cache where the requested data is stored and accessed on the first query.

The improvement in computational time in maximizing "cache hits" is provided by the speed of access to the data stored in the CPU cache as compared to other storage media. For example, level 1 cache references are on the order of 0.5 nanoseconds and level 2 cache references are on the order of 7 nanoseconds. By comparison, a random read from a solid state hard drive takes on the order of 150,000 nanoseconds, or 300,000 times slower than a level 1 cache reference.

Accordingly, particular configurations and methods of iterative feature selection are disclosed. However, it will be apparent to those skilled in the art that many modifications other than those already described can be made without departing from the inventive concept in this application. Therefore, the body of the invention should not be limited except in the spirit of the disclosure. Moreover, in interpreting this disclosure, all terms should be interpreted in the broadest possible manner consistent with the context. In particular, the term “comprising” should be construed as non-exclusively referring to an element, component, or step, and the referenced element, component, or step is present or utilized, Or, it may be combined with other elements, components, or steps not explicitly mentioned.

Claims (18)

A computing system for increasing cache hits when running an iterative model development program,
A processor,
Cache memory,
A second memory, the second memory using the processor,
Generating a first model (a), the first model comprising a first model component from a first model component pool;
Generating (b) a second model, the second model comprising a second model component from a second model component pool (b);
Performing an iterative model development process using the first model and the second model to produce an improved model set, (c).
The first model component comprising a ratio of (1) the amount of models that the first model component is present in the improved model set to (2) the amount of model component pools where the first model component is present; (D) calculating a first utility metric of the model component of
The second model component comprising a ratio of the amount of models present in the improved model set to the amount of model component pool present in the second model component; (E) calculating a second utility metric of the model component of
Excluding the first model component and the second model component from the first model component pool and the second model component pool based on the first utility metric and the second utility metric; Thereby increasing the likelihood that the processor will access the cache memory when performing step (c), (f)
A second memory comprising executable instructions for performing
A computing system comprising: The computing system of claim 1, wherein the first model component is randomly selected from the first model component pool for use in the first model. The first model component pool and the second model component pool are calculation operators, mathematical operators, constants, predictors, features, variables, ternary operators, algorithms, mathematical expressions, binary operators, hidden nodes. The computing system of claim 1, comprising at least one of:, weights, biases, slopes, and hyperparameters. The computing system of claim 1, wherein steps (a) and (b) comprise an iterative modeling process. The iterative modeling process comprises at least one of an evolutionary computing process, a genetic programming process, a genetic algorithm process, a neural network process, a deep learning process, a Markov modeling process, a Monte Carlo modeling process, and a stepwise regression process. The computing system according to claim 4. Retaining first model components and second model components from the first model component pool and the second model component pool based on the first utility metrics and the second utility metrics. The computing system of claim 1, further comprising: The second memory is
Excluding the first model component and the second model component from the first model component pool and the second model component pool based on the first utility metric and the second utility metric , Generating a third model component pool,
Generating a third model, the third model comprising a third model component from the third model component pool;
A third utility of the third model component, comprising (1) a ratio of the amount of the model in which the third model component is present to (2) the amount of the model component pool in which the third model component is present. Calculating the metric,
Excluding the third model component from the third model component pool based on the third utility metric;
The computing system of claim 1, further comprising instructions for performing. A computing system for increasing cache hits when running an iterative model development program,
A processor,
Cache memory,
The second memory,
Generating a first model (a), the first model comprising a first model component from a first model component pool;
Generating (b) a second model, the second model comprising a second model component from a second model component pool (b);
Performing (c) an iterative model development process to generate an improved model set, the improved model set comprising: an improved first model from the first model and the second model. Step (c) with an improved second model from the model,
Calculating (1) a first model attribute metric corresponding to the improved first model, and (2) a second model attribute metric corresponding to the improved second model. ,
The first model component comprising a ratio of (1) the amount of models that the first model component is present in the improved model set to (2) the amount of model component pools where the first model component is present; (E) calculating a first utility metric of the model component of
The second model component comprising a ratio of the amount of models present in the improved model set to the amount of model component pool present in the second model component; (F) calculating a second utility metric of the model component of
Calculating (g) a first weighted utility metric corresponding to the first model component, wherein the first weighted utility metric is (1) a model for a model in which the first model component resides An attribute metric, and (2) a step (g) comprising the result of a first function incorporating the first utility metric,
Calculating (h) a second weighted utility metric corresponding to the second model component, wherein the second weighted utility metric comprises: (1) a model for a model in which the second model component exists. An attribute metric and (2) a step (h) comprising the result of a second function incorporating the second utility metric,
Excluding the first model component and the second model component from the first model component pool and the second model component pool based on the first weighted utility metric and the second weighted utility metric (I) thereby increasing the likelihood that the processor will access the cache memory when performing step (c);
A second memory comprising instructions for performing
A computing system comprising: 9. The model attribute metric of claim 8, wherein the model attribute metric comprises at least one of accuracy, sensitivity, specificity, area under the curve (AUC) from a receiver operating characteristic (ROC) metric, and algorithm length. Computing system. The computing system of claim 8, wherein the second memory further comprises instructions for repeating the step (c) after performing the step (i). The iterative model development process comprises at least one of an evolutionary calculation process, a genetic programming process, a genetic algorithm process, a neural network process, a deep learning process, a Markov modeling process, a Monte Carlo modeling process, and a stepwise regression process. The computing system according to claim 10. A computing system for increasing cache hits when running an iterative model development program,
A processor,
Cache memory,
Has a dataset containing predictors, and
Generating a model comprising model components (a),
(B) performing an iterative model development process to produce an improved model set, the improved model set including a first improved model based on the model;
Calculating (c) model attribute metrics corresponding to the model using a subset of the dataset;
Calculating (d) a utility metric of the model component comprising a ratio, wherein the numerator of the ratio comprises the amount of the model in which the model component is present in the improved model set, and the denominator of the ratio. (D) is incremented when the model component is present in the model component pool;
Calculating (e) a weighted utility metric corresponding to the model component, the weighted utility metric comprising (1) the result of a function incorporating the model attribute metric and (2) the utility metric. )When,
Excluding the model component from the model component pool based on the weighted utility metric, thereby increasing the likelihood that the processor will access the cache memory when performing step (b) (f) )When,
A second memory further comprising instructions for performing
A computing system comprising: The computing system of claim 12, wherein the second memory further comprises instructions for holding model components from the model component pool based on the weighted utility metrics. The computing system of claim 12, wherein the model component is randomly generated. The model component includes a calculation operator, a mathematical operator, a constant, a predictor, a feature, a variable, a ternary operator, an algorithm, a mathematical expression, a binary operator, a hidden node, a weight, a bias, a gradient, and a hyperparameter. The computing system of claim 12, comprising at least one of: The model component pool comprises at least one of a computational operator, a mathematical operator, a constant, a predictor, a feature, a variable, a ternary operator, an algorithm, a mathematical expression, and a binary operator. The described computing system. The computing system of claim 12, wherein the function comprises at least the product of the model attribute metric and the utility metric. 13. The model attribute metric of claim 12, wherein the model attribute metric comprises at least one of accuracy, sensitivity, specificity, area under the curve (AUC) from a receiver operating characteristic (ROC) metric, and algorithm length. Computing system.

Images