JP6824450B2 - Iterative feature selection method - Google Patents

Description

This application is a partial continuation of US Patent Application No. 15/636394 filed on June 28, 2017 and International Patent PCT / US2017 / 39812 filed on June 28, 2017. Claims priority over. All ancillary materials listed in this application are incorporated herein by reference in their entirety.

The field of the present invention is iterative feature selection.

Descriptions of the background art include information that may be useful in understanding the present invention. Any document that is prior art, that any of the information provided in the present application is related to the claimed invention, or that is specifically or implicitly referred to is prior art. It does not approve that.

As data becomes more available and the size of datasets grows, many analytical processes are subject to a "curse of dimensionality". Richard E. The phrase "curse of dimensionality", coined by Bellman ("Adaptive control variables: a guided tour", 1961, Princiton University Press), does not occur in low-dimensional settings, in superdimensional spaces (eg, hundreds, thousands, etc.). Or a problem that arises when analyzing and organizing data in a dataset that has millions of features or variables.

All documents herein are likewise incorporated herein by reference, as each document or patent application is specifically and individually indicated to be incorporated by reference. If the definition or use of a term in the incorporated references conflicts with or differs from the definition of that term provided herein, the definition of that term provided herein applies. , Definitions of terms in the bibliography do not apply.

Computer technology continues to advance, but processing and analyzing superdimensional datasets is computationally intensive. For example, in an iterative modeling process, the computational time required to find all possible model component combinations increases exponentially with each additional model component added. In particular, there is a need to reduce computational requirements in superdimensional space by making techniques such as iterative modeling processes more suitable for solving complex problems with large datasets. One way to reduce the computational requirements in an iterative modeling process is to reduce the universe of algorithmic components available in the modeling process.

It is not yet understood that the number of algorithmic components available to the iterative modeling process can be dramatically reduced by determining which components are significant or insignificant to the solution.

Therefore, iterative feature selection methods applied to iterative modeling processes are still needed in the art.

The present invention provides devices, systems and methods in which model components are excluded as possible model components for the development of models in an iterative modeling process.

In one aspect of a key part of the invention, methods are contemplated to reduce the computational time required to improve the model that associates the predictors with the results in the dataset. This method involves several steps. First, a model is generated using the model components from the pool of model components. Using a subset of the dataset, model attribute metrics (eg, accuracy, sensitivity, specificity, area under the curve (AUC) from the receiver operating characteristic (ROC) metric, and algorithm length) are assigned to each model. It is generated for. Next, for some model components, utility metrics are calculated that are (1) the ratio of the amount of models in which each model component is present to (2) the amount of model component pool in which each model component is present. The weighting utility metric corresponding to the model component can then be calculated.

The weighted utility metric is, in some embodiments, the result of a function that includes (1) model attribute metrics for models in which model components are present, and (2) utility metrics for these model components. Certain model components from the pool of model components are eliminated or retained based on the weighted utility metric. In some embodiments, the function comprises the product of a model attribute metric and a utility metric.

In some embodiments, model components are randomly generated. Model components can be, among other things, computational operators, mathematical operators, constants, predictors, features, variables, ternary operators, algorithms, formulas, binary operators, weights, gradients, nodes, or hyperparameters.

The main part disclosed is that by dramatically reducing the computational cycles required to perform a particular task (eg, genetic programming), it has beneficial technical effects, including improved computer behavior. Offering should be appreciated. In the absence of the main part of the invention, iterative modeling methods are coherent due to exorbitant computational requirements, which can often require computational time, primarily months and years. It does not provide a solution.

The various objectives, features, aspects and advantages of the main parts of the invention will become more apparent from the detailed description of the following preferred embodiments along with the accompanying drawings (similar numbers represent similar components). There will be.

It is a figure which shows the general-purpose framework of an iterative modeling process. FIG. 5 illustrates an intended method for determining utility metrics for model components. It is a figure which shows the planned method for determining a model attribute metric. FIG. 5 illustrates an intended method for calculating weighted utility metrics. FIG. 5 illustrates an intended method for excluding or retaining a given model component from a pool of model components. It is a figure which shows one planned embodiment which includes a model, a series of generations, and a run which has a "best" model. It is a figure which shows the pool of the model component corresponding to the run of FIG. FIG. 5 illustrates one intended embodiment, each comprising a model, a series of generations, and a series of runs having a "best" model. It is a figure which shows the series of model component pools corresponding to the run of FIG. FIG. 5 illustrates an intended method for excluding or retaining a given model component from a pool of model components. FIG. 5 illustrates another intended method for excluding or retaining a given model component from a pool of model components. It is a figure which shows the method of incorporating a model component from one model set into another model set. It is a figure which shows another method which incorporates a model component from one model set into another model set. It is a figure which shows the method of incorporating one or more unchanged models from one or more generations of a model to another generation of a model.

The following description provides exemplary embodiments of the main parts of the invention. Although each embodiment represents one combination of elements of the invention, the main part of the invention is believed to include all possible combinations of disclosed elements. Thus, if one embodiment comprises elements A, B, and C and a second embodiment comprises elements B and D, the main part of the invention is further not explicitly disclosed. Is also considered to include other remaining combinations of A, B, C, or D.

As used herein and throughout the claims, the meanings of "one," "one," and "that" may be more than one, unless the context clearly indicates otherwise. It shall include. Also, as used in the description within this application, the meaning of "inside" shall include the meanings of "inside" and "above" unless the context clearly indicates otherwise.

Also, as used in this application, the term "bonded to" is a direct bond (two elements that are joined to each other contact each other), unless the context clearly indicates a different meaning. And indirect binding (at least one additional element is placed between the two elements) shall be included. Therefore, the terms "combined with" and "combined with" are used synonymously.

In some embodiments, numbers representing properties such as the amount, concentration, reaction conditions, etc. of the components used to describe and claim a particular embodiment of the invention are, in some cases, the term "about". Should be understood as modified by. Therefore, in some embodiments, the numerical parameters described in the specification and the appended claims may vary depending on the desired properties required to be obtained by the particular embodiment. Is. In some embodiments, the numerical parameters should be interpreted by taking into account the reported significant digits and applying conventional rounding techniques. While the numerical ranges and parameters that indicate the broad range of some embodiments of the invention are approximations, the numerical values described in the particular embodiment are reported as accurately as possible. The numbers presented in some embodiments of the invention may include certain errors that inevitably arise from the standard deviation found in each test measurement. In addition, all scopes described in this application should be construed as including those endpoints, unless the context indicates the opposite meaning, and open-ended scopes are commercially. It should be construed to include only practical values. Similarly, a list of all values should be considered as containing intermediate values, unless the context indicates the opposite meaning.

Any language targeted to a computer is any suitable computing device, including servers, interfaces, systems, databases, agents, peers, engines, controllers, or other types of computing devices that operate individually or collectively. It should be noted that it should be interpreted as including various combinations. A computing device comprises a processor configured to execute software instructions stored on a tangible, non-transitory computer-readable storage medium (eg, hard drive, solid state drive, RAM, flash, ROM, etc.). You should understand that. Software instructions preferably configure the computing device to provide the roles, responsibilities, or other functions described below with respect to the disclosed device. In a particularly preferred embodiment, the various servers, systems, databases, or interfaces are based on HTTP, HTTPS, AES, public private key exchange, web service APIs, known financial transaction protocols, or other electronic information exchange methods. Exchange data using a seemingly standardized protocol or algorithm. Data exchange is preferably done via packet-switched networks, the Internet, LAN, WAN, VPN, or other types of packet-switched networks. The following description includes information that may be useful in understanding the present invention. Any document that is prior art, that any of the information provided in the present application is related to the claimed invention, or that is specifically or implicitly referred to is prior art. It does not approve that.

As used in this application, terms such as "set" or "subset" should be construed to include one or more items. A "set" does not necessarily include more than one item unless otherwise noted.

One of the main objectives of the present invention is to identify low performance (eg, unnecessary or unnecessary) model components used to create models that describe the relationship between predictors and results in a target dataset. To eliminate. Pruning the number of possible model components improves computational efficiency by reducing the computational time required to converge on a high performance model in an iterative modeling process.

There are several phases in the main part of the invention, and these phases can be implemented as method steps.

In one intended embodiment of the main part of the invention, the first phase is to use an iterative modeling process to generate a set of models from a pool of model components. FIG. 1 shows a general-purpose iterative modeling framework, in which the set {c ₁ , ... .. .. The model component in, c _z } undergoes a modeling process to generate models m _{1 to} m _n .

As used herein, the term "iteration modeling process" is used in a target dataset that includes repeatable or loopable subroutines or processes (eg, runs, for loops, epochs, cycles). Refers to a modeling method for creating one or more models to describe the relationship between a predictor and a result.

The intended iterative modeling process is artificial neural networks (ANN), convolutional neural networks (CNN), recursive neural networks, deep Boltzmann machines (DBM), deep belief networks (DBN), stacked self-encoders. Includes deep learning methods such as, and other modeling techniques derived from neural network frameworks.

Additional or alternative, the intended iterative modeling process includes genetic algorithms and genetic programming (eg, tree-based genetic programming, stack-based genetic programming, linear (including machine code) genetic programming. Includes evolutionary programming methods, including grammatical evolution, extended compact genetic programming (ECGP), embedded Cartesian genetic programming (ECGP), probabilistic incremental program evolution (PIPE), and typed genetic programming (STEP). Other evolutionary programming methods include gene expression programming, evolutionary strategies, differential evolution, neuroevolution, learning classifier systems, or enhanced learning systems, where the solution is binary, real, neural net. , Or a set of classifiers (rules or conditions) that can be of type S. For learning classifier systems, suitability is either a strength or accuracy-based reinforced learning approach or a supervised learning approach. Can be decided.

Additional or alternative intended iterative modeling processes include Monte Carlo methods, Markov chains, and sequential linear logistic regression as long as the process contains repeatable or loopable subroutines or processes (eg, run, foreloop, epoch, cycle). , Decision trees, random forests, support vector machines, Bayesian modeling techniques, or gradient boosting techniques.

In the next phase, utility metrics are calculated for the selected model component and model attribute metrics are calculated for the selected model. The weighted utility metric is then calculated using each utility metric and one or more model attribute metrics. Based on the weighted utility metric, some model components are excluded from the model component pool and others are left untouched. This pruning process improves the computer's ability to perform iterative modeling methods by reducing the dimensions of the search space by reducing the number of model components, which will be described in more detail below. ..

In some embodiments, each model component has a utility metric calculated for it. The utility metric, one embodiment of which is shown in FIG. 2, shows the ratio where the molecule is the number of times the model component appears in the model and the denominator is the number of times the model component appears in the model component pool.

In some intended embodiments, the model components are, for example, computational operators (eg, logical statements such as IF, AND, OR), mathematical operators (eg, such as multiplication, division, subtraction, addition). Arithmetic operations), triangle operations, logistic functions, quantification operations, "floor" or "ceiling" operators, or any other mathematical operator), constants (eg, constant numbers including values such as integers or π) , Predictor (eg, observed or measured or mathematical formula), feature (eg, characteristic), variable, ternary operator (eg, operator that takes three factors, where the first argument is the comparison Arguments, the second argument is the result of a true comparison and the third argument is the result of a false comparison), algorithms, formulas, literals, functions (eg, one-variable functions, two-variable functions, etc.) , Binary operator (for example, an operator that operates on two operands and returns the result in that operand), weights and weight vectors, nodes and hidden nodes, gradient descent, sigmoid activation function, hyperparameters, and bias. Can include.

FIG. 3 shows how the model attribute metric can be determined. In some intended embodiments, the model attribute metric is intended to be able to describe the model's ability to predict outcomes using predictors, the accuracy of which is expressed as a percentage. The data from the dataset is used to determine the model attribute metrics, where the dataset contains predictors and results, the model attributes give the model only predictors, and then the results from the model. Is determined by comparing the actual results from the dataset. For example, if a model uses a set of predictors to accurately predict results with a 35% probability, the model attribute metric for that model will be 35%.

In other embodiments, the model attribute metric additionally or alternatives is sensitivity, specificity, area under the curve (AUC) from the receiver operating characteristic (ROC) metric, root mean square error (RMSE), algorithm. Can be the length of the algorithm, the algorithm calculation time, the variables or components used, or other suitable model attributes. Model attribute metrics can be determined using one or more of the identified model attributes, but it is contemplated that model attribute metrics are not limited to these attributes alone.

Determine if a model component can contribute sufficiently to model performance (for example, if a particular model component affects the model's ability to use a set of predictors to determine results). To this end, weighted utility metrics are generated as a function of each utility metric and one or more model attribute metrics, as shown in FIG.

It is intended that whether a model component is "important" or "insignificant" is determined by whether the weighting utility metric is below or above the threshold. In some embodiments, the threshold is obtained by first averaging all of the weighting utility metrics for the model components that appear in the set of models (eg, {m ₁ , ..., _mn } in FIG. 1). Can be calculated. Each weighted utility metric is then divided by a summary statistic (eg, mean, tri-mean, variance, standard deviation, mode, median) for all weighted utility metrics.

When the result of dividing the weighted utility metric by the summary statistics of all the weighted utility metrics is less than a certain threshold (eg, the result is 1.2, 1.1, 1, 0.9, 0.8, 0. If less than 7, 0.6, 0.5, or 0.4, the model component corresponding to that weighting utility metric is excluded from consideration (eg, any used to generate a new lancet). You cannot put that model component in your new model component pool). This process is shown in FIG.

Other suitable methods of deciding whether to maintain or eliminate model components are also contemplated. For example, in some embodiments, the weighted utility metric is compared to a threshold without undergoing any operation (eg, averaging, division, and comparison, or any of the other processes described above) prior to comparison. To. The threshold can be any value or can be chosen based on an understanding of the expected weighting utility metric values. In these embodiments, when a weighted utility metric for a model component is calculated, the weighted utility metric is then compared to a predefined threshold, and based on that comparison, the model corresponding to the weighted utility metric. The component is either excluded from all model component pools (eg, the weighted utility metric is below the threshold) or left for use in future runs.

Eventually, some model components prove to be less useful than others, based on their corresponding weighted utility metrics, and if the utility shortage is below the threshold, then these model components are It is intended to be destroyed.

In some embodiments, after excluding the model components from consideration, a new pool of model components is created without the excluded model components. In other embodiments, the model component is removed from the existing model pool and the same model pool is reused to generate a set of models for the new lancet. In yet another embodiment, the model component is not excluded from the model component pool, but is simply excluded from consideration. From this point on, the process is repeated and eventually more model components can be eliminated. This process can be repeated as needed until all the remaining model components are found to contribute significantly to the "best" model in each iteration or run.

Through this process, model components are pruned from one or more pools of model components. By pruning the model components according to the main part of the invention, the computational time required to perform iterative modeling (and related tasks) is dramatically reduced.

Although not desired to be limited to one particular type of iterative modeling, a subset of the embodiments of the main parts of the invention exclude model components as possible model components for the development of models in the genetic programming process. Provide devices, systems and methods to be used. Examples of applying the main part of the present invention to genetic programming are useful for understanding the application to other iterative modeling techniques.

For example, in this subset of the intended embodiments, the first phase is to use a genetic programming process to generate a set of models that make up a "run." The term "run" refers to a set of models that are manipulated to converge to the "best" model. In the run, a model set is generated using the model components from the pool of model components.

This model set is called the model generation. In the next phase, the (randomly generated) first generation models are competed to determine which model (one or more) in that generation will perform best, and then one. The part uses a model from the previous generation (eg, based on or by duplicating) to generate the next model generation. These phases are iteratively completed across multiple generations within each run until one or more models are developed that adequately describe the relationship between predictors and results in the dataset.

In the next phase, utility metrics are calculated for the selected model components and model attribute metrics are calculated for the selected model in each run.

First-generation runs require model set generation. The model of the main part of the present invention is described using the notation _mabc , where a is the number of runs, b is the generation number, and c is the number of models. FIG. 6 shows a run having a run number of 1 and shows a first generation composed of models m _{111 to} m _11i . The value of i is the number of models in that generation. It is contemplated that i can be 10 to 1,000,000, more preferably 100 to 10,000, most preferably 1,000 to 5,000.

Models m _{111 to} m _11i are randomly generated using various model components from the model component pool, as shown in FIG. The model is an algorithm and the model components are intended to be used to construct the algorithm. The model components in FIG. 7 are set {c ₁ , ... .. .. , _Cz }. All model components in a pool are available for use in the model corresponding to that model pool, but not all model components need to be used. Moreover, when one model component is used within one model, that model component is available for use within another model.

As described in another paragraph, weighted utility metrics are shown in Figure 4 to determine if a particular model component affects the ability of a model to determine results using a predictor set. Created as a function of each utility metric and one or more model attributes, as shown.

In one aspect of the main part of the invention, the first model generation {m ₁₁₁ ,. .. .. , _{M 11i} } are generated and compete with each other for their first generation models to determine which model performs best. For example, competition can be a comparison of model performance (eg, the ability of a model to predict results from a set of predictors). In some embodiments, the best set of execution models is identified after the models in each generation of the run compete with each other. In other embodiments, one best execution model is identified. The top percentages of performance-based models (eg, top 1-5%, 5-10%, 10-20%, 20-30%, 30-40%, or 40-50%) are the best execution models for each generation. It is intended that it can be considered.

The best execution model can be described in several ways. For example, if the model uses a predictor to predict a percentage of the result (for example, by comparing the result of the model with the actual result from the dataset), that percentage. Can be used to determine if the model is a model that performs better than other models of a generation. In such an embodiment, the models in one generation "break" the model with the lower percent accuracy with respect to each other, with the model having the higher percent accuracy in determining the result from the predictor. Competing. When some (or all but one) of the models in a generation are eliminated (eg, the losing model is removed from the set), the best multiple (or one) models remain.

In another embodiment, the "best" model of a generation can be a model with one or more favorable properties when compared to other models of that generation. For example, the "best" model runs the model "shortest" in terms of algorithm length (eg, the model uses the fewest model components in terms of either quantity, type, or non-overlapping model components). It can be a model with the minimum computational time required to do, the best training accuracy, the best standard process training verification, or the best training verification. In addition, the "best" model can be determined by the combination of these and any other factors discussed in this application.

A second model generation can be generated using one or more models from the first generation in the run identified as the best performer. The second model generation can consist of several subsets of the model. For example, a subset of models in the next generation can be randomly generated using model components from a model pool (shown in Figure 7), while another subset of models can be models from previous generations (eg, one). It can be produced by mutation of (the best model above), and another subset can be produced by creating offspring (also called crossovers) using models from previous generations.

In some embodiments, a subset of models from one generation (eg, one or more models) is included in the next generation (eg, any next generation) without modification. For example, one or more models from the previous generation (eg, one or more "best" models) to any next generation to reduce the time required to converge to the "best" model for the run. Can be introduced. Therefore, for example, in FIG. 6, when the generation a is reached, any of the models from the generation 1 to the a-1 can be included in the generation a. One embodiment of this concept is shown in FIG.

By incorporating one or more models from one generation into the next generation without modification, the components of the model or the characteristics of the model (eg, a group of model components that collectively affect the performance of the model) are described by that generation. It can be introduced to the next generation to improve the ability of the next generation to converge on the "best" model for existing runs. It is suggested that any model introduced from one run or generation to another run or generation is saved first (eg, in computer memory) so that it can be recalled later.

In some embodiments, the model introduced to the next generation is not always the optimal model (eg, according to any of the model performance characteristics discussed in this application, others in the same run in other generations. It does not perform well compared to the model of). The goal is to ensure that the model component or feature is not accidentally excluded, even if it exists only in the model that would have resulted in the model component or feature being discarded or excluded from consideration for another reason. It is only to make.

Introducing a model from one generation to the next generation (eg, the "best" model) without change is repeated by several generations (eg, 10-100 generations, 100-150 generations, 150-250 generations). It is intended that it can be flagged to occur only afterwards. For example, in some embodiments, the "best" model from any of the previous generations may be incorporated into the 100th generation. In other embodiments, if the run is flagged so that the "best" model from one generation can only be taken over after the 100th generation, then in the 101st generation the "best" model from the 100th generation ( One or more) can be incorporated. In these embodiments, after the 100th generation, models from generations prior to the 100th generation can be incorporated into subsequent generations.

The term "crossover" refers to a combination of one or more models for generating new models from one generation to the next. This is similar to the regenerative and biological crossovers on which genetic programming is based. In some embodiments, the model further has a fitness function (eg, as a figure of merit, a particular type of purpose used to summarize the achievement of a given design solution for a set purpose. It can be modified between generations using functions), or multi-generational evolution to solve user-defined tasks (eg, describing the relationship between predictors and results).

Model mutation is the creation of a new model based on one existing model. The mutation model is intended to be a model that is subtly altered or altered from its original form. Mutations can be used to maintain diversity from one generation to the next generation of the model population. This is similar to a biological DNA mutation and involves changing one or more aspects of the model from its initial state.

An example of mutation involves implementing the probability that any bit in the model will change from its original state. A common way to implement mutations involves generating a random variable for each bit in the sequence. This random variable indicates whether a particular bit is modified. This mutation procedure is called a single point mutation, based on a biological point mutation. Other types include inversion and floating point mutations. Other types of mutations include swap, inversion, and scrambling.

Creating a model's offspring is creating a new model based on two or more existing models. Descendants of two or more parent models take features from the parent model and combine those features to create a new model. Embodiments of the main part of the invention use offspring to change the characteristics of the model from one generation to the next. This is similar to the regenerative and biological crossovers on which the models of key parts of the invention (eg, genetic algorithms) are based. Crossover is the process of acquiring multiple (eg, two or more) parent models and generating child models from the parent model.

Using any number or combination of techniques described above, the model {m ₁₂₁ ,. .. .. , _{M 12j} } are created as the second generation run shown in FIG. In some embodiments, each next generation contains fewer models than the previous generation (eg, j <i), but in other embodiments, each next generation has as many models as the previous generation. (For example, j = i) is intended. Similarly, each next generation of models may contain more models than the previous generation (eg j> i), or each generation may contain a different number of models (eg second generation). It has fewer models than the first generation, the third generation may have more models than the second generation, or it may have more models than the first generation, etc.).

The process of iterating through generations of models in the run can be completed as many times as desired. In FIG. 6, the number of generations is represented as a variable a. Preferably, a is large enough that the resulting number of models is sufficient to traverse the dataset. For example, there should be enough generations for the model to consider all possible variables (eg, predictors) from the dataset. For example, a larger dataset may require more model generations than a smaller dataset. In some embodiments, a may be 10 to 10,000 generations, more preferably 50 to 1,000 generations, and most preferably 100 to 500 generations. Generational evolution, as described in the main part of the invention, can be taxonomically described as genetic programming. Since the main part of the invention allows efficient elimination of model components, the method of the main part of the invention can be useful for dramatically improving computational efficiency in any way of iterative programming. Is intended.

After repeating through generation a, the final generation of orchids in FIG. 6 is reached. In some embodiments, the final generation of orchids (eg, generation a in FIG. 6) is composed of one model, but it is also contemplated that the final generation of orchids may consist of a set of models. In embodiments where the final generation of orchids comprises a set of models, one or more "best" models are based on any of the criteria described above with respect to determining which model is "best" in a generation. Will be decided again. It is also envisioned that all models in the final generation can be considered the "best" models of those runs. In embodiments where only one model is present in the final generation of the run, that model is necessarily considered the "best" model of that generation, and thus the "best" model of the run.

Model attributes are calculated for the "best" model using the "best" model (one or more) of the identified runs (eg, in FIG. 6, the best model is labeled as _m1a1 ).

Since each model in the run is created using the model components identified in the model pool, the "best" model (one or more) from a particular run is similarly modeled by the first model generation. Use the model component from the same model pool from which the component was taken. For example, FIG. 7 shows a pool of model components with model components that can be used to generate the model in the run shown in FIG. Therefore, the model components used within the "best" model of the run shown in FIG. 6 will necessarily be taken from the pool of model components shown in FIG.

In some embodiments, each model component used within a run (eg, used in any generation of the run) has a utility metric calculated for it. In other embodiments, only each model component used within the "best" model has a utility metric calculated for it. In yet another embodiment, the utility metric is a model found within a subset of the model from the run (eg, only the latest 10%, 20%, 30%, 40%, 50%, 60%, 70% generations). Can be calculated for a component.

For example, in FIGS. 6 and 7, if a model component from a pool of model components appears in the "best" model (eg, m _1a1 ), the numerator of the utility metric for that model component is 1. If a model component appears multiple times within a model (or within multiple models that make up the "best" generation), the count is incremented by 1 for that model (or for that run). .. For example, if the "best" generation contains two models and both models contain the model component C _g , the numerator for C _g is incremented by 1 for that run.

As for the denominator of utility metrics, each time a model component appears in the pool of model components, the denominator is incremented by 1. For example, all model components in the pool of model components in FIG. 7 have a denominator of 1 for their utility metrics. The denominator of the utility metric can be greater than 1 if there are two or more pools of model components.

8 and 9 show embodiments of the main part of the invention that implement X runs and Y pools of model components. There is one pool of model components per run (eg X = Y), and each pool of model components is specifically intended to correspond to a particular run, but there are fewer model component pools than the run. It is similarly contemplated that there may be more model component pools to obtain (eg, XY), or more than runs (eg, XY).

In determining the utility metrics of the model components appearing in runs 1-X of FIG. 8, the numerator can be between 0-X (eg, the total number of runs) and the denominator can be 1-Y (eg, model). Total number of component pools). For example, if a model component appears in the "best" model in two runs, but the same model component is in four model component pools, the utility metric is 0.5 (20 divided by 4). The utility metric is calculated for each model component, but if the model component does not appear in any of the "best" models in any run, then that model component has a numerator of 0, so the utility metric is 0. It becomes.

It is contemplated that utility metrics can be calculated for any model component in every pool of model components. However, in some embodiments, utility metrics are calculated only for model components that appear in one or more models in the "best" generation of runs. Intuitively, if the model component does not appear in the "best" model, the numerator will necessarily be zero. Therefore, it is possible to skip computing utility metrics for model components that do not appear in at least one "best" model, and instead, at least one from all model component pools without using excessive processor cycles. All model components that do not appear in one "best" model can be eliminated.

For example, in FIG. 8, there are X runs, where each run is one of the best models (ie, the model in the set {m _1a1 , ..., m _Xc1 }, i.e. the final generation of each run). Has. Since the pool of model components shown in FIG. 9 is intended to have overlapping model components, model component c _1g may be present in all or part of the other model pools. .. If model component c _{1 g} appears in 5 model component pools (ie, Y ≧ 5) and c _{1 g} also appears in 3 of the “best” models of these runs, then for model component c _{1 g} . The utility metric will be 3: 5 or 0.6.

A model attribute metric is required for each model in which _{1 g of} utility metric c appears. To calculate the weighting utility metric, a utility metric c _{1 g} is multiplied by some function of the model attribute models c _{1 g} appears. The model attributes of the model in which c _{1 g} appear can be averaged, for example. It is also contemplated that in other embodiments the median model attributes may be used, in other embodiments modes may be used, and in yet other embodiments geometric mean may be implemented.

Also, if there are many "best" models in which a particular model component appears, outliers can be eliminated before calculating the mean, median, or mode (eg, calculating the mean of the model attributes, or It is also envisioned that a certain number of highest and lowest model attributes can be ignored before the median is determined). In some embodiments, other known mathematical operations or functions are applied to the set of model attributes to reach the manipulated model attributes that can be used in calculating weighted utility metrics for a particular model component. Can be done.

Therefore, returning to the above example, if the utility metric for c _{1 g} is 0.6 and the average of the model attributes is 30%, the weighted utility metric is 0.18. This process involves a set of "best" models {m _1a1 , _... .. .. , _Mxc1 } is repeated for all model components, resulting in a weighted utility metric corresponding to each model component in the "best" set of models.

The next phase of the method of the main part of the invention requires determination of which model components are considered important and which are not. Model components that are "important" are to be reused and placed within a new set of model pools that will be used to generate the next lancet. The "insignificant" model components are discarded and not reused in the new model pool set, so certainly no "insignificant" model components are used to create new models.

It is intended that whether a model component is "important" or "insignificant" is determined by whether the weighting utility metric is below or above the threshold. In some embodiments, the threshold first averages all of the weighted utility metrics for the model components that appear in the "best" set of models (eg, {m _1a1 , ..., m _xc1 } in FIG. 8). Can be calculated by doing. The individual weighted utility metrics are then divided by their average. If the result of dividing the weighted utility metric by the average of all weighted utility metrics is less than a certain threshold (eg, the result is 1.2, 1.1, 1, 0.9, 0.8, 0.7). , 0.6, 0.5, or less than 0.4, the model component corresponding to the weighting utility metric is excluded from consideration (eg, the model component to generate a new lancet). Cannot be in any new model component pool used). This process is shown in FIG.

Other suitable methods of deciding whether to maintain or eliminate model components are also contemplated. For example, in some embodiments, the weighted utility metric is compared to a threshold without undergoing any operation (eg, averaging, division, and comparison, or any of the other processes described above) prior to comparison. To. The threshold can be any value or can be chosen based on an understanding of the expected weighting utility metric values. In these embodiments, when a weighted utility metric for a model component is calculated, the weighted utility metric is then compared to a predefined threshold, and based on that comparison, the model corresponding to the weighted utility metric. The component is either excluded from all model component pools (eg, the weighted utility metric is below the threshold) or left for use in future runs.

In some embodiments, model components may be excluded (eg, excluded from consideration) based on their corresponding utility metrics. To do this, when utility metrics are calculated for some components, the utility metrics for those model components are analyzed, for example, using summary statistics. Possible summary statistics are position (eg, arithmetic mean, median, mode, and interquarter mean), skewness (eg, standard deviation, variance, range, quadrant range, absolute deviation, mean absolute). Includes differences (and distance standard deviation), shape (eg skewness or kurtosis, and alternatives based on the L-moment method), and dependencies (eg Pearson's product moment correlation coefficient or Spearman's rank correlation coefficient). ..

The utility metric can then be compared to the summary statistic to determine whether it should be maintained or excluded. For example, if a utility metric for a model component is compared to the arithmetic mean calculated from a set of utility metrics (for example, the utility metric is divided by the average of the set of utility metrics), then the model component has that utility metric. Less than 1 (indicating that the model component is less influential or less useful than half the total number of model components with utility metrics that contribute to the mean) can be excluded. In another embodiment, if the utility metric is less than one standard deviation from the mean, the model component corresponding to that utility metric may be excluded. Its main goal is to facilitate the elimination of model components that do not contribute as much to the "best" model as other model components when compared to other model components. FIG. 11 generally illustrates this concept, where thresholds are determined using summary statistics, as described above.

In many situations, utility metrics are intended to be compared to summary statistics by dividing individual utility metrics by a summary statistic for a set of utility metrics. This is useful for some summary statistics (eg, location summary statistics), while other summary statistics are whether the utility metric falls within the desired range (eg, dispersal degree summary statistics). Needs a comparison between the utility metric value and the range of values to find out.

Instead of calculating the average of the weighted utility metrics, it is also contemplated that the weighted utility metrics for each model component in the "best" model set can be manipulated in other ways. For example, in some embodiments, individual weighted utility metrics may be divided by the median of a set of weighted utility metrics. In other embodiments, the mode of the set of weighted utility metrics may be used instead of the mean or median.

Eventually, some model components prove to be less useful than others, based on their corresponding weighted utility metrics, and if the utility shortage is below the threshold, then these model components are It is intended to be destroyed.

In some embodiments, after excluding the model components from consideration, a new pool of model components is created without the excluded model components. In other embodiments, the model component is removed from the existing model pool and the same model component pool is used again to generate a set of models for the new lancet. In yet another embodiment, the model component is not excluded from the model component pool, but is simply excluded from consideration. From this point on, the process is repeated and eventually more model components can be eliminated. This process can be repeated as needed until all the remaining model components are found to contribute significantly to the "best" model in each run.

It is also envisioned that when the next run is generated using the model component pool that has been trimmed of the model components, the "best" set of models from the previous run may be incorporated into subsequent runs. Therefore, if the "best" model from a previous run contains a model component that would be discarded because it was determined to be insignificant for another reason, then that model component is "best" from the previous run. Even if the model is less favorable than the "best" model of the other run, it can be reintroduced by the introduction of the "best" model from the previous run.

A model component or feature found within a "best" model from some runs is that the "best" model is another "best" (eg, with respect to any of the performance characteristics described in this application). It is intended that it may be excluded from consideration because it is less favorable than the model. These model components or features are reconsidered by introducing a mechanism that allows the model components or features to be excluded for another reason (by excluding the model components from consideration or by excluding the entire model from consideration). As a result, it can result in faster convergence times for high performance models.

This concept can be extended to iterative model generation in any form described in this application. As shown in FIG. 12, the model components from the first model set can be introduced into the second model set by incorporating the models from the first model set into the second model set. It is contemplated that a "set" of models may include generations of models as described in this application.

For example, the result of the first run produces the "best" model, and the second run (which starts with a set of randomly generated models that use the model components from the pruned model component pool). May include the "best" model of the first run in the first set of randomly generated models. By doing so, elements of the previously identified valid model can be introduced into the new run (eg, one or more model components that would otherwise have been discarded) can be revived. As a result, the ability of the second run to evolve the "best" model generationally is improved. The "best" model from the first run is the second, even if it is not the best of the other runs in the group of runs that include the first run and the second run described in this example. Can be introduced in the run of. The goal is for another reason (eg, by eliminating the model component by the method described in this application, or by excluding the model itself from consideration due to its lower performance than other models). By giving a model that does not always perform best in some runs the opportunity to be introduced into a new run in order to incorporate useful model components or model features that would have been eliminated in is there.

Through this process, model components are pruned from one or more pools of model components. By pruning the model components according to the main part of the invention, the computational time required to perform genetic programming (and related tasks) is dramatically reduced.

The inventor intends that model components from the "best" model from previous runs may be additionally reincorporated. In these situations, the "best" model from one run may not be the "best" model when compared to the "best" model from the next run. Even if a model from a previous run is no longer considered the "best", it can be important to reintroduce the model into one or more subsequent runs. For example, in the case of genetic programming, the characteristics of the "best" model from the previous run (eg, a group of model components that collectively affect the performance of the model) can result in a more accurate model in subsequent runs. For some reason, these features did not provide a better model than before. Therefore, by introducing the "best" model from the previous run into the next run, features from the "best" model from the previous run can be incorporated into the models in the next run. For example, in an embodiment that implements a genetic programming technique, this ensures that good model features are introduced into the new run, and as a result, the model containing those features as a whole is from other runs. Those features are not lost if they are discarded or ignored because they are not as powerful as the model in.

Therefore, the model components eliminated through the process described above can be taken into account again. The "best" model from past runs (eg, one or more models form each run that turns out to be "best") can, for example, meet a threshold to leave in consideration. It may contain model components or features that are excluded because they cannot. These "best" models can be considered in the next run (as described above and as shown in FIG. 13), thus reconsidering the otherwise excluded model components. Can be done. In the light of the drawings, for example, as shown in FIG. 8, a model within any generation (eg, final generation) of run 2 can incorporate the "best" model (m _1a1 ) from run 1. As a result, it is intended to reintroduce any model component within model _m1a1 , which was otherwise excluded from consideration. This process is shown in FIG.

In an embodiment where the model component is thus reconsidered, instead of excluding the model component from one or more pools of model components when the model component does not meet the threshold, the model component is simply taken into account. It is intended to be removed (eg, it can remain in all model component pools, but cannot be used in any model). Thus, when the "best" model from one run is reintroduced into the next run, the denominator of the utility metric is no longer zero and that model component has the opportunity to be taken into account again. For example, if a model component is initially excluded from consideration, but then reintroduced and then its weighting utility metric exceeds a threshold, the model component is retaken into consideration and used in later generated models. Can be done.

A major part of the present invention is an improvement in the state of the art, in part because computational methods that process large data sets are subject to the "curse of dimensionality". The curse of dimensionality is the idea that as the number of rows (eg, observations) and / or columns (eg, predictors) increases, the dimension of the problem increases. As the dimensions increase, the volume of space increases so quickly that the available data becomes sparse. This sparsity is problematic for any method that requires statistical significance. This sparsity is problematic for any analytical method in several important ways.

First, if statistically relevant and reliable results are desired, the amount of data required to support the results often increases exponentially with dimensions. Second, many methods of organizing and retrieving data often rely on detecting areas where objects form groups with similar properties. However, with high dimensional data, all objects are sparse and appear different in many ways, which can reduce the efficiency of common data organization procedures.

In the context of iterative modeling techniques, higher dimensions raise further issues. Each additional dimension exponentially increases the size of the solution search space. Many iterative methods randomly sample the search space for possible solutions, so adding each model component to the problem exponentially measures the amount of time (physical and computational) required to converge to the solution. To increase.

In applying the main parts of the invention, we iteratively reduce the number of input features (eg, model components) available in the iterative modeling process, in some cases reaching convergence. We have found that the time required to do this can be reduced by 100x, or, in other ways, the "search space" or depth that the process can consider in the same amount of time can be significantly increased.

One reason for this performance improvement is the reduction of model components available in the iterative modeling process, which allows any individual model component to be RAM or another form of electronic storage device (eg, hard drive flash or another). ) (Called a "cache miss"), as opposed to being stored in the CPU cache and then called (called a "cache hit"). A major part of the invention increases the likelihood of a "cache hit" and, in some cases, a "cache hit" for any given model component rather than a "cache miss".

As briefly described above, the "cache hit" is a state in which the data (for example, a model component) requested for processing by a program is found in the cache memory of the CPU. Cache memory is fairly fast when transferring data to a processor. When executing a command, the CPU looks for data in the closest accessible memory location (usually the primary CPU cache). If the requested data is found in the cache, it is considered a "cache hit". A "cache hit" provides data more quickly due to the speed of the CPU cache when transferring data to the CPU. A "cache hit" may also refer to the retrieval of data from a disk cache where the requested data is stored and accessed in the first query.

The improvement in calculation time when maximizing "cache hits" is brought about by the speed of access to the data stored in the CPU cache as compared to other storage media. For example, a level 1 cache reference is on the order of 0.5 nanoseconds and a level 2 cache reference is on the order of 7 nanoseconds. By comparison, random reads from solid-state hard drives require orders of 150,000 nanoseconds, or 300,000 times slower than level 1 cache references.

Therefore, specific configurations and methods of iterative feature selection are disclosed. However, it will be apparent to those skilled in the art that many modifications other than those already described can be made without departing from the concept of the invention in this application. Therefore, the main part of the present invention should not be limited except in the spirit of the present disclosure. Moreover, in interpreting this disclosure, all terms should be construed in the broadest possible sense consistent with the context. In particular, the term "contains" should be construed as non-exclusively referring to an element, component, or step, and the element, component, or step mentioned is present or utilized. Or indicate that it can be combined with other elements, components, or steps not explicitly mentioned.

Claims (18)

A method of reducing the computational time required to improve the model that associates predictors and results in a dataset that utilizes a processor in a computing system, said method.
Steps to generate at least one model with at least one model component,
A step of performing an iterative model development process to generate an improved model set, including a first improved model based on the at least one model, wherein the improved model set has at least two model generations. In preparation, the iterative model development process is a deep learning method with steps.
Using a subset of the dataset to calculate model attribute metrics for at least one of the models.
A step of calculating at least one utility metric of the at least one model component having a ratio, wherein the molecule of the ratio is the amount of models in which the at least one model component is present in the improved model set. The denominator of the ratio is a step that is incremented when the at least one model component is in the pool of model components.
A step of calculating a weighting utility metric corresponding to the at least one model component, wherein the weighting utility metric comprises the result of a function incorporating the model attribute metric and the at least one utility metric.
A step of excluding the at least one model component from the pool of model components based on the weighted utility metric.
A step of identifying a model from one of the at least two model generations based on at least one criterion.
Steps to save the identified model and
A method. The method further comprises the step of introducing the identified model into the next model run by generating the next model run with the plurality of models, the next model generation in the next model run. The method of claim 1, comprising the identified model. The method of claim 1, wherein the model run comprises a randomly generated model. The method of claim 2, wherein the next model run further comprises crossing at least two models from the model run and at least one of a mutation model from the model run. The method of claim 2, wherein the next model run further comprises a randomly generated model. The at least one criterion is model accuracy compared to other models in the generation, characteristics compared to other models in the generation, model length compared to other models in the generation, and the generation. The method of claim 1, comprising at least one of the calculation times compared to the other models in. The method of claim 2, wherein the identified model comprises a model component that is not in the next model run. A method of reducing the computational time required to improve the model that associates predictors and results in a dataset that utilizes a processor in a computing system, said method.
Steps to generate at least one model with at least one model component,
A step of performing an iterative model development process to generate an improved model set, including a first improved model based on the at least one model, wherein the improved model set has at least one model generation. In preparation, the iterative model development process is a deep learning method with steps.
Using a subset of the dataset to calculate model attribute metrics for at least one of the models.
A step of calculating at least one utility metric of the at least one model component having a ratio, wherein the molecule of the ratio is the amount of models in which the at least one model component is present in the improved model set. The denominator of the ratio is a step that is incremented when the at least one model component is in the pool of model components.
A step of calculating a weighting utility metric corresponding to the at least one model component, wherein the weighting utility metric comprises the result of a function incorporating the model attribute metric and the at least one utility metric.
A step of excluding the at least one model component from the pool of model components based on the weighted utility metric.
A step of identifying a first model from the at least one model generation based on at least one criterion, wherein the first model is a step that is not a preferred model from the at least one model generation.
Steps to save the identified model and
A method. The method of claim 8, further comprising the step of introducing the first model into the next model generation by generating the next model generation comprising the first model. The method of claim 9, wherein the next model generation further comprises a randomly generated model. The at least one criterion is model accuracy compared to other models in the generation, characteristics compared to other models in the generation, model length compared to other models in the generation, and the generation. 8. The method of claim 8, comprising at least one of the calculation times compared to the other models in. The method of claim 9, wherein the next model generation further comprises crossing at least two models from the model generation and at least one of a mutant model from the model generation. A step of identifying a second model from the next model generation based on the at least one criterion, wherein the second model is not a preferred model from the next model generation.
A step of introducing the second model and the first model into the new next model generation by generating a new next model generation comprising the second model and the first model.
9. The method of claim 9. A method of reducing the computational time required to improve the model that associates predictors and results in a dataset that utilizes a processor in a computing system, said method.
Steps to generate at least one model with at least one model component,
A step of performing an iterative model development process to generate an improved model set, including a first improved model based on the at least one model, wherein the improved model set comprises multiple model generations. , The iterative model development process is a deep learning method,
Using a subset of the dataset to calculate model attribute metrics for at least one of the models.
A step of calculating at least one utility metric of the at least one model component having a ratio, wherein the molecule of the ratio is the amount of models in which the at least one model component is present in the improved model set. The denominator of the ratio is a step that is incremented when the at least one model component is in the pool of model components.
A step of calculating a weighting utility metric corresponding to the at least one model component, wherein the weighting utility metric comprises the result of a function incorporating the model attribute metric and the at least one utility metric.
A step of excluding the at least one model component from the pool of model components based on the weighted utility metric.
A step of identifying a model for each generation within the subset of the plurality of model generations based on the criteria, and each identified model is a step that is a preferred model from the corresponding generation.
Steps to save each identified model,
A method. 14. The method of claim 14, further comprising the step of introducing each identified model into a final model run generation by generating a final model generation with each identified model. 14. The method of claim 14, wherein the next model generation additionally comprises a randomly generated model. The criteria are model accuracy compared to other models in the generation, characteristics compared to other models in the generation, model length compared to other models in the generation, and other models in the generation. 14. The method of claim 14, comprising at least one of the calculation times compared to the model. 14. The method of claim 14, wherein the next model generation further comprises crossing at least two models from said model generation and at least one of a mutant model from said model generation.

Images