JP2013025349A - Feature selection device, method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide technique to select an appropriate combination of features in a practical time under a condition that relation between variables is not restricted to linear correlation, even if the number of combinations of features exponentially increases when the number of features is large.SOLUTION: A feature selection device comprises: non-dense relation learning means for learning non-dense relation in order to increase a value of an evaluation function calculated on the basis of observation data including a criterion variable and explanatory variables and non-dense relation between the criterion variable and the respective explanatory variables; and prediction function learning means for learning a prediction function in which every variable has correlation with any of variables, on the basis of the learned non-dense relation.

Description

  The present invention relates to a feature selection apparatus, method, and program.

Consider a regression problem consisting of an objective variable Y and explanatory variables X1,. When a linear model is used, the prediction function is expressed by the following equation.
y ^ = β ₁ X ₁ + β ₂ X ₂ +..., β _p X _p
Using least squares estimation, the parameter estimators are: X = (X ₁ ,..., X _p ), β = (β ₁ ,..., Β _p ) ^T
β = (X ^T X) ⁻¹ X ^T _y
It is obtained.

However, if regression analysis is performed using the parameters obtained above as they are, all explanatory variables will be used to estimate the objective variable.
・ The estimation accuracy of the objective variable may not be the best.
・ The relationship between objective variables and explanatory variables is difficult to understand.
There is a problem. To solve this problem, it is only necessary to select some explanatory variables for use in estimating the objective variable. This is equivalent to setting some elements of β = (β ₁ ,..., Β _p ) to 0. Here, an explanatory variable that is important for estimation of the objective variable is called a feature, and selection of a variable used for estimation of the objective variable is called feature selection.

  Non-Patent Document 1 section 3.3.1 describes an example of a feature selection system. This feature selection system includes feature combination means, function learning means, accuracy evaluation means, and optimum feature selection means. The feature combination means generates a combination of all features. The function learning means learns the prediction function from the feature combination to the objective variable for all the feature combinations obtained by the feature combination means. The accuracy evaluation means evaluates the accuracy of the learned prediction function. The optimum feature selection unit selects the combination of features with the highest accuracy.

  Non-Patent Document 1 Section 3.3.2 describes another example of a feature selection system. This feature selection system includes feature combination means, function learning means, accuracy evaluation means, and optimum feature selection means. The feature combination means adds a new feature to the feature combination obtained at the present time or generates a feature combination with reduced features. The function learning means learns a prediction function from the feature combination to the objective variable with respect to the feature combination generated by the feature combination means. The accuracy evaluation means evaluates the accuracy of the learned prediction function. The optimum feature selection unit selects the combination of features with the highest accuracy.

Trevor Hastie, Robert Tibshirani, Jerome Friedman, “The Elements of Statistical Learning,” Springer, 2009. Tosaka, Yoshiba, “Explanation of concrete utilization method of copula in financial practice”, Financial Research, Bank of Japan Financial Research Institute, December 2005. S. Kirshner, “Learning with tree-averaged branches and distributions,” Advances in Neural Information Processing Systems, NIPS 2007, December 2007. G. Elidan, “Copula Bayesian Networks,” Advances in Neural Information Processing Systems, NIPS 2010, December 2010.

  The first problem of the feature selection system described in Non-Patent Document 1 is that evaluation cannot be performed for all combinations when the number of features is large. The reason is that, when the number of features is large, a method of selecting an optimum feature combination for a combination of features that exponentially increases is not sufficiently considered.

  The second problem of the feature selection system described in Non-Patent Document 1 is that the relationship between the objective variable and the feature is limited to linear correlation. This is because a method for expressing a nonlinear correlation in the model used is not sufficiently considered.

  In view of the above, the present invention provides a realistic time for a combination of features that increase exponentially when the number of features is large under the condition that the relationship between variables is not limited to linear correlation. Another object of the present invention is to provide a technique capable of generating a prediction function by selecting an appropriate combination of features.

  The present invention for solving the above-mentioned problems improves the value of the evaluation function calculated based on the observation data including the objective variable and the explanatory variable and the sparse relationship between the objective variable and each of the explanatory variable. The sparse relationship learning means for learning the sparse relationship, and a prediction function in which all variables are correlated with one of the variables based on the learned sparse relationship And a prediction function learning means for learning.

  The present invention for solving the above-mentioned problems improves the value of the evaluation function calculated based on the observation data including the objective variable and the explanatory variable and the sparse relationship between the objective variable and each of the explanatory variable. A sparse relationship learning step for learning the sparse relationship, and a prediction function in which all variables are correlated with any one of the variables based on the learned sparse relationship And a prediction function learning step for learning.

  The present invention for solving the above-mentioned problems is a program of a prediction function generation device, wherein the program uses the prediction function generation device for each of observation data including an objective variable and an explanatory variable, and the objective variable and the explanatory variable. The sparse relationship learning means for learning the sparse relationship and the learned sparse relationship so that the value of the evaluation function calculated based on the sparse relationship between On the basis of this, it is characterized by functioning as a prediction function learning means for learning a prediction function in which all variables are correlated with any one of the variables.

  According to the present invention, it is possible to generate a prediction function by selecting an appropriate combination of features in a realistic time even for a combination of features that exponentially increases when the number of features is large.

The block diagram which shows the prediction function production | generation apparatus of one Embodiment of this invention. The flowchart which shows an operation | movement procedure. (A) And (b) is a figure of the model which shows a relationship.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 shows a prediction function generation device (feature selection system) according to an embodiment of the present invention. The feature selection system 100 includes sparse relationship learning means 102 and prediction function learning means 103. The feature selection system 100 can be configured by a computer system that operates by a program operation. Functions of each unit in the feature selection system 100 can be realized by a computer operating according to a predetermined program.

  The input device 101 takes in data to be calculated. Data captured by the input device 101 includes an explanatory variable X and an objective variable Y. The input device 101 inputs the acquired data to the feature selection system 100. The observation data storage unit 111 stores observation data input from the input device 101.

  The sparse relationship learning unit 102 reads the observation data from the observation data storage unit 111 and learns a sparse relationship with the observation data. Here, a sparse relationship refers to a relationship in which there is no correlation between all elements between data, and there is no correlation between some elements. The sparse relationship learning unit 102 stores data representing the learned sparse relationship model in the learning result storage unit 112. Note that the observation data storage unit 111 and the learning result storage unit 112 are only required to be referenced from the feature selection system 100, and may be inside or outside the feature selection system 100.

  The sparse relationship learning unit 102 reads the observation data from the observation data storage unit 111 and also reads the learning result from the learning result storage unit 112. Based on the observation data and the learned sparse relationship, the sparse relationship learning unit 102 The value of the evaluation function set to is calculated. Here, the evaluation function is a function for quantifying the distance between the distribution of the input data and the distribution of the learning model.

  The sparse relationship learning means 102 learns a sparse relationship so that the value of the calculated evaluation function is improved. The feature selection system 100 repeatedly performs sparse relationship evaluation and learning until the learning result converges. When the learning result converges, the sparse relationship learning unit 102 notifies the prediction function learning unit 103 that learning has ended. When the learning is completed, the prediction function learning unit 103 reads the learning result of the sparse relationship from the learning result storage unit 112, and learns the prediction function based on the sparse relationship learned by the sparse relationship learning unit 102. . The prediction function learning unit 103 outputs the learned prediction function via the output device 104.

  FIG. 2 shows an operation procedure. The input device 101 takes in data to be calculated (observation data) (step S1). The sparse relationship learning unit 102 calculates the value of the evaluation function based on the observation data and the learned sparse relationship (step S2). The learning result storage unit 112 stores an initial value of a sparse relationship before the sparse relationship learning unit 102 performs learning. The sparse relationship learning means 102 reads the initial value of the arbitrarily set sparse relationship from the learning result storage unit 112 and performs an evaluation using the initial value when performing the evaluation in step S2 at the first execution. Calculate the value of the function. The sparse relationship learning means 102 estimates random variables (hidden variables) that are not directly observed when calculating the value of the evaluation function.

  The sparse relationship learning means 102 learns the sparse relationship so that the calculated evaluation function value is high (S3). When the sparse relationship learning means 102 evaluates the sparse relationship in step S2, the sparse relationship learning means 102 estimates a random variable that is not directly observed based on the learning result of the previous sparse relationship. In step S3, the sex learning means 102 learns a sparse relationship different from the previous learning result. The feature selection system 100 repeatedly performs the evaluation of the learning result in step S2 and the learning in step S3 until the learning result converges.

  When the learning result converges, the prediction function learning unit 103 reads the learning result from the learning result storage unit 112, and learns the prediction function based on the sparse relationship learned by the sparse relationship learning unit 102 (step S4). ). The prediction function learning unit 102 outputs the prediction function learned in step S4 via the output device 104 (step S5).

The evaluation and learning of the sparse relationship between S2 and S3 will be described in more detail. For example, a mixed model of Chow-Liu tree (maximum spanning tree probability model) can be used as the model representing the sparse relationship. This is a method of considering the simultaneous distribution between variables showing a strong correlation when modeling the relationship between variables. The Chow-Liu tree mixed model is represented by the following formula 1 as a probability model.

  In Equation 1, X is the observed data, p (X) is the probability distribution of the observed data, K is the number of components, πi is the mixing probability of components, and p (Ti) is the probability of the tree structure (sparse relationship) in component i. The distribution, p (θi | Ti) is the probability distribution of the parameter under the given tree structure in component i, and p (X | θi, Ti) is under the given tree structure and parameter in component i. Represents the probability distribution of the observed data. Here, the tree structure is a structure in which a line is drawn between variables having a large partial correlation, and the remaining partial correlations are made zero. The parameter is, for example, the average and variance of each variable. The mixing probability is a weight for each tree when the tree is viewed as one component.

  When the objective variable and the explanatory variable are given, the sparse relationship learning means 102 uses them as multivariate observation data, and estimates the probability model of the maximum spanning tree, for example. Estimating the tree structure, parameters, and mixing probabilities is “learning sparse relationships”, and “evaluating sparse relationships” estimates the specific components to which observation data belongs when they are fixed. Do. In the sparse relationship evaluation, when the component to which the observed data belongs is estimated, the likelihood difference between the distribution of the input data and the distribution using the learned sparse relationship model (between distributions) Distance) is calculated as the value of the evaluation function. Here, the simultaneous distribution is given in order from among the variables having strong correlations while restricting the relationship between the objective variable and the explanatory variable so as not to form a loop. This simultaneous distribution can be expressed using a function called a copula. This copula can take into account non-linear correlations and can also represent rank correlations such as Kendall's tau and Spearman's low. In addition, there is an index called a tail dependency coefficient that represents a dependency relationship at the bottom of the distribution, and it is possible to select features based on the strength of the correlation. Note that the method described in any of Non-Patent Documents 2 to 4 can be used for the probability model.

  The feature selection system 100 performs sparse relationship evaluation and learning until the learning result converges. For example, the feature selection system 100 determines that the learning result has converged when the value of the evaluation function does not change even when the number of learnings is increased, or when the change is smaller than a predetermined threshold, and the sparse relationship End sex assessment and learning. By repeatedly evaluating and learning the sparse relationship, a sparse relationship including the objective variable and the entire explanatory variable is obtained. Learning a function from an explanatory variable to an objective variable based on this relationship corresponds to “learning a prediction function”.

  As a result, the prediction function learning unit 103 obtains a sparse relationship including the objective variable and the entire feature, selects a feature used for prediction of the target variable based on this sparse relationship, and learns the prediction function. At this time, a regression equation using a copula can be constructed and used as a prediction function.

FIG. 3 shows a model representing the relationship. Y is an objective variable, and X _{1 to} X ₄ are explanatory variables. FIG. 3A shows a model used in Non-Patent Document 1. In this model, the explanatory variables Xi and Xj (i ≠ j) are all connected. If the number of data is small, the estimation accuracy does not increase so much, and if there is a strong correlation between Xi and Xj, the calculation is Sometimes it broke down. FIG. 3B shows a model used in this embodiment. As input data, five variables including Y and X _{1 to} X ₄ are input. Of these five variables, there are 5 × 4/2 = 10 combinations between two variables. With respect to all of these 10 combinations, the magnitude of correlation is obtained through the processing of S2 and S3 described above. Connect the variables with lines in descending order of correlation. However, the loop should not be allowed at this time. The process is complete when all variables are connected by at least one line. Then, at S4, selects the Y directly tethered variables of X ₁ to X ₄ determines characterized give a great influence on Y. It correlation of strong influence of X ₄ to X _3, the influence of X ₃ is included in X _2, it comes to may be used X ₂ and X ₁ is when estimating Y.

  Although the present invention has been described with reference to the embodiments and examples, the present invention is not necessarily limited to the above-described embodiments and examples, and various modifications can be made within the scope of the technical idea. I can do it.

  The present invention can be used as follows. In finance and economy, a nonlinear correlation structure is observed in the behavior between various measured quantities. For example, if a company's stock price falls, the stock prices of other related companies will also fall. Usually these correlations are often linear and not very large. However, for example, in a recession or a financial crisis, there is an extreme situation that shows a non-linear correlation in which the stock price of a certain company drops significantly, and the stock price of another related company also drops significantly. In such a case, if a conventional method is used to predict a stock price of a certain company, an extreme situation cannot be expressed sufficiently and the prediction performance deteriorates. In addition, there are many companies that become features (explanatory variables) in the prediction, and it is difficult to select an appropriate feature. These problems can be solved by using the proposed method.

100: Feature selection system (prediction function generator)
101: input device 102: sparse relationship learning means 103: prediction function learning means 104: output device 111: observation data storage unit 112: learning result storage unit

Claims (5)

Learning the sparse relationship so that the value of the evaluation function calculated based on the observation data including the objective variable and the explanatory variable and the sparse relationship between each of the objective variable and the explanatory variable is improved. Sparse relationship learning means,
A prediction function generation device comprising: a prediction function learning unit that learns a prediction function in which all variables are correlated with one of the variables based on the learned sparse relationship.   The prediction function generation apparatus according to claim 1, wherein the prediction function learning unit forms a correlation between variables in descending order of the correlation. Learning the sparse relationship so that the value of the evaluation function calculated based on the observation data including the objective variable and the explanatory variable and the sparse relationship between each of the objective variable and the explanatory variable is improved. Sparse relationship learning step,
A prediction function generation method comprising: a prediction function learning step that learns a prediction function in which all variables are correlated with one of the variables based on the learned sparse relationship.   The prediction function generation method according to claim 3, wherein the prediction function learning step forms a correlation between variables in descending order of the correlation. A program for a prediction function generation device, the program comprising:
Learning the sparse relationship so that the value of the evaluation function calculated based on the observation data including the objective variable and the explanatory variable and the sparse relationship between each of the objective variable and the explanatory variable is improved. Sparse relationship learning means,
A program that functions as a prediction function learning unit that learns a prediction function in which all variables are correlated with one of the variables based on the learned sparse relationship.

Images