JP2014041566A - Device, method, and program for linear regression model estimation - Google Patents

Abstract

PROBLEM TO BE SOLVED: To estimate more reliable linear regression model.SOLUTION: A linear regression modeling unit 31 models a linear regression model which represents an observation value y by an explanatory variable x and a linear sum of regression coefficients β for the explanatory variable, a regression coefficient estimation unit 32 estimates a regression coefficient β for each of parameters (λ, λ) designating magnitudes of a penalty term by minimization of a cost function in which a penalty term of which the magnitude is larger for a smaller norm of the explanatory variable x is added to a residual between the linear regression model and the observation value, and a model identification unit 33 selects a regression coefficient which maximizes a contribution rate when used, to identify the linear regression model.

Description

  The present invention relates to a linear regression model estimation apparatus, method, and program.

  The linear regression model captures actual observed data fluctuations as a linear sum of explanatory variables. Modeling algorithms represented by the least square method are easy to implement, and the estimated regression coefficients represent partial correlations between observed data and explanatory variables, and are easy to interpret. ing. However, in actual analysis, the number of explanatory variables increases, and modeling with the least square method becomes impossible. For this reason, a modeling method with a penalty term has been proposed in which a penalty term is attached to the magnitude of the regression coefficient and a regression model soft-fitted to the data is estimated.

  For example, in the estimation of the linear regression model, the cost function considering the L1 and L2 norms of the regression coefficient in addition to the square error with the observation data is used, and only the explanatory variables having a high correlation with the observation data are used for the data. Regression modeling of variable selection for soft fitting has been proposed (see, for example, Non-Patent Document 1). This method is called Elastic Net Regression, and an effective algorithm called LARS (Least Angle Regression) is often used for estimating regression coefficients.

Hui Zou and Trevor Hastie, "Regularization and Variable Selection via the Elastic Net", J.R.Statist.Soc.B, pp0301-320, 2005.

  An object of the present invention is to provide a linear regression model estimation apparatus, method, and program capable of estimating a more reliable linear regression model.

  In order to achieve the above object, a linear regression model estimation apparatus according to the present invention includes an explanatory variable and a linear regression model that represents an observed value by a linear sum of regression coefficients for the explanatory variable. Regression coefficient estimation means for estimating the regression coefficient by minimizing a cost function with a penalty term that increases as the norm of the explanatory variable increases in the residual, and for each parameter that specifies the size of the penalty term Model identifying means for identifying the linear regression model by selecting a regression coefficient that provides the largest explanation rate when using each regression coefficient from the regression coefficients estimated by the regression coefficient estimation means, Can be configured.

  According to the linear regression model estimation apparatus of the present invention, first, a linear regression model that represents an observation value by a linear sum of an explanatory variable and a regression coefficient with respect to the explanatory variable is defined. The regression coefficient estimation means estimates the regression coefficient by minimizing the cost function with a penalty term that increases as the norm of the explanatory variable decreases to the residual between the linear regression model and the observed value. The linear regression model is selected by selecting the regression coefficient that provides the highest explanation rate when using each regression coefficient from the regression coefficients estimated by the regression coefficient estimation means for each parameter that specifies the size of the penalty term. Is identified.

  In this way, to estimate the regression coefficient by minimizing the cost function with a penalty term that increases as the norm of the explanatory variable directly related to the reliability as data becomes smaller, a highly reliable linear regression model must be estimated. Can do.

  The linear regression model estimation method of the present invention is a linear regression model estimation method in a linear regression model estimation device including a regression coefficient estimation means and a model identification means, wherein the regression coefficient estimation means includes an explanatory variable and the In a linear regression model that represents an observation value as a linear sum of regression coefficients with respect to an explanatory variable, a cost function with a penalty term attached to the residual between the linear regression model and the observation value that increases as the norm of the explanatory variable decreases. The regression coefficient is estimated by minimization, and the model identification unit uses each regression coefficient from the regression coefficient estimated by the regression coefficient estimation unit for each parameter that specifies the size of the penalty term. In this method, the linear regression model is identified by selecting a regression coefficient that maximizes the explanation rate.

  The linear regression model estimation program of the present invention is a program for causing a computer to function as each means constituting the linear regression model estimation apparatus.

  As described above, according to the linear regression model estimation apparatus, method, and program of the present invention, the cost function with a penalty term that becomes larger as the norm of the explanatory variable directly related to the reliability as data becomes smaller is minimized. Thus, since the regression coefficient is estimated, an effect that a highly reliable linear regression model can be estimated is obtained.

It is the schematic which shows the structure of the linear regression model estimation apparatus which concerns on this Embodiment. It is a figure which shows an example of the data structure of the time series data of an observation value. It is a flowchart which shows the content of the linear regression model estimation process routine in this Embodiment. It is a graph which shows the observation data used for experiment. Is a graph showing explanatory variables (x ₁₎ used in the experiment. Is a graph showing the explanatory variable (x ₄₎ used in the experiment. Is a graph showing explanatory variables (x ₇₎ used in the experiment. It is a table | surface which shows the estimation result of a regression coefficient.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Outline of the present embodiment>
In the linear regression model estimation apparatus according to the present embodiment, first, a linear regression model that explains fluctuations in observed values is assumed and prepared. Here, it is assumed that the observed value is standardized to mean 0, the explanatory variable is standardized to mean 0, and variance 1. Next, in Elastic Net Regression, prepare a cost function that penalizes the size (scale) of the explanatory variable. In the present embodiment, this technique is called Scale Penalized Elastic Net Regression (SPEN). Then, an estimated value of the regression coefficient that minimizes the cost function is obtained by taking in the observed value. In the estimation of the regression coefficient, a modification of LARS can be used similarly to the estimation of Elastic Net Regression. Finally, the model is identified by the explanation rate (determination coefficient) of the regression model, and the estimated values of the parameters of the linear regression model are output.

<System configuration>
The linear regression model estimation apparatus 10 according to the present embodiment is configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing a linear regression model estimation processing routine described later. As shown in FIG. 1, this computer can be functionally represented by a configuration including an input unit 20, a calculation unit 30, and an output unit 40.

The input unit 20 receives time-series data of input observation values. The observed value is a scalar value, that is, the time series data is a one-dimensional time series. For example, as shown in FIG. 2, the observed value is a set of time t _i , observed value y _i, and explanatory variable x _{i, j} (i = 1,..., N, j = 1,. p) is time-series data.

  The calculation unit 30 can be expressed by a configuration including a linear regression modeling unit 31, a regression coefficient estimation unit 32, and a model identification unit 33.

  The linear regression modeling unit 31 receives the set value input from the input unit 20 by the operator, and defines the linear regression models shown in the following equations (1) and (2).

Where y is a vector of n × 1 observation values, X is a matrix of n × p explanatory variables, and β is a vector of p × 1 regression coefficients. x _{i, i = 1,... p} are p explanatory variables, each of which is an n × 1 vector. When this linear regression equation is standardized, the following equations (3) and (4) are obtained.

Here, || X _{i, i = 1,... P} || is the L ₂ norm of each explanatory variable, and ￣A (“￣” on symbol A in the equation) is the average of A. . (4) has an average of the observed values 0, since it has an average be 0, L ₂ norm even 1 for each explanatory variable, called the standardized linear regression model with this format. The purpose of this embodiment is to estimate the regression coefficient β ′ of the equation (4) from time series data of observed values.

  The regression coefficient estimation unit 32 estimates the parameter β ′ that minimizes the cost function of Scale Penalized Elastic Net Regression (SPEN) shown in the following formula (5) by optimization shown in the following formula (6).

The second term on the right side of equation (5) is the L ₁ penalty term, the third term is the L ₂ penalty term, and λ ₁ and λ ₂ are parameters that specify the size of the L ₁ penalty term and the L ₂ penalty term. . W is a diagonal matrix having a function of the size (scale) of the explanatory variable as a diagonal component as shown in the following equation (7).

However, w (|| x _p ||) is a function of the L ₂ norm of x _p and is defined as a function that approaches 1 (that is, there is no penalty) when the norm is large and becomes large when the norm is small (the penalty is large). To do. For example, w (|| x _p ||) can be assumed in the form shown in the following equation (8), and γ = 50 and m = 3.

  The cost function of the above equation (5) is transformed into the following equation (9).

Here, β ″ = G ⁻¹ β ′, and G is represented by the following equation (10).

  Further, y ″ and x ″ are enlargements of y ′ and x ′, and are respectively expressed by the following formula (11).

Be careful, however, "a, which (5) of the _{L 1} are substituted into penalty term | f (W) Gβ" | β '= Gβ next, as f (W) G = I ( 9) formula _{L 1} penalties term | beta "|. lies in that the words, equation (5) f (W) becomes the following equation (12).

(9) the form of the cost function is in the form of the same type Lasso only L ₁ penalty term (Least Absolute Shrinkage and Selection Operator) , it is possible to estimate the beta "using the deformation of LARS. Observations The time series data is taken in, β ″ is estimated, and β ″ → β ′ → β is converted to estimate the regression coefficient of the linear regression model in equation (1). Also, in equation (9) When the following equation (13) is considered as the cost function minimization, the estimated regression coefficients can all be set to 0 or more: The cost function can be minimized by the modification of LARS, and the regression coefficient is 0. This is equivalent to performing an iterative calculation of LARS using only the explanatory variables that become larger.

The model identification unit 33 assigns λ ₁ and λ ₂ in the equation (5), and selects the model having the highest explanation rate (determination coefficient) of the regression model. For example, the explanation rate (adjusted R ² ) defined by the following (14) can be used as the explanation rate.

  The numerator of the two items of the equation (14) is the sum of squares of the residuals of the linear regression model, and the denominator is the sum of squares of deviations from the average of the observed values. n is the number of observation values, and p is the number of explanatory variables of the model. The explanation rate adjusted for the degree of freedom is a scale obtained by adjusting the degree of fitting of the regression model to the observed value by the degree of freedom.

<Operation of linear regression model estimation device>
Next, the operation of the linear regression model estimation apparatus 10 according to the present embodiment will be described. First, when time series data including observation values at times t _{1 to} t _N is input to the linear regression model estimation apparatus 10 by the operator, the input time series data is stored in the memory by the linear regression model estimation apparatus 10. (Not shown). Then, the linear regression model estimation apparatus 10 executes the linear regression model estimation processing routine shown in FIG.

  First, in step 100, time series data of observation values stored in the memory is acquired. In step 102, the linear regression modeling unit 31 uses the explanatory variables included in the time series data to define the linear regression models shown in the above formulas (1) and (2), standardizes them, (4) Define a standardized linear regression model as shown in equation (4).

Next, at step 104, a regression coefficient estimator 32, the explanatory variable size (scale) of the cost function SPEN which gave a penalty term that takes into account (5) the expression of L ₁ penalties section and L ₂ penalties term Parameters (λ ₁ , λ ₂ ) for specifying the magnitudes are set, and in the next step 106, the parameter β ′ is estimated by minimizing the cost function of equation (5) using equation (6).

  Next, in step 108, the model identification unit 33 calculates an explanation rate (determination coefficient) shown in, for example, the equation (14) based on the linear regression model using the parameter β ′ estimated in step 106. , The parameter β ′ is once stored in a predetermined storage area.

Next, in step 110, the regression coefficient estimation unit 32 determines whether the parameter β ′ has been estimated for all combinations of the parameters (λ ₁ , λ ₂ ). If there is an unprocessed (λ ₁ , λ ₂ ), the process returns to step 104, the next (λ ₁ , λ ₂ ) is set, and the processes of steps 106 and 108 are repeated.

When the processing is completed for all (λ ₁ , λ ₂ ), the process proceeds to step 112, and the parameter β ′ when the explanation rate calculated in step 108 is the largest is selected (4 ) Identify a standardized linear regression model of the equation. Also, the selected parameter β ′ is converted into β.

  Next, in step 114, the output unit 40 outputs the parameter β obtained in step 112 as a parameter estimated value of the linear regression model of equation (1), and the linear regression model estimation processing routine is terminated.

<Experimental result>
Here, an experimental result of SPEN, which is a technique according to the present embodiment, using artificial observation data generated by the following equation (15) will be described.

  The four explanatory variables that generated this observation data are independent. Furthermore, six explanatory variable candidates shown in the following equations (16) to (21) are prepared.

  In total, the magnitudes (scales) of the ten explanatory variables are the following formulas (22) and (23), respectively.

In this set of explanatory variables, the explanatory variables are strongly correlated with each other and are data having multicollinearity. In addition, x ₄ , x ₇ , and x ₈ are explanatory variables that are smaller in size (scale) than other explanatory variables and have low reliability as data. Plots of the observed value y and the explanatory variables x ₁ , x ₄ , x ₇ are shown in FIGS.

FIG. 8 shows the result of modeling this data with SPEN which is the method of the present embodiment. The values of λ ₁ and λ _{2 in} the equation (5) are determined by the processing of the model identification unit 33 described above, and are λ ₁ = 0.1651 and λ ₂ = 0.0131. The comparison method is Ridge and Elastic Net (EN). Ridge the regression only L ₂ penalties term of the regression coefficients is a technique for soft-fitted to the observed data. It matches when λ ₁ of SPEN is 0 and W = I. EN is a technique when SPEN W = I, and does not consider penalties due to the scale of the explanatory variables. Ridge can stably estimate the regression coefficient even when multicollinearity exists, but it can be seen that there are many false estimations. EN is made up estimated regression coefficients may have been estimated regression coefficients erroneous for explanatory variable x ₇ and x _8.

On the other hand, in SPEN which is the method of the present embodiment, since the explanatory variable with a small scale is regarded as having low reliability as data, the regression coefficients for the explanatory variables x ₄ , x ₇ , and x ₈ are estimated to be 0. . The explanation rate is almost the same in either method, but the linear regression model estimated by SPEN is closest to the true model, and the penalties for the scale of the explanatory variable, which is the effect of this embodiment, are working well. I understand.

  As described above, according to the linear regression model estimation apparatus according to the present embodiment, the SPEN cost function with a penalty term considering the size (scale) of the explanatory variable directly related to the reliability as data is obtained. Since the regression coefficient is estimated by minimization, a highly reliable linear regression model can be estimated.

  Linear regression models are mathematically simple and easy to interpret, and are used by many analysts, supported by powerful estimation algorithms. However, the actual data has complicated fluctuations, and the number of explanatory variables used for regression increases, making appropriate modeling impossible. Therefore, modeling methods with penalties for regression coefficients have been proposed for the purpose of soft fitting to observation data. In the present embodiment, a linear regression modeling method for penalizing the size (scale) of explanatory variables has been proposed. This allows modeling using only explanatory variables with sufficient size (scale) as explanatory variables, that is, highly reliable explanatory variables as data, and estimation of robust regression equations from complex data sets. Will be able to do.

  Note that the present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

  In the present specification, the embodiment has been described in which the program is installed in advance. However, the program can be provided by being stored in a computer-readable recording medium.

DESCRIPTION OF SYMBOLS 10 Linear regression model estimation apparatus 20 Input part 30 Operation part 31 Linear regression modeling part 32 Regression coefficient estimation part 33 Model identification part 40 Output part

Claims (3)

In a linear regression model that represents an observed value by a linear sum of an explanatory variable and a regression coefficient for the explanatory variable, a penalty term that increases as the norm of the explanatory variable decreases is added to the residual between the linear regression model and the observed value. A regression coefficient estimating means for estimating the regression coefficient by minimizing the cost function;
The linear regression is selected by selecting a regression coefficient having the highest explanation rate when using each regression coefficient from the regression coefficients estimated by the regression coefficient estimation means for each parameter that specifies the size of the penalty term. A model identification means for identifying the model;
An apparatus for estimating a linear regression model. A linear regression model estimation method in a linear regression model estimation apparatus including a regression coefficient estimation unit and a model identification unit,
In the linear regression model in which the regression coefficient estimation means represents an observation value as an explanatory variable and a linear sum of the regression coefficients for the explanatory variable, a norm of the explanatory variable is small in a residual between the linear regression model and the observation value Estimate the regression coefficient by minimizing the cost function with a penal term that increases
The model identification unit selects a regression coefficient that provides the highest explanation rate when each regression coefficient is used from the regression coefficients estimated by the regression coefficient estimation unit for each parameter that specifies the size of the penalty term. A linear regression model estimation method for identifying the linear regression model.   The linear regression model estimation program for functioning a computer as each means which comprises the linear regression model estimation apparatus of Claim 1.

Images