JP2001022722A - Method and device for finding number law conditioned by qualitative variable and storage medium stored with finding program for number law conditioned by qualitative variable - Google Patents

Abstract

PROBLEM TO BE SOLVED: To find a number law conditioned by a qualitative variable even if an original data distribution is based upon different polynomials by partial spaces when the relation that an input vector consisting of multiple qualitative variables and quantitative variables and a target output value have is find as a number law conditioned by a qualitative variable as to a set of instances consisting of the input vector and the target output value of the quantitative variable value. SOLUTION: By this number law finding method, a learning target function is set by adding a term obtained by multiplying the sum of squares of the combination weight of a neural network by a normalization coefficient to the term of the sum of square errors between target output values of respective instances and the output value of the neural network, which is learnt by using a secondary learning method; and the number of intermediate units of the neural network and the normalization coefficient are determined by using a crossing verifying method and a neural network as the learning result is converted into a number law conditioned by a qualitative variable by using a rule decomposing method.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and an apparatus for discovering a number law conditional on qualitative variables, and a storage medium storing a program for finding a number law conditional on qualitative variables. For a set of examples consisting of an input vector consisting of variable values and quantitative variable values and a target output value of a quantitative variable value, qualitatively qualify the relationship lurking between the input vector and the target output value. The present invention relates to a method and apparatus for discovering a number law conditioned by a qualitative variable to be discovered as a number law obtained, and a storage medium storing a program for finding a number law conditioned by a qualitative variable.

[0002] Supporting scientific discovery is extremely useful not only in academic fields but also in a wide range of industries.
The task of discovering number laws from data is central. For example, if a rule discovery system is used, Kepler's third law T = kr ^3/2 can be discovered from numerical data.

[0003]

2. Description of the Related Art As a method of discovering a number law, a pioneering study of the BACON system [P. Langley, HA Simon,
G. Bradshaw, J. Zytokow: "Scientific discovery: co
mputational explorations of the creative process ".
MIT Press (1987)], followed by various law discovery methods [P. Langl
ey, J. Zytkow: "A robust approach to numeric disco
very ", Proc. seventh International Machine Leaning
Conference, 411-418 (1990)], [C. Schaffer: "Bivari
ate scientific function finding in a sampled, real-
data testbed ", Machine Learning, 12 (1/2/3): 167-18
3 (1993)].

[0004]

However, in the search of the above proposal, a combination explosion occurs. There are drawbacks such as the need to define an appropriate function in advance and the vulnerability to data noise. As described above, in the conventional method, a single polynomial rule is applied. However, in a case where the original data distribution follows different polynomials for each subspace, an attempt to apply a single polynomial does not work.

For example, the clone law F = 4πεq ₁ q
₂ / γ ² defines the magnitude of the force F acting between the charges q ₁ and q ₂ separated by γ. This law defines the dielectric constant ε of the medium as a proportionality constant. For example, in water, F = 8897.3
52q ₁ q ₂ / γ ² , F = 111.280 in air
q ₁ q ₂ / γ ² . In such a case, separate rules should be prepared for each subspace.

The present invention has been made in view of the above points, and has an input vector composed of a plurality of qualitative variable values and quantitative variable values, and a set of cases composed of target output values of the quantitative variable values. In contrast, when finding the relationship lurking between the input vector and the target output value as a qualitative variable and a conditional law, when the original data distribution follows a separate polynomial for each subspace However, the present invention provides a method and apparatus for discovering a number law conditioned by a qualitative variable, and a storage medium storing a program for finding a number law conditional on a qualitative variable. The purpose is to:

[0007]

FIG. 1 is a diagram for explaining the principle of the present invention. The present invention (claim 1) provides an input vector composed of a plurality of qualitative variable values and quantitative variable values, and a set of cases composed of target output values of quantitative variable values. In a method of finding a number law conditioned by qualitative variables to discover the relationship lurking between target output values as a number law conditioned by qualitative variables, the square of the target output value and the output value of the neural network in each case is used. A learning objective function is set by adding a term obtained by multiplying the sum of squares of the connection weights of the neural network by the regularization coefficient to the term of the error sum (step 1), and learning of the neural network is performed using the secondary learning method. (Step 2), determine the number of intermediate units of the neural network and the regularization coefficient using the cross-validation method (Step 3), and use the rule decomposition method to condition the neural network of the learning result with qualitative variables. Into a number law eclipsed (Step 4).

According to the present invention (claim 2), as a rule decomposition method, zero shift of a non-variable weight, extraction of a basic rule, and synthesis of the basic rule are performed. FIG. 2 is a diagram illustrating the principle of the present invention. The present invention (Claim 3) provides an input vector composed of a plurality of qualitative variable values and quantitative variable values, and a set of cases composed of target output values of quantitative variable values. A numerical law finding apparatus conditioned by a qualitative variable for discovering a relation lurking between target output values as a numerical law conditioned by a qualitative variable, wherein the target value and the output value of the neural network in each case are obtained. A learning objective function setting means 10 for setting a learning objective function by adding a term obtained by multiplying a sum of squares of connection weights of a neural network by a regularization coefficient to a term of a square error, and a neural learning method using a secondary learning method A neural network learning means 20 for learning the net; a model determining means 30 for determining the number of intermediate units of the neural network and a regularization coefficient using a cross-validation method; and a learning result using a rule decomposition method. The number rule conversion means 4 for converting the neural net number law is conditioned by qualitative variables
0.

According to the present invention (claim 4), the rule decomposition method used in the number law conversion means includes a zero shift of a non-variable weight,
The basic rules are extracted and the basic rules are synthesized. The present invention (claim 5) provides an input vector composed of a plurality of qualitative variable values and quantitative variable values, and a set of cases composed of target output values of the quantitative variable values. A storage medium storing a discovery program of a number law conditioned by qualitative variables for discovering a relationship lurking between target output values as a number law conditioned by qualitative variables. A learning objective function setting process for setting a learning objective function by adding a term obtained by multiplying a sum of squares of connection weights of the neural network by a regularization coefficient to a term of a sum of square errors of output values of the neural network, and a secondary learning method , A neural network learning process for learning the neural network, a model determination process for determining the number of intermediate units and a regularization coefficient of the neural network using a cross-validation method, And a number rule conversion process for converting Lumpur decomposition neural network learning result using several law is conditioned by the qualitative variables.

According to the present invention (claim 6), as a rule decomposition method used in the number law conversion process, zero shift of non-variable weight, extraction of basic rules, and synthesis of the basic rules are performed. As described above, according to the present invention, learning is performed by adding a term obtained by multiplying a sum of squares of connection weights of a neural network by a regularization coefficient to a term of a square error between a target output value and an output value of a neural network in each case. The objective function is set, the secondary learning of the neural network is performed, the number of intermediate units of the neural network and the regularization coefficient are determined using the cross-validation method, and the number rule conditioned by the qualitative variables of the learning result neural network. Convert to At this time, as a rule division method, zero shift of the non-variable weight, extraction of basic rules, and synthesis of basic rules are performed.

Thus, even when the original data distribution follows different polynomials for each subspace, it is possible to find a number law conditional on qualitative variables.

[0012]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS FIG. 3 shows a configuration of a number law finding apparatus according to the present invention. The number law finding device includes a learning objective function setting unit 10, a learning unit 20, a coefficient determination unit 30, and a number law conversion unit 40. The learning objective function setting unit 10 calculates a term obtained by multiplying the sum of the square errors of the neural network connection weights by the regularization coefficient by the sum of the square errors of the target output value and the output value of the neural network in each case of the input case set. Is added to set the learning objective function.

The learning section 20 learns a neural network by a secondary learning method. The coefficient determining unit 30 determines the number of intermediate units and the regularization coefficient of the neural network using the cross-validation method. The number-law conversion unit 40 converts the learning result into a number law conditioned by the qualitative variables of the neural network. The conversion includes a rule decomposition method such as a zero-shift method of a constant weight, an extraction method of a basic rule, and a basic rule. Is used.

Hereinafter, the above components will be described in detail. 1. Basic framework: Each sample is a set of variables (q ₁ , ...,
q _K1, x _1, ..., and is represented by x _K2, y). q _k is a qualitative variable, x _k is a quantitative variable, and y is a reference variable. Also, let L _k be the number of possible categories. For convenience of processing, q _k is _represented by the following binary dummy variable q _kl .

[0015]

(Equation 1)

Here, it is assumed that the true data model has the following rules (γ = 1,..., R). The condition part of each rule is a logical product of qualitative variables. However,

[0017]

(Equation 2)

Is a set of dummy variables appearing in the condition part.

[0019]

(Equation 3)

The execution unit includes a parameter

[0021]

(Equation 4)

Let be a polynomial having

[0023]

(Equation 5)

2. Numerical expression of conditional part In order to numerically express the conditional part, the following function g is introduced.

[0025]

(Equation 6)

Here,

[0027]

(Equation 7)

Is a vector composed of parameters appearing on the right side of the above equation. Now, this parameter value is set as follows.

[0029]

(Equation 8)

If β is a large positive number, when the condition part is satisfied,

[0031]

(Equation 9)

And if not,

[0033]

(Equation 10)

Exp (−β) 0 becomes effective,

[0035]

[Equation 11]

Thus, the following approximates equation (2) with sufficient accuracy.

[0037]

(Equation 12)

Therefore, the following equation obtained by adding them approximates the true data model with sufficient accuracy. However, Ψ
Is the parameter group

[0039]

(Equation 13)

Consists of

[0041]

[Equation 14]

3. Learning by Neural Network Next, learning by the neural network in the learning unit 20 will be described. Available data

[0043]

(Equation 15)

It is assumed that At this time, the following calculation formula can completely express formula (6) if f is appropriately prepared.

[0045]

(Equation 16)

[0046] Θ is a parameter _{vector, w i, j = 0,} ..., J, v jkl, j = 1, ..., J, k = 1, ..., K 1, l = 1, ..., L k, and _{, w jk, j = 1,} ..., J, k = 1, ..., consisting of K _2. Also, the order is M. Equation (7) is J
This can be interpreted as a feedforward calculation formula of a three-layer neural network having three hidden units.

In the neural network learning, the error function E
It is known that adding a penalty term Ω (Θ) to (Θ) significantly improves generalization performance. Furthermore, the effect of adding unnecessary weights to the penalty term can be expected to be zero, which is convenient from the viewpoint of finding rules. According to the experiment, the combination of the square penalty and the secondary learning algorithm has a remarkable effect of improving the generalization performance. Therefore, the following objective function can be considered. Where λ is a penalty coefficient,
θ _m ∈Θ.

[0048]

[Equation 17]

However, the square penalty having a single penalty coefficient has a drawback that a weight that is not invariant to variable transformation is obtained. In fact,

[0050]

(Equation 18)

In order to obtain an invariant weight for variable conversion as described above, it is necessary to prepare a different penalty coefficient for each weight, but this is not easy because the number of coefficients is too large.
Therefore, here, a penalty is imposed only on the weight from the input layer to the hidden layer. In this way, a weight that is invariant to the above-described variable conversion can be obtained with a single penalty coefficient. That is, the problem of finding the rule according to the equation (7) can be formulated as a problem of finding a parameter that minimizes the following objective function.

[0052]

[Equation 19]

3. Criteria for Model Selection Next, a case will be described in which the coefficient determination unit 30 determines the number of intermediate units of the neural network and the result of the regularization coefficient. With the data given, equation (7)
The optimal number of hidden units J ^{* and} the optimal penalty coefficient λ ^{* in} equation (9) are unknown. To find the optimal model and model parameters that describe the data, J ^*
And criteria for selecting λ ^* are required. In the present invention, the cross-validation method cross-validation is used as this criterion. Cross-validation is often used to evaluate the generalization performance of learning machines such as neural networks. Here, generalization refers to the performance for data not used for learning. In the cross-validation method, given data D is randomly divided into S segments (G _s , s = 1,..., S). And
S-1 segments are used for training and the remaining one is used for (generalization) testing. If the number of segments is equal to the number of samples, it is called leave-one-out. This procedure is repeated S times, and finally the following mean square error MSE _CV is obtained.

[0054]

(Equation 20)

Next, the number law conversion unit 40 will be described. 5. Rule decomposition algorithm It is assumed that one neural net can be selected as the best rule candidate using the above criteria. It needs to be decomposed into the original rule set. The weight W may be used to form a polynomial, but the way of using the weight V is not so simple.
The following procedure is used for rule decomposition. It should be noted that a parameter ε that ignores minute fluctuations is introduced to prevent the occurrence of useless rules.

Zero shift of non-variable weight: The following procedure is repeated for each hidden unit j. When the weight of the qualitative variable k is regarded as no variation as _{max l (v jkl) -min l} (v jk) ≦ ε is a constant sentences weights w _j
And then shift to zero (v _jkl = 0).

[0057]

(Equation 21)

Extraction of basic rules: The following procedure is repeated for each hidden unit j. For a qualitative variable group having fluctuating weights, all combinations of each dummy variable are created and numbered with u. Since a set Q _{j: u} of dummy variables is determined for the u-th combination, the corresponding weight contribution is calculated and multiplied by the weight w ′ _j .

[0059]

(Equation 22)

In this way, the following basic rule having the logical product of the qualitative variables in the condition part can be extracted. | W
_For weights where _{j: u} | <ε, t _{j: u} = 0.

[0061]

(Equation 23)

Synthesis of basic rules: hidden unit j,
The basic rules determined by j 'are synthesized. Composition is performed for all combinations, and the conditional part is composed as a logical product.
The execution unit simply sums.

[0063]

(Equation 24)

A combination whose condition part is always "false" is discarded. If the above combination is repeated until there are no hidden units, and a common constant term w ₀ is added to each execution unit, the combination can be decomposed into a rule such as Expression (2).

[0065]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A description will now be given, with reference to the drawings, of an embodiment of a method for finding a number law conditional on qualitative variables using the above logic. FIG. 4 shows a configuration of a number-law finding device according to an embodiment of the present invention, and FIG. 5 shows a configuration of a learning unit according to an embodiment of the present invention.

A learning unit 20 for learning a neural network using a regularization term includes a setting unit 21 for setting an initial value of a regularization coefficient μ and its lower limit, an initialization unit 22 for initializing connection weights, Stop condition determination unit 2 that determines the stop condition of repetition
3. Search direction calculation unit 2 for calculating the correction direction of the connection weight
4, a search width calculating unit 25 for calculating a correction width of the connection weight, a connection weight updating unit 26 for updating the connection weight, an error calculating unit 27 for calculating the cross-validation error, and an end determination unit 28 for the algorithm.

Here, a group of polynomials conditioned by a qualitative variable q ₂ is considered as artificial data for evaluation of the present invention.

[0068]

(Equation 25)

The qualitative variables are q ₁ , q ₂ , q ₃ , the quantitative variables are x ₁ ,..., X ₉ , and the number of categories L _k is 2, 3, 4. q ₁ , q ₃ , x ₆ ,..., x ₉ are variables unrelated to the rule. Each variable value is randomly generated, but the quantitative variable value is a value in the section [0, 1], and a small normal random number is added to the reference variable value. Set the number of samples to 400 (N
= 400).

In the learning, the initial values of the weights v _{jkl and} w _jk follow a normal distribution having an average of 0 and a standard deviation of 1. The initial value of the weight w _j is 0, and the initial value of the constant term w ₀ is a reference value. The average value of the variable values was used. Learning starts when the gradient is small enough.
(‖ ▽ J (Θ) ‖ ² / M <10 ^-8 ) or when the processing time exceeded 100 seconds, the process was terminated. Penalty factor λ
From 10 × 10 ⁰ to 10 × 10 ^-9 in 10 ^-1 increments. The number of hidden units was changed from 1 to 4 (J = 1, 2, 3, 4). The trial was repeated 10 times for the same J and λ.

FIG. 6 shows the learning result. The error RMSE with respect to the learning data is minimized when J = 4, but the test data (10,000) generated separately from the learning data.
Sample), it becomes minimum when J = 3. From this, it can be inferred that J ^* = 3. When J = 3, an example of the rule discovered in the present invention is shown.

[0072]

(Equation 26)

By applying the above rule decomposition algorithm to this, the following rule was found.

[0074]

[Equation 27]

Unlike the existing method AVACUS, there is no need to prepare a specific function system in advance, and almost the same rule as the original rule can be restored. Another feature of the present invention is that a neural network having one conditional polynomial group can be learned. Next, the present invention is evaluated using vehicle data as real data. Automobile data (using the UCI machine learning database) is data on the specifications and prices of automobiles and trucks imported to the United States in 1985. There are 11 qualitative explanatory variables, 14 quantitative explanatory variables, and a reference variable (price). Except for samples with missing values, 159 samples are available (N = 159). In the conventional analysis, a linear analysis and a k-NN method were applied to obtain MDE = 14.2.
% And 11.8%. However, MDE (mean deviat
ion error)

[0076]

[Equation 28]

Is defined as All data was used for training and for testing. The experimental conditions for applying the present invention were the same as in the previous experiment. However, J = 1, 2, and 3.
In addition, each variable was transformed as follows, but it can be shown that the rule to be found is invariant to this.

[0078]

(Equation 29)

The present invention was applied to three cases, namely, data of only quantitative variables, data of qualitative variables + 1 data of qualitative variables, and data of all variables. As a result, it was found that in the case, over-learning occurred and the generalization performance deteriorated as compared with the case, and that the generalization performance of the case exceeded the case. Qualitative variables used in the case is a manufacturer q _2. In the case, the number of hidden units J = 1
Had the best generalization performance. When j = 1, the rule discovered by the present invention is shown. It is easy to break this down into rules. The MDE for this rule was 8.4%. In addition,
The k-th manufacturer (q _2k ) was simply denoted as q _k .

[0080]

[Equation 30]

Although the above description has been made based on the configurations shown in FIGS. 3 and 4, the learning objective function setting unit 10, the learning unit 20, the coefficient determination unit 30, and the number law conversion unit 4 shown in FIGS.
0 is constructed as a program and stored in a portable storage medium such as a disk device connected to a computer used as a number law finding device, a floppy (registered trademark) disk, or a CD-ROM. The present invention can be easily realized by installing the software at the time of installation.

The present invention is not limited to the above embodiments, but can be variously modified and applied within the scope of the claims.

[0083]

As described above, according to the present invention, an input vector composed of a plurality of qualitative variable values and quantitative variable values,
When the relationship between the input vector and the target output value is discovered as a number law conditioned on qualitative variables for a set of cases consisting of A learning objective function is set by adding a term obtained by multiplying the sum of squares of the connection weights of the neural network by the regularization coefficient to the term of the square error of the output value of the neural network and the secondary learning of the neural network. Then, the number of intermediate units of the neural network and the regularization coefficient are determined by using the cross-validation method, and the result is converted into a number rule conditioned by qualitative variables of the learning result neural network. At this time, as a rule division method, zero shift of the non-variable weight, extraction of basic rules, and synthesis of basic rules are performed.

As a result, even when the original data distribution follows different polynomials for each subspace, it is possible to find a number law conditional on qualitative variables.

[Brief description of the drawings]

FIG. 1 is a diagram for explaining the principle of the present invention.

FIG. 2 is a principle configuration diagram of the present invention.

FIG. 3 is a configuration diagram of a number law finding device of the present invention.

FIG. 4 is a configuration diagram of a number law finding device according to an embodiment of the present invention.

FIG. 5 is a configuration diagram of a learning unit according to an embodiment of the present invention.

FIG. 6 shows a result of applying one embodiment of the present invention to artificial data.

[Explanation of symbols]

 Reference Signs List 10 learning objective function setting means, learning objective function setting unit 20 neural network learning means, learning unit 21 setting unit 22 initialization unit 23 stop condition determination unit 24 search direction calculation unit 25 search width calculation unit 26 connection weight update unit 27 error calculation Unit 28 end determination unit 30 model determination unit, coefficient determination unit 40 number law conversion unit, number law conversion unit

Claims (6)

[Claims] 1. An input vector composed of a plurality of qualitative variable values and quantitative variable values, and a set of cases composed of target output values of quantitative variable values, the input vector and the target output value In the method of finding the number law conditioned by qualitative variables to discover the relationship between them as a number law conditioned by qualitative variables, the method of finding the sum of the squared error between the target output value and the output value of the neural network in each case A learning objective function is set by adding a term obtained by multiplying the sum of the squares of the connection weights of the neural net by the regularization coefficient to the term, and the neural network is trained using the secondary learning method, and the cross-validation method is used. The number of intermediate units of the neural network and the regularization coefficient are determined, and the neural network resulting from the learning is converted into a number law conditional on qualitative variables using a rule decomposition method. Number law discovery methods of which is conditioned on. 2. The method according to claim 1, wherein the rule decomposition method includes zero-shifting a non-variable weight, extracting a basic rule, and synthesizing the basic rule. 3. An input vector composed of a plurality of qualitative variable values and quantitative variable values, and a set of cases composed of target output values of the quantitative variable values, wherein the input vector and the target output value A numerical law finding device conditioned by a qualitative variable for discovering the relationship between them as a numerical law conditioned by a qualitative variable, comprising a sum of a squared error between a target output value and a neural network output value in each case. A learning objective function setting means for setting a learning objective function by adding a term obtained by multiplying the sum of squares of the connection weights of the neural network by the regularization coefficient to the term, and learning the neural network using the secondary learning method A neural network learning means, a model determining means for determining the number of intermediate units and a regularization coefficient of the neural network using a cross-validation method, and a neural network learning result using a rule decomposition method. The number rule discovery device is conditioned by the qualitative variables and having a number laws conversion means for converting the number law is conditioned by the qualitative variables. 4. The number rule finding apparatus conditioned on qualitative variables according to claim 3, wherein the rule decomposition method used in the number rule conversion means performs zero shift of a constant weight, extracts a basic rule, and synthesizes the basic rule. . 5. An input vector composed of a plurality of qualitative variable values and quantitative variable values, and a set of cases composed of target output values of the quantitative variable values. A storage medium that stores a program for finding a number law conditioned by qualitative variables for discovering the hidden relationship between them as a number law conditioned by qualitative variables, the target output value and the output of the neural network in each case A learning objective function setting process for setting a learning objective function by adding a term obtained by multiplying the sum of the squares of the connection weights of the neural network by the regularization coefficient to the term of the sum of square errors of the values, A neural network learning process for learning the neural network, a model determination process for determining the number of intermediate units and a regularization coefficient of the neural network using a cross-validation method, and a rule. Storage medium storing a number laws discovery programs is conditioned by qualitative variables, characterized in that it comprises a number law conversion process of converting the neural net learning result in several law is conditioned by the qualitative variables using a solution. 6. The discovery of a number law conditioned on qualitative variables according to claim 5, wherein the rule decomposition method used in the number law conversion process includes zero shift of a constant weight, extraction of a basic rule, and synthesis of the basic rule. A storage medium that stores programs.