JPH0973438A - Device and method for statistical learning, and information storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a regression curve estimation part to estimate an unknown system with a minimum error by generating data (x) minimizing the learning error of the regression curve estimation part and setting it to a learning data output part. SOLUTION: The regression curve estimation part 3 has a linear model 'f(x:θ)=Σiθifi(x)' having an M-dimensional parameter θ set previously in a model storage part 8 so as to estimate the unknown system 7 as a regression curve 'E(p(y1x))', and has a parameter estimation part 9 which estimates the M-dimensional parameter θ of the linear model. A learning data output part 5 outputs a vector (y) outputted as, a vector (x) inputted to the unknown system 7 to a regression curve setting part 3 as learning data (x, y), and a learning data generation part 6 generates the vector (x) minimizing the learning error of the regression curve estimation part 3 and sets it to the learning data output part 5. Therefore, the unknown system 7 outputs a K-dimensional vector (y) to an L-dimensional vector (x) by operating a function and adding Guassian noise.

Description

Detailed Description of the Invention

[0001]

TECHNICAL FIELD The present invention relates to a statistical learning device, a statistical learning method, and an information storage medium.

[0002]

2. Description of the Related Art Currently, a statistical learning device for externally estimating an unknown system functioning as a black box is being studied. Such a statistical learning device has, for example, a regression curve estimation unit and a learning data output unit, and the regression curve estimation unit estimates the unknown system from the learning data output by the learning data output unit.

In more detail, the unknown system uses the data x
When the data y is output by the operation of the function and the addition of Gaussian noise with respect to the input of M, the M linear models “f (x; θ) =
Σ _i θ _i f _i (x) ”is set. The learning data output unit inputs the data x to the unknown system, collects the output data y, and sets the combination of these data x, y as the learning data. The regression curve estimation unit executes the maximum likelihood estimation method using a linear model on the input of the learning data (x, y), and outputs the unknown system to the regression curve “E [p (y | x)] ”.

Further, when the data x is inputted to the unknown system from a one-dimensional real number straight line, a polynomial having an M-dimensional parameter θ can be set in the average estimating section and the unknown system can be estimated by this polynomial. Is. Such a polynomial is set as follows, for example.

[0005]

(Equation 7)

[0006]

In the statistical learning device as described above, the unknown system can be satisfactorily estimated by accurately inputting the learning data to the estimation unit.

However, since the learning data cannot be created accurately, it is necessary to input a huge amount of learning data in order to sufficiently estimate the unknown system.

[0008]

According to the statistical learning device of the invention described in claim 1, when the unknown system outputs the data y by the operation of the function and the addition of Gaussian noise with respect to the input of the data x, A learning data output unit that outputs the data x input to the unknown system and the output data y as learning data (x, y) is provided, and the M-dimensional parameter θ is set for the input of the learning data (x, y). Statistics provided with a regression curve estimation unit for estimating the unknown system as a regression curve "E [p (y | x)]" by the linear model "f (x; θ) = Σ _i θ _i f _i (x)" In the dynamic learning device, there is provided a learning data creation unit that creates data x that minimizes the learning error of the regression curve estimation unit and sets it in the learning data output unit.

The data x, data y, etc. referred to in the present invention are data indicating numerical values such as vectors and scalars.

In the statistical learning device of the invention described in claim 2, in the statistical learning device of the invention described in claim 1, the learning data creating section has the parameter v and the learning data.
Probability density function "r that generates data x of (x, y)
(x; v) ”holding the input distribution holding unit, and the estimated amount“ hat q ”of the density function of the distribution of the data x input from the outside.
The probability density function “r (x; v)” is calculated by using the estimator holding unit that holds (x) ”and the density function estimator“ hat q (x) ”.
An error minimization unit that calculates a parameter v that reduces the learning error estimation value E (v) of the regression curve estimation unit when the data x of the learning data (x, y) is generated according to The probability density function “r (x;
v) ”to generate the data x of the learning data (x, y).

The "hat A" in the text of this specification is a code expression of "A" to which "^" is added. There is.

In the statistical learning device of the invention described in claim 3, in the statistical learning device of the invention described in claim 2, the density function estimator "hat q (x)" is used.

[0013]

(Equation 8)

A first matrix calculation unit for calculating is used, and the probability density function "r (x; v)" is used.

[0015]

[Equation 9]

A second matrix calculation unit for calculating is provided, and the error minimization unit calculates the estimated value E (v) of the learning error.

[0017]

(Equation 10)

Calculate as

In the statistical learning device of the present invention as defined in claim 4, in the statistical learning device of the invention as defined in claim 3, the error minimization section minimizes the estimated value E (v) of the learning error. Is calculated by a sequential method using the gradient direction.

In the statistical learning device of the invention described in claim 5, in the statistical learning device of the invention described in claim 4, the probability density function "r (x; v)" is "M" in the input distribution holding unit.
It is set as a discrete distribution of the number of (M + 1) / 2 "or less.

In the statistical learning device of the invention described in claim 6, in the statistical learning device of the invention described in claim 4, the probability distribution function "r (x; v)"

[0022]

[Equation 11]

The following function is set as a discrete distribution of the number of real numbers that are linearly independent.

A statistical learning device according to a seventh aspect of the invention is
When an unknown system outputs data y by inputting data x from a one-dimensional real straight line by calculating a function and adding Gaussian noise, the data x input to this unknown system and the output data y are A learning data output unit that outputs the learning data (x, y) is provided, and the learning data (x, y
y) polynomial with M-dimensional parameter θ for input

[0025]

(Equation 12)

According to the statistical learning device provided with the average estimation unit for estimating the unknown system, data x which minimizes the learning error of the average estimation unit is created in accordance with the preset mixture distribution of two distributions. A learning data creating unit to be set in the learning data output unit is provided.

The statistical learning method according to the invention of claim 8 is
When the unknown system outputs the data y by the operation of the function and the addition of Gaussian noise with respect to the input of the data x, the data x input to this unknown system and the output data y are learned data (x, y). As a linear model “f (x; θ) = Σ _i having an M-dimensional parameter θ for the input of this learning data (x, y).
In the statistical learning method in which the regression curve estimation unit estimates the unknown system as a regression curve “E [p (y | x)]” by θ _i f _i (x) ”, learning of the regression curve estimation unit The learning data creation unit creates the data x having the minimum error and sets it in the learning data output unit.

A statistical learning method according to a ninth aspect of the invention is
When an unknown system outputs data y by inputting data x from a one-dimensional real straight line by calculating a function and adding Gaussian noise, the data x input to this unknown system and the output data y are The learning data (x, y) is output to the learning data output unit, and the learning data (x, y
y) polynomial with M-dimensional parameter θ for input

[0029]

(Equation 13)

In the statistical learning method according to which the average estimating unit estimates the unknown system, the learning data creating unit minimizes the learning error of the average estimating unit according to a preset mixture distribution of two distributions. The data x was created and set in the learning data output section.

An information storage medium according to the invention of claim 10 is
An information storage medium in which software for operating a microcomputer is written.
Various functions of the statistical learning device described in 4, 5, 6 or 7 are written as software.

According to the invention described in claims 1 and 8, when the unknown system outputs the data y by the operation of the function and the addition of the Gaussian noise with respect to the input of the data x, the data x input to the unknown system and the output Output the learned data y as learning data (x, y) to the learning data output unit,
A linear model “f (x; θ) = Σ _i θ _i f having an M-dimensional parameter θ with respect to the input of this learning data (x, y)
_The regression curve estimation unit estimates the unknown system as a regression curve “E [p (y | x)]” by “ _i (x).” At this time, the data x that minimizes the learning error of the regression curve estimation unit is the learning data. Since the creation unit creates and sets it in the learning data output unit, the regression curve estimation unit satisfactorily estimates the unknown system with a small number of learning data.

According to the second aspect of the present invention, the probability density function "r (x; v)" for generating the data x of the learning data (x, y) with the parameter v is held in the input distribution holding unit. The estimated amount “hat q (x)” of the density function of the distribution of the data x input from the outside is held in the estimated amount holding unit. Learning error of the regression curve estimation unit when the data x of the learning data (x, y) is generated according to the probability density function “r (x; v)” using the density function estimator “hat q (x)” When the error minimization unit calculates a parameter v having a smaller estimated value E (v) of the learning data (x, y), the calculated parameter v is set to the probability density function “r (x; v)”. Since the data x is generated, the data x that minimizes the learning error of the regression curve estimation unit is easily created.

According to the third aspect of the invention, the first matrix calculation unit calculates the following matrix using the density function estimator "hat q (x)",

[0035]

[Equation 14]

The second matrix calculator calculates the following matrix using the probability density function "r (x; v)",

[0037]

(Equation 15)

The error minimization unit calculates the learning error estimated value E (v) as follows.

[0039]

(Equation 16)

Therefore, the estimated value E (v) of the learning error of the regression curve estimator is easily calculated from the two matrices.

According to the fourth aspect of the invention, the error minimization unit calculates the parameter v for minimizing the estimated value E (v) of the learning error by the sequential method using the gradient direction. Therefore, the estimated value E (v The parameter v that minimizes) is simply calculated.

In the invention according to claim 5, since the probability density function "r (x; v)" of the input distribution holding unit is set as a discrete distribution of a number of "M (M + 1) / 2" or less, learning The number of data is reduced.

According to the sixth aspect of the present invention, since the probability density function "r (x; v)" of the input distribution holding unit is set as a discrete distribution of the number of real independent primary values by the following function, the learning data Is limited to the necessary minimum.

[0044]

[Equation 17]

According to the seventh and ninth aspects of the present invention, when the unknown system outputs the data y by the operation of the function and the addition of Gaussian noise with respect to the input of the data x from the one-dimensional real number straight line, the unknown system The input data x and the output data y are output as learning data (x, y) to the learning data output unit, and the following data having an M-dimensional parameter θ with respect to the input of this learning data (x, y) The average estimator estimates the unknown system by the polynomial of.

[0046]

(Equation 18)

At this time, since the learning data creating unit creates the data x having the minimum learning error of the average estimating unit according to the preset mixed distribution of the two distributions and sets it in the learning data output unit, the regression curve estimation is performed. The department estimates the unknown system well with a small number of training data.

According to the invention of claim 10, claim 1
Since the various functions of the statistical learning device described in 2, 3, 4, 5, 6 or 7 are written as software, the microcomputer operated by the software of this information storage medium is defined in claim 1, 2, 3, It functions as the statistical learning device described in 4, 5, 6 or 7.

[0049]

FIG. 1 shows a first embodiment of the present invention.
It will be described below based on. First, the statistical learning device 1 of the present embodiment has an external data input unit 2, a regression curve estimation unit 3, an external data output unit 4, a learning data output unit 5, and a learning data creation unit 6, It is connected to the unknown system 7.

When the L-dimensional vector x is input as the data x from the external vector space, the unknown system 7 executes the operation of a predetermined function and the addition of Gaussian noise, and the data y is the K-dimensional vector. Output y to the outside. The external data input unit 2 inputs the vector x input from the outside to the regression curve estimation unit 3, and the external data output unit 4 outputs the vector y output by the regression curve estimation unit 3 to the outside. .

The regression curve estimator 3 estimates the unknown system 7 as a regression curve "E [p (y | x)]", and therefore a linear model "f" having an M-dimensional parameter θ is used.
(x; θ) = Σ _i θ _i f _i (x) ”is preset in the model storage unit 8 and has a parameter estimation unit 9 for estimating the M-dimensional parameter θ of the linear model.

The learning data output section 5 outputs the vector x input to the unknown system 7 and the output vector y to the learning curve (x, y) regression curve estimation section 3,
The learning data creation unit 6 creates a vector x that minimizes the learning error of the regression curve estimation unit 3 and sets it in the learning data output unit 5.

In such a configuration, the unknown system 7
Outputs the K-dimensional vector y with respect to the input of the L-dimensional vector x by the calculation of the function and the addition of the Gaussian noise. Therefore, the statistical learning device 1 according to the present embodiment includes the regression curve estimation unit 3 Estimates the unknown function of the unknown system 7.

More specifically, the learning data output unit 5 outputs the vector x input to the unknown system 7 and the output vector y to the regression curve estimation unit 3 as learning data (x, y). The regression curve estimator 3 uses the least square error estimation method, which is the maximum likelihood estimation method using the linear model “f (x; θ) = Σ _i θ _i f _i (x)”, for example, to minimize the following calculation. The function of the unknown system 7 is estimated as a regression curve “E [p (y | x)]” by obtaining θ that satisfies

[0055]

[Equation 19]

At this time, the learning data creation unit 6 creates the vector x with which the learning error of the regression curve estimation unit 3 is minimized and sets it in the learning data output unit 5, so that the learning data output unit 5 Vector x set by the creating unit 6
The learning data (x, y) is sampled and output to the regression curve estimation unit 3. Therefore, the regression curve estimation unit 3 is required to input accurate learning data (x, y) and sufficiently,
The unknown system 7 can be well estimated with a minimum error without requiring enormous learning.

Since the regression curve estimator 3 estimating the unknown system 7 in this way can output the vector y to the input of the vector x in the same manner as the unknown system 7,
Even if the vector y output by the unknown system 7 is not directly collected, it is possible to estimate it and control the unknown system 7.

As a specific method of using such a statistical learning device 1, characteristic identification of a color scanner can be assumed. For example, L, which is the physical quantity of the actual color of a certain document
A statistical learning device that identifies the regression curve estimator 3 with the conversion function from the * a * b * value to the YMC (Yellow, Magenta, Cyanide) value output from the color scanner as the unknown system 7. 1 can be assumed.

In such a case, a standard manuscript in which the L * a * b * values are changed in a necessary range in a sufficiently fine unit is prepared, and the learning data provided by the learning data creating section 6 is observed. L * a * b * closest to the input vector x for
The original of the value is read by the color scanner, and the YMC value output by this is set as the vector y, and one learning data (x, y)
Create By repeating such an operation up to N times, N pieces of learning data can be created by the method of the present invention, and the regression curve estimation unit 3 estimates the parameter θ based on the learning data to determine the characteristics of the color scanner. Can be identified.

In the final adjustment of the color scanner to be shipped as a product, if the regression curve estimation unit 3 is identified by the color scanner which is the unknown system 7, the learning data output unit 5 and the learning data creation unit 6 are products. It does not need to be installed on the color scanner.

Further, the statistical learning device 1 as described above is
This can be realized by manufacturing each of the units 2 to 6 as unique hardware, but the statistical learning device 1 can also be realized by writing the functions of the units 2 to 6 as software in an information storage medium and operating the microcomputer. Can be realized. Similarly, it is possible to manufacture a part of each of the units 2 to 6 with hardware and write the other part as software into the information storage medium.

More specifically, the data x input to the unknown system and the output data y are output as learning data (x, y) to the information storage medium.
The unknown system 7 is subjected to the regression curve “E [p (y) by a linear model“ f (x; θ) = Σ _i θ _i f _i (x) ”having an M-dimensional parameter θ for the input of (x, y). | X)] ", writing and setting data x that minimizes this learning error, etc. are written as software. The information storage medium in which the various functions of the statistical learning device 1 are written as software can be handled as a single product, for example, a RAM (Random Access Memory).
emory), ROM (Read Only Memory), CD-ROM (Com
It can be supplied in the form of a pact disk-ROM), an FD (Floppy Disk), or the like.

Next, a second embodiment of the present invention will be described with reference to FIG.
The following description is based on FIG. Regarding the statistical learning device 11 of the second embodiment, the same parts as those of the statistical learning device 1 of the first embodiment described above will be described in detail by using the same names and reference numerals. Is omitted.

The statistical learning device 11 of the present embodiment specifically explains the structure and operation of the learning data creating unit 12, and as shown in FIG.
Is an estimation amount holding unit 13, an input distribution holding unit 14, a first matrix calculation unit 15, a second matrix calculation unit 16, an error minimization unit 17,
have.

The estimated amount holding unit 13 estimates the estimated amount “hat q” of the density function of the distribution of the vector x input from the outside.
(x) ”is held, and the first matrix calculation unit 15 calculates the following matrix using the estimated amount“ hat q (x) ”of the density function. The added symbol means in the present case that it is an estimate.

[0066]

(Equation 20)

The input distribution holding unit 14 has a parameter v
Holds a probability density function “r (x; v)” for generating a vector x of learning data (x, y), and the second matrix calculation unit 16 holds the probability density function “r (x V) ”is used to calculate the following matrix.

[0068]

(Equation 21)

The error minimization unit 17 uses the probability density function “r (x;
v) ”, a vector x of the learning data (x, y) is generated.
A parameter v that reduces (v) is calculated. For this reason,
The estimated value E (v) of the learning error is calculated as follows, and the parameter v that minimizes the estimated value E (v) of the learning error is calculated by the sequential method using the gradient direction, such as the mountain descent method. calculate.

[0070]

(Equation 22)

Therefore, the learning data creating section 12 uses the parameter v calculated by the error minimizing section 17 as
The probability density function “r (x; v)” held in the input distribution holding unit 14 is set to generate a vector x of learning data (x, y).

In the statistical learning device 11 of the present embodiment having such a configuration, the learning data creation unit 12 creates the vector x and outputs it to the learning data output unit 5 as in the statistical learning device 1 described above. When set, this learning data output unit 5 collects the learning data (x, y) with the set vector x and outputs it to the regression curve estimation unit 3. Therefore, the regression curve estimation unit 3 receives the input learning data. Estimate the function of the unknown system 7 according to the data (x, y).

Here, the process in which the learning data creating unit 12 generates the vector x of the learning data (x, y) will be sequentially described in detail below. First, in the input distribution holding unit 14, a probability density function “r which generates a vector x of learning data (x, y)
Since (x; v) ”is held in advance, the vector x is also fixed when the parameter v is determined.

Therefore, the first matrix calculating unit 15 calculates the following matrix using the estimated amount “hat q (x)” of the density function preset in the estimated amount holding unit 13,

[0075]

(Equation 23)

The second matrix calculation section 16 uses the probability density function "r
The following matrix is calculated using (x; v) ”.

[0077]

(Equation 24)

The error minimization unit 17 calculates the estimated value E of the learning error.
(v) is calculated as follows, and the estimated value E of this learning error is calculated.
The parameter v that minimizes (v) is calculated by a sequential method using the gradient direction.

[0079]

(Equation 25)

Then, the learning data creating unit 12 uses the probability density function "r" for the parameter v calculated as described above.
Since the vector x is generated by setting (x; v) ", the learning data (x, y) by this vector x can minimize the learning error of the regression curve estimation unit 3.

Here, the statistical learning device 1 of the present embodiment
The effectiveness of 1 is verified below. First, it is assumed that the true regression curve f (x) of the unknown system 7 to be estimated is included in the M linear models f (x; θ) set in the regression curve estimation unit 3, and f (x) = f (x; θ ₀ ). Next, the density function set in the input distribution holding unit 14 is set to r (x), and the parameter θ acquired by the least square error estimation by the regression curve estimation unit 3 from the learning data (x, y) created from this is calculated as “hat θ ” The average learning error in such a state can be calculated as follows.

[0082]

(Equation 26)

Letting E _{0 be} the expected value corresponding to the appearance of the learning data of this learning error, this is approximately expressed as follows.

[0084]

[Equation 27]

In this case, the following is established by a general calculation method in statistical estimation.

[0086]

[Equation 28]

Therefore, the expected value E ₀ of the learning error is as follows.

[0088]

(Equation 29)

If the expected value E ₀ of the learning error can be minimized, the learning error of the regression curve estimator 3 can be minimized. Therefore, the statistical learning device 11 of the present embodiment does the above. The learning data creation unit 12 is formed so as to satisfy

Incidentally, the statistical learning device 11 of the present embodiment.
Then, since the error minimization unit 17 calculates the parameter v of the estimated value E (v) of the learning error by the sequential method as described above, even if the proper parameter v cannot be analytically calculated, the proper parameter v Can be earned.

Further, when the error minimization unit 17 calculates the parameter v by the sequential method in this way, the probability density function “r (x; v)” in the input distribution holding unit 14 is “M (M +
1) / 2 "or less, the calculation cost of the trace Tr of the error minimization unit 17 can be reduced. Furthermore, the probability density function" r (x;
v) ”can be set as a discrete distribution of the number of real independent primary values by the following function, the error minimization unit 1
The calculation cost of 7 can be reduced to the necessary minimum.

[0092]

[Equation 30]

Here, the statistical learning device 11 as described above is used.
A specific example of the statistical learning method according to is briefly described below with reference to FIG. First, as described above, the first matrix calculating unit 15 uses the estimated amount “hat q (x)” of the density function preset in the estimated amount holding unit 13 to calculate the first matrix “hat I”. Calculate (step S1). Next, the second matrix calculation unit 16 calculates the second matrix “Jab (v)” using the probability density function “r (x; v)” stored in the input distribution storage unit 14 in advance ( In step S4), the error minimizing unit 17 calculates the estimated value E (v) of the learning error (step S5).

At this time, since the parameter v that minimizes the estimated value E (v) of the learning error is calculated by the sequential method using the gradient direction (step S6), the parameter v is first initialized by a random number. (Step S2), the number of iterations k of the steepest descent method that is a sequential method is repeated.
Is initialized to "0" (step S3), and the parameter v is calculated by the second matrix "Jab (v)" and the estimated value E (v) of the learning error until the number of repetitions k _end is reached (step S3). Steps S4 to S8). In addition, “α” in the figure is a convergence coefficient.

When the parameter v that minimizes the estimated value E (v) of the learning error is calculated in this way, the learning data creation unit 12 sets the parameter v to the probability density function “r (x; v)”. Then, "N-1" vectors x are generated (step S9), and at the same time, "N" which the unknown system 7 actually outputs.
Since -1 "vectors y are acquired (step S1
0), the parameter θ of the linear model f (x; θ) of the unknown system 7 is estimated from these “N−1” pieces of learning data (x, y) (step S11).

Next, a third embodiment of the present invention will be described with reference to FIG.
It will be described below based on. Regarding the statistical learning device 21 of the third embodiment, the same parts as those of the statistical learning device 11 of the second embodiment described above will be described in detail by using the same names and reference numerals. Is omitted.

The statistical learning device 21 of the present embodiment specifically explains the estimated amount holding unit 23 of the learning data creation unit 22, and the estimated amount holding unit 23 has a parametric model “parameter u”. q (x; u) ”is preset and is connected to the external data input unit 2. Therefore, the estimated amount holding unit 23 collects an (N-1) -dimensional vector x that is actually input to the external data input unit 2 from the outside, and estimates the density function of the distribution "hat q (x)". ”
Is parametrically estimated and held by the parametric model “q (x; u)”.

Note that this parametric model "q
As (x; u) ”, various models for realizing parametric estimation can be used, and are set as follows by, for example, a mixture distribution that is a normal distribution.

[0099]

[Equation 31]

"N (x; m, s ² )" in this equation is the average m
And the variance s ² of the normal distribution probability density function. Since this is expressed as follows, the parametric model “q
(x; u) "parameter u of," it is _{u = (m t, s t} ) ".

[0101]

(Equation 32)

In such a configuration, also in the statistical learning device 21 of the present embodiment, similarly to the above-described statistical learning device 11, the learning data creation unit 22 holds the vector x held in the estimated amount holding unit 23. Vector x of the training data (x, y) using the estimated value of the density function of the distribution “hat q (x)”
Create

Since the estimated amount holding unit 23 parametrically estimates and holds the estimated amount "hat q (x)" from the vector x input from the outside, the regression curve estimation unit 3 uses it at the time of this parametric estimation. The vector x equivalent to the situation is input to the external data input unit 2. Then, the estimated amount holding unit 23 collects the vector x input from the outside to the external data input unit 2 and uses the parameter u of the preset parametric model “q (x; u)” as the maximum likelihood estimation method or the like. By performing parametric estimation with, the estimated amount “hat q (x)” of the density function of the distribution of the vector x is estimated and held.

The statistical learning device 21 of the present embodiment stores the distribution density of the vector x in the estimated amount holding unit 23 of the learning data creation unit 22 based on the vector x equivalent to the actual use condition as described above. An estimator "hat q (x)" of the function can be set.

Next, a fourth embodiment of the present invention will be described with reference to FIG.
It will be described below based on. Regarding the statistical learning device 31 of the fourth embodiment, the same parts as those of the statistical learning device 21 of the third embodiment described above will be described in detail by using the same names and reference numerals. Is omitted.

The statistical learning device 31 of the present embodiment also specifically describes the estimated amount holding unit 33 of the learning data creation unit 32. The estimated amount holding unit 33 is a non-parametric model "q (x ) ”Is preset and the vector x that is actually input to the external data input unit 2 from the outside is sampled, and the estimated amount“ hat q (x) ”of the distribution density function is nonparametrically estimated and stored. To do.

The non-parametric model "q
Various models that realize non-parametric estimation can be used as (x) ”. The vector x is input as follows.

[0108]

[Expression 33]

In this case, the non-parametric model "q
(x) ”is set as follows by a normal distribution, for example.

[0110]

(Equation 34)

In such a configuration, the statistical learning device 31 of the present embodiment also, like the statistical learning device 21 described above, based on the vector x equivalent to the actual usage condition, the learning data creation unit 32. The estimated amount “hat q (x)” of the density function of the distribution of the vector x is set in the estimated amount holding unit 33. The estimated amount holding unit 33 samples the vector x input from the outside to the external data input unit 2 and non-parametrically estimates and holds the estimated amount “hat q (x)” of the density function of the distribution of the vector x. .

Incidentally, the statistical learning device 31 of the present embodiment.
In the above, the non-parametric model “q (x) using a normal distribution is set in the estimation amount holding unit 33, but the present invention is not limited to the above-described embodiment, and such a non-parametric model“ q As q (x) ”,
Various kernel functions such as a rectangular function can also be used. For example, the following calculation using a delta function as the kernel function can also be used.

[0113]

(Equation 35)

Next, a fifth embodiment of the present invention will be described with reference to FIG.
It will be described below based on. Regarding the statistical learning device 41 of the fifth embodiment, the same parts as those of the statistical learning device 11 of the second embodiment described above will be described in detail by using the same names and reference numerals. Is omitted.

The statistical learning device 41 of the present embodiment specifically describes the learning data creating unit 42. The learning data creating unit 42 does not have the estimated amount holding unit 13, The one-row calculation unit 43 is connected to the external data input unit 2. The first matrix calculation unit 43 collects the (N'-1) -dimensional vector x that is actually input to the external data input unit 2 from the outside and executes the following calculation.

[0116]

[Equation 36]

With such a configuration, also in the statistical learning device 41 of the present embodiment, the learning data creation unit 42 uses the calculation result of the first matrix calculation unit 43 as in the statistical learning device 11 described above. To create a vector x of learning data (x, y). Since the first matrix calculation unit 43 uses the vector x input from the outside to perform the calculation by addition without performing the integration, the calculation load can be reduced and the time can be shortened. Such a method corresponds to the case where the delta function is used as the kernel function.

Next, a sixth embodiment of the present invention will be described with reference to FIG.
This will be described below with reference to FIG. With regard to the statistical learning device 51 of the sixth embodiment, the same parts as those of the statistical learning device 1 of the first embodiment described above will be described in detail by using the same names and reference numerals. Is omitted.

As shown in FIG. 7, the statistical learning device 51 of the present embodiment corresponds to an unknown system 52 which outputs data y for an input of a vector x from a one-dimensional real number straight line. A polynomial having an M-dimensional parameter θ is set in the model storage unit 54 of the average estimation unit 53 that estimates the polynomial as follows.

[0120]

(37)

Then, the learning data creating unit 55 for creating the vector x of the learning data (x, y) of the average estimating unit 53 as described above, has a mixture distribution “r” of two preset distributions.
(x: v) ”according to the learning error E of the average estimation unit 53.
A vector x that minimizes (v) is created and set in the learning data output unit 5.

Note that such a mixture distribution "r (x: v)"
Is formed by two different distributions, but here "0
<A <1 "by the two normal distributions are represented by real number a made" an (x; 0, τ 1 2) + (1-a) n (x; 0, τ 2
² ) ”in this case. In this case, two normal distributions have respective linear combination coefficients of“ a, 1-a ”and respective variances of“ τ ₁ ² , τ ₂ ² ”. "0 <a <0.1"
Is set so as to satisfy the tau ₁ is satisfied is "aτ ₁ ^2> 10" to "τ ₁ ^2> 10" with small a that satisfies.

With such a configuration, in the statistical learning method of the statistical learning device 51 of the present embodiment, as shown in FIG. 8, the learning data creation unit 55 causes the preset distribution distribution of the two distributions to be set. Learning error E according to "r (x: v)"
Since the vector x that minimizes (v) is created (step T1), the average estimating unit 53 favorably estimates the unknown system 52 by using the polynomial f (y; θ) from the learning data (x, y). Can be performed (steps T2 to T4).

Here, the statistical learning device 5 of the present embodiment.
The effectiveness of 1 is verified below. First, mean m, variance τ ²
The density function of the normal distribution of

[0125]

(38)

Then, the mixture distribution “r (x: v)” of the two normal distributions set in the average estimation unit 53 is defined as “an (x; 0,
τ ₁ ² ) + (1-a) n (x; 0, τ ₂ ² ) ”, and
If τ ₂ is a bounded value and the limit is set as “a → 0” “τ ₁ → ∞” so that “a τ ₁ ² → ∞” is maintained, the learning error E (v) trace value is become that way.

[0127]

[Equation 39]

When the extreme operation is executed as described above,
Since the trace value of the learning error E (v) larger than "1" approaches infinitely "1", the optimum input distribution can be obtained. However, since it is not possible to operate indefinitely, it is necessary to set a specific numerical value.

In order to confirm this, the statistical learning device 51 of this embodiment was simulated as follows, and a sufficient effect was confirmed. First, let U be the uniform distribution on [-1, 1], Z be an independent random variable that follows a normal distribution with mean 0 variance 1, and set the input distribution of the real system by q (x) to "U + 0.3Z". did. And the unknown system 52
When the true function of is defined as “f (x) = x + 2”, “a =
A sufficient effect was confirmed at 0.1, τ ₁ = 30, τ ₂ = 1 ”.

Therefore, the statistical learning device 5 uses these numerical values.
Set to 1 and change the estimation model from first-order to fifth-order, and collect 1000 pieces of learning data (x, y) by the two density functions “r (x: v), q (x)” described above. The estimated amount of the parameter, “hat θ”, was calculated by the above, and the estimation error after the learning was evaluated by the mean square error by the following formula.

[0131]

(Equation 40)

Two such density functions “r (x: v),
As shown in FIG. 9, when the density function “q (x)” was executed 500 times each, the error increased in proportion to the increase of the model order. r (x: v) ”
In the case of, the error did not increase even if the order of the model increased. Moreover, the error of the density function “r (x: v)” is equivalent to the minimum error of the density function “q (x)” (when the order is 0th order), and as described above, the mixed distribution of the two distributions. "R (x:
v) ”is used to create the vector x, the learning error E
It was confirmed that (v) was the minimum.

Incidentally, the statistical learning device 51 of the present embodiment.
Corresponds to the case where the input space of the vector x is a one-dimensional real straight line as described above, but such an input space may be limited to the bounded section [-A, B]. In such a case, the learning data (x, y) at the both ends of the ratio "a /
2 "each, and the remaining" 1-a "ratio is" (A +
B) ^2/4 "to generate the distribution of smaller dispersion is preferred. In this way, since approximately taken close to the optimal distribution of the real line is possible, to create a better learning data be able to.

[0134]

According to the first and eighth aspects of the present invention, when the unknown system outputs the data y by the operation of the function and the addition of Gaussian noise with respect to the input of the data x, the data x input to this unknown system is input. And output data y as learning data (x, y) to the learning data output unit, and a linear model “f (x (x) y) having an M-dimensional parameter θ with respect to the input of this learning data (x, y). ; Θ) = Σ _i θ _i f
_In the statistical learning device and the statistical learning method in which the regression curve estimation unit estimates the unknown system as the regression curve “E [p (y | x)]” by _i (x) ”, the learning error of the regression curve estimation unit is minimum. Since the learning data creation unit creates the data x to be set in the learning data output unit, the learning data output unit learns in the regression curve estimation unit based on the data x output by the learning data creation unit. Since the data (x, y) is output, this regression curve estimation unit can estimate the unknown system with the minimum error.

According to the second aspect of the invention, the learning data creation unit holds the probability density function "r (x; v)" for generating the data x of the learning data (x, y) with the parameter v. The input distribution holding unit, the estimation amount holding unit that holds the density function estimation amount “hat q (x)” of the distribution of the data x input from the outside, and the density function estimation amount “hat q (x)” Learning data according to probability density function "r (x; v)"
an error minimization unit that calculates a parameter v that reduces the learning error estimation value E (v) of the regression curve estimation unit when the data x of (x, y) is generated. By setting v to the probability density function “r (x; v)” and generating the data x of the learning data (x, y), the learning data (x, y) that minimizes the learning error of the regression curve estimation unit. )
The data x can be easily created.

According to the third aspect of the invention, the density function estimator "hat q (x)" is used.

[0137]

[Equation 41]

A first matrix calculation unit for calculating is provided and the probability density function "r (x; v)" is used.

[0139]

(Equation 42)

A second matrix calculation unit for calculating is provided, and the error minimization unit calculates the estimated value E (v) of the learning error.

[0141]

[Equation 43]

By calculating as, the estimated value E (v) of the learning error required for creating the optimum data x can be easily calculated.

According to the fourth aspect of the invention, the error minimization section minimizes the estimated value E (v) of the learning error by the parameter v.
The parameter v that minimizes the learning error E (v) can be easily calculated by calculating γ by a sequential method using the gradient direction.

According to the fifth aspect of the invention, in the input distribution holding unit, the probability density function “r (x; v)” is “M (M + 1) / 2”.
The number of learning data can be reduced by setting the following discrete distributions.

According to the sixth aspect of the invention, the probability distribution function “r (x; v)” is

[0146]

[Equation 44]

Since the function is set as a discrete distribution of the number of real numbers which is linearly independent, the number of learning data can be limited to the necessary minimum.

In the inventions according to claims 7 and 9, when the unknown system outputs the data y by the operation of the function and the addition of Gaussian noise with respect to the input of the data x from the one-dimensional real number straight line, the unknown system The input data x and the output data y are output as learning data (x, y) to the learning data output unit, and a polynomial having an M-dimensional parameter θ with respect to the input of this learning data (x, y)

[0149]

[Equation 45]

According to the statistical learning device and the statistical learning method in which the average estimating unit estimates the unknown system, the learning data creating unit creates the data x having the minimum learning error of the average estimating unit and outputs it to the learning data output unit. By setting the learning data, the learning data output unit outputs the learning data to the average estimation unit based on the data x output by the learning data creation unit.
Since (x, y) is output, this average estimation unit can estimate the unknown system with the minimum error.

An information storage medium according to the invention of claim 10 is
An information storage medium in which software for operating a microcomputer is written.
Since various functions of the statistical learning device according to claim 4, 5, 6 or 7 are written as software, if the microcomputer is operated by this software, the claim 1, 2, 3, 4, 5, 6 or The statistical learning device described in 7 can be easily realized.

[Brief description of drawings]

FIG. 1 is a block diagram showing a statistical learning device according to a first embodiment of the present invention.

FIG. 2 is a block diagram showing a statistical learning device according to a second embodiment of the present invention.

FIG. 3 is a flowchart showing the statistical learning method.

FIG. 4 is a block diagram showing a statistical learning device according to a third embodiment of the present invention.

FIG. 5 is a block diagram showing a statistical learning device according to a fourth exemplary embodiment of the present invention.

FIG. 6 is a block diagram showing a statistical learning device according to a fifth embodiment of the present invention.

FIG. 7 is a block diagram showing a statistical learning device according to a sixth embodiment of the present invention.

FIG. 8 is a flowchart showing the statistical learning method.

FIG. 9 is a characteristic diagram showing test results of the estimation error.

[Explanation of symbols]

 1,11,21,31,41,51 Statistical learning device 3 Regression curve estimation unit 5,22 Learning data output unit 6,12,32,42 Learning data creation unit 7,52 Unknown system 13,23,33 Estimated amount Storage unit 14 Input distribution storage unit 15 First matrix calculation unit 16 Second matrix calculation unit 17 Error minimization unit 53 Average estimation unit 55 Learning data creation unit

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] June 13, 1996

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name]

[Correction method] Change

[Correction contents]

[0084]

[Equation 27]

Claims (10)

[Claims] 1. When an unknown system outputs data y by an operation of a function and addition of Gaussian noise with respect to an input of data x, the data x input to this unknown system and the output data y are learned data. A learning data output unit for outputting as (x, y) is provided, and a linear model “f” having an M-dimensional parameter θ with respect to the input of learning data (x, y)
(x; θ) = Σ _i θ _i f _i (x) ”In a statistical learning device provided with a regression curve estimation unit that estimates the unknown system as a regression curve“ E [p (y | x)] ”, A statistical learning device comprising: a learning data creation unit that creates data x that minimizes the learning error of the regression curve estimation unit and sets the data x in the learning data output unit. 2. The learning data creation unit, an input distribution holding unit holding a probability density function “r (x; v)” for generating data x of learning data (x, y) with a parameter v, The probability density function “using the estimated quantity holding unit that holds the estimated quantity“ hat q (x) ”of the density function of the distribution of the data x input from the outside and the estimated quantity“ hat q (x) ”of the density function r
When the data x of the learning data (x, y) is generated according to (x; v) ", the estimated value E of the learning error of the regression curve estimation unit E
and an error minimization unit that calculates a parameter v in which (v) becomes smaller, and the calculated parameter v is set to the probability density function “r (x; v)” of the learning data (x, y). The statistical learning device according to claim 1, wherein the data x is generated. 3. A density function estimator “hat q (x)” is used to obtain: The probability density function "r
(x; v) ” A second matrix calculation unit for calculating is provided, and the error minimization unit calculates the estimated value E (v) of the learning error as 3. The statistical learning device according to claim 2, wherein 4. The error minimization unit estimates the learning error E
4. The statistical learning device according to claim 3, wherein the parameter v that minimizes (v) is calculated by a sequential method using the gradient direction. 5. The probability distribution function “r
5. The statistical learning device according to claim 4, wherein (x; v) "is set as a discrete distribution of a number of" M (M + 1) / 2 "or less. 6. The probability distribution function “r
(x; v) ”is 5. The statistical learning device according to claim 4, wherein the function is set as a discrete distribution of the number of real numbers that are linearly independent. 7. When the unknown system outputs the data y by inputting the data x from the one-dimensional real number straight line by the calculation of the function and the addition of Gaussian noise, the data x input to the unknown system is output. The learning data output unit that outputs the learned data y and the learning data (x, y) is provided, and the M-dimensional parameter θ is input to the learning data (x, y).
Polynomial with In the statistical learning device provided with the average estimation unit for estimating the unknown system according to the above, the learning data is created by creating the data x with which the learning error of the average estimation unit is minimized in accordance with the preset mixed distribution of two distributions. A statistical learning device, characterized in that a learning data creation unit for setting the output unit is provided. 8. When the unknown system outputs data y by inputting data x by calculating a function and adding Gaussian noise, the data x input to this unknown system and the output data y are learned data. (x, y) is output to the learning data output unit, and a linear model “f” having an M-dimensional parameter θ for the input of this learning data (x, y)
(x; θ) = Σ _i θ _i f _i (x) ”In the statistical learning method, the regression curve estimation unit estimates the unknown system as a regression curve“ E [p (y | x)] ”. The statistical learning method is characterized in that the learning data creation unit creates data x that minimizes the learning error of the regression curve estimation unit and sets it in the learning data output unit. 9. When an unknown system outputs data y by inputting data x from a one-dimensional real number straight line by calculating a function and adding Gaussian noise, it is output as data x input to this unknown system. Output data y as learning data (x, y) to the learning data output unit, and an M-dimensional parameter θ for the input of this learning data (x, y).
A polynomial with According to the statistical learning method in which the average estimation unit estimates the unknown system, the learning data creation unit calculates the data x with which the learning error of the average estimation unit is minimized according to the mixture distribution of two distributions set in advance. A statistical learning method, which is created and set in the learning data output unit. 10. An information storage medium in which software for operating a microcomputer is written, and various functions of the statistical learning device according to claim 1, 2, 3, 4, 5, 6 or 7 are written as software. An information storage medium characterized by being present.