JPH0962652A - Maximum likelihood estimation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a maximum likelihood estimation method capable of solving the problem of local optimality in EM algorithm and obtaining a further improved parameter estimated value by the EM algorithm even for the optional initial value of a parameter. SOLUTION: The logarithmic likelihood function based on observation data is calculated by using a combined probability density function where the combined probability density function of an observation data set and a non- observation data set is supplied parametrically, and the parameter for maximizing the function is obtained by a nonlinear numerical solution method.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a maximum likelihood estimation method for estimating parameters from observed data which is incomplete data including non-observed data in which some data are missing or unobservable, and more particularly Relates to a maximum likelihood estimation method which is effective for a basic technique of point estimation of parametric statistics, such as parameter estimation of a probability density function such as parameter estimation of a mixed density function.

[0002]

2. Description of the Related Art Now, when a certain stochastic model is given as a parametric model, p (x; Θ), and unknown parameters are estimated from the data set X = {x ₁ , ..., X _N },
In the maximum likelihood estimation method, a parameter that maximizes the log-likelihood function defined by the following equation is used as an estimated value. This estimated value is called the maximum likelihood estimated value.

[0003]

[Equation 1] This is because, since the data set X originates from the probability density function p (x; Θ), conversely, given the data set X, the value that is most likely to yield X, that is, p
It is based on the likelihood principle that Θ with the largest (X; Θ) is likely. Here, the logarithm is taken
It is intended to facilitate mathematical handling and is not essential.

Next, the maximum likelihood estimation in the case where the data set X has a missing portion or an unobservable portion will be considered.
That is, it is assumed that the data set X consists of the observation data set X _obs and the non-observation data set X _mis . At this time, the data set X is called complete data (set), and the observation data set X _obs is called incomplete data (set). The parametric probability density function of complete data is p (X _obs , X
_mis ; Θ), the log-likelihood function based on the observed data is

L (Θ; X _obs ) = ∫log p (X _obs , X _mis ; Θ) dX _mis (2)

The maximum likelihood estimate is the maximization of L, that is, the likelihood equation:

(Equation 3) However, since it is generally a nonlinear equation, it is difficult to obtain an analytical solution, and it is usually solved numerically by some iterative algorithm. Further, in the case of maximum likelihood estimation from incomplete data, there is a more general and efficient known method called EM (Expectation Maximization) algorithm described below instead of directly solving the likelihood equation numerically (APDempster , NMLaird, and DBRubin, "Ma
ximum likelihood from incomplete data via the EM a
lgorithm ", Journal of RoyalStatistics, Ser.B, vol.39,
pp.1-38,1977).

In the EM algorithm, the parameters are estimated by successive iterations. Now, if the estimated value after the Tth iteration is Θ ^(t) , Θ ^{(t + 1)} is obtained by the following E and M steps.

[0007]

(Equation 4) Must be met. Where p (X _mis | X
_obs ; Θ ^(t) ) is the posterior probability density function, which is calculated from the complete data probability density function as shown in the following equation.

[0008]

(Equation 5) In the EM algorithm, it is theoretically proved that the maximization of Θ of the sequential Q function is equivalent to the maximization of the log-likelihood function (APDempster, NMLai
rd, and DBRubin, "Maximum likelihood from incomple
te data via theEM algorithm ", Journal of RoyalStati
stics, Ser.B, vol.39, pp.1-38, 1977).

[0009]

The above-mentioned conventional EM algorithm merely converges to the local optimum solution (maximum value) of the log-likelihood function, and does not necessarily find the maximum value of the log-likelihood function. In general, since there is no effective guideline for initial value setting, the inefficient method of actually executing the EM algorithm for various initial values and selecting the best solution among them is taken. There is.

The present invention has been made in view of the above,
The object is to provide a maximum likelihood estimation method that can solve the problem of local optimality in the EM algorithm and obtain a better parameter estimate than the EM algorithm for any initial value of the parameter. Especially.

[0011]

In order to achieve the above object, the present invention according to claim 1 parametrically provides a joint probability density function of an observed data set and an unobserved data set, and uses the joint probability density function. Then, a log-likelihood function based on the observation data is calculated, and a parameter maximizing the function is obtained by a nonlinear numerical solution method.

Further, the present invention according to claim 2 is based on claim 1.
In the described invention, for appropriate initial parameter values,
The first step of forming a first-order smoothed log-likelihood function smoothed so that the maximum value of the log-likelihood function is unique, and a parameter for maximizing the first-order smoothed log-likelihood function are A second step of setting the first optimum parameter by a non-linear numerical solution method and a second smoothed log-likelihood function in which the degree of smoothing is weakened from the first step using the first optimum parameter value as an initial value The second-order optimal parameter is obtained in the same manner as above on the third step of configuring and the second-order smoothed log-likelihood function until the degree of smoothing becomes zero, that is, the original log-likelihood function. The fourth step is to recursively perform a smoothing process in which a similar process is performed while weakening the degree of smoothing, and to obtain the optimum parameter value of the original log-likelihood function.

Further, the present invention according to claim 3 provides the invention according to claim 2.
In the invention described above, in the first to fourth steps, a smoothed log-likelihood function is obtained using a posterior probability density function calculated from a joint probability density function of observed data and non-observed data. To do.

A fourth aspect of the present invention is characterized in that, in the first, second or third aspect of the invention, the iterative method is used as the nonlinear numerical solution.

According to a fifth aspect of the present invention, in the invention according to the first, second, third or fourth aspect, the degree of smoothing of a certain smoothed log-likelihood function and the following smoothed log-likelihood function of The point is to gradually reduce the degree of smoothing in a geometric series.

In the maximum likelihood estimation method of the present invention, the smoothing uses a new posterior probability density function defined by the following equation instead of equation (7) when calculating the Q function of the E step of the EM algorithm. Realized by using.

[0017]

(Equation 6) Get. On the other hand, comparing the first term on the right side of equation (10) with the equation (4), the first term on the right side of equation (10) is
The posterior probability density function p (X _mis | X _obs ; Θ ^(t) ) is f
It has been replaced with (X _mis | X _obs ). Also, considering that the second term on the right side of the equation (10) is independent of Θ, the maximization of Q function with respect to Θ is p
It can be seen that, except for the difference between (X _mis | X _obs ; Θ ^(t) ) and f (X _mis | X _obs ), it is equivalent to maximization of Θ on the right side of equation (10). Furthermore, p (X _mis | X _obs ;
Sequentially estimating Θ using f (X _mis | X _obs ) instead of Θ ^(t) ) means that instead of _obtaining the parameter maximizing the likelihood function of equation (2), equation (10) Likelihood function L _β controlled by β defined on the left side of:

(Equation 7) It means that the parameter that maximizes is successively estimated. Then, by setting β to a positive value smaller than 1, smoothing of L can be realized. Apparently, when β = 1, L ₁ (Θ; X _obs ) ≡L (Θ; X _obs ), that is, equation (11) agrees with equation (2), and the closer β is to zero, the smoother The degree of conversion becomes large.

Now, the value of β is

(Equation 8) It is assumed that smoothing likelihood function groups L _{β 1} , L _β2 , ..., L _βmax corresponding to the value of β and having different degrees of smoothing are obtained. At this time, if the value of β ₁ is set so that L _{β 1} has a single peak, the global optimum parameter (referred to as Θ ₁ ^* ) that gives the maximum value can be easily obtained. Next, from the similarity of the shapes of L _β1 and L _β2 , it is considered that the optimum parameter (θ ₂ ^* ) of L _β2 is in the vicinity of Θ ₁ ^*.
₂ ^* can be easily obtained with Θ ₁ ^* as the initial value. By sequentially executing the following for β ₃ , β ₄ , ..., L _βmax , that is, the optimum parameter that maximizes the original likelihood function can be finally obtained. As an empirical rule, it is known that an estimated parameter can be efficiently obtained by reducing the value of β in geometric progression.

[0019]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 is a block diagram showing the functional arrangement of an apparatus for carrying out the maximum likelihood estimation method according to an embodiment of the present invention. In the figure, the parameter estimation unit 1 receives training observation data from the outside, and estimates parameters using this data. The parameter estimation unit 1 sequentially executes steps 1 to 3 shown below to obtain estimated parameters.

(Procedure 1) Initial values of parameters and β
Set the value of to an appropriate value. Let t ← 0.

(Procedure 2) Until an appropriate convergence condition is satisfied,
Repeat steps 2-1 and 2-2.

[0023]

[Equation 9] The parameters maximizing the likelihood function of the equation (2) are estimated by the above procedures 1 to 3.

2 to 4 show experimentally the effectiveness of the present invention. In the experiment, a two-parameter estimation (in this case, a search in a two-dimensional parameter plane) problem in which the search for the solution can be visualized was used. Specifically, a mixture distribution of two one-dimensional Gaussian distributions:

(Equation 10) The parameter estimation problem of is used. In this case, the parameters to be estimated are m ₁ and m ₂ . 100 pieces of training data X _obs used in the experiment were artificially generated from the mixture distribution with m ₁ ^* = − 2 and m ₂ ^* = 2 in the above equation.

FIG. 2 shows contour lines of the likelihood function L _β (θ; X _obs ) with β in the equation (11) with respect to the equation (13) for various values of β. The x mark in the figure shows the maximum smoothing likelihood function at each β value (m ₁ , m ₂ ).
The coordinates are shown. As described above, when β = 1, since the equation (11) is equivalent to the equation (2), the contour line of β = 1 in FIG. 2 corresponds to the contour line of the original likelihood function. From FIG. 2, it can be seen that the smaller the value is, the smoother the likelihood function is, and that between the smoothed likelihood functions in which the difference in the value of β is small,
It can be confirmed that the values of the global optimum parameters that give the maximum value of the function are close to each other.

In the conventional method (EM algorithm), as shown in FIG.
In contrast to the parameter estimation (maximum value search of the likelihood function) performed on the likelihood function for β = 1 in (f), the method according to the present invention first searches for the maximum value in (a) of FIG. Then, using the estimated parameter value as a new initial value, the maximum value search of the likelihood function of FIG. Hereinafter, the same maximum value search is performed up to FIG. 6F, and the optimal parameter that finally gives the maximum value of the original likelihood function is estimated.

FIGS. 3 and 4 compare the estimation process and the estimation result of the method according to the present invention and the conventional method for the above problem. As described above, the method according to the present invention performs the sequential search with different β as shown in FIG. 2, but the estimation process is represented by β = 1, that is, on the original likelihood function, for convenience. . Figure 3 shows the initial value

[Equation 11] The estimation process and the estimation results are shown. In the conventional method, all are local optimum values.

(Equation 12) While the method according to the present invention converges to the neighborhood of

(Equation 13) Has converged near.

A typical application of the maximum likelihood estimation method of the present invention is cluster analysis. Cluster analysis is to find out some "clusters" from a set of multidimensional data. This allows the data sets to be roughly classified into similar groups. For example, a two-dimensional data set as shown in FIG. 5 (each data consists of two attributes)
In this case, the elliptical cluster shown in the figure will be found. When each cluster is parametrically expressed by an analytic function such as a Gaussian function and a data set is expressed by a linear sum of those functions, the parameters and linear weights of each function are automatically estimated from only given data according to the present invention. can do.

[0029]

As described above, according to the present invention,
Parametrically gives the joint probability density function of the observed data set and the non-observed data set, calculates the log-likelihood function based on the observed data by using the joint probability density function, and uses the nonlinear numerical solution method for the parameter that maximizes the function. According to the above, the EM algorithm can obtain a good parameter estimation value even for an initial value with which a local optimum solution is obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing a functional configuration of an apparatus that implements a likelihood estimation method according to an embodiment of the present invention.

FIG. 2 is a diagram showing the effect of the present invention in which contour lines of the likelihood function with β in equation (11) with respect to equation (13) are displayed for various values of β.

FIG. 3 is a diagram showing a comparison of an estimation process and an estimation result between the method according to the present invention and a conventional method.

FIG. 4 is a diagram showing a comparison of an estimation process and an estimation result between the method according to the present invention and the conventional method.

FIG. 5 is a diagram showing an elliptical cluster found in a cluster analysis of a two-dimensional data set.

[Explanation of symbols]

 1 parameter estimation unit

Claims (5)

[Claims] 1. Parametrically providing a joint probability density function of an observed data set and an unobserved data set, calculating a log-likelihood function based on observed data using the joint probability density function, and maximizing the function. A maximum likelihood estimation method characterized in that parameters are obtained by a non-linear numerical solution. 2. A first smoothed value such that the maximum value of the log-likelihood function is unique to an appropriate initial parameter value.
A first step of forming a second-order smoothed log-likelihood function, a second step of obtaining a parameter that maximizes the first-order smoothed log-likelihood function by the nonlinear numerical solution method, and setting the parameter as a first-order optimum parameter; A third step of forming a second-order smoothed log-likelihood function in which the degree of smoothing is weakened from the first-step by using the first-order optimum parameter value as an initial value, and the second-order smoothed log-likelihood function described above The second-order optimal parameter is obtained in the same manner as, and thereafter, the degree of smoothing is zero,
That is, the optimum parameter value of the original log-likelihood function is obtained by performing the fourth step of recursively performing the smoothing process of performing the same processing while weakening the degree of smoothing until the original log-likelihood function is obtained. The maximum likelihood estimation method according to claim 1, wherein 3. The smoothed log-likelihood function is obtained in the first to fourth steps using a posterior probability density function calculated from a joint probability density function of observed data and non-observed data. The maximum likelihood estimation method according to claim 2. 4. The maximum likelihood estimation method according to claim 1, wherein an iterative method is used as the nonlinear numerical solution. 5. The smoothing degree of a certain smoothed log-likelihood function and the smoothing degree of the next smoothed log-likelihood function are reduced in a geometric progression. The maximum likelihood estimation method described in 3 or 4.