WO2009081707A1

WO2009081707A1 - Statistical model learning device, method, and program

Info

Publication number: WO2009081707A1
Application number: PCT/JP2008/071983
Authority: WO
Inventors: Yoshifumi Onishi
Original assignee: Nec Corporation
Priority date: 2007-12-20
Filing date: 2008-12-03
Publication date: 2009-07-02
Also published as: JP5493867B2; JPWO2009081707A1

Abstract

It is possible to provide a statistical model learning device which can calculate a prediction distribution considering affects of the state density while avoiding the problem of a statistical amount attributed to model parameter integration. The statistical model learning device includes: self-response function calculation means (102) which calculates a self-response function of a model parameter; eigenvalue analysis means (103) which calculates eigenvalues and eigenvectors of a matrix having self-response functions between M model parameters as components; and prediction distribution calculation means (104) which calculates a weight of a model parameter by using values of M' eigenvalues and values of eigenvectors corresponding to M' which is smaller than M in the descending order of the eigenvalues among the eigenvalues and the eigenvectors calculated by the eigenvalue analysis means (103), and calculates a prediction distribution as a weighted sum, assuming that the model parameter has the weight.

Description

Statistical model learning apparatus, method and program

The present invention relates to a statistical model learning apparatus, method, and program, and more particularly, to a technique for calculating a predicted distribution in consideration of the influence of the state density of model parameters.

Statistical models are used for phoneme models in speech recognition systems and speaker models in speaker verification systems. Specifically, a hidden Markov model (hereinafter referred to as “HMM (Hidden Malkov Model)”) is often used as a phoneme model, and a mixed Gaussian model (hereinafter referred to as “GMM (Gaussian Mixture Model)”) as a speaker model. It is used.

These learning methods include a framework based on Bayesian estimation. For example, Non-Patent Document 1 describes mathematical explanations.

Learning data, which is a learning example of the random variable x, Equation 1, Numerical model parameter, which is a parameter of the statistical model, Equation 2, Learning model which gives a probability density function in which the random variable x appears when the model parameter is given, When describing the prior distribution of model parameters as Equation 4, in the framework of Bayesian estimation, the posterior distribution, which is the probability density of the model parameters when learning data is given, is expressed by Equation 5. Here, the denominator Z (X ⁿ ) in Equation 5 is called a partition function and is expressed by Equation 6.

At this time, the probability density function in which the random variable x appears when the learning data is given is expressed by Equation 7 as a predicted distribution.

However, since integration with the model parameters of Equation 6 is difficult, a probability density distribution learning method that is often used in practice is a posterior probability maximization method (maximum a posteriori: MAP estimation method, hereinafter referred to as “MAP method”) or There is a maximum likelihood estimation method.

In the MAP method, the log likelihood function is calculated as Equation 8, the model parameter that maximizes the log likelihood function is set as the estimated parameter number 9, and the prediction distribution is calculated as Equation 10.

In the maximum likelihood estimation method, the prior distribution is not used in the MAP method.

In the MAP method or the maximum likelihood estimation method, this corresponds to the prediction distribution using only one model parameter having the maximum likelihood in Equation 7 in Bayesian estimation. A statistical model learning apparatus using such a maximum likelihood estimation method will be described with an example.

Fig. 4 shows a conventional statistical model learning apparatus using the maximum likelihood estimation method. The conventional statistical model learning apparatus based on the maximum likelihood estimation method includes a learning data input unit 401, parameter calculation means 402, and predicted distribution calculation means 403. A conventional statistical model learning apparatus based on the maximum likelihood estimation method having such a configuration operates as follows.

The statistical model learning apparatus in FIG. 4 calculates model parameters satisfying Equation 9 in the parameter calculation unit 402 from the learning data input from the learning data input unit 401. For example, a learning model such as HMM or GMM is calculated using an EM algorithm. The predicted distribution calculating unit 403 calculates the predicted distribution as Equation 10 using the calculated model parameter.

In addition, as a background art related to the present invention, there is a general acoustic model learning method using an HMM described in Non-Patent Document 2. The “statistical model creation method” described in Patent Document 1 is also a technique related to the present invention.
JP 2001-083986 A Sumio Watanabe, "Algebraic Geometry and Learning Theory", Morikita Publishing Co., Ltd., April 20, 2006 L.Rabiner, BH.Juang, "Basics of Speech Recognition", NTT Advanced Technology, 1995

The problem with the learning method based on the conventional MAP method or maximum likelihood estimation method is that the performance of the prediction distribution may be lowered. The reason for this is that only the likelihood is considered, the prediction distribution is calculated from only one point having the maximum likelihood, and the model parameter distribution is not considered. Since the parameter distribution is not taken into consideration, the density of parameters having the same likelihood (state density expressed by Equation 12) is ignored.

As pointed out in Non-Patent Document 1, learning models such as HMM and GMM are called singular models, and there are singular points having the same distribution satisfying Equation 13 in different parameters.

Since the state density is a large value around these singular points, the prediction distribution is strongly affected, and the estimation method using only one parameter may lower the performance. That is, even if the model parameter does not have the maximum likelihood, the model parameter that increases the density of states for calculating the same likelihood has a large effect on the prediction distribution expressed by Equation 7, and the maximum likelihood In other words, the MAP method or the maximum likelihood estimation method using only one point may not give a sufficient prediction distribution.

An object of the present invention is to provide a statistical model learning apparatus that calculates a prediction distribution that avoids the problem of the amount of calculation due to integration by the model parameter of Equation 6 and that takes into account the influence of the state density. .

In order to achieve the above object, the present invention calculates a self-response function calculation means for calculating a self-response function of model parameters, and calculates eigenvalues and eigenvectors of a matrix having the values of self-response functions between M model parameters as components. Of the eigenvalue analysis means and the eigenvalues and eigenvectors calculated by the eigenvalue analysis means, from the larger eigenvalue, M ′ eigenvalues smaller than M and eigenvector values corresponding to M ′ are used. And a predictive distribution calculating means for calculating a predictive distribution as a weighted sum by calculating a weight and assuming that the model parameter has the weight.

According to the present invention, it is possible to provide a statistical model learning apparatus that calculates a predicted distribution that avoids the problem of calculation amount and considers the influence of the state density.

Embodiments of the present invention will be described in detail with reference to the drawings.

Referring to FIGS. 1 and 2, a statistical model learning apparatus according to the present embodiment is shown. The statistical model learning apparatus according to the present embodiment includes a learning data input unit 201, a learning data dividing unit 202, a divided learning data storage unit 203, a parameter calculation unit 204,

parameter storage units

205 and 101, and a self-response function calculation. Means 102, eigenvalue analysis means 103, and predicted distribution calculation means 104 are configured. The statistical model learning device according to the present embodiment is realized by a personal computer, for example. Here, the

parameter storage units

205 and 101 may be the same.

The learning data input unit 201 is a program that inputs learning data, for example, from its own computer or from another computer through a network. The learning data dividing unit 202 is a program that divides the learning data input by the learning data input unit 201 into M pieces. The divided learning data storage unit 203 is, for example, a hard disk device or a memory, and stores M pieces of divided data.

Learning data dividing means 202 randomly divides all learning data into M subpopulations, assuming that overlapping is allowed. When there is a sufficient amount of learning data, division that does not allow overlap may be used. The division number M can be set and specified so as to have an expected calculation amount, or set and specified so as to increase performance experimentally, or can be specified in view of a balance between the calculation amount and performance.

The parameter calculation means 204 is a program that estimates M model parameters for each of the divided M learning data. The

parameter storage units

205 and 101 are, for example, hard disk devices or memories, and store the estimated M model parameters.

The parameter calculation unit 204 learns model parameters so that the likelihood of the learning model expressed by Equation 3 is maximized for the given learning data. Not only maximum likelihood estimation but MAP estimation or the like may be used, which are represented by Equations 8 and 9.
As a learning model, for example, a complex model having hidden variables such as HMM and GMM can be used, and maximum likelihood or MAP estimation of these model parameters can be executed by using an EM algorithm. The estimated M model parameters are expressed by Equation 14.

This makes it possible to obtain M model parameter candidates having high likelihood for the learning data.

The self-response function calculating means 102 is a program for calculating a self-response function through a random variable x between model parameters. The self-response function calculating unit 102 calculates a self-response function between the model parameters ω and ω ′ through the random variable x, which is expressed by Equation 15.

Here, the integration of the random variable x can be calculated analytically when the learning model is GMM. In the case of a more complex model such as an HMM, it can be executed by replacing it with the sum of the values of learning data. It is not necessary to use all of the sum of the learning data, or conversely, the sum of the points other than the learning data can be calculated by including a point such as a linear combination of the learning data.

The eigenvalue analyzing means 103 is a program for calculating eigenvalues and eigenvectors in a matrix having values calculated by the self-response function calculating means 102 among the M model parameters calculated by the parameter calculating means 204.

The eigenvalue analyzing means 103 constructs a matrix of M rows and M columns, the component of which is the value of the self-response function between the M model parameters, and an eigenvector (Equation 17) corresponding to the eigenvalue (Equation 16) of this matrix. Is calculated.

The prediction distribution calculation unit 104 uses M ′ eigenvalues (M ′ <M) having large eigenvalues and eigenvectors corresponding to the M eigenvalues and eigenvectors calculated by the eigenvalue analysis unit 103 to obtain M pieces of eigenvalues and eigenvectors. It is a program that calculates weights of model parameters and calculates a predicted distribution as a sum of these weights.

The predicted distribution calculation means 104 calculates the predicted distribution using Equation 18.

Here, it can be seen from Equation 19 that the square of the component of the first eigenvector corresponds to the eigenvalue in the learning model.

M ′ can be set and specified so as to have an expected calculation amount, or set and specified so as to increase performance experimentally, or can be specified in view of the balance between the calculation amount and performance.

Comparing Eq. 18 and Eq. 7, in the present invention, the weighted value is obtained using the eigenvalue and eigenvector value calculated from the self-response function of the model parameter instead of the posterior distribution of the model parameter (Equation 5). ing.

This avoids the difficulty of computational complexity such as the integration of the model parameters of Equation 6, and increases the weight to model parameters that have a high likelihood of learning data and a large self-response function, that is, model parameters that give the same likelihood. By doing so, a predicted distribution in consideration of the influence of the state density is calculated.

Next, another embodiment of the present invention will be described. Referring to FIG. 3, the configuration of another embodiment of the present invention is shown. Another embodiment of the present invention is a configuration including a learning data input unit 301, a parameter initial value storage unit 302, a parameter calculation unit 303, and a parameter storage unit 304.

The parameter calculation unit 303 calculates M model parameters using the input learning data and M different initial parameters, and stores them in the parameter storage unit 304.

The parameter calculation means 303 learns model parameters so that the likelihood of the learning model expressed by Equation 3 is maximized for the given learning data. Not only maximum likelihood estimation but MAP estimation or the like may be used, which are represented by Equations 8 and 9.

As a learning model, for example, a complex model having hidden variables such as HMM and GMM can be used, and maximum likelihood or MAP estimation of these model parameters can be executed by using an EM algorithm. Since the EM algorithm depends on the initial parameters, even if the same learning data is used, different model parameters are estimated when different initial values are used. Alternatively, M model parameters can be calculated using a combination of the present embodiment and the above-described embodiment, using learning data division and different initial values. As a result, M model parameter candidates having high likelihood can be obtained for the learning data.
This completes the description of another embodiment of the present invention.

Next, the structure of the principal part of each embodiment mentioned above and its effect | action are demonstrated again.
The statistical model learning apparatus according to the embodiment of the present invention described above includes a parameter storage unit 101 that stores M model parameters, a self-response function calculating unit 102 that calculates a self-response function of model parameters, and M model parameters. And eigenvalue analysis means 103 for calculating eigenvalues and eigenvectors of a matrix having self-response function values as components, and prediction distribution calculation means 104 for calculating a prediction distribution using the calculated eigenvalues and eigenvectors. Has been.

In this embodiment, the self-response function calculation means 102 calculates the self-response function through the random variable x between the M model parameters stored in the model parameter storage unit 101 in this way. Since the value of the self-response function becomes a large value when the overlap of the probability density functions of the random variables x obtained in the respective model parameters is large, the value becomes large between model parameters giving the same likelihood.

An M-row and M-column matrix having the values of the self-response function between the M model parameters as components is constructed, and eigenvalues and eigenvectors of this matrix are calculated by the eigenvalue analysis means 103, and M ′ having a large eigenvalue is obtained. The weights of the M model parameters are calculated using the individual (M ′ <M) eigenvalues and the corresponding eigenvectors, and the prediction distribution is calculated as the weighted sum. As a result, the present embodiment considers the influence of the state density by increasing the weight to the model parameter that increases the self-response function without performing integration with the model parameter of Equation 6, that is, the model parameter that gives the same likelihood. The predicted distribution is calculated.

Therefore, the effect of this embodiment is that a predicted distribution can be calculated in consideration of the influence of the state density of the model parameter without performing integration by the model parameter. This is because the weight of model parameters is calculated using eigenvalue analysis of a matrix whose value is a self-response function between model parameters, and the predicted distribution is calculated as a weighted sum thereof.

Next, the best mode for carrying out the present invention will be described using specific examples.
This embodiment uses an HMM as an acoustic model for speech recognition, and in particular, when the GMM is a probability density function that outputs a feature vector from each state of the HMM, this GMM parameter is learned as an example. To do. The drawings showing the configuration of this embodiment are shown in FIGS.

In speech recognition, speech data is made into feature vectors by cepstrum analysis or the like and used as learning data.
By performing Viterbi alignment using an existing HMM, learning data can be assigned to each state of the HMM. Focusing on the GMM in the current state s, the feature vector assigned to the state s is input to the learning data input unit 201.

The learning data dividing unit 202 divides the input data into M pieces at random. The parameter calculation means 204 calculates M model parameters by the EM algorithm using the learning data divided into M pieces.
A model parameter is a set of GMM mixture weights, mean vectors, and covariance matrices.

The self-response function between the model parameters is calculated using the formula 15 in the self-response function calculation means 102. The integration of the feature vector x can be performed analytically in the case of GMM. Alternatively, it can also be executed approximately by replacing it with the sum of the feature vectors input to the learning data input unit 201.

The eigenvalue analysis means 103 performs eigenvalue analysis. In the simplest case, “M ′ = 1” is used, and the maximum eigenvalue and the value of its eigenvector are used. In the predicted distribution calculation means 104, M ′ = 1 in the equation 18, and the predicted distribution is calculated.
In this case, it can be seen from Equation 18 that since the weighted sum of M GMMs is formed, the entire prediction distribution is again a GMM. It should be noted that this property is not limited to GMM, and particularly when learning a mixed distribution type model, this property is maintained by Equation 18, even if M ′ = 1.

This application claims priority based on Japanese Patent Application No. 2007-328435 filed on December 20, 2007, the entire disclosure of which is incorporated herein.

It is a block diagram (the 1) which shows the structure of the statistical model learning apparatus which concerns on embodiment of this invention. It is a block diagram (the 2) which shows the structure of the statistical model learning apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of another embodiment of this invention. It is a figure for demonstrating a prior art.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Parameter memory | storage part 102 Self-response function calculation means 103 Eigenvalue analysis means 104 Prediction distribution calculation means 201 Learning data input part 202 Learning data division means 203 Division learning data storage part 204 Parameter calculation means 205 Parameter storage part 301 Learning data input part 302 Parameter Initial value storage unit 303 Parameter calculation unit 304 Parameter storage unit 401 Learning data input unit 402 Parameter calculation unit 403 Predictive distribution calculation unit

Claims

A self-response function calculating means for calculating a self-response function of the model parameter;
Eigenvalue analysis means for calculating eigenvalues and eigenvectors of a matrix having components of the self-response function between M model parameters;
Of the eigenvalues and eigenvectors calculated by the eigenvalue analysis means, the model parameter weight is calculated using M ′ eigenvalues smaller than M and the eigenvector values corresponding to M ′ from the larger eigenvalues, A prediction distribution calculating means for calculating a prediction distribution as a weighted sum, assuming that the parameter has its weight;
A statistical model learning apparatus comprising:
Having model parameter calculation means for calculating M model parameters;
2. The statistical model learning apparatus according to claim 1, wherein the self-response function calculating means uses the calculated M model parameters as model parameters.
3. The statistical model learning device according to claim 2, wherein the model parameter calculation means calculates M model parameters using learning data and M different initial values.
Learning data dividing means for dividing the learning data into M subsets;
3. The statistical model learning device according to claim 2, wherein the model parameter calculation means calculates M model parameters using the divided learning data.
A self-response function calculation process for calculating a self-response function of a model parameter;
An eigenvalue analysis process for calculating eigenvalues and eigenvectors of a matrix having self-response function values among M model parameters as components;
Of the eigenvalues and eigenvectors calculated in the eigenvalue analysis process, the model parameter weight is calculated using M ′ eigenvalues smaller than M and eigenvector values corresponding to M ′ from the larger eigenvalues, Prediction distribution calculation processing that calculates a prediction distribution as a weighted sum, assuming that the parameter has its weight,
A statistical model learning method characterized by comprising:
In the information processing device,
A self-response function calculation process for calculating a self-response function of a model parameter;
An eigenvalue analysis process for calculating eigenvalues and eigenvectors of a matrix having self-response function values among M model parameters as components;
Of the eigenvalues and eigenvectors calculated in the eigenvalue analysis process, the model parameter weight is calculated using M ′ eigenvalues smaller than M and eigenvector values corresponding to M ′ from the larger eigenvalues, Prediction distribution calculation processing that calculates a prediction distribution as a weighted sum, assuming that the parameter has its weight,
A statistical model learning program characterized in that