JP2006126398A - Method of compressing plurality of probability density function kernels of probability model, and computer program therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable effective compression of a probability model, while maintaining performance. <P>SOLUTION: A method of compressing an HMM model includes the steps of clustering the means of the probability density function (pdf) kernels 20, 22, 24, 26, 28 and 30 of the HMM model into first centroid kernels 40 and 42; clustering the variances of the pdf kernels 20, 22, 24, 26, 28 and 30 into second centroid kernels 50 and 52; and redefining each of the pdf kernels 20, 22, 24, 26, 28 and 30 by means of one of the first centroid kernels 40 and 42 that is nearest the original mean of pdf kernels, and by dispersion of one of the second centroid kernels 50 and 52 that is nearest the original pdf kernel. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to efficient modeling of probability, and more particularly to compression of a probability model such as HMM (Hidden Markov Model) for reducing storage capacity and computational complexity.

For efficient automatic speech recognition, there is always a desire for a smaller model that has a small capacity and a small amount of calculation, and that can maintain favorable recognition performance. By applying state sharing [see non-patent document 1] or distribution sharing [see non-patent document 2] for so-called semi-continuous HMMs during training, a small model can be constructed. The constructed HMM basic pdf (probability density function or probability density function) kernel can also be clustered in the feature space after the training procedure. It has been proposed that the pdf kernel average can be clustered in each feature dimension [see non-patent document 3] or subspace [see non-patent document 4].
S. J. et al. Young and P.C. C. Woodland, “Use of State Sharing in Continuous Speech Recognition”, Eurospeech-1993, pp. 2203-6, 1993 (SJ Young and PC Woodland, "The Use of State Tying in Continuous Speech Recognition", Eurospeech-1993, pp. 2203-6, 1993.) X. D. Han, “Phonology classification using semi-continuous hidden Markov model”, IEEE ASSP transaction, Vol. 40, no. 5, pp. 1062-7, 1992 (XD Huang, "Phoneme Classification Using Semicontinuous Hidden Markov Models", IEEE Trans. ASSP, Vol. 40, No. 5, pp. 1062-7, 1992.) S. Takahashi and S.H. Sagayama, “4-Level Shared Structure for Efficient Representation of Acoustic Modeling”, ICASSP-1995, pp. 520-3, 1995 (S. Takahashi and S. Sagayama, "Four-Level Tied-Structure for Efficient Representation of Acoustic Modeling", ICASSP-1995, pp. 520-3, 1995.) E. Bocchieri and K. W. Mac, “Hidden Markov Model with Sub-Space Model Clustering”, IEEE Acoustic Processing Transactions, Vol. 9, no. 3, pp. 264-75, March 2001 (E. Bocchieri and KW Mak, "Subspace Distribution Clustering Hidden Markov Model", IEEE Trans. Speech Audio Proc. Vol. 9, No. 3, pp. 264-75, Mar. 2001. ) T. T. et al. A. Millfort and F.M. K. Soon, “Optimal Clustering of Multivariable Normal Distribution Using Divergence and Application to HMM and Its Application”, ICASSP-2003, pp. 552-5, 2003 (TA Myrvoll and FK Soong, "Optimal Clustering of Multivariate Normal Distributions Using Divergence and Its Application to HMM Adaptation", ICASSP-2003, pp. 552-5, 2003.) J. et al. Kim, R.A. Haimi-Cohen and F.M. K. Soon, “Hidden Markov Model with Divergence-Based Vector Quantization Variables”, ICASSP-1999, pp. 125-8, 1999 (J. Kim, R. Haimi-Cohen, and FK Soong, "Hidden Markov Models with Divergence based Vector Quantization Variances", ICASSP-1999, pp. 125-8, 1999.)

  In Non-Patent Document 3, high model resolution is maintained by clustering only the average in each feature dimension. However, a large memory space and a large amount of calculation are still necessary.

  In Non-Patent Document 4, clustering is performed by combining the average and the variance. However, due to the resulting quantization error, a good model representation cannot be guaranteed by a centroid clustered together. In order to explain this point, an example is shown in FIG. Here, six Gaussian distribution kernels are clustered into two clusters. Kernels 20, 22, 24 are clustered into one cluster indicated by centroid kernel 400, and kernels 26, 28, 30 are clustered into another cluster indicated by centroid kernel 402.

  The resulting centroid does not represent well the individual elements in the cluster, especially with respect to dispersion.

  Accordingly, one object of the present invention is to provide a method for effectively compressing a probabilistic model while maintaining the performance of the original model.

  One aspect of the present invention relates to a method for compressing a plurality of pdf kernels of a probability model. Each pdf kernel is defined by a first parameter and a second parameter. The method includes clustering a first parameter of a plurality of pdf kernels into one or more first centroid kernels, and a second parameter of the plurality of pdf kernels into one or more second centroid kernels. The first parameter of the first centroid kernel and the first centroid kernel having the first parameter closest to the first parameter of each of the plurality of pdf kernels; And redefining each of the plurality of pdf kernels with a second parameter having a second parameter closest to each of the plurality of pdf kernels.

  The plurality of pdf kernels are clustered into a first centroid kernel and a second centroid kernel. Each of the pdfs is then redefined by the first parameter of the closest one of the first centroid kernels and the second parameter of the closest one of the second centroid kernels. Since the pdf kernel is redefined with fewer centroid kernel parameters, the total amount of storage required by the model is reduced.

  Preferably, each of the plurality of pdf kernels includes a Gaussian pdf kernel. The first and second parameters include the mean and variance of the Gaussian pdf kernel.

  The mean and variance of the Gaussian pdf kernel are clustered separately. For this reason, the storage capacity is reduced and the calculation amount is also reduced.

  More preferably, the step of clustering the first parameter is the sum error between the redefined Gaussian pdf kernel and each corresponding original pdf kernel (Kullback-Leibler Divergence (KLD)). Clustering the average of the Gaussian pdf kernel so that) is minimized.

  More preferably, the step of clustering the second parameter is such that the sum of errors between the redefined Gaussian pdf kernel and each corresponding original pdf kernel with zero mean is minimized. Clustering the variances.

  The error of a given two probability density functions may be calculated as a symmetric Kalbach-librarian divergence between the two given probability density functions.

  By clustering the Gaussian pdf and redefining the Gaussian pdf using the clustered centroid kernel, the model becomes smaller and less computationally intensive.

  Another aspect of the present invention relates to a computer program that, when executed on a computer, causes the computer to execute all the steps of any of the methods described above.

[Introduction]
In order to find a good trade-off between model resolution and resource allocation, in this embodiment, according to the optimal cluster centroid calculation [see Non-Patent Document 5], the symmetric Kalbach-Librer divergence is an error measure. We propose to cluster the mean and variance separately in each scalar dimension. The approximation of the optimal centroid has already been proposed in Non-Patent Document 6.

  Specifically, as shown in FIG. 1, the average of kernels 20, 22, 24, 26, 28, and 30 is clustered into two average centroid kernels (hereinafter referred to as “average cluster kernels”) 40 and 42. . That is, the kernels 20, 22 and 24 are clustered into the average cluster kernel 40, and the kernels 26, 28 and 30 are clustered into the average cluster kernel 42. Further, the distributions of the kernels 20, 22, 24, 26, 28 and 30 are clustered into distributed kernels (hereinafter referred to as “distributed cluster kernels”) 50 and 52. That is, the kernels 20 and 28 are clustered into the distributed cluster kernel 50, and the kernels 22, 24, 26 and 30 are clustered into the distributed cluster kernel 52. By clustering the mean and variance separately, a higher model resolution is obtained.

[Optimal centroid based on KLD]
The symmetric Kalbach-Librer divergence used here to measure the error (distance) between two given pdfs, f and g is defined by the following equation:

以下の式に従ってクラスタセントロイドとクラスタ内の全てのカーネルとの合計ＫＬＤを最小化することにより、最適クラスタセントロイドｆ_ｃが得られる。

By minimizing the total KLD with all kernel in a cluster centroid and the cluster according to the following equation, the optimal cluster centroid f _c is obtained.

多変数ガウス分布では、式（１）の対称ＫＬＤの閉じた式があり、これは以下で表される。

In the multivariable Gaussian distribution, there is a closed equation for the symmetric KLD in equation (1), which is expressed as:

ここでμ及びＲは対応する分布の平均と分散とである。最適セントロイドは一組のリカッティ行列式[非特許文献５を参照]を解くことで得られる。対角共分散の特殊な事例では、最適セントロイドのｉ次元の平均と分散、すなわちμ_ｃｉ及びσ_ｃｉ ^２は以下の通りである。

Here, μ and R are the mean and variance of the corresponding distribution. The optimal centroid can be obtained by solving a set of Riccati determinants [see Non-Patent Document 5]. In the special case of diagonal covariance, the i-dimensional mean and variance of the optimal centroid, ie, μ _ci and σ _ci ² are as follows:

[Why do clustering of mean and variance separately?]
As shown in FIG. 9, the mean and variance of the Gaussian distribution kernel can be clustered together. However, the resulting centroid does not represent well the individual elements in the cluster, especially with respect to dispersion.

  This model resolution problem can be overcome by clustering the mean and variance separately. The variance can be clustered by setting the corresponding average to zero. As shown at the bottom of FIG. 1, the variances of kernels 20 and 28 are clustered into the left variance indicated by the distributed cluster kernel 50, and the variances of kernels 22, 24, 26, 30 are indicated by the distributed cluster kernel 52 Clustered into a variance of

  Each of the averages of the original kernels 20, 22, 24, 26, 28 and 30 is approximated by the average of the nearest neighbors of the average cluster kernels 40 or 42. Similarly, each of the variances of kernels 20, 22, 24, 26, 28 and 30 is approximated by the variance of the nearest neighbor of variance kernels 50 or 52. By clustering the variance separately from the average clustering, obviously a high model resolution can be maintained.

[Construction]
FIG. 2 shows the structure of an automatic speech recognition (ASR) system 60 according to an embodiment of the present invention. Referring to FIG. 2, ASR system 60 trains an HMM acoustic model using training corpus 70 including segmented speech data with speech notation and speech data in training corpus 70 as training data. And a compression module 76 for compressing the HMM acoustic model 74 by separately clustering the mean and variance of each kernel of the states in the HMM acoustic model 74 as described above. . The resulting compressed HMM acoustic model 78 is used for ASR.

  The ASR system 60 further receives the language model 80 and speech data of the input utterance 82, recognizes the input speech using the HMM acoustic model 78 and the language model 80, and outputs the resulting text 86. Including.

  The compression module 76 can be realized by software. The overall control structure of the software is shown in FIG.

Referring to FIG. 3, when this program is started, it repeatedly initializes control variable i to zero in step 100. In step 102, the variable i is incremented by one. In step 104, it is determined whether the variable i is larger than the dimension number _{Ndim of the} HMM parameter. If the variable i is greater than N _dim , execution of this program ends. Otherwise, control proceeds to step 106.

  At step 106, the averages of the i th dimension kernels are clustered. Details of this step will be described later with reference to FIG. When the average clustering is finished, the distribution of kernels in the i-th dimension is clustered in step 108. Here, the kernel variance is clustered with the kernel average fixed at zero. Details of step 108 will be described later with reference to FIG.

  After step 108, control proceeds to step 116 where the kernel obtained in steps 106 and 108 is written to the codebook page for the i-th kernel of the HMM model. At step 118, the mean and variance are assigned to the i-th kernel of the HMM.

  Each i th dimension Gaussian kernel is assigned the average of the nearest neighbor of the clustered averages and the variance of the nearest neighbor of the clustered variances. In other words, each original Gaussian kernel is redefined by the average of the nearest neighboring average cluster kernel and the variance of the nearest neighboring distributed cluster kernel.

  After step 118, control returns to step 102, and compression of the (i + 1) -dimensional kernel is performed.

  FIG. 4 shows details of step 106. Referring to FIG. 4, the average of the i-th kernel is clustered by the following steps. At step 130, the iterative control variable j is initialized to zero and another variable Q_old is initialized to the maximum value that the computer can handle.

  In step 132, one kernel is added to the clustering kernel. That is, clustering starts by clustering the average of the i-dimensional Gaussian kernels into one clustering kernel. At step 134, variable j is incremented by one.

  At step 136, an optimal single (or multiple) average cluster kernel is calculated utilizing the KLD-based optimization described above. The resulting KLD value is stored as Q_new.

  At step 138, ΔQ is calculated as the difference between Q_old and Q_new. That is, ΔQ = Q_old−Q_new.

  In step 140, it is determined whether or not ΔQ is smaller than a predetermined threshold value δ. If ΔQ is less than δ, control proceeds to step 144. Otherwise, control proceeds to step 142.

  In step 142, the variable Q_new is saved as Q_old and control returns to step 132.

  In step 144, the kernel obtained in the jth iteration is selected as the optimal average cluster kernel. After step 144, control exits this routine and returns to step 108 of FIG.

  FIG. 5 shows details of step 108 of FIG. During this process, the kernel average is fixed to zero. Referring to FIG. 5, the distribution of the i-th kernel is clustered in the following steps. At step 160, the iteration control variable j is initialized to zero, and another variable Q_old is initialized to the maximum value that the computer can handle.

  In step 162, one kernel is added to the clustering kernel. That is, clustering is started by clustering the variance of the i-th Gaussian kernel into one clustering kernel. At step 164, variable j is incremented by one.

  At step 166, an optimal single (or multiple) distributed cluster kernel is calculated utilizing the KLD-based optimization described above. The resulting KLD value is stored as Q_new.

  At step 168, ΔQ is calculated as the difference between Q_old and Q_new.

  In step 170, it is determined whether ΔQ is smaller than δ. If ΔQ is less than δ, control proceeds to step 174. Otherwise, control proceeds to step 172.

  In step 172, the variable Q_new is saved as Q_old and control returns to step 162.

  In step 174, the kernel obtained in the jth iteration is selected as the optimal distributed cluster kernel. After step 174, control exits this routine and returns to step 116 of FIG.

[Realization by computer]
The above-described embodiment can be realized by a computer system and a computer program executed by the system. The software control structure has been described with reference to FIGS. FIG. 6 is an external view of the computer system 330 of this embodiment, and FIG. 7 shows the system 330 in a block diagram.

  Referring to FIG. 6, a computer system 330 includes a computer 340 having an FD (Flexible Disk) drive 352 and a CD-ROM (Compact Disc Read-Only Memory) drive 350, a keyboard 346, a mouse 348, a monitor 342, and the like. including.

  Referring to FIG. 7, the computer 340 further includes a CPU (Central Processing Unit) 356, a bus 366 connected to the CPU 356, the CD-ROM drive 350 and the FD drive 352, and a ROM (Read) for storing a bootup program and the like. -Only Memory) 358, a RAM (Random Access Memory) 360 connected to the CPU 356 for storing application program instructions, system programs and data, and a printer 344.

  Although not shown here, the computer 340 may further include a network adapter board that provides a connection to a local area network (LAN).

  A program for causing the computer system 330 to execute the method for compressing the HMM model is stored in the CD-ROM 362 or the FD 364 and the like, and when these are inserted into the CD-ROM drive 350 or the FD drive 352, the program is further transferred to the hard disk 354. . Alternatively, the program may be transmitted to the computer 340 via a network (not shown) and stored in the hard disk 354. The program is loaded into the RAM 360 when executed. The program may be loaded directly into the RAM 360 from the CD-ROM 362, the FD 364, or the network.

  The program described with reference to FIGS. 3 to 5 includes a plurality of instructions for causing the computer 340 to perform the method of this embodiment. Some of the basic functions required to perform this method are performed by modules such as an operating system (OS) or a third party program running on computer 340, or an HMM toolkit installed on computer 340. As provided, the program need not necessarily include all the basic functions necessary to implement the embodiments of the present invention. The program only needs to include a portion of an instruction that performs compression processing by calling an appropriate function or “tool” in a controlled manner to obtain a desired result. Since the operation of the computer 330 is well known, a description thereof will be omitted.

[Operation]
The ASR system 60 of this embodiment operates as follows. Referring to FIG. 2, a training corpus 70 is prepared. Training corpus 70 includes segmented speech data and associated phonetic notations.

  The training module 72 trains the HMM acoustic model 74 using the HMM acoustic model 74 as training data. Tools for training the HMM model are readily available, so training details will not be described here.

  Next, the compression module 76 compresses the HMM acoustic model 74 as follows. First, the compression module 76 clusters the average of the Gaussian kernels of the first parameter dimension in each state of the HMM acoustic model 74 so as to minimize the KLD to obtain the optimum number of average cluster kernels (FIG. 3, step). 106). Next, the compression module 76 clusters the Gaussian kernels of the first parameter dimension in each state of the HMM acoustic model 74 so as to minimize the KLD to obtain the optimum number of distributed cluster kernels (FIG. 3, step). 108). At this time, the average of the kernel is fixed to zero.

  In step 116, the cluster kernel thus obtained is written into the first dimension codebook page. Next, the first dimension Gaussian kernel is redefined by an average cluster kernel, each having an average closest to the average of the original Gaussian kernel, and a distributed cluster kernel, each having a variance closest to the variance of the original Gaussian kernel. The

  The above steps are then repeated for the other parameter dimensions of each state of the HMM acoustic model 74 to compress the HMM acoustic model. When all parameter dimensions of the HMM acoustic model are compressed, the compressed model is output from the compression module 76 as an HMM acoustic model 78.

  When the HMM acoustic model 78 and the language model 80 are available, the ASR module 84 is ready to receive the input utterance 82 and recognizes speech data using the HMM acoustic model 78 and the language model 80. The recognized text 86 is output.

[Experimental result]
The clustering method of this embodiment was tested on a DARPA (The Defense Advanced Research Projects Agency) 991 word resource management database. For this implementation, a standard SI-109 training data set (3,990 utterances) was used. Using CMU (Carnegie Mellon University) 48 phone sets, context-independent (CI) phoneme models were generated, each with 3 states and 12 Gaussian mixing elements per state. The features were conventional 39-dimensional mel-frequency cepstrum counts (MFCC) (12 static MFCCs and logarithmic energy, and their first and second derivatives).

  The Perplexity 60 standard word-pair grammar was evaluated using the Feb89 test set. The baseline recognition performance of the original, unquantized HMM was 92.82% in word accuracy.

  (1) Separate clustering in which the mean and variance are clustered separately (this embodiment), (2) Clustering that combines the mean and variance (Non-Patent Document 4), and (3) The original variance Comparison was made with only the average clustering used (Non-Patent Document 3). The results are shown in FIG.

  As can be seen in FIG. 8, when the average and variance were clustered separately, the performance of clustering combining the average and variance exceeded the performance. Also, when the variances were clustered separately, the recognition performance was equal (or slightly better) with 16 or more clusters per dimension than the variance without clustering.

表１はこの実施の形態に従った最適にクラスタリングされたモデル（平均と分散との別個のクラスタリング）の記憶容量及び計算量の、元のクラスタリングされていないＨＭＭに対するパーセンテージを示す。容量は、クラスタの数により１２％〜２４％までに減少した。演算量に関しては、ｊ番目の状態の対数尤度は以下のように計算される。 [Storage capacity and computational requirements]

Table 1 shows the percentage of storage capacity and complexity of the optimally clustered model (mean and variance separate clustering) according to this embodiment relative to the original unclustered HMM. Capacity decreased from 12% to 24% depending on the number of clusters. Regarding the computational complexity, the log likelihood of the jth state is calculated as follows.

平均と分散とをクラスタリングしているので、第３項(o_ti-μ_jmi)²/2σ_jmi ²は予め計算して異なる出力ｐｄｆで共有する。表１に示すように、乗算／除算は２％〜１９％まで減少し、加算／減算は５２％〜５４％までに減少し、性能の劣化はわずかであった。

Since the mean and variance are clustered, the third term (o _ti -μ _jmi ) ² / _{2σ jmi} ² is calculated in advance and shared by different output pdfs. As shown in Table 1, multiplication / division decreased from 2% to 19%, addition / subtraction decreased from 52% to 54%, and the performance degradation was slight.

[Conclusion]
A multivariate, diagonal co-fraction based HMM Gaussian distribution kernel was optimally clustered by minimizing the corresponding symmetric Kalbach-Librer divergence in each scalar dimension. By clustering the mean and variance separately, we maintained the high model resolution of the original HMM. In the evaluation using the resource management database, the amount of storage and the amount of calculation could be reduced considerably without degrading performance.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the embodiment described above. The scope of the present invention is indicated by each claim in the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

It is a figure which shows roughly the clustering of the average and dispersion | distribution according to one embodiment of this invention. It is a block diagram which shows the structure of ASR system 60 according to one embodiment of this invention. It is a flowchart of the clustering program according to the embodiment. It is a flowchart of the average clustering according to the embodiment. It is a flowchart of distributed clustering according to an embodiment. It is a figure which shows the computer system 330 which executes a clustering program. 2 is a block diagram showing a structure of a computer system 330. FIG. It is a figure which shows the ASR performance according to embodiment of this invention. It is a figure which shows the clustering by a prior art schematically.

Explanation of symbols

20, 22, 24, 26, 28, 30 Gaussian kernel 40, 42 Average cluster kernel 50, 52 Distributed cluster kernel 60 ASR system 70 Training corpus 72 Training module 74 HMM acoustic model 76 Compression module 78 Compression HMM acoustic model 80 Language model 84 ASR module

Claims (6)

A method of compressing a plurality of probability density function kernels of a probability model, wherein the plurality of probability density function kernels are each defined by a first parameter and a second parameter,
Clustering first parameters of the plurality of probability density function kernels into one or more first centroid kernels;
Clustering a second parameter of the plurality of probability density function kernels into one or more second centroid kernels; and a first parameter of each of the plurality of probability density function kernels of the first centroid kernels. A first parameter having a first parameter closest to one parameter and a second parameter having a second parameter closest to each of the plurality of probability density function kernels among the second centroid kernels. Redefining each of the plurality of probability density function kernels with two parameters. The method of claim 1, wherein each of the plurality of probability density function kernels includes a Gaussian probability density function kernel, and the first and second parameters include a mean and variance of a Gaussian probability density function kernel. Clustering the first parameter comprises averaging the Gaussian probability density function kernels such that an error between the redefined Gaussian probability density function kernel and each corresponding original probability density function kernel is minimized. The method of claim 2, comprising clustering. The step of clustering the second parameter includes the Gaussian probability such that the sum of errors between the redefined Gaussian probability density function kernel and each corresponding original probability density function kernel having zero mean is minimized. 4. A method according to claim 2 or 3, comprising clustering the variance of the density function kernel. 5. A method according to claim 3 or claim 4, wherein the error of a given two probability density functions is calculated as a symmetric Kalbach-Librarian divergence between the two given probability density functions. A computer program that, when executed on a computer, causes the computer to execute all the steps according to any one of claims 1 to 5.

Images