JPH09244689A - Parallel estimating method for hmm parameter - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain a parallel estimating method averagely equivalent to probable processings by preparing plural pieces of estimating series of parameters based on algorithm and making respective microcomputers execute respective estimating series parallel. SOLUTION: At first, plural pieces of estimating series of parameters based of algorithm are prepared (procedures 411, 421) and respective microcomputers PEs are made to execute the respective estimating series parallel (procedures 412, 422). The respective estimating series are finally made to be converged to a global optimum by correcting estimating series while applying such a probable method as, for example, GA with respect to these plural estimating series. At second, a processing is executed by a pseudo establishing process by limiting a selection object to an adjacent PE in order to reducing the load of the data communications among the PEs. Moreover, combinations of selections which do not perform a communication confliction are extracted preliminarily and a means selecting the pattern probably is used in order to avoid the confliction among the communications. At third, the exchanging of processing PEs is performed at the time of comparing and correcting of estimating series. Thus, the parallel estimating method becomes equivalent to the probable processings.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a hidden Markov model (HM) used for recognition processing in a voice recognition device using a plurality of microcomputers (hereinafter abbreviated as PE).
M) relates to a method for efficiently estimating the parameters.

[0002]

[Prior art]

1. HMM Parameter Estimation Problem in Speech Recognition 1.1 HMM in Speech Recognition Speech recognition is configured by a system as shown in FIG. 1, for example. A voice signal input by the voice input device 102 such as a microphone is converted into a digital signal by AD conversion. Then, in order to express the stationary nature of the sound, the feature is extracted for each certain section (hereinafter, frame). In the recognition processing, recognition is performed based on the time-series property of the parameter group (hereinafter, feature vector) from which the feature is extracted. An HMM is used as a model that expresses the time-series sound properties.

The present invention envisions a process of estimating the HMM parameter based on the feature-extracted signal when the HMM parameter used in this recognition process is dynamically adjusted.

1.2 Basic Principle of HMM-based Speech Recognition Here, the HMM will be briefly described within the range necessary for explaining the present invention.
The voice recognition method by will be described. First, consider a Markov model as shown in FIG. The Markov model means “a stochastic model in which the state at the time point n + 1 is given only by the state at the time point n”. This stochastic behavior is expressed by a transition probability between states. In speech recognition, a unique output probability (specifically, the probability of outputting a pattern of a certain feature vector) is given as a state. Then, it is compared with the input sequence of feature vectors (hereinafter, observation sequence),
Guess the state transition process. Specifically, a state transition sequence having the highest probability of outputting a feature vector such as an observation sequence is calculated. Each recognition element corresponds to a type of a state transition series. With this series of procedures, the recognition element with the highest probability can be calculated. Note that HMM Hidden means that the direct state transition cannot be observed (thus estimated).

1.3 Parameter Estimation Problem in HMM In the HMM described above, it is necessary to calculate the parameters of the HMM (specifically, the state transition probabilities a _ij and b _ij ). Baum has realized an HMM parameter estimation method using the maximum likelihood estimation principle.

In the maximum likelihood estimation method, the system is expressed as a function of a certain parameter (hereinafter, a likelihood function),
For the observation series as a teacher, the parameters are determined so that the output of the system becomes maximum. Specifically, Baum's algorithm is executed in the following procedure.

Step 1 Selection of initial value Step 2 Iterative estimation of parameters Step 3 Determination of state transition probability and output probability Of these, iterative calculation is required in Step 2. This procedure is Step 2-1 Calculation of forward variables Step 2-2 Calculation of backward variables Step 2-3 Estimation of state transition probability (calculation of intermediate estimated values) Step 2-4 Output Probability estimation (Same as above) In Baum's algorithm, Forwar of Steps 2-1, 2-2
The calculation of the d variable and the Backward variable depends on the state transition probability and the output probability, as in the following Expression 1.

[0008]

[Equation 1]

The calculation of the state transition probabilities and output probabilities in Steps 2-3 and 2-4 depends on the Forward variable and the Backward variable as shown in the following Expression 2.

[0010]

[Equation 2]

In such iterative calculation, whether to converge to the optimum solution depends on the initial setting. Such problems are often found in nonlinear optimization problems.

In the present invention, the stochastic search method in the nonlinear optimization problem is partially applied. Therefore, next, a conventional technique of the stochastic search method in the nonlinear optimization problem is shown.

1.4 Stochastic Search Method in Nonlinear Optimization Problem In order to solve the optimization problem of a multimodal function, a probabilistic search method is generally used. Here, a representative simulated annealing method (hereinafter, SA) method and a genetic algorithm (GA method) will be briefly described.

(1) SA Method The SA method is in the limelight as a method for avoiding local optimization. This stochastically updates (transitions) states from the analogy of thermodynamics. That is, the evaluation function to be minimized is regarded as an energy function, and the old state and the new state are compared. Then, if the energy of the new state is low, the transition is made with a probability of 1 (that is, to the new state without fail), and if the energy of the new state is high, the transition is made with the probability of exp (-ΔE / T) according to the Boltzmann distribution. Here, the temperature T is a parameter that determines the transition probability.

According to the analogy with thermodynamics, in the SA method, the temperature T is initially increased, and the temperature T is gradually decreased to reach the minimum energy state. In other words, it is a method in which the transition probability is initially set large and the transition probability is gradually reduced to reach the global optimum solution.

This is likened to the fact that the annealing that slowly cools from the high temperature to the rapid cooling (which is likened to the steepest descent method) is close to the minimum state in terms of energy. Although the SA method is an approximate solution,
It is known that a relatively good quality solution can be obtained. most,
The calculation time of this processing is enormous, which is a problem when the SA method is applied.

(2) GA method The GA method has been in the spotlight in recent years as one method of the stochastic search method. This GA method was proposed by Holland mainly from the analogy of biological evolution. The GA method is calculated by each process of "proliferation / crossover / mutation / selection" corresponding to the evolution of living things.

For example, an example in which the GA method is simply applied to a nonlinear optimization problem will be shown. First, a plurality of points (hereinafter, individuals) are randomly selected on the domain. Here, the value of each point (for example, a binary bit pattern) is used as a gene.

In the GA method, the same individual (bit pattern) as the parent is generated. This is called proliferation. Propagation is also a kind of stochastic process and affects the probability of being selected in the next cross. Next, constant mating, that is, gene recombination, is performed from the generated individuals. This is equivalent to updating the value in the usual iterative method. After this, a stochastic gene change called mutation is performed. By moving the solution at a certain probability frequency in this way, it is possible to prevent the solution from falling to the local optimum. Individuals generated by these processes are selected or disappear based on a certain evaluation. As these processes pass through generations, they gradually converge to an optimal solution.

Therefore, the GA method, like the SA method, requires a very long calculation time. However, each processing is relatively independent and is a suitable method for parallel computing. In addition, the framework of the GA method is a series of probabilistic methods that basically include each process of "proliferation / crossover / mutation / selection", and various methods depend on "how to modify what". It is possible.
In addition, it is sufficiently possible to perform processing in combination with other calculation methods, and various approaches are currently being attempted.

In the present invention, such a competitive stochastic search method is applied. Here, the competitive probabilistic search is
Run multiple searches in parallel, compare and correct stochastically,
This is a method for improving the probability of searching to reach the global optimum solution. GA is a typical example of the method, but other methods include search by Monte Carlo simulation.

(3) Parallel computing system There are two types of parallel computers: shared memory type and distributed memory type. In the shared memory type, all the processors (PE) can access the common memory, which is convenient for calculation processing, but the hardware becomes complicated. On the other hand, in the distributed memory type, a memory is associated with each processor (local memory). Therefore, the hardware is simple and the degree of parallelism can be large, but it is difficult to bring out the calculation efficiency due to the data consistency problem and the communication problem.
The present invention provides a method for efficiently learning HMM parameters using such a distributed memory type computer.

The distributed memory type computer is shown in FIG.
There is a hypercube type topology as shown in FIG. 2 and a mesh type topology as shown in FIG. In the hypercube type, communication between PEs can be efficiently executed, but since it is mechanically complicated, there are many mesh type parallel computers at present.

[0024]

Since Baum's algorithm is not guaranteed to converge to a global optimum solution, the setting of initial values greatly affects the accuracy of estimation parameters. However, the selection of the initial value is heuristically determined, and a systematic local solution avoidance method has not been proposed yet.

Further, the probabilistic search method represented by the SA method / GA method has structurally many parallel processing parts, and several parallel processing methods have been proposed. However, in the distributed memory type computer, since the communication load is large, a calculation method that suppresses communication as much as possible is required.

It is an object of the present invention to provide a method suitable for a parallel memory-in particular, a distributed memory type computer, by incorporating an element of the stochastic search method into Baum's algorithm which is a typical method for estimating HMM parameters. It is in.

[0027]

In order to solve the above-mentioned problems, the present invention uses the following means. First, a plurality of parameter estimation sequences based on Baum's algorithm are prepared, and each estimation sequence is executed in parallel by each PE.
A probabilistic method such as GA is applied to the plurality of estimated sequences to correct the estimated sequences and finally converge to the global optimum. Secondly, in order to reduce the load of data communication between PEs, the selection target is limited to adjacent PEs and processing is executed by a pseudo stochastic process. Further, in order to avoid the competition of the communication between PEs, a means for extracting a combination of selections that do not cause communication competition in advance and selecting the pattern stochastically is used. Such means are not strictly probabilistic, but
It is sufficient as a means to reach a global solution. However, there is a risk that the selected parties will be overemphasized only by the above means. Therefore, as a third means, the processing PEs are exchanged when comparing and correcting the estimated sequences. This is on average equivalent to stochastic processing after a considerable number of iterations.

According to the present invention, an optimal solution close to a stochastic search method such as the SA method or the GA method can be obtained with the convenience of the conventional Baum algorithm. Further, it can be efficiently realized by a distributed memory type parallel computer.

[0029]

BEST MODE FOR CARRYING OUT THE INVENTION A first embodiment of the present invention will be described. In this embodiment, an example will be described on the assumption that a parallel processing type microcomputer having two calculation units is used. Here, as the simplest example for realizing the present invention, processing with two estimation sequences will be considered.

FIG. 4 is a flow chart showing the outline of the processing of the first embodiment. In this embodiment, Phase I is calculated by each calculation unit independently, specifically, steps 411 to 411.
413 and steps 421 to 423 and phase II in which each calculation unit cooperates to calculate, specifically steps 402 and 40
Repeat 4 and 4 alternately to obtain the optimum solution. Hereinafter, description will be made separately for phase I and phase II.

Processing is started from START 401. As a result, each calculation unit simultaneously starts calculation. And
Enter Phase I. Focusing on the estimated sequence 1 which the first calculation unit is in charge of, the necessary initialization is performed in step 411.

In this embodiment, a system using two PEs will be considered. Both of them start estimation from different initial estimation values. The procedure 411 is the initial setting of the estimated sequence 1, and the procedure 421 is the initial setting of the estimated sequence 2. Step 4
At 12, the estimated value is updated based on the update of the estimated value in the conventional Baum method. Specifically, for each observation series (feature vector sequence), the calculation of the Forward variable and the Backward variable by the equation 1 and the calculation of the state transition probability and the output probability by the equation 2 are repeated.

The same processing as in the search sequence 1 is also performed in the estimated sequence 2 in charge of the second calculation unit, that is, in steps 421, 422 and 423. Here, procedure 421 corresponds to the processing of procedure 411 of estimated sequence 1. Similarly, procedure 422 corresponds to procedure 412 and procedure 423 corresponds to procedure 413.

Next, the phase II, that is, the processing that the calculation units cooperate with each other will be described. This phase II is basically a procedure 402 for reconstructing the estimated sequence.
And a procedure 404 for determining convergence. This step 4
Correct the inferior sequence parameters with 04. This judgment can be evaluated by a method such as comparing the likelihoods of the two.

FIG. 5 illustrates a configuration example of procedure 402. In the first embodiment, as the simplest example, the estimated sequence has the minimum configuration of two systems, so the estimated sequences to be compared are fixed to the sequences 1 and 2. Therefore, step 41
If the execution shifts to the procedure 402 from 3 and 423, that is, if the execution of the phase I is completed for both the series 1 and 2, the superiority or inferiority of the estimated sequences is compared by the procedure 501.

The superiority or inferiority of the estimated sequence can be executed by, for example, setting an evaluation function for superiority or inferiority comparison in consideration of the likelihood, etc. Here, for the necessary values, the values calculated in step 412 and step 422 are set in the table 511, respectively.
It may be stored in 512 and 512 and used.

Based on this result, procedure 502 modifies the estimated sequence evaluated as recessive. There are several possible methods for this modification, but the simplest method is to randomly select the search points. Further, the property of series 1 and the property of series 2 are stochastically modified by a GA-like method. For example, a method of appropriately mating binary search points to generate a new search point is also possible. Here, the table 511 or 512 is updated with the reconstructed search points.

Second Embodiment of the present invention using a plurality of PEs
Will be described. FIG. 6 shows an outline of the processing flow of the second embodiment. As in the first embodiment, S is selected from step 401.
TART. It is assumed that the nth PE is in charge of each of the n estimated sequences. Procedures 611 to 613 of estimation sequence 1
The same processing as in the first embodiment may be performed. Also in the search system n, that is, in steps 691 to 693, the same processing as the estimation sequence 1 shown in the first embodiment is performed. Where step 69
1 corresponds to the process of step 611 of the estimated sequence 1. Similarly, procedure 692 is procedure 612 and procedure 693 is procedure 61.
Corresponds to 3, respectively.

Next, the phase II, that is, the processing in which the respective calculation units cooperate with each other will be described. This phase II is basically the same as the first embodiment, and the procedure 601 for reconstructing the estimated sequence and the procedure 60 for determining the global optimum solution.
It consists of two. However, because of the n-sequence, several different reconstruction methods are possible.

FIG. 7a shows a processing flow of procedure 601 for reconstructing the estimated sequence. In step 701, the n estimated sequences are classified into a pair of two persons (hereinafter referred to as a comparison pair),
Therefore, comparison of superiority and inferiority is performed. It is theoretically possible to randomly select this comparison pair. And such an implementation method is the simplest. Here, an efficient comparison pair selection method will be described on the premise of a hypercube type computer as shown in FIG. 3a.

Attention is paid to a node (PE) in FIG. 3a. This is number 0 (binary expression 0000). In the hypercube, a node with a number in which any one bit is changed in binary representation-that is, the first node (0001 in binary representation)
-No. 2 (0010 in binary representation) -No. 4 (010 in binary representation)
It is adjacent to 0) and 8 (1000 in binary representation). This is shown in FIG. 7b.

FIG. 8 shows this procedure 701 in more detail. In a distributed parallel computer, PE
There is a need to efficiently handle data communication between them. In the data transfer between PEs, it is necessary to reduce the number of transfers and suppress the competition of communication paths. In the present invention, the number of data transfers is reduced because of the structure in which each PE processes independently in Phase I. Therefore, a method of suppressing the competition of communication paths will be described below.

The first method considers a method of managing a combination of comparison pairs in which communication paths do not compete as a table. There are many combinations, but it is better to prepare as many tables as possible. At run time, Figure 8a
In step 821, the index of this table is randomly selected. This allows the comparison pair to be determined according to the selected table.

FIG. 9 shows an example of this. FIG. 9a shows the combination of comparison pairs in pattern i. In this example,
No communication contention occurs. Each PE is
The partner PEs to be paired when the index is given may be set in the table. This is done in step 822 of Figure 8a. For example, PE1 has a table 731. Here, PE2 is selected for i. Similarly, the PE 16 has a table 739. Here, PE12 is selected for i. Then, if it is determined in advance that the PE having the smallest PE number in the comparison pair executes the reconstruction 601 of the estimated sequence, each PE can execute the process without communication conflict.

The second method aligns all communication directions.
In the hypercube, this can be realized by inverting the bit at the same position when the PE number is expressed in binary notation. In the case of the 16PE system, it is only necessary to randomly select an arbitrary position of 1 to 4 and invert the corresponding bit accordingly. This corresponds to the procedure 821 in FIG. For example, if the second bit is inverted, (0,2) (1,3) (4,6) (5,7) (8,
10) (9, 11) (12, 14) (13, 15) pairs will be obtained.

In step 822, superiority / inferiority comparison of estimated sequences is performed for each comparison pair obtained in step 821. This process
Basically, the same procedure as step 702 may be performed. However, processing is performed for all comparison pairs. Each set of comparisons can be performed in parallel. It is theoretically possible that the calculation of each set is performed in parallel by the corresponding two PEs. However, since it is not a heavy calculation, an example in which one PE is executed will be shown here. That is, in the above example, PE2 → PE
0, PE3 → PE1, PE6 → PE4, PE7 → PE
5, PE10 → PE8, PE11 → PE9, PE14 →
Data transfer from PE12, PE15 to PE13, P
E0, PE1, PE4, PE5, PE8, PE9, PE
12, PE13 may be compared for superiority or inferiority of the estimated sequences.

In step 703, the estimation series evaluated to be inferior in the above evaluation for each comparison pair is performed. This is also for each PE shown above-ie PE0, PE1, PE4, P
E5, PE8, PE9, PE12, and PE13 can be executed in parallel.

In this method, pseudo stochastic processing is performed. That is, instead of generating a comparison pair completely randomly, an efficient set pattern is determined in advance,
The method of selecting the pattern stochastically is used. Of course, completely random combinations are also possible.

[0049]

The present invention is effective in avoiding local solutions in Baum's algorithm. Hereinafter, the effect of the present invention will be described with respect to three points of convergence speed, parallel efficiency, and property of solution.

(1) Convergence speed The reason why it takes time in the stochastic search method such as the SA method or the GA method is that the stochastic processing for avoiding the local optimization is, so to speak, a "backtracking" processing. It has the effect of delaying the convergence. In the first place, the meaning of introducing a stochastic process is avoidance of local optimization, so it is sufficient to correct the fallen local solution, and there is no reason to introduce a stochastic process in all procedures.

In the present invention, a plurality of parameter estimation sequences are prepared, and each sequence is basically calculated independently (Phase I), and only as a stochastic process for eliminating the local solutions that fall. It employs a GA-like competitive pseudo-probabilistic search method in which PE-to-PE communication occurs (Phase II). Therefore, the weight of the stochastic processing is smaller than that of SA or GA, and the decrease of the convergence speed is suppressed to a small extent as a whole.

(2) Concerning parallel efficiency In phase I, parameter estimation for each PE is
This is done by Baum's algorithm. In this process, P
Since no inter-E communication occurs, efficient execution can be expected even in a loosely coupled system using a plurality of microcomputers.

On the other hand, in phase II, data communication occurs between the PEs, but since these are not repetitive processes as in phase I, the proportion of them in phase II as a whole is small.

(3) Property of Solution Since the inferior estimated sequence among the estimated sequences of a plurality of parameters that exist is revised or abolished and updated by a certain stochastic process, it is possible to avoid local optimization and obtain a good quality solution. Is obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing an example of a voice recognition device.

FIG. 2 is an explanatory diagram showing a configuration example of a left-to-right type HMM.

FIG. 3 is a model diagram showing an example of a connection form of a device using a plurality of microcomputers.

FIG. 4 is a processing flowchart of an HMM parameter estimation method according to the first embodiment.

FIG. 5 is a detailed process flow diagram of step 402 in the first embodiment.

FIG. 6 is a processing flowchart of an HMM parameter estimation method according to the second embodiment.

FIG. 7 is a detailed diagram of a procedure 601 according to the second embodiment.

FIG. 8 is a detailed diagram of a procedure 601 according to the second embodiment.

9 is a combination of patterns and P in Example 2. FIG.
Explanatory drawing which shows the structure of E table.

[Explanation of symbols]

401 ... START, 402 ... Parameter re-evaluation, 4
03 ... END, 404 ... Convergence judgment, 411 ... Estimated sequence 1
Initial setting, 412 ... Update of estimated value of sequence 1, 413 ...
Sequence 1 iteration count determination, 421 ... Estimated sequence 2 initialization, 422 ... Sequence 2 estimate update, 423 ... Sequence 2 iteration count determination, 501 ... Parameter evaluation, 502 ...
Correction of inferiority estimation series, 511 ... Data of past estimated value of series 1, 512 ... Data of past estimated value of series 2, 61
1 ... Initialization of sequence 1, 612 ... Update of estimated value of sequence 1, 613 ... Determination of local solution of sequence 1, 691 ... Initialization of sequence n, 692 ... Update of estimated value of sequence n, 693 ... Sequence n Determination of local solution of 601 ... Reconstruction of estimated sequence, 60
2 ... Judgment of global solution, 701 ... Selection of series pair to be compared,
702 ... Comparison of superiority / inferiority of estimated sequences, 703 ... Modification of inferiority estimated sequences, 711 ... Evaluation data of series 1, 719 ... Evaluation data of series n, 731 ... PE table of series 1, 739 ...
PE table of series n, 821 ... Index selection,
822 ... Selection of estimation sequence.

Claims (5)

[Claims] 1. Baum-Welch parameter estimation with different initial values in a voice recognition device using a plurality of microcomputers,
This is a calculation method that each microcomputer (hereinafter PE) executes independently and in parallel, and compares the result of the parameter estimation sequence executed by each PE with the first procedure in which each PE independently estimates parameters. Then, the second step of modifying the estimation sequence executed in each PE is repeated to obtain an optimal solution, and a parameter estimation method for a Hidden Markov Model (HMM). 2. A first procedure for selecting a sequence to be compared, a second procedure for comparing superiority or inferiority of an estimated sequence, and a third procedure for correcting an estimated sequence determined to be inferior by the second procedure. A method for modifying an estimated sequence that modifies the estimated value that is being calculated or the parameter that affects convergence. 3. An estimation that is preliminarily provided with a plurality of PE combination tables in which communication competition between PEs does not occur, selects an arbitrary combination from the table when selecting an estimation sequence, and is processed by a corresponding PE. A method for selecting an estimated sequence that stochastically selects an estimated sequence from among the sequences. 4. A fourth procedure for transferring data of an estimated sequence executed in each PE to a partner PE after the third procedure of correcting the estimated sequence determined to be inferior in the second procedure of claim 2. The method of estimating a parameter, which comprises the procedure of 1. and takes over the calculation to be processed by the PE of the partner compared with each PE. 5. The method according to claim 2 or 3, wherein a system in which PEs are combined in a hypercube type is used to randomly select a specific position when a PE number is expressed in binary, and each PE. A method of selecting an estimated sequence that stochastically selects an estimated sequence from a plurality of estimated sequences by a second procedure of inverting a specific position selected in the first procedure in the PE number.