KR100764247B1 - Apparatus and Method for speech recognition with two-step search - Google Patents

Abstract

Disclosed are a speech recognition apparatus using a two-stage search and a method thereof.

The present invention provides a fast search unit for generating a plurality of candidate words by performing a Viterbi search using a Gaussian distribution having a predetermined number or less included in a pool, and selecting candidate words in an order of high reliability among the candidate words. And an N-best candidate generator to extract and a precision search unit to output the recognized word by performing a Viterbi search using a predetermined number or more Gaussian distributions of the extracted candidate words.

According to the present invention, the speed of speech recognition can be improved without degrading the speech recognition rate, and the performance of the entire system can be improved.

Description

Apparatus and Method for speech recognition with two-step search}

1 is a block diagram of the present invention.

FIG. 2 is a detailed block diagram of FIG. 1.

3 is a flowchart of the present invention.

4 is a detailed flowchart of the fast searching process of FIG. 3.

FIG. 5 is a detailed flowchart of an N-best candidate word extraction process of FIG. 3.

6 is a detailed flowchart of the precise search process of FIG.

<Description of the symbols for the main parts of the drawings>

100: fast search unit 110: N-best candidate generation unit

130: precision search unit

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to speech recognition, and more particularly, to a speech recognition apparatus and a method using two-stage search in an embedded platform.

Recently, voice recognizer implementation in embedded has become a big issue. This is because the development of information and communication technology has increased the use of personal mobile devices, the development of products such as home appliances, vehicles, toys, etc. applied with advanced technologies, and the interest in the introduction of voice interfaces has increased.

However, in the case of an embedded system, resources are extremely limited than the development environment of a general PC, and the computation speed is slow, so it is not easy to implement a speech recognizer based on a large vocabulary or a continuous word. In particular, the conventional speech recognition methods detect a single word having an optimal probability value through one sophisticated Viterbi search, and thus have a high recognition performance and a fast execution speed to perform a large vocabulary recognition in a resource-limited embedded environment. Difficult to obtain

In addition, the conventional continuous density hidden markov model (HMM) is not suitable for designing an acoustic model for an embedded system since it cannot allocate a large amount of memory and is not easy to minimize the degradation of recognition performance.

Therefore, the conventional speech recognition method has a problem that it is not possible to perform high-speed speech recognition while maintaining high recognition performance.

Accordingly, the first technical problem to be achieved by the present invention is to provide a speech recognition apparatus using a two-stage search that can perform speech recognition at high speed while maintaining high recognition performance.

The second technical problem to be achieved by the present invention is to provide a speech recognition method using a two-stage search applied to the speech recognition device.

In order to achieve the first technical problem, the present invention provides a fast search unit for generating a plurality of candidate words by performing a Viterbi search using a Gaussian distribution having a predetermined number or less included in a pool, among the candidate words. N-best candidate generator for extracting candidate words in order of high reliability and Viterbi search using a predetermined number or more Gaussian distributions of the extracted candidate words and outputting the recognized words It includes a search unit.

In order to achieve the second technical problem, the present invention performs a Viterbi search using a Gaussian distribution having a predetermined number or less included in a pool, for generating a plurality of candidate words, and having a reliability among the candidate words. Extracting candidate words in ascending order and performing a Viterbi search using a predetermined number or more of Gaussian distributions included in the pool to output the recognized words.

Hereinafter, with reference to the drawings will be described a preferred embodiment of the present invention.

1 is a block diagram of the present invention.

The fast search unit 100 generates a plurality of candidate words by performing a Viterbi search using a Gaussian distribution having a predetermined number or less included in a pool on the input voice. Where pool is a set of possible Gaussian distributions.

The N-best candidate generator 110 extracts candidate words in the order of high reliability among the candidate words generated by the fast search unit 100.

The precision search unit 130 performs a Viterbi search using a predetermined number or more Gaussian distributions of the candidate words extracted by the N-best candidate generator 110 and outputs the recognized words.

FIG. 2 is a detailed block diagram of FIG. 1.

The fast search unit 200 generates a plurality of candidate words by performing a Viterbi search using a Gaussian distribution having a predetermined number or less included in a pool on the input voice. Where pool is a set of possible Gaussian distributions.

The fast search unit 200 includes a Mahalanobis distance calculator 201, a Gaussian selector 202, and a log sum calculator 203.

The Mahalanobis distance calculator 201 calculates Mahalanobis distance values for all Gaussian distributions included in the pool. Preferably, the Mahalanobis distance calculator 201 may calculate Mahalanobis distance values between the feature vectors extracted for each frame of the voice and all Gaussian distributions included in the pool.

The Gaussian selector 202 selects Gaussian distributions having a predetermined number or less from the Gaussian distributions included in the pool in order of Mahalanobis distance values in ascending order. In this case, the predetermined number is a threshold value for performing fast recognition without lowering the speech recognition rate.

The log sum operator 203 performs a log sum operation using the Gaussian distributions selected by the Gaussian selector 202.

The N-best candidate generator 210 extracts candidate words in the order of high reliability among candidate words generated by the fast search unit 200.

The N-best candidate generator 210 includes an NLLR verification unit 211 and a search space generator 212.

The NLLR verification unit 211 calculates reliability of candidate words and selects candidate words having a reliability greater than or equal to a threshold among candidate words. At this time, the threshold value is predetermined by a person skilled in the art to ensure that the result of speech recognition is a reliable level. Preferably, the NLLR verification unit 211 may calculate candidate normalized log likelihood ratios of the candidate words to select candidate words whose normalized log likelihood ratio is equal to or greater than a threshold. At this time, the selected candidate words are defined as N-best candidate words.

The search space generator 212 outputs candidate words selected by the NLLR verification unit 211 to the precision search unit 230.

The Gaussian cache storage unit 220 stores the calculated Mahalanobis distance values. The Gaussian cache storage unit 220 may include a volatile memory device for storing Mahalanobis distance values.

The precision search unit 230 performs a Viterbi search using a predetermined number or more of Gaussian distributions of the candidate words extracted by the N-best candidate generator 210 and outputs the recognized words. Preferably, the precision search unit 230 may output a recognized word by performing a Viterbi search after speech is completed.

The precision search unit 230 includes a Gaussian cache application unit 231, a log sum calculator 232, and a 1-best search unit 233.

The Gaussian cache application unit 231 reads Mahalanobis distance values from the Gaussian cache storage unit 220 and outputs the Mahalanobis distance values to the log sum calculator 232.

The log sum operator 232 performs a log sum operation during the output probability calculation process of the Viterbi search. In addition, the log sum calculator 232 performs a log sum operation using a predetermined number or more of Gaussian distributions.

The 1-best search unit 233 uses the result of the log sum operation by the log sum operator 232 and the Mahalanobis distance values by the Gaussian cache application unit 231 to have 1 having the highest likelihood among the candidate words. Words are extracted and the extracted words are output as recognized words.

Preferably, the fast search unit 200 and the precision search unit 230 may apply the same acoustic model.

3 is a flowchart of the present invention.

First, a voice is input (step 300). Preferably, the process 300 may include extracting a feature vector for each frame of speech.

Next, a plurality of candidate words are generated by performing a Viterbi search using a Gaussian distribution having a predetermined number or less included in the pool on the input voice (step 310). Where pool is a set of possible Gaussian distributions.

When candidate words are generated, candidate words are extracted in order of high reliability among candidate words (step 320).

Finally, a Viterbi search is performed on the extracted candidate words using a predetermined number of Gaussian distributions included in the pool (step 330).

Preferably, the process of generating candidate words (step 310) and outputting the recognized words (step 330) may apply the same acoustic model.

4 is a detailed flowchart of the fast discovery process 310 of FIG. 3.

In the fast search, the search is performed using a small amount of acoustic modeling to ensure the reliable speed of complex large vocabulary search.

First, Mahalanobis distance values are calculated for all Gaussian distributions included in the pool (step 411). This process (411) may be a process of calculating the Mahalanobis distance values between the feature vector extracted for each frame of the voice and all Gaussian distributions.

Next, among the Gaussian distributions included in the pool, the Mahalanobis distance values are selected in order of increasing number of Gaussian distributions having a predetermined number or less (step 412).

Finally, a log sum operation is performed using the selected Gaussian distributions (step 413). Preferably, the process 413 may include generating a candidate word according to the output probability when the output probability is calculated according to the log sum operation.

FIG. 5 is a detailed flowchart of the N-best candidate word extraction process 320 of FIG. 3.

This process (step 320) generates N-best candidate words for the rescan process to prevent the recognition rate from being lowered.

First, the reliability of candidate words is calculated (step 521). This process (521) may be a process of calculating the normalized log likelihood ratio (NLLR) of the candidate words. For candidate N-best generation, we need to select the candidate candidates that are reliable. The Normalized Log Likelihood Ratio (NLLR) method can be used to efficiently calculate a confidence measure. That is, NLLR can be defined as a measure of reliability.

The normalized log likelihood ratio may be calculated by Equation 1 below.

Where NLLRv is the normalized log likelihood ratio of the Vth candidate word, LKv is the likelihood of the Vth candidate word, LLv is the log likelihood of the vth candidate word, LKmax is the maximum likelihood of all candidate words, and LLmax is all The maximum log likelihood of candidate words.

Next, it is determined whether the calculated reliability is greater than or equal to the threshold (step 522). At this time, the threshold value is predetermined by a person skilled in the art to ensure that the result of speech recognition is a reliable level. The formula for determining a candidate candidate is as follows.

NLLRv> Th

The threshold Th is predetermined by experiment. At this time, the v- th word whose NLLR is larger than the threshold Th is selected as candidate words for the fine search.

Finally, candidate words having a reliability greater than or equal to a threshold among candidate words are selected (step 523). At this time, the selected candidate words are defined as N-best candidate words.

6 is a detailed flowchart of the precise search process 330 of FIG.

First, a log sum operation is performed using more than a predetermined number of Gaussian distributions (step 631). The log sum operation process (631) is a process required for calculating the output probability in the Viterbi search.

Next, one word having the highest likelihood among the candidate words is extracted using the result of the log sum operation and the Mahalanobis distance values calculated in the fast search process 310, and the extracted word is recognized. (Step 632). In this case, the recognized word is defined as a 1-best word. In operation 632, after the speech is completed, the Viterbi search may be performed to output the recognized word.

For the search process required for the voice recognition of the present invention, it is necessary to select an appropriate acoustic model to be used in the embedded platform.

The present invention introduces a semi-continuous HMM (SCHMM) model technique, and can be implemented through a shared-mixture modeling method. In addition, by applying the context-dependent model (triphone) to reflect the articulation effects according to the context of the front and back, to prevent the lack of data when training the context-dependent model (triphone) while solving the problem of lack of data that does not appear in the learning To do this, state treeing based on decision trees can be applied. The acoustic model thus constructed has a size of about 60% of the size of the CHMM model, and the recognition performance is reduced to less than 1%.

In the present invention, the search method uses the Viterbi search method. In Viterbi search, most of the computation time is spent calculating output probability. The output probability calculation can be divided into the Mahalanobis distance calculation (Euclidean distance calculation considering dispersion) and the log-add operation of this value within each Gaussian. Since the state-probability calculation process based on the shared-mixture shares the Gaussians existing in the pool, which is a set of possible Gaussians, and calculates by adding up the weights of each distribution, the log sum operation is the most practical. It requires a lot of computation time.

According to an embodiment of the present invention, in order to shorten the decoding time, the Gaussians participating in the log sum may be appropriately selected and calculated. That is, if a feature vector is input every frame, the Mahalanobis distance calculation is performed on all Gaussians present in the pool, and the values are sorted in ascending order to select the threshold values from the top. Only the distributions are involved in the log sum operation, which can reduce the amount of computation. The threshold value is selected to suit the characteristics of the fast search and the fine search.

In addition, since the Mahalanobis calculation process calculates both the fast search and the fine search by the same feature vector, the value calculated in the fast search can be applied to the precision search as it is. Therefore, rescanning time can be shortened significantly.

Equation 3 is a formula for Gaussian selection based on a shared-mixture.

Where bs is the selected Gaussian, i is the index, Ws is the weight, Xt is the observed value of the current voice, μi is the mean value, and Σi is the variance value.

According to another embodiment of the present invention, a higher Gaussian distribution through Gaussian distance value calculation may be selected for a fast search. For example, in the fast search, the output probability value may be calculated by selecting only the top four distributions.

According to another embodiment of the present invention, the result of arranging the Gaussian Mahalanobis distance calculation values according to the size may be stored in a cache, which is a temporary storage space, every frame, and applied to the precise search.

In the precise search, the re-search is performed with the N-best candidate words obtained from the fast search. For example, since there are a maximum of 20 N-best candidate words, a more sophisticated search process can be performed with a small search space. Viterbi search can also be performed in fine search. For example, to obtain an output probability, the number of Gaussian distributions can be calculated by selecting the top 32 more than the fast search.

According to another embodiment of the present invention, it is important to minimize the user waiting time because the fast search is a re-search process performed at a non-real time on the result after the voice utterance is finished. If the length of the speaker's voice is 1x real-time (xRT), the maximum waiting time that the user can wait while thinking that it is real time is 0.4xRT after the speech is finished.

According to another embodiment of the present invention, since the implemented system uses the values obtained from the same acoustic model, the Gaussian distance calculated from the fast search can be used as it is in the precise search. Accordingly, the rescanning time can be significantly shortened. For example, since the precision search is completed within 0.4xRT, the recognition result of the large vocabulary system can be obtained within the real time felt by the user.

Table 1 is a table showing the speed and recognition rate of speech recognition results according to the conventional Viterbi search and the search of the second stage of the present invention.

Search process Gaussian Count Recognition rate (%) Average execution time (μsec) Fast navigation Precise navigation Step 2 navigation 4 32 93.31 1,805,876 8 32 93.64 1,966,436 16 32 93.72 2,242,341 32 32 93.76 2,458,194 Conventional Viterbi Search 32 93.74 2,418,897

Table 1 shows the experimental results of increasing the number of Gaussians selected from the high-speed search from 4 to 32 compared to the conventional Viterbi search method, and selecting 32 from the detailed search. For each step, we recorded the average execution time and examined the speed of recognition time.

The results show that the two-stage search is slightly lower than the conventional Viterbi search, but the average execution time is 0.7 times faster. It can be seen that the overall recognition speed has been greatly improved without a significant drop in the recognition performance.

Preferably, a program for executing the voice recognition method using the two-stage search of the present invention on a computer may be provided recorded on a computer-readable recording medium.

The invention can be implemented via software. When implemented in software, the constituent means of the present invention are code segments that perform the necessary work. The program or code segments may be stored on a processor readable medium or transmitted by a computer data signal coupled with a carrier on a transmission medium or network.

Although the present invention has been described with reference to one embodiment shown in the drawings, this is merely exemplary and will be understood by those of ordinary skill in the art that various modifications and variations can be made therefrom. However, such modifications should be considered to be within the technical protection scope of the present invention. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

As described above, according to the present invention, when the speech recognition of the large vocabulary is implemented in a limited computing environment such as an embedded platform, the candidate word generated through the fast searching process is recognized through a precise searching process, thereby lowering the speech recognition rate. The speed of speech recognition can be improved without improving the performance of the entire system.

Claims (21)

A fast search unit configured to generate a plurality of candidate words by performing a Viterbi search using a specific number of Gaussian distributions (hereinafter referred to as "N1") included in the pool with respect to the input voice; An N-best candidate generator configured to extract candidate words in an order of high reliability among the candidate words; And The precision search unit outputs a recognized word by performing a Viterbi search using a specific number of Gaussian distributions (hereinafter referred to as "N2") included in the pool with respect to the extracted candidate words. Speech recognition device using a two-step search comprising. The method of claim 1, N1 is less than or equal to the N2 voice recognition device using a two-step search. The method of claim 2, The fast search unit A Mahalanobis distance calculator for calculating Mahalanobis distance values for all Gaussian distributions in the pool; A Gaussian selector which selects the N1 Gaussian distributions in order of the Mahalanobis distance values among the Gaussian distributions included in the pool; And And a log sum calculator configured to perform a log sum operation using the selected Gaussian distributions. The method of claim 3, wherein Further comprising a Gaussian cache storage for storing the calculated Mahalanobis distance values, The precision search unit And a Gaussian cache application unit for reading Mahalanobis distance values from the Gaussian cache storage unit. The method of claim 3, wherein The Mahalanobis distance calculation unit And a Mahalanobis distance value between all the Gaussian distributions and the feature vector extracted for each frame of the voice. The method of claim 3, wherein The precision search unit A log sum operator configured to perform a log sum operation using the N2 Gaussian distributions; And A 1-Best search unit for extracting one word having the highest likelihood among the candidate words by using the log sum operation and the Mahalanobis distance values, and outputting the extracted word as the recognized word Speech recognition device using a two-step search, characterized in that it comprises a. The method of claim 2, The N-best candidate generation unit An NLLR verification unit for calculating candidates of the candidate words and selecting candidate words of the candidate words whose threshold is greater than or equal to a threshold value; And And a search space generator for outputting the selected candidate words to the precision search unit. The method of claim 7, wherein The NLLR verification unit And calculating candidate normalized log likelihood ratios of the candidate words to select candidate words whose normalized log likelihood ratio is greater than or equal to a threshold. The method of claim 2, The fast search unit and the precision search unit Speech recognition device using a two-stage search, characterized in that the same acoustic model is applied. The method of claim 2, The precision search unit After the speech is completed, the speech recognition apparatus using the two-stage search, which performs a Viterbi search to output the recognized word. Generating a plurality of candidate words by performing a Viterbi search using a specific number of Gaussian distributions (hereinafter referred to as "N1") included in the pool with respect to the input voice; Extracting candidate words in an order of high reliability among the candidate words; And And performing a Viterbi search using a specific number of Gaussian distributions (hereinafter, referred to as "N2") included in the pool for the extracted candidate words, and outputting a recognized word. Speech recognition method using two-stage search. The method of claim 11, The N1 is less than or equal to the N2 voice recognition method using a two-stage search. The method of claim 12, Generating the candidate words Calculating Mahalanobis distance values for all Gaussian distributions included in the pool; Selecting the N1 Gaussian distributions in ascending order of the Mahalanobis distance values among the Gaussian distributions included in the pool; And And performing a log sum operation using the selected Gaussian distributions. The method of claim 13, Computing the Mahalanobis distance values And calculating Mahalanobis distance values between the feature vector extracted for each frame of the voice and all Gaussian distributions. The method of claim 13, The step of outputting the recognized word is Performing a log sum operation using the N2 Gaussian distributions; And Extracting one word having the highest likelihood among the candidate words by using the log sum operation and the Mahalanobis distance values, and outputting the extracted word as the recognized word Speech recognition method using two-step search. The method of claim 12, Extracting the candidate words Calculating the reliability of the candidate words; And And selecting candidate words of the candidate words having a reliability greater than or equal to a threshold value. The method of claim 16, Computing the reliability And a step for calculating a normalized log likelihood ratio of the candidate words. The method of claim 17, Computing the normalized log likelihood ratio T is the number of frames of speech to be recognized, NLLRv is the normalized log likelihood ratio of the Vth candidate word, LKv is the likelihood of the Vth candidate word, LLv is the log likelihood of the vth candidate word, and LKmax is all candidate words. Is the maximum likelihood of, and LLmax is the maximum log likelihood of all candidate words,

Computing the normalized log likelihood ratio using the equation of the speech recognition method using a two-stage search. The method of claim 12, Generating the candidate words and outputting the recognized words Speech recognition method using two-stage search characterized by applying the same acoustic model. The method of claim 12, The step of outputting the recognized word is And after the speech is completed, performing the Viterbi search and outputting a recognized word. A computer-readable recording medium having recorded thereon a program for executing the invention according to any one of claims 11 to 20.

Images