KR20070109314A - Method of selecting the training data based on non-uniform sampling for the speech recognition vector quantization - Google Patents

Abstract

A method for selecting the training data based on non-uniform sampling for the speech recognition vector quantization is provided to select the training based on non-uniform sampling considering frequency of each phoneme. When sample speech data are received, phonetic information for each phoneme is obtained by performing forced alignment on the speech data(201). A phoneme frequency list is generated based on the obtained phonetic information(202). The frequency rate of each phoneme is statistically computed according to corresponding language, based on the generated phoneme frequency list(203). Training data are deduced considering the computed statistic value, wherein the training data minimizes a difference between the total frequency for each phoneme and the frequency rate of each phoneme(204).

Description

Method of selecting the training data based on non-uniform sampling for the speech recognition vector quantization}

1 is a configuration diagram of an embodiment of a distributed speech recognition system to which the present invention is applied.

2 is a flowchart illustrating a method of selecting training data based on non-uniform samples in quantizing a speech feature vector according to the present invention.

Figure 3 illustrates one embodiment of a speech recognition process used in the present invention.

4 is a flow chart of an embodiment of a learning process for generating an acoustic model in FIG. 3.

5 is a graph showing speech recognition performance using a speech feature vector quantizer to which the present invention is applied to Korean.

6 is a graph showing speech recognition performance using a speech feature vector quantizer to which the present invention is applied to English.

* Explanation of symbols on the main parts of the drawing

11: A / D Converter 12: Speech Feature Vector Extractor

13: Voice Feature Vector Quantizer

21: speech feature vector dequantizer 22: speech recognizer

The present invention relates to a method for selecting the training data in speech feature vector quantization for speech recognition, and more particularly, to selecting the training data used to train the speech feature vector quantizer. For example, the present invention relates to a method of selecting training data based on a nonuniform sample in quantizing a speech feature vector for speech recognition, which selects training data based on a nonuniform sample in consideration of the occurrence frequency of each phoneme.

Distributed Speech Recognition (DSR) is a technology that enables the speech recognition function to be implemented in low-end terminals such as mobile phones. In low-end terminals, voice signals are recognized, and in high-end servers, voices received from low-end terminals It consists of a dual processing system that performs speech recognition based on features.

In general, in the speech recognition field as described above, MFCC (Mel-Frequency Cepstrum Coefficient) is commonly used. The MFCC represents a form of the frequency spectrum expressed in melscale as a sinusoidal component, and refers to a speech feature vector (aka speech feature parameter) representing a speech received from a user. In order for the MFCC to be transmitted from the terminal to the server through a communication network, a quantization process must be performed. The speech recognition process will be described briefly with reference to the aforementioned distributed speech recognition system.

The terminal extracts the speech feature vector from the speech input from the user through the MFCC and quantizes the speech feature vector to process the speech feature vector into a form suitable for transmission to a communication network. In other words, the MFCC corresponding to the voice input from the user is selected from among the finite number of center vectors of the codebook and the vector having the closest distance is transmitted as bit data. In other words, a codebook having a center value for a cluster having similar values is required for speech recognition spoken by a user, and the codebook is used as a reference value in speech feature vector quantization.

On the other hand, the server side dequantizes the voice feature vector received from the terminal, and recognizes a word corresponding to the voice by using a Hidden Markov Model (HMM) as the voice model. Here, the HMM is a process of modeling a basic unit, for example, a phoneme, for speech recognition. The HMM combines phonemes coming into the speech recognition engine and phonemes registered in a DB in the speech recognition engine to form words and sentences.

As described above, the speech feature vector quantization process is a major factor that determines the performance of the speech recognition system. The design of the speech feature vector quantizer may be referred to as codebook generation. In particular, such a codebook should be selected as the learning data of the speech feature vector quantization as the most representative value among the extracted training data after extracting the learning data (learning MFCC vector) from the numerous speech data.

In other words, the codebook has a center vector of speech data as an index, and a large amount of training data is required to increase the speech recognition rate. As mentioned above, the result of the speech feature vector quantization means a data string expressed by a codebook index, and the speech feature vector quantization is given the closest codebook index through the distance calculation between the vectors.

However, to date, most of speech recognition technologies have been proposed only for processing algorithms of speech feature vector quantizers, and practically no algorithms regarding selection of training data for training speech feature vector quantizers have been proposed.

For example, in the conventional technology related to learning data selection in speech feature vector quantization, only arbitrary unscheduled sample learning data selected by a manager or the like is used.

In addition, as mentioned above, HMM is adopted in most speech recognition systems because of the excellent recognition performance of HMM. However, when the training data of the HMM is low in reliability, accurate acoustic model estimation is difficult on the server side. Extensive prior knowledge of phonemic context is required. For example, if the training feature is highly reliable in quantizing the speech feature vector on the terminal side, no additional speech recognition processing is required on the server side, and only the speech feature vector dequantization is performed through the existing technology.

Therefore, there is an urgent need for a technique for quantizing a speech feature vector with a smaller number of bits, and it is easy for those skilled in the art to develop a learning data selection algorithm that is used as training data in quantization of speech feature vectors. I can understand.

The present invention has been proposed to solve the above problems and to meet the above demands, and to select learning data used to train the speech feature vector quantizer, the characteristics of speech according to language, for example, the frequency of appearance of each phoneme, The purpose of the present invention is to provide a method for selecting training data based on non-uniform samples in quantizing speech feature vectors for speech recognition, which selects training data based on non-uniform samples.

In order to achieve the above object, the present invention provides a method for selecting the learning data in speech feature vector quantization for speech recognition. When sample speech data is received, the speech data is forced to be aligned. A force alignment step of obtaining pronunciation information for the phoneme; A phoneme appearance list generation step of generating a phoneme appearance list based on the obtained pronunciation information; A phoneme appearance frequency ratio calculation step of statistically calculating an appearance frequency ratio of phonemes according to a corresponding language based on the generated phoneme appearance list; And a training data derivation step of deriving training data for minimizing an error between the total frequency and the phoneme frequency ratio for each phoneme with reference to the calculated statistical value.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a configuration diagram of an embodiment of a distributed speech recognition system to which the present invention is applied.

As shown in FIG. 1, the distributed speech recognition system to which the present invention is applied is composed of a terminal (client) such as a low specification mobile phone and a high specification server.

The terminal includes an A / D converter 11, a voice feature vector extractor 12, a voice feature vector quantizer 13, and the like, and the server includes a voice feature vector dequantizer 21 and a voice recognizer 22. And the like, and descriptions of other basic components are omitted. In addition, in describing the present invention, a distributed speech recognition system will be described as an example, but the present invention provides a single speech recognition system such as an A / D converter, a speech feature vector extractor, a speech feature vector quantizer / dequantizer and a speech. It will be apparent to those skilled in the art that the present invention also applies to other speech recognition apparatus configured as a recognizer.

A feature of the present invention is that, in selecting learning data used to train the voice feature vector quantizer 13 (aka design), the characteristics of the voice according to the language (eg, Korean, English, etc.), for example, the frequency of appearance of each phoneme. We select the training data based on a non-uniform sample.

That is, the present invention uses the statistical characteristics of the language in selecting the learning data to quantize the speech feature vector represented by the MFCC. For example, the characteristics of speech appear differently depending on the language. Based on this, the learning data are selected according to the characteristics of the speech, a codebook is generated based on the learning data, and then applied to the speech feature vector quantizer 13. To increase the speech recognition rate.

Prior to describing the learning data selection algorithm of the present invention, the components of the distributed speech recognition system will be briefly described as follows.

The A / D converter 11 receives an analog voice signal from a user and converts it into digital voice data.

The speech feature vector extractor 12 extracts a speech feature vector representing the speech feature from the speech data converted by the A / D converter 11 based on the MFCC.

The speech feature vector quantizer 13 quantizes the speech feature vector extracted by the speech feature vector extractor 12 in the form of a bit stream based on the training data previously trained through the training data. In addition, the quantization process applied to the speech feature vector quantizer 13 uses a split vector quantization scheme proposed in the ETSI standard "ETSI ES 201 108".

The speech feature vector dequantizer 21 dequantizes the bit stream data received from the terminal.

The speech recognizer 22 recognizes data dequantized by the speech feature vector dequantizer 21 based on the HMM, for example, a speech feature vector, and outputs a speech recognition result such as a word or a sentence. In addition, the server side may perform a function of reconstructing and reproducing speech using the speech feature vector.

In the distributed speech recognition system as described above, the speech feature is extracted through MFCC to express a speech feature vector as a 14th order coefficient. The 14th order MFCC vector undergoes a quantization process. In other words, the MFCC vector is not represented as a coefficient of a particular order. In general, when a MFCC vector is expressed as a 14th order or 15th order coefficient, the MFCC vector is expressed as a coefficient of such order because its speech recognition performance is excellent. Reveal.

Hereinafter, a method of selecting learning data based on a non-uniform sample presented in the present invention will be described in detail with reference to FIG. 2, and a voice recognition process used in the present invention with reference to FIGS. 3 and 4 and the same. A process of generating an acoustic model will be described, and a speech recognition performance rate to which the present invention is applied in Korean and English will be described with reference to FIGS. 5 and 6.

2 is a flowchart illustrating a method for selecting training data based on a non-uniform sample in quantizing a speech feature vector according to the present invention.

First, when sample speech data used for speech feature vector quantizer learning is input, the speech data is forced alignment to obtain pronunciation information about each phoneme (201). That is, the voice data is force-aligned to obtain a utterance sentence, pronunciation transcription, phoneme type, start time for each phoneme, and end time for each phoneme. Table 1 below shows the results of force alignment of the sample voice data.

In [Table 1], the sample voice data is the voice that "and people are jumping out". It is transcribed into "ㅗ ㄷ ㅉ ㅗ", and you can see the start time and end time of each phoneme.

Then, a phoneme-specific appearance list is generated based on the pronunciation information obtained through the forced alignment process (202). [Table 2] shows the appearance list of each phoneme in the result of performing the post alignment on the voice data of the Korean DB used to verify the present invention, and the last two rows represent the total phonemes of the frequency and learning data for each phoneme. Indicates a number.

Thereafter, based on the generated appearance list for each phoneme, a frequency ratio of occurrences of phonemes according to the language (the language) is calculated statistically (203). Here, in statistically calculating the frequency ratio of phonemes, the ratio of any of the pronunciation strings is analyzed based on the pronunciation dictionary. For example, to analyze the frequency of phonemes in Korean, based on pronunciation dictionary data such as "study frequency of speech in adult daily conversation" and "quantitative linguistic studies on functional burden and phoneme chain of Korean phonemes" Perform phoneme frequency analysis for each consonant and each vowel. On the other hand, in order to analyze the frequency of phonemes in Korean, the phoneme frequency of each consonant and each vowel is analyzed based on pronunciation dictionary data such as "CMU dictionary".

[Table 3] shows the frequency ratio of phonemes according to Korean, and [Table 4] shows the frequency frequency of phonemes according to English.

In Table 2 and Table 3 (or Table 4 in English), learning data for minimizing the error between the total frequency [Table 2] and the appearance frequency ratio of each phoneme [Table 3] is derived for each phoneme (204). . This learning data derivation process ("204" process) will be described later in more detail, for example.

The training data selected through the above "201" to "204" process is determined as the training data of the speech feature vector quantizer 13. The training data thus determined is used as a parameter for codebook generation. Here, the algorithm used for generating the codebook may be any algorithm such as the K-means algorithm.

To improve the understanding of the above-described learning data selection algorithm of the present invention, for example, the English frequency ratio of any phoneme "/ a /" in the pronunciation dictionary is 2%, and the sample learning data [sample voice data] ], The number of occurrences of the phoneme "/ a /" is "40" and the total number of phonemes generated from the sample training data is "1,000", and the percentage of occurrences of the phoneme "/ a /" in the sample training data is 4%. to be. Therefore, only the "1, 3, 5, ..., 35, 37, 39th data" from the sample training data list is selected for the phoneme "/ a /" generated after the forced alignment, and the training data of the speech feature vector quantizer is selected. It can be used as a (derivation).

For example, to minimize the difference between the frequency of appearance for a particular phoneme in the phonetic dictionary and the ratio of phoneme appearance to the total number of phonemes in the sample training data [4% => 2%, half difference], Half of the sample training data ("1, 3, 5, ..., 35, 37, 39th data") of 40 sample training data is selected as training data for a speech feature vector quantizer training.

FIG. 3 is a diagram illustrating an embodiment of a speech recognition process used in the present invention, and FIG. 4 is a flowchart of an embodiment of a learning process for generating an acoustic model in FIG. 3.

3 illustrates a speech recognition process using an HMM, extracting a speech feature from a user's speech (301), and an acoustic model (303), a language model (304), and a pronunciation dictionary (305) with respect to the extracted speech feature. After searching for the pattern matching 302 through the process of recognizing the words, sentences corresponding to the voice.

In the speech feature extraction 301, the method proposed by the European Telecommunication Standards Institute (ETSI) standard proposal "ETSI ES 201 108" is used. That is, the speech feature is extracted from the speech data through the MFCC to form a speech feature vector as a 14th order coefficient, and the acoustic model 303, the language model 304 and the pronunciation dictionary 305 are formed for the speech feature vector. The pattern matching is used to find a word string having the maximum probability.

In the acoustic model 303, an HMM is used. In particular, in the present invention, a phoneme model according to a language is used as the acoustic model. The learning process for generating the phoneme model is described below with reference to FIG. 4.

First, a monophone based model, which is a phoneme independent model, is generated using a speech feature vector extracted from the selected training data according to the present invention described with reference to FIG. 2 (401).

Then, a new phoneme label file is generated by performing forced alignment based on the generated monophone-based model (402).

Meanwhile, the generated monophone based model is extended to generate a triphone based model, which is a phoneme dependent model (403).

Then, state sharing is performed in consideration of the small amount of training data for the generated triphone-based model (404).

Then, the model obtained as a result of the state sharing is increased by mixing density to generate a final acoustic model (405).

Meanwhile, in the language model 304 illustrated in FIG. 3, a statistic-based method is used. Here, the statistics-based method refers to a method of statistically estimating the probability value of possible word strings from a DB of speech spoken in a given situation. A representative of such statistical-based language models is n-gram. This en-gram method approximates the word sequence probability by the product of the previous n conditional probabilities, and uses bigram in FIG.

In the pronunciation dictionary 305, in Korean, a pronunciation dictionary provided by SiTEC's "CleanSent01" is used, and in English, "CMU dictionary V.0.6" provided by Carneige Mellon University is used.

On the other hand, in the speech DB used in FIG. 3, in the case of Korean, the "reading sentence speech DB (CleanSent01)" provided by the Voice Information Industry and Technology Center (SiTEC) is used, and in the case of English, "AURORA" constructed by ETSI. 4 DB (Wall Street Journal).

FIG. 5 is a graph showing speech recognition performance using a speech feature vector quantizer to which the present invention is applied to Korean. FIG. 6 is a graph showing speech recognition performance using a speech feature vector quantizer to which the present invention is applied to English.

In FIG. 5, the graph on the left shows the average word recognition rate of Korean in the case of training the speech feature vector quantizer using unplanned sample learning data, and the graph on the right shows the learning data according to the frequency of occurrence of each phoneme presented in the present invention. The average word recognition rate of Korean is shown in case of selecting and using this to train the speech feature vector quantizer.

In FIG. 6, the graph on the left shows the average word recognition rate in English when the speech feature vector quantizer is trained using unplanned sample learning data, and the graph on the right shows the learning data according to the frequency of occurrence of each phoneme presented in the present invention. The average word recognition rate of English is shown in the case of selecting and using this to train the speech feature vector quantizer.

Hereinafter, the speech recognition performance test of the present invention shown in FIGS. 5 and 6 will be described.

In English, "multi-condition training data from Wall Street Journal DB (WSJ0)" was used as sample training data, and 7,138 spoken voice data recorded as "Sennheiser Near Microphone" and several types of remote microphones as voice input conditions. The spoken speech data was stored at a sampling rate of 16 kHz.

In addition, "Set1 to Set7" which is a part of 14 sets of multi-condition evaluation data among the evaluation data of "WSJ0" corresponding to 5,000 words is used for speech recognition performance evaluation. Here, each set is composed of 330 voice data, which is about 40 voices per speaker, which is recorded by eight speakers.

In addition, a 39th-order speech feature vector was used. For this purpose, 12th-order MFCC and log energy were extracted, and first- and second-order differential coefficients were used. In the speech feature vector extraction, "distributed speech recognition front-end standard algorithm proposed in ETSI" is used.

In addition, the cepstrum mean normalization and energy normalization methods were used for the speech feature vectors used for learning.

In addition, the acoustic model used a "left-to-right model" with three states, and used four context-independent mixing densities and a "cross-word triphone model". These acoustic models were trained using the "HTK version 3.2 toolkit". For example, starting with 41 monophonic acoustic models with two silent models and extending to a triphone-based acoustic model, state sharing using binary trees was performed to reduce the number of states. This model consists of 8,360 triphone acoustic models and 5,356 acoustic models.

After setting the speech recognition system under the above conditions, the speech recognition performance without the speech feature vector quantization process, and the speech recognition performance when the speech feature vector quantizer is trained using unplanned sample training data, and the present invention In the case of training the speech feature vector quantizer by selecting the learning data according to the frequency of each phoneme presented in the above, the speech recognition performance is compared as follows.

[Table 5] below shows word false recognition rate among speech recognition performances that do not undergo speech feature vector quantization. In Table 5, it can be seen that the mean word recognition rate is 21.8% for the seven sets.

Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 Avg 14.3 17.6 22.1 27.0 25.1 23.1 23.5 21.8

Next, the word misrecognition rate of the codebook generated from the unplanned training data and applied to the speech feature vector quantizer, and the codebook generated from the selected training data using the algorithm proposed by the present invention and applied to the speech feature vector quantizer. Comparing the word misrecognition rate of is as follows.

[Table 6] shows the word misrecognition rate of the codebook generated from the unplanned training data and applied to the speech feature vector quantizer, and the following [Table 7] is selected from the training data selected using the algorithm of the present invention. A codebook is generated and applied to the speech feature vector quantizer.

Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 Avg 17.8 18.9 25.1 29.4 26.5 24.8 25.9 24.1

Set 1 Set 2 Set 3 Set 4 Set 5 Set 6 Set 7 Avg 13.6 16.8 23.0 27.6 24.9 21.8 24.5 21.7

In [Table 6] and [Table 7], the conventional unplanned learning data method has a mean word misrecognition rate of 7 sets for 7 sets, whereas the learning data selection method according to the phoneme occurrence frequency proposed in the present invention. The average word misrecognition rate of 7 sets is 21.7%.

For example, as a result of using the algorithm proposed in the present invention, it can be seen that the word misrecognition rate is reduced by about 10% compared to the existing unplanned learning data selection algorithm. It can be seen that this is almost the same as the average word misrecognition rate of 21.8% by the method that does not go through the speech feature vector quantization process, the present invention is that the speech recognition performance is not degraded due to the speech feature vector quantization process Would be easy to understand.

As described above, the learning data selection algorithm proposed in the present invention can be applied to any device or system equipped with other speech feature vector quantizers as well as the distributed speech recognition system illustrated in FIG. 1. For example, the present invention can be applied to a quantizer of a linear prediction coefficient (LPC) used for speech encoding, and the multimedia processing systems such as video and audio are considered in consideration of the characteristics of the vector to be quantized. It will be readily understood by those skilled in the art that the present invention can be applied to a processing device.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

As described above, the present invention has an effect of further improving the speech recognition performance under the condition that the number of bits required for quantizing the speech feature vector is the same as compared with the conventional unplanned sample learning data.

In addition, the present invention can achieve the same speech recognition performance compared to the system that does not perform the conventional speech feature vector quantization process, which has the effect of preventing the speech recognition performance is degraded due to the speech feature vector quantization process. .

In addition, the present invention can increase the reliability of the training data in the speech recognition, and thus, by accurately estimating an accurate acoustic model through the HMM, a better speech recognition success rate is achieved even through an existing speech recognition device that does not have a separate processor and device. There is a guaranteed effect.

Claims (10)

In the method for selecting the training data in the speech feature vector quantization for speech recognition, A force alignment step of obtaining alignment information for each phoneme by force-aligning the voice data when receiving the sample voice data; A phoneme appearance list generation step of generating a phoneme appearance list based on the obtained pronunciation information; A phoneme appearance frequency ratio calculation step of statistically calculating an appearance frequency ratio of phonemes according to a corresponding language based on the generated phoneme appearance list; And Learning data derivation step of deriving training data for minimizing the error between the total frequency and the frequency of appearance frequency for each phoneme with reference to the calculated statistical value Training data selection method based on a non-uniform sample in speech feature vector quantization for speech recognition comprising a. The method of claim 1, The pronunciation information includes speech data, pronunciation transcription, phoneme type, start time for each phoneme, and end time for each phoneme. Selection method. The method of claim 1, The method for selecting learning data based on non-uniform samples in quantizing a speech feature vector for speech recognition, characterized in that the phoneme appearance list includes a frequency of each phoneme and a total phoneme number of training data. The method of claim 1, The speech feature vector is a learning data selection method based on a non-uniform sample in the speech feature vector quantization for speech recognition, characterized in that it is expressed as Mel-Frequency Cepstral Coefficient (MFCC). The method of claim 1, Learning language based on a non-uniform sample in speech feature vector quantization for speech recognition, characterized in that the language is Korean. The method of claim 1, Learning language based on a non-uniform sample in speech feature vector quantization for speech recognition, wherein the language is English. The method of claim 1, The training data selected by the training data derivation step is based on a non-uniform sample in speech feature vector quantization for speech recognition, characterized in that it is used as training data for generating an acoustic model in a Hidden Markov Model (HMM). Learning data selection method. The method of claim 1, The phoneme appearance frequency ratio calculation step, A method for selecting training data based on a non-uniform sample in quantizing a speech feature vector for speech recognition, characterized by analyzing a ratio of any pronunciation among pronunciation strings based on a pronunciation dictionary of the language. The method according to any one of claims 1 to 8, The learning data derivation step, Speech recognition, characterized in that for selecting the number of frequencies corresponding to the difference between the appearance frequency ratio for a specific phoneme and the phoneme appearance frequency ratio of the total phoneme in the sample speech data in the sample learning data A method for selecting training data based on non-uniform samples in speech feature vector quantization. In a speech processing device having a processor, A function of obtaining alignment information for each phoneme by force-aligning the voice data when receiving sample voice data; Generating a phoneme appearance list based on the acquired pronunciation information; A function of statistically calculating an appearance frequency ratio of phonemes according to a corresponding language based on the generated phoneme appearance list; And A function of deriving learning data for minimizing the error between the total frequency and the frequency of appearance frequency for each phoneme by referring to the calculated statistical value A computer-readable recording medium having recorded thereon a program for realizing this.

Images