KR101424193B1 - System And Method of Pronunciation Variation Modeling Based on Indirect data-driven method for Foreign Speech Recognition - Google Patents

Abstract

A non-direct data-based pronunciation variation modeling system and method for enhancing the performance of a speech recognition system for a non-native speaker voice are provided. A modeling system for speech recognition, comprising: a feature extraction unit for extracting a feature vector from an externally input speech; An acoustic model database for storing acoustic data obtained in advance; A pronunciation model database for storing pronunciation data; And a speech recognition unit for finding a word sequence having the highest phoneme probability based on the feature vector from the word sequence generated by synthesizing the sound data read from the acoustic model database and the pronunciation data read from the pronunciation model database. Therefore, it is possible to improve the recognition performance of the voice of the native speaker without reducing the recognition performance of the native speaker voice.

Speech recognition system, non-direct data base, pronunciation variation modeling

Description

TECHNICAL FIELD The present invention relates to a non-direct data-based pronunciation variation modeling system and method for enhancing the performance of a speech recognition system for a non-native speaker voice,

The present invention relates to speech recognition technology for continuous speech recognition, and more particularly, to a non-direct data-based pronunciation variation modeling system and method for improving the performance of a speech recognition system for non-native speaker speech.

In general, the speech recognition system in the natural language processing field is mainly optimized for the native speaker voice because it is learned by the native speaker's voice. Therefore, speech recognition performance by native speakers is good, but speech recognition performance by other speakers is not good.

A description of a recognition performance of a speech of a non-native speaker to improve the problem of the speech recognition system will be described with reference to FIG.

10 is a block diagram schematically showing a general continuous speech recognition system. Contineous Voice Recognition System (Contineous Voice Recognition) of FIG. 10 can be roughly divided into two modules, and can be divided into a feature extraction module 11 and a voice recognition module 12. More specifically, the speech recognition module 12 of the continuous speech recognition system is composed of three models: an acoustic model 13, a pronunciation model 14, and a language model 15.

In the continuous speech recognition system, when a voice input is received, only the feature vectors useful for recognizing the input voice through the feature extraction module 11 are extracted. The feature vector searches the acoustic model 13, the pronunciation model 14, and the language model 15 in the speech recognition module 12. The acoustic model 13 and the pronunciation model 14 are used for word unit search and the language model 15 is used for sentence unit search. Further, the voice recognition module 12 searches the acoustic model 13, the pronunciation model 14, and the language model 15 and outputs the recognized result.

Here, research on speech recognition can be considered as an acoustic modeling viewpoint, a pronunciation modeling viewpoint, and a language modeling viewpoint.

In the pronunciation modeling viewpoint, two representative methods among the various methods proposed so far are H. H. Strik, H. Strik and C. Cucchiarinin, "Modeling pronunciation variation for ASR: A survey of the literature," Speech Comm. , Vol. 29, nos. 2-4, pp. 225-246, Nov. 1999, hereinafter referred to as Document 1).

In reference to Document 1, the pronunciation model includes a knowledge-based method and a data-driven modeling method. The knowledge-based method uses a currently available linguistic knowledge , The data-based method is a method of extracting pronunciation variation from various types of pronunciation variations such as phonological shortening, phonological elimination, typical eruption, and utterance errors from speech signals.

In the case of the knowledge-based method, the pronunciation variation in the above-mentioned dialogue is considered in consideration of the general pronunciation variation, so that the pronunciation sequences according to the phoneme variation rule are generated more than necessary, thereby increasing the confusability. In addition, the knowledge-based method requires a lot of time and effort to define the phoneme rules that are appropriate for other speakers. For this reason, it is more effective to use the data-based method.

See the description of pronunciation modeling in speech technology introduced in T. Svendsen's paper describing the data-based method. (T. Svendsen, "Pronunciation modeling for speech technology," in Proc. Of SPCOM , pp. 11-16, Dec. 2004, hereinafter referred to as document 2).

Referring to Document 2, the data-based method described in the pronunciation model introduced in Document 1 is divided into a direct data driven method and an indirect data-driven method.

In Document 2, a direct data-based method forcibly recognizes individual speech data for pronunciation variation extraction or performs phoneme recognition to directly use pronunciation variation for words in the pronunciation dictionary. However, the direct data-based method shows good performance when the words in the pronunciation dictionary are sufficiently contained in the individual speech data of the pronunciation variation, but it shows weak performance when not.

On the other hand, the non-direct data-based method uses pronunciation data (training data) based on pronunciation rules that can be applied by extracting phoneme variation rules from individual speech data for pronunciation variation extraction in pronunciation modeling There is an advantage that the pronunciation variation can also be generated.

Thus, according to the pronunciation model of Document 2, the non-direct data-based method is mainly used in dialogue voice because it has an advantage of generating a pronunciation variation for a word. However, when the pronunciation model of the document 2 is used for the voice of a non-native speaker, the performance of the voice of the native speaker is good, but the performance of the native speaker is frequently deteriorated due to an increase in complexity. The voice of the native speaker influences the voice of the speaker of the other language, thereby deteriorating the performance of the entire pronunciation model. That is, the pronunciation model of Document 2 is advantageous in generating a pronunciation variation, but it has a problem of deteriorating speech recognition performance of a native speaker and a native speaker. Therefore, it is required to model a pronunciation dictionary which is good not only for the speech recognition of the non-native speaker but also for the native speaker voice recognition.

Therefore, it is an object of the present invention to provide a modeling system using a phonetic dictionary in order to solve the problem of deteriorating native speaker's speech recognition performance as well as speech recognition of a non-native speaker speaker, The present invention provides a non-direct data-based pronunciation variation modeling method using speaker speech data to improve recognition performance of a native speaker speech.

According to the pronunciation variation modeling system of the present invention for achieving the above object,

A feature extraction unit for extracting a feature vector from an externally input speech;

An acoustic model database for storing acoustic data obtained in advance;

A pronunciation model database for storing pronunciation data; And

And a speech recognition unit for finding a word string having the highest phoneme probability based on the feature vector from a word string generated by synthesizing the sound data read from the acoustic model database and the pronunciation data read from the pronunciation model database.

Further, the speech recognition unit according to the pronunciation variation modeling system of the present invention includes a dynamic alignment unit,

A first phoneme recognition unit for recognizing speech data for native speaker development, a first standard phoneme recognition unit for generating a first standard phoneme string by transferring text data for native speaker development, A native speaker dynamic programming unit for arranging the first recognized phoneme string and the first standard phoneme string generated by the transfer in the first standard phoneme string portion by dynamic programming;

A second phonemic recognition unit for recognizing speech data for development of a second language speaker, a second standard phoneme recognition unit for generating a second standard phoneme string by transferring the text data for development of a second language speaker, And a second language dynamic programming unit for arranging the recognized phoneme string recognized and sent and the second standard phoneme string generated by transferring in the second standard phoneme string portion by the dynamic programming method.

에 의해서 발음 변이 패턴을 정의한다. (여기에서 L1과 L2는 상기 제1 및 제2 표준음소열 각각의 왼쪽 음소들이고, R ₁ 과 R ₂ 는 상기 제1 및 제2 표준음소열 각각의 오른쪽 음소들이고, 또한 X 는 상기 제1 및 제2 표준음소열의 음소로서 음소 인식 결과 음소열의 음소인 Y 에 대응됨.) Further, in the dynamic programming of the dynamic alignment unit according to the pronunciation variation modeling system of the present invention, each of the first and second standard phonemes is expressed by the following equation

To define a pronunciation variation pattern. (Wherein L1 and L2 are the first and second deulyigo standard phoneme each left column phoneme, R ₁ and R ₂ are the first and second deulyigo each of the right phoneme standard phoneme heat, and X is the first and And corresponds to the phoneme Y of the second standard phoneme string and the phoneme of the phoneme string.

In addition,

And a phoneme decision tree (phoneme decision tree) is generated from the mutated phoneme string pattern corresponding to the phoneme string aligned by the native speaker dynamic programming section and the mutated phoneme string pattern corresponding to the phoneme string aligned by the non- Create a rule.

In addition, the pronunciation variation modeling system of the present invention generates a variation phoneme by performing pruning through Rule Accuracy in order to extract the phoneme variation rules effectively, and generates a new phoneme phoneme and a new A standard pronunciation dictionary adaptation unit for generating a multi-phonetic dictionary;

A pronunciation dictionary unit for storing the phonemes generated by the standard pronunciation dictionary adaptation unit; And

And a pattern matching unit for forming a pattern by combining mutation phonemes stored in the pronunciation dictionary unit, stored sound data stored in the acoustic model database, and language information stored in the language model database.

In addition, the speech recognition modeling method using the modeling system for speech recognition according to the present invention includes a first recognition phoneme string recognizing speech data for native speaker development and a first standard phoneme string generated by transferring text data for native speaker development Arranging the columns into native speaker phonemes by dynamic programming;

Arranging the second standard phoneme string generated by transferring the second recognized phoneme string recognizing the speech data for the second language speaker development and the text data for developing the second language speaker to the second language speaker phoneme according to the dynamic programming method;

Generating a first variation phoneme string pattern corresponding to a phoneme string arranged by a native speaker phoneme by the dynamic programming method and a second variation phoneme string pattern corresponding to a phoneme string aligned with a non-native speaker phoneme by the dynamic programming method;

Generating a phoneme mutation rule using the generated first and second mutated phoneme string patterns using a phoneme tree;

Generating a mutation phoneme by performing pruning through Rule Accuracy to effectively extract the phoneme mutation rule and generating a multi-phonetic dictionary for a native speaker phoneme and a non-native speaker phoneme; And

And pattern matching the variable phoneme by combining the stored sound data stored in the acoustic model database and the language information stored in the language model database.

The first and second standard phoneme strings according to the pronunciation variation modeling method of the present invention are generated using at least one of a representative phoneme string, a knowledge base, and a direct transfer mode.

Further, according to the pronunciation variation modeling method of the present invention, each of the first and second standard phoneme strings is expressed by the following equation

에 의해서 발음 변이 패턴을 정의한다. 여기에서 L1과 L2는 상기 제1 및 제2 표준음소열 각각의 왼쪽 음소들이고, R ₁ 과 R ₂ 는 상기 제1 및 제2 표준음소열 각각의 오른쪽 음소들이고, 또한 X 는 상기 제1 및 제2 표준음소열의 음소로서 음소 인식 결과 음소열의 음소인 Y 에 대응된다.

To define a pronunciation variation pattern. Here, L1 and L2 are the first and second deulyigo standard phoneme each left column phoneme, R ₁ and R ₂ are the first and second deulyigo each of the right phoneme standard phoneme heat, and X is the first and the 2 is a phoneme of a standard phoneme sequence, and corresponds to a phoneme Y of a phoneme string.

Therefore, the pronunciation variation modeling system and method of the present invention can improve the recognition performance of the speaker speech of the non-native speakers, and enable the non-direct data-based pronunciation variation modeling using the speaker data of the non-native speakers and the native speech data of the native speaker .

As described above, according to the non-direct data-based pronunciation variation modeling system and method using the non-native speaker voice data of the present invention and the native speaker's voice data, it is possible to improve the performance of the speech recognition system for the non- The performance of the speech recognition system with respect to the speaker voice is not deteriorated.

Further, the pronunciation variation modeling system and method according to the present invention can reduce the human effort and time for analyzing the pronunciation variation of a non-native speaker voice based on data, and can extract pronunciation variation of a non- It enables extraction of pronunciation variations even in situations where knowledge is lacking. The present invention can also generate pronunciation variations for new words using a non-direct data-based method.

While the invention is susceptible to various modifications and alternative forms, specific embodiments thereof are shown by way of example in the drawings and will herein be described in detail. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. Like reference numerals are used for like elements in describing each drawing.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically illustrating an embodiment of a non-direct data-based pronunciation variation modeling system according to an embodiment of the present invention.

Referring to FIG. 1, in a non-direct data-based pronunciation variation modeling method according to an embodiment of the present invention, the continuous speech recognition system 100 can be roughly divided into two parts, and includes a feature extraction unit 101, As shown in FIG. The speech recognition unit 102 of the speech recognition system 100 further includes a word unit search unit 90 and a sentence unit search unit 95 that is connected to the word unit search unit 90 and transmits the word .

The feature extraction unit 101 extracts only information that is useful for recognition from an externally input voice and converts it into a feature vector fv. That is, in the speech recognition system 100, the feature extraction unit 101 extracts the input speech as the feature vector fv when the speech input comes from the outside. The feature vector (fv) can be extracted from the speech for speech recognition, for example using a 12th order mel-cepstrum (MFCC), log energy, or a first order, second order derivative thereof.

The word model synthesis unit 93 synthesizes the sound data p1 stored in the acoustic model database 102 and the pronunciation data p2 stored in the pronunciation model database 103 to generate a word sequence ws.

The word unit search unit 90 of the speech recognition unit 102 refers to the feature vector fv provided from the feature extraction unit 101 and the generated word sequence ws for word unit search.

The grammar unit 97 and the semantic analysis unit 99 generate a sentence string ps as the language data p3 stored in the language model database 102. [

In the speech recognition unit 102, the sentence unit search unit 95 receives the word string ws through the word unit search unit 90 and outputs the generated sentence string ps to the word unit search (90).

The word unit search unit 90 and the sentence unit search unit 95 exchange word sequence ws and sentence string (ps) data with each other.

The speech recognition unit 102 searches for a word string having the highest phoneme probability from the word string ws and the sentence string ps based on the feature vector fv using the Viterbi algorithm. Here, the Viterbi Algorithm is a search method for finding the most frequently occurring phonemes among the observed phonemes.

FIG. 2 is a block diagram showing a configuration for modeling a pronunciation in order to form a pronunciation dictionary in a speech recognition system for a non-native speaker voice in a non-direct data-based pronunciation variation modeling system.

The dynamic alignment unit 200 required for forming a phonetic dictionary in the non-direct data-based pronunciation variation modeling system according to the present invention includes a native speaker dynamic programming unit 220 and a dynamic speaker unit 220 connected to the native speaker dynamic programming unit 220 And a programming unit 210. The dynamic arranging unit 200 arranges the phonemes of the native speech dynamic programming unit 220 and the other language dynamic programming unit 210 by a dynamic programming method. Here, the dynamic programming algorithm refers to an optimal approach for aligning the phonemes of the input phonemes and phonemes in order to optimally achieve the recognition rate of a certain speaker.

The dynamic arranging unit 200 programs the non-native speaker phonemes and the native speaker phonemes to be recognized by the dynamic programming method for modeling the non-direct data-based pronunciation variation, and to arrange them into the phoneme strings.

The transposed phoneme rule generator 230 receives the phonemes v1 and v2 aligned in the dynamic arrangement unit 200 and patterns the phoneme strings v1 and v2 into a phoneme sequence in which a variation occurs, A phoneme decision tree (phoneme decision tree) is used to generate a variation phoneme rule from the mutated phoneme train pattern. That is, the dynamic arrangement unit 200 combines the mutated phoneme string v1 output through the native speaker dynamic programming unit 220 and the mutated phoneme string v2 output through the other language dynamic programming unit 210, After generating the rate pattern, the phoneme rule is generated using the phoneme decision tree from the rate pattern. In other words, when the mutation is performed for the aligned phonemes (v1, v2) in the dynamic arrangement unit 200, the variation rule generator 230 patterns the variation based on the mutation phoneme, Generate phoneme rules.

The standard pronunciation dictionary adaptation unit 240 generates a variation phoneme by performing pruning through Rule Accuracy in order to extract a phoneme variation rule efficiently and generates a phoneme corresponding to the native speaker phoneme and the other non- A new multi-phonetic dictionary is generated and applied to the phonetic dictionary unit 254.

The pronunciation dictionary unit 254 receives the variable phonemes vpd pruned and adapted from the standard pronunciation dictionary adaptation unit 240 and stores the received phonemes vpd.

The pattern matching unit 260 stores the phoneme vpd stored in the pronunciation dictionary unit 254, the acoustic data stored in the acoustic model database 253, and the language information stored in the language model database 252 To form a pattern.

In addition, the native dynamic programming unit 220 in the dynamic alignment unit 200 The recognition phoneme string n1 recognized by the phoneme recognition unit 202 and the standard phoneme string n2 obtained by transferring the standard phoneme string 204 are aligned by a dynamic programming method.

In the dynamic arrangement unit 200, the dynamic language programming unit 210 of the other language is configured to include a recognition phoneme string e1 recognized by the phoneme recognition unit 201 and a standard phoneme string e1 obtained by transferring from the standard phoneme string unit 203 (e2) by dynamic planning.

The standard phoneme strings e2 and n2 recognized by the standard phonetic column units 203 and 204 constituting the dynamic arrangement unit 200 of the present invention are stored in the memory unit 200 using the dynamic programming method of the dynamic arrangement unit 200 As shown in Fig. 4, It is preferable to construct a phoneme string for arranging a standard phoneme string using any one of a representative phoneme transfer, a knowledge-based transcription, and a direct transcription mode.

Meanwhile, the non-direct data-based pronunciation variation modeling method can be processed in five steps using the speech recognition system shown in FIG.

First, the non-direct data-based pronunciation variation modeling method of the first speech recognition system is arranged by the dynamic sorting unit 200. The dynamic sorting unit 200 firstly outputs the speech data for native speaker development to the phoneme recognition unit 202, And the standard phoneme string n2 transferred from the standard phoneme string unit 204 by the dynamic programming of the native speaker dynamic programming unit 220 to the native speaker's phoneme string n1, .

Secondly, the voice data for the development of other language speakers is also arranged in the same manner as the method of arranging by the dynamic programming method of the dynamic programming section 220. That is, the recognition phoneme string e1 that is recognized by the phoneme recognition unit 201 and the standard phoneme string e2 that is transferred from the standard phoneme string unit 203 to the non-language speaker development text data, And arranges the phonemes of other non-native speakers according to the dynamic programming method of the dynamic language programming unit 210 of the other language.

Third, the transitional phoneme strings v1 and v2, which are rearranged results obtained by the dynamic programming of the native speaker phoneme dynamic programming unit 210 and the non-native speaker phonetic dynamic programming unit 220, Generator 230 to acquire a variation phoneme pattern.

When the mutation phoneme rule generator 230 inputs the mutation phoneme string pattern to the variation rule generator 230, the variation rule generator 230 generates a pronunciation variation rule from the mutation phoneme string pattern using the phoneme decision tree do. In other words, the non-direct data-based phoneme mutation rule is generated by using the phoneme decision tree (described in detail in FIG. 6) as a mutation phoneme pattern in which the mutation phoneme rule generator 230 generates a pronunciation mutation.

Lastly, a pronunciation variation string vp is generated according to the non-direct data-based phoneme mutation rule of the variation rule generator 230, and the standard pronunciation dictionary adaptation unit 240 receives the native speaker phoneme and the non- To create a new multi-phonetic dictionary.

The generated variation phoneme vpd is stored in the pronunciation dictionary unit 254 and the phoneme vpd or d1 stored in the pronunciation dictionary unit 254 is stored in the acoustic model unit 230 and the language model unit 255 And the speech data d2 and the language data d3 output from the speech recognition unit 260 and pattern matching unit 260 in the pattern matching unit 260 to improve recognition performance of the speech of the native speaker without degrading the recognition performance of the native speaker speech .

A method for modeling pronunciation variation based on a non-direct data base on speech of a non-native speaker for speech recognition by a non-native speaker of the present invention VLSI chip, IP of SoC, and embedded software.

Now, the non-direct data-based pronunciation variation modeling method of the speech recognition system of the present invention will be described. 1. Phoneme recognition and alignment procedures are described by dynamic programming of dynamic arrangements. 2. Rule extraction using phoneme decision tree And a pronunciation dictionary adaptation procedure includes steps of acquiring a phoneme string pattern, generating a phoneme mutation rule using a phoneme tree with a phoneme pattern, and generating a new multi-phonetic dictionary by applying the phoneme mutation rule to the standard pronunciation dictionary adaptation unit 3. The recognition performance of the phonetic dictionary applied to the speech recognition system according to the phoneme recognition procedure, phoneme recognition and alignment procedure is compared and evaluated.

1. Phoneme recognition and alignment procedure

FIG. 3 is a table showing the sound list used in the pronunciation variation modeling of the non-direct data base of the present invention, and shows a Hangul pronunciation symbol transcribed to the Roman pronunciation symbol.

In the pronunciation variation modeling method of the present invention, a phoneme list of vocabulary consecutive is referred to in order to model speech by using the speech recognition system 100 for speech data for development of a non-native speaker and speech data for native speaker development.

3, when the phoneme recognition is performed using, for example, a Korean continuous speech recognition system, the phoneme recognition unit 202 of the speech recognition system 100 of FIG. The triphone model is 10,138, and the phonemes used are a total of 40 phonemes including 21 neutral vowels including 9 short vowels and 12 double vowel syllables, and 19 consonants consisting of the first and last consonants. (back-off bigram) language model. Here, the BackoPavigar language model is the same word in the indexing process, but it measures the similarity between two words of the index target and judges the same class if it exceeds the threshold value.

Referring to FIG. 2 again, the non-direct data-based pronunciation variation modeling method of the speech recognition system according to the present invention is a method of modeling phoneme strings (e1, n1) and standard phoneme strings (e2, dynamic programming algorithm), the corresponding phonemes represent the phoneme patterns as shown in the following Equation (1).

[Equation 1]

(Where L1 and L2 are the left phonemes of the standard phoneme string, R ₁ and R ₂ are the right phonemes of the standard phoneme string (e2, n2), and X is the phoneme of the standard phoneme string (e2, n2) One result corresponds to the phoneme Y of the phoneme string v1, v2.)

However, when a phoneme string is acquired by the pronunciation variation modeling method represented by the phoneme pattern shown in Equation (1), it is possible to acquire a free variation phoneme, but errors of the recognition system 100 itself are frequently generated, It may be difficult to distinguish between an error or a speech recognition system itself error. Thus, the embodiments of the present invention have solved these problems in two ways.

In order to reduce the error of the recognition system 100, the first method of complementing 100 results in the search using the Viterbi algorithm, the phoneme recognition rate increased from 71.5% to 76.8%. In order to classify whether the recognition system 100 itself is an error or an error due to a mutated phoneme, a method of interpolating neighboring phonemes of a standard phoneme string (e2, n2) and a phoneme corresponding to a phoneme corresponding to a resultant phoneme string As a result of using the phoneme patterns only when more than half of the phonemes are the same, the recognition rate of phonemes is significantly increased.

FIG. 4 is a table expressing the arrangement by the dynamic arranging unit illustrated in FIG. 2, in which a phoneme sequence obtained by phoneme recognition using a standard phoneme list as shown in FIG. 3 as a reference sequence is sorted The table shows an example.

Referring to FIG. 4, in the phoneme recognized by the dynamic sorting unit 200 shown in FIG. 2, since the phonetic transitions occurring in the words in the phoneme string and the phonetic transitions occurring between the words and the words are different, Quot; The boundary of each word is marked with an @ symbol as shown in FIG.

In the dynamic alignment unit 200, the standard phoneme string 204 includes: For example, by expressing alignment by dynamic alignment, it is because "it is a month with many different meanings." As shown in Fig.

Examples of the standard phoneme sequence include a canadian phoneme sequence represented by a representative pronunciation of the speaker, a knowledge-based phoneme sequence in which the speaker's voice is changed, a hand-held phoneme in which the pronunciation of the speaker is directly transferred Three standard reference sequences obtained by the column and replaceable phoneme sequences aligned phonemically with respect to these standard phoneme strings are represented.

In Fig. 4, the representative phoneme string corresponds to each phoneme in Korean as shown in Fig. 3. The knowledge base phoneme sequence is a phoneme sequence generated by applying the phoneme variation rule based on the Korean standard pronunciation method. The direct transcriptional phoneme sequence is a phoneme sequence that is pronounced by the person and is pronounced by the person. The alternative phoneme string is a substitute phoneme string that can replace the recognition result of the voice to be pronounced in the pronunciation rule generator 230.

5 is a table showing a phoneme rule pattern extracted based on a phoneme string by the dynamic arrangement unit illustrated in FIG.

을 사용하여 얻은 결과 규칙패턴은, 예컨대 도 4의 정렬테이블에 기재된 한국어 "@달이기@"에 해당하는 음소패턴이 제거된다. 즉 상호 이웃하고 있는 음소의 절반이상이 상이한 경우 규칙패턴 리스트에서 하나의 규칙패턴이 제거됨을 이해할 수 있다. Referring to FIG. 5, according to a method for removing a phoneme pattern due to a recognition error of the recognition system,

For example, the phoneme pattern corresponding to the Korean word " @ " described in the alignment table of FIG. 4 is removed. That is, when more than half of mutually neighboring phonemes are different, one rule pattern is removed from the rule pattern list.

For example, in Fig. 5, da-l + i + g -> l, which is one phoneme rule pattern, is removed. This is because the phoneme string 6 Because the phonemes 'd', 'a', and 'i' are different. That is, if the number of neighboring phonemes is different by more than half, it is determined that the phoneme recognition unit generates an error, so that it can be confirmed that the phoneme determination tree can not be used as the structure of the phoneme determination tree described in the following section.

2. Rule extraction and pronunciation dictionary using phoneme decision tree

6 illustrates a phoneme determination tree by a non-direct data-based pronunciation variation modeling method. The phoneme variation rules are extracted by the phoneme rule generator 230 using the phoneme decision tree of FIG. 6, and one of CART, CHAID, and C4.5 programming methods can be used for the phoneme decision tree. In the embodiment of the present invention, the phoneme decision tree program C4.5 is used.

This C4.5 phoneme tree program was developed by J. Ross Quinlan as an extension of the existing ID3 algorithm. C4.5 The phoneme tree program can vary the number of branches without binary separation like CART. C4.5 The phoneme tree program uses a method similar to CART for continuous variables, but uses a different method for categorical types.

In other words, according to the present invention, the phoneme mutation rules are extracted through the C4.5 phoneme decision tree by the phoneme pattern generated after the alignment according to the example of the standard phoneme sequence using the dynamic programming by the phoneme recognition and alignment procedure as shown in FIG.

Separation based on the C4.5 phoneme decision tree was used for statistical class (statistical classifier) and selected with the left and right phonemes of the phoneme X, separating the left and right phoneme phoneme two each two items.

에 있어서 표준음소열 L ₁, L ₂, R ₁, 그리고 R ₂ 가 분리기준이다. C4.5 음소결정트리 프로그램에 의한 결과 클래스는 수학식 (1)에서 Y 로 표시되는 결과 음소열의 대응되는 음소이다. 표준음소열 L ₁, L ₂, R ₁, 그리고 R ₂는 도 4에 도시된 정렬테이블에서 40개 음소들이 각각 사용되었다. That is, in Equation (1)

The standard phoneme strings L ₁ , L ₂ , R ₁ , and R ₂ are separation criteria. The result class by the C4.5 phoneme tree program is the corresponding phoneme in the resultant phoneme string indicated by Y in Equation (1). The standard phonemes L ₁ , L ₂ , R ₁ , and R ₂ have 40 phonemes in the alignment table shown in FIG. 4, respectively.

C4.5 Phoneme Decision Tree If you create a phoneme decision tree in a format appropriate to your program and create rules with the options provided, each phoneme decision tree generates a rule by tracing back from the root to the leaf of each tree (see FIG. 6).

FIG. 6 is an example of generating a phoneme determination tree for the phoneme 'k'. C4.5 Phoneme Decision Tree In the phoneme decision tree generated by the phoneme pattern for 'k (k)' using the program, the nodes of the result class are 'k' and 'g', which are determined by the separation criteria L ₁ and R ₁ do.

The phoneme decision tree of FIG. 6 is a phoneme decision tree determined and searched by the following Algorithm expression example 1, in which the standard phoneme string L ₁ is 'n' or 'jv' or the standard phoneme string L ₁ If 'a' or 'ⓐ' and the standard phoneme string R ₁ is 'v' or 'U' then the node of the result class will be changed from 'k' to 'g'.

[Expression 1]

In addition, if the rule is extracted according to the options provided by the C4.5 phoneme tree program of the phoneme decision tree of the phoneme 'k (k)', the following rules can be obtained.

Rule N :

R ₁ = 'v'-> class 'g' [Rule Accuracy]

                          Default: class 'k'

(Where N is the rule number and N = 1 in the rule for phoneme 'k (k)').

In the above rule, the rule accuracy [Rule Accuracy] represents the relative frequency applied to the rule in which all the phoneme patterns for the phoneme 'k (k)' are generated. The default is applied when there is nothing to be applied to the generated rule of the phoneme pattern. In order to effectively extract the phoneme rules, we used pruning rule accuracy and used 25% as a threshold value in this experiment.

Lastly, in the speech recognition system shown in FIG. 2, the standard pronunciation dictionary adaptation unit 240 applies a pruned rule to the pronunciation dictionary unit 254. The pronunciation dictionary unit 254 is composed of a word and a phoneme string corresponding to the word. When a mutation phoneme is generated through a rule in the phoneme string, a phoneme string of a mutation in the phoneme string of the standard pronunciation dictionary is added to the pronunciation dictionary portion 254 to generate a new pronunciation dictionary.

For example, when the standard phoneme string of 'bigger' is 'k v z i d a', the first phoneme 'k' applies to the rule of Equation 3, so that the phoneme 'k' can be changed to the phoneme 'g'. Thus, the phoneme string 'g v z i d a' can be obtained, and a new phonetic dictionary can be created by adding it.

3. Performance evaluation of pronunciation dictionary adaptation procedure

7, 8, and 9, recognition performance of speech recognition in other languages and native speakers is evaluated using the pronunciation dictionary adaptation procedure by the non-direct data-based pronunciation variation modeling system and method of the speech recognition system of the present invention.

FIG. 7 is a graph showing an example of recognition performance displayed as a word recognition rate which is mistaken when a phonetic dictionary is used in the speech recognition system of the present invention.

Referring to FIG. 7, in the pronunciation dictionary adaptation procedure rule of the present invention, three pronunciation dictionary units 254 are used, which are a representative phoneme string, a knowledge base, and a direct pronunciation pronunciation dictionary. When these dictionaries were used for speech recognition of other speakers, word recognition rates were 28.33%, 27.73%, and 27.73%, respectively, and 43.47%, 34.43%, and 35.00% respectively when used for native speaker speech recognition.

As described in FIG. 7, according to the arrangement by the dynamic arranging unit illustrated in FIG. 2, the word recognition rate (%) is higher than that of the other speaker speech recognition because it is a large capacity continuous speech recognition in the case of native speaker test speech In the case of actual test voice data, it is confirmed that the word recognition rate of the voice recognition of the other speaker is much higher.

FIG. 8 is a table showing examples of recognition performance when a phonetic dictionary applied only to a native speaker voice and a phonetic dictionary applied only to a non-native speaker voice are used.

In FIG. 8, when a phonetic dictionary according to a non-native speaker rule is used, the speech recognition rate of the non-native speaker test voice is 22.87% for the representative phoneme string, 22.40% for the knowledge base, and 22.33% 5.46%, 5.33%, and 5.4%, respectively, compared to the reference dictionary mentioned above.

On the other hand, the word recognition rate of the native speaker test voice is 46.65% at the representative phoneme, 36.19% at the knowledge base, and 34.94% at the direct phonetic level. Likewise, when the pronunciation dictionary according to the native speaker rule is used, the performance of the native speaker test voice improves, but the performance of the other language speaker test voice improves less than that of the other language speaker rules.

8 and 9, when the phoneme mutation rule developed only for the native speaker voice is used, it is referred to as a 'native speaker rule', and in the case of using the phoneme mutation rule developed only by the voice of a non-native speaker, "He said. In addition, the phoneme mutation rule developed using both the native speaker and the voice of another linguistically speaking speaker is called a 'binding rule'. In addition, each standard transcription is divided into a representative phoneme sequence, a knowledge base, and a direct transcription.

FIG. 9 shows an example of recognition performance when a combination rule using both native speaker rules and non-native speaker rules is applied. Referring to FIG. 9, in the non-direct data-based pronunciation variation modeling method of the speech recognition system, recognition results of the speech dictionary test speech 254 adapted to the non-native speaker test speech using the combination rule, 22.40% for standard transcription, 23.53% for knowledge base, and 22.60% for direct transcription. In addition, the results of recognizing the native speaker test voice are 39.49%, 35.40%, and 34.60%, respectively, which are similar to those when the native speaker rules are applied.

Therefore, through the non-direct data-based pronunciation variation modeling system and method of the speech recognition system of the present invention, it is possible to improve the performance of the native speaker speech recognition as well as the non-native speaker speech recognition, .

While the present invention has been described in connection with what is presently considered to be practical exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagram showing a configuration of a continuous speech recognition system according to the present invention. Fig.

FIG. 2 is a diagram illustrating a non-direct data-based pronunciation variation modeling process of the speech recognition system illustrated in FIG. 1;

FIG. 3 is a table showing voice-by-voice configuration of the pronunciation dictionary modeling in non-direct data-based pronunciation variation modeling of the present invention,

FIG. 4 is a table representing an example of alignment by the dynamic alignment unit illustrated in FIG. 2,

3 is a diagram illustrating a pronunciation variation modeling method of a non-direct data-based method using voice data of a non-native speaker and voice data of a native speaker suggested in the present invention.

FIG. 5 is a table showing examples of phoneme patterns extracted based on the alignment table by the dynamic alignment unit illustrated in FIG. 2,

FIG. 6 is a diagram showing a phoneme determination based on the phoneme pattern table of FIG. 4 in a tree structure;

FIG. 7 is a graph showing the word recognition rate in the pronunciation dictionary when sorting by the dynamic sorting unit illustrated in FIG. 2,

FIG. 8 is a table showing a test according to the pronunciation pre-adaptation procedure rule at the time of sorting by the dynamic arranging unit illustrated in FIG. 2;

FIG. 9 is a table showing the tests according to the pronunciation pre-combination rule at the time of sorting in the test illustrated in FIG. 8; and

10 is a block diagram schematically showing a general continuous speech recognition system.

Description of the Related Art

90: word unit search unit 93: word model unit

95: sentence unit search unit 97: grammar unit

99: Semantic analysis unit 100: Speech recognition system

101: Feature extraction unit 102: Voice recognition unit

200: dynamic alignment unit

Claims (7)

A feature extraction unit for extracting a feature vector from an externally input speech; An acoustic model database for storing acoustic data obtained in advance; A pronunciation model database for storing pronunciation data; And And a speech recognition unit for finding a word string having the highest phoneme probability based on the feature vector from a word string generated by synthesizing sound data read from the acoustic model database and pronunciation data read from the pronunciation model database, Wherein the pronunciation model database is generated by generating a phoneme mutation rule non-directly from data for native speaker and non-native speaker development, and applying a multi-phonetic dictionary generated using the phoneme mutation rule. The method according to claim 1, Wherein the speech recognition unit includes a dynamic alignment unit, A first phoneme recognition unit for recognizing speech data for native speaker development, a first standard phoneme recognition unit for generating a first standard phoneme string by transferring text data for native speaker development, A native speaker dynamic programming unit for arranging the first recognized phoneme string and the first standard phoneme string generated by the transfer in the first standard phoneme string portion by dynamic programming; A second phonemic recognition unit for recognizing speech data for speech recognition for another language speaker development, a second standard phonemic recognition unit for generating a second standard phoneme string by transferring the text data for developing another speaker's language, And a second language dynamic programming unit for arranging, by the dynamic programming method, the recognized phoneme string derived from the first standard phoneme string part and the second standard phoneme string generated by transferring from the second standard phoneme string part. 3. The method of claim 2, Wherein the first and second standard phonemes in the dynamic arrangement scheme are each expressed by the following equation

And a pronunciation variation pattern is defined by the speech recognition model. (Wherein L1 and L2 are the first and second deulyigo standard phoneme each left column phoneme, R ₁ and R ₂ are the first and second deulyigo each of the right phoneme standard phoneme heat, and X is the first and And corresponds to the phoneme Y of the second standard phoneme string and the phoneme of the phoneme string. 3. The method of claim 2, Wherein the dynamic alignment unit comprises: Wherein the phoneme string is generated by using a phoneme decision tree from a mutated phoneme string pattern corresponding to a phoneme string aligned by the native speaker dynamic programming section and a mutation phoneme string pattern corresponding to a phoneme string aligned by the non- A variation phoneme rule generator for generating a variation rule; In order to effectively extract the phoneme variation rule, pruning is performed through Rule Accuracy to generate a mutation phoneme, and a standard pronunciation dictionary for generating a new multi-phonetic dictionary for native speaker phonemes and other non- Adaptation part; A pronunciation dictionary unit for storing the phonemes generated by the standard pronunciation dictionary adaptation unit; And Further comprising a pattern matching unit for forming a pattern by combining mutation phonemes stored in the pronunciation dictionary unit, acoustic data stored in the acoustic model database, and language information stored in a language model database . In a speech recognition modeling method, Arranging a first standard phoneme string generated by transferring a first recognized phoneme string recognizing speech data for native speaker development and text data for native speaker development into a native speaker phoneme by a dynamic programming method; Arranging the second standard phoneme string generated by transferring the second recognized phoneme string recognizing the speech data for the second language speaker development and the text data for developing the second language speaker to the second language speaker phoneme according to the dynamic programming method; Generating a first variation phoneme string pattern corresponding to a phoneme string arranged by a native speaker phoneme by the dynamic programming method and a second variation phoneme string pattern corresponding to a phoneme string aligned with a non-native speaker phoneme by the dynamic programming method; Generating a phoneme mutation rule using the generated first and second mutated phoneme string patterns using a phoneme tree; Generating a mutation phoneme by performing pruning through Rule Accuracy to effectively extract the phoneme mutation rule and generating a multi-phonetic dictionary for a native speaker phoneme and a non-native speaker phoneme; And And pattern matching the variable phoneme by combining stored speech data stored in an acoustic model database and language information stored in a language model database. The speech recognition modeling method of claim 5, wherein the first and second standard phoneme strings are generated using at least one of a representative phoneme string, a knowledge base, and a direct transfer mode. 6. The method of claim 5, Wherein each of the first and second standard phoneme strings is represented by the following equation

Wherein L1 and L2 are left phonemes of the first and second standard phonemes, and R ₁ and R ₂ are the left and right phonemes of the first and second standard phoneme strings, respectively, And X is the phoneme of the first and second standard phonemes corresponding to the phoneme Y of the phoneme string resulting from the phoneme recognition).

Images