KR100777569B1 - The speech recognition method and apparatus using multimodal - Google Patents

Abstract

A multimodal voice recognition method and an apparatus thereof are provided to ensure a high success rate of voice recognition even in a low-spec portable terminal as well as a high performance computing device by performing voice recognition based on a multimodal synchronized consonant and a user's voice. A multimodal voice recognition method comprises the following steps of: receiving a consonant from a user and receiving a user's voice, and synchronizing the consonant and the user's voice(300,301); extracting vocabularies, started from the synchronized consonant, from a pronunciation dictionary(302); creating the extracted vocabularies as a voice recognition dictionary(303); and searching a corresponding vocabulary from the created voice recognition dictionary and recognizing a word or a sentence corresponding to the user's voice(304).

Description

Speech Recognition Method and Apparatus Using Multimodal TECHNICAL FIELD

1 is a configuration diagram of an embodiment of a terminal to which the present invention is applied.

Figure 2 is an illustration of an embodiment for showing a speech recognition process used in the present invention.

Figure 3 is a flow chart of an embodiment of a speech recognition method using multi-modal in accordance with the present invention.

* Explanation of symbols on the main parts of the drawing

10: key button 20: microphone

30: text processing unit 40: voice processing unit

50: multi-modal synchronization unit 60: speech recognition unit

70: recognition DB

The present invention relates to a speech recognition method and apparatus, and more particularly to multimodal, for example, when a user inputs a character through a key button or the like and utters a voice through a microphone or the like. The present invention relates to a voice recognition method using multi-modal and a device for recognizing the voice that has been spoken by the user by reducing the voice recognition target as a vocabulary.

Multimodal means that various types of input / output methods are used in the interface between a person and a machine. In the case of multimodal input, voice recognition, a keyboard, a pen, and a sensor can be used together. In the present invention, the user inputs specific information by hand and mouth, such as a user pressing a key button to input text and a microphone voice. Particularly, in the present invention, in the voice recognition, a start button of a vocabulary corresponding to a voice to be spoken by the user is input using a key button and the voice is uttered as a microphone.

Recently, in order to meet the trend of digital convergence, terminals equipped with various functions have been released. In order to use these specific functions, the user must input some information through the interface of the terminal.

For example, in a navigation terminal, a user is provided with traffic information from a starting point to a destination by inputting a word such as a building name or a place name by pressing a key button.

As another example, in order to inquire a schedule of a specific date in a PDA or the like, a user selects an icon on the LCD screen using a pen tool to obtain desired information.

As another example, in view of the fact that the above-described key button input method and pen tool input method is troublesome for the user due to the small size of the portable terminal, a virtual keyboard is displayed on the LCD screen so that the user can easily enter text. There is also a technique to make it possible.

Recently, considering the performance of the terminal, the limitation of the text-based information input method as described above [Example; You need to combine several key button inputs to complete a word, and it is not easy to use the pen tool to click on the LCD screen, especially in an environment where your eyes and hands are not free, such as while moving or driving. In order to overcome the hassle and accuracy, etc.], a terminal equipped with a speech recognition function (hereinafter referred to as a "speech recognition device") has been actively studied. Such a speech recognition apparatus recognizes speech spoken by a user and processes information as a result of the recognition.

The most important problem for the above-mentioned speech recognition device is how much the user's voice can be recognized [voice recognition success rate], but current technology does not guarantee 100% speech recognition success rate even for high-performance computing devices. In addition, the mobile terminal has a low voice recognition success rate.

Meanwhile, Korean Patent No. 474253 (name of the invention: "a voice recognition method using the first consonant of a word and a recording medium storing the same") as a related art for increasing the speech recognition success rate presented above.

In the prior art, the first consonant of a word corresponding to a speech to be spoken is first asked, and then a user is asked to speak a substantial word, thereby reducing the object of speech recognition as a vocabulary starting with the first consonant spoken by the user, thereby realizing the actual speech to be spoken later. Disclosed is a method of recognizing speech.

However, the prior art can reduce the number of word lookups stored in the recognition DB by reducing the speech recognition target as a vocabulary starting with the first consonant, thereby reducing the speech recognition processing time, but guaranteeing the success rate of the speech recognition. There is no way down.

For example, in the field of speech recognition, finding the starting point of a word to be recognized greatly influences the success rate of speech recognition. In Korean, the first consonant of a word is correctly spoken considering that the first consonants are mostly consonants, especially unvoiced sounds. It's unrecognizable.

In detail, the speech recognition technology uses voice energy detection to find the starting point of the voice to be recognized, and then extracts the feature vector from the user's spoken voice to perform the voice recognition function. On the other hand, considering that unvoiced sound is produced through the arteries of the lungs (lips, teeth, tongue, nasal cavity, oral cavity), unvoiced sounds with small energy, especially "ㄱ", "ㅅ", "ㅎ" Detecting as speech recognition is a very difficult problem in reality.

The present invention has been proposed to solve the above problems and to meet the above demands, and when a user inputs a character through a key button and utters a voice through a microphone or the like, based on the character input time point. The present invention provides a method and apparatus for recognizing speech using a multi-modal, which is prepared to find a starting point of a user's voice and reduces the object of speech recognition as a vocabulary starting with this character to recognize the speech spoken by the user. have.

According to a method of the present invention for achieving the above object, in a speech recognition method using multimodal, if a consonant is input from a user and the user's spoken voice is simultaneously received, the consonant and the user's spoken voice are synchronized. Making a step; Extracting vocabulary starting from the synchronized consonant from a pronunciation dictionary; Grouping the extracted vocabularies and generating them as a speech recognition target dictionary; And retrieving a corresponding vocabulary from the generated voice recognition target dictionary for the synchronized user spoken voice while generating the voice recognition target dictionary and recognizing a word or sentence corresponding to the user spoken voice. do.

On the other hand, the apparatus of the present invention, a speech recognition apparatus using a multimodal, a character processing unit for recognizing a text signal received from the user and processing a consonant corresponding to the consonant as a character to the multi-modal synchronization unit; A voice processor for recognizing a voice signal input from a user and processing the voice signal and transmitting the signal to a multimodal synchronizer; A multi-modal synchronization unit for synchronizing the consonant received from the text processing unit with the user spoken voice received from the voice processing unit; A recognition DB in which a pronunciation dictionary is stored; And extracting the vocabulary starting with the consonant from the pronunciation dictionary, combining the extracted vocabularies, generating them as a speech recognition target dictionary, and searching for the vocabulary from the generated speech recognition target dictionary. It includes a speech recognition unit for recognizing a word or sentence corresponding to the spoken voice.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an embodiment of a terminal to which the present invention is applied.

As shown in FIG. 1, the terminal to which the present invention is applied, that is, the voice recognition apparatus, includes a key button 10, a microphone 20, a text processing unit 30, a voice processing unit 40, and a multi-modal synchronization unit 50. , Speech recognition unit 60 and recognition DB 70.

Meanwhile, in describing the present invention, the text-based information input means (interface) will be described using the key button 10 as an example, but the pen tool 11, the virtual keyboard 12, the digitizer (not shown), and the haptic ( Any text input means such as not shown) can be used in the present invention. Here, there may be a limit in recognizing a letter or a word in a text input method through a digitizer or a haptic, but as described below, the digitizer or haptic used in the present invention only needs to recognize a first consonant of a word. It will be readily understood by those skilled in the art that there is no great difficulty in text recognition.

The present invention provides a user with a terminal (i.e. a speech recognition apparatus, a main; Such a speech recognition device has a speech recognition success rate while avoiding the hassle of inputting information into any terminal such as a mobile phone, a PDA, a navigation terminal, a portable media player, a laptop, a desktop computer, etc. To increase.

In particular, the present invention is to solve the problem of finding the starting point of speech recognition, that is, unvoiced voice and the voice to be recognized as a representative difficulty of speech recognition, as a technique devised to properly ensure the success rate of speech recognition, such as words, sentences, The multi-modal method is used to simultaneously receive texts and voices from the user and process them, while reducing the object of speech recognition as a vocabulary starting with the letters to recognize the voices spoken by the user.

Next, each component of the speech recognition apparatus of the present invention will be described.

The character processor 30 recognizes a key signal input from the user through the key button 10, processes the first consonant of a word corresponding to the character, and transmits the first consonant as a character to the multimodal synchronization unit 50.

For example, the user presses a key button corresponding to the first consonant of a word to input a character, and simultaneously or sequentially inputs a voice corresponding to the word, that is, information to be uttered, into the microphone.

On the other hand, the present invention is most preferably applied to a speech recognition device implemented in Korean, it is also applicable to a speech recognition device implemented in English, in this case, if the user inputs a letter by pressing the first alphabet of words with a key button do.

In addition, one word will be described as an example for convenience of explanation of the present invention, but the present invention is also applicable to a sentence composed of several words, which will be described later in detail.

The voice processor 40 recognizes a voice signal input from the user through the microphone 20, processes the voice signal, and transmits the voice signal to the multimodal synchronizer 50. In this case, the voice processing unit 40 simply performs signal processing, such as a DSP. For example, in the present invention, it is apparent that the actual signal processing for speech recognition is performed in the following speech recognition unit 60.

The multi-modal synchronization unit 50 performs text / voice synchronization so that the front and back of the first consonant of the word received from the text processing unit 30 and the voice received from the voice processing unit 40 are not reversed. For example, if a user inputs "a" as a character and "speaks" a voice, the user recognizes the letter "a" and this voice "potato" as one and synchronizes it with the voice recognition unit 60. Pass it over.

A typical speech recognition unit extracts a feature vector from a speech spoken by a user and performs speech recognition through a Hidden Markov Model (HMM) and a recognition DB. The speech recognition unit 60 of the present invention is also known in a speech recognition process. Do this.

In particular, the speech recognition unit 60 according to the present invention performs a speech recognition process using the first consonant of the word synchronized by the multi-modal synchronization unit 50 and the user spoken voice as parameters, and performs such speech recognition. The speech recognition process is performed by reducing (limiting) only the vocabulary beginning with the first consonant among the vocabulary on the pronunciation dictionary stored in the recognition DB. The speech recognition process proposed by the present invention will be described later in detail with reference to FIGS. 2 and 3.

As described above, the voice recognition result recognized by the voice recognition unit 60 is output in the form of text through the display unit or in the form of a voice through the speaker, so that the user can check whether the recognition of the speech spoken by the user is successful. do. Of course, when the user inputs the departure / destination information in the above-described multi-modal manner in the navigation terminal, it will be easily understood by those skilled in the art that the traffic guidance is provided through the display and the map from the starting point to the destination. .

FIG. 2 is a diagram illustrating an embodiment of a speech recognition process used in the present invention, and FIG. 3 is a flowchart of an embodiment of a speech recognition method using multi-modal according to the present invention.

As shown in FIG. 2, in a typical speech recognition process, the speech recognition unit 60 uses a Hidden Markov Model (HMM), extracts a speech feature vector 61 from a user speech, and extracts the extracted speech feature vector. A process of recognizing a word or sentence corresponding to the voice through pattern matching 62 is performed by searching the pronunciation dictionary 71, the language model 72, and the acoustic model 73 on the recognition DB 70 for the feature vector. Rough

In the pronunciation dictionary 71, a speech recognition target vocabulary is stored. In the case of Korean, the pronunciation dictionary provided by SiTEC's "CleanSent01" may be used. In the case of English, the pronunciation dictionary provided by Carneige Mellon University may be used. CMU dictionary V.0.6 "may be used, and any pronunciation dictionary may be used.

In the present invention, before the speech recognition unit 60 performs the actual speech recognition, only the vocabulary starting with the consonant received from the user is extracted from the pronunciation dictionary 71 stored in the recognition DB 70, and the extracted vocabulary. Are generated as a single speech recognition target dictionary.

The speech recognition target dictionary generated as above has a data structure in which a pronunciation dictionary is reduced, and searches for a vocabulary corresponding to a user spoken speech from the speech recognition target dictionary generated as described above, and performs the actual speech recognition process.

In particular, in constructing the recognition DB 70 in order to increase the performance of speech recognition, the speech recognition target vocabulary is provided with information necessary for recognition of each vocabulary.

For example, information about which phoneme sequence is formed for each vocabulary, information about which class each word belongs to, and information indicating which classes may precede or follow each word in performing sentence recognition. It is included on the 70, so that only the corresponding vocabulary can be extracted accurately and quickly from the pronunciation dictionary 71, so that the speech recognition target dictionary can be generated.

In the above-mentioned process of generating a speech recognition target dictionary, a consonant input by a user, such consonants may be input in order of one or more consonants, and are generated as a speech recognition target dictionary by grouping vocabularies composed of only the consonants. Such a voice recognition target dictionary may be generated in advance at the time of the multimodal synchronization or the dictionary. In the former case, if only the first consonant of a word is synchronized, the consonant is inputted. The moment the consonant is input will be the time to generate the speech recognition target dictionary.

As mentioned above, in the present invention, reducing the speech recognition target vocabulary is a key element, and the speech recognition success rate can be increased by further reducing the speech recognition target vocabulary. For example, the case where the user inputs the first consonants of each syllable constituting one word is described as follows.

That is, when there are vocabulary beginning with "h" to "ㅎ" in the pronunciation dictionary on the recognition DB, when the user wants to utter "cold", the consonant input as the letter becomes "ㄱ".

Accordingly, in the present invention, the vocabulary of two ("2") syllables in which the first consonant of each syllable is "a" is selected as a material for generating a speech recognition target dictionary from the pronunciation dictionary. For example, the primary speech recognition target vocabulary may be "furniture", "going", "cold", "meat", "air", "grilling", and the like.

As described above, in the state of selecting a primary speech recognition target vocabulary, unnecessary words of the primary speech recognition target vocabulary may be removed by using information of the word.

That is, when the word corresponding to the voice at the time of the user's speech corresponds to the predicate, the word "go" corresponding to the predicate except for "furniture", "cold", "meat", and "air" in the target speech recognition words Only "," grilling can be selected as the secondary speech recognition target vocabulary to generate the speech recognition target dictionary.

The above method may be used to rescan the recognition result as a second consonant when the user fails to recognize the spoken voice.

However, if the information of the word corresponding to the voice of the user's speech, for example, information related to the relationship on the sentence, is unclear, it is preferable to generate the speech recognition target dictionary using only the consonant information presented above.

Next, the flow of the speech recognition process of the present invention mentioned above with reference to FIG. 3 will be described.

When the consonant is input from the user and the user's spoken voice is simultaneously input (300), the consonant and the user's spoken voice are synchronized with each other (301).

Thereafter, only the vocabulary starting with the consonant with the multimodal synchronization is extracted from the pronunciation dictionary on the recognition DB (302), and the extracted vocabularies are bundled and generated as one speech recognition target dictionary (the speech recognition target vocabulary reduction) (303). ).

In the state where the speech recognition target dictionary is generated as described above, the user searches for the corresponding vocabulary from the generated speech recognition target dictionary for the user spoken speech with the multi-modal synchronization, and recognizes a word or sentence corresponding to the speech spoken by the user [substantially. Performing the speech recognition process] (304).

In addition, in the above-described process "300", a text / voice is input from the user. If a user's voice is not input within a certain time since a specific key signal is input through the key button, the uni-modal method, that is, the existing key button Process user input information as an input method.

Of course, in the above-described process "300", when only a user voice is input without a specific key signal being input through the key button, the normal voice recognition process described with reference to FIG. 2 is performed.

In the above-described embodiment of the present invention, the speech recognition of one word has been described as an example, but in order to further improve the speech recognition performance of one word, only the first consonants of each syllable forming one word are key buttons. A voice corresponding to a word composed of syllables starting from each of the consonants may be recognized by the user. That is, in this case, the vocabulary composed of syllables starting with each consonant is extracted from the phonetic dictionary and generated as a speech recognition target dictionary. However, in English, receiving each syllable constituting a single word does not mean that the word itself is eventually input. In this case, it is preferable to receive only the first alphabet of the word.

In addition, another embodiment of the present invention is a method of speech recognition for one sentence, and only the consonants on the first syllable of each word constituting each sentence may be recognized by the user through a key button. That is, in this case, words commenced with each consonant are combined, and the vocabulary corresponding to each word is extracted from the pronunciation dictionary and generated as a speech recognition target dictionary.

According to the present invention described above, since the user can input text and voice at the same time, there is no need to select a menu several times for text / voice input, which has the advantage that information can be conveniently input to the terminal.

On the other hand, in the application of the present invention to a portable terminal, due to the characteristics of the small size terminal can not be provided with key buttons corresponding to all consonants, vowels, it is possible to input several consonants, vowels in one key button universally Interface is implemented. For example, a celestial keyboard method, a naragle keyboard method, and the like can be given. In such a keyboard, "a", "ㅋ" and "ㄱ" are inputted as one key button.

Accordingly, the present invention also considers the keyboard structure of the portable terminal, in order to save the user from having to press the same key button several times in entering the first consonant of the word, each consonant corresponding to one key button on the terminal The starting vocabulary is set as the speech recognition target vocabulary.

For example, when the user utters "black", the user should input "ㅋ". However, in the present invention, even if "a" is input, the vocabulary beginning with "ㅋ" can be searched. This has the advantage of eliminating the hassle of having to press the "a" button twice consecutively in order to press "k" when "a" and "k" correspond to the same button.

As described above, the method of the present invention may be implemented as a program and stored in a recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto-optical disk, etc.) in a computer-readable form. Since this process can be easily implemented by those skilled in the art will not be described in more detail.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

The present invention as described above by performing the speech recognition based on the consonant and the user uttered speech with the multi-modal synchronization, there is an effect of ensuring a very high speech recognition success rate not only in a high-performance computing device but also a low-end portable terminal.

In addition, the present invention has the effect of accurately finding the starting point of the words to be recognized by the speech recognition using a multi-modal method, and accurately perform the speech recognition even for unvoiced initial speech difficult to recognize in Korean.

In addition, the present invention does not require the user to enter a number of key inputs, and even in an environment where the eyes and hands are not free, such as during driving or driving, the effect of enabling the user to accurately input the information without any inconvenience have.

Claims (10)

In the speech recognition method using multimodal, Synchronizing between the consonant and the user spoken voice when the user receives a consonant from the user and simultaneously receives the user spoken voice; Extracting vocabulary starting from the synchronized consonant from a pronunciation dictionary; Grouping the extracted vocabularies and generating them as a speech recognition target dictionary; And In the state of generating the voice recognition target dictionary, retrieving a corresponding vocabulary from the generated voice recognition target dictionary for the synchronized user spoken voice and recognizing a word or sentence corresponding to the user spoken voice Speech recognition method using a multi-modal comprising a. The method of claim 1, The consonant is a voice recognition method using multi-modal, characterized in that the first consonant of the first syllable of the word corresponding to the user spoken voice. The method of claim 1, In the case of receiving a consonant about a word from the user, the first consonant of each syllable constituting the single word is input from the user, And a vocabulary composed of syllables beginning with each of the input consonants from the phonetic dictionary to be generated as a speech recognition target dictionary. The method of claim 1, In the case of receiving a consonant about a sentence from the user, the consonants on the first syllable of each word constituting the single sentence are input from the user, Combining words starting with the input consonants, and extracting vocabulary corresponding to each word from a pronunciation dictionary to generate a speech recognition target dictionary; The method of claim 1, In the case where a consonant is input from a user through a key button, but at least one consonant is implemented in one key button, When a specific key button is selected by the user, the vocabulary starting with each of the consonants implemented in the key button is extracted from the phonetic dictionary. The method of claim 1, In the case of receiving the first consonants of each syllable forming a word from the user, as a process of generating the speech recognition target dictionary, Selecting vocabulary corresponding to the first consonant of each syllable input from the user as a speech recognition target vocabulary from a pronunciation dictionary; Removing words that do not correspond to the context information from the selected speech recognition target vocabulary based on the context information of the word input from the user; And Generating the speech recognition target vocabulary from which the specific vocabulary has been removed as a speech recognition target dictionary Speech recognition method using a multi-modal comprising a. In the speech recognition apparatus using multimodal, A text processing unit which recognizes a text signal input from a user, processes a consonant corresponding to the text as a text, and transmits the text to a multi-modal synchronization unit; A voice processor for recognizing a voice signal input from a user and processing the voice signal and transmitting the signal to a multi-modal synchronization unit; A multi-modal synchronization unit for synchronizing the consonant received from the text processing unit with the user spoken voice received from the voice processing unit; A recognition DB in which a pronunciation dictionary is stored; And After extracting the vocabulary starting with the synchronized consonants from the pronunciation dictionary, the extracted vocabularies are combined and generated as a speech recognition target dictionary, and then the corresponding speech is searched from the generated speech recognition target dictionary to synchronize the user. Speech recognition unit for recognizing words or sentences corresponding to voice Speech recognition apparatus using a multi-modal comprising a. The method of claim 7, wherein The speech recognition unit, The multi-modal synchronization unit performs a speech recognition process using the first consonant and the user's spoken voice of the synchronized words as parameters, and reduces only the vocabulary starting with the first consonant among the vocabulary in the pronunciation dictionary to the speech recognition target. Speech recognition device using modal. The method according to claim 7 or 8, And the text signal is input from a user through at least one of a key button, a pen tool, and a virtual keyboard. In a speech recognition device having a processor, Synchronizing between the consonant and the user spoken voice when the consonant is input from the user and the user spoken voice is received at the same time; Extracting vocabulary starting from the synchronized consonants from a pronunciation dictionary; A function of grouping the extracted vocabularies and generating them as a speech recognition target dictionary; And A function of recognizing a word or a sentence corresponding to the user spoken voice by searching the corresponding vocabulary from the generated speech recognition target dictionary for the synchronized user spoken voice while generating the voice recognition target dictionary. A computer-readable recording medium having recorded thereon a program for realizing this.

Images