KR101092489B1 - Speech recognition system and method - Google Patents

Abstract

The present invention relates to a speech recognition system and method, wherein the system generates a vision information by photographing a user, and whether the recognition word information is adopted according to the recognition word information and the vision information generated by recognizing the speech spoken by the user. It includes an image-based speech recognition device for determining the. According to the present invention, by using the vision information of the user and the speech spoken by the user together, the user's spoken word is recognized to reduce the probability of misrecognition, thereby improving the performance of the speech recognition system.

Speech Recognition, Vision Information, Shape, Lip Shape, Phoneme Group

Description

Speech Recognition System and Method {SPEECH RECOGNITION SYSTEM AND METHOD}

The present invention relates to a speech recognition system and method.

Rapidly developing robotic technology improves the quality of human life. Industrial robots using sophisticated control technology brought material abundance to human beings, which gradually developed into service robot technology, bringing convenience in daily life of humans through cleaning robots and guide robots. While technology for controlling speed and precise movement to increase productivity and synchronization technology are important issues for industrial robots, human-robot interaction (HRI) for service robots to provide appropriate services to humans Technology is an important issue.

The robot has a variety of sensors that can receive human instructions. Direct commands can be delivered using external input devices such as buttons, keyboards, and touch screens. However, it is difficult to obtain command extensibility when receiving a command from a human using only the input information of such hardware. Therefore, a technology for transmitting a command using voice information through a conversation with a person has been developed. In the case of using the voice information, various commands can be recognized and executed. However, since the voice signal is affected by environmental factors, there is a high probability of false recognition due to the deterioration of the voice signal due to voice noise.

The problem to be solved by the present invention is to provide a speech recognition system and method that can lower the false recognition rate for the speech spoken by the user.

According to an aspect of the present invention, there is provided a speech recognition system for generating vision information by photographing a user, and recognizing the speech according to the recognition word information and the vision information generated by recognizing speech spoken by the user. It includes an image-based speech recognition device for determining whether to adopt the word information.

The apparatus may further include a speech-based speech recognition apparatus configured to recognize the speech spoken by the user to generate the recognition word information and to provide the recognition speech information to the image-based speech recognition apparatus.

The image-based speech recognition apparatus may classify the vowels spoken by the user from the vision information and classify them by phoneme group.

The phoneme groups may be divided according to the similarity of the feature elements of each vowel.

The image-based speech recognition apparatus extracts first phoneme order information from the recognized word information, extracts second phoneme order information from the vision information, and extracts the phoneme order information from the recognized word information according to the first and second phoneme order information. Adopt a decision.

The image-based speech recognition apparatus analyzes an ASCII code of the recognized word information to extract a vowel for each syllable, determines a phoneme group corresponding to the extracted vowel, and determines the first phoneme order according to the determined phoneme group. Information can be generated.

The image-based speech recognition apparatus may calculate a value of a lip-shaped feature element based on the vision information, and generate the second phoneme order information according to a phoneme group corresponding to the calculated feature element value.

The feature element may include at least one of a lip width, a lip height, a ratio of the lip width and the lip height, and a lip area.

The image-based speech recognition apparatus may classify each phoneme group in a feature element space based on the feature element, and extract a phoneme group corresponding to the calculated feature element value using a maximum likelihood value.

The robot system according to another aspect of the present invention includes any of the above speech recognition systems.

A conversation system according to another aspect of the present invention includes any of the speech recognition systems described above.

According to another aspect of the present invention, there is provided a speech recognition method, comprising: photographing a user to generate vision information, and recognizing the speech word information generated by recognizing a speech signal uttered by the user and the recognition word information according to the vision information. Determining whether to adopt.

The method may further include generating the recognized word information by recognizing the voice spoken by the user.

The determining may include extracting the vowels spoken by the user from the vision information and classifying the phonemes into phoneme groups.

The determining may include extracting first phoneme order information from the recognized word information, extracting second phoneme order information from the vision information, and the recognition word information according to the first and second phoneme order information. Determining whether to adopt may include.

The extracting of the first phoneme order information may include extracting a vowel for each syllable by analyzing the ASCII code of the recognized word information, determining a phoneme group corresponding to the extracted vowel, and according to the determined phoneme group. Generating first phoneme order information.

The extracting of the second phoneme order information may include calculating a value of a lip feature element based on the vision information, and generating the second phoneme order information according to a phoneme group corresponding to the calculated feature element value. It may include the step.

The extracting of the second phoneme sequence information may further include extracting a phoneme group corresponding to the calculated feature element value using a maximum likelihood value by dividing each phoneme group in a feature element space based on the feature element. Can be.

A computer-readable medium according to an embodiment of the present invention records a program for causing a computer to execute any one of the above methods.

As described above, according to the present invention, by using the speech spoken by the user and the vision information obtained by photographing the user together, the speech word of the user is recognized, thereby reducing the probability of misrecognition, thereby improving the performance of the speech recognition system.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention.

First, a voice recognition system according to an exemplary embodiment of the present invention will be described in detail with reference to FIG. 1.

1 is a block diagram illustrating a speech recognition system according to an embodiment of the present invention.

The speech recognition system 100 according to the embodiment of the present invention includes a speech-based speech recognition apparatus 110 and an image-based speech recognition apparatus 130.

The voice-based speech recognition apparatus 110 receives the speech spoken by the user U through an input module (not shown) such as a microphone. The speech-based speech recognition apparatus 110 generates recognition word information based on the recognized words from the input speech and provides the same to the image-based speech recognition apparatus 130.

The speech-based speech recognition apparatus 110 estimates the similarity between the input speech and reference words in a pre-built and stored word database (not shown). Information such as pronunciation and frequency of the input voice (hereinafter referred to as 'voice information') becomes a major factor for determining similarity. Hidden Markov Models (HMM) may be used to model speech signals, but are not limited thereto. The Hidden Markov Model is one of the statistical language model techniques used to implement speech recognition. A word or syllable may be represented by a hidden Markov model learned with a lot of speech information in advance. Meanwhile, before modeling the speech signal, the speech information may be converted into the speech feature vector, and the speech signal may be modeled based on the speech information. As such, when the speech signal is modeled using the speech feature vector, speech recognition performance may be further improved.

The speech-based speech recognition apparatus 110 may estimate similarity by comparing the input speech signal with a word in a word database using a pruning algorithm, but is not limited thereto. By using this pruning algorithm, even if the word database is large, speech recognition can be performed with fewer operations.

The speech-based speech recognition apparatus 110 extracts a recognition word based on the estimated similarity and generates recognition word information based on the recognition word. In this case, the speech-based speech recognition apparatus 110 processes a speech recognition word representing a similarity below a predetermined level as a failure of recognition, and a speech recognition word representing a similarity above a predetermined level is regarded as a successful recognition to recognize the recognition word information. Create

The image-based speech recognition apparatus 130 adopts the recognition word information based on the recognition word information provided from the speech-based speech recognition apparatus 110 and vision information obtained by photographing the face portion of the user U who speaks the voice. Determine whether or not. When adopted, the image-based speech recognition apparatus 130 determines the recognized word as a valid word and provides the valid word information to an external device (not shown) such as a robot system or a conversation system. On the other hand, if not adopted, the image-based speech recognition apparatus 130 may provide a recognition failure message to the user U.

Meanwhile, although these devices 110 and 130 are illustrated in FIG. 1 as separate independent devices, the devices 110 and 130 may be implemented in one form according to an exemplary embodiment. In addition, although these devices 110 and 130 have been described as being included in the speech recognition system 100, they may be divided into individual devices and implemented independently.

Next, the image-based speech recognition apparatus 130 according to an exemplary embodiment of the present invention will be described in detail with reference to FIGS. 2 to 5.

FIG. 2 is a block diagram illustrating the image-based speech recognition apparatus of FIG. 1 in more detail. FIG. 3 is a view for explaining an operation of extracting first phoneme order information from recognized word information according to an exemplary embodiment of the present invention. 4 is a view for explaining an operation of extracting a lip part from vision information according to an embodiment of the present invention, and FIG. 5 is an operation for extracting second phoneme order information from vision information according to an embodiment of the present invention. It is a figure for demonstrating.

Referring to FIG. 2, the image-based speech recognition apparatus 130 includes an image photographing unit 131, a recognition word adoption determining unit 133, and a valid word providing unit 135.

The image capturing unit 131 includes a capturing module (not shown) such as a camera, and when a voice is uttered from the user U, a vision made of a video frame by capturing a face part of the user U through the capturing module. Generate information.

The recognition word adoption determiner 133 extracts a collection of syllables by analyzing an ASCII code of the recognition word information provided from the speech-based speech recognition apparatus 110. The recognition word adoption determining unit 133 determines the phoneme group for each syllable based on the extracted syllable-by-syllable group to generate first phoneme sequence information.

Hangul creates syllables using a combination of 19 consonants and 21 vowels. In order to sound each syllable, the user U pronounces the proper tongue position and mouth shape. Consonant pronunciation is mainly influenced by the location of the tongue and vowel pronunciation by mouth shape. In particular, the short vowels 'ㅓ', 'ㅓ', 'ㅗ', 'TT', 'ㅡ', 'ㅣ', 'ㅐ', 'ㅔ' and the consonant 'ㅁ' have representative lips. Thus, all vowel and lip sounds can be grouped into the same phoneme group around them.

Here, each phoneme group may be composed of a plurality of phonemes. Phonemes belonging to the same phoneme group have similar main features. Key features include the width, height, relative ratio of width to height, area, pixel values, and position inside or outside the lip shape. As shown in [Table 1], eight important vowels may be classified into four phoneme groups based on the similarity of the main feature elements, but are not limited thereto and may be classified into eight phoneme groups or phoneme groups of less than eight. Table 1 shows the important vowels belonging to the four phoneme groups and the values of the main feature elements of each phoneme group.

As an example in which the recognition word adoption determining unit 133 generates the first phoneme order information, as shown in FIG. 3, when the speech recognition word is 'Korea', an ASCII code of the recognition word information is analyzed to analyze the syllables of the recognition word. Star collections 'ㅐ', 'ㅏ', 'ㅣ', 'TT' are extracted and the syllable group is determined by the syllables. For example, the ASCII code '0xebb4' of the syllable 'large' is analyzed to extract the vowel 'ㅐ', and the phoneme group 'second phoneme group' including the extracted vowel 'ㅐ' is determined. If the phoneme group corresponding to each vowel is determined in the same manner, the first phoneme order information becomes the second-second-first-third-phoneme group.

2 and 4, the recognition word adoption determiner 133 extracts a main feature element of a lip shape based on the vision information generated by the image capturing unit 131. That is, the recognition word adoption determiner 133 finds the face position of the user U in each video frame of the vision information, extracts the mouth region, and selects it as a region of interest (ROI). It is usually found in the lower third of the face of the wearer. As shown in Fig. 4, a region of interest (ROI) is extracted from each video frame of vision information. Wherein (a), (b), (c), (d), (e), (f), (g) and (h) of Figure 4, the vowel 'ㅏ', 'ㅓ', 'ㅐ', It represents a video frame obtained when the user U pronounces ',', 'TT', 'ㅗ', 'ㅡ' and 'ㅣ'.

The recognition word adoption determiner 133 extracts a lip segment LS from the ROI using the lip color (RGB value) as a threshold. Normally, pixels having a color similar to the lip color exist sporadically in the region of interest (ROI). Therefore, in order to extract the lip region LS, a plurality of pixels are selected in advance, and each pixel of the ROI checks whether there is a selected pixel among adjacent pixels, and if there is a selected pixel, combines the selected pixel with one clustering group. To form. The clustering group expands the region by repeatedly combining with adjacent pixels. Of course, a combination may be formed between the clustering groups and merged into one clustering group. As a result, a plurality of clustered segments are generated, and among them, the clustering group located at the center of the ROI is selected as the lip region LS.

The recognition word adoption determining unit 133 extracts the main feature elements of the lip shape using the pixels of the lip area LS. Representative values of the main feature elements of each phoneme may be precomputed and stored in the phoneme database for reference. Since the information units between the main features may be different, the standardization of the feature information may be equalized through normalization.

The recognition word adoption determination unit 133 extracts a phoneme group for each syllable using the extracted main feature elements of the lip shape and the main feature element space, and generates second phoneme order information based on the syllable group.

As shown in FIG. 5, the main feature element space is a main feature element with axes A1, A2, and A3, and the axes A1, A2, and A3 are orthogonal to each other. The main feature element space is divided into first to fourth phoneme group areas PG1, PG2, PG3, and PG4 based on pre-calculated main feature elements for each phoneme group, and the phoneme group areas PG1, PG2, PG3, The boundary between PG4) can be known through the group boundary line BL.

That is, the recognition word adoption determining unit 133 calculates a maximum likelihood value with each phoneme group based on the extracted main feature elements, and selects a group having the greatest similarity based on the calculated maximum likelihood value. Select the phoneme group of the syllable.

The recognition word adoption determining unit 133 determines whether to adopt the recognition word based on the first and second phoneme order information. That is, the recognition word adoption determining unit 133 checks whether a syllable phoneme group based on the first phoneme order information and a syllable phoneme group based on the second phoneme order information match each other. The recognition word adoption determination unit 133 adopts the recognition word if the order matches, and rejects the recognition word if it does not match.

The valid word providing unit 135 determines the recognized word as a valid word when the recognized word is adopted by the recognized word adopting determination unit 133. The valid word providing unit 135 provides valid word information based on the valid word to an external device such as a robot system or a conversation system. On the other hand, the valid word providing unit 135 may provide a recognition failure message to the user (U) when the recognition word is not adopted.

Next, the results of testing the performance of the speech recognition system 100 according to the present invention will be described.

Voice and vision information according to an embodiment of the present invention by using a system (hereinafter, referred to as a 'comparison system') that recognizes a word spoken from a user U (hereinafter, referred to as a 'recognition target word') only through voice. Was tested to improve the performance of the speech recognition system 100 (hereinafter, referred to as 'the present system'). The comparison system used a commercial product 'Voiceware'.

In the case of recognizing a word using only a voice, the shorter the length of the word to be recognized, the more likely it is to be mistaken as a similar pronunciation word. Therefore, we used isolated words consisting of two or three syllables for the performance test accuracy of this system, and the target words were selected based on words with similar syllables that are easy to misunderstand.

The results of measuring the false recognition rate when the comparison system is used and the false recognition rate when the system is used, using about 100 words including important phonemes for each syllable as words to be recognized are shown in [Table 2].

As shown in [Table 2], the comparison system has misrecognition rates of 8.2% and 22.9% for words in distinguished word DB and similar word DB, respectively. It can be seen that the present system improves the false recognition rate over the comparison system.

So far, the speech recognition system 100 according to the embodiment of the present invention has been described with respect to recognizing a user's speech in Korean, but may be equally applicable to English, Japanese, Chinese, and the like. In addition, the voice recognition system 100 according to the embodiment of the present invention may be included in various systems such as a robot system, a dialogue system, and an education providing system.

Next, a voice recognition method according to an embodiment of the present invention will be described with reference to FIG. 6.

6 is a flowchart illustrating a speech recognition method according to an embodiment of the present invention.

First, the voice-based speech recognition apparatus 110 receives a spoken speech from the user U through an input module, and generates recognition word information based on words recognized from the input speech (S210). That is, the speech-based speech recognition apparatus 110 models a speech signal using a hidden Markov model and the like, estimates the similarity with words in a word database using a pruning algorithm or the like on the input speech signal, Recognition word information is generated based on the recognition word extracted based on the estimated similarity.

In operation S220, the image-based speech recognition apparatus 130 generates a vision information formed of a video frame by photographing a face portion of the user U who utters a voice through a photographing module.

Thereafter, the image-based speech recognition apparatus 130 extracts first phoneme order information from the recognized word information (S230). That is, the image-based speech recognition apparatus 130 extracts a collection of syllables by analyzing ASCII codes of recognized word information. The image-based speech recognition apparatus 130 generates first phoneme sequence information by determining a syllable group for each syllable based on the extracted syllable-by-syllable collection.

The image-based speech recognition apparatus 130 extracts second phoneme sequence information from vision information (S240). That is, the image-based speech recognition apparatus 130 finds the face position of the user U in each video frame of the vision information, extracts the mouth region, and selects it as the ROI. The image-based speech recognition apparatus 130 extracts the lip region LS from the ROI based on the lip color (RGB value) as a threshold. The image-based speech recognition apparatus 130 extracts main feature elements of the lip shape by using pixels of the lip area LS. The image-based speech recognition apparatus 130 extracts a phoneme group for each syllable using the extracted main feature elements and the main feature element space, and generates second phoneme order information based on the extracted syllable group. That is, the image-based speech recognition apparatus 130 calculates a maximum likelihood value with each phoneme group based on the extracted main feature elements, and selects a group having the greatest similarity based on the calculated maximum likelihood value. Select as a group.

The image-based speech recognition apparatus 130 determines whether to adopt recognition word information based on the first and second phoneme order information (S250). That is, the image-based speech recognition apparatus 130 checks whether a syllable phoneme group based on the first phoneme order information and a syllable phoneme group based on the second phoneme order information match each other. The video-based speech recognition apparatus 130 adopts a recognized word if the order matches, and rejects the recognized word if the order does not match.

When the recognition word is adopted, the image-based speech recognition apparatus 130 determines the recognition word as a valid word, and provides valid word information based on the valid word to an external device such as a robot system or a conversation system (S260).

On the other hand, it has been described that step S220 is performed after step S210, but not limited thereto, and these steps S210 and S220 may be simultaneously performed according to an embodiment, and step S220 may be performed in step S210. It may be performed first. In addition, it has been described that the step S240 is performed after the step S230, but the present invention is not limited thereto, and according to an exemplary embodiment, the steps S230 and S240 may be simultaneously performed, and the step S240 may be performed before the step S230. It may also be performed.

Embodiments of the invention include a computer readable medium containing program instructions for performing various computer-implemented operations. This medium records a program for executing the speech recognition method described so far. The media may include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of such media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CDs and DVDs, floppy disks and program commands such as magnetic-optical media, ROM, RAM and flash memory. Hardware devices configured to store and perform such operations. Alternatively, the medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

Although the preferred embodiments of the present invention have been described in detail above, the scope of the present invention is not limited thereto, and various modifications and improvements of those skilled in the art using the basic concepts of the present invention defined in the following claims are also provided. It belongs to the scope of rights.

1 is a block diagram illustrating a speech recognition system according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating the image-based speech recognition apparatus of FIG. 1 in more detail.

3 is a view for explaining an operation of extracting first phoneme order information from recognized word information according to an exemplary embodiment of the present invention.

4 is a diagram for describing an operation of extracting a lip part from vision information according to an exemplary embodiment of the present invention.

5 is a diagram for describing an operation of extracting second phoneme order information from vision information according to an exemplary embodiment of the present invention.

6 is a flowchart illustrating a speech recognition method according to an embodiment of the present invention.

<Explanation of symbols for the main parts of the drawings>

100: speech recognition system, 110: speech-based speech recognition device,

130: an image-based speech recognition device, 131: an image capturing unit,

133: recognition word adoption determination unit, 135: valid word provider

Claims (21)

An image-based speech recognition apparatus for generating vision information by photographing a user and determining whether to recognize the recognition word information according to the recognition word information and the vision information generated by recognizing the speech spoken by the user. Including, The image-based speech recognition apparatus extracts first phoneme order information from the recognized word information, extracts second phoneme order information from the vision information, and extracts the phoneme order information from the recognized word information according to the first and second phoneme order information. Decide whether to accept, The image-based speech recognition apparatus analyzes an ASCII code of the recognized word information to extract a vowel for each syllable, determines a phoneme group corresponding to the extracted vowel, and determines the phoneme order information according to the determined phoneme group. To generate Speech recognition system. In claim 1, And a speech-based speech recognition apparatus for generating the recognition word information by recognizing the speech spoken by the user and providing the recognition word information to the image-based speech recognition apparatus. In claim 1, And the image-based speech recognition apparatus extracts the vowels spoken by the user from the vision information and classifies them by phoneme group. 4. The method of claim 3, The phoneme group is divided according to the similarity of the feature elements of each vowel. delete delete In claim 1, The image-based speech recognition apparatus is configured to calculate a value of a lip-shaped feature element based on the vision information, and generate the second phoneme sequence information according to a phoneme group corresponding to the calculated feature element value. . 8. The method of claim 7, The feature element comprises at least one of lip width, lip height, ratio of lip width to lip height, and lip area. 8. The method of claim 7, And a phoneme group corresponding to the calculated feature element value using a maximum likelihood value by dividing each phoneme group in a feature element space based on the feature element. 10. A robotic system comprising the speech recognition system of any of claims 1-4 and 7-9. 10. A conversation system comprising the speech recognition system of any of claims 1-4 and 7-9. Photographing the user to generate vision information, and Determining whether to adopt the recognition word information according to the recognition word information and the vision information generated by recognizing the speech signal spoken by the user. Including, The determining step, Extracting first phoneme order information from the recognized word information; Extracting second phoneme order information from the vision information, and Determining whether to adopt the recognized word information according to the first and second phoneme order information; The first phoneme order information extraction step, Extracting a vowel per syllable by analyzing the ASCII code of the recognized word information; Determining a phoneme group corresponding to the extracted vowels, and Generating the first phoneme order information according to the determined phoneme group. Speech recognition method. delete The method of claim 12, The determining may include extracting a vowel spoken by the user from the vision information and classifying the vowels by phoneme group. The method of claim 14, The phoneme group is divided according to the similarity of the feature elements of each vowel. delete delete The method of claim 12, The second phoneme order information extraction step, Calculating a value of a lip feature element based on the vision information, and Generating the second phoneme order information according to a phoneme group corresponding to the calculated feature element value. The method of claim 18, The feature element comprises at least one of a lip width, a lip height, a ratio of the lip width to the lip height, and a lip area. The method of claim 18, The extracting of the second phoneme order information further comprises: dividing each phoneme group into a feature element space based on the feature element and extracting a phoneme group corresponding to the calculated feature element value using a maximum likelihood value. Speech recognition method. A computer readable medium having recorded thereon a program for causing a computer to execute the method of any one of claims 12, 14, 15, and 18-20.

Images