KR100873920B1 - Speech Recognition Method and Device Using Image Analysis - Google Patents

Abstract

The present invention comprises the steps of receiving a sound and a speaker's speech screen from the outside, analyzing the received sound, selecting at least one moving zone of the speaker's speech screen, at least one selected moving Selecting at least one vocalization screen candidate zone corresponding to the zone, determining a vocalization screen zone among at least one utterance screen candidate zone according to a preset reference screen, and voice analysis of the sound analyzed according to the utterance screen zone The method may include determining a signal and performing voice recognition, wherein the preset reference screen is a screen including a nose portion.

Description

Method and apparatus for speech recognition using image analysis {Method and Apparatus for voice recognition using image analysis}

1 is a schematic illustration to which the present invention is applied.

2 is a conceptual diagram illustrating a method of recognizing a speech in a speech recognition apparatus according to an exemplary embodiment of the present invention.

3 is a schematic flowchart of recognizing speech according to a preferred embodiment of the present invention.

4 is a flowchart illustrating a procedure of analyzing a speech screen for speech recognition according to an exemplary embodiment of the present invention.

5 is an exemplary view of analysis of a vocalization screen for speech recognition through lip movement compared with the present invention.

6 is an exemplary view illustrating analysis of a speech screen for speech recognition through movement of a nose according to an embodiment of the present invention.

7 is an overall flowchart of a speech recognition method using image analysis according to an exemplary embodiment of the present invention.

<Explanation of symbols for main parts of the drawings>

101: speech recognition device

103: voice input device

105: screen input device

107: service device

The present invention relates to a method and apparatus for speech recognition using image analysis.

With the rapid development of information technology, convergence between unit technologies is progressing rapidly. As various information devices including PCs are being widely distributed, it is necessary to utilize mutually necessary information between them. In particular, PC video cameras are becoming a necessity for long-distance video communications, making it easy to utilize video information.

On the other hand, one of the most problematic parts in the speech recognition field is acoustic noise collected with voice. This acoustic noise varies from the driving noise of the device itself to the network noise and the environmental noise. In particular, the static noise having a constant size and frequency is mostly removed, but the dynamic noise having a constant size and frequency is a conventional voice. It is difficult to remove the voice recognition method using only.

This dynamic noise is particularly problematic in continuous speech recognition.

An object of the present invention is to provide a speech recognition method and apparatus using image analysis.

Another object of the present invention is to provide a voice recognition method and apparatus capable of removing unnecessary voice noise by using screen information including a nose, among screen information during speech.

In order to achieve the above objects, according to one aspect of the present invention, a). Receiving a sound and a speech screen of a speaker from the outside, b). Analyzing the received sound, c). Selecting at least one moving area of the voice screen of the received speaker; d). Selecting at least one spoken screen candidate zone corresponding to the at least one selected moving zone, e). Determining a utterance screen zone among the at least one utterance screen candidate zone corresponding to a preset reference screen; and f). And performing voice recognition by determining the analyzed sound of step b) as a voice signal corresponding to the speech screen region, wherein the preset reference screen is a screen including a nose part. And a speech recognition method combined with the present invention.

In a preferred embodiment, the received sound may be divided into a voice signal section and a noise signal section. The voice screen candidate zone may be selected from above the selected moving zone. The speech screen region may be characterized by selecting a region having the highest matching rate among the reference screen and the speech screen candidate region.

Referring to another aspect of the present invention, a). Input unit for receiving the sound and the speech screen of the speaker (Speaker) from the outside, b). An acoustic analyzer for analyzing the received sound; c). An image selection unit for selecting at least one moving area of the received talker's speech screen; d). A speech screen candidate zone selection unit for selecting at least one speech screen candidate zone corresponding to the at least one selected moving zone, e). A speech screen region selection unit for determining a speech screen region among the at least one speech screen candidate region corresponding to a preset reference screen; and f). And a voice recognizer configured to perform voice recognition by determining the sound analyzed by the sound analyzer as a voice signal corresponding to the voiced screen area, wherein the preset reference screen is a screen including a nose part. It is possible to provide a speech recognition device combined with the analysis.

In a preferred embodiment, the received sound may be divided into a voice signal section and a noise signal section. The voice screen candidate zone may be selected from above the selected moving zone. In addition, the speech screen region may be characterized by selecting the region having the highest matching rate among the reference screen and the speech screen candidate region.

According to another aspect of the present invention, a computer-readable recording medium that records a program for speech recognition combined with image analysis, the program receiving sound and a speech screen of a speaker from the outside; Analyze the received sound, select at least one moving area of the spoken screen of the received speaker, select at least one speech screen candidate area corresponding to the at least one selected moving area, and preset criteria The speech screen region is determined among the at least one speech screen candidate region corresponding to the screen, and the speech recognition is performed by determining the analyzed received sound as a speech signal corresponding to the speech screen region. A computer-readable recording medium characterized by a screen including a nose portion Can be provided.

Next, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a schematic illustration to which the present invention is applied.

Referring to FIG. 1, first, the speaker 100 transmits a voice to a voice input device 103 such as a microphone in order to recognize and process a voice pronounced by the speaker.

In this case, since the voice input device 103 continues to receive sound, the voice input device 103 collects not only the voice of the speaker 100 but also the noise generated in the surrounding environment.

Meanwhile, while the speaker 100 inputs a voice, the screen input device 105 such as an image camera captures the voice screen of the speaker 100. The screen shot as described above is used by the voice recognition apparatus 101 to clearly recognize the voice of the speaker 100.

The sound signal received by the voice input device 103 and the voice screen of the speaker 100 photographed by the screen input device 105 are all screened by the voice recognition device 101 to detect the voice of the speaker 100. Will be recognized.

Then, the service device 107 may provide various services using the voice signal recognized by the voice recognition device 101.

2 is a conceptual diagram illustrating a method of recognizing a speech in a speech recognition apparatus according to an exemplary embodiment of the present invention.

Referring to FIG. 2, a sound signal input to a voice input device is illustrated. In this acoustic signal graph, the graph of reference numeral 201 represents the voice intended by the speaker, and the graph of reference numeral 203 represents the ambient noise.

When the sound including the noise section is input to the voice recognition apparatus 220 without the screen input device, the noise signal 203 is mistaken for the voice signal 201 and recognized as the wrong voice 205.

For example, assuming that the voice signal 201 is "stop", "mute", "start", and the noise signal 203 occurs in the "mute" period, the speech recognition device 220 is "stop", " Start "," start ". Because the speech database stored in the speech recognition apparatus 220 is limited, the speech signal that is determined to be the most similar to the noise signal 203 among the databases existing in the speech recognition apparatus 220 when the noise signal 203 is recognized. Because it is recognized as.

This can have a big impact on actual operation, which can lead to very large problems caused by noise.

On the other hand, if there is an image input device, the speaker's speech screen 213 may be analyzed to determine whether the speaker is actually speaking. Therefore, the noise signal 203 may be analyzed like the voice signal using only the voice signal, but the screen 213 indicates that the speaker does not speak in the noise signal 203 section by analyzing the image.

Therefore, even in this case, the voice recognition apparatus 230 having the screen input device recognizes only the signal intended by the speaker (215).

Therefore, assuming that the voice signal 201 is "stop", "mute", "start", and if the noise signal 203 occurs in the "mute" period, the voice recognition device 230 according to the present invention is the voice of the speaker The sound generated in the 'mute' section by analyzing the screen 213 may not be recognized as a voice, and thus may be accurately recognized as "stop", "mute", "start".

3 is a schematic flowchart of recognizing a voice according to an exemplary embodiment of the present invention.

Referring to FIG. 3, first, a sound signal is received from a voice input device (step 301). The received sound signal may be a voice signal or a noise signal. Since the voice input device does not operate only when the speaker speaks the voice, a noise signal is inevitably added before or after the speaker speaks the voice.

Thereafter, the sound signal input from the voice input device is analyzed (step 303). The analysis of the speaker voice directional signal is a step of analyzing the acoustic signal in order to use a conventional speech recognition method. Therefore, the details of analyzing the acoustic signal will not be described in detail.

When the acoustic signal is analyzed as described above, it is checked whether the speaker is actually uttered by comparing recognition information of the speaker (step 311) obtained by analyzing the voice screen of the speaker inputted from the screen input device with the extracted speaker voice candidate (step 311). 307). At this time, if the speaker is actually speaking, the analyzed sound signal is recognized as the voice intended by the speaker (step 309) and voice recognition is performed. Otherwise, if the talker has not actually spoken, the analyzed section is a noise section, and therefore, the process returns to analyzing the sound again (step 303) without proceeding with the speech recognition and performing the voice analysis again.

4 is a flowchart illustrating a procedure of analyzing a speech screen for speech recognition according to an exemplary embodiment of the present invention.

Referring to FIG. 4, first, a voice input image of a speaker is received by the screen input device (step 401). Thereafter, the speech image is analyzed to select a moving portion of the speech image, for example, an eye or a lip portion (step 403). The vocalization screen candidate region is determined by comparing the movement characteristics of the previously analyzed lips with the selected moving portion (step 405). This is to select the vocalization screen area compared to the reference screen. However, since the movement of the lips is so diverse that the calculation processing capacity for registration is excessively increased, in the present invention, the reference screen for registration is determined to be a portion of the nose where contrast is clearly distinguished with little movement.

Thereafter, the reference screen to be compared with the vocalization screen candidate zone is called (step 413), and compared with the called reference screen and the vocalization screen candidate zone to determine whether there is a part that matches well with the reference screen (step 407), and a certain level of If there is a portion that satisfies the matching ratio above the threshold value, this portion is determined as the speech screen region (step 409).

In the present invention, since the reference screen is selected as part of the nose, the screen compared to the reference screen will be the upper portion of the vocalization screen candidate area.

Thereafter, the movement of the utterance screen candidate region is analyzed to determine whether the actual speaker is uttered (step 411).

5 is an exemplary diagram of analysis of a speech screen for speech recognition through lip movement compared to the present invention.

Referring to FIG. 5, first, an entire face of a utterance utterance screen 510 input through a screen input device enters. Then, the voice recognition apparatus first selects a portion in which the movement occurs in the captured utterance screen 510 (520). In general, the large moving parts of the vocalization screen are the areas of both eyes 521 and 523 and the lips 525. Thereafter, each of the selected portions becomes the speech screen candidate region, and the speech screen candidate region is determined by comparing with the preset feature of the lips screen 530.

In this case, there is a disadvantage that the movement of the lips is very diverse when speaking, so that the movement screen of the lips to be set on the reference screen should be very large according to each movement. In order to overcome this disadvantage, the present invention can determine the speech screen in the manner of FIG. 6.

6 is an exemplary diagram illustrating analysis of a speech screen for speech recognition through movement of a nose according to an exemplary embodiment of the present invention.

Referring to FIG. 6, the entire face of the talker's utterance screen 610 input through the screen input device enters.

Thereafter, the voice recognition apparatus according to the present invention first selects a portion in which the movement occurs in the photographed utterance screen 610 (620). This selection will be the eye area and the lip area as the selection in FIG. 5. However, thereafter, the upper part of the selected moving part is selected as the vocal screen candidate region to be compared with the reference screen (630). Accordingly, the upper portions 631 and 633 of the eye movement region and the upper portion 635 of the lip movement region are selected for comparison. That is, the reference numeral 635 includes the nose portion of the speaker's face.

 Thereafter, each of the selected portions becomes a speech screen candidate region, and determines a speech screen region that actually determines movement by comparing with a preset screen 640 of the nose.

This is because the contrast of the nose screen is clear and there is almost no movement, so it is easier to calculate than the direct movement of the lips.

7 is an overall flowchart of a speech recognition method using image analysis according to an exemplary embodiment of the present invention.

Referring to FIG. 7, first, a sound signal is received from a voice input device (step 701). Thereafter, the sound signal input from the voice input device is analyzed (step 703).

After analyzing the acoustic signal, the speaker is actually compared with the extracted voice speech candidate (step 721) obtained by analyzing the voice screen of the speaker inputted from the screen input device to determine whether the analyzed signal is recognized. It is checked whether the voice is spoken (step 707). At this time, if the speaker is actually speaking, the speaker voice candidate is recognized as the voice intended by the speaker (step 709), and if not, the process returns to analyzing the sound (step 703) and analyzes the sound again.

On the other hand, while receiving a sound signal from the voice input device, the screen input device receives a voice image of the speaker (step 711). Thereafter, the vocal image is analyzed to select a moving portion of the vocal image, for example, an eye or a lip portion (step 713). Then, a predetermined upper portion of the selected moving parts is determined as the speech screen candidate area (step 715).

Then, the reference screen to be compared with the utterance screen candidate zone is called (step 723), and compared with the called reference screen and the vocalization screen candidate zone to determine whether there is a part that matches well with the reference screen (step 717), and a constant matching rate If there is a matching part as above, the part is determined as a speech screen area (step 719). Thereafter, the movement of the vocalization screen region is analyzed to determine whether the speaker is talking or not (step 721). The voice determined in this way is used to determine whether to recognize the sound analyzed in step 703.

The present invention is not limited to the above embodiments, and many variations are possible by those skilled in the art within the spirit of the present invention.

According to the present invention, a voice recognition method and apparatus using image analysis can be provided.

In addition, according to the present invention, it is possible to provide a voice recognition method and apparatus capable of removing unnecessary voice noise by using screen information including a nose, among screen information during speech.

Claims (9)

a). Receiving a sound and a speech screen of a speaker from the outside; b). Analyzing the received sound; c). Selecting at least one moving area of the voice screen of the received speaker; d). Selecting at least one spoken screen candidate zone corresponding to the at least one selected moving zone; e). Determining a utterance screen zone among the at least one utterance screen candidate zone corresponding to a preset reference screen; And f). Determining the analyzed sound of the step b) as a voice signal corresponding to the speech screen region to perform voice recognition, The preset reference screen is a screen including a nose portion, and the vocalization screen zone is selected from the reference screen and the vocalization screen candidate zone having the highest matching ratio. Speech recognition method combined with image analysis, characterized in that. The method of claim 1, The received sound is divided into a voice signal section and a noise signal section Speech recognition method combined with image analysis, characterized in that. The method of claim 1, Selecting the vocalization screen candidate zone above the selected moving zone. Speech recognition method combined with image analysis, characterized in that. delete a). An input unit configured to receive a sound and a utterance screen of a speaker from the outside; b). An acoustic analyzer analyzing the received sound; c). An image selecting unit which selects at least one moving area of the received speech screen of the speaker; d). A utterance screen candidate zone selection unit that selects at least one utterance screen candidate zone corresponding to the at least one selected moving zone; e). A utterance screen zone selection unit that determines a utterance screen zone among the at least one utterance screen candidate zone corresponding to a preset reference screen; And f). And a voice recognition unit configured to perform voice recognition by determining the sound analyzed by the sound analyzer as a voice signal corresponding to the speech screen region. Wherein the preset reference screen is a screen including a nose portion and the vocalization screen zone selects a region having the highest matching ratio between the reference screen and the vocalization screen candidate zone. Speech recognition device combined with image analysis, characterized in that. The method of claim 5, The received sound is divided into a voice signal section and a noise signal section Speech recognition device combined with image analysis, characterized in that. The method of claim 5, Selecting the vocalization screen candidate zone above the selected moving zone. Speech recognition device combined with image analysis, characterized in that. delete A computer-readable recording medium having recorded thereon a program for speech recognition combined with image analysis, The program,  Receive sound and a speaker's speech screen from the outside, analyze the received sound, select at least one moving zone of the received speaker's speech screen, and correspond to the at least one selected moving zone Select at least one utterance screen candidate zone, determine a utterance screen zone among the at least one utterance screen candidate zone according to a preset reference screen, and use the analyzed received sound as a voice signal according to the utterance screen zone; And perform voice recognition, wherein the preset reference screen is a screen including a nose portion and the speech screen region selects the region having the highest matching ratio between the reference screen and the speech screen candidate region. Readable recording medium.

Images