JP3798530B2 - Speech recognition apparatus and speech recognition method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a voice recognition apparatus that performs voice recognition using a video signal including a lip of a speaker and a voice signal, and a voice recognition method thereof, and particularly, to improve a recognition rate.
[0002]
[Prior art]
When performing speech recognition, a speech recognition device that uses not only speech signals but also images including the lips of the speaker is an “An Isolated Word Speech Recognition Using Fusion of Auditory and Visual Information” (Sintani et al.) IEICE Trans. Fundamentals, Vol. E79-A, No. 6, p777-783 (1996)). In speech recognition using only a speech signal, if noise is mixed, the recognition accuracy is drastically reduced. However, when the image of the lips is used together, the degree of reduction in recognition accuracy can be reduced.
[0003]
FIG. 6 shows a schematic configuration of this conventional speech recognition apparatus. This apparatus includes a video input unit 1 such as a video camera that inputs an image including a lip portion of a speaker, an audio input unit 3 such as a microphone that inputs sound uttered by the speaker, and a lip portion that utters various words. A video processing unit 2 for calculating the similarity between the video standard data and the input lip image, and outputting the similarity to each word included in the video standard data, and the audio standard data of various words and the input voice Obtaining the similarity and calculating the word having the highest similarity from the similarity input from the audio processing unit 4 that outputs the similarity to each word included in the audio standard data and the video processing unit 2 and the audio processing unit 4; And a voice recognition unit 5 for outputting the result as a recognition result.
[0004]
The video processing unit 2 of this apparatus extracts, for example, the vertical and horizontal lengths of the lip portion and the ratio of the vertical and horizontal lengths as feature amounts from the input video. The similarity (R) between the feature quantity corresponding to the i-th word and the feature quantity extracted from the input video among the video standard data of a plurality of words prepared in advance for calculating the similarity._i,_Image) Is calculated and output by a hidden Markov model (hereinafter abbreviated as HMM), which is well known as a pattern recognition technique.
[0005]
Also, the speech processing unit 4 extracts feature quantities from the input speech by cepstrum analysis, and extracts from the feature quantities corresponding to the i-th word and the input speech among the speech standard data of a plurality of words prepared in advance. Similarity with feature value (R_i,_Sound) Is calculated by the HMM and output.
[0006]
In addition, the voice recognition unit 5 outputs (R_i,_Image) And the output of the voice processing unit 4 (R_i,_Sound) From the similarity (R_i,_Total) Is calculated by the following equation (1).
R_i,_Total= Α ・ R_i,_Image+ (1-α) · R_i,_Sound    ......... (Formula 1)
Here, α (0 ≦ α ≦ 1) is a coefficient set in advance so that the recognition rate is maximized by using sampled video and audio data for coefficient determination (apart from similarity calculation). is there.
[0007]
The voice recognition unit 5 performs the similarity R for all words included in the video standard data and the voice standard data._i,_TotalThis similarity R_i,_TotalThe word with the maximum is output as the recognition result.
[0008]
As described above, since the voice recognition device uses the video signal including the lip information together with the voice signal, even when noise is present, the voice recognition device can avoid the sudden decrease in the recognition rate. Can be extended to devices used in noisy environments.
[0009]
[Problems to be solved by the invention]
It has been studied to apply a voice recognition device to a car navigation device and give a command to the device by voice. However, in order to incorporate it in a device used in such a noise environment, the noise of the voice recognition device is considered. It is necessary to further increase the recognition rate below.
[0010]
The present invention responds to such a demand, and it is an object of the present invention to provide a speech recognition apparatus capable of realizing a high recognition rate in speech recognition under a noisy environment and to provide a speech recognition method thereof. .
[0011]
[Means for Solving the Problems]
  Therefore, in the speech recognition apparatus of the present invention, the video data of the speaker including the lipsTheVideo input means to input and speaker's voice dataTheVideo processing that calculates the similarity between the input audio data and the standard video data of the lips that utter each single syllable and the input video data, and outputs each single syllable of the standard video data and the similarity to it Sound processing means for calculating the similarity between the voice standard data of the voice uttering each single syllable and the input voice data, and outputting each single syllable of the voice standard data and the similarity thereto, and video A speech recognition means for identifying a single syllable having the largest overall similarity using the similarity output from the processing means and the speech processing means;The video data relating to various single syllables is input to the video input means in advance, and the single syllable of the video standard data having the maximum similarity output from the video processing means corresponding to each input video data is selected as a candidate single syllable. The ratio of the number of candidate single syllables that match the single syllable of the input video data in the total number of identical candidate single syllables output from the video processing means calculated based on As data of correct answer rate for the single syllable of the dataMeans for holding the correct rate of video standard data to be held;The voice data of various single syllables is input to the voice input means in advance, and the single syllable of the voice standard data having the maximum similarity output from the voice processing means corresponding to each input voice data is set as a candidate single syllable. The ratio of the number of candidate single syllables that match the single syllable of the input voice data out of the total number of identical candidate single syllables output from the voice processing means calculated and calculated based on the voice standard data As the correct answer rate data for that single syllableVoice standard data correct answer rate holding means to hold, and voice recognition means is video processing meansFromOutput similarityWhenVideo standard data correct answer rate retention meansFromRead correct answer rate andAnd the sum of products of the similarity output from the voice processing means and the correct answer rate read from the voice standard data correct answer rate holding meansOverall similarity to each syllableAsI want to ask.
[0012]
In this device, when the single syllable to be identified is a type of single syllable that can be identified with high accuracy when identified based on the image of the lips, the identification result by the image greatly contributes to the final determination, When the single syllable to be identified is a single syllable that can be identified with higher accuracy when identified based on speech, the identification result by speech greatly contributes to the final determination. Therefore, highly reliable voice recognition is possible even in a noisy environment.
[0015]
DETAILED DESCRIPTION OF THE INVENTION
  According to the first aspect of the present invention, video data of a speaker including a lip is provided.TheVideo input means to input and speaker's voice dataTheVideo processing that calculates the similarity between the input audio data and the standard video data of the lips that utter each single syllable and the input video data, and outputs each single syllable of the standard video data and the similarity to it Sound processing means for calculating the similarity between the voice standard data of the voice uttering each single syllable and the input voice data, and outputting each single syllable of the voice standard data and the similarity thereto,The video data relating to various single syllables is input to the video input means in advance, and the single syllable of the video standard data having the maximum similarity output from the video processing means corresponding to each input video data is selected as a candidate single syllable. The ratio of the number of candidate single syllables that match the single syllable of the input video data in the total number of identical candidate single syllables output from the video processing means calculated based on As data of correct answer rate for the single syllable of the dataMeans for holding the correct rate of video standard data to be held;The voice data of various single syllables is input to the voice input means in advance, and the single syllable of the voice standard data having the maximum similarity output from the voice processing means corresponding to each input voice data is set as a candidate single syllable. The ratio of the number of candidate single syllables that match the single syllable of the input voice data out of the total number of identical candidate single syllables output from the voice processing means calculated and calculated based on the voice standard data As the correct answer rate data for that single syllableAudio standard data correct answer rate holding means to hold, and the video processing meansFromOutput similarityWhenSaid video standard data correct answer rate holding meansFromRead correct answer rate andAnd the sum of products of the similarity output from the speech processing means and the correct answer rate read from the speech standard data correct answer rate holding means.Overall similarity to each syllableAsThe speech recognition device is provided with speech recognition means for identifying the single syllable having the highest overall similarity, and the single syllable to be identified is more accurately identified based on the shape and movement of the lips. In the case of a single syllable that can be identified, the contribution rate of the identification by video is increased, and the single syllable to be identified is a single syllable that can be identified with higher accuracy when identified based on speech. By increasing the contribution rate of discrimination by voice, it is possible to perform voice recognition with high reliability even in a noisy environment.
[0020]
  Claim2The voice standard data correct answer rate holding means holds a plurality of types of correct answer rate data corresponding to the signal-to-noise ratio as correct answer rate data, and the voice recognition means is comprehensive for each single syllable. When obtaining a high degree of similarity, the correct answer rate data corresponding to the signal-to-noise ratio of the input voice data is read from the voice standard data correct answer rate holding means, and the reliability of the identification using the voice signal However, the point which fluctuates by the signal-to-noise ratio of the input voice signal can be improved.
[0024]
  Claim3The video data of the speaker including the lipsTheVideo input means to input and speaker's voice dataTheVoice input means for input;Single syllables with common utterance mechanisms are grouped in advance, and each groupStandard video data for lips that speak single syllablesWhenCalculate the similarity between the input video data andThe maximum similarity among them is set as the similarity of the group, and the group identification information and theVideo processing means for outputting the similarity of the group;
  For each group,Voice standard data for voices uttering each single syllableWhenCalculate the similarity between the input audio data andThe maximum similarity among them is set as the similarity of the group, and the group identification information and theVoice processing means for outputting the similarity of the group;In advance, the video data relating to various single syllables is input to the video input means, and the total number of outputs of each group output from the video processing means corresponding to each input video data is calculated and calculated based thereon. Of the total number of outputs, the ratio of the number of correct outputs in which the input single syllable is included in the group is used as data of the correct answer rate for the group.Means for holding the correct rate of video standard data to be held;Preliminarily input the voice data of various single syllables to the voice input means, totaled the total number of outputs of each group output from the voice processing means corresponding to each input voice data, calculated based on it, Of the total number of outputs, the ratio of the number of correct outputs in which the input single syllable is included in the group is used as the correct answer rate data for the group.Audio standard data correct answer rate holding means to hold, and the video processing meansFromOutput similarityWhenSaid video standard data correct answer rate holding meansFromRead correct answer rate andAnd the sum of products of the similarity output from the speech processing means and the correct answer rate read from the speech standard data correct answer rate holding means.Overall similarity for each groupAsAnd a voice recognition device including a voice recognition unit that identifies a group having the largest overall similarity, and can accurately identify a group to which a voice to be recognized belongs, thereby efficiently performing voice identification processing. Can be
[0038]
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0039]
(First embodiment)
As shown in FIG. 1, the speech recognition apparatus according to the first embodiment includes a video input unit 1 such as a video camera that inputs a video including a speaker's lip, a microphone that inputs a voice uttered by the speaker, and the like. The video input unit 3 and the video processing for obtaining the similarity between the video standard data of the lip part that utters various single syllables and the video of the input lip part, and outputting the similarity to each single syllable included in the video standard data Based on the image of the lip portion, the speech processing unit 4 that obtains the similarity between the unit 2, the speech standard data of various single syllables and the input speech, and outputs the similarity to each single syllable included in the speech standard data A video processing unit correct answer rate data holding unit 6 that holds correct answer rate data of each single syllable that has been voice-recognized (that is, data representing a probability that the single syllable is correct), and each single syllable that has been voice-recognized based on the voice. Preserve syllable accuracy data The voice processing unit correct answer rate data holding unit 7, the similarity input from the video processing unit 2 and the audio processing unit 4, and the video processing unit correct answer rate data holding unit 6 and the voice processing unit correct answer rate data holding unit 7 are read out. A speech recognition unit 5 that obtains a total similarity based on the correct answer rate data and outputs a single syllable having the highest similarity as a recognition result.
[0040]
FIG. 2 is a drawing for explaining the correct answer rate in the present invention. This figure shows, for example, when a single syllable is input to the speech processing unit 4 (here, for the sake of simplicity, the input single syllable is set to 5 of “A”, “I”, “U”, “E”, “O”. The single syllable is output as a “single syllable with the highest degree of similarity” (this is called a candidate single syllable) by the speech processing unit 4.
[0041]
This candidate single syllable appearance frequency data is prepared as a sample of single syllable speech separately from the standard speech data for calculating the similarity, and is input to the speech processing unit 4 to obtain the actual value. Seeking.
[0042]
For example, when a single syllable “a” is input 100 times to the speech processing unit 4, “a” is a candidate single syllable 97 times, and “u” is a candidate single syllable once, There was one case where “o” was a candidate single syllable (the candidate single syllable could not be specified for the remaining one).
[0043]
From the frequency of appearance of candidate single syllables obtained in this way, when a specific single syllable is output, the ratio that the single syllable is correct, that is, the number of outputs of the single syllable and the number of outputs The ratio of the number of outputs that match a single syllable is calculated as the correct answer rate.
[0044]
For example, with respect to the number 105 in which the speech processing unit outputs “a”, the number of input single syllables “a” is 97. Therefore, in the example of FIG. .924.
[0045]
In this way, the speech data of all single syllables to be recognized is input to the speech processing unit 4, the correct answer rate for these single syllables is calculated, and the correct answer rate data (S_i,_Sound) Is stored in the voice processing unit correct answer rate data holding unit 7. Similarly, with respect to video, video data including the lip portion of the speaker who utters all single syllables to be recognized is input to the video processing unit 2, and correct answer rate data (S for these single syllables)._i,_Image) Is calculated and stored in the video processing unit correct answer rate data holding unit 6.
[0046]
It should be noted that the video data and audio data used for calculating the correct answer rate data are preferably provided by a voice recognition target person or a plurality of speakers as data providers. It is desirable that the number of input data has little variation.
[0047]
The video processing unit 2 of this apparatus extracts, for example, the vertical and horizontal lengths of the lip portion and the ratio of the vertical and horizontal lengths as feature amounts from the video input from the video input unit 1. The similarity between the feature quantity corresponding to the i-th single syllable and the feature quantity extracted from the input video among the standard video data of a plurality of single syllables prepared in advance for calculating the similarity. (R_i,_Image) Is calculated by the HMM, and the result is output.
[0048]
Further, the speech processing unit 4 extracts a feature amount from speech input from the speech input unit 3 by cepstrum analysis, and corresponds to the i-th single syllable among a plurality of prepared single speech syllable standard data. Similarity between the feature quantity and the feature quantity extracted from the input speech (R_i,_Sound) Is calculated by the HMM, and the result is output.
[0049]
The voice recognition unit 5 outputs the video processing unit 2 (R_i,_Image) And the output of the voice processing unit 4 (R_i,_Sound) And correct answer rate data (S) stored in the video processing unit correct answer rate data holding unit 6_i,_Image) And the correct answer rate data (S) held in the voice processor correct answer rate data holding unit 7_i,_Sound) And the total similarity of video and audio (R_i,_Total) Is calculated by the following equation (2).
R_i,_Total= S_i,_Image・ R_i,_Image+ S_i,_Sound・ R_i,_Sound  (Formula 2)
The voice recognition unit 5 performs similarity R for all single syllables included in the video standard data and the voice standard data._i,_TotalThis similarity R_i,_TotalThe single syllable with the maximum is output as the recognition result.
[0050]
As described above, the speech recognition apparatus of this embodiment performs speech recognition by combining the similarity between video and audio and the correct answer rate. Combining this correct answer rate means that when identifying single syllables, among the identification methods using audio or video, more weight is given to the method that can effectively identify the single syllable. By doing so, highly accurate voice recognition can be realized even in a noisy environment.
[0051]
For calculating the similarity in the video processing unit 2 and the audio processing unit 4, other methods generally used for voice recognition such as a neural network may be used in addition to the HMM.
[0052]
(Second Embodiment)
The speech recognition apparatus of the second embodiment has the same configuration as that of the first embodiment (FIG. 1), but only the overall similarity (R_i,_TotalOnly the calculation operation is different.
[0053]
The voice recognition unit 5 of this apparatus is S_i,_Image・ R_i,_Image> S_i,_Sound・ R_i,_SoundWhen
R_i,_Total= S_i,_Image・ R_i,_Image                    ……… (Formula 3)
While the value of S_i,_Image・ R_i,_Image≦ S_i,_Sound・ R_i,_SoundWhen
R_i,_Total= S_i,_Sound・ R_i,_Sound                    ......... (Formula 4)
Is calculated. And R_i,_TotalThe single syllable with the maximum is output as the recognition result.
[0054]
As described above, in the apparatus of this embodiment, the one having higher reliability among the identification results based on the video data or the audio data is selected and used for identification. By doing so, it is possible to simplify arithmetic processing in speech recognition while maintaining high recognition accuracy.
[0055]
(Third embodiment)
The speech recognition apparatus according to the third embodiment can perform highly accurate speech recognition even when the S / N of the input speech varies. In speech recognition using speech, the correct answer rate varies with the S / N of the input speech. This device is configured to cope with such changes.
[0056]
As shown in FIG. 3, this apparatus detects an utterance section (speaking section) and a non-speaking section (non-speech section) from the lip image extracted by the video processing unit 2. A detection unit 8; a sound pressure level detection unit 9 that calculates a signal-to-noise ratio (S / N) from the sound pressure level in the utterance interval and the sound pressure level in the non-utterance interval; Syllable correct rate data (S_i,_Sound) As a voice processing unit correct answer rate data holding unit 7 that holds a plurality of types of data corresponding to the S / N of the input voice signal, and the correct answer held by the voice processing unit correct answer rate data holding unit 7 Rate data (S_i,_Sound), The correct answer rate data corresponding to the S / N detected by the sound pressure level detection unit 9 is output to the speech recognition unit 5. Other configurations are the same as those of the first embodiment (FIG. 1).
[0057]
In this apparatus, the video processing unit 2 extracts the feature value of the lip from the input video, and the similarity (R) between the feature value and the feature value in the video standard data of each single syllable_i,_Image) Is calculated and output to the speech recognition unit 5.
[0058]
The utterance section detection unit 8 samples feature amounts such as the vertical and horizontal lengths of the lips extracted by the video processing unit 2 or a ratio thereof at regular intervals, and changes in the feature amounts over time. When the amount exceeds the set threshold, it is identified as an utterance interval, and when the amount does not exceed the threshold, it is identified as a non-utterance interval, and the identification result is output to the speech processing unit 4.
[0059]
The speech processing unit 4 divides the speech signal input from the speech input unit 3 into a speech section and a non-speech section and outputs the speech signal to the sound pressure level detection unit 9, and extracts a feature amount from the input speech signal, The similarity (R) to the feature value in the speech standard data of each single syllable_i,_Sound) Is calculated and output to the speech recognition unit 5.
[0060]
The sound pressure level detection unit 9 detects an average sound pressure level in each time interval of the utterance interval and the non-utterance interval, and signals the sound pressure level of the utterance interval as the signal level and the sound pressure level of the non-utterance interval as the noise level. A noise-to-noise ratio (S / N) is calculated and output to the speech processing unit correct answer rate data holding unit 7.
[0061]
The speech processing unit correct answer rate data holding unit 7 stores the correct answer rate data (S_i,_Sound) Holds a plurality of types of data corresponding to a plurality of S / Ns of the input audio signal. Then, when the S / N is input from the sound pressure level detection unit 9, the voice processing unit correct answer rate data holding unit 7 receives the correct answer rate data (S of the type corresponding to the S / N)._i,_Sound) Is prepared as output data.
[0062]
The voice recognition unit 5 outputs the outputs (R) of the video processing unit 2 and the voice processing unit 4._i,_Image), (R_i,_Sound) And the correct answer rate data (S) read from the video processor correct answer rate data holding unit 6_i,_Image) And the correct answer rate data (S) corresponding to S / N selected from the voice processing unit correct answer rate data holding unit 7_i,_Sound) And the overall similarity of video and audio (R_i,_Total) Is calculated by equation (2) and R_i,_TotalThe single syllable with the maximum is output as the recognition result.
[0063]
Thus, in the speech recognition apparatus of this embodiment, correct answer rate data (S_i,_SoundSince a value corresponding to S / N is used as), voice recognition processing can be performed more reliably even for input voice signals having different S / N.
[0064]
(Fourth embodiment)
The voice recognition device according to the fourth embodiment uses a video of a speaker taken from various directions for voice recognition processing.
[0065]
As shown in FIG. 4, this apparatus includes a video input unit 1 for inputting a video obtained by photographing a speaker from the front, a video input unit 10 for inputting a video obtained by photographing the speaker from the side, and a video input unit 1. A video processing unit 2 that extracts the feature amount in the lip portion of the input front image and obtains the similarity between the feature amount and the feature amount of the video standard data composed of the front image of the lip portion speaking each single syllable; A video processing unit that extracts the feature value in the lip portion of the side image input from the input unit 10 and calculates the similarity between the feature value and the feature value of the video standard data composed of the side image of the lip portion that speaks each single syllable 11 and correct answer rate data (S_i,_Image1) That holds the image processing unit correct answer rate data holding unit 6 and the correct answer rate data (S_i,_Image2And a video processing unit correct answer rate data holding unit 12. Other configurations are the same as those of the first embodiment (FIG. 1).
[0066]
In this apparatus, a video signal obtained by photographing a speaker from the front is input to the video input unit 1, and a video signal obtained by photographing the speaker from the side is input to the video input unit 10.
[0067]
The video processing unit 2 extracts the feature value of the lip portion from the front video input from the video input unit 1, and the feature value of the video standard data including the front video of the lip portion that utters each single syllable. Similarity (R_i,_Image1) Is calculated and output to the voice recognition unit 5, and the video processing unit 11 extracts the feature value of the lip from the side image input from the video input unit 10, and calculates the feature value and each single syllable. Similarity with the feature value of the video standard data consisting of the side video of the lip that speaks (R_i,_Image1) Is calculated and output to the speech recognition unit 5.
[0068]
The video processing unit correct answer rate data holding unit 6 stores the correct answer rate data (S) of each syllable recognized by voice based on the front image of the lip._i,_Image1) Is stored, and the correct response rate data 12 of each single syllable that is voice-recognized based on the side image of the lip portion (S)_i,_Image2) Is held.
[0069]
The voice recognition unit 5 outputs the video processing unit 2 (R_i,_Image1), Output of the video processing unit 11 (R_i,_Image2), And the output (R_i,_Image) And the correct answer rate data (S) read out from the video processor correct answer rate data holding unit 6_i,_Image1), Correct answer rate data (S) read from the video processing part correct answer rate data holding unit 12_i,_Image2), And correct answer rate data (S) read out from the voice processing unit correct answer rate data holding unit 7_i,_Sound), The overall similarity between video and audio (R_i,_Total) Is calculated by equation (5).
R_i,_Total= S_i,_Image1・ R_i,_Image1+ S_i,_Image2・ R_i,_Image2
+ S_i,_Sound・ R_i,_Sound                  ......... (Formula 5)
And R_i,_TotalThe single syllable with the maximum is output as the recognition result.
[0070]
As described above, in the apparatus of this embodiment, it is possible to perform more reliable voice recognition by using the video of the speaker taken from a plurality of directions.
[0071]
In this embodiment, the case where the front and side images of the speaker are used has been described. However, by using more images such as images from oblique directions in addition to the front and side surfaces, more reliable voice recognition is possible. It becomes possible.
[0072]
(Fifth embodiment)
In the fifth embodiment, a voice recognition device that identifies a group to which a single syllable of spoken voice belongs will be described.
[0073]
For example, there are features common to single syllables belonging to the group of “A” (“A”, “I”, “U”, “E”, “O”), and “K”, “S” that contain the same consonant. If there is a feature common to the single syllables belonging to each group of “line”,..., The similarity between the single syllable feature of the input speech and the feature of each group is compared. It can be specified whether it belongs to a group.
[0074]
For example, when the word “mole” is uttered, a plurality of single syllables are selected as candidate single syllables corresponding to each of “mo”, “gu”, and “la”. There is known a method of identifying a finally uttered word based on whether combinations of candidates “M”, “G”, and “La” are hit in order, and whether the combination has a meaning as a word.
[0075]
In such a case, for example, if the group to which the leading single syllable belongs can be identified, the number of combinations of candidates to be examined can be greatly reduced, and the speech recognition process can be made more efficient.
[0076]
The voice recognition apparatus according to the fifth embodiment can specify a group to which a single syllable belongs with high accuracy by using voice data and video data together.
[0077]
As in the first embodiment (FIG. 1), this apparatus includes a video input unit 1, video processing unit 2, audio input unit 3, audio processing unit 4, video processing unit correct answer rate data holding unit 6, audio processing. A correct answer rate data holding unit 7 and a voice recognition unit 5 are provided.
[0078]
However, the video processing unit 2 does not compare the features extracted from the input video of the speaker's lip part with the standard video data of the lip part that utters each single syllable, but each group consisting of a plurality of single syllables. Compared with the feature of, the similarity for each group is output.
[0079]
Further, the speech processing unit 4 does not compare the features extracted from the input speech with the speech standard data of individual single syllables, but compares the features extracted from the input speech with respect to each group. Outputs similarity.
[0080]
In order to obtain the similarity with each group, the video processing unit 2 and the audio processing unit 4 calculate, for example, the similarity between the feature value of the input single syllable and the feature values of all the single syllables included in the group. The similarity of a single syllable having the maximum similarity is defined as the similarity of the group. Alternatively, the feature amount pattern common to all single syllables included in the group is used as the feature amount of the group, and the similarity between the feature amount of the input single syllable is calculated.
[0081]
The video processing unit correct answer rate data holding unit 6 and the audio processing unit correct answer rate data holding unit 7 hold the correct answer rates of the groups output from the video processing unit 2 or the audio processing unit 4. In order to obtain the correct answer rate, as illustrated in FIG. 5, a single syllable video or audio (“ka” “ki” “ku”) sample is input from the video input unit 1 or the audio input unit 3. Actually, which group (“A row” “Ka row”... “Wa row”) is output from the video processing unit 2 or the audio processing unit 4 is measured. The ratio of the number of correct answers (the number of outputs in which the input single syllable was included in the group) is calculated.
[0082]
The voice recognizing unit 5 outputs i from the output from the video processing unit 2 and the voice processing unit 4 and the correct answer rate data held in the video processing unit correct answer rate data holding unit 6 and the voice processing unit correct answer rate data holding unit 7. Total similarity to the second group (R_i,_Total) Is calculated by the above (Formula 2). And R_i,_TotalThe group with the largest value is output as the recognition result.
[0083]
In this way, this apparatus can specify the group to which the single syllable of the uttered voice belongs with high accuracy.
[0084]
Examples of grouping include three syllable groups including lip sounds (/ b /, / m /, / p /), single syllable groups including stuttering (/ y /), and groups not including lip sounds and stuttering. It can also be divided into two.
[0085]
In this case, since the characteristics of the lip sound appear in the shape of the lips, the correct answer rate of the group including the lip sound tends to be higher for the video processing unit 2 and lower for the voice processing unit 4. On the contrary, since the stuttering is characterized by the utterance sound, the correct answer rate of the group including the stuttering tends to be higher in the voice processing unit 4 and lower in the video processing unit 2. Therefore, when the total similarity for each group is calculated by (Equation 2), the contribution of the similarity output from the video processing unit 2 is high for the lip sound target, and for the stuttering target. The contribution of the similarity output from the voice processing unit 4 is increased.
[0086]
Therefore, it is possible to more surely recognize whether the input single syllable includes a lip sound or does not include a stuttering sound by performing audio recognition using both video and audio.
[0087]
In this way, when grouping single syllables to be recognized, by selecting a group with a high correct answer rate of the output of the video processing unit 2 and a group with a high correct answer rate of the output of the audio processing unit 4, respectively, only the audio is selected. Or, compared with the case of identifying a group only by video, it becomes possible to recognize a more detailed group.
[0088]
【The invention's effect】
As is clear from the above description, the speech recognition apparatus of the present invention performs speech recognition by combining the similarity between the input audio data and video data and the standard data, and the correct answer rate, so that noise is generated. Even in an existing environment, more reliable speech recognition can be realized.
[Brief description of the drawings]
FIG. 1 is a schematic configuration diagram of a speech recognition apparatus according to a first embodiment of the present invention;
FIG. 2 is a diagram for explaining a calculation method of correct answer rate data in the first embodiment;
FIG. 3 is a schematic configuration diagram of a speech recognition apparatus according to a third embodiment of the present invention;
FIG. 4 is a schematic configuration diagram of a speech recognition apparatus according to a fourth embodiment of the present invention;
FIG. 5 is a diagram for explaining classification of groups of single syllables in the fifth embodiment of the present invention;
FIG. 6 is a schematic configuration diagram of a conventional speech recognition apparatus.
[Explanation of symbols]
1 Video input section
2 Video processing section
3 Voice input part
4 Voice processing part
5 Voice recognition unit
6 Video processing part correct answer rate data holding part
7 Voice processing part correct answer rate data holding part
8 Voice detection section
9 Audio level detector
10 Second video input section
11 Second video processing unit
12 Second video processing part correct answer rate data holding part

Claims (3)

Video input means for inputting the video data of the speaker including the lips;
Voice input means for inputting the voice data of the speaker;
Video processing means for calculating the similarity between the video standard data of the lips that utter each single syllable and the input video data, and outputting each single syllable of the video standard data and the similarity thereto;
A speech processing means for calculating the similarity between the speech standard data of the speech uttering each single syllable and the input speech data, and outputting each single syllable of the speech standard data and the similarity thereto;
The video data relating to various single syllables is input to the video input means in advance, and the single syllable of the video standard data having the maximum similarity output from the video processing means corresponding to each input video data is selected as a candidate single syllable. The ratio of the number of candidate single syllables that match the single syllable of the input video data out of the total number of identical candidate single syllables output from the video processing means calculated based on the Video standard data correct answer rate holding means for holding data as correct answer rate data for that single syllable ;
The voice data of various single syllables is input to the voice input means in advance, and the single syllable of the voice standard data having the maximum similarity output from the voice processing means corresponding to each input voice data is set as a candidate single syllable. The ratio of the number of candidate single syllables that match the single syllable of the input voice data out of the total number of identical candidate single syllables output from the voice processing means calculated and calculated based on the voice standard data Voice standard data correct answer rate holding means for holding as a correct answer rate data for the single syllable ,
Product of the image processing similarity output from means the video standard data correct rate holding means from the read right answer rate, and, from the similarity output from the audio processing means the audio standard data correct rate holding means A speech recognition apparatus comprising speech recognition means for obtaining a sum of products of read correct answer rates as an overall similarity to each single syllable and identifying a single syllable having the largest overall similarity.   The voice standard data correct answer rate holding means holds a plurality of types of correct answer rate data corresponding to a signal-to-noise ratio as the correct answer rate data, and the voice recognition means has a comprehensive similarity to each single syllable. The speech recognition apparatus according to claim 1, wherein when determining the correct answer rate, the correct answer rate data corresponding to a signal-to-noise ratio of the input speech data is read from the speech standard data correct answer rate holding unit. Video input means for inputting the video data of the speaker including the lips;
Voice input means for inputting the voice data of the speaker;
Single syllables with common utterance mechanisms are grouped in advance, and for each group, the similarity between the video standard data of the lips that utter each single syllable of the group and the input video data is calculated, and the largest of them is calculated . Video processing means for setting the similarity to the similarity of the group and outputting the group identification information and the similarity of the group;
For each group, calculate the similarity between the voice standard data of the voice that utters each single syllable of the group and the input voice data, and set the maximum similarity among them as the similarity of the group Voice processing means for outputting the group identification information and the similarity of the group;
In advance, the video data relating to various single syllables is input to the video input means, and the total number of outputs of each group output from the video processing means corresponding to each input video data is calculated and calculated based thereon. Video standard data correct answer rate holding means for holding the ratio of the number of correct outputs included in the group of the input single syllables among the total number of outputs, as correct answer rate data for the group ,
Preliminarily input the voice data of various single syllables to the voice input means, totaled the total number of outputs of each group output from the voice processing means corresponding to each input voice data, calculated based on it, A voice standard data correct answer rate holding means for holding the ratio of the number of correct outputs included in the group of the input single syllables among the total number of outputs, as correct answer rate data for the group ,
The video product of processing the similarity output from the means and the video standard data correct rate holding read from the means percentage of correct answers, and the similarity output from the audio processing means the audio standard data correct rate holding means obtains the sum of the product of the read correct rate from the overall similarity for each group, the speech recognition apparatus and a speech recognition means for identifying a group greatest overall similarity.

Images