JP2002169586A - Composite model generating device for voice and image, environment adapting device for composite model of voice and image, and voice recognizing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a composite model generating device for voice and image for voice recognizing device which can performs voice recognition at a high voice recognition rate and the voice recognizing device. SOLUTION: In the composite model generating device 100 for voice and image, an HMM composition part 16 computes the products of the output probabilities of the voice and image in all combinations of states of a voice HMM and an image HMM and generates and composites a composite HMM having a composited Gaussian mixture distribution including the products of the output probabilities in the respective states. Then, an HMM learning part 17 performs connected learning maximizing the output likelihood by using a labeled AV signal in a learning AV data memory 31 according to the generated and composite HMM to generate a composite HMM of the learnt voice and image. A voice recognition part 200 of the voice recognizing device 200 performs voice recognition by using the composite HMM of the learnt voice and image according to the feature quantity of a feature-extracted spoken voice signal and the feature quantity of an image signal.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to an uttered voice signal,
A speech and image synthesis model generation device used for speech recognition based on the lip image signal at the time of speech, an environment adaptation device for a speech and image synthesis model, and the synthesis model generation device and / or The present invention relates to a speech recognition device using an environment adaptation device.

[0002]

2. Description of the Related Art As a speech recognition system more suitable for a real environment, a bimodal speech recognition system using speech and a moving image around the lips has been studied in recent years. The signal-to-noise ratio (hereinafter referred to as SN) of a voice such as a situation where it is difficult to speak with a loud voice such as in a train or a public place, or the surrounding environment is noisy.
Called R. The advantage of using a moving image around the lips for voice recognition is that using a moving image around the lips in a situation where the ratio is lower than that in the case of using the moving image around the lips gives a higher recognition performance than using only the voice. In recent years, modeling of bimodal speech recognition includes a hidden Markov model (hereinafter referred to as HM).
It is called M. ) And its effects have been reported (for example, Prior Art Document 1 "Tetsu Nakamura et al.," Speech recognition and lip image generation by integrating speech and lip images using HMM "), Information Processing Society of Japan, Spoken Language Information Processing, Vol.
-17, February 8, 1997 ". ).

In the conventional bimodal speech recognition of speech and image based on HMM, an initial integration method of integrating image data and speech data at a feature vector stage and assigning a weighting factor to an output probability, There is a result integration method in which processing is performed in separate processes and a weighting factor is assigned to the likelihood of the result. Specifically, in the initial integration method, the parameters of the audio data and the image data are set as independent parameter streams, and the product of the output probabilities of the respective HMMs is calculated in each state, and is calculated as the output probability of that state. At this time, a weight coefficient of the power of the output probability of each stream is given. On the other hand, in the result integration method, contrary to the above-mentioned initial integration method, the likelihood for all words is separately calculated for the voice data and the image data, and finally, the logarithm of the voice data for the same word is calculated. The likelihood and the log likelihood of the image data are weighted and added, and the result is calculated as the log likelihood of the word. These two methods are referred to as conventional methods as comparative examples.

[0004]

However, in the case of the initial integration method, the state transition probability of the HMM is shared,
Since the result integration method processes speech and images in separate processes, they do not consider the relationship between speech speed and lip movement, and even when speech with the same phoneme is spoken, Since the movements of the lips may not always match, there is a problem that the speech recognition rate is still low.

[0005] Further, when integrating the voice HMM and the image HMM, it is important to determine which information should be prioritized according to the surrounding environment. Yes not shown yet.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above problems and to provide a speech and image synthesis model generation apparatus for a speech recognition apparatus capable of recognizing speech at a higher speech recognition rate than a conventional example, and It is another object of the present invention to provide a speech recognition apparatus using the speech and image synthesis model generation apparatus.

Another object of the present invention is to integrate a speech HMM and an image HMM according to the surrounding environment, and achieve a higher speech recognition rate than the conventional example. An object of the present invention is to provide an environment adaptation device capable of generating a speech and image synthesis model capable of speech recognition, and a speech recognition device using the environment adaptation device.

[0008]

According to the present invention, there is provided an apparatus for generating a synthesized model of voice and image, which stores an AV signal including an uttered voice signal and an image signal of a lip of a speaker at the time of utterance. Means, first generating means for generating an audio HMM based on an uttered voice signal of the AV signal so that the output likelihood is maximized, and based on an image signal of the AV signal, In order to maximize the output likelihood, the image HMM
A second generation unit for generating a voice HMM generated by the first generation unit; and a second storage unit for storing the speech HMM generated by the first generation unit.
A third storage unit for storing the image HMM generated by the second generation unit, a voice HMM stored in the second storage unit, and an image HMM stored in the third storage unit. , The product of the output probabilities of speech and image in all combinations of each state of these two HMMs, and the synthesized HMM containing the product of the output probabilities in each state
And a combination learning unit that generates the maximum likelihood by using the labeled AV signal stored in the first storage unit based on the generated composite HMM. And a learning means for generating a synthesized HMM of the learned voice and image.

Further, the speech recognition apparatus according to the present invention, based on an input AV signal including an uttered voice signal and an image signal of a lip of a speaker at the time of utterance, obtains a characteristic amount of the uttered voice signal and Extracting means for extracting a feature amount of the image signal;
Based on the extracted feature amount of the uttered speech signal and the feature amount of the image signal, the speech and the synthesized speech and image synthesized HMM generated by the synthesized model of the speech and image are used to generate a speech. Voice recognition means for recognizing and outputting a voice recognition result.

[0010] Furthermore, the environment adaptation apparatus for a speech and image synthesis model according to the present invention stores an AV signal including an uttered speech signal and an image signal of a speaker's lip at the time of utterance with a phoneme label. Fourth storage means for storing the environment adaptation signal data to be processed, and second speech recognition for calculating the likelihood when the stored environment adaptation signal data is subjected to speech recognition using a predetermined HMM. Means, and the weighting factors of each phoneme in the synthesized speech and image synthesized HMM generated by the audio and image synthesis model generation device are clustered into a plurality of classes using a predetermined clustering criterion. An environment adapting means for adapting the synthesized HMM to the environment by re-learning the weighting factors of the phonemes belonging to the class based on the calculated likelihood so as to reduce misrecognition; Characterized by comprising a.

Still further, in the above-mentioned environment adaptation apparatus for a speech and image synthesis model, preferably, the environment adaptation apparatus is arranged such that the number of environment adaptation signal data of each class is less than a predetermined threshold value. It is characterized in that re-learning of the conversion means is repeated.

Further, another speech recognition apparatus of the present invention includes an utterance speech signal and an image signal of a lip of a speaker at the time of utterance.
Extracting means for extracting the feature amount of the speech sound signal and the feature amount of the image signal based on the input AV signal, the feature amount of the extracted speech sound signal and the feature amount of the image signal Using the synthesized HMM for speech and image, which is environment-adapted by the environment adaptation apparatus for speech and image synthesis model based on the above, to perform speech recognition and output a speech recognition result Means.

[0013]

Embodiments of the present invention will be described below with reference to the drawings.

<First Embodiment> FIG. 1 is a block diagram showing a configuration of a speech and image synthesis model generation apparatus 100 and a speech recognition apparatus 200 according to a first embodiment of the present invention. The speech and image synthesis model generation device 100 according to this embodiment includes an HMM synthesis unit 16 that synthesizes an audio HMM in the audio HMM memory 32a and an image HMM in the image HMM memory 32b,
And an HMM learning unit 17 that generates a combined HMM learned by performing connection learning using learning AV (Audio and Visual) data with phoneme labels in the data memory 31 based on . Further, the speech recognition device 200 includes a speech recognition unit 26 that performs speech recognition using the synthesized HMM learned by the HMM learning unit 17.

In order to solve the above-mentioned problems of the conventional example, the present inventors use an integrated method using HMM synthesis that can describe the relationship between the utterance speed and the lip movement. It is proposed to generate a synthesis model of voice and image by learning by using learning AV data with phoneme labels after synthesizing.

In the speech and image synthesis model generating apparatus 100 shown in FIG. 1, the learning AV data memory 31 with phoneme labels stores speech waveform data when a specific speaker speaks a plurality of predetermined words, At the time of speech, the head of the speaker is fixed, and image data (hereinafter, referred to as image data) in which an image around the lips is recorded is stored in advance. Next, the data separation unit 11 outputs the learning AV data memory 31
The mixed data of the audio data and the image data is separated into audio data and image data and output to the synchronization unit 12. At this time, the frame period of the image data is 33.3 m.
sec, and the frame period of the audio data is 8 msec.
Therefore, a frame shift process is performed so as to synchronize with each other, the audio data after the synchronization process is output to the preprocessing unit 13a, and the image data after the synchronization process is output to the preprocessing unit 1a.
3b.

The pre-processing unit 13a has a sampling frequency of 4
Maximum frequency 2 sampled at 4.1 kHz 2
2.05kHz audio data, maximum frequency 12kHz
Is downsampled and output to the feature extracting unit 14a. Next, the feature extracting unit 14a performs, for example, an LPC analysis on the input audio data, thereby obtaining a 16th-order mel-cepstral coefficient and a 16th-order Δ
The feature vector including the mel cepstrum coefficient and the Δ power is extracted and output to the speech HMM generation unit 15a.

On the other hand, the preprocessing unit 13b converts an RGB JPEG image signal (for example, 160 × 120 pixels) for each frame into a 256-level gray-scale image signal (gray scale signal) based on the input image data. Then, after the histogram is flattened, normalization at the lip position is performed in order to minimize the difference in luminance from the reference frame. Next, the feature extraction unit 14b performs a two-dimensional FFT process on the data after the pre-processing, for example, in an area of 256 × 256 pixels. Here, a dynamic spectrum is calculated by calculating a power spectrum in a spatial frequency domain and calculating a difference between frames. Specifically, for example, 35
A feature vector including parameters of the next smoothed logarithmic power spectrum and the 35th smoothed logarithmic Δ power spectrum is extracted and output to the image HMM generation unit 15b.

The speech HMM generation unit 15a uses a known EM (Expectation-maximi) based on the input feature vector and the phoneme label in the learning AV data memory 31.
zation), a connected HMM having a three-state Gaussian mixture distribution is generated by performing connected learning with a label such that the output likelihood is maximized, and is output to and stored in the voice HMM memory 32a. I do. On the other hand, the image H
The MM generation unit 15b performs the labeled connection learning using the EM algorithm based on the input feature vector and the phoneme label in the learning AV data memory 31 so that the output likelihood is maximized. As a result, an image HMM having a two-state Gaussian mixture distribution is generated, output to the image HMM memory 32b, and stored.

FIG. 2A shows the voice HMM memory 32 shown in FIG.
FIG. 2 is a state transition diagram showing an example of a voice HMM in FIG.
(B) is an image HMM in the image HMM memory 32b of FIG.
FIG. 6 is a state transition diagram showing an example of. The audio HMM shown in FIG. 2A has three states in the state transition direction of the audio data, and each state has a transition that returns to itself and a transition that proceeds to the next state. Further, the image HMM shown in FIG. 2B has two states in the state transition direction of the image data, and in each state, there is a transition that returns to itself and a transition that proceeds to the next state.

The HMM synthesizing unit 16 includes the voice HMM memory 3
The voice HMM in 2a and the image HMM in the image HMM memory 32b are combined using the synthesis integration method according to the present invention to calculate the product of the output probabilities of the voice and the image in all combinations of these two HMM states. Calculated and combined by generating a combined HMM having a combined Gaussian mixture distribution including the product of the output probabilities in each state, and the combined HMM is output to the combined HMM memory 33 and stored. At this time, the output probabilities of the respective states of the synthesized HMM are synthesized and integrated as a product of the output probabilities of voice and image as shown in the following equation.

[0022]

## EQU1 ## b _ij (O _t ) = b _i ^(a) (O _t ^(a) ) ^{λ a} × b
_j ^(v) (O _t ^(v) ) ^{λ v}

Where b _i ^(a) (O _t ^(a) ) is the time t,
The probability b _j ^(v) (O _t ^(v) ) of outputting the feature vector _Ot ^(a) in the state i of the voice HMM is the probability of outputting the feature vector _Ot ^(v) in the state j of the image HMM. Yes, λa, λv
Is a weight coefficient of the audio data stream and a weight coefficient of the image data stream, respectively. Also, synthetic HM
M, the transition probability a from the state S _ij to the state S _{k λ
ij, k λ} is _{calculated using} a transition probability a _ik ^(a) from the state S _{i of the} voice HMM to the state S _k and a transition probability a _{j λ} ^(v) of the image HMM from the state S _j to the state S _λ . Is represented by the following equation.

[0024]

## EQU2 ## a _{ij, k λ} = a _ik ^(a) × a _{j λ} ^(v)

FIG. 3 is a state transition diagram showing an example of the combined HMM in the combined HMM memory 33 of FIG. As shown in FIG. 3, the combined HMM has three states in the state transition direction of the audio data.
There are two states in the state transition direction of the image data, that is, a total of six states having a Gaussian mixture distribution in each state. Here, the synthesized HMM is a voice HMM and an image H
A MM and a three-dimensional trellis in the time direction are formed. However, since the HMM parameters at the time of synthesis are independently learned for voice and image, synchronization between voice and image is not considered. Therefore, in the present embodiment, the HMM learning unit 1
7, using the synthesized HMM as an initial model, using an audio-video synchronous mixed vector obtained by synthesizing a feature vector of audio and an image, using an EM algorithm so that the output likelihood is maximized, and a learning AV data memory 31. The connected learning is performed using the phoneme labels stored in the learning HMM, and the learned synthesized HMM after the learning is output to the learned synthesized HMM memory 34 and stored. The synthesized H by the HMM learning unit 17
By learning the MM, synchronism can be expressed between the voice and the image.

FIG. 4 is a diagram showing a three-dimensional search space for synthesis integration in the speech and image synthesis model generating apparatus 100 of FIG. As shown in FIG. 4, at the time of speech recognition, the search space based on the synthesis integration searches the state of the voice HMM and the image HMM and the three-dimensional trellis in the time frame direction, and asynchronously changes the state of the voice and the image. It becomes searchable.

In FIG. 1, a speech recognition apparatus 200 includes an input AV data memory 41, a data separation section 21, a synchronization section 22, pre-processing sections 23a and 23b, and a feature extraction section 2.
4a and 24b, a feature synthesizing unit 25, and a voice recognition unit 26. Here, the processing from the data separation unit 21 to the feature extraction units 24a and 24b is the same as the processing from the data separation unit 11 to the feature extraction units 14a and 14b in the audio and image synthesis model generation device 100, and is described in detail. Detailed description is omitted.

In the voice recognition device 200, the input AV data memory 41 stores the waveform data of the utterance voice signal of the utterance voice sentence to be voice-recognized and the image data in which the image around the lips of the speaker at the time of the utterance is recorded. In the same manner as the data separation unit 11, the data separation unit 21 separates the AV data input from the input AV data memory 41 into the audio data of the uttered voice signal and the image data and synchronizes them. Output to the conversion unit 22. The feature synthesizing unit 25 synthesizes data of the feature vector of the voice extracted by the feature extracting unit 24a and the feature vector of the image extracted by the feature extracting unit 24b (each data is converted into one feature in the same frame). Synthesized as a vector), speech recognition unit 2
6 is output. The speech recognition unit 26 inputs the feature vector input from the feature synthesizing unit 25 to the learned synthesized HMM in the learned synthesized HMM memory 34, so that the likelihood in the HMM (specifically, Is calculated, and the maximum likelihood phoneme is determined. Further, based on the determined maximum likelihood phoneme, the speech recognition unit 26 calculates the likelihood for the word using the phoneme-based word HMM stored in the word HMM 42 and determines the character string of the maximum likelihood word. Thereby, the voice recognition processing is executed, and the character string of the voice recognition result is output.

In the speech and image synthesis model generating apparatus 100 configured as described above, the HMM learning unit 17 performs connection learning on the synthesized HMM synthesized by the HMM synthesis unit 16 using phoneme labels. , It is possible to establish synchronism between voice and image, and to generate a learned synthesized HMM having synchronism between voice and image. By performing the speech recognition processing by the speech recognition unit 26 using this, the speech recognition can be performed at a higher speech recognition rate than the initial integration method and the result integration method according to the related art.

[0030]

EXAMPLES <Examples of First Embodiment> The present inventors
Synthetic model generation device 1 for voice and image according to first embodiment
An evaluation experiment was performed as follows using the voice recognition apparatus 200 and the speech recognition apparatus 200, and the following results were obtained. Table 1 shows the experimental conditions.

[0031]

[Table 1] Experimental conditions ――――――――――――――――――――――――――――――――― Voice Sampling frequency: 12 kHz Analysis window function : Hamming window Frame length: 32 msec Frame shift: 8 msec Parameter: MFCC 16 dimension, ΔMFCC 16 dimension ―――――――――――――――――――――――――――――――― ――― Image Frame shift: 33 msec Preprocessing 1: Conversion from RGB signals to 256-level grayscale image signal Preprocessing 2: Histogram flattening Preprocessing 3: Lip position normalization Processing Parameter: Smoothed log power spectrum 35 dimensions and smoothed logarithmic Δ power spectrum 35 dimensions ――――――――――――――――――――――――――――――――― HMM status Numerical result integration method, synthesis integration: voice 3, Image 2 Initial integration method: 3 ――――――――――――――――――――――――――――――――― Probability density function Gaussian distribution: 2 mixture HMM Phoneme environment independent 55 phoneme model ――――――――――――――――――――――――――――――――――― Learning data Voice and image synchronization data Female One speaker, 4740 words ――――――――――――――――――――――――――――――――――― Test data 200 words (3 sets) (Open condition) ―――――――――――――――――――――――――――――――――――

In this experiment, in a sound laboratory, a specific speaker (one female speaker) was placed on the utterance list 52 owned by the patent applicant.
A database speaking 40 words was used. Since the frame shift between sound and image is 1: 4, the image is 4
Embed the same frame and adjust the audio and image frame shift. The recorded image data shows differences in lighting conditions, face inclination, and the like depending on the utterance word. Therefore, as pre-processing, histogram flattening and lip position normalization were performed so as to minimize the difference in luminance from the reference frame. To create a speech HMM, a mel-cepstrum coefficient was obtained from clean speech data recorded in an acoustic laboratory, and a model was created using the coefficient as a feature vector.
Further, the image HMM performs a two-dimensional FFT on the preprocessed image, calculates a logarithmic power spectrum, performs a 6 × 6 region division on the frequency region, and smoothes the logarithm of the region excluding the DC component. A model was created using the power spectrum as a feature vector. In this experiment, the combined HMM of audio and image is trained with the weighting factor of each stream being equal to 1: 1.

FIGS. 5 to 7 show experimental results of the speech recognition apparatus using the initial integration method and the result integration method, which are conventional examples, and the synthesis integration method according to the first embodiment, respectively. Weight factor λ of the audio stream at 10 dB, 20 dB, and for clean audio without noise
6 is a graph showing a word recognition rate for a. Here, the weight coefficient λa of the audio data stream and the weight coefficient λv of the image data stream are changed so as to satisfy the following equation.

[0034]

Λa + λv = 1

That is, FIG. 5 is a graph showing the word recognition rate when the SNR is 10 dB and the horizontal axis is the weight coefficient λa of the audio stream. FIG. 6 is a graph showing the SNR when the SNR is 20 dB.
FIG. 7 is a graph showing the word recognition rate when the horizontal axis is the weight factor λa of the audio stream at B, and FIG. 7 shows the horizontal axis of the weight coefficient λa of the audio stream when the voice is clean without noise. 6 is a graph showing a word recognition rate at the time. 5 to 7 show data in the following cases. (A) The synthesis integration method according to the first embodiment (with learning), (b) The synthesis integration method of the comparative example (without learning), (c) the result integration method of the conventional example, (d) the initial integration of the conventional example (E) When only audio data, (f) When only image data. As is clear from FIG. 5, when the SNR is 10 dB,
Even if the weight coefficient λa of the audio stream is changed, the speech recognition rate of the synthesis integration method according to the first embodiment of the present invention is higher than that of the other integration methods. As is clear from FIG. 6, when the SNR is 20 dB, even if the weight coefficient λa of the audio stream is changed, the speech recognition rate of the synthesis and integration method according to the first embodiment of the present invention is different from that of the first embodiment. Higher than that of the integration method. Further, as is clear from FIG. 7, even when the weight coefficient λa of the audio stream is changed in the case of clean audio without noise, the speech recognition rate of the synthesis integration method according to the first embodiment of the present invention is , Higher than that of other integration laws. Therefore,
It can be seen that even when the SNR is changed from 10 dB to infinity, the synthesis integration method according to the first embodiment of the present invention has a higher recognition rate than other integration methods. It can be said that the synthetic integration method of this method is effective in integrating different modalities.

<Second Embodiment> FIG. 8 is a block diagram showing a configuration of an environment adapting apparatus 300 according to a second embodiment of the present invention. In the environment adapting apparatus 300 according to the second embodiment, in the same format as the learning AV data memory 31 of FIG. 1, a plurality of utterance voice signals and an image signal of the speaker's lips at the time of utterance are included. AV word data memory for environment adaptation with word AV signal data having phoneme labels
In the input AV data memory 41 of FIG.
Voice recognition is performed using the voice recognition device 200 of FIG. 1 based on the environment adaptation AV signal instead of the signal data. The environment adaptation processing unit 50 executes the environment adaptation processing of the stream weight coefficients shown in the flowchart of FIG. 9, and specifically, learns the learned speech generated by the HMM learning unit 17 of FIG. 1. And a weighting factor of each phoneme in the image-synthesizing HMM is clustered into a plurality of classes using a predetermined clustering criterion such as a binary tree tree clustering tree shown in FIG. Based on the log likelihood calculated by the speech recognition unit 26 in the speech recognition apparatus 200, the weighting coefficient is set so that misrecognition is reduced (specifically, the misclassification measure d _x shown in Expression 6 is reduced). As described above, the synthetic HMM is adapted to the environment by re-learning. Here, preferably, the re-learning of the environment adaptation process is repeated so that the number of environment adaptation signal data of each class becomes less than a predetermined threshold value. Then, the speech recognition device 200 in FIG. 1 performs a speech recognition process using the re-learned synthesized HMM.

First, an environment adaptation method which is a relearning method according to the present embodiment will be described below.

As described in the related art section, the HM
When integrating M and the HMM of an image, it is important to determine which information is to be prioritized according to the surrounding environment. Specifically, the problem is that, by the synthesis integration method according to the first embodiment, the stream weight of the audio and the image at which the recognition rate of the integrated HMM reaches a peak is determined from the adaptation data spoken by the user. A method of adapting according to the situation can be considered. However, it is difficult to estimate the SNR of speech, so other criteria are needed to estimate the stream weight. Normally, it is known that good performance cannot be obtained by learning based on the maximum likelihood criterion (ML criterion) because the dynamic range of the likelihood between voice and image greatly differs. Therefore, in the present embodiment, using the learning based on the minimum classification error criterion (MCE criterion), specifically, a known GPD (Generalized Probabilistic Desc
ent method; generalized stochastic descent method) algorithm (for example, see Prior Art Document 2 “Gerasimos Potamianos et
al., ”Discriminative training of HMM stream expone
nts for Audio-Visual speech recognition ”, Proceedi
ng of ICASSP-98, Vol. 6, pp. 3733-3736, May 1998 ", and prior art document 3" Chiyomi Miyajima et al., "Audio-
Visual speech recognition using MCE-basedHMMs and
model-dependent stream weights ”, Proceeding of ICS
LP2000, Vol.2, pp.1023-1026,2000 ". ) To adapt the environment by re-learning the composite HMM. The reason for using the GPD algorithm is as follows. The weighting factors of the audio and image streams are different for each phoneme. Therefore, it is considered that the clustering unit of the stream weighting factors should be divided for each phoneme class according to the number of adaptive data. In contrast to the direct search method, which is one of the methods based on the MCE, the GPD algorithm is an algorithm that can be applied to multiple variables and has high applicability.

Next, the adaptation of the stream weight coefficient to the environment will be described.

In the stream weighting coefficient estimation method using the GPD algorithm, the weight of an HMM stream is minimized so as to minimize a smooth loss function including a misclassification measure indicating information on a distance between a correct classification and an incorrect classification. Estimate the coefficient. Here, the process of estimating the weight coefficient of the stream for each phoneme based on the GPD algorithm will be described below.

First, a feature vector sequence O of uttered voice data x of a certain word is

[Number 4] _{O = [o x (1)} , ..., o x (t), ..., o x (T x)] and. Here, t is the time frame, o _x (t) is a vector with the S streams (modalities).
Next, a set of stream weighting factors for a set C with HMM states is

Λ _c = [λ _c1 ,..., Λ _cs ,..., Λ _cS ], and the weight coefficient set of the entire stream is

６ = [λ ₁ ,..., Λ _c ,..., Λ _C ]. Here, C is the number of classes of the weight coefficient of the stream for each phoneme. At this time, the utterance voice data x of a certain word is converted to the corresponding word HMM (the word HMM in FIG. 1).
For example, the state sequence of the HMM at the time of voice recognition by the Viterbi algorithm is

Assuming that Q _x = {q _x (t); t = 1,..., T _x }, the log likelihood L _x ^{R at} that time is represented by the following equation.

[0042]

(Equation 8)

(Equation 9)

Thus, it can be expressed as a function of the set of stream weighting factors Λ. In _Equation 9, b _js [ox _{, s} (t)] is the probability q _x of observing the feature vector ox _{, s} (t) of the stream s in the state j.
If (t) is q _x (t) = j, δ ^j _{qx (t)} =
1 and if q _x (t) ≠ j, δ ^j _{qx (t)} =
0. Similarly, for the utterance voice data x of the word,
The log likelihood L _x ^Fn when the word HMM is erroneously recognized by the nth candidate is represented by the following equation.

[0044]

(Equation 10)

Next, the misclassification measure d _x is defined as follows.

[0046]

[Equation 11]

[0047] The misclassification measure d _x expresses that about classification error small, i.e. erroneous recognition is reduced. However, Equations 9 and 10 above may calculate non-smooth functions in order to calculate the likelihood in the maximum likelihood state sequence. Then, using the misclassification measure d _x , the form is converted into a sigmoid function as in the following equation, and a smooth loss function is defined.

[0048]

(Equation 12)

In order to stabilize the direction of the gradient, a loss function of the following equation is set for the entire adaptive data.

[0050]

(Equation 13)

Where X is the total number of adaptive data. The weight coefficient の of the entire stream is updated by the following equation using the GPD algorithm.

[0052]

１４ _{k + 1} = Λ _k −ε _k E _k ∇L (Λ) | _{Λ = Λ k} , k
= 1, 2, ...

Here, E _k is a unit matrix.

[0054]

(Equation 15)

(Equation 16)

It has been proved that this algorithm converges if the above equation is satisfied (for example, see W. Chou et al., A minimum error rate pattern re
cognition approach to speech recognition ”, Journal
of Pattern Recognition and artificial intelligenc
e, Column VIII, pp. 5-31, 1994 ". ).

Further, a description will be given of a formula for updating the weight coefficient of the stream. Here, a description will be given of the expansion of the equation for actually calculating the above equation (14). However, (Λ) is omitted for simplicity. First, in the process of the GPD algorithm, the weight coefficient class c of each stream is

[Equation 17] To add

(Equation 18) A transformation that satisfies

Λ h _cs = log λ _cs is performed. Then, in order to update the weight coefficient of the stream by the above equation (14), the following equation is calculated from the above equations (12) and (13).

[0057]

(Equation 20)

Here,

(Equation 21)

(Equation 22) It is.

Here, B = R or Fn, and C is the HM when the values of the weighting factors of the stream are clustered.
A set of M states. Equations (12) and (20) to (22) are calculated, and the weight coefficient of the stream is updated by the equation (14). Finally, after updating each step,
Is converted by

Further, the subdivision of the unit of clustering of the weighting factor of the stream using the tree structure will be described. In the present embodiment, a method is considered in which the weighting factor of the stream of the phoneme HMM is used as a basic unit, and the clustering unit of the weighting factor of the stream is divided top-down according to the number of adaptive data.

First, a tree structure as a reference for performing HMM clustering is created. As a procedure for creating a tree structure, a plurality of questions are prepared, and HMM clustering is performed on those questions. Examples of the questions (assigned to each node) used in this experiment are the following three items. (1) Is the HMM a vowel or a consonant? (2) Is it voiced or unvoiced? (3) Whether the articulation position is around the lips? By performing the clustering as described above, the foresight knowledge of the voice can be incorporated in the weight coefficient estimation of the stream.

Then, from a plurality of questions prepared in advance,
Select one question and cluster the HMM. As the question, the question "whether voiced or unvoiced" with the best recognition performance in the preliminary experiment was selected. HM at this time
FIG. 10 shows an example of a clustering tree having a binary tree structure as a reference when M is clustered. In this example,
At the root node 101, it is determined whether or not it is a voiced sound. If YES, the process proceeds to the cluster node 102 to perform clustering, whereas if NO, the process proceeds to the cluster node 103 to perform clustering. Then, the clustering process is repeated toward lower layers. In addition, a method of selecting a question that minimizes the loss function (Equation 12) at the time of environmental adaptation can be considered. However, the loss function does not always match the recognition performance, and the amount of calculation at the time of adaptation is increased. . Therefore, the clustering unit of the weighting factor of the stream for each phoneme is divided at the time of environment adaptation using the tree structure created in advance.

Next, the configuration and operation of the environment adapting apparatus 300 shown in FIG. 8 will be described below. In FIG. 8, the environment-adaptive AV word data memory 51 has a format similar to that of the learning AV data memory 31 of FIG. 1, and includes a plurality of utterance voice signals and an image signal of the lips of the speaker at the time of utterance. The word AV signal data is stored in the environment adaptation AV word data memory 51 with a phoneme label. The speech recognition apparatus 200 shown in FIG. 1 performs speech recognition based on the AV signal for environment adaptation instead of the AV signal data in the input AV data memory 41 shown in FIG. Processing unit 5
Output to 0. Then, the environment adaptation processing unit 50 executes the processing shown in FIG.
Of the stream weight coefficient of the stream.
Specifically, the weighting factor of each phoneme in the synthesized HMM of the learned speech and image generated by the HMM learning unit 17 in FIG. 1 is determined by a predetermined value such as a clustering tree having a binary tree structure shown in FIG. Is clustered into a plurality of classes using the clustering criterion, and the weighting factor of each phoneme belonging to each class is reduced based on the log likelihood calculated by the voice recognition unit 26 in the voice recognition device 200. (Specifically, so that the misclassification measure d _x shown in Expression 6 is reduced) so that the combined H
Environmental adaptation of MM. Here, the re-learning of the environment adaptation process is repeated so that the number of environment adaptation signal data of each class becomes less than a predetermined threshold value. Then, the speech recognition device 200 in FIG. 1 performs a speech recognition process using the re-learned synthesized HMM.

FIG. 9 is a flowchart showing the process of adapting the environment of the stream weighting coefficients performed by the environment adaptation processing unit 50 of FIG.

In FIG. 9, first, in step S1, the initial stream weighting factor for each phoneme is set to be the same for all HMMs (initialization processing). That is, for all HMMs, the weighting factor of each phoneme stream at the root node of the clustering tree is set to one class, and is initialized to, for example, 0.5. Then
In step S2, for each node in the next lower hierarchy in the clustering tree, the weighting factors of the HMM streams belonging to the same cluster are clustered into one class, and in step S3, the initial weighting factor of the weighting factor of each phoneme stream is clustered. Let the value be the value estimated in the upper layer. Then, in step S4, it is determined whether or not the number of adaptive data of each class <the threshold value (for example, 20). Here, if YES, it is determined that clustering has been sufficiently performed, and in step S5,
The process proceeds to step S7, using the weight coefficient of the stream as the estimated value of the estimated constant. On the other hand, if NO is determined in the step S4, it is determined that the clustering is not sufficiently performed, and in step S6, the weight coefficient of the stream is set as the estimated value of the variable update target, and the process proceeds to step S7. In step S7, it is determined whether or not the process of step S4 has been performed for all the classes. If NO, the process returns to step S4 and executes the process of step S4. On the other hand, if YES is determined in the step S7, it is determined whether or not there is a weight coefficient of the stream to be updated in a step S8. If YES, in a step S9, the GPD algorithm is used (as described above). After the update of the stream by a predetermined number of times (using the formula) and updating of the weighting factor of the stream, the combined HMM stored in the combined HMM memory 34 is environment-adapted.
Return to 2. On the other hand, if NO in step S8, the environment adaptation process ends.

The reason for using this method is that a stream is estimated in order from the root node and is used as an initial value, so that a stable solution is estimated. A good HMM adaptation is performed. However, the calculation time increases as the depth of the tree increases, and becomes enormous. In the present embodiment, the first purpose is to confirm the effectiveness of the division, the tree depth is set to a maximum of 2, and the GPD
The number of repetitions n of the processing by the algorithm was set to a maximum of eight.

In the above embodiment, a clustering tree is used as a reference for clustering. However, the present invention is not limited to this, and another reference such as a predetermined reference formula may be used.

[0068]

Example <Example of Second Embodiment> The present inventors
As an evaluation experiment for the environment adaptation apparatus according to the second embodiment, a recognition experiment of 200 words × 2 sets was performed. As an evaluation, an average of two sets of word recognition rates was used. Table 2
Shows the experimental conditions. In this experiment, a database of audio data in which a specific speaker (one female speaker) uttered 5,240 words in an utterance list owned by the present applicant was used in an acoustic laboratory.

[0069]

[Table 2] Experimental conditions ――――――――――――――――――――――――――――――――― Voice Sampling frequency: 12 kHz Analysis window function : Hamming window Frame length: 32 msec Frame shift: 8 msec Parameter: MFCC 16 dimension, ΔMFCC 16 dimension ―――――――――――――――――――――――――――――――― ――― Image Frame shift: 33 msec Preprocessing 1: Conversion from RGB signals to 256-level grayscale image signal Preprocessing 2: Histogram flattening Preprocessing 3: Lip position normalization Processing Parameter: Smoothed log power spectrum 35 dimensions and smoothed logarithmic Δ power spectrum 35 dimensions ――――――――――――――――――――――――――――――――― HMM status Number 3 Audio, Image 2 ―――――――― ―――――――――――――――――――――――――― Probability density function Gaussian distribution: 2 mixture HMM Phoneme environment independent 55 phoneme model ――――――――― ―――――――――――――――――――――――――― Learning data Voice and image synchronization data One female speaker, 4740 words ――――――――― ―――――――――――――――――――――――――― Test data 200 words (2 sets) (open condition) ―――――――――――― ――――――――――――――――――――――― Adaptive data Word data other than training data and test data ―――――――――――――――― ――――――――――――――――――― Recognition dictionary at the time of adaptation 500 words dictionary containing vocabulary of test set ―――――――――――――――― ―――――― ------------

Since the frame shift of the image is longer than that of the sound, the same frame is embedded in the image, and the frame shift of the sound and the image is adjusted. The recorded image data shows differences in lighting conditions, face inclination, and the like depending on the utterance word. Therefore, as preprocessing, histogram flattening and lip position normalization were performed so as to minimize the difference in luminance from the reference frame.

To create the audio HMM, an MFCC was obtained from clean audio data recorded in an acoustic laboratory, and a model was created using the MFCC as a feature vector. Also, the image H
The MM performs a two-dimensional FFT on the preprocessed image to obtain a logarithmic power spectrum. Then, the frequency range is set to 6
A × 6 area division was performed, and a model was created using the smoothed logarithmic power spectrum of the area excluding the DC component as a feature vector. In this experiment, the synthesized HMM of voice and image is
Learning is performed with a weight coefficient of each stream being equal to 1: 1.

As a comparison, a recognition experiment was performed in which only voice, only image, and voice and image were initially integrated. The experiment using only speech used a 3-state HMM, the experiment using only images used a 2-state HMM, and the initial integration method used a 3-state HMM. The shape of each HMM is ref from left to right.
It is a t-to-right type.

As the experiment conditions at the time of environmental adaptation, word utterance data other than learning data and test data was used as adaptation data. Therefore, the adaptation data differs from the test data in the utterance content. In addition, the number of adaptive data is set to 15, 2,
Stream weighting factors were estimated for 5, 50, 75 and 100 words. However, when the number of adaptive data is 15 words, the weighting factors of the streams estimated based on the utterance contents are significantly different. Therefore, when the number of adaptive data is 15 words, the average of the recognition rates for three sets of adaptive data is used. As a dictionary at the time of adaptation, a dictionary of 500 words including words of adaptation data and words of test data was used.

In Equation 11 of the misclassification measurement degree, the number of error candidates is N = 1, and in Equation 12 of the GPD algorithm, α is 0.1. In Equation 14, when the weighting factors of all the streams are clustered, ε _k = 200 / k, and after dividing the unit of the clustering of the weighting factors of the stream, ε _k = 100 / k.
And converges more slowly than when the weighting factors of all the streams are clustered.

Next, the results of experiments for environmental adaptation will be described below.

First, the recognition rates of the synthesis integration method and other integration methods are compared. FIGS. 11, 12 and 13 show the initial integration method and the synthesis integration when the weight factor of the audio stream is changed so that the weight coefficient of the audio stream and the weight coefficient of the image stream satisfy Equation 17 above. The recognition result of the law is shown. In addition, the recognition rates of only voice and only image are also shown. Here, FIG. 11 is a recognition result when white Gaussian noise is added to the voice so that the SNR becomes 10 dB.
FIG. 12 shows a recognition result when the SNR is 20 dB. FIG. 13 shows a recognition result when no noise is added to the recorded data. 11 to FIG.
3 shows the value of the weight coefficient of the stream estimated by the GPD algorithm from the adaptive data of 50 words. However, the weight coefficients of the estimated streams are for the synthesis integration method in which relearning has been performed.

From FIGS. 11 to 13, it can be seen that the bimodal speech recognition system tends to have a peak in the recognition rate at the value of the weight coefficient of a certain stream, and by estimating this peak, the bimodal speech recognition system is higher than the single-modal speech recognition system. It can be seen that recognition performance can be obtained. Then, it can be seen that the value of the weight coefficient of the stream close to the peak of the recognition rate can be estimated by the GPD algorithm. Also, a synthesis integration method for re-learning a synthesized HMM of synthesized voice and image (with re-learning)
It can be seen that the recognition performance obtained is higher than the initial integration method and the synthesis integration method without re-learning (no re-learning). This is because the initial integration method assumes that the audio and the image are synchronized, and the synthesis integration method that does not re-learn does not learn the synchronization, but the synthesis integration method that re-learns does not synchronize the audio and the image. This is probably because they are learning the relationship. Also, in the preliminary experiment, if the learning is performed with the voice and image data by simply increasing the number of states of the HMM and changing the shape without synthesizing the voice and the image HMM, the parameter estimation is not successful, and the synthesized HMM
Higher performance was not obtained than those learned based on. From this, it is considered that a good initial model can be given by synthesizing the voice and the image HMM. Further, even when learning is not possible, the composite model can be used as an initial model as it is.

Next, an experimental result in the case where the weight coefficient values of the audio and video streams are estimated by the GPD algorithm without dividing the weight coefficients of the streams will be considered. Table 3
When the sound is clean and the SNR of the sound is 20d
B shows the recognition rate when the environment adaptation is performed without dividing the weight coefficients of the audio and video streams when white Gaussian noise is added so as to be 10 dB. The adaptation data is selected at random.

[0079]

[Table 3] Word recognition rate results when weighting factors of audio and video streams are adapted to the environment ――――――――――――――――――――――――――――― ―――――― Number of adaptation data Clean with no noise SNR = 20dB SNR = 10dB (phonemes) ――――――――――――――――――――――――――― ―――――――― 15 words (108) 96.86% 77.15% 56.35% 25 words (193) 97.28% 89.36% 69.06% 50 words (366) 97.28 % 87.38% 68.57% 75 words (521) 97.03% 83.42% 65.60% 100 words (697) 97.03% 87.38% 68.81% ――――――― ――――――――――――――――――――――――――――

As is clear from Table 3, when the number of adaptive data is 15 words, the recognition rate is low. This is because when estimating the weight coefficients of the audio and video streams from the small number of adaptive data, the values deviate from the optimal stream weight coefficient values for the test set. Therefore, in order to examine how much the recognition rate changes depending on the content of the adaptation data, three recognition experiments were performed for the case where the adaptation data was 15 words and 50 words, and the variance of the recognition rate was examined. When the number of adaptive data is 15 words, the standard deviation of the recognition rate is 10.18, and the recognition rate varies depending on the utterance contents of the adaptive data. On the other hand, the standard deviation of 50 words was 0.57, and there was almost no variation in the recognition rate due to the difference in the adaptation data. Here, the standard deviation is an average value when the voice is clean without noise, when SNR = 20 dB, and when 10 dB. Therefore, when estimating the weighting factors of the audio and video streams from a small number of adaptive data, it is understood that the utterance content of the adaptive data must be carefully selected. Also, from Table 3, it can be seen that the larger the number of adaptive data, the more the appropriate stream weighting factor is estimated. Finally, consider the experimental results when the stream weighting factor is divided into two.

Table 4 shows that the unit of clustering of stream weighting factors is divided into two by changing the number of adaptive data.
The recognition rate when the weight coefficient of the audio and image streams is estimated is shown. Note that the same adaptive data as those in Table 3 are selected. In the experiment, the threshold of the number of adaptive data to be updated is not limited. Accordingly, the weight coefficient classes of all the streams have been updated.

[0082]

[Table 4] Word recognition rate results when the weighting factors of audio and video streams are divided into two parts ――――――――――――――――――――――――――――― ―――――― Number of adaptation data Clean with no noise SNR = 20dB SNR = 10dB ――――――――――――――――――――――――――――― ―――― 15 words (58,50) 97.03% 77.39% 61.96% 25 words (104,89) 97.52% 89.36% 68.57% 50 words (197,169) 97.28 % 87.87% 66.34% 75 words (266,255) 97.03% 83.91% 65.85% 100 words (365,332) 97.28% 87.38% 68.57% ――――――― ―――――――――――――――――――――――――――― (Note) The number of adaptive data (,) is (the number of phonemes of voiced sound, unvoiced sound) It represents the number of phonemes).

When Table 3 and Table 4 are compared, the adaptive data is 50
Above words, the recognition rate may be slightly higher, but there is not much difference. 15 adaptive data
When the word is a word, the recognition rate is higher than when the weight coefficient of the stream is not divided. This is because, by dividing the unit of the clustering of the weighting factor of the stream, the value of the class of the weighting factor of one stream is estimated to be a value close to the optimal value for the test set. This is because the number of repetitions has increased.

As described above, according to the present embodiment, voice and image information are synthesized and integrated using the HMM,
Further, the weight coefficient of the stream of the synthesized and integrated HMM was adapted to the environment. As a result, it is possible to generate a re-learned composite HMM that can establish synchronism between speech and images and reduce erroneous recognition during speech recognition. Further, better speech recognition performance can be obtained as compared with the conventional example and the first embodiment.

In the first and second embodiments described above, each operation or processing unit 11-17, 21-26, 50
And the like, and may be constituted by a hardware circuit or may be constituted by a software program. Each of the memories 31-34, 41, 42,
Reference numeral 51 denotes a storage device such as a hard disk memory.

[0086]

As described above in detail, according to the speech and image synthesis model generating apparatus of the present invention, the speech HMM,
The image HMM is calculated by calculating the product of the output probabilities of the voice and the image in all combinations of the states of these two HMMs, and generating a synthesized HM including the product of the output probabilities in each state.
After generating and synthesizing M, based on the generated synthesized HMM, connection learning is performed using the labeled AV signal stored in the first storage means so as to maximize the output likelihood. Thus, a synthesized HMM of the learned voice and image is generated. Therefore, it is possible to establish synchronism between the voice and the image, and to generate a learned synthesized HMM having the synchronism between the voice and the image.

According to the speech recognition apparatus of the present invention, the synthesized speech and image synthesis H based on the extracted speech speech signal feature and image signal feature are extracted.
By performing voice recognition using the MM, voice recognition can be performed with a higher voice recognition rate than the initial integration method and the result integration method according to the related art.

Further, according to the environment adapting apparatus for a combined voice and image model according to the present invention, the likelihood when speech recognition is performed on the environment adapting signal data using a predetermined HMM is calculated. The weighted coefficients of each phoneme in the learned synthesized HMM are clustered into a plurality of classes using a predetermined clustering criterion, and the weighted coefficients of each phoneme belonging to each class are calculated based on the calculated likelihood. By performing re-learning so as to reduce erroneous recognition, the combined H
Environmental adaptation of MM. Here, preferably, re-learning of the environment adapting means is repeated so that the number of environment adaptation signal data of each class becomes less than a predetermined threshold value. Therefore, it is possible to generate a re-learned composite HMM that can establish synchronization between voice and image and reduce erroneous recognition during voice recognition.

Further, according to another speech recognition apparatus according to the present invention, speech recognition is performed using the synthesized HMM of speech and image which has been environment-adapted by the environment adaptation apparatus. The speech recognition can be performed at a higher speech recognition rate than the initial integration method, the result integration method, and the above-described synthesis integration method.

[Brief description of the drawings]

FIG. 1 shows a speech and image synthesis model generation apparatus 100 and a speech recognition apparatus 2 according to a first embodiment of the present invention.
FIG. 2 is a block diagram showing a configuration of a 00.

2A is a state transition diagram illustrating an example of a voice HMM in a voice HMM memory 32a in FIG. 1, and FIG. 2B is a state transition diagram illustrating an example of an image HMM in a image HMM memory 32b in FIG. FIG.

FIG. 3 is a combined HMM in a combined HMM memory 33 of FIG. 1;
FIG. 6 is a state transition diagram showing an example of.

4 is a synthesized model generation device 1 for audio and images in FIG.
It is a figure which shows the three-dimensional search space of the synthesis integration in 00.

FIG. 5 is a graph showing experimental results of the speech recognition apparatus according to the conventional example and the first embodiment.

FIG. 6 is a graph showing experimental results of the speech recognition apparatus according to the conventional example and the first embodiment.

FIG. 7 is a graph showing experimental results of the speech recognition apparatus according to the conventional example and the first embodiment.

FIG. 8 is a block diagram illustrating a configuration of an environment adaptation apparatus 300 according to a second embodiment of the present invention.

9 is a flowchart showing a stream weighting coefficient environment adaptation process executed by the environment adaptation processing unit 50 of FIG. 8;

10 is a diagram illustrating a tree structure of a binary tree that is a reference when clustering the HMMs in the environment adaptation processing of FIG. 9;

FIG. 11 is a graph showing experimental results of the conventional example and the speech recognition devices according to the first and second embodiments.

FIG. 12 is a graph showing experimental results of the conventional example and the speech recognition devices according to the first and second embodiments.

FIG. 13 is a graph showing experimental results of the conventional example and the speech recognition devices according to the first and second embodiments.

[Explanation of symbols]

 11: Data separation unit, 12: Synchronization unit, 13a, 13b: Preprocessing unit, 14a, 14b: Feature extraction unit, 15a: Voice HMM generation unit, 15b: Image HMM generation unit, 16: HMM synthesis unit, 17 ... HMM learning unit, 21: data separation unit, 22: synchronization unit, 23a, 23b: preprocessing unit, 24a, 24b: feature extraction unit, 25: feature synthesis unit, 26: speech recognition unit, 31: learning with phoneme label AV data memory for use, 32a: voice HMM memory, 32b: image HMM memory, 33: synthesized HMM memory, 34: learned HMM memory, 41: input AV data memory, 42: word HMM memory, 50: environment adaptation Processing unit, 51: AV word data memory for environment adaptation, 100: Synthetic model generation device for voice and image, 200: Voice recognition device.

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) G10L 15/24 G10L 3/00 521C 571Q (72) Inventor Satoshi Nakamura 2-chome Kodai, Seika-cho, Soraku-gun, Kyoto Prefecture 2nd 2 AT R Co., Ltd. Spoken Language Communication Research Laboratory F-term (reference) 5D015 GG01 GG03 HH23 LL07 5L096 BA16 BA18 JA11 JA16 KA04

Claims (5)

[Claims] A first storage unit for storing an AV signal including an uttered voice signal and an image signal of a lip of a speaker at the time of utterance; and an output likelihood based on the uttered voice signal of the AV signal. First generating means for generating a hidden Hidden Markov Model so as to maximize the degree, and generating an image Hidden Markov Model based on an image signal of the AV signal so as to maximize the output likelihood. A second generation unit that stores the speech hidden Markov model generated by the first generation unit; and a second storage unit that stores the image hidden Markov model generated by the second generation unit. 3, a hidden Hidden Markov Model stored in the second storage means, and an image Hidden Markov Model stored in the third storage means, for each state of the two hidden Markov models. Synthesizing means for calculating the product of the output probabilities of the voice and the image in each combination and generating a synthesized hidden Markov model that includes the product of the output probabilities in each state; Based on the Markov model, by using the labeled AV signal stored in the first storage means and performing joint learning so that the output likelihood is maximized, a synthesized hidden Markov model of the learned speech and image is obtained. And a learning unit for generating a composite model of voice and image. 2. The method according to claim 1, further comprising: determining a feature amount of the speech sound signal and a feature amount of the image signal based on an input AV signal including the speech sound signal and an image signal of a speaker's lip at the time of speech. 2. A learned voice generated by the voice and image synthesis model generating apparatus according to claim 1, based on extraction means for extracting, and the extracted voice voice signal feature amount and the image signal feature amount. And a first speech recognition means for performing speech recognition using a synthesized Hidden Markov Model of an image and outputting a speech recognition result. 3. Fourth storage means for storing environment adaptation signal data for storing an audio-video signal including an uttered voice signal and an image signal of a lip of a speaker at the time of utterance with a phoneme label, and the storage means. A second speech recognition unit for calculating a likelihood when speech recognition is performed on the obtained environment adaptation signal data using a predetermined hidden Markov model; and a speech and image synthesis model generating apparatus according to claim 1. The generated weighted coefficients of the phonemes in the synthesized speech and image synthesized Hidden Markov Model are clustered into a plurality of classes using a predetermined clustering criterion, and the weighted coefficients of the phonemes belonging to each class are calculated as described above. Based on the calculated likelihood, re-learning so as to reduce misrecognition by environmental adaptation means for adapting the composite hidden Markov model to the environment. That environmental adaptation device for the synthesis model of speech and image. 4. The voice and image according to claim 3, wherein re-learning of said environment adapting means is repeated so that the number of environment adaptation signal data of each class becomes less than a predetermined threshold value. Environment Adaptation System for Composite Model 5. A feature amount of the speech sound signal and a feature amount of the image signal based on an input AV signal including a speech sound signal and an image signal of a lip of a speaker at the time of speech. An extracting means for extracting, and an environment adapting device for a speech and image synthesis model according to claim 3 or 4, based on the extracted feature amount of the speech voice signal and the feature amount of the image signal. A speech recognition apparatus comprising: a third speech recognition unit that performs speech recognition using an adapted synthesized Markov model of speech and an image and outputs a speech recognition result.