DE4435272C2 - Method and device for extracting a visual feature vector from a sequence of images and speech recognition device - Google Patents

Description

The invention relates to a method for extracting a visual Feature vector from a sequence of images according to the generic term of Claim 1 further relates to an apparatus for extracting a visual feature vector from a sequence of raster-scanned Video images according to the preamble of claim 6. Further relates the invention relates to the field of machine speech recognition and is particularly concerned with facial feature extraction to improve acoustic speech recognition with the help of Ge visual characteristics in connection with acoustic data in one ronal network classification system.

The goal of automatic or machine speech recognition is in designing a system or device so that it is human Ability is approximated to spoken language when changes in the language Emphasis on the speaker, gender, speaking speed, to understand the degree of articulation, even if disturbing background noises and noise distract the system. The currently existing automated devices are less accurate and robustness than what is necessary to the very high Requirements in such use cases as a computer  Speech-to-text conversion, automatic translation and at control systems based on language are sufficient. Typical Lö solutions contain hidden Markov models, at which transition probabilities in links between nodes (States) are encoded, which represent phonetic segments and "Blackboard" methods, in which several special phonologi Czech, lexical and grammar based subsystems be linked to work synergistically to speech recognizers maximize results. Lately there are neural networks works with some success in limited areas as described by A. Waibel in an article with the Ti tel "Modular Construction of Time-Delay-Neural Networks for Speech Recorgnition ", which is described in Neural Computation 1, Stn. 39 to 46 (1989).

A predicted source of information and any restrictions conditions that are reasonably placed in an artificial system that can generally increase the detection accuracy and it it is therefore desirable that it be used in a speech recognition system are included. Usually, research has mainly focused on that Inclusion of high level linguistic information, like grammatical and syntactic data. People can also use information other than the acoustic signal turn to improve understanding. For example, use hearing impaired or hearing impaired people often visual information for "speech reading" to improve recognition accuracy. (See, for example, Dodd, B. and Campbell, R. (eds.), "Hearing by Eye: The Psychology of Lipreading ", Hillsdale, N.J., Lawrence Erlbaum Press (1987); or DeFilippo, C.L. and Sims, D.G. (eds.), "New Reflections on Speechreading ", special issue of The Volta Review 90 (5), (1988).

A voice reading can provide direct information about language segments and Phonemes as well as the speaking speed, the gender of the Speakers and their identity as well as subtle information manage to separate speech from background noise. The be  knew "Cocktail Party Effect", in which language, which by Noise from the crowds has been falsified, noticeably becomes more understandable when you see the speaker's face can be a strong indication that people have visual information Use speech recognition.

Several speech reading and face classification systems have been described recently, namely

• a) Petajan, E.D., et al., "An Improved Automatic Lipreading System to Enhance Speech Recognition ", ACM SIGCHl-88, Stn. 19 to 25, (1988);
• b) Pentland, A., et al., "Lip Reading: Automatic Visual Recognition of Spoken Words ", Proc. Image Understanding and Machine Vision, Op tical Society of America, June 12 to 14 (1984) and
• c) Yuhas, B.P., et al., "Integration of Acoustic and Visual Speech Signals Using Neural Networks ", Nov. 1989, IEEE Communications Magazine (1989).  
• d) R. Buhr, "Analysis and Classification of Facial Images", ntzArchiv Bd. 8 (1986) H. 10, pp. 245-256.
• e) Kaneko et. al., U.S. Patent No. 5,136,659
• f) Welsh et. al., U.S. Patent No. 4,841,575

Petajan, et al. used thresholded images (pixels) of a speaker's face while creating a word together with a dictionary of previously saved and marked Speeches and a standard distance classifier for visual Er identifier.

Pentland, et al. used an optical gradient technique to measure the Ge dizziness of the upper lip, lower lip and the two corners of the mouth from the unprocessed pixel video image of the mouth. she then used a principal component analysis and a mini mum spacing classifier with three or four digit phrases.

Yuhas, et al., Trained a neural network using static Images of the mouth shape for vowel recognition along with one Control unit with free parameters to determine the relative weights of visual and acoustic contributions to the best recognition when Presence of different levels of acoustic noise  to deliver.  

Buhr uses knowledge-based image segmentation to extract specific ones Pixels, from their coordinates feature vectors for the classification of facial images be formed.

Kaneko et. al. applies threshold processing to a digitized face image, to determine a small number of parameters from the binary image so determined, which are characteristic of the facial image.

Welsh et. al. uses the so-called Nagao method, which uses a column value preprocessed digitized facial image is summed up from the sum values characteristic points in the picture and from them code words, which different Specify mouth shapes to determine.

According to the invention, the performance of speech recognition system, which is only acoustic or visual Lip position information especially in an unfavorable ge use a smoky environment. Furthermore, a new time-delayed neural network speech reading system for classifying Aus language and ways of speaking created only with the help of visual data will. Furthermore, an acoustic speech recognition system is said to be verbal be replaced by a delayed neural network using visual speech recognition system in connection with the acoustic speech recognition system is used.

The classifier is also intended to have a continuous stream of related, visual and acoustic data are supplied from which the acoustic way of speaking is detected and classified that can. In addition, a time-delayed neural network Classifier architecture can be created which is a gradient gradients or other learning methods, such as re-propagation training (backpropagation training). Furthermore, an acoustic and vi source signal processing before the delayed neural Network classifier can be created.

In addition, an initial classification, i.e. H. a probability distribution, which is likely represents every possible candidate spoken in one predetermined time interval has occurred. In addition, a simple process for extracting dynamic, visual Ge visual information using a selected group of facial marks are created.

According to the invention, this is in a method for extracting egg a visual feature vector from a sequence of images after the The preamble of claim 1 characterize the features in Part of claim 1 achieved. Advantageous further developments are Ge  subject matter directly or indirectly related to claim 1 gene claims 2 to 5. Furthermore, this is with a device for Ex trace a visual feature vector from a sequence of image data according to the preamble of claim 6 by the features in the kenn drawing part of claim 6 achieved. Advantageous further training conditions are the subject of claim 6 directly or indirectly related claims 7 to 9. In addition, this is with a Speech recognition system or device according to the preamble of claim 10 the characteristics achieved in its characteristic part. Beneficial Developments are the subject of claim 10 directly or indirectly related claims 11 to 14.

According to the invention, a speech recognition system is created in which processes both acoustic and visual speech signals to the probability of a correct classification of the Increase speech or pronunciation.

Furthermore, a speech recognition system for recognizing speech point to a previously set group of allowed channels didate speech modes include an acoustic feature extraction Facility, a dynamic, visual feature extraction facility and a classifying device comprising a neural network. The acoustic feature extractor converts acoustic speech signals that represent a way of speaking into a corresponding one spectral feature vector set. The dynamic, visual characteristic Extracting device converts the dynamic facial features that are assigned to the generation of acoustic speech, in one dynamic, visual feature vector set around. The one neural Network-wide classifying facility transforms the dynamic, acoustic and visual feature vectors into a conditional probability distribution around the the probability of each candida ten-way of speaking that has been spoken by the be observed acoustic and visual data have been specified.

The invention based on preferred embodiment  shape with reference to the accompanying drawings explained. It shows:

Figure 1 is a block diagram of an acoustic and visual dynamic speech recognition system.

Fig. 2, the placement of visual comparison marks or markings on and around the lips of the speaker;

Fig. 3 is a front view of a speaker face and physiological identifier;

FIG. 4 shows the geometry for introducing a symmetry axis and ROI (regions of interest);

Figure 5 is a pixel-defined boundary for the left eye area.

Figures 6 (a) to 6 (d) show a vertical gray scale profile of the closed and open mouth and its relationship to the facial features;

Figures 7 (a) and 7 (b) show a horizontal gray scale face profile of the closed and open mouth along the mouth line;

Figures 8 (a) and 8 (b) show the significant peaks and valleys of the vertical and horizontal grayscale profiles;

Figure 9 shows a reduced set of data points used in generating a visual feature vector;

FIG. 10 is a speech classification system using video images without trust indicator;

Fig. 11 countries, the relationship between the steady-Mel Frequenzbän and the corresponding non-uniform, linear frequency bands;

Figures 12 (a) and 12 (b) show two equivalent designs of a time-delay, single-layer neural network (TDNN);

Fig. 13 (a) and 13 (b) the electrical analogue of a typical Ausspra che together with their spectogram;

Fig. 14 only the architecture of a visual (VO) multi-ply TDNN;

FIG. 15 is a hardware one TDNN architecture;

FIG. 16 is a block diagram of detection system audible and visual Spracher which links the output signals of separate AKU stischen and visual recognition systems;

Fig. 17 is a hardware of an embodiment of a AxV-TDNN- speech classifier;

FIG. 18 is a hardware of an embodiment of another TDNN- speech classifier;

Figure 19 is a hardware of an embodiment of a full AV speech classifier.

FIG. 20 is the architecture of a typical McCulloch-Pitts neuronal cell;

Fig. 21 (a) to 21 (c) examples of data transfer activation function characteristic;

FIG. 22 is the architecture of a typical sigma Pi (πII) artificial ronalen new cell;

Fig. 23 is a set of learning curves showing the training progress, and

Fig. 24 (a) to 24 (d) a group of Confusion matrices for four sy stems.

Now several embodiments of speech recognition systems described systems, each system being trained to be a to recognize limited amount of speech or pronunciations. The different embodiments differ in the use type of information and / or how the information signa le are processed. In any case, it is the goal of speech recognition system, a probable at the classifier output distribution that each of the possible candidate Speaking styles is assigned.

In view of the clarity of the description that follows, spe used specific examples of the embodiment according to the invention, to recognize one of ten candidate ways of speaking. Here should however, it should be noted that this is not necessarily a Be restriction to a number or certain types of speech or expressions that can be reconciled.

The present invention is a speech recognition system in which dynamically changing acoustic and visual signals are used for improved speech recognition efficiency. Fig. 1 is a simplified block diagram showing one embodiment. Acoustic speech data are recorded by means of a microphone 28 which, via an output line 22, applies the electrical signal replicas of the acoustic information to a spectrum analyzer, from which the acoustic energy is applied to an arrangement of 14 filters. The briefly temporally changing energy which is emitted by each of the 14 filters is measured, normalized and applied to the 14 output lines 25 as a vector with 14 components which changes over time.

At the same time, in one embodiment, a video camera 27 captures the associated visual data generated from a set of markings used to describe lip and mouth movement, as evidenced by a given speech or pronunciation has been generated. The untreated video information is delivered to a lip and mouth position vector generator 14 , which normalizes the temporally changing position of the lip and mouth markings for a head inclination and size, which results in a temporally changing vector on the five output lines 15 results, which describes the lip and mouth activity, which is assigned to the acoustic speech.

The lip and mouth position vector generator 14 has a tracking system 16 to track the position of light emitting or light reflecting fiducial marks placed on and around the speaker's lips, and a vector generator 19 that has selected distances between fiducial marks ex tracted and standardized. This facility was implemented using the computer-based, two-dimensional motion analysis system manufactured by Motion Analysis Corporation, 3650 North Laughlin Road, Santa Rosa, California 95403 under the trademark "Expert Vision".

The Expert Vision video tracking system uses a video setup to capture and record the movement of objects. Right reflective marks are placed at points of interest. The system can capture and analyze data from a video camera (or tape) with a video standard frame rate of 60 Hz up to 200 Hz have been obtained.

The unprocessed video data from the video camera is fed to a video processor subsystem which isolates the marker images and converts them in real time into digitized video outlines. The video processor provides the function of the tracking system 16 by generating x and y coordinates of the marking edges in the host computer for subsequent processing. The detected screen positions are expressed in video frame rate intervals in a rectangular grid that is 256 pixels wide (in the x direction) and 240 pixels high (in the y direction).

The video outline is reduced to focal points using the host computer software. The center of gravity of each reflective marking is determined by averaging the coordinates of the individual pixels that form the outline of the marking. The focal points of each marker are marked from picture to picture and connected to form a trajectory. Additional host computer software is used to generate and normalize the set of visual distance vectors, as will be described below and functionally indicated by the vector generator 19 of FIG. 1.

The two time-changing vectors on lines 15 and 25 are applied to a voice classifier 200 having a time-delaying neural network ("TDNN"), the neural network generating ten output signals on lines 17 . Each of the output lines 17 is exclusively assigned to one of ten possible speeches or pronunciations, so that the TDNN 200 is trained to recognize them. The signal size on each of the ten lines represents the probability that the associated speech or pronunciation has been made. An output network 18 (shown in dashed lines) is an optional (known) one-of-ten classifier that selects the most likely pronunciation by choosing the pronunciation or speech that is the line with the highest level (probability) of the ten Lines 17 corresponds. Since the TDNN classifier 200 can check a data stream that changes over time, precise knowledge of the time position of the acoustic and visual information is not necessary.

Fig. 2 shows the placement of the ten reflective markings (M0, 9) on the respective speaker face, which were used in one embodiment ver. The positions of the ten comparison marks were the tip of the nose (M0), the center of the chin (M1), the left corner of the mouth (M2), the right corner of the mouth (M3), the top center of the lips (M4), the bottom center of the lips (M5) and four places on the lips, approximately half between the center of the upper lip and the left and right corners of the mouth (M6, M7) and between the center of the lower lip and the left and right corners of the mouth (M8, M9 ) are. This set of benchmarks was used to determine the general shape and movement of the lips and mouth area. This special arrangement has the great advantage of simplicity, in comparison to the computationally complex problem of deriving the same information from the enormous amount of data that has been generated by following video pixel images.

The position of the reflective fiducial marks on each face of a speaker was recorded by a video camera 27 at a 60 Hz frame rate and then transformed by a position vector generator 14 into visual feature vectors which have five components as shown in the table below .

This special selection of vector component features was made so that the different distances across the mouth and face can be made independent of any movements and rotations of the head. This was done with the vector generator 14 by first turning the head so that a line drawn through the fiducial marks M0 and M1 was vertical, then the three vertical distances (M0-M1, M4-M5 and the average distance from M6-M8 and M7-M9) and the two horizontal distances (M2-M3 and the average distance from M6-M7 and M8-M9).

A further standardization of the component characteristics was due to cross Object differences in face dimensions required. As well it was noticed that the mouth of each speaker was sometimes in one partially open, comfortable and relaxing position. At the Nor step was a lot of characteristic facial dimensions the size of each speaker's face Averaging the vectors generated all the rested position time correspond to ten. Then each visual vector subsequently became nor by subtracting the mean rest from each component and the size was scaled so that each component value a value of plus or minus one is assigned if its level plus or minus two standard deviations from the average corresponds.

Although the specific group of facial enumerated above Motion features was used to identify speech in the Reducing practice could also be characteristic of others Sets of dynamic dimensions are used to measure the we to convey substantial visual information. This could become one another, yet effective, dynamic visual data vector. For example, a video camera could have a dynamic pixel image of the speaker's mouth and lips. The dynamic form could be measured by measuring the distance between the shadows under the Lower and upper lips are measured, which are indicated by an over headlights are caused. A lot of such dimensions  across the mouth would describe the dynamic shape.

A system or device for extracting dynamic facial feature vectors from Video camera images that are mathematically simple and only from one Gray scale (light and shadow) information depends is implemented been.

This system uses an NTSC (National Television Standards Committee) camera to capture and capture the front view of a speaker's face, which is illuminated by an overhead light source. The gray-scale video images are recorded by the video camera and broken down into sections so that a (150 × 150) pixel image results at 30 frames (e.g. images) per second. FIG. 3 shows a front view image of a face illuminated from above and the larger shadows and highlights that have been used for facial feature extraction, and which include: orbital area shadow 10 , a nasal highlight area 20 one Nose and nostril shadow area 30 , an upper lip highlight area 40 , a shadow mouth area 50 , a lower lip highlight area 60 , a lower lip shadow area 70 and an upper chin highlight area 80 . The eye shadow regions 10 lie along an eye line 11 , which runs perpendicular to the axis of symmetry 9 , which divides the front view into two parts. The mouth line 51 runs through the middle of a Mundschat tenbereich 50 at right angles to the axis of symmetry.

Critical language-related drop shadows are the nose area 20 , the upper lip area 40 , the lower lip area 60 and the chin area 80 together with the respectively adjacent shadow areas 30 , 50 and 70 . To set these critical features, area shadows 10 are used to create an eye line 11 . Before performing facial feature extraction, it has been found advantageous to preprocess the video images to reduce spatial noise using a two-dimensional (3 × 3) pixel convolution kernel with a Fourier transform low-pass filter characteristic.

Next, an edge sharpening filter with a (3 × 3) pixel trap core used to regenerate the shadow-light transitions ren that have been "softened" by the low pass filter operation. This operation is followed by an anti-aliasing filter with an effekti ven (9 × 9) pixel convolution kernel to make an image with smooth transitions like to manufacture. A final preprocessing step includes a time smoothing over groups of three adjacent video image frames a. Typical convolution cores used for preprocessing pixel image frames The following have been used:

• 1) a spatial Gaussian (3 × 3) filter core with a standard deviation of 0.391 pixels with the following distribution values:
• 2) a spatial (3 × 3) edge enhancement filter core with the following distribution
• 3) a spatial (16 × 16) low pass that has been implemented by repeating an order 1 binomial filter (1,1) 15 times long and perpendicular to each grid line, which is ei effective core with an approximate Gaussian weight quantity results, and
• 4) a temporal filter core in order to match corresponding pixel elements filter three neighboring ones with core weights of (1,4,1).

Since these three filter operations only include linear processes, the various filter operations in any order turned or even combined in a two-step process, the one edge-sharpening filter step and one, the Rau  has reducing or smoothing filter step or can can also be linked in a single filter step, in which a only (9 × 9) pixel convolution kernel is used which results when the three two-dimensional nuclei in a single equivalent Filter core can be folded.

After the spatial filtering operations mentioned above, the image gray scale is scanned to determine the maximum light intensity value I _max and the minimum intensity (maximum shadow) value I _min . A threshold intensity value I _T is determined which lies between the two extremes (I _max and I _min ) and is used to select shadow areas which have a light intensity which is less than I _T. An average value of I _T = I _max - I _min ) / 2 has been found to be useful for this purpose. Since the eye and mouth shadows tend to have the deepest shadows, thresholding amounts to setting three areas, as shown by the hatched areas in FIG. 4, namely the eye area shadows 10 and a mouth area shadow 40 .

Since the mouth area shadow 40 cannot have any symmetry due to a head tilt, a physiological asymmetry or because of the illumination with respect to a vertical line, the eye line 11 is first defined, thereby defining the axis of symmetry 9 .

Eye shadow areas 10 are easily identified since they occur in pairs and are set in a strictly horizontal proximity for an approximately vertical alignment of the actual axis of symmetry. The center of each eye shadow area 10 is determined in that the center of gravity is found relative to an arbitrarily selected coordinate system (x, y), which has been set at coordinates (0, 0) with tig, as shown in FIG. 5. For example, the center of gravity (x₁, y₁), which is assigned to the left eye shadow region 10 , is given by:

and

where (x _lk , y _{lk are} the coordinates of the kth pixel center of gravity of the left eye shadow area 10 , w _{lk is} a weight associated with the kth pixel, and K _{l is} the number of pixels that are in the left eye shadow area are included.

Likewise, the coordinates (x _r , y _r ) of the right eye shadow region 10 can be expressed as:

and

where (x _rk , y _rk ) are the coordinates of the kth pixel center of gravity of the right eye shadow area, w _{rk is} a weight assigned to the kth pixel and K _{r is} the number of pixels contained in the right eye shadow area 10 .

The weights {w _lk } and {w _rk } are usually selected accordingly to represent the value of the shadow in the kth pixel, which can be determined as the complement of the lighting values I _lk and I _rk in the following way:

w _lk = I _max - I _lk
w _rk = I _max - I _k

where I _{max is} the previously found maximum illumination intensity I _lk , I _{rk is} the illumination intensity _{value of} the kth pixel of the left and right eye shadow areas.

Further definitions for the weights {w _lk } and {w _rk } can be specified. For example, the complement can be found using I _lmax and I _rmax , which represent the maximum lighting values for the left and right eye shadow areas, respectively. Weights could also include scale factors that represent the significance of the kth pixel location by assigning larger values to pixels that are fixed at coordinates that are more likely to be eye shadow areas. Thus, if the weights were scaled to 0 if the pixel at the corresponding (x, y) coordinates was unlikely to get an eye shadow pixel and was otherwise scaled by 1, this would add an additional threshold when separating the eye shadows. Create areas. Or it could be used to replace the initial thresholding that has been used to generate the image of FIG. 5.

If the weights {w _lk } and {w _rk } are assigned the same values, for example a unit, only the coordinates of the centers of gravity are determined by the pixel allocation of the left and right eye shadow regions of FIG. 5 and are simplified the following:

and

A further simplification has been found to reduce the processing required to determine the focal points and still provide acceptable results by restricting the summation to only those pixels that have the edge (boundary line) of the threshold values Define fixed left and right eye shadows as shown in Fig. 5 for the left eye shadow area. When the left and right eye area coordinates are found, the axis of symmetry 9 can be determined by first executing the eye line 11 which passes through the centers of gravity (x _r , y _r ) and (x _l , y _l ). By halving the line section or line segment between the two centers of gravity at point 12 and by constructing a line running at a right angle ver through point 12 , the axis of symmetry 9 is obtained.

Thus, the eye line 11 is related to the focal points by the following expression:

The corresponding expression for the axis of symmetry 9 is:

The center of gravity of the mouth shadow area is found with the help of expressions which correspond to those previously used, so that the center of gravity (x _m , y _m ) of the mouth area 40 is:

where (x _mk , y _mk ) are the coordinates of the kth pixel center of gravity, w _{mk is} the weight assigned to the kth pixel, and the sum is made over all mouth area pixels, which is in the calculation are included. The weights {w _mk } can take various evaluations, as explained above for the eye shadow area focal points, and in one embodiment the summation is limited to those pixels which define the boundary line of the mouth shadow area, as shown in FIG. 4 .

After the center of gravity of the mouth region (x _m , y _m ) has been determined, the mouth line 51 can be constructed by drawing a line through the center of gravity which is also perpendicular to the axis of symmetry 9 and can be expressed as follows:

A region which is referred to as a region of interest (ROI) can now be defined as a rectangle in which two sides ( 52 and 53 ) run parallel to the eye line 11 or also to the mouth line 51 and in the same Distance above half and below the mouth line 51 are arranged. The others on the sides ( 54 and 55 ) are arranged in parallel and at the same distance on each side of the axis of symmetry 9 .

The dimensions of ROI are made large enough to include the tip tip area 20 , the nasal and nostril area 30 , the upper lip area 40 , the mouth area 50 , the lower lip area 70 and the upper chin area 80 . As shown in FIG. 6, a practical size for ROI (32 × 64) pixels has been found.

The pixels belonging to ROI can be found by defining two coordinate systems (x, y) and (x ′, y ′), the origin of the (x ′, y ′) system fittingly being the intersection of the Axis of symmetry 9 and the mouth line 51 can be taken, which is designated in Fig. 4 with (x₀, y₀). Coordinates in the x, y system, which correspond to a point in the x ′, y′ system, can be obtained by rotating and shifting the coordinates in the x ′, y′ plane. Therefore:

where θ = tan ^-1 {(y _l - y _r ) / (x _l - x _r )} represents the desired transformation to any coordinates of ROI related to the system (x ′, y ′) into coordinates convert that are related to the system (x, y) which are the "natural" coordinates of the video image pixels. Thus, by selecting the origin of the system at an appropriate point on the video image raster, such as a corner, the points of interest in ROI can be readily accessed.

If the angle of rotation θ is small, (θ «π / 2 radian), sinθ ≈ 0 and cosθ ≈ 1, so that:

Hence, if the speaker's head axis of symmetry is in is a small angle to the vertical, a simple relationship between the points of interest in the (x ′, y ′) system without white teres are converted into the (x, y) coordinate system, what is highest at least two additions are required.

Since the raster samples have been sampled by a 150 pixel to generate a wide image, the entire coordinate information, which, as stated above, was obtained by calculations is rounded down to the nearest pixel coordinates, if from the (x ′, y ′) coordinate system to the (x, y) coordinate system is converted.  

Fig. 6 (a) and 6 (c) show two typical Abtastverläufe along the Sym metrieachse. 9 Fig. 6 (a) shows the gray scale change along the axis of symmetry for the closed mouth shown in Fig. 6 (b). The vertical dimension of ROI in this example is 63 pixels. The height of the bars corresponding to each pixel represents the intensity of the illumination (the gray scale complement). Similarly, Figures 6 (c) and 6 (d) show a typical open mouth scan. The physiological change is evident in the changes in the two bar graphs that clearly reflect the shift in the valleys and peaks near the mouth line 51 .

Fig. 7 (a) and 7 (b) show the respective closed and open mouth positions as it passes through the bar graph of image Abtastver along the mouth line 51 in FIG. 6 (b) and 6 (d ) reproduced ROI are shown. Again, physiological changes are clearly reflected in the change in central peaks and valleys.

Figure 8 (a) shows typical vertical gray level scans and shows the selection of gray scale pixel values associated with each significant peak and valley. Peak values are labeled P1 to P3 and valleys are labeled V1 to V3. Figure 8 (b) shows the identification of two significant valleys V4 and V5, which represent the corner of the mouth positions as a function of a pixel position.

In this way (i.e. by peak and valley identification) the vertical profile scan vector of 64 pixels on six signifi edge elements P1 to P3 and V1 to V3 reduced, and the 32 horizon tal pixel scans are reduced to two significant elements graces. This data represents a set of elements of a visual language vector and can be used for speech recognition.

However, a further reduction can be achieved by restricting the pixel image vector to components that are more closely related to the mouth. It has been found that an effective speech classification (recognition) can be made with the help of the five points shown in FIG. 9, the mouth angles V4 and V5 denoted by 1 ′ and 2 ′ corresponding to FIG. 8 (b) , and wherein the upper lip, mouth and lower lip positions, which are denoted by 3 ', 4' and 5 ', correspond to P1, V2 and P2 in Fig. 8 (a). This data set formed from five elements could also be used as a visual feature vector, which is suitable for processing by means of a speech recognition classifier. Alternatively, the five-element data set could be additionally processed before it is applied to a language classification unit, with the expectation that the additional processing would reduce the complexity of the classifier unit.

In a preferred embodiment, in which the five positions P1, V2, P2, V4 and V5 are used, additional features are Vector elements from frame-to-frame (picture-to-picture) observation generated by P1, V2, P2, V4 and V5. The seventeen additional ones Elements are as follows:

• 1) the distance between V4 and V5 (in pixel units), which the distance between the left and right corners of the mouth poses;
• 2) the distance between P1 and P2, which is the vertical Ab stood between the upper and lower lips;
• 3) a vertical speed of P1, V2 and P2, which on Interframe changes ΔP₁ / Δt, and ΔV₂ / Δt, and ΔP2 / Dt based, where Δt is the interframe interval (1/30 s);
• 4) gray level values G (P1), G (V2) and G (P2);
• 5) Changes in gray level over time caused by G (P1) / ΔT, G (V2) / Δt and G (P2) / Δt are given;
• 6) the horizontal speeds ΔV4 / Δt and ΔV5 / ΔT;
• 7) Gray levels G (V4) and G (V5) and
• 8) a change in gray level over time by what ΔG (V4) / Δt and ΔG (V5) / Δt is given.

The resulting visual feature vector contains 22 elements which are used as the visual data vector which has been input to a TDNN language classifier shown in FIG. 10. A video camera 27 creates an electrical (150x150) pixel image (with a plurality of horizontal lines) of the speaker's face, which is then fed to a visual feature vector generator 90 that includes spatial and temporal filters 91 for preprocessing the image . Thresholding, ROI selection and coordinate conversion are performed in a unit 92 , the selected elements along the axis of symmetry and the mouth line are passed to a visual feature processor 93 which generates the 22 feature visual feature vector, which is then called an input value is supplied to the TDNN language classifier 200 .

In the combined acoustic and visual speech recognition system of FIG. 1, the acoustic data signals occurred in a time window with a duration of one second and were simultaneously recorded by a cardioid microphone 28 and were then recorded at a sampling frequency of 8012 Hz with an 8 Bit (mu-law) coding sampled and digitized (which is equivalent to a digital telephone line quality). A fourteen-valued Mel power spectrum analysis was performed using a spectrum analyzer 24 using a discrete Fourier transform on 256 data point frames centered at 5 msec intervals around the fourteen Mel power spectral analysis filters to simulate. Each of the 256 data point frames was windowed by a Hamming window function to reduce spectral loss. Applying the discrete Fourier transform to the "windowed" data and forming the squared size of the Fourier coefficients resulted in a performance spectrum with a uniform resolution with 128 points. However, the Mel scale 14 band power spectrum has a frequency scale F which is logarithmically related to the normal linear frequency scale f in the following way:

F = ln (1 + f / 700).

This mel frequency scale is based on psycho-acoustic frequency Scaling experiments which show that human assessments of the relative distance from two sine curves not equivalent to what they did objective arithmetic relationship. Hence, the concerns above equation to assess the perceived distance F. the physical measurements of frequency f. Although the Mel-Lei stung spectrum can be used with a variety of Lei Performance spectrum types that include those that are alike remain or are based on exponentially scaled band intervals be effective.

Fig. 11 shows a graphical curve of the Mel frequency F over the actual frequency f. Fourteen uniform bandwidths on the Mel scale are shown on the vertical scale together with their projection onto the actual frequency scale as non-uniform bands, which become larger with increasing frequency f in accordance with the expression given above. The lower abscissa scale refers to a Fourier coefficient frequency index with respect to the actual frequency. As a result, the uniform 14-mel scale bandwidth filters are simulated by summing the spectral power in the corresponding linear, non-uniform bands, as shown. A statistically more consistent mel-scale power spectrum is generated every 5 ms or 200 times per second by averaging sequential pairs of spectra, which reduces the effective output spectrum rate to 10 ms (100 spectra per second). The next processing step performed by the spectrum analyzer 24 is to generate the logarithm of each Mel spectrum to obtain a 14 component Mel spectral vector logarithmic. A standardization of each set of logarithmic Mel spectra, which correspond to a given speech / pronunciation, is achieved by calculating the mean and extracting the maximum and minimum of all values in the set. The log mel spectral sets were then scaled to cause the new mean to be zero and the maximum value of each log spectrum to be one. The standardized logarithmic Mel spectral vectors are available on output lines 25 of the spectrum analyzer 24 .

In one embodiment, since the video frame rate of the camera 27 in Fig. 1 was 60 frames / s, while the spectral generation frequency of the spectrum analyzer was 100 spectra per second, an additional processing step was required to adjust the two frequencies. This is illustrated graphically in FIG. 1 by the dash-dotted block 26 labeled "interpolator", which is only required if the video frame rate and the spectral frame rate are not the same. In this case, successive video data frames (vectors), each having the five vector components described above, were interpolated with a (sin x) / x interpolation core to produce a new sequence of frequencies at a speed of 100 per second to create. In other words, the value of the interpolated kth sample, which occurs in a time = k / 100 per second (k = 0, ± 1, ± 2,...), Ie in uniform time intervals of 10 ms, is given by:

where x (m / 60) is the value of the vector component at a time = m / 60, m is the original integer time index, which in Inter increments of 1/60 s; k is the new time index, which has been incremented at 1/100 s intervals and X (k / 100) is the interpolated values at the 1/100 s intervals.

This operation can be done using normal folding (digital filters techniques are carried out. Other possible interpolations cores including linear, square and polynomial cores higher order could also be used for interpolation that without noticeably affecting the efficiency.  

Thus, the five output lines 15 of the visual vector generator 14 and the 14 output lines 25 of the acoustic spectrum analyzer 24 are combined to form a time-varying, audio-visual ("AV") vector with 29 components, which is connected to the TDNN 200 is to be created, as shown in FIG. 1.

To adapt styles of speech of different lengths and et what can be unpredictable in terms of when they started, became the neural network used for a language classification plant selected, which is a time-delayed neural network (TDNN) which corresponds to that of Waibel, A. in an article with entitled "Modular Construction of Time-Delay Neural Networks for Speech Recognition "in Neural Computation 1, Stn. 39 to 46 (1989) be is written.

Fig. 12 (a) and 12 (b) show two embodiments of a functionally equivalent TDNN. Fig. 12 (a) has a uniformly tapped delay line 103 which receives an input data sequence x (t) on an input line 101 . The signals on taps 102 of the delay line form a time sequence vector which represents the time sequence x (t) which has been applied to delay line 103 , the older data being on the right-hand side, ie each tap has an output , which represents an input subsequent value which is applied τ seconds earlier than a value immediately to the left of it. At a predetermined point in time, a time course of the input sequence is available at the output taps 102 . A selected group of adjacent taps is connected to lines 105 of a neuron 110 labeled N1. Likewise, a set which has the same number of inputs as taps 102 , but is shifted by a delay unit τ, is applied to a neuron 110 designated N2. As shown, the TDNN structure can have a number of additional neuron units 110 connected in the same manner, each shifted by a delay unit τ. The neurons 110 are each identical in structure and use the same group of synaptic weights, which are applied via lines 107 . Consequently, the output value of each neuron 110 on output lines 109 represents a data sequence {y _k (t)} which is shifted in time by the interval τ from its neighboring neuron 110 output.

The hardware of the TDNN structure shown in Fig. 12 (b) uses a single neuron 110 in a multiplex mode. The input data sequence, which has been applied via the input line 101 to the tapped delay line, forms an image of its past values at intervals of τ seconds on output lines 105 which are applied in parallel to a neuron cell 110 . A set of weights (identical to those used in Fig. 12 (a)) are applied through input lines 107 . In every interval τ, a neuron cell 110 generates an output value on line 108 , which is applied to a second, tapped delay line 103 . If the second delay line 103 , which belongs to the next higher layer, has n taps as shown, each of (n-1) output taps 109 has a new sample output vector at intervals of τ seconds which is identical to that wel would be generated by the TDNN of FIG. 12 (a) for the same input data weights.

Both structures of FIG. 12 are suitable for receiving a single, time-changing vector component at its input connection 101 . Consequently, the structures shown must be divided for each component of the input vector. In general, the weights entered for each component will be unique, as determined by the training procedure.

The time delay unit τ is chosen so that aliasing of input data is minimized. If the value of τ is substantially less than the Nyquist sampling interval, the values at adjacent output taps 109 would be highly interrelated and would result in training instabilities due to the excess degrees of freedom (redundancy) used in the neural network .

If the interval τ is larger than the Nyquist sampling interval, introduced an aliasing that leads to an incorrect classification of Can lead to input data. An acceptable one has been found Value is about 10 ms.

The number of delay elements per delay line must be sufficient to ensure that a complete speech section through the total delay of (n-1) τ for the delays line structure is spanned with n output taps. For the class of speech or pronunciation values used found that a span of 100 elements with a Delay of 10 ms (thus a total of 1000 ms) is appropriate. Shorter observation periods can worsen a classification tern since they do not contain the entire section and longer span The classification can deteriorate because it overflows many contain some data.

In a practical speech recognition system there is a multilayer Vector input TDNN required. This can be done by a cascade Arrangement of a number of single-layer TDNNs can be realized, so that the output sequence of a given layer is the input sequence the next higher layer. In this way each layer forms an entered time sequence into another output time sequence and bil last but not least the entered time sequence of features (acoustic and visual) into a time sequence of attributes (classifications). There the TDNN is a subset of optimal neural networks it using the known back propagation or other gradients downward algorithms trainable. In addition, the TDNN has required another property, an identical transformation of a sequence of Feature time windows at incrementally changing time periods to be carried out one after the other. In contrast to more conventional neuro nalen Optimalwert networks the entered data sequence does not have to to be exactly matched to an absolute time reference value allow proper training. The TDNN structure confirms that inter eating patterns somewhere in time in the entered data sequence  can occur and while learning the TDNN tries for example to find sticky patterns in the training sequence regardless of when they occur.

However, if the network output is required to be a single one Is set of attributes, a sequence of features can have a fixed duration in the Input delay line can be saved. The duration should be like this be chosen that the period (the window) of the delay line can certainly contain the sequence of interest. If everyone follows de layer of the TDNN is a function of the window period, the Output for each characteristic / attribute of the output stage to a single one Value of the window interval can be averaged.

Fig. 13 (a) and 13 (b) show an example of a typical spoken expression (the manner of speaking). In Fig. 13 (a), an electrical signal is shown which represents the sound pressure, which starts at about 15.4 s and ends at 17.0 s. Fig. 13b shows the five components of the vector vi suellen same utterance, which is also at about 15,4 s be gins and about 1.6 sec later (see 17.0) ends. As a result, a window of one second would appear reasonable to contain both most of the audible and visual signals.

In the embodiments described below, ten possible expression / pronunciation types in the speech recognition system spoken, and then ten output values were generated at the output, each the probability of one of the ten possible speeches represent wise men who have been spoken. The network structure, had two layers of the TDNN type, to which a step to Time averaging and probability standardization followed.

The five different embodiments of one to be described Speech recognition system included:

• a) the VO system, which uses only visual data
• b) the AO system, which only uses acoustic data;
• c) the AxV system, which is a Kombina at a high level  tion of the AO and VO system;
• d) the AV system, which is at medium level (classification layer) links the AO and VO systems, and
• e) the full AV system with integrated processing of acoustic and visual data.

The architecture of a TDNN network is shown in FIG. 14 as used in a VO embodiment of a speech classifier 200 , which is based only on reading the visual comparison markers, using the five distance features previously described. No acoustic data was used in this embodiment. The delay line input stage 210 has five delay line units 103 each with 100 output taps, each delay line 103 storing one of the five components of the visual vector which is created by means of the vector generator 14 of FIG. 4. The delay lines were simulated in software. The time increment τ between stored values is 10 ms, so that the entered time sequence window spans 1000 ms.

A buried layer 220 has four rows of neuron cells, each having 81 neuron cells with (5 x 20) inputs on each row. Each neuron in a given row is connected to 20 adjacent delay line values of the five delay lines. The reduction from five to four rows created a modicum of data compression in the row dimension.

The classification layer 230 has ten rows, one for each of the pronunciations to be detected, and is 72 neuron cells long. Each neuron receives an array of 40 input lines: four rows from the hidden layer 220 with ten adjacent neuronal outputs in the hidden layer. Each of the ten resulting rows has 72 neuron cells, which correspond to the 72 possible values available from the shifted exit of the hidden layer. Each value at the output of the neurons in a given row of the classification layer 230 corresponds to the probability that the corresponding pronunciation / speech has occurred during one of 72 time intervals of 290 ms, which are arranged at intervals of 10 ms and the original (input step ) Span data window of 1000 ms. The resolution of 290 ms is obtained from the 20 ms window that has been used as an input to the ver buried layer 220, and from the following ten window units which are det USAGE in the hidden layer 220th

The averaging layer 240 has 10 neuron cells, each of which is assigned to a specific row of the classification layer and consequently to a specific one of the ten permissible speeches / pronunciations. The values in each output row of classification layer 230 indicate the likelihood that the particular pronunciation / speech occurred during a 290 ms interval that corresponds to that cell. The 72 inputs are linked in the corresponding neuron cell in the averaging layer to produce a value at their output which represents the probability that this pronunciation / speech was spoken somewhere during the 1000 ms observation window of input stage 210 .

The only acoustic (AO) speech recognition system uses the same type of architecture as the VO system architecture shown in FIG. 14, with parameters being set to match the 14-component logarithmic Mel Ska la spectral feature vectors obtained from the spectrum analyzer 14 have been creating the Figure 1 at intervals of 10. ms ge. Thus, the delay line input stage 210 has an array of 14 rows of 100 data values as inputs to the hidden layer 220 . Each row corresponds to a 1000 ms value of logarithmic mel scale values from one of the group of 14 output filters of the spectrum analyzer 14 .

The buried AO layer 220 had four rows of 96 neuron cells for feature compression from 14 to 4. Each neuron 221 is connected to 5 adjacent input stage time samples of each of the 14 input features (a total of 70 input feature values per neuron). The input window of 5 units (50 ms) is a typical time interval for the most important acoustic features, such as bursts and transitions. Thus, each subsequent neuron in a given row of hidden layer 220 receives an (14 rows by 5 columns) array of input values from an input stage 210 that are shifted by an interval of 10 ms.

Each neuron 231 of the AO classification layer 230 assumes an input factor of 4 feature rows and 25 time columns from the hidden layer 220 . Each subsequent column of neurons 231 receives inputs from the hidden layer 220 which are shifted by a column time unit of 10 ms. The 25 column window was chosen so that the total effective time window seen by the classification layer 230 is 290 ms (((5-1) + 25) × 10 ms); the same effective window was used in the VO speech recognition network described above. A total of 720 neurons, which the AO classification layer 230 contains, are each arranged in ten rows of 72 neurons. As in the case of the VO speech recognition network, each of the ten rows corresponds to one of the ten permitted speeches / pronunciations.

The output layer 240 of the AO-TDNN classifier 200 has a column with 10 neuron cells 241 , each cell which is connected to a unique series of features of the classification layer 230 being an input value of 72. The values at the output of the averaging layer 240 are each representative of the likelihood that the corresponding pronunciation / way of speaking occurred somewhere in the 1000 ms data window in the input stage 210 .

Figure 15 is another embodiment of the AO-TDNN, a functional equivalent to the network described above, that minimizes the amount of hardware required for neuron cell multiplexing. The 14 log. Mel-scale spectral data components of the acoustic vector signal are applied to a delay line arrangement (DLA) 302 , which has 14 delay lines with taps, each of which has a capacity of five data values. (In this context, a delay line is the equivalent of an analog or digital shift register). Once the delay line is filled, the neuron cell array (NCA1) 310 , which has four neuron cells (N1 through N4), picks up and connects the array of (14 x 5) values from the taps of the DLA 302 70 values -Field with each of the 70 input connections of the four cells (N1 to N4). The NCA1 310 generates four unique output values, one for each of the neuron cells N1 to N4 and transfers them to the input of the DLA 304 of the delay line arrangement in the hidden layer. The DLA 304 clocks the four values. 10 ms later, a new value reaches the input of the DLA 302 , whereby its content is shifted one unit to the right and the oldest set of 14 spectral data values is therefore output. The new set of (14 x 5) data locations is then applied to each of the cells of the NCA 310 , creating a new four-component vector at its four output ports which is input to the four delay lines of the DLA 304 . This process continues and ultimately results in the DLA 304 being filled. At this point in time, the ten neuron cells (N5 to N14) of the NCA 312 each calculate an initial value for the ten pronunciation categories and, for storage, deliver the ten-component vector to the DLA 306 , which each have ten delay lines with a capacity of 72 values.

Thus, a new vector is calculated by the NCA 310 after 10 ms, which is pushed into the DLA 304 . The NCA 312 then successively calculates a new ten-component vector, which is pushed into the DLA 306 . When the capacity of the DLA 306 is felt, the contents of each of the ten (1 × 72) delay lines are applied to the associated averaging neuron (N15 to N24) of the NCA3 14 . The first group of ten (10) outlets represents the probability that the associated phrase was spoken during the first 290 ms.

Since spectral data continuously arrive at intervals of 10 ms, new values of output probabilities are calculated. Episode The second calculated value of the probability would be one Expression that occurs in a 290 ms window around 10 ms is shifted from the first. Subsequent edition probabilities correspond to 290 ms windows relative to that previous windows are shifted by 10 ms.

Although the operation of the TDNN of Fig. 15 has been described with an AO Spra cherkennungssystems that same Operationsprin applies zip course, even if a similar structure in a VO speech recognition system were used, the operating descriptions made with the previous respect to Fig. 14 correspond agrees .

The described AO and VO systems work independently of acoustic or visual speech data. In other words, they are the equivalent of the system shown in Figure 1, in which either the visual or the acoustic data is nulled. It is to be expected that an increase / enlargement of an AO system by visual data would increase the classification accuracy, in particular under unfavorable acoustic environmental conditions. Three embodiments of a combined acoustic and visual speech recognition system will now be described.

The first system, which is referred to as the A × V system and is shown in FIG. 16, has an AO language classifier 400 and a VO language classifier 500 together with an additional network layer 250 to determine the output probabilities of the to combine both systems in a single set of probabilities.

An OA classifier 400 has a log. Mel scale spectrum analyzer 24 which picks up the electrical analogue of a speech signal and produces at its output a ten component acoustic vector which is used as the input to an acoustic TDNN classifier 410 as described above. The output of the AO classifier 400 is a probability vector that describes the probability that each of the ten permissible acoustic speeches / pronunciations were expressed only based on the acoustic data. This is then fed to a probability combiner 250 .

Likewise, a VO classifier 500 uses an electrical signal of the fiducial markers to set a lip and face feature vector associated with the corresponding acoustic feature vector as an input to the lip and mouth position vector generator 14 . The output visual vector, which has been generated by the comparison marker search system and the vector generator unit, is applied to an acoustic TDNN classifier 510 , which generates a ten-component visual probability vector, which is supplied to the probability combiner 250 , which represents the probability that each of the ten possible speeches / pronunciations was based solely on the visual data.

Now p (c: A) should be the conditional probability that the speech / pronunciation c with the associated acoustic data vector A occurred, and p (c: V) should be the conditional probability that the pronunciation / speech 10 with the associated visual data vector V occurred. Thus, p (c: A) corresponds to the c-th value of the vector output of the classifier 410 based on a vector A generated by the spectrum analyzer 24 , and p (c: V) corresponds to the c-th value of the vector output des Klassifika tors 510 , based on a vector V, which has been supplied by the generator 14 . If the vectors A and V are independent of each other, then the linked probability of the c-th pronunciation / speech given by A and V is:

p (c: A, V) = p (c: A) p (c: V) / p (c)

where p (c) is the a priori probability of the pronunciation c occurring. This is then the desired output from the probability combiner 250 .

Since p (c: A) and p (c: V) cannot be strictly independent and because of mistake in estimating these values and since p (c) cannot be known, the probability combiner creates the scaled product:

p (c: A, V) = kp (c: A) p (c: V)

where k is a normalization factor used to scale the ten output probabilities so as to ensure that their sum is one. Hence, the probability combiner 250 forms the product of corresponding terms of each of the ten component input vectors and then sums the ten resulting product terms:

around the normalized linked probability estimator form:

in which

The product term in the above expression can be different  Facilities including analog or digital multiplication units and a Sigma-Pl (ΣΠ) neural cell network in which corresponding p (c: A) and p (c: V) terms are used to link them together. The same term can also be used generated by a ΣΠ-neuronal cell network with 20 inputs in which corresponding A and V probability vectors gate components are linked together to create product terms form, and the initial value is then determined by the sum of the product Terms formed. Dividing each of the products by the sum term can be in a compatible technology using analog or numeri computing units are carried out.

Fig. 17 shows a hardware embodiment of the AxV system TDNN- languages classifier 200th Acoustic and visual data vectors are entered into delay lines (DL) 302 and 301 at 10 ms intervals. The acoustic and visual data flows follow different paths up to the last NCA3 level. The input acoustic data vector in FIG. 17 is processed analogously to FIG. 15 in the same manner and using the same hardware as in an AO process up to the unit NCA3 314 , which is arranged in the averaging layer. Likewise, the visual data vector V in FIG. 17, apart from the end unit NCA3 314 , is processed in the same way as is described in FIG. 14 for the VO system. The final probabilities p (c: A, V) are formed by the ten Sigma-Pi-Neuron cells, which have the unit NCA3 314 and the normalization unit 255 . The normalization unit 255 applies the scaling factor k by the ten output vector components p (C: A, V) = k * p (c: A) * p (c: V) for c = 1, 2. . ., 10 to form the.

Although an acoustic AO processor based on TDNN is described ben has been, this is not essential in the AxV embodiment Lich. The AxV system can recognize any acoustic speech tion processor, which is functionally equivalent even though he doesn't use a TDNN. The outputs of the AO and VO Subsystems would be linked to one exactly as described  to provide more classification than they deliver each system for themselves would. An acoustic vector generator can also be used effective acoustic vector, which speaks / off know the language with acceptable statistical reliability records as an acoustic preprocessor in one of the combined acoustic and visual speech recognition systems are used.

FIG. 18 shows a further embodiment for a TDNN classifier 200 from FIG. 1, which is referred to as an AV system classifier. In this case, the A and V vector inputs are processed separately by hidden layer 220 , which has NCA1 310 and NCA 309 . The output values from the hidden layer are stored in DLA 304 and DLA 303 , from where the acoustic (4 × 25) value field from DLA 304 and the (4 × 10) value field from DLA 303 to the ten 140-input neuron cells ( N35 to N44) of the NCA2 313 . The previous mix of acoustic and visual data results in a richer set of cross terms in which resolution can be sought in the training process, resulting in an overall improved performance of the overall system.

After processing by NCA2 313 , the results are passed to averaging layer 240 . NCA3 315 has ten neuron cells ( 950 to 955 ), each of which operates on one of ten delay lines in DLA 307 . The non-normalized probability product p (c: a) .p (c: V) is formed by MCA3 315 and passed to a normalization unit 255 to form the final initial probabilities p (c: A, V).

Fig. 19 shows a further development step in the process of combining the processing of the acoustic visual data at the earliest possible level. Upon receipt and storage of sufficient input acoustic and video data in the DLA 302 and 301 , the formation of transverse terms can occur in the hidden layer 220 , with each of the eight (8) neurons on the combined (14 × 5) and (5 × 20) works data fields available from DLA 302 and 301 . The hidden layer output is stored in DLA 308 and passes through NCA3 313 of classification layer 230 . The initial value is then traded as described above with reference to FIG. 18.

The 15 basic types of neuron cells are used in the TDNN structures of FIGS. 15, 17, 18 and 19. NCA1 of hidden layer 220 and NCA3 of averaging layer 240 have conventional McColuch-Pitts (MP) neuron cells, as shown in FIG. 20. An input data vector X, which is provided at an input connection 601 , is processed by a weighting factor w with the aid of multiplication elements 603 . A summing network 605 sums the x _i w _i terms from the multiplication elements 603 , which results in the vector point product:

which is formed at the output of the summing network 605 . The vector point product and offset is applied to an activation functional unit 609 , which forms the output signal f (z) = f (x ^T w + offset). An offset value can be obtained by an input 607 on the summing network 605 .

Activation functional unit 609 may have one of several transfer characteristics, such as that shown in FIG. 21. The transfer characteristic in Fig. 21a is f (z) = tanh z, ie a hyperbolic Tan function z of the input signal. This is an example of a large group of semi-linear sigmoidal squashing functions that can be used in hidden layer NCA1. Fig. 21 (b) shows a linear transfer function f (z) = mz + offset. This form of an activation function is used in the NCA3 neuron cells of the middle layer 240 (the offset being zero).

Fig. 21 (c) shows an exponential activation function f (z) = e ^{z / T} , where T is a positive value "temperature" parameter which effectively changes the basic value e ^{1 / T} , which is determined by the input value z is potentiated. The exponential value of the activation function is used to estimate the likelihood of a set of unknown variables (the set of permissible speeches) corresponding to Parallel Distributed Processing: Explorations in the Micro structure of Cognition. Vol.1: Foundations, Rumelhart, et al., The MIT Press, Cambridge, MA 1986, Chapter 6, Stn. 194 to 281. The exponential activation function is used in the MP-Neuron cells of NCA2 of the classification layer 230 .

The second type of neuron cell used in the NCA4 of the TDNN structure of FIG. 17 is known as a Sigma Pi (ΣΠ) cell, as shown in FIG. 22. In contrast to the MP cell of FIG. 20, in which a single input data vector is applied to a set of controllable weights {w _j }, ΣΠ-neuron cells in their simplest form take two input data vectors x and y at input connections 701 or 702 . Corresponding input vector components x _i and y _i are applied to multiplier units 703 to form the x _i · y _i product terms, which are summed by a sum network to form an offset term which is applied to an offset input terminal 707 . Consequently, the output of the summing unit

the vector point product of x and y plus offset, which is applied as the input to an activation functional unit 709 with a transfer characteristic f (0). The Σπ neuron cell of FIG. 22 is used in the AxV network of FIG. 17 to calculate the sum of the product of the ten p (c: A) values and the corresponding ten p (c: V) values in NCA4 the averaging layer 240 (with the offset value equal to zero).

Because of the differentiable activation functional units that have been used in the neuron cells (see FIGS. 20 to 22), one form of the known back propagation algorithm (described by Rum melhart, DE et al., In "Learning Internal Re presentations by Error Propagation ", Stn. 318 to 362 (Chapter 8), of Parallel Distributed Processing Vol.1: Foundations, The MIT Press, Cambridge, MA, 1960) used to train the neural network weights. The error function of the initial value during training has a considerable difference from that which was described in the reference given above.

When a neural network task is out of class select to which an entered pattern belongs, if all entered pattern to one and only one class (one from one of one n classification) and if the initial value of the neural network plant consists of n values, which is a probability distribution representation (i.e. the sum of values equals one) is the mistake lermetrix for a single training classification attempt

Error = -ln {(c _l )}

where c _{l is} the correct classification of the lth class member and p (c _l ) is the experimental value of the output, which represents the probability that the data entered belong to class c _l . Since the correct probability of the pronunciation / speech c _{l entered is} one (certainty), the error reproduced above can be rewritten as a difference in the logarithms of the actual and test probabilities (p (c _l ) or (c _l )) or

Error = ln {p (c _l )} - ln {(c _l )} = ln {p (c _l )} / {(c _l )}.

This error metric comprises the ratio of probabilities. As stated above, if the probabilities were formed by the potentiating activation function ( Fig. 21 (c)), the substitution of

where z _l and _{l are} the correct or test values of the input z on the lth classification neuron cell activation function network. Thus, the negative logarithm of the empirical probability held is a measure of the error generated by the signal arriving at the classification level, which includes the error caused by the weights of the lth neuron cell of NCA2 is included. Consequently, with the help of the training patterns, which represent all speech styles / pronunciations to be classified, this logarithmic error function can be used to train the TDNN network.

Experiments were carried out with the aid of comparison marks for visual information in the systems described in order to distinguish between the spoken letters B, D, F, M, N, P, S, T, V and Z. Acoustic and visual information was stored by four male people who are native speakers of American English. Each person spoke each letter five times. An example of the letter P, which was spoken by a person, was made unusable by an error in the visual tracking system, which resulted in a total of 199 and not 200 bars (token). Each bar was converted into visual (VO), acoustic (AO), and acoustic and video (AV) vectors suitable for a classification layer 230 , as described above.

Four of the five repeated letter pronunciations for each of the ten Letters spoken by the four speakers (160 Patterns) were used for training. The remaining 39, wel che four examples for each letter except three for the Letters P, as stated above, were used for Testing used.  

Each of the three networks was trained with the back propagation algorithm to find the 05514 00070 552 001000280000000200012000285910540300040 0002004435272 00004 05395 set of weights that cause a minimal error for the training set examples. A learning rate constant (η = 0.001 and a moment constant of α = 0.09 were used for the VO and AO cases, while constants η = 0.002 and α = 0.3 were used for the AV case AV network was formed by mixing the AO and VO trained networks and adding the cross-model connections between the hidden A and V layers and the classification layer. Learning was allowed to progress, until the training error showed a negligible change over the past several hundred training sections. The software simulation of the spectrum analyzer 24, the lip-mouth position vector generator 14 and the ADNN language classifier 200 ran on a Sun Sparcstation 2 computer, which was manufactured by Sun Microsystems, Inc., 2550 Garcia Avenue, Mountain, View, California 94043-1100 The training took approximately 20 seconds per section for the most complex AV network and we for the other networks. For every 50 training sessions, the weights were recorded in the network and the test group error was calculated. Fig. 23 shows the progress of error reduction as a function of the number of training sections that have been used for the VO, AO and full AV cases. The dashed line represents the error when the test group of input data is used, while the dashed line is the error that has been observed when the training group is used for testing. As was to be expected, the latter generally produced fewer errors.

After the "best" weight group was determined by the training process for each of the three cases, confusion matrices were generated for the test data. The average of the distributed distributions over all test group examples of this letter was calculated for the respective letter speech. Figure 24 shows the resulting confusion matrices for the VO, AO and AxV and full AV networks. Each vertical column is labeled with the spoken letter, which is represented as the input value; each horizontal line represents the output letters which have been selected with their associated probabilities which have been selected by the location on each plate. The percentage under each confusion matrix is the overall accuracy of this system. Note the confusion or confusion with the / bi / and / di / phonemes in the AO network and the unbalanced confusion of / bi / and / pi / in the VO system. As a result, although / pi / can easily be confused with / bi / in the VO system, these are easily distinguished in the OA system. Conversely, the VO system easily distinguishes between / bi / and / di /. Likewise, the acoustic (AO) similarity of the / em / - and / en / -phonems is to be distinguished, which mainly differ in the acoustic subtle nasal characteristic and are clearly distinguishable in the VO system. In the VO system, / di / and / ti / are easily confused in the OA system. The AxV system usually eliminates these and other confusions, as evidenced by the high classification probability along the main diagonal.

Although it is reasonably expected that the full AV system with an overall accuracy of 87% should be better than the AxV System with a measured accuracy of 91% because the full AV System associations at a lower (hidden layer) level can learn, the test results show the following. This result strongly emphasizes that the full AV vector of 19 Ele not exactly synchronous between the ten (10) acoustic Elements and the five (5) visual element parts of the complete Vector behaves because it is separate through its respective preprocessors were processed. In the case of the AxV system, we were synchronized niger critical, because the acoustic and visual vectors by geson most TDNNs have been processed. In other words, low level cor relations in the training patterns could have been changed, so that the full AV network could not use this information.  

A corresponding performance for the AxV system is demonstrated if the "ROI" technique was used, in which none Compare marks are required.

The relative insensitivity of the AxV system to acoustic Noise was demonstrated by a simulation in which Ro thereby adding sa noise with a standard deviation of 0.4 added any variable to each A vector was added, which were generated by the spectrum analyzer is. Under these conditions, the AO network had an overall ge 43% accuracy while the AV network is 24% accuracy would have. This improved accuracy was achieved without any adjustment a free parameter, such as modifying the ver Ratio from acoustic to visual, achieved.

Claims (15)

1. A method for extracting a visual feature vector from a sequence of images, each having a number of horizontal lines of front views of a speaker's face, for use in a speech recognition device, comprising the following steps:

• a) the image is scanned and quantized at regular intervals along each horizontal line of the image to produce an image of pixels that represent gray level levels centered on the uniform intervals;
• b) the pixel image is pretreated by spatially smoothing and improving edge separating areas with higher or lower gray scale intensity using spatial folding methods;
• c) a threshold value is formed on the pretreated pixel image in order to determine a left and right eye area and a mouth area;
• d) the left and right eye area and the mouth area are calculated;
• e) an eye line is introduced as a straight line that passes through the left and right eye area locations;
• f) a vertical axis of symmetry is introduced as a straight line which bisects an eye line portion connecting the left and right eye area locations;
• g) a mouth line is introduced by passing a straight line through the mouth area, the line being perpendicular to the vertical axis of symmetry;
• h) image pixels along the axis of symmetry are selected that fall near the mouth line to form a vertical profile of gray scale pixel values;
• i) image pixels are selected along the mouth line that fall near the axis of symmetry to form a horizontal profile of gray-scale pixel values; and
• j) as a set of elements of a visual feature vector, a set of pixels and associated pixel values is selected that occur at vertices and troughs (maxima and minima) of the vertical and horizontal gray scale pixel value profiles.

2. The method according to claim 1, characterized in that the Step of selecting image pixels along the mouth line selected pixels, left and correspond to a right corner of the mouth, and that the Step of selecting image pixels along the symmetry axis results in the selected pixels, that of an upper lip, one mouth and one lower lip position chen. 3. The method according to claim 2, characterized by the following steps:

• a) a mouth angle-to-mouth angle distance measure is calculated by forming the difference between the locations of the selected pixels, which correspond to the left and right mouth angle positions;
• b) a vertical mouth pitch is calculated by forming a difference between the positions of the selected pixels, which correspond to the upper lip and lower lip positions;
• c) a horizontal mouth angle-to-mouth angle speed is calculated by forming the difference between mouth angle-to-mouth angle distances from adjacent successive images or picture frames;
• d) a set of vertical speeds of an upper lip position, a mouth area position and a lower lip position is calculated by forming a difference between pixel positions, the upper lip, mouth area and lower lip positions of adjacent, successive images or Correspond picture frame;
• e) a set of pixel gray level value changes over time is calculated by differentiating in pixel values of the selected set of pixels between adjacent successive frames; and
• f) a visual feature vector is formed from the values calculated in steps (a) through (e) and from the selected set of pixel values.

4. The method according to claim 1, characterized in that the Step of selecting image pixels along the mouth line selected pixels, left and correspond to the right corner of the mouth and that the step the selection of image pixels along the axis of symmetry selected pixels, that of a lower nose area position, an upper lip position, a mouth area  rich position, a lower lip position and a chin area position. 5. The method according to claim 1, characterized by the step tt a temporal picture-to-picture smoothing for noise reduction tion by using successive pictures or picture frames a prescribed low-pass filter core are folded. 6. Apparatus for extracting a visual feature vector from a sequence of raster-scanned video images from front views of a speaker's face for a speech recognition device for recognizing speech patterns which belong to a previously introduced set of permissible candidate speech patterns, characterized by

• a) analog-to-digital converting means for sampling and quantizing the raster-scanned video image at regular intervals along each raster scan to produce an image of pixels representing gray level levels centered on the regular intervals;
• b) a filter device for pretreating the pixel image by spatially smoothing and improving edge separating areas with higher or lower gray scale intensity using spatial convolution methods;
• c) thresholding means for thresholding the pretreated pixel image to determine left and right eye area and mouth area;
• d) a calculation device for calculating the left and right eye area location and the mouth area location;
• e) calculating means for introducing an eye line as a straight line passing through the left and right eye area locations;
• f) calculating means for introducing a vertical axis of symmetry as a straight line which bisects an eye line portion connecting the left and right eye area locations;
• g) a calculator to introduce a mouth line by passing a straight line through the mouth area, the line being perpendicular to the vertical axis of symmetry;
• h) computing means for selecting image pixels along the axis of symmetry which fall near the mouth line to form a vertical profile of gray scale pixel values;
• i) computing means for selecting image pixels along the mouth line that fall near the axis of symmetry to form a horizontal profile of gray scale pixel values; and
• j) computing means for selecting a set of pixels and associated pixel values as an element set of a visual feature vector, the pixels and associated pixel values at vertices and troughs (maxima and minima) of the vertical and horizontal grayscale pixel value profiles occurrence.

7. The device according to claim 6, characterized in that the calculation device for selecting image pixels selects the pixels along the mouth line, those of a left and correspond to a right corner of the mouth, and that the calculation device for selecting image pixels along the symmetry axis selects the pixels that an upper lip, one mouth and one lower lip position chen. 8. The device according to claim 7, characterized by:

• a) a calculation device for calculating a mouth angle-to-mouth angle distance measure by forming the difference between the locations of the selected pixels which correspond to the left and right mouth angle positions,
• b) calculating means for calculating a vertical mouth pitch by difference between the positions of the selected pixels corresponding to the upper lip and lower lip positions;
• c) a calculation device for calculating a horizontal mouth angle-to-mouth angle speed by forming the difference between mouth angle-to-mouth angle distances from adjacent successive images or image frames;
• d) a calculation device for calculating a set of vertical speeds of an upper lip position, a mouth area position and a lower lip position by forming the difference between pixel positions, the upper lip, mouth area and lower lip positions of adjacent successive images or Correspond to picture frames;
• e) calculation means for calculating a set of pixel gray level changes with respect to time by differentiating in pixel values of the selected set of pixels between adjacent successive frames; and
• f) calculation means for forming a visual feature vector from the values calculated in steps (a) to (e) and from the selected set of pixel values.

9. The device according to claim 6, characterized in that the calculation device for selecting image pixels selects the pixels along the mouth line, those of a left and right corner of the mouth position, and that the Calculator for selecting image pixels along the axis of symmetry selects the pixels of a lower one Nose area position, an upper lip position, a mouth area position, a lower lip position and a chin area position.   10. The device according to one of claims 6 to 9, characterized by a Device for temporal smoothing over successive pixel images or Pixel picture frame for noise reduction using a prescribed Low pass filter core. 11. The device according to one of claims 6 to 10, characterized in that the filtering, thresholding, and computing devices are computers. 12. The apparatus according to claim 11, characterized in that the device is a computer for smoothing over successive pixel images. 13. The apparatus according to claim 11 or 12, characterized in that all Computers are a jointly programmable computer.   14. Use of a visual feature vector extractor according to any one of claims 6 to 13 in a speech recognition device for recognizing speeches belonging to a previously introduced set of allowable candidate speeches, wherein

• a) the visual feature vector extracting device generates a sequence of visual feature vectors from a sequence of raster-scanned video images from before the views of a speaker's face

and the speech recognition device

• b) comprises an acoustic feature vector extracting device for converting signals, which represent an acoustic speech, which occur accompanying the raster scanned video images, into a corresponding sequence of acoustic feature vectors, and
• c) a neural network classification device for generating a conditional probability distribution of the permissible modes of speech by the acoustic or visual feature vector sequences provided by the acoustic or visual feature vector extracting device as input variables for the neural network Classifying device used and processed by this.