JPH08249447A - Expression detector - Google Patents

Abstract

PURPOSE: To exactly detect facial expression with high robustness in real time. CONSTITUTION: This detector is provided with an extraction part 16 extracting only the part distinct in facial expressions from a facial video signal DVL, a wavelet filter 18 generating a spatial frequency F by performing wavelet conversion for the extracted video signal DVLe, an average power calculation part 20 calculating the average power Pc of the spatial frequency F for every prescribed band and a difference calculation part 24 obtaining a feature vector FV by calculating the difference of the average calculated power Pc and the average power Pn of expressionless which is preliminarily stored in a stationary state storage memory 22.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a facial expression detecting device,
More specifically, facial expression detection of participants in video conferences using computer graphics connecting remote places,
The present invention relates to a facial expression detection device that detects a human facial expression in real time, such as a facial expression detection created by computer graphics in real time.

[0002]

2. Description of the Related Art Human facial expression recognition is an important basic technology in research on advanced coded communication and active computer interfaces. For example, it is disclosed in "Otani, Kitamura, Takemura, Kishino:" Real-time display of 3D face image in realistic communication conference ", IEICE Technical Report, HC-92-61, pp.23-28 (1993.1.)". In such a computer-controlled communication conference system, if facial expression recognition is realized, it is possible to achieve a high compression ratio of information to be transmitted, and it can be expected to be applied to more advanced emphasizing work.

In the conventional facial expression detecting method in the above-mentioned realistic communication conference, the facial expression is detected by attaching markers to several places on the face and tracking the movements of the markers. However, since such a method requires a detection assisting tool such as a marker, there is a problem that it takes time to prepare for performing facial expression detection. In addition, since the robustness is low, there is also a problem that malfunction is likely to occur due to disturbance factors such as displacement of the helmet and pulse.

In the field of computer vision,
In many cases, the positions and shapes of the characteristic parts of the face (eyes, nose, mouth, etc.) are extracted from the image and facial expression recognition is attempted based on the measured amount. However, it is very difficult to accurately extract the shape and position of each part from a still image such as a photograph as a feature amount of an image because it strongly depends on the imaging conditions and the aspect of the subject, and at the same time, complicated processing is required. An algorithm was required, which was an obstacle to real-time processing.

By the way, some reports have been made on the recognition of facial expressions by paying attention to the change with time. For example, “M. Rosenblum, Y. Yacoob and L. Davis:“ Human Emotio
n Recognition form Motion Using a Radial Basis Fun
ction Network Architecture ”, Proceedings of the IE
EE Workshop on Motion of Non-rigit and Articulated
"Objects, pp.43-49 (1994)" focuses on changes in the face surface due to facial expressions in moving images, and attempts recognition using optical flow, but since it is a very multidimensional feature,
Hidden Markov Model (HMM)
However, there is a problem that the processing is complicated and the calculation amount is large.

As an example of moving image recognition using HMM,
For example, “Yamato, Otani, Ishii:“ Recognizing human behavior from moving images using hidden Markov models ””, IEICE Transactions D-
II, Vol.J76-D-II, No.12, pp2556-2563 (1993.12.) ”, Attempts to recognize the movement of a tennis player, but the mesh features of the brightness values used to characterize the image It is not always effective when the degree of similarity is high like facial expressions, and there are problems such as weakness in classification into similar categories.

The present invention has been made to solve the above problems, and an object of the present invention is to provide a facial expression detection device having high robustness.

Another object of the present invention is to provide a facial expression detecting device capable of detecting the facial expression in real time without preparation such as attaching a marker to the face.

Still another object of the present invention is to provide a facial expression detection device which is simple in processing and has a small amount of calculation.

Still another object of the present invention is to provide a facial expression detecting device capable of obtaining a high facial expression recognition rate by applying an HMM.

[0011]

A facial expression detecting apparatus according to a first aspect of the present invention comprises a photographing means, a wavelet transforming means, an average power calculating means, and a difference calculating means. The shooting method is
An image of a person's face is captured and a video signal is generated. The wavelet transform means wavelet transforms the video signal to generate a frequency signal in the spatial frequency domain for each predetermined band. The average power calculation means calculates the average power of the frequency signal for each band. The difference calculation means calculates a difference between the average power sequentially given by the average power calculation means and the corresponding average power obtained from the face of the person when the face is expressionless.

A facial expression detecting apparatus according to a second aspect further comprises an extracting means in addition to the configuration of the first aspect. The extraction means extracts a portion corresponding to a predetermined area where the facial expression of the person is likely to appear from the video signal given from the video means and gives it to the wavelet transformation means.

[0013]

In the facial expression detecting apparatus according to the first aspect, the face of a person is photographed and a video signal is generated. The generated video signal is wavelet transformed to generate a frequency signal in the spatial frequency domain for each predetermined band, and the average power of the generated frequency signal is calculated for each band. Then, the difference between the calculated average power and the corresponding average power obtained from the expressionless face is calculated. Since the video signal is once wavelet transformed in this way, facial expression detection with high robustness is possible.

In the facial expression detecting device according to claim 2,
In addition to the effect of the first aspect, more accurate facial expression detection is possible because only the portion where the facial expression is likely to appear is extracted from the video signal and wavelet transformed.

[0015]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of a facial expression detecting device according to the present invention will be described below in detail with reference to the drawings. In the drawings, the same reference numerals indicate the same or corresponding parts.

FIG. 1 is a block diagram showing the overall structure of a facial expression detection device according to an embodiment of the present invention. Referring to FIG. 1, this facial expression detection device includes a CCD (Charge Coupled Device) camera 10 for photographing a human face, and a CC.
Low-pass filter (LPF) 1 for removing high frequency components from the analog video signal AV given from the D camera 10
2, an A / D converter 14 for converting the analog low-pass video signal AVL given from the low-pass filter 12 into a digital low-pass video signal DVL, and a digital low-pass video signal DVL given from the A / D converter 14. An extracting unit 16 for extracting only a predetermined portion, which will be described later, and an extracting unit 1.
A wavelet filter 18 that generates a frequency signal F in the spatial frequency domain for each predetermined band by performing a wavelet transform on the digital low-frequency video signal DVLe extracted by 6, and the generated frequency for each band. An average power calculation unit 20 that calculates the average power Pc of the signal F, a steady-state storage memory 22 that stores in advance the average power Pn when there is no expression corresponding to this average power Pc, and the average power Pc and The difference calculation unit 24 that generates the feature vector FV by calculating the difference of Pn.

Next, the operation of this facial expression detecting device will be described. First, the CCD camera 10 is attached to the front of the subject's face. The CCD camera 10 is fixed to a helmet or the like attached to the subject so that the subject's face is always photographed from a fixed position. FIG. 2 shows an image of the subject's face taken by the CCD camera 10.

Prior to detecting a facial expression, the average power Pn when there is no facial expression is stored in the steady state storage memory 22 in advance. Details of the average power Pn will be described later.

To detect a facial expression, first, a CCD
The subject's face is photographed by the camera 10, and the analog video signal AV generated by the photograph is taken by the low-pass filter 1.
Given to 2. The low-pass filter 12 removes high frequency components from the supplied analog video signal AV and supplies the low frequency analog video signal AVL to the A / D converter 14. Generally, since a human face has pores, irregularities, and hair loss, the analog video signal AV contains a large amount of high-frequency components caused by these signals. It is completely unnecessary for detecting facial expressions. Further, although the analog video signal AV having a low SN ratio contains a large amount of noise components, most of such noise components have a high frequency. Therefore, the low-pass filter 12 is for preventing malfunction of this device by passing only the low-frequency component necessary for detecting the facial expression.

The A / D converter 14 converts the supplied low frequency analog video signal AVL into the low frequency digital video signal DVL.
Convert to. This low-frequency digital video signal DVL is given to the extraction unit 16, and a part DVLe of this low-frequency digital video signal DVL is extracted. In this embodiment, a detection area 161 including eyes and eyebrows and a detection area 162 including a mouth are cut out as shown in FIG. That is, the video signal D
Only the part DVLe corresponding to the detection areas 161 and 162 is extracted from the VL. The eyes, the eyebrows, and the mouth are defined as the detection areas in this way because these are the portions in which the change due to the facial expression is remarkable. Therefore, the detection area is not limited to the above-described area, and for example, a wrinkle-prone frame may be set as the detection area.

The video signal DVL thus extracted
The e is given to the wavelet filter 18, where the wavelet transform is performed. The wavelet transform is one kind of transforming method to the frequency domain of an image, and is considered to be an overlap transform in which the transform base width becomes narrower at higher frequencies. The wavelet filter 18 for converting an image into the frequency domain can be considered as a band division filter, and can be realized by combining filter banks as shown in FIG. 3 in multiple stages. As shown in FIG. 3, one filter bank includes a low-pass filter 26 and a high-pass filter 28 for horizontal division, low-pass filters 34 and 42 and high-pass filters 36 and 44 for vertical division, and a down sampler 30, 32,3
8, 40, 46, 48.

The video signal DVLe from the extraction unit 16 is given to the low-pass filter 26 and the high-pass filter 28 as an original image. The signal that has passed through the low-pass filter 26 is compressed by the down sampler 30, and is further supplied to the low-pass filter 34 and the high-pass filter 36. The signal that has passed through the low-pass filter 34 is compressed by the down sampler 38, and is further given to the filter bank of the next stage. The signal passed through the high pass filter 36 is compressed by the down sampler 40,
Appears in the frequency band 183 shown in FIG.

On the other hand, the signal that has passed through the high pass filter 28 is compressed by the down sampler 32, and is further given to the low pass filter 42 and the high pass filter 44. The signal passed through the low pass filter 42 is compressed by the down sampler 46 and appears in the frequency band 182 shown in FIG. The signal passed through the high pass filter 44 is compressed by the down sampler 48 and appears in the frequency band 181 shown in FIG.

Similarly, the video signal supplied to the filter bank of the next stage has three frequency bands 184 to 184 shown in FIG.
It is divided into 186. Similarly, the video signal given to the filter bank at the next stage is also divided into three frequency bands 1.
It is divided into 87 to 189. The signal passed through the low-pass filter 34 in this final stage filter bank is compressed by the down sampler 38 and appears in the frequency band 190 shown in FIG. Therefore, frequency band 181 contains the highest spatial frequencies and frequency band 190 contains the lowest spatial frequencies.

FIG. 5 shows a detection area 161 including eyebrows and eyes.
2 shows an image when the video signal of 3 has passed through the above three-stage filter bank. This image is biased for each band for visualization.

The low-pass filters 26, 34, 42 and the high-pass filter 2 in the wavelet filter 18
8, 36, and 44 require conditions such as perfect reversibility, phase characteristics close to linear phase, steep cutoff characteristics, and orthogonality of frequency response, but there is a trade-off in these conditions. In this embodiment, a 32nd-order quadrature mirror filter having a linear cutoff characteristic but a relatively low cutoff characteristic is used.

In the image that has passed through the wavelet filter 18 as shown in FIG. 5, the frequency characteristic and the position information on the image are stored. Since this amount of information is the same as the original image, it is difficult to obtain features from this. Moreover, if it is used as a feature as it is, C
The position of the CD camera 10 is strongly affected by variations and individual differences. Therefore, in the present invention, a method characterized by only increasing or decreasing the average power for each band is adopted. Therefore, the spatial frequency F generated by the wavelet filter 18 for each band is given to the average power calculation unit 20, where the average power Pc of the spatial frequency F is calculated for each band.

When the facial expression is expressed, the shape, size, inclination, etc. of the facial constituent elements change. It is characterized by the effect of this change within each frequency band. For example, when the subject closes his / her eyes, the horizontal frequency component in which electric power has been present decreases, and conversely, the vertical high frequency component in which electric power is small increases.

When the facial expression changes, the high frequency band 181
No significant power change is observed in ~ 183, but a significant power change is observed in the low frequency bands 184-189. This is because the high frequency components after passing through the wavelet filter 18 mainly include the edge information of the image, and the low frequency components include the shape information of the face constituent elements. This shows that the change of the shape itself is more effective as a feature than the change of the edge direction and the intensity accompanying the change of the facial expression. Therefore, in this embodiment, the average power Pc in the frequency bands 184-189 is preferably used.

FIG. 6 is a graph showing changes in the average power Pc of the frequency band 185 obtained from the detection region 162 including the mouth when the subject is surprised. The horizontal axis of this table shows the number of frames, and the vertical axis shows the average power. When a surprising expression appears, the mouth is opened vertically, so that the power of the horizontal low-frequency component appearing in the band 185 is significantly increased.

FIG. 7 is a graph showing changes in average power of the frequency bands 185 and 186 obtained from the region 161 including the eyebrows and eyes when the subject blinks. Figure 7
As shown in (1), when blinking occurs, the eyes become narrower, so the average power of the horizontal low-frequency components appearing in the band 185 instantaneously decreases, and the average power of the vertical low-frequency components appearing in the band 186 instantaneously decreases. Increase.

As described above, the average power in the low frequency component of the spatial frequency obtained by the wavelet transform changes according to the facial expression. Such average power Pc calculated by the average power calculation unit 20 is given to the difference calculation unit 24 for each frame. The difference calculator 24 calculates the difference between the calculated average power Pc and the expressionless average power Pn given from the steady-state storage memory 22. Here, the steady-state storage memory 22 must store an average power Pn corresponding to the calculated average power Pc. Therefore, prior to detecting the facial expression, an expressionless face is first photographed, the wavelet transform is performed in the same manner as described above, and then the bands 184 to 189 are added.
It is necessary to calculate the average power of the above and store it in the steady state storage memory 22 in advance. Therefore, the difference between the calculated average power Pc and the expressionless average power Pn is the difference calculation unit 2
4 is output as a feature vector FV. The feature vector FV indicates the displacement of the face based on the expressionlessness.

As described above, according to this embodiment, since the face image signal is wavelet-transformed once, the robustness is high, and even if the face moves a little on the screen, or the illumination conditions are slightly different. However, the facial expression can be accurately detected. Moreover, since the wavelet transform is performed by the filter having a simple structure, the processing is simple and the calculation amount is small. Therefore, the facial expression can be detected in real time.

Further, since only the detection areas 161, 162 including the eyebrows, eyes, and mouths, which are likely to express facial expressions, are extracted, a malfunction does not occur due to a facial movement unrelated to facial expression changes, and a more accurate facial expression is obtained. It becomes possible to detect.

It is also possible to perform vector quantization on the feature vector FV obtained in this embodiment and recognize the facial expression using the HMM. In this way, a higher recognition rate can be obtained by combining with the time change by the HMM.

[0036]

According to the facial expression detecting device of the first aspect,
The face image signal is wavelet-transformed, and the average power of the spatial frequency obtained thereby is compared with the average power when there is no expression. Therefore, facial expression detection with high robustness is possible.

According to the facial expression detecting apparatus of the second aspect, since only the video signal in which the facial expression is likely to appear is extracted, the facial expression can be detected more accurately.

[Brief description of drawings]

FIG. 1 is a block diagram showing an overall configuration of a facial expression detection device according to an embodiment of the present invention.

2 is an image taken by the CCD camera in FIG. 1, in which a detection region extracted by the extraction unit in FIG. 1 is shown.

FIG. 3 is a block diagram of the wavelet filter in FIG.
It is a block diagram which shows the structure of one filter bank.

FIG. 4 is a structural diagram of an image of spatial frequency obtained by the wavelet filter in FIG.

5 is an image of a spatial frequency when the video signal in the detection region including the eyebrows and eyes shown in FIG. 2 actually passes through the wavelet filter in FIG.

FIG. 6 is a graph showing changes in average power obtained from the average power calculation unit in FIG. 1 when the subject is angry.

FIG. 7 is a graph showing changes in average power obtained from the average power calculation unit in FIG. 1 when the subject blinks.

[Explanation of symbols]

 10 CCD camera 12 Low-pass filter 14 A / D converter 16 Extraction unit 18 Wavelet filter 20 Average power calculation unit 22 Steady-state storage memory 24 Difference calculation unit 181 to 190 Frequency band AV analog video signal AVL Low frequency analog video signal DVL, DVLe Low range digital video signal F Spatial frequency Pc, Pn Average power

Claims (2)

[Claims] 1. A photographing means for photographing a face of a person to generate a video signal, and a wavelet transforming means for wavelet transforming the video signal to generate a frequency signal in a spatial frequency domain for each predetermined band. An average power calculation means for calculating the average power of the frequency signal for each of the bands, the average power sequentially given from the average power calculation means,
A facial expression detection device comprising: a difference calculation unit that calculates a difference from a corresponding average power obtained from the face when the person's face has no expression. 2. An extraction means for extracting a portion corresponding to a predetermined region where the facial expression of the person is likely to appear from the video signal given from the photographing means and giving it to the wavelet transformation means. The facial expression detection device according to claim 1.