JP2000268161A - Real time expression detector - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce burdens on a user without contacting, to be solid relating to the rotation of a face and to mitigate the limitation of an operation. SOLUTION: The face image of a testee is picked up by a CCD camera and the CCD camera is controlled so as to match the centroid of a skin color area and the center of an input image in an image pickup control part 7. Also, in a feature point detection part 4, the positions of the eyes and the mouth are estimated from face image signals. The inclination angle of the face is estimated based on the estimated positions of both eyes, the face image inside a detection area is rotated for the angle and an expression estimation part 5 performs two-dimensional discrete cosine transformation to the rotated detection area and obtains the change of spatial frequency components from the time of non expression detection. In an expression reproducing part 6, by using a parameter learned by genetic algorithm beforehand, the obtained change data of the spatial frequency components are transformed to the deformation of a three-dimensional face model and an expression is reproduced.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a real-time facial expression detection device, and more particularly to a real-time facial expression detection device for realizing communication between a plurality of persons at a distance through a virtual space. The present invention relates to a real-time expression detection device.

[0002]

2. Description of the Related Art A realistic communication conference system in which a communication environment is limited to a communication conference has already been proposed.
In such a system, for example, when a person wears a helmet to which a small camera is fixed, the expression of the face of the other person can always be observed from a fixed position. This eliminates the need for automatic face tracking processing.

On the other hand, as a technique for detecting a feature point of a face,
Others using feature point detection using an edge image, matching using a two-dimensional template, and the like have been proposed.

[0004]

In the above-described method in which a person wears a helmet to which a small camera is fixed, the burden of attaching the helmet remains, and in the method using an edge image or a two-dimensional template, calculation costs are reduced. There was a problem.

[0005] Therefore, a main object of the present invention is to provide a real-time facial expression detecting device which is non-contact, requires less load on the user, is robust with respect to the rotation of the face, and alleviates the restriction on operation. It is to be.

[0006]

The invention according to claim 1 is
What is claimed is: 1. A real-time facial expression detecting device for detecting a facial expression of a subject in real time, comprising: an imaging unit for capturing a facial image of the subject; An imaging control unit that controls the imaging unit so that the center of the face coincides with the center of the input image; and a feature point detection unit that detects the positions of the eyes and the mouth from the face image signal output from the imaging unit. Image rotation means for estimating the face inclination angle based on the positions of both eyes and rotating the face image in the detection area by the angle, and two-dimensional discrete cosine for the detection area in which the face image is rotated by the image rotation means Perform the conversion,
Using a facial expression estimating means for obtaining a change in the spatial frequency component from the time of detection of no expression and a parameter learned in advance by a genetic algorithm, the spatial frequency component change data obtained by the facial expression estimating means is converted into a three-dimensional face model. Facial expression reproducing means for converting the expression into a deformation and reproducing the expression.

According to a second aspect of the present invention, the feature point detecting means of the first aspect creates a skin color and other binary images from the image signal from the imaging means as pre-processing, and searches from the position of the center of gravity of the image. Then, the element whose pixel value changes is found, the contour is traced, and the center of gravity of the area inside the contour is found to find the positions of the eyes and the mouth.

[0008]

FIG. 1 is a block diagram of an apparatus for implementing an embodiment of the present invention. In FIG.
A face image of a subject is imaged by a CCD camera 1 as an imaging means, an image signal of the imaged output is given to an image processing device 2, and a real-time expression of the face image according to the present invention is detected and displayed on a display unit 3. You.

FIG. 2 is a diagram showing a processing procedure by the image processing apparatus shown in FIG. 1, and FIG. 3 is a flowchart showing a more specific processing procedure of the feature point detecting section in FIG.
FIG. 4 is a diagram for explaining a method of rotating an image in a detection area.

An image signal of a face imaged by the CCD camera 1 shown in FIG. As shown in FIG. 2, the image processing device 2 has functions as a feature point detecting unit 4, a facial expression estimating unit 5, a facial expression reproducing unit 6, and an imaging control unit 7. These functions are realized by software processing.

The imaging control unit 7 performs a binary conversion on the input image based on the skin color information of the subject which has been learned in advance, specifies the face position using the binary image, and matches the center of the input image with the center of the face. The feedback control of the CCD camera 1 is performed in such a manner.

The feature point detecting section 4 specifies the position of the eye by one-dimensional template matching based on the luminance information of the face image signal, and estimates the position of the mouth from the positions of both eyes. Then, the input image is changed to 2 based on the skin color information of the subject that has been learned in advance.
Value conversion, the face position is specified using the binary image, and the CCD camera 1 is moved so that the center of the face matches the center of the input image.
Feedback control. In addition, the feature point detection unit 4 estimates the inclination angle of the face based on the estimated positions of the eyes, and rotates the face image in the detection area by the angle.

The facial expression estimating unit 5 performs a DCT (two-dimensional discrete cosine) transform on a region important for capturing a facial expression change, and obtains a spatial frequency component change from the time when no facial expression is detected. The facial expression reproducing unit 6 converts the change data of the spatial frequency component obtained by the facial expression estimating unit 5 into a deformation of the three-dimensional face model using the parameters learned in advance by the genetic algorithm, To reproduce the expression.

Further, specific operations of the feature point detecting section 4, the facial expression estimating section 5, and the facial expression reproducing section 6 will be described. First,
In order to detect the facial expression, the feature point detection unit 4 can more accurately detect the feature region by capturing the feature region with as many pixels as possible. However, when the face is displayed on the entire screen, the subject's movement of the face is suppressed, and the burden imposed increases. Therefore, the head movement is tracked by the CCD camera 1 so that the subject can be in a free posture while capturing the face of the person as large as possible.

Basically, the movement of the head is tracked by tracking the subject's pupil. However, a state in which tracking is impossible occurs due to intense head movement, depression, or turning. In order to prevent this as much as possible and to quickly perform the tracking process, a process of optimizing the background color to the skin color serving as an index of the subject is performed. That is, a difference image based on chromaticity information is obtained from the image including only the background and the image including the subject, and the region is divided into the human region and the background region by the threshold value.

Each chromaticity value (I, Q value) of the chromaticity region is centered on the chromaticity value of a representative skin color obtained empirically, and the chromaticity value is included in the human region. Number of pixels:
The number of pixels in which the chromaticity value is included in the background area is obtained. If the ratio of classification into the human area is greater than a certain value, the chromaticity value is regarded as the skin color.

The face motion is tracked using the subject's unique skin color information obtained by the above processing. Next, an initial method for specifying an eye-mouth area will be described.

In accordance with the skin color information obtained as described above from the image of the subject, skin color and other binary information are created. Starting from the position of the center of gravity of the image, an element whose pixel value changes in the direction of a fixed angle (is not a skin color) is searched. The outline composed of the found pixels is tracked, and if the outline length is out of a certain range, the point where the outline intersects with the extension of the search start point and the found point is set as a new search start point. . If the search fails and reaches the end of the image, the search starting point is changed and the search is performed again. When the desired outline is obtained, the center of gravity of the area inside the outline is obtained, and this is set as the position of the eye.
Similarly, the initial position of the mouth is determined, and the relative positional relationship between the eyes and the mouth of the person is stored as initial information.

In this embodiment, the search range of the eye position is a rectangular area of 100 × 100 pixels centered on the position of the previous eye. Based on the coordinates of the pupil obtained by the above-described processing, a one-dimensional template of luminance information having a length of 100 pixels in the y-axis direction is created. When the next frame to be tracked is input, a correlation is calculated for the pixels in the search area according to the following equation (1), and the average coordinates of the three points having a high degree of correlation are calculated for a new eye. Position.

[0020]

(Equation 1)

For the mouth area, template matching is not used because the shape is severely deformed. However, since the position of the eye is already known and the relative position between the eye and the mouth is determined in the process of the eye-mouth position initialization process,
Based on the relative coordinate positions, the position of the mouth is calculated from the positions of both eyes.

If the movement of the head is slight, the entire face is contained in the screen, and the expression can be detected. However, in this case, the subject is forced to fix his face so that it always fits in the viewfinder of the camera. For example, despite a relatively natural movement such as shaking the head, with a fixed camera, a part of the face goes out of the screen, it is difficult to capture the entire face, and the facial expression is correctly detected Not. Therefore, in this embodiment, this problem is solved by controlling the camera with the electric pan head and controlling the CCD camera 1 so as to always capture the subject.

In controlling the CCD camera 1, the coordinate system of the control system of the camera and the coordinate system of the input image are normal when the subject is located at about 1 m from the CCD camera 1, which is the current usage condition. The coordinate matching function is optimized in advance so as to operate. By calculating the center of gravity of the binarized image obtained by extracting the skin color from the input image and controlling the camera platform so that the point is located at the center in the camera coordinate system, the center of the person's face can always be captured. it can.

When capturing the features of the facial expression from the eye and mouth regions, there is no problem if the face is directly facing the CCD camera 1, but the subject tilts his / her head as shown in FIG. In this case, the eyes in the rectangular area are inclined, and it is not possible to obtain a correct facial expression.

Therefore, in this embodiment, when the face does not face the CCD camera 1, the feature point detecting section 4 performs a process of rotating the image in the detection area. That is, the steps shown in FIG. 3 (abbreviated as S in the drawing)
In S1, the inclination angle θ of the face is determined based on the binocular detection positions as shown in FIG. 4B, and in FIG.
The image in the eye area is rotated by −θ as shown in FIG. Similarly, the mouth area position is determined in step S3,
In step S4, the mouth area image is rotated by -θ.

Next, the facial expression estimation by the facial expression estimation unit 5 will be described in more detail. The luminance information of the face image of the person is converted into the spatial frequency domain, and the change is analyzed to estimate the facial expression. As the facial expression changes, the spatial frequency component of the face image also changes.However, in order to uniquely determine what kind of change has occurred due to which part of the face, the spatial frequency is divided into several areas and the spatial frequency is changed. Convert.

In this embodiment, a certain rectangular area is allocated to the image area of both eyes and the mouth where the change is particularly noticeable among the parts forming the facial expression of the person, and that part is independently subjected to the spatial frequency. The facial expression is reproduced by conversion. However, since the movement of the mouth of a person is more complicated than that of the eyes,
It is further divided into six areas.

As an example of the spatial frequency conversion, when an image of an eye is used, when the eyes are closed, the space between the upper and lower eyelids is reduced, so that the spatial frequency component of the image in the vertical direction is increased. In the horizontal direction, the degree of horizontal movement of the outline of the eyelid in the horizontal direction decreases, so that the spatial frequency component in the horizontal direction decreases in inverse proportion to the vertical direction.

When the eyes are open, the spatial frequency component in the horizontal direction, which is completely opposite to the case where the eyes are closed, increases,
Vertical spatial frequency components are reduced. When the detection of these spatial frequency components is started, an expressionless state is first presented to the subject, and the facial expression is detected as a relative change from the spatial frequency component data.

In this method, since an image is converted into a spatial frequency component and the facial expression is detected by analyzing the electric energy without using the information on the position, the image is converted into a rectangular area for performing the spatial frequency conversion. As long as the eye (or mouth) is closed, the estimation can be performed normally even if the rectangular area is erroneously acquired.

As described above, in this embodiment, since the detection is performed based on the amount of power in the spatial frequency domain, no phase information is required. From this point, DCT transform, which is excellent in calculation cost, is used.

The luminance image f (x, y) (x = 0,..., M−
1; y = 0, ..., N-1) DCT result F
(U, v) (u and v are spatial frequency components in x and y axis directions)
Is shown in equation (2).

[0033]

(Equation 2)

For the size (M, N) for performing DCT, the rectangular area to be transformed is divided into small areas of M = N = 8 pixels in order to reduce the calculation cost.
Perform CT conversion.

FIG. 5 is a diagram for explaining vectorization of a spatial frequency component by DCT. In this embodiment, in order to capture the change in the shape of the face part, the sum of power amounts is calculated for the spatial frequency components in the vertical, horizontal, and oblique directions in the range shown in FIG. As shown in FIG. 5, in the converted spatial frequency domain, the high frequency domain is not used because the amount of power variance is large due to the inclusion of noise or the like.
The difference from the expressionless state of the sum of the electric energy in each of the vertical, horizontal, and diagonal directions obtained in this manner is defined as the characteristic amount of the facial expression.

Next, the expression reproducing section 6 will be described in more detail. In this embodiment, a facial expression is reproduced by deforming a three-dimensional face model created in advance in accordance with the estimated facial expression data. Therefore, a rule for deforming the three-dimensional face model is required. In this embodiment, taking into account the movement of the muscle corresponding to the change in the facial expression of the person, 10
Types of three-dimensional model deformation rules (hereinafter, referred to as basic expressions) are determined. Basically, deformation rules are defined in a manner that simulates the movement of human muscles.However, since facial expression detection is performed on typical characteristic regions, some regions do not capture facial expression changes. . Therefore, the quality of the expression screen is improved by slightly emphasizing and reproducing the expression. Therefore, some of the ten basic facial expressions have conversion rules expressing movements impossible for humans, and the facial expressions of a person are reproduced by properly mixing these basic facial expressions.

In order to reproduce the expression of a person on a three-dimensional face model, data serving as an index is required. In this embodiment, first, various facial expressions are expressed by a model person, and their facial images are recorded (for example, 1903 frames). The basic expression is adjusted for each frame to reproduce the expression, and the mixture ratio of each basic expression is stored as index data. These data mixing ratio vertex vector wireframe face model as P _j, when the mixing ratio and C _j, the movement vector W _j of the three-dimensional coordinates of the wireframe each vertex is expressed by the equation (3).

[0038]

(Equation 3)

In order to reproduce a facial expression with a three-dimensional face model, it is necessary to convert a change in spatial frequency component from the time of no expression to a mixture ratio of basic expressions. This association is the fourth (4)
The value of the coefficient is determined by using a conversion formula represented by the following formula, and a genetic algorithm (GA) is determined.
This is performed using Formula (4) shows a basic formula for calculating the mixture ratio to be learned by the GA.

[0040]

(Equation 4)

[0041] C _j is a mixing ratio, ABCD is a coefficient for determining, Hdi, Vdi, Ddi is the change amount of the spatial frequency component of the horizontal vertical diagonal directions. In practice, the entire face is divided into six regions (eye, eyebrows, nose, etc.), and independent mixing ratios are obtained for each region. In addition, the mixing of basic facial expressions that require spatial frequency component changes in some regions In the ratio formula, the parameter increases accordingly. The number of parameters actually used in this embodiment is 115.

FIG. 6 is a diagram showing an example of facial expression reproduction according to this embodiment. In FIG. 6, the left is the face image of the subject, the center shows the case where the teacher information is input, and the right shows an example reproduced by an expression learned by the genetic algorithm. As is clear from FIG. 6, according to this embodiment, it is understood that the expression of the person can be accurately reproduced.

The embodiments disclosed this time are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

[0044]

As described above, according to the present invention, a face image of a subject is imaged, a flesh-colored area is extracted from the image, the imaging means is controlled based on the center of gravity, and the positions of the eyes and the mouth are determined. The face inclination angle is estimated based on the estimated position of the eyes, the face image in the detection area is rotated by the angle, and the two-dimensional discrete cosine transform is performed on the rotated detection area. The change of the spatial frequency component from the time of detecting the facial expression is obtained, and the change data of the spatial frequency component is converted into the deformation of the three-dimensional face model by using the parameter learned by the genetic algorithm, so that the facial expression can be reproduced. Therefore,
By performing face expression detection based on an image obtained from an image input independent of the head, non-contact detection can be realized.
In addition, since the image calculation corresponding to the rotation of the head corresponds to the stage before the detection of the facial expression feature amount, the degree of freedom of the user's movement can be improved, and the facial expression detection accuracy can be improved.

[Brief description of the drawings]

FIG. 1 is a block diagram of an apparatus for realizing an embodiment of the present invention.

FIG. 2 is a diagram showing a processing process in the image processing apparatus shown in FIG.

FIG. 3 is a flowchart illustrating a processing procedure of an image rotation processing process in a feature point detection unit in FIG. 2;

FIG. 4 is a diagram for explaining a method of rotating an image in a detection area.

FIG. 5 is a diagram for explaining vectorization of a spatial frequency component by DCT.

FIG. 6 is a diagram showing an example of facial expression reproduction according to the embodiment of the present invention.

[Explanation of symbols]

 Reference Signs List 1 CCD camera 2 Image processing device 3 Display unit 4 Feature point detection unit 5 Expression estimation unit 6 Expression reproduction unit

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] December 27, 1999 (1999.12.
27)

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claims

[Correction method] Change

[Correction contents]

[Claims]

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] 0006

[Correction method] Change

[Correction contents]

[0006]

The invention according to claim 1 is
A real-time facial expression detection device for detecting a facial expression of a subject in real time, comprising: an imaging unit for capturing a face image of the subject; and a skin color region extracted from a face image signal output from the imaging unit, and a center of gravity thereof is calculated. calculated, an imaging control means for controlling the imaging means in such a manner that the center of the face matches the center of the input image, the pre-treatment
Skin color and other binary images from the image signal from the imaging means
Create an image, search from the center of gravity of the image
Finds the changing element of and traces the contour,
A feature point detection means for determining the position of the eyes and mouth from the center of gravity of the side area, on the basis of the positions of both eyes was determined al estimates the inclination angle of the face, rotating the face image of the angle only detection region image Rotation means , two-dimensional discrete cosine transform for a detected area where the face image is rotated, and a facial expression estimating means for obtaining a change in a spatial frequency component from a time when no expression is detected; and a parameter learned in advance by a genetic algorithm. And a facial expression reproducing means for converting the change data of the spatial frequency component obtained by the facial expression estimating means into a deformation of the three-dimensional face model to reproduce the facial expression.

[Procedure amendment 3]

[Document name to be amended] Statement

[Correction target item name] 0007

[Correction method] Deleted

[Procedure amendment 4]

[Document name to be amended] Statement

[Correction target item name] 0044

[Correction method] Change

[Correction contents]

[0044]

As is evident from the foregoing description, according to the present invention, captures the face image of the subject, to extract the skin color area from the image and controls the imaging unit based on the centroid, the imaging hands as a pretreatment
Create skin color and other binary images from image signal from stage
Search from the position of the center of gravity of the image, and
Traces the contour by finding the element that
The position of the eyes and the mouth is obtained from the center of gravity, the inclination angle of the face is estimated based on the obtained positions of the two eyes, the face image in the detection area is rotated by that angle, and the two-dimensional discrete The cosine transform is performed to find the change in the spatial frequency component from the time when the expressionless expression is detected. Using the parameters learned by the genetic algorithm , the spatial frequency component change data is converted into a three-dimensional face model deformation to express the facial expression. Can be reproduced. Therefore, non-contact detection can be realized by performing face expression detection based on an image obtained from an image input independent of the head. In addition, the image operation corresponding to the rotation of the head corresponds to the previous stage of the facial expression feature amount detection,
The degree of freedom of the user's operation can be improved, and the expression detection accuracy can be improved.

 ────────────────────────────────────────────────── ─── Continued on the front page (72) Inventor Atsushi Atsushi Otani 5 Shiraya, Seika-cho, Soraku-cho, Kyoto Pref. No. 5 Sanraya, Inaya, Seika-cho, Gunma F-term (reference) 5A050 AA08 BA09 BA12 CA07 DA04 EA12 EA13 EA28 GA06 5B057 CB13 CD03 CD11 CG05 DA07 DA08 DC06 DC17 DC25 DC33 DC40

Claims (2)

[Claims] 1. A real-time facial expression detection device for detecting a facial expression of a subject in real time, comprising: an imaging unit for imaging a face image of the subject; and a skin color region from a face image signal output from the imaging unit. Imaging control means for extracting and calculating the center of gravity and controlling the imaging means so that the center of the face coincides with the center of the input image; feature inspection for detecting the positions of eyes and mouth from a face image signal output from the imaging means Output means; an image rotation means for estimating a tilt angle of a face based on the positions of both eyes estimated by the feature point detection means, and rotating the face image in the detection area by the angle; and a face image by the image rotation means. Expression estimating means for performing a two-dimensional discrete cosine transform on the rotated detection region to obtain a change in spatial frequency component from the time of expressionless detection, and a parameter learned in advance by a genetic algorithm Using data, with a facial expression reproduction means for reproducing the expression to convert the change data of the spatial frequency components obtained by the facial expression estimator to deformation of the three-dimensional face model, real-time facial expression detection device. 2. The feature point detecting means generates a skin color and other binary images from an image signal from the image pickup means as preprocessing, and searches from the center of gravity of the image to search for an element whose pixel value changes. 2. The real-time facial expression detecting device according to claim 1, wherein the real-time facial expression detecting device is obtained by finding the position of the eye and the mouth by finding the center of gravity and finding the center of gravity of the region inside the outline.