JP2004505353A - Facial feature special method and device by accurate landmark detection on expressionless facial image - Google Patents

Abstract

The present invention provides a method and apparatus for customizing a visual sensor for tracking a facial feature using an expressionless facial image of an actor. This method creates a correction graph to improve the performance of the sensor tracking the performer's facial features.

Description

[0001]
[Related application]
This application claims priority to US Provisional Patent Application No. 60/220288, filed July 24, 2000, entitled "Special Features and Apparatus for Facial Features by Accurate Mark Detection on Expressionless Face Images," Claims priority of co-pending U.S. patent application Ser. No. 9 / 188,079 entitled "Wavelet-Based Facial Motion Capture for Avatar Video."
[0002]
BACKGROUND OF THE INVENTION
The present invention relates to avatar video, and more particularly to facial feature tracking technology.
[0003]
Avatar-filled virtual spaces are an attractive way to experience a common environment. However, photographic avatar animation generally requires cumbersome tracking of the performer's movements, especially tracking of the facial features.
[0004]
Therefore, there is a need for an improved facial feature tracking technology. The present invention fulfills this need.
[0005]
Summary of the Invention
The present invention provides a method and associated apparatus for customizing a visual sensor utilizing an expressionless face of an actor. This method includes capturing the performer's expressionless face image and automatically detecting the position of the screen feature on the expressionless face image using an elastic bunch graph matching method. A node (reference point: node) is automatically assigned to the position of the facial feature on the expressionless facial image of the performer. Thereafter, the node position is manually corrected on the expressionless facial image of the performer.
[0006]
Further, the method can create a correction graph based on the corrected node positions.
[0007]
Other features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings.
[0008]
DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
The present invention provides a method and apparatus for customizing a visual sensor for tracking a facial feature using an expressionless facial image of an actor. This method creates a correction graph that is used to improve the performance of the sensor used for the actor's facial feature tracking.
[0009]
As shown in FIG. 1, in this method, a facial image of an actor is captured (block 12). The expressionless facial image is captured using the visual sensor customization wizard 22 as shown in FIG. An example image 24 is shown to the actor to show the integrity of the captured image 26.
[0010]
Next, the facial feature position is automatically detected using a telescopic bunch graph matching method (block 14). A facial feature detection method using a stretchable bunch graph matching method is disclosed in U.S. Patent Application No. 09 / 188,079. In the expansion / contraction graph matching technique, an image is converted into a Gabor space using a wavelet transform based on a Gabor wavelet. The transformed image is represented by a complex wavelet component value associated with each pixel of the original image.
[0011]
As shown in FIG. 3, node 28 is automatically scored on the facial image at a particular facial feature location (block 16). Due to the intervening actor-specific image features, the facial features graph placed on the actor's face image will include node positions that are not properly located on the face image. For example, the four-point node of the performer's eyebrows is positioned slightly above the eyebrows of the facial image.
[0012]
The apparatus of the present invention uses the vision sensor customization wizard 22 to pick up and move the node 28. The node is manually moved over the expressionless facial image using a pointing device such as a mouse and pulled to the desired selected position (block 18).
[0013]
For example, as shown in FIG. 4, the node arrangement points on the eyebrows of the actor image are adjusted to exactly match the actor eyebrows according to the example image 24.
[0014]
As shown in FIG. 5, after the nodes 28 of the facial features from A to E have been accurately scored on the expressionless facial image 24, the image jets are recalculated for each facial feature and the bunch graph Is compared with the corresponding jet of the gallery 32. The bunch graph gallery includes a multi-person (N) sub gallery. Each person in the sub-gallery has a jet of expressionless facial images 34 and facial expression facial images 36 to 38 representing, for example, a smile or wonder.
[0015]
Each facial feature jet from the corrected actor image 24 is compared to the corresponding facial feature jet from several sub-gallerys' expressionless jets. The expressionless jet of the sub gallery of the feature (feature A) that is closest to the jet of facial feature A is selected to create the jet gallery of feature A in the correction graph 40.
[0016]
In another embodiment, feature E is of interest, and the sub gallery for the number of people (N) has a featureless jet of feature E that most closely resembles the jet of feature E from expressionless image 24. . A feature graph E jet is created using features E jets from expressionless jets along with features E jets from respective expressive features jets 36-38 from sub-gallerys N.
[0017]
Thus, for the expressionless facial image 24, the correction graph 40 is formed using the best jet from the gallery 32 forming a bunch graph.
[0018]
The resulting correction graph 40 provides a more powerful sensor for tracking node positions. A custom-built workplace tracking sensor that incorporates a correction graph further provides a photographic avatar and an enhanced virtual space experience.
[0019]
Although the foregoing description discloses embodiments of the present invention, those skilled in the art will be able to modify those embodiments as appropriate within the scope of the present invention.
[Brief description of the drawings]
FIG. 1 is a flowchart illustrating a method of customizing face feature tracking by accurate landmark detection on a faceless facial image according to the present invention.
FIG. 2 is a visual sensor customization wizard showing an actor's camera image and a general model image.
FIG. 3 is a visual sensor specialization wizard after automatic detection of a node position on a camera image of an actor's face and a scoring process;
FIG. 4 is a visual sensor customization wizard showing correction node positions for creating a correction graph according to the present invention.
FIG. 5 is a block diagram showing a correction graph creation technique using an expressionless facial image according to the present invention.

Claims (8)

A method of customizing facial features tracking,
Capturing the actor's expressionless facial image;
Automatically detecting the facial feature position on the expressionless facial image using a telescopic bunch graph matching method,
Automatically arranging nodes at the facial features on the expressionless facial image;
Manually correcting the position of the node on the expressionless facial image. The method of claim 1, further comprising creating a correction graph based on the corrected node positions. A device that specializes in facial feature tracking,
Means for capturing an expressionless facial image of the performer;
Means for automatically detecting the position of the facial features on the expressionless facial image using a stretchable bunch graph matching method,
Means for automatically arranging a node at the facial feature position on the expressionless facial image;
Means for manually correcting the position of the node on the expressionless face image;
An apparatus characterized by comprising: The apparatus of claim 3, further comprising means for creating a correction graph based on the corrected node positions. A method of customizing facial features tracking,
Capturing the actor's expressionless facial image;
Automatically detecting face features on the expressionless face image using image analysis based on wavelet component values created by the wavelet transform of the expressionless face image;
Means for automatically arranging a node at the facial feature position on the expressionless facial image;
Means for manually correcting the position of the node on the expressionless face image;
A method comprising: The method of claim 5, wherein the wavelet transform utilizes Gabor wavelets. A method of customizing facial features tracking,
Capturing the actor's expressionless facial image;
Detecting the position of the facial features on the facial image using image analysis based on wavelet component values created by the wavelet transform of the expressionless facial image;
Creating a correction graph for providing an expressive feature based on the wavelet component values at the facial feature location on the face image;
A method comprising: The method of claim 7, wherein the wavelet transform utilizes Gabor wavelets.

Images