JP3432816B2 - Head region extraction device and real-time expression tracking device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image of a person.
A head region extraction device that extracts the head region of a person from an image
To do. In addition, the present invention uses C instead of transmitting the face of the person.
It is applied to a communication system such as a videophone that communicates person images with each other by transmitting the image of the G character to the other party, and in particular, the three-dimensional posture information of the head and the facial expression are obtained from the face image captured by the camera. The present invention relates to a real-time facial expression tracking device using a proxy response that measures and controls the movement of a CG character based on the measurement result.

[0002]

2. Description of the Related Art For example, FIG.
1 is a view showing a conventional virtual transformation device (first conventional technique) disclosed in Japanese Patent No. 1190, which includes a video camera for inputting a face image, an electric pan head for rotating the video camera, A face image recognition device that detects the rotation of the axis of the face or the rotation around the axis of the face and the direction of the line of sight from the face image input from the video camera, and detects the shape change of both eyes and mouth, and based on this measurement result A virtual environment composition device for controlling a character in a virtual space constructed by CG (computer graphics).

In the first prior art, a face image input from a video camera is binarized by setting a skin color to 1 and a color other than the skin color to 0 according to a skin color model constructed on a preset RGB space. . Next, the center of gravity of the binarized face area is obtained, the electric pan head device is controlled so that the center of gravity is at the center of the image, and the angle of the camera is corrected. Next, the holes present in the face area are detected as both eyes and mouth based on the position of the center of gravity. Next, the eye region is tracked by template matching using a preset template, and the line-of-sight direction is obtained from the position of the black eye. Also, the angle between the straight line connecting both eyes and the horizontal axis of the image is measured, and the rotation of the face around the axis is detected from the distance between both eyes. Then, the shape change of both eyes and mouth is measured by capturing the power change in each frequency band when the discrete cosine transform is performed on the surrounding images of both eyes and mouth. The head and facial expression of the character in the virtual space constructed by CG are controlled based on the above measurement results.

Further, in the facial expression detection device (second prior art) of Japanese Patent Laid-Open No. 2000-259831, a plurality of selected feature points are tracked in images of consecutive frames, and the plurality of feature points are tracked for each frame. A Delaunay network having feature points as vertices is constructed, and by using this Delaunay network, the facial muscle model is displaced based on the movement of the characteristic point to obtain a change in the facial muscle model.

Further, in Japanese Unexamined Patent Publication No. 11-306348 (third prior art), a window mask having a fixed size is scanned over the entire image, and the luminance dispersion in the mask is normalized, thereby illuminating conditions. An invention relating to a target object detection apparatus capable of stably extracting a feature amount of a target object is disclosed even when is changed.

[0006]

SUMMARY OF THE INVENTION In the first prior art,
The face image taken by the camera is binarized based on the skin color model, holes in the face area are found, and they are made to correspond to the eyes and mouth from the position of the center of gravity of the face area. However, since the shadows and highlights caused by the unevenness of the face are inherently affected, it is very difficult to detect only eyes and mouths as holes in the first conventional technique unless the lighting conditions are carefully set. is there. In addition, the first conventional technique is based on the three axes (X
Axis, Y-axis, Z-axis) rotation cannot be measured at the same time, and the rotation of the face around the axis is determined by the distance between the eyes. For example, when the face moves away from or approaches the camera, Since the distance between the eyes inevitably changes, there was a problem in that it was considered as rotating even though it was not actually rotating.

Further, in the second conventional technique, it is necessary to trace a large number of feature points in the face image in order to measure the three-dimensional posture, so that it is difficult to perform real-time processing with hardware having low calculation ability. was there.

Further, in the third prior art, since the mask area of which size is fixed is used, it is difficult to deal with the change of the size of the face area due to individual difference or photographing distance.

The present invention has been made in view of the above,
Head region extraction that can accurately extract the head region in real time from hardware images with low calculation power by simple calculation from the face image of an unspecified person taken under arbitrary lighting conditions
The purpose is to get the device. Further, the present invention is
The head region is extracted by a simple calculation, the three-dimensional movement of the head is measured, the open / closed states of both eyes and mouth are measured, and the result is used to control the movement and expression of the head of the CG character. The purpose is to obtain a real-time facial expression tracking device.

[0010]

In order to achieve the above object, the head region extracting apparatus according to the present invention images a person.
A head region extraction device that extracts the head region of a person from video
In addition, each pixel data of the image of the target person is R,
Normalized according to the following equations for each of the G and B components: c1 = arctan (R / max (G, B)) c2 = arctan (G / max (R, B)) c3 = arctan (B / max (R, G)) The normalized data c1, c2, c3
The normalizing means to be obtained and the normalized data c1, c2, c3
Each of the included pixel data is converted into pixel data including the data of the C1-C2 space according to the following formula C1 = c2 / c1 C2 = c3 / c2.
Data conversion means for converting each and C of the converted pixel data
1 data and C2 data satisfy the following formulas th1 <C1 <th2 th1, th2; skin color extraction parameter th3 <C2 <th4 th3, th4; skin color extraction parameter , this pixel data is judged as a skin color pixel.
By extracting the head region from the captured image
And a partial area extracting unit.

A head region extraction device according to the next invention is
In the above invention, it is the same as when capturing the target person.
In the same lighting environment, an image of a predetermined area of a part of the target person's face is displayed.
Skin color sampling means for sampling an image, and the skin color
Predetermined area sampled by sampling means
Each pixel data of the image is normalized using the normalizing means.
After that, using the data conversion means,
A plurality of pixels in the predetermined area converted into pixel data
The maximum value for C1 data and
And minimum and maximum and minimum values for C2 data
Then, the skin color extraction parameters are calculated with these maximum and minimum values.
Correct meters th1, th2, th3 and th4
A flesh color extraction parameter adjusting means is further provided.
To collect.

In the head area extraction according to the next invention, in the above invention, the head area extraction means is a skin color area extraction.
The head area is extracted by extracting the maximum area from the output result.
Characterized by issuing .

The extraction of the head region according to the next invention is performed by
In the invention, the head area extracting means is
Characterized by adding expansion / contraction processing to the binary image after output
It

A head area extraction device according to the next invention is
In the above invention, the head region extraction means obtains an exclusive OR of the binary image after the expansion / contraction process and a mask image whose pixel values are all logical value levels corresponding to skin color, and the exclusive OR is performed. The entire head region is extracted by obtaining the logical sum of the image having the logical value level corresponding to the non-skin color other than the head region of the image logically ORed with the binary image after the expansion / contraction processing. It is characterized by doing.

Real-time facial expression tracking according to the next invention
The device sequentially processes the images input at a predetermined frame rate.
Video input means to capture and the captured image
A head region extracting means for extracting a head image from the image;
Candidate areas for each part including both eyes and mouth from the head area
Region candidate extraction means for extracting regions and the extracted candidate regions
Part detection tracking means for detecting the position of each part from the region
Based on the detected positions of both eyes and mouth,
Measures three-dimensional posture and opens and closes both eyes and mouth
A head three-dimensional posture / facial expression measuring means for measuring a state,
The measured three-dimensional posture of the head and the opening of both eyes and mouth
Rear that controls the movement of the CG character based on the closed state
A real-time facial expression tracking device, wherein the head region extraction means
Represents each pixel data of the image obtained by capturing the target person as R, G,
For each B component, normalize and normalize according to the following formula c1 = arctan (R / max (G, B)) c2 = arctan (G / max (R, B)) c3 = arctan (B / max (R, G)) The converted data c1, c2, c3
The normalizing means to be obtained and the normalized data c1, c2, c3
Each of the included pixel data is converted into pixel data including the data of the C1-C2 space according to the following formula C1 = c2 / c1 C2 = c3 / c2.
The data conversion means for each conversion and the converted pixel data
Expression th1 <C1 <th2 th1, th2; Skin color extraction parameter th3 <C2 <th4 th3, th4; When the skin color extraction parameter is satisfied, this pixel data is determined as a skin color pixel.
By extracting the skin color area from the captured image
And a color area extracting means.

A real-time facial expression tracking device according to the next invention, in the above invention, images the target person.
In the same lighting environment as when, a predetermined part of the target person's face
Skin color sampling means for sampling a region image
And sampled by the skin color sampling means
Each pixel data of an image of a predetermined area is used by the normalizing means.
And then normalize, and then C1-
Converted to pixel data in the C2 space, and the converted predetermined data
For C1 data using multiple pixel data of the area
Maximum and minimum values and maximum and minimum values for C2 data
And minimum value, and the maximum and minimum values of the
Color extraction parameters th1, th2, th3 and th4
And a flesh color extraction parameter adjusting means for correcting
It is characterized by

Real-time facial expression tracking according to the next invention
In the above invention, the device is the head region extracting means.
From the skin color region extraction result by the skin color region extraction means
Extracting the head region by extracting the maximum region
Is characterized by.

The real-time facial expression tracking device according to the next invention is the head region extracting means in the above invention.
Is a binary value after the skin color area is extracted by the skin color area extracting means.
It is characterized in that the image is subjected to expansion / contraction processing.

Real-time facial expression tracking according to the next invention
In the above invention, the device is the head region extracting means.
Is the binary image after the expansion / contraction process and all the pixel values
Exclusive with the mask image, which is the logical level corresponding to the color
The head of the image for which this exclusive OR is calculated
An image with logical values corresponding to non-skin colors other than the area
Obtaining a logical sum with the binary image after the expansion / contraction processing
Then, the entire head region is extracted.

[0020]

[0021]

[0022]

[0023]

[0024]

[0025]

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of a head region extracting device and a real-time facial expression tracking device according to the present invention will be described in detail below with reference to the accompanying drawings. This real-time facial expression tracking device is applied to a communication system such as a videophone that communicates person images with each other by transmitting the image of the CG character to the other party instead of transmitting the face of the person.

The embodiments of the present invention will be described below with reference to FIGS.
Will be explained. FIG. 1 shows a conceptual configuration of the real-time facial expression tracking device according to the present embodiment.

The real-time facial expression tracking device shown in FIG. 1 shows the functional configuration of a program executed by a personal computer or workstation, for example. The real-time facial expression tracking device shown in FIG. 1 includes a video input unit 1 for capturing a video input from a video capturing unit such as a video camera 80, and a head from a human video input via the video input unit 1. Head region detecting means 2 for detecting a region, part region candidate extracting means 3 for extracting candidate regions for both eyes and mouth from the head region extracted by the head region detecting means 2, and part region candidate extracting means 3 Both the eyes and the mouth area are detected from the candidate area extracted in 1., the position that changes every hour is tracked, and the open / closed state of each part is measured, and the position of both eyes and the mouth detected by the area detection and tracking means 4 A head 3D posture / facial expression measuring means 5 for measuring the 3D posture and facial expression of the head.

Further, the head region detecting means 2 is based on the skin color sampling means 6 for sampling the skin color of a person in the environment where the image is taken (such as under the lighting environment) and the skin color information sampled by the skin color sampling means 6. Skin color extraction parameter adjusting means 7 for adjusting extraction parameters,
Among the extracted skin color regions, the skin color region extraction unit 8 extracts the skin color pixels from the input image based on the skin color extraction parameter adjusted by the skin color extraction parameter adjustment unit 7 and classifies the extracted pixels into clusters (regions). Head region extracting means for selecting all the pixels relating to the head of the person as regions by selecting a head region from the region and filling all small regions such as holes and crevices in the head region (replacement with skin color) 9 and 9.

The part area candidate extracting means 3 comprises a head area brightness averaging means 10 for averaging the brightness values of the head area, and a pixel selecting means 11 for extracting candidate areas for both eyes and mouth.

The part detecting / tracking means 4 specifies a region corresponding to each of the eye and mouth candidate regions extracted by the part region candidate extracting means 3, and the part detecting means 12 detects both eyes and It comprises an initial position setting means 13 for storing the initial position of the mouth, and a part tracking means 14 for detecting the position in the current frame from the position of each part stored up to the previous frame.

The head three-dimensional posture / facial expression measuring means 5 sets an affine basis which is a reference for obtaining the three-dimensional head posture based on the initial position of each part set by the initial position setting means 13. The setting means 15, the head rotation amount estimating means 16 for obtaining a provisional rotation amount of the head around the horizontal axis and the vertical axis, and the position of each part detected by the part detecting means 12 are set by the affine base setting means 15. Virtual 3
The facial expression is tracked by measuring a three-dimensional posture of the head from a two-dimensional point in the image corresponding to a point in the three-dimensional space and the open / closed state of each part (both eyes and mouth). The open / close state measuring means 18 is provided.

The character control device 90 uses the three-dimensional posture of the head and the open / closed state of each part (both eyes and mouth) input from the three-dimensional posture / facial expression measuring means 5 to determine the three-dimensional C.
By controlling the G character, the movement and facial expression of the CG character are changed in real time by following the movement and facial expression of the user captured by the video camera 80.

FIG. 2 is a flow chart for explaining the outline of the operation of the calibration phase of the real-time facial expression tracking device of FIG. FIG. 3 is a flowchart for explaining the outline of the operation of the tracking phase of the real-time facial expression tracking device of FIG. The outline of the operation of the real-time facial expression tracking device will be described with reference to FIGS. 2 and 3.

The operation procedure performed by the real-time facial expression tracking device includes a calibration phase for acquiring the positions of both eyes and mouth and a state of no expression as information for tracking the movement of the head, and an actual head part. There is a tracking phase that tracks the movements of the eyes and both eyes and mouth, and measures the head posture and the open / closed state of both eyes and mouth, that is, the facial expression.

In the calibration phase, first,
An image from the video camera 80 is captured by the image input means 1 (step S100). When a video image of a person is taken by the video camera 80, by instructing the user to "face the camera, open both eyes, close the mouth", a person's image without a facial expression can be obtained. obtain. Next, the head area detection unit 2 samples the skin color of the user under the photographing environment (step S1).
10), using this sampling data, the preset skin color extraction parameter is adjusted (step S12).
0). Then, the skin color area is actually extracted using the adjusted skin color extraction parameter (step S130), and the head area is detected from the extracted area (step S14).
0). Next, the part area candidate extraction means 3 extracts both eye and mouth candidate areas from the extracted head area (step S150), and the part detection and tracking means 4 detects both eye areas and mouth area, respectively (step S160). The position, size, and initial value of the template of each part from the detected eyes and mouth area are stored (step S170). Finally, in the head 3D posture / facial expression measuring means 5, based on the obtained positions of both eyes and mouth, an affine basis (a virtual point in a 3D space) for obtaining 3D posture information of the head in the tracking phase. ) Is set (step S180).

In the tracking phase, the image input means 1 captures an image from the video camera 80 (step S200). The head area detecting means 2 extracts the skin color from the captured image using the skin color extraction parameter set in the calibration phase (step S210), and detects the head area from the extracted area (step S220). Next, the part area candidate extraction means 3 extracts candidate areas for both eyes and mouth (step S230). Next, the part detection tracking means 4
Is based on the position of both eyes and mouth detected in the previous frame,
Both eye and mouth areas in the current frame are detected from the candidate areas extracted by the part area candidate extracting means 3 (step S240). Next, the head 3D posture / facial expression measuring means 5
In step 3, the three-dimensional posture information of the head is measured from the positions of both eyes and mouth (two-dimensional image points) detected by the part detection / tracking means 4 and preset virtual points in the three-dimensional space (step S
250), and measures the open / closed state of both eyes and mouth based on the measurement information (step S260). Finally, the measured open / closed state information of both eyes and mouth and the posture information of the head are input to the character control device 90, and the movement and expression of the head of the CG character are controlled by the character control device 90 (step S270).

[Calibration Phase] Next, the operation in the calibration phase of the real-time facial expression tracking device shown in FIG. 1 will be described in detail with reference to FIGS.

(A) Processing by the head area detecting means 2 First, the details of the processing of steps S110 to S140 of FIG. 2 performed by the head area detecting means 2 will be described with reference to FIGS.

FIG. 4 is a diagram for explaining the operation of the skin color sampling means 6 in the head area detecting means 2.
FIG. 5 is a flow chart for explaining the operations of the skin color sampling means 6 and the skin color extraction parameter adjusting means 7.

First, as shown in FIG. 4, in order to sample the skin color of the user in the lighting environment used, as shown in FIG.
A sampling window 20 for designating a sampling area is displayed so as to overlap the captured image 19 (step S300). Next, the user moves the sampling window 20 to a position where only the skin color such as cheek or forehead can be extracted by using a mouse or other pointing device, a keyboard or the like, and informs the system that sampling is possible (step S310). ).
Note that the user may move his / her head to adjust the position according to the position of the sampling window 20 displayed first.

Next, the colors of all the pixels in the sampling window 20 are mapped to a color space (skin color model space) for skin color extraction (step S320), and the maximum and minimum values of the mapped pixels in the mapping space are calculated. The preset skin color extraction parameter is adjusted by using (step S330).

Here, as the skin color extraction space, for example, a color space relatively robust against a change in brightness is newly constructed, or a color data space of pixels (R, G, B space) is constructed. To use. Here, the following color space that is relatively robust to luminance changes is used.

Assuming that R (red), G (green), and B (blue) are the components of the three primary colors of each pixel, the colors are first normalized by the following equation.

[0051] c1 = arctan (R / max (G, B)) ... Equation (1) c2 = arctan (G / max (R, B)) ... Equation (2) c3 = arctan (B / max (R, G)) ... Equation (3)

The color normalized by the above equation is further converted by the following equation.

[0053] C1 = c2 / c1 ... Formula (4) C2 = c3 / c2 ... Formula (5)

In the skin color area extracting means 8, the color converted from the RGB space to the C1-C2 space by the equations (4) and (5) falls within the skin color range defined by the following equations (6) and (7). A flesh color area is extracted from the input image by determining whether or not there is any. th1 <C1 <th2 ... Equation (6) th3 <C2 <th4 ... Equation (7)

The skin color extraction parameter adjusting means 7 uses the skin color extraction parameter (threshold value) th used in the skin color extraction.
1 to th4 are made variable by using the sampling data of the skin color sampling means 6 in accordance with different illumination conditions or the difference in skin color of each person. That is, the skin color extraction parameter adjusting means 7 is the skin color sampling means 6
The RGB data of the pixels sampled in 1. is mapped to the C1-C2 space, the maximum value and the minimum value at that time are obtained for C1 and C2, and the minimum value for C1 is the threshold value th1.
, The threshold value th2 is changed with the maximum value for C1, the threshold value th3 is changed with the minimum value for C2, and the threshold value th4 is changed with the maximum value for C2.

As described above, the flesh color extraction performance can be improved by sampling the flesh color of the user in the lighting environment in which it is used, and a simple color space can be provided by using a robust color space for changes in the brightness of the illumination. It is possible to further improve the skin color extraction performance even by adjusting the parameters.

Next, the operations of the skin color area extracting means 8 and the head area extracting means 9 will be described with reference to FIGS. Figure 6
3 is a flow chart for explaining the operations of the skin color area extraction means 8 and the head area extraction means 9.

Even if the skin color extraction parameters adjusted by the skin color extraction parameter adjusting means 7 are used, highlights may still occur on a part of the face depending on the lighting environment, and the head region may be accurately extracted only by skin color extraction due to wrinkles or shadows. It is difficult to extract. Therefore, the largest area of the skin color areas extracted by the skin color area extraction unit 8 is determined as the head area, and small areas other than the areas such as holes and crevices due to omission of extraction, the nose, and the mouth are head areas. By performing the head region repair process for removing from the region, the entire head can be appropriately extracted.

The skin color area extracting means 8 maps the color data of all the pixels of the captured image onto the skin color model space (step S400), and formulas (6) and (7) are used.
Pixels within the threshold values th1 to th4 defined in step S410 are extracted (step S410), and a labeling process for integrating the extracted pixels into four connections or eight connections (processing for grouping consecutive figures into numbers) is executed. By doing so, the area is divided into individual block areas (lumps) (step S4).
20). Then, as a result of the labeling process, a region having the largest area (number of pixels) is selected from the obtained block regions, and the selected region is set as the head region (step S430).

FIG. 7 shows an image including the head region thus selected. At this point, highlights and shadows,
Since dark parts such as eyes, mouth, and nose are not extracted,
As shown in FIG. 7, small areas 21 such as holes and crevices often occur in the head area.

Therefore, the head region extracting means 9 first repairs the crevice. The repair of the torn portion is achieved by performing expansion / contraction processing on the binary image in which the skin color pixels after the skin color region extraction are set to 1 and the rest are set to 0. In the expansion / contraction process, the expansion mask 22 and the contraction mask 23 as shown in FIG. 8 are set, and the expansion process and the contraction process described below are repeatedly performed to fill the above-mentioned cracks and small holes. The expansion processing expands the area by replacing the pixel value in the vicinity of the pixel of interest with the pixel value set by the expansion mask 22. The contraction processing leaves the target pixel when the pixel value of the non-zero pixel set by the contraction mask 23 is the same value as the pixel value of the contraction mask 23 among the neighboring pixels of the target pixel, and when the pixel value is not the same, By setting the value to 0, the area is shrunk. By the expansion and contraction process, the crevice 24 as shown in FIG. 9A is repaired, and it becomes as shown in FIG. 9B. Further, this processing can also fill a minute hole.

Since the rift generated in the head region has been repaired by the expansion / contraction process, the entire head is extracted as one region by filling the small regions corresponding to all the holes in the head region. Is possible. A logical operation process as shown in FIG. 10 is used for this filling process.

First, the exclusive OR of the head region image 26 obtained by the rip repair processing and the mask 27 having all 1 pixel values is obtained. As a result, holes in the background area and the head area are obtained. Next, the area (background area) in contact with the outer edge of the image is removed from the obtained image 28, and the logical sum of the removed image 29 and the original head area image 26 is obtained to obtain the head. It is possible to extract the whole as one area (30 is an image obtained by logical OR, step S44).
0).

As described above, since the head region can be extracted by the simple logical operation processing, the high speed processing becomes possible.

(B) Processing by the part area candidate extracting means 3 Next, the details of the processing of step S150 of FIG. 2 performed by the part area candidate extracting means 3 will be described with reference to FIGS. 11 and 12. FIG. 11 is a flow chart for explaining the operation of the part region candidate extraction means 3.

The region candidate extraction unit 3 is adaptive to the head region extracted by the head region detection unit 2 in order to robustly cope with a change in brightness according to a change in illumination condition. The histogram averaging method is used to keep the contrast of the head region constant. First, the head area brightness averaging means 10 obtains a circumscribed rectangle of the head area and divides the circumscribed rectangular area into, for example, 8 × 8 small areas (step S500). Next, the head region brightness averaging means 10 performs a histogram averaging process for each small region (step S510).

The histogram averaging process is performed as follows. First, a histogram showing the relationship between the pixel value and the frequency is obtained for each small area. Next, the cumulative frequency (the cumulative total of each frequency up to each class (pixel value)) is obtained, and each cumulative frequency is divided by the maximum cumulative frequency to obtain the ratio of each cumulative frequency. Then, the obtained ratio is multiplied by the maximum value of the pixel values in the small area and rounded off to the right of the decimal point. The value obtained here becomes the pixel value after averaging. Finally, the frequency of pixel values after averaging is obtained from the frequency before averaging.

For example, when the pixel values in the small area are in the range of 0 to 7 as shown in FIG. 13 and the frequency is as shown in FIG. 13, the frequency of each pixel value after averaging. Is as shown in FIG. For example, when the pixel value after averaging is 4, the pixel value before averaging corresponding to the pixel value 4 is 2
And 3, the frequency is 9 + 2 = 11.

Here, in the adaptive histogram averaging method as described above, most pixel values in the region are assigned to the maximum points of the histogram, especially in a small region where the contrast is low, so that a lot of noise may occur. There is. Therefore, if there is a pixel value 31 having a frequency exceeding a certain threshold as shown in FIG.
As shown in (b), a process of dispersing those frequencies to other pixel values is performed, which can suppress the generation of noise.

Since a constant contrast can always be obtained by the above processing, the pixel selection means 11 uses a constant threshold value tha and logically determines pixels (dark pixels) whose luminance value in the head region is equal to or lower than the threshold value tha. Level 1 is set, and the others are set to logical level 0 (step S520). Further, pixels having a pixel value of 1 are connected by 4 connection or 8 connection to divide into regions (step S530). Finally, by removing the minute area, the candidate area of each part (both eyes, mouth, and nose) can be extracted (step S540).

As described above, the whole head is extracted as one area, and the processing for making the contrast of the head area always constant is performed. Can be performed using. Therefore, it is possible to construct a system that enables high-speed processing and is robust against changes in brightness.

(C) Processing by the part detecting / tracking means 4 Next, the part detecting / tracking means 4 will be described with reference to FIGS. 15 and 16.
The operation of steps S160 and S170 of FIG. 2 performed in the calibration phase will be described. Figure 15
6 is a flowchart for explaining the operation of the part detection / tracking means 4 in the calibration phase.

First, the part detecting means 12 obtains the center of gravity of the head area extracted by the head area detecting means 2 (step S).
600). The position of the center of gravity is obtained by using a known distance conversion process or the like.

The distance conversion process is a conversion process for replacing each pixel value of the object in the image with the shortest distance from each pixel position to the background area. As the concept of distance, the simplest city distance (4 connection distance) and chessboard distance (8 connection distance) are often used. Here, an algorithm using the city distance will be described.

Step 1. First, assuming that each pixel data obtained by binarizing the input image is f _{i, j} and D _{i, j} is the initialized multi-valued data, initialization conversion is performed as follows. That is, the pixels in the head region having a pixel value of 1 are multivalued data ∞
(Actually, a large value such as 100) is replaced, and a background pixel having a pixel value of 0 is replaced with 0.

[Equation 1]

Step 2. The initialized image is scanned from the upper left to the lower right, and D' _{i, j} is sequentially updated according to the following rule. D ″ _{i, j} = min (D′ _{i, j} , D ″ _{i-1, j} + 1, D ″ _{i, j-1} + 1) ... Equation (9)

Step 3. With respect to D ″ _{i, j} obtained in the previous Step 2, scanning is performed from the lower right side to the upper left side, and D ″ _{i, j} is sequentially updated according to the following rule. D _{i, j} = min (D ′ _{i, j} , D ″ _{i + 1, j} +1, D ″ _{i, j + 1} +1) ... Equation (10)

D _{i, j} obtained by the above equation (10) becomes each pixel data of the range image. Therefore, a pixel having the maximum distance value is obtained from these obtained distance images, and this pixel is set as the center of gravity of the head region.

A feature of the range image conversion is that a stable center of gravity position can be obtained even if the shape of the area changes. The center of gravity may be obtained by averaging the coordinate values of pixels without using the distance image conversion.

The part detecting means 12 selects the candidate area closest to the center of gravity of the head area obtained in the previous step S600 from the candidate areas for both eyes, mouth and nose extracted by the part area candidate extracting means 3. Consider as nose area (step S61)
0).

Next, as shown in FIG. 16, the part detecting means 12 causes the left-eye mask 33 having a size proportional to the size of the head region in a certain direction and distance from the specified nose region,
The right eye mask 34 and the mouth mask 35 are set.

Among the set mask areas, the areas closest to the barycentric position are set as the right eye, the left eye, and the mouth area, respectively (step S620).

Next, the initial position setting means 13 stores the center position of each region and the positions of the outer end points 36a and 37a of both eyes (step S630). Finally, the right eye,
Of the detection regions related to the left eye and the nose, a region area mask image in which the pixel values in the right eye, left eye, and mouth region are set to 1 and other pixels are set to 0 is generated for each region, and these region region mask images are stored. . (Step S640). This part region mask image is used for part tracking processing for the first frame in the tracking phase. In addition, the part detecting unit 12 uses the image vertical direction (Y direction) at the center position of each part region (left eye, right eye, mouth).
Of the initial position setting means 13
Remember. This memorized part region (left eye, right eye,
The length of the mouth in the image vertical direction (Y direction) is used to obtain the open / closed state information of each part in the subsequent tracking phase.

(D) Processing by Head 3D Posture / Facial Expression Measuring Means 5 Next, with reference to FIGS. 17 to 19, the steps of FIG. 2 performed by the head 3D posture / facial expression measuring means 5 in the calibration phase. The operation of S180 will be described. FIG. 17 is a flowchart for explaining the operation of the three-dimensional head posture / facial expression measuring means 5 in the calibration phase.

As shown in FIG. 18, the affine basis setting means 15 obtains a straight line 38 connecting the outer end points 36a and 37a of both eyes obtained by the part detection / tracking means 4 (step S).
700). Next, the image is rotated so that the straight line 38 becomes horizontal based on the end point of either the left eye or the right eye (step S710). Then, a straight line 39 that passes through the center position of the mouth and is parallel to the obtained straight line and has the same length is obtained (step S720). The end points of these two straight lines 38, 39,
That is, the center coordinates 40 of the rectangle formed by the four points 36a, 37a, 36b, 37b are obtained (step S730). Further, the relative coordinates of the four vertices of the rectangle are obtained with reference to the center 40 of the rectangle 39, and these are stored as virtual points in the three-dimensional space (step S740).

The virtual point on the three-dimensional space serves as a reference point for measuring the three-dimensional head posture in the tracking phase.

Next, the head rotation amount estimating means 16 is operated as shown in FIG.
As shown in, the coordinate system is defined with a straight line connecting the end points 36a and 37a of the eyes as the X axis, and a straight line passing through the center of the mouth and perpendicular to the X axis as the Y axis, and the X axis of a circumscribed rectangle circumscribing the head region. When the length in the direction is 1, the distances La and Lb between the inner end point of the left eye or the right eye and the left and right sides of the circumscribed rectangle are obtained (step S750). Similarly, when the length of the circumscribed rectangle in the Y-axis direction is 1, the distances Lc and Ld from the center position of the mouth to the upper and lower sides of the circumscribed rectangle are obtained (step S76).
0).

This relative position serves as a reference for predicting the amount of rotation of the head in the vertical and horizontal directions in the tracking phase.

The above is the operation of the real-time facial expression tracking device in the calibration phase.

[Tracking Phase] Next, the operation in the tracking phase of the real-time facial expression tracking device in FIG. 1 will be described in detail with reference to FIGS.

(A) 'Processing by the head area detecting means 2 In the head area detecting means 2, the skin color area extracting means 8 and the head area extracting means 9 are operated to make a predetermined operation via the image inputting means 1. The same processing as in the calibration phase is performed on the images of the current frame that are sequentially input at the frame rate of 1 to extract the skin color area and the head area (see FIG. 3).
Steps S200 to S220). However, in this tracking phase, the skin color sampling by the skin color sampling unit 6 and the skin color parameter adjustment by the skin color extraction parameter adjusting unit 7 are not performed.

(B) 'Processing by the region candidate extraction unit 3 The region candidate extraction unit 3 executes the same process as in the calibration phase to obtain the region (eye, mouth, nose) from the image of the current frame. Region candidates are extracted (Fig. 3
Step S230). That is, the head region extracted by the head region detecting means 2 is subjected to a process of keeping the contrast of the head region constant by using the adaptive histogram averaging method, and further, a constant threshold value tha is used to Pixels (dark pixels) whose brightness value is less than or equal to the threshold value tha in the partial area are set to 1, other pixels are set to 0, and pixels having a pixel value of 1 are 4
Each area (both eyes, mouth and nose) is divided by connecting or connecting 8 areas to divide the area and finally removing the micro area.
The candidate area of is extracted.

(C) 'Processing by the part detecting / tracking means 4 The operation of the part detecting / tracking means 4 in the tracking phase will be described in detail with reference to FIGS. 20 and 21 are flowcharts for explaining the operation of the part detecting / tracking means 4 in the tracking phase.

The region tracking means 14 sets a rectangular region of a certain size centered on the center coordinates of the region region of the stored previous frame. A candidate area of the current frame existing in the rectangular area is obtained (step S82).
0). Next, the evaluation value E is obtained using the following discriminant (11) for each candidate area.

[Equation 2]

Here, E is the evaluation value, SP is the number of pixels in the region of the previous frame, SC is the number of pixels of the candidate region in the current frame, and OP is the mask image of the candidate region in the current frame (only the pixels in the candidate region are 1 and 0 otherwise
Image) and the mask image of the part region in the previous frame (image of which only 1 is the pixel of the part region and is 0 otherwise), the number of pixels having a pixel value of 1, D is the distance between the center of the part region and the center of the candidate region in the previous frame.

By selecting, as the target area, the one having the smallest value E calculated by the above equation (11), among the candidate areas of the current frame existing within a certain range based on the position of the partial area of the previous frame. The target area is specified from (step S830). That is, even if the small noise region 47 as shown in FIG. 22 is completely included in the partial region of the previous frame, in that case, | SP-SC | of the equation (11).
Since the value of and OP becomes large, such a noise area can be removed.

Such processing is executed for each of the left eye, right eye and mouth areas (steps S810 to S84).
0).

When all the parts can be detected by the above processing, the part area mask image is updated with that of the current frame, and the center position of the detection area for each part (left eye, right eye, mouth) is set. Obtain and store this (step S
850 and S860).

If there is a part that cannot be found (step S870), the position in the current frame of the part that cannot be detected is predicted from the movement vector of the part detected in the current frame. For example, as shown in FIG. 23, when there is a site (target site) 54 that could not be detected in the current frame, from the position of another site 48 detected in the current frame and the position 49 of the site in the previous frame. A movement vector 50 between frames is obtained. Then, the movement vector 50 obtained from the detection positions of other regions is added to the position 51 of the target region 54 in the previous frame to obtain the estimated position in the current frame (step S890). Then, paying attention to a pixel in a predetermined rectangular area (for example, 16 × 16) 53 including the obtained position, and executing the processing of step S820 and step S830 described above on the pixel in this rectangular area, 54 is detected (step S90)
0).

If there is no candidate area in the rectangular area 53, it is assumed that the face is hidden due to the inclination of the face, and the position estimated in step S890 is set as the position of the target portion in the current frame. It stores itself as its part area (steps S910 and S920).

If the part region of the current frame is not detected at all in step S870, the part detecting means 12 performs the processes of steps S600 to S640 of FIG. 15 again to detect the part region again (step S88).
0).

As described above, if one part can be detected, even if the other part is missed, the position of the missed part in the current frame is predicted from the movement vector of the detected part. Part tracking is possible. Furthermore, even if the target part does not appear in the image due to hiding, etc., since the temporary part region is set, it becomes possible to immediately trace the part when the hidden part appears, that is,
The smooth movement of each part of the head can be reproduced.

(D) 'Head 3D posture / facial expression measuring means 5
Next, the operation of the head three-dimensional posture / facial expression measuring means 5 in the tracking phase will be described in detail with reference to FIGS. 24 and 27 are flowcharts for explaining the operation of the three-dimensional head posture / facial expression measuring means 5 in the tracking phase.

First, in the head rotation amount estimating means 16, as shown in FIG. 25, from the both eye regions of the current frame obtained by the part detecting and tracking means 4, the end points 7 on the outer sides of both eyes are detected.
0,71 is calculated, and a straight line 55 connecting these end points 70,71
Is calculated (step S1000). Further, a straight line 56 that is orthogonal to the straight line 55 and passes through the center position 59 of the mouth is obtained (step S1010). The calculated straight line 55 is used as the X-axis, and the straight line 5
A local coordinate system having 6 as the Y axis is set, and a circumscribed rectangle 57 having sides parallel to the X axis 55 and the Y axis 56 and circumscribing the extracted head region is obtained (step S1).
020). The length of the side of the circumscribed rectangle 57 in the X-axis direction is set to 1, and the relative distances La ′ and Lb ′ between the end point 58 inside the eye measured in the calibration phase and the two sides 72 and 73 parallel to the Y-axis. Respectively (step S103
0). Similarly, the length of the circumscribed rectangle in the Y-axis direction is set to 1, and the relative distances Lc ′ and Ld ′ between the center 59 of the mouth and the two sides 74 and 75 parallel to the X-axis are obtained (step S104).
0).

Next, the intersections (two points) 76 and 77 of the outer end points 70 and 71 of the eyes and a straight line passing through the end points 70 and 71 and parallel to the Y axis and a straight line passing through the center of the mouth and parallel to the X axis. A rectangle 60 formed by and is obtained (step S1050).

Here, with respect to the X axis, the right direction is the positive direction,
Regarding the Y-axis, when the upward direction is the positive direction, the relative distance dec (= Lb ′) in the X-axis positive direction of one eye and the relative distance dei in the X-axis positive direction stored in the calibration phase.
From (= Lb), the rotation amount of the head in the left-right direction is calculated by the following equation (12). Rf _E = dec / dei Equation (12)

Here, Rf _E is the amount of rotation in the left-right direction, de
c is the relative distance in the positive direction of the X-axis of the current frame, dei
Is the relative distance in the positive direction of the X axis stored in the calibration phase.

If the rotation amount Rf _E is larger than 1, the head is rotating to the left. On the contrary, when the rotation amount Rf _E is smaller than 1, the head is rotating to the right.

Similarly, the relative distance dmc of the mouth in the positive direction of the Y-axis
From (= Ld ') and the relative distance dmi (= Ld) in the Y-axis positive direction stored in the calibration phase, the vertical rotation amount of the head is obtained by the following equation (13). Rf _m = dmc / dmi ...... formula (13)

Here, Rf _m is the amount of vertical rotation, dm
c is the relative distance of the mouth in the Y-axis positive direction in the current frame, dmi
Is the relative distance in the Y-axis positive direction of the mouth stored in the calibration phase.

If the rotation amount Rf _m is greater than 1, the head is rotating downward. On the contrary, when it is smaller than 1, it means that the head is rotating upward.

Next, the rectangle 60 is distorted as follows based on the left, right, up and down rotation amounts Rf _E and Rf _m obtained by the equations (12) and (13) (step S1060).

When Rf _E > 1: The length of the left side of the rectangle (the side parallel to the Y axis and in the negative direction of the X axis) is calculated by the following equation (1)
Use 4) to shorten. l = w · Rf _E · ol (Equation (14)) l is the calculated length, ol is the original length, and w is the weighting factor.

When Rf _E <1: The length of the right side of the rectangle (the side parallel to the Y axis and in the positive direction of the X axis) is calculated by the formula (1)
Use 4) to shorten.

When Rf _m > 1: The length of the lower side of the rectangle (the side parallel to the X axis and in the negative direction of the Y axis) is calculated by the following equation (1)
Use 5) to shorten. l = w · Rf _m · ol (Equation (15)) l is the calculated length, ol is the original length, and w is the weighting factor.

When Rf _m <1: The length of the upper side of the rectangle (the side parallel to the X axis and in the positive direction of the Y axis) is calculated by the formula (1
Use 5) to shorten.

For example, when the head is rotated to the left as shown in FIG. 26A, the left side of the rectangle 60 is shortened, and the rectangle 60 is rotated upward as shown in FIG. In this case, the rectangle 60 has a shorter upper side. Then, the vertex coordinates of the thus transformed rectangle are obtained with reference to the center coordinates of the rectangle 60 before the transformation.

The posture measuring means 17 then calculates the four vertex coordinates (two-dimensional coordinates) thus obtained and the virtual points in the three-dimensional space set by the affine base setting means 15 corresponding to them. Based on, the three-dimensional posture of the head is measured. Here, three-dimensional posture measurement is performed using the following method.

The relationship between the image taken by the camera and the object in the three-dimensional space is as shown in FIG. In FIG. 28, 63 is a plane in the three-dimensional space set by the affine base setting means 15, 64 is a camera image plane, and 65 is a plane.
Is the camera coordinate system.

The point (X _f , Y _f , Z _f ) in the coordinate system of the plane 63 in the three-dimensional space and the corresponding point (X _c , Y _c , Z _c ) in the camera coordinate system 65 are as follows: There is a relationship of 16).

[Equation 3]

In the equation (16), R represents a rotational component and T represents a translational component, which is equal to the three-dimensional posture information of the head.

On the other hand, the point (X _c , Y _c , Z _c ) in the three-dimensional space in the camera coordinate system 65 and the two-dimensional point (dX _c , dY _c ) in the camera image plane 64 are expressed by the following equation (17). ) Has the relationship shown in.

[Equation 4]

Here, the matrix including P is a perspective projection matrix of the video camera 80 to be used and can be obtained in advance by using a well-known camera calibration technique.

Now, the rectangle (camera image plane 64) obtained by the head rotation amount estimating means 16 has the upper and lower sides and the left and right sides in parallel in the three-dimensional space. From these two sets of parallel sides, the direction vector (X axis, Y axis) in the vertical direction and the horizontal direction in the rectangular three-dimensional space can be obtained.

When the equation of a straight line on the camera image plane 64 of parallel sides is a ₁ x + b ₁ y + c ₁ = 0 equation (18) a ₂ x + b ₂ y + c ₂ = 0 equation (19), the camera coordinates The equation of the three-dimensional plane including these straight lines in the system 65 can be expressed by the following equations (20) and (21).

A ₁ P ₁₁ X _c + (a ₁ P ₁₂ + b ₁ P ₂₂ ) Y _c + (a ₁ P ₁₃ + b ₁ P ₂₃ + c ₁ ) Z _c = 0 Equation (20) a ₂ P ₁₁ X _{c + (a 2 P 12 +} b 1 P 22) Y c + (a 2 P 13 + b 2 P 23 + c 2) Z c = 0 ...... formula (21)

The normal vectors (X,
When the cross product of the Y and Z coefficients is calculated, the direction vector (X
Axis, Y-axis).

As described above, the direction vector corresponding to the X axis and the Y axis of the rectangle in the camera coordinate system 65 can be obtained. However, due to the error of the information obtained from the image, the obtained direction vector is as shown in FIG. May not be orthogonal to. Therefore, when the obtained direction vectors are S1 and S2, the orthogonal vector V is obtained based on the vectors S1 and S2.
1, V2 is calculated. The vector in the Z-axis direction is the calculated V1
And the cross product of V2. These three direction vectors become the rotation component R in equation (16).

If the rotation component R is known, the translational component T can be obtained by substituting the corresponding points of the two-dimensional coordinates and the three-dimensional coordinates into the equations (16) and (17).

In the posture measuring means 17, first, the formula (18) is calculated from the coordinates of the four vertexes of the rectangle obtained by the head rotation amount estimating means 16.
The linear parameter (equation) of each side shown in (1) is obtained (step S1100), and the obtained linear parameter is used to obtain the equation (2).
0) and the equation (21), the X axis and the Y axis of the virtual three-dimensional plane set by the affine basis setting means 15 are obtained (step S1110). Then, as described above, the corrected axes are corrected so that they are orthogonal to each other, and the corrected X
The Z-axis is obtained from the axes, Y-axis, and these three axes (X-axis, Y-axis, Z-axis)
The rotation matrix (rotation component) R is obtained from the direction vector of the axis (step S1120), and the corresponding points of the two-dimensional coordinates and the three-dimensional coordinates obtained by using this rotation component R are given by the formula (1).
6) The translation sequence (translation component) by substituting into (17)
T is calculated (step S1130).

Using the projection matrix obtained as described above, the projection matrix is corrected according to the error when the virtual points in the three-dimensional space are actually projected on the camera image plane (step S1140). The projection matrix when it becomes equal to or less than the threshold is set as the three-dimensional posture information of the head (step S1150),
By outputting this three-dimensional posture information to the character control device 90, the three-dimensional posture of the head of the CG character is controlled.

In this way, a rectangle (virtual plane) in the three-dimensional space is defined from the three points of both eyes and mouth detected from the face image, and the rectangle created from the three points of both eyes and mouth at the time of tracking is used to move the head. The three-dimensional plane becomes 2 by distorting according to
3D posture information that cannot be obtained unless there are 4 or more corresponding 3D and 2D points is obtained by pseudo-reproducing the distortion when projected onto 3D. I'm trying to estimate it.

Next, the operation of the open / close state measuring means 18 will be described. The open / closed state measuring unit 18 reproduces and reproduces both the eyes and the mouth region in the camera image when the user faces the front by using the projection matrix obtained by the posture measuring unit 17, that is, the three-dimensional posture information of the head. The ratio between the length of the region in the image vertical direction (Y direction) and the length of each region in the initial state stored in the initial position setting means 13 in the image vertical direction is obtained. This ratio serves as open / closed state information indicating how much the eyes and mouth are opened and closed.

As described above, since the three-dimensional posture information is used to estimate the eyes and mouth area in the camera image when the user faces the front, for example, even in an image in which the head faces sideways or upwards, It is possible to estimate the image when facing the eye, and more accurately determine the open / closed state of both eyes and mouth from only the two-dimensional image.

The three-dimensional posture information of the head and the open / closed state information of both eyes and mouth thus obtained are input to the character control device 90. Character control device 9
0 is variably controlled by the input three-dimensional posture information of the head and the open / closed state of both eyes and mouth to variably control the movement of the head of the CG character and the open / closed state of both eyes and mouth, so that the video camera 80 captures the image. The movement and facial expression of the CG character are changed in real time by following the movement and facial expression of the user.

[0136]

As described above, according to the present invention, each pixel data of an image obtained by picking up a target person is converted into R, G,
After normalizing for each B component, the data of C1-C2 space is included.
C of the converted pixel data.
1 data and C2 data are predetermined skin color extraction parameters
If it falls within the range of, the pixel data is judged as a skin color pixel.
To extract the head region from the captured image by
Since it is set to
Can be accurately extracted. Therefore,
Head movements and facial expressions of CG characters according to movements
Can be controlled accurately. In addition, the skin color of the target person (user) is sampled under the lighting environment to be used, and the skin color extraction parameter for skin color extraction is adjusted using this sampling data. The head region of the user can be accurately extracted by adapting to the individual difference of each user.

[0137]

[0138]

[0139]

[0140]

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[Brief description of drawings]

FIG. 1 is a block diagram showing an embodiment of a real-time facial expression tracking device according to the present invention.

FIG. 2 is a flowchart for explaining an outline of an operation in a calibration phase of the real-time facial expression tracking device of FIG.

FIG. 3 is a flowchart for explaining an outline of an operation in a tracking phase of the real-time facial expression tracking device of FIG.

FIG. 4 is a diagram for explaining skin color sampling.

FIG. 5 is a flowchart for explaining the operation of the skin color sampling means and the skin color extraction parameter adjusting means.

FIG. 6 is a flowchart for explaining the operations of the skin color area extraction means and the head area extraction means 9.

FIG. 7 is a diagram showing an example of a result of extracting a skin color area by a skin color area extracting unit.

FIG. 8 is a diagram illustrating an expansion mask and a contraction mask.

FIG. 9 is a diagram for explaining a process of filling a crack generated in the detected head region.

FIG. 10 is a diagram for explaining a logical operation process for filling all holes in the head region.

FIG. 11 is a flow chart for explaining the operation of the part area candidate extraction means.

FIG. 12 is a diagram for explaining processing for suppressing noise generation, which is a drawback of the adaptive histogram averaging method.

FIG. 13 is a diagram for explaining an adaptive histogram averaging method.

FIG. 14 is a diagram for explaining an adaptive histogram averaging method.

FIG. 15 is a flowchart for explaining the operation of the part detection / tracking means in the calibration phase.

FIG. 16 is a diagram showing a mask region used when identifying both eyes and a mouth region in the region detecting means.

FIG. 17 is a flowchart for explaining the operation of the three-dimensional head posture / facial expression measuring means 5 in the calibration phase.

FIG. 18 is a diagram showing virtual points on a three-dimensional space set by affine base setting means.

FIG. 19 is a diagram for explaining the relative positions of the end points of both eyes and the center point of the mouth with respect to the circumscribed rectangle of the head region, which are obtained by the head movement amount estimation means.

FIG. 20 is a flowchart for explaining the operation of the part detecting / tracking means in the tracking phase (No. 1).

FIG. 21 is a flowchart for explaining the operation of the part detecting / tracking means in the tracking phase (No. 2).

FIG. 22 is a diagram for explaining a method of tracking a part region in the current frame by part tracking means.

FIG. 23 is a diagram for explaining a process of predicting a part region that could not be detected from the position of the part region that could be detected.

FIG. 24 is a flowchart for explaining the operation of the head three-dimensional posture / facial expression measuring means in the tracking phase.

FIG. 25 is a diagram for explaining a process of estimating the head rotation amount in the left-right and up-down directions by the head rotation amount estimation means.

FIG. 26 is a diagram for explaining a process of obtaining corresponding points corresponding to virtual points (affine bases) in a three-dimensional space in the head rotation amount estimation means.

FIG. 27 is a flowchart for explaining the operation of the three-dimensional head posture / facial expression measuring means in the tracking phase.

[Fig. 28] Fig. 28 is a diagram for explaining a process of obtaining three-dimensional posture information of the head from three-dimensional and two-dimensional corresponding points in the posture measuring means.

FIG. 29 is a diagram for explaining a process of correcting an error when obtaining orientation information.

FIG. 30 is a diagram showing a conventional technique.

[Explanation of symbols]

1 image input means, 2 head area detecting means, 3 part area candidate extracting means, 4 part detecting and tracking means, 5 3D posture / facial expression measuring means, 6 skin color sampling means, 7 skin color extraction parameter adjusting means, 8 skin color area extraction Means, 9
Head region extraction means, 10 Head region brightness averaging means, 1
1 pixel selection means, 12 part detection means, 13 initial position setting means, 14 part tracking means, 15 affine base setting means, 16 head rotation amount estimation means, 17 posture measuring means, 18 open / closed state measuring means, 20 sampling window, 22 expansion mask, 23 contraction mask, 33
Left eye mask, 34 Right eye mask, 35 mouth mask, 50
Movement vector, 53 rectangular area, 57 circumscribed rectangle, 64
Camera image plane, 80 video camera, 90 character controller.

─────────────────────────────────────────────────── ─── Continuation of the front page (56) Reference JP 2000-331190 (JP, A) JP 8-272948 (JP, A) JP 8-272973 (JP, A) JP 11-85988 (JP, A) JP 11-15947 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 1/00 340 G06T 7/00 100 G06T 7/20 300 H04N 7 / 14 JSTPlus file (JOIS)

Claims (10)

(57) [Claims] 1. A head region of a person from a video image of the person.
In the head region extraction device for extracting the R, G, and B pixel data of the image of the target person.
The formula c1 = arctan per minute (R / max (G, B )) c2 = arctan (G / max (R, B)) c3 = arctan normalized by normalizing according to (B / max (R, G )) Collect data c1, c2, c3
The normalizing means to be obtained and each pixel data including the normalized data c1, c2, c3 are
According to the formula C1 = c2 / c1 C2 = c3 / c2, the pixel data including the data in the C1-C2 space is added.
The data conversion means for converting each and the C1 data and C2 data of the converted pixel data are
Expression th1 <C1 <th2 th1, th2; Skin color extraction parameter th3 <C2 <th4 th3, th4; When the skin color extraction parameter is satisfied, this pixel data is determined as a skin color pixel.
By extracting the head region from the captured image
Head region extraction apparatus characterized by comprising: a part region extraction means. 2. The same illumination as when capturing the target person.
Under the environment, an image of a certain area of the target person's face is supported.
Skin color sampling means for sampling and a predetermined color sampled by the skin color sampling means
Each pixel data of the image of the area of
After normalization, C1-C2 is obtained using the data conversion means.
Converted to space pixel data, and the converted predetermined area
Maximum for C1 data using multiple pixel data of
Value and minimum value and maximum and maximum value for C2 data
Obtain a small value and use the maximum and minimum values to extract the skin color.
Output parameters th1, th2, th3 and th4 are supplemented
The head according to claim 1, further comprising a correct skin color extraction parameter adjusting unit.
Area extraction device. 3. The head area extracting means extracts skin color areas.
Extract the head area by extracting the maximum area from the result
The head region according to claim 1 or 2, characterized in that
Area extraction device. 4. The head area extraction means extracts the head area.
Characterized by adding expansion / contraction processing to the subsequent binary image
The head region extraction device according to claim 1.
Place 5. The head region extracting means obtains an exclusive logical sum of the binary image after the expansion / contraction process and a mask image in which all pixel values are logical value levels corresponding to skin color, and the exclusive OR is performed. The entire head region is extracted by obtaining the logical sum of the image having the logical value level corresponding to the non-skin color other than the head region of the image logically ORed with the binary image after the expansion / contraction process. The head area extraction device according to claim 4 , wherein 6. Sequentially input at a predetermined frame rate
Video input means for capturing video and head region for extracting head image from the captured image
Region extraction means, and from the extracted head region of each part including both eyes and mouth
Region candidate extracting means for extracting a candidate region and a region for detecting the position of each region from the extracted candidate regions
The detection / tracking means and the tertiary of the head based on the detected positions of the detected eyes and mouth.
Measure the original posture and check the open / closed state of both eyes and mouth.
A head 3D posture / facial expression measuring means for measuring;
Measured 3D posture of head and open / closed state of both eyes and mouth
Real-time control that controls the movement of CG characters based on movement
In the facial expression tracking device, the head region extracting means may generate R, G, and B pixel data of each pixel of an image of a target person.
The formula c1 = arctan per minute (R / max (G, B )) c2 = arctan (G / max (R, B)) c3 = arctan normalized by normalizing according to (B / max (R, G )) Collect data c1, c2, c3
The normalizing means to be obtained and each pixel data including the normalized data c1, c2, c3 are
According to the formula C1 = c2 / c1 C2 = c3 / c2, the pixel data including the data in the C1-C2 space is added.
When the data conversion means for performing the conversion and the converted pixel data satisfy the following equations th1 <C1 <th2 th1, th2; skin color extraction parameter th3 <C2 <th4 th3, th4; Judge
By extracting the skin color area from the captured image
A real-time facial expression tracking device comprising: a color area extracting unit . 7. The same illumination as when capturing the target person
Under the environment, an image of a certain area of the target person's face is supported.
Skin color sampling means for sampling and a predetermined color sampled by the skin color sampling means
Each pixel data of the image of the area of
After normalization, C1-C2 is obtained using the data conversion means.
Converted to space pixel data, and the converted predetermined area
Maximum for C1 data using multiple pixel data of
Value and minimum value and maximum and maximum value for C2 data
Obtain a small value and use the maximum and minimum values to extract the skin color.
Output parameters th1, th2, th3 and th4 are supplemented
7. The rear according to claim 6, further comprising : a correct skin color extraction parameter adjusting unit.
Letime facial expression tracking device. 8. The head region extraction means is the skin color region.
The maximum area is extracted from the skin color area extraction result by the extraction means.
Claim to extract the head region by
Item 7. The real-time facial expression tracking device according to item 6 or 7. 9. The head region extraction means is configured to perform the skin color region.
Expansion / contraction processing is performed on the binary image after the skin color area is extracted by the extraction means.
9. The method according to claim 6, further comprising:
Real-time facial expression tracking device. 10. The head region extraction means obtains an exclusive OR of the binary image after the expansion / contraction processing and a mask image in which all pixel values are logical value levels corresponding to skin color,
The entire head region is obtained by obtaining the logical sum of the image having the logical value level corresponding to the non-skin color other than the head region of the image obtained by the exclusive OR and the binary image after the expansion / contraction processing. The real-time facial expression tracking device according to claim 9 , wherein