JP2004021924A - Face feature extraction method, apparatus and information storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To extract feature objects, such as a mouse, a nose and eyes, from 3-dimensional face data. <P>SOLUTION: In this method, using a high-frequency component removal means 100, data D200, in which high-frequency components related to shapes are removed from 3-dimensional original face data D100 having 3-dimensional shape data and its color/luminosity information, is generated. Based on patterns D300 related to colors and shapes of face feature objects, such as a mouse, a nose and eyes, stored in a color and luminosity/shape pattern storage means 300, face feature objects D400 are extracted by a feature extracting means 200 from the data D200 without high-frequency components. In the color and luminosity/shape pattern storage means 300, shape patterns and color/luminosity patterns related to desired feature objects, and position relations D300 between other feature objects are pre-stored. To find a plurality of feature objects, not only pre-stored information, but position information D500 on other feature objects that are acquired using the feature extraction means 200 is also stored. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a method for extracting a characteristic part such as a mouth, a nose, and an eye from three-dimensional face data, an apparatus and an information storage medium.
[0002]
[Prior art]
In technology for extracting characteristic parts such as mouth, nose, eyes, etc. from face three-dimensional range data in which shape and color / luminance information is stored, it is generally performed manually by a person using a dedicated user interface or two-dimensionally. The mainstream is to apply image processing for extracting a characteristic part on the image from the image and to extract using only the color / luminance information.
[0003]
As other methods using shape information, Japanese Patent Application Laid-Open No. 02-311962 and the document "Curture-based face surface recognition using spherical correlation. Principal Directions for curved agonism, recruitment, reconciliation, and the like. T. Ikeda, M .; , Chiaki, H .; , Proceedings of Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998, pp. 372-377. However, these are not features of the face such as the mouth, nose, and eyes, but are features that are extracted as features of the entire face having characteristic irregularities.
[0004]
[Problems to be solved by the invention]
However, the above method has the following problems.
[0005]
The method using a dedicated user interface can be avoided by mistake because obvious manual extraction is performed manually, but the burden on the work is large, and detailed extraction depends on the completeness of the interface and the skill of the operator. The problem arises.
[0006]
In addition, since the method using the color luminance information is easily affected by the photographing state of the image such as a change in lighting and posture, and individual differences between photographed persons, it is difficult to expect stable extraction with this method alone.
[0007]
On the other hand, the shape information includes useful information independent of the color / luminance information. In particular, the color of the nose does not change much from the surrounding portion, but the shape of the nose, such as that represented by the curvature, is rich in change. However, since the shape information such as the curvature is obtained by the differential operation, there is a problem that the measurement device for acquiring the shape is easily affected by noise if the performance is low.
[0008]
In addition, it has not been known what kind of information of a three-dimensional feature can be used to extract a characteristic portion such as a mouth, a nose, and an eye stably.
[0009]
Therefore, according to the method of the present invention, a characteristic pattern relating to the shape and color of a characteristic portion such as a mouth, a nose, and an eye is categorized into a template, and the template is compared with input three-dimensional data to extract a characteristic portion. Frequency conversion is performed on data affected by noise due to the low performance of the measurement device for acquiring a shape, and the inversely converted data is used for processing, leaving only the low-frequency portion.
[0010]
By performing such pre-processing, the general features of the human face are mainly used while removing the differential geometric features in high-frequency components, which are mainly affected by individual differences and errors of the measuring device. There is an advantage that only differential geometric features at low frequency components can be used.
[0011]
In addition, for parts such as the mouth, where both the shape and color luminance are largely different from the surroundings, by using both color luminance information and shape information, erroneous extraction that cannot be prevented in principle with color luminance information alone, for example, lip due to illumination It is possible to prevent a case where the surroundings have a similar color and the surroundings are also extracted.
[0012]
It is an object of the first to eleventh aspects of the present invention to reduce the burden on an operator when extracting a part from three-dimensional face data.
[0013]
An object of the first, third, fourth, fifth, sixth, eighth, ninth, and eleventh aspects of the present invention is to reduce the influence of noise added to a shape when extracting a part from three-dimensional face data. It is in.
[0014]
It is an object of the first to eleventh aspects of the present invention to reduce the influence of illumination and posture fluctuations in extracting a part from three-dimensional face data.
[0015]
[Means for Solving the Problems]
A first facial feature extraction method of the present invention is a method for extracting a facial feature portion including a mouth, a nose, and an eye from three-dimensional face data in which shape information and color luminance information are stored,
A first procedure for removing high frequency components from the shape information of the face three-dimensional data;
A second procedure for storing a shape pattern, a color pattern, and face information of a search range of a face characteristic portion;
A third procedure of extracting a face feature from the low-frequency component face three-dimensional data output in the first procedure using the face information in the second procedure;
Is provided.
[0016]
A second facial feature extraction method according to the present invention is a method for extracting a facial feature portion including a mouth, a nose, and an eye from three-dimensional face data in which shape information and color luminance information are stored,
A first procedure for storing face information including a shape pattern, a color pattern, and a search range of a face characteristic portion;
A second procedure for extracting a face feature from the face three-dimensional data and the face information according to the first procedure;
Is provided.
[0017]
A third facial feature extraction method according to the present invention according to the first or second aspect, comprises extracting a mouth as an extracted feature portion.
[0018]
A fourth face feature extraction method according to the present invention according to the first or second invention, further comprising extracting a nose as an extracted feature portion.
[0019]
A fifth face feature extraction method according to the present invention according to the first or second invention, comprises extracting an eye as an extracted feature portion.
[0020]
A first facial feature extraction device of the present invention is a device for extracting a facial feature portion including a mouth, a nose, and an eye from three-dimensional face data in which shape information and color luminance information are stored,
A high-frequency component removing means for removing high-frequency components from the shape information of the face three-dimensional data; a color luminance / shape pattern accumulating means for storing a face pattern of a characteristic feature, a color pattern, and face information of a search range;
Feature extraction means for extracting a face feature from the low-frequency component face three-dimensional data output in the high-frequency component removal procedure using the face information,
Is provided.
[0021]
A second facial feature extraction device of the present invention is a device for extracting a facial feature portion including a mouth, a nose, and an eye from three-dimensional face data in which shape information and color luminance information are stored,
A color-brightness / shape-pattern storage means for storing a shape pattern, a color pattern, and face information of a search range of a face characteristic portion;
Feature extraction means for extracting a face feature from the face three-dimensional data and the face information;
Is provided.
[0022]
A third facial feature extraction device according to the present invention according to the first or second aspect, further comprises extracting a mouth as an extracted feature portion.
[0023]
According to a fourth aspect of the present invention, in the first or second aspect, the facial feature extracting device includes extracting a nose as an extracted characteristic portion.
[0024]
According to a fifth aspect of the present invention, in the first or second aspect, an eye is extracted as an extracted characteristic part.
[0025]
A first aspect feature extraction information storage medium according to the present invention is an information storage medium storing computer-readable software, wherein the face feature extraction method according to the first, second, third, fourth, or fifth aspect is provided. Is stored in the computer.
[0026]
BEST MODE FOR CARRYING OUT THE INVENTION
FIG. 1 shows an embodiment of a face feature extraction method according to the first invention.
[0027]
The present method includes three means for performing the following processing.
[0028]
The high-frequency component removing means 100 is a low-frequency component face three-dimensional data obtained by removing a high-frequency component related to a shape from original face three-dimensional data D100 which is original face image information having three-dimensional shape data and color / luminance information associated therewith. D200 is created and sent to the feature extracting means 200.
[0029]
The feature extracting means 200 extracts the face characteristic part D400 based on the color / shape pattern D300 which is the face information on the face characteristic part such as the mouth, nose, and eyes stored by the color luminance / shape pattern storing means 300.
[0030]
The color / luminance / shape pattern storage unit 300 stores in advance a color / shape pattern D300 representing a shape pattern relating to a desired facial feature portion, a color / luminance pattern, and a positional relationship between other feature portions. When finding a plurality of facial feature parts, not only the information previously stored but also the facial feature part position data D500 previously obtained by the feature extracting means 200 are stored.
[0031]
Next, the operation of each means will be described in detail.
[0032]
The high-frequency component removing means 100 functions as a high-frequency removing filter for removing only high-frequency components mixed as noise from the shape information portion of the original face three-dimensional data D100.
[0033]
Various methods are conceivable for the filter to be used, but a frequency conversion such as Fourier transform, DCT, or wavelet transform may be used as a widely used method, or only a main term is converted into an eigenspace such as principal component analysis. The method of leaving is good.
[0034]
The high-frequency component and the low-frequency component can be separated by applying any one of the filters to the original face three-dimensional data D100, so that only the low-frequency component is left, and the low-frequency component face three-dimensional data is inversely transformed. Converted to D200.
[0035]
There are several ways to apply the filter, and an appropriate method depends on how the face three-dimensional range data is stored.
[0036]
For example, when three-dimensional coordinate data (X, Y, Z) of a face is projected and stored in cylindrical coordinates (U, V) as in the Mercator projection as shown in FIG. For each block, a two-dimensional Fourier transform or DCT, which is often performed, may be used. Alternatively, if it is known that the face is parallel to the XY plane and faces to the front as shown in FIG. One-dimensional Fourier transform and DCT may be performed. Even after performing such a conversion, the shape feature indicating the characteristic portion of the face is a low-frequency change.If only an area below an appropriate frequency is left, only the high-frequency noise is removed, and the change meaning the unevenness of the face is not changed. Can be left.
[0037]
The feature extracting means 200 calculates the color of each of the mouth, nose and eyes stored in the color luminance / shape pattern storing means 300 for the low-frequency component face three-dimensional data D200 obtained by the high-frequency component removing means 100. A characteristic part is extracted based on a color / shape pattern D300 which is differential geometric information such as relative positional relationship, magnitude of coordinates, poles, curvature, and the like. In particular, the differential geometric information is less susceptible to noise because high-frequency components have been removed by the above processing. More specific processing will be described in the second embodiment for the mouth, the third invention for the nose, and the fourth embodiment for the eyes.
[0038]
The color / brightness / shape pattern storage means 300 stores in advance a color / shape pattern D300 which is differential geometric information such as color, relative positional relationship, magnitude of coordinates, poles, curvature, and the like for each of the mouth, nose, and eyes. Not only that, the position information of the feature found earlier is also input and stored. This helps to limit the search location. For example, when the mouth and nose have already been found, the eyes are on the left and right of the nose above the mouth, so that the search range is greatly reduced, and it is possible to measure the reduction of extraction errors and the reduction of extraction time.
[0039]
The second invention is obtained by removing the high-frequency component removing means 100 from the first invention. This configuration is used when the three-dimensional data D100 is obtained by a high-precision three-dimensional data acquisition device and the effect of noise can be ignored. It is valid.
[0040]
The third embodiment of the present invention specializes in mouth extraction in the first or second embodiment of the present invention. This can be extracted, for example, by the following method.
[0041]
When the face is parallel to the XY plane and faces forward and the mouth is closed as shown in FIG. 2, the vertex of the nose is the minimum point on the Z coordinate, and the nose is closed on the V axis passing through this point. When the high-frequency component is removed from the original face three-dimensional data D100 by the high-frequency component removing means 100 on a straight line, the result is as shown in FIG. 3 on the VZ plane. As can be seen from this figure, the first maximum point below the nose apex is below the nose, and the next maximum point is the portion where the upper and lower lips overlap when the mouth is closed. From this information, the position of the mouth in the V direction is obtained. Then, if you find the red part in the vicinity, you can find the upper and lower lips.
[0042]
The fourth embodiment of the present invention is directed to the first or second embodiment, which specializes in the extraction of a nose. Since the nose extraction hardly uses color luminance information, it uses only differential geometric shape information. Assuming that the same data as in the third embodiment is used, a portion where the curvature increases in both directions of the U coordinate from the vertex of the nose corresponds to the nose muscle (see FIG. 4). Further, by tracking the point of maximum curvature near the U coordinate above and below the V coordinate of this point, a candidate for a nose ridge can be found (see FIG. 5). The upper limit of the nose ridge is determined at the point where the Z coordinate of the maximum curvature point is minimized (see FIG. 6). Since this portion corresponds to the protruding portion just above the depression under the eye, it is appropriate as a real feeling. The lower limit of the nose muscle is searched for a point at which the curvature of the maximum curvature greatly changes, based on the V coordinate of the local maximum at the top of the nose found at the time of extracting the mouth.
The fifth embodiment of the present invention specializes in eye extraction in the first or second embodiment. If the positions of the nose and mouth of the eyes are known, it is possible to narrow down the area considerably, and it is possible to stably extract only the color in the narrowed down area.
[0043]
Embodiments of the sixth, seventh, eighth, ninth, and tenth aspects of the invention can be implemented by an apparatus including the face feature extraction method of the first, second, third, fourth, or fifth aspect.
[0044]
According to an embodiment of the eleventh invention, each procedure of the first, second, third, fourth or fifth invention is configured as a program executable on a computer, and the program is stored in an information storage medium readable by the computer. Then, it can be implemented by executing on the computer.
[0045]
【The invention's effect】
An advantage of the first to eleventh aspects of the present invention is to reduce a burden on an operator when extracting a characteristic portion in three-dimensional face data. The reason is that the user does not need to directly specify the position of the part on the three-dimensional coordinate data, and the desired characteristic part is automatically extracted by using the face color luminance / shape pattern.
[0046]
The effects of the first, third, fourth, fifth, sixth, eighth, ninth, and eleventh aspects of the present invention reduce the influence of noise added to a shape in extracting a characteristic portion in three-dimensional face data. Is to do. The reason is that high-frequency noise is reduced by high-frequency component removing means or a device or a program equipped with the same.
[0047]
An advantage of the first to eleventh aspects is to reduce the influence of fluctuations in lighting and posture in extracting a characteristic part in three-dimensional face data. The reason is that three-dimensional shape information is used.
[0048]
According to the effects of the first, fourth, sixth, ninth, and eleventh aspects of the present invention, it is possible to extract a nose for which use of a color feature is not expected. The reason is that three-dimensional shape information is used.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.
FIG. 2 is an explanatory diagram of an example of a coordinate system of face three-dimensional data and a coordinate system of a storage format.
FIG. 3 is an explanatory diagram of a VZ plane cross section passing through a nose vertex of face three-dimensional data from which high-frequency components have been removed.
FIG. 4 is an explanatory diagram of a UZ plane cross section passing through a nose vertex of face three-dimensional data from which high-frequency components have been removed.
FIG. 5 is an explanatory diagram of searching for a nose ridge candidate by tracing a point having a large curvature.
FIG. 6 is an explanatory diagram of a method of determining an upper limit of a nose ridge.
[Explanation of symbols]
100 high frequency component removing means 200 feature extracting means 300 color luminance / shape pattern storage means D100 original face three-dimensional data D200 low frequency component face three-dimensional data D300 color / shape pattern D400 face characteristic part D500 face characteristic part position data

Claims (11)

A method for extracting a facial feature portion including a mouth, a nose, and eyes from face three-dimensional data in which shape information and color luminance information are stored,
A first procedure for removing high frequency components from the shape information of the face three-dimensional data;
A second procedure for storing a shape pattern, a color pattern, and face information of a search range of a face characteristic portion;
A third procedure of extracting face features from the low-frequency component face three-dimensional data output in the first procedure using the face information in the second procedure;
A facial feature extraction method comprising: A method for extracting a facial feature portion including a mouth, a nose, and eyes from face three-dimensional data in which shape information and color luminance information are stored,
A first procedure for storing face information including a shape pattern, a color pattern, and a search range of a face characteristic portion;
A second procedure for extracting a face feature from the face three-dimensional data and the face information according to the first procedure;
A facial feature extraction method comprising: 3. The face feature extracting method according to claim 1, wherein a mouth is extracted as an extracted feature portion. 3. The method according to claim 1, wherein a nose is extracted as an extracted characteristic part. 3. The face feature extracting method according to claim 1, wherein eyes are extracted as extracted feature portions. An apparatus for extracting a facial feature portion including a mouth, a nose, and eyes from face three-dimensional data in which shape information and color luminance information are stored,
A high-frequency component removing means for removing high-frequency components from the shape information of the face three-dimensional data, a color luminance / shape pattern accumulating means for storing a face pattern, a color pattern, and face information of a search range;
Feature extraction means for extracting a face feature from the low-frequency component face three-dimensional data output in the high-frequency component removal procedure using the face information,
A facial feature extraction device comprising: An apparatus for extracting a facial feature portion including a mouth, a nose, and eyes from face three-dimensional data in which shape information and color luminance information are stored,
A color-brightness / shape-pattern storage means for storing a shape pattern, a color pattern, and face information of a search range of a face characteristic portion;
Feature extraction means for extracting a face feature from the face three-dimensional data and the face information;
A facial feature extraction device comprising: 8. The facial feature extracting device according to claim 6, wherein a mouth is extracted as an extracted feature portion. 8. The facial feature extracting device according to claim 6, wherein a nose is extracted as an extracted feature portion. 8. The facial feature extraction device according to claim 6, wherein an eye is extracted as an extracted feature portion. An information storage medium storing software that is readable by a computer, wherein a program for causing the computer to execute the face feature extraction method according to claim 1, 2, 3, 4, or 5 is stored. Information storage medium.

Images