JPH01314385A - Method and device for detecting face image - Google Patents

Abstract

PURPOSE:To detect eyes, eyebrows and a mouth area irrespective of the direction of a face which is being reflected by detecting a non-skin color area surrounded by a skin color area as an area candidate corresponding to a feature of the face, at the time of detecting a feature of a face image. CONSTITUTION:The title device is provided with a face structure collating means 12 for collating a face candidate group which as combined as area stored in an eye candidate group storage means 11 and an area stored in a mouth candidate group storage means 17, with a structure of a face stored in a face structure storage means 15. A candidate of a feature of a face is detected as an area which is not a skin color in a skin color area. That is, for instance, a mouth area being one example of a feature of a face is scarcely hidden by hair, etc., therefore, it is contained stably in a skin color area in a face image, and as for a lip, its color is varied frequently by rouge, etc., therefore, since it is difficult to specify a color of a lip, for instance, a candidate of a mouth is detected as an area being not a skin color which has been surrounded by a skin color area. In such a way, feature areas corresponding to eyebrows, a nose and a mouth can be set correctly.

Description

DETAILED DESCRIPTION OF THE INVENTION (Field of Industrial Application) The present invention relates to a face image detection technique that is necessary when automating the detection of a face image part from a TV camera image, which is important in introductory management and the like.

(Prior art) An example of a conventional method for detecting face images is given in the literature [Takashi Anezaki, Hiroyuki Sakemi, [Image processing for portrait robots (I) - Processing overview, feature extraction], 1985 IEICE General Report I will explain what is written in the National Convention, P5-83.1.

For example, in the conventional method for extracting facial features from an image input by the left camera of a TV, the first step is to calculate the characteristics for each group of black areas, which are divided into black areas of the image as shown in Figure 9. Regions that form a pair of eyes are identified based on conditional expressions regarding the area, circumference, center of gravity, etc., and the relative positional relationship of the region group. In the second step, feature regions containing the eyebrows, nose, and mouth were created based on the coordinates of the mouth, and in the third step, features were extracted from each of the regions.

(Problem to be Solved by the Invention) However, when using conventional methods to identify, for example, a face image that is upside down, it is difficult to detect the top and bottom of the face image using only the eye pair regions. Therefore, it was difficult to correctly set the characteristic regions corresponding to the eyebrows, nose, and mouth.

An object of the present invention is to provide a face image detection technique that does not depend on how the face image appears.

(Means for Solving the Problems) The face image detection method according to the first aspect of the present invention is characterized in that a candidate area corresponding to a facial feature is detected as an area other than a skin color area within a skin color area.

A second face image detection method of the present invention is characterized in that a set of facial feature candidates is selected based on the area ratio of a figure whose apex is the position of the candidate with respect to the face.

A first face image detection device of the present invention includes a face structure storage means for storing a facial structure, a face image storage means for storing a face image, and an area corresponding to an eye from an image stored in the face image storage means. eye candidate group finding means for detecting candidates for the eye candidate group, eye candidate group storage means for storing the region groups detected by the eye candidate group finding means, and candidates for the region corresponding to the mouth from the image stored by the face image storage means. a mouth candidate group discovery means for detecting a region group detected by the mouth candidate group discovery means; a mouth candidate group storage means for storing a region group detected by the mouth candidate group discovery means; The present invention is characterized by comprising a face structure matching means for matching a group of face candidates obtained by combining the above with a face structure stored in the face structure storage means.

A second facial image detecting device of the present invention is characterized in that the first facial image detecting device includes threshold determining means for calculating parameters that define a black area and a skin colored area of a facial image.

(Operation) The first aspect of the present invention detects facial feature candidates as non-skin-colored areas within the skin-colored area.

That is, firstly, the eye area, which is an example of a facial feature, is rarely hidden by hair or the like, so it is stably included in the skin-tone area in a face image, and secondly, the lips are hidden by lipstick, etc. Since the color often changes, it is difficult to specify the color of the lips. Therefore, in the present invention, for example, a mouth candidate is detected as a non-skin-colored area surrounded by a skin-colored area.

A second aspect of the present invention is to select a set of facial feature candidates based on the area ratio of a figure whose apex is the position of the candidate with respect to the face.

That is, for example, the area ratio of a triangle with respect to the face, whose vertices are the center of gravity of both eyes and mouth, is stable even if the line of sight to the face changes slightly, and since the face can be detected as a skin-colored area, for example, the area ratio of the face to the face is stable. It can be used as a scale to measure size. Therefore, for example, the area ratio to the face of a triangle whose vertices are the positions of the centers of gravity of the incorrect eyes and mouth does not fall within a preset range, and by examining the above area ratio, candidates for the incorrect eyes and mouth can be selected. I can do it.

The third aspect of the present invention provides, for example, eyes obtained by processing an input image.
By comparing the shape and positional relationship of the candidate region group for each of the eyebrows and mouth with a facial structure model representing the eyes, eyebrows, and mouth, it is possible to determine the shape and positional relationship of the candidate region group corresponding to the eyes, eyebrows, and mouth from among the above candidate region groups. The purpose of this is to select the area that will be used.

That is, in the present invention, among the region groups constituting the input image, the following parameter group L in Section 1 regarding the relative position of the facial structure illustrated in FIG. , distance L2 from the straight line passing through the center of gravity of both eyes: Distance between the center of gravity of the mouth and the perpendicular bisector of the line segment between the centers of gravity of both eyes B3: Perpendicular bisector of the line segment between the center of gravity of the stone chips and the center of gravity of both eyes Distance to the dividing line B4: Distance between the left center of gravity and the perpendicular bisector of the line between the centers of gravity of both eyes B5: A line that passes through the stone chips, the center of gravity, and the center of gravity of the right eye and is perpendicular to the line between the centers of gravity of both eyes. Distance from minute B6: Distance between the left center of gravity and a line passing through the center of gravity of the left eye and perpendicular to the line segment between the centers of gravity of both eyes PBR: Position of stone chips, PBL: Position of left side PER: Position of right eye, PEL: Left eye position PM: For example, the conditional expression below regarding the position of the mouth, and the following second parameter group θ1 that does not depend on the way the image is captured, for example regarding the shape of the facial structure illustrated in FIG. 8(B): Line between the centers of gravity of both eyes Angle 02 between the minute and the direction of the principal axis of the right eye area: Angle θ3 between the line segment between the centers of gravity of both eyes and the direction of the principal axis of the left eye area O4: Angle between the line segment between the centers of gravity of both eyes and the direction of the principal axis of the eye area O4: Between the centers of gravity of both eyes Angle θ5 between the line segment and the principal axis direction of the cloth layer region: For example, an area that satisfies the following conditional expression regarding the angle formed by the line segment between the centers of gravity of both eyes and the principal axis direction of the left eyebrow area is detected as an area corresponding to the eyes, eyebrows, and mouth. It is something to do.

The fourth invention corresponds to the skin color area of a face image by utilizing the fact that the brightness histogram of a face image captured under a bright background has three peaks corresponding to a black area, a skin color area, and a background. Detect areas.

That is, in the present invention, B1, B4, and B6 illustrated in FIG.
From the histogram 81 of the brightness of the face image in which maximum values corresponding to black color, skin color, and background appear, for example, the minimum value B2 of the histogram 81, which has a larger pixel value than B1 illustrated in the figure and is the smallest, is obtained. Then, find the minimum value B3 of the histogram 81 whose pixel value is smaller and largest than B4 illustrated in the same figure, and the minimum value B5 of the histogram 81 whose pixel value is larger and smallest than B4, for example. Pixel value B is B<B2
The pixel (2-1) is included in the black area,
For example, pixel value B is B3≦B<B5
It is determined that the pixel (2-2) is included in the skin color area.

(Example 1) Hereinafter, a first example of the present invention will be described in detail using the drawings.

FIG. 1 shows a first embodiment in which the third invention is used to output regions corresponding to the eyes, eyebrows, and mouth among the regions constituting the human-powered image illustrated in FIG. 2(A), for example. FIG.

In FIG. 1, 14 is a face image storage means for storing the face image 39 shown in FIG. 2(A) inputted by a TV camera, and 15 is shown in FIGS. 8(A) and 8(B). This is a face image storage means that stores models of facial structures expressed by equations (1-1) and (1-2), respectively.

The operation begins, for example, when the control means 18 activates the eye candidate group finding means 10. As a first step, the activated eye candidate group finding means 10 detects, when the face image 39 is expressed as fCx, y), a pixel group having a value of 1 using, for example, a threshold value TL determined in advance through an experiment. For example, the black area image g (x, y) shown in FIG. Patent application 1986-190
The image g(x, y) shown in FIG. ), the eye candidate group image 4.8 illustrated in FIG. After the composition, the eye candidate group image 45 is stored in the eye candidate group storage means 11, and the process ends.

When the eye candidate group finding means 10 completes the processing, the control means 18 activates the mouth candidate group finding means 16 using, for example, the first principle. The activated mouth candidate group discovery means 16
As a first step, when the face image 39 is expressed as f(x, y), for example, a threshold value TS determined in advance through an experiment is determined.
Using O, TSI (5), the pixel group having a value of 1 represents a skin color area, for example, the skin color area illustrated in FIG. y
) is synthesized by assigning the value 1 to the pixel group corresponding to the hole area included in the mouth candidate group pixel 49 illustrated in FIG. 4(B).
After synthesizing the mouth candidate group images 49, the mouth candidate group image 49 is stored in the mouth candidate group storage means 17, and the process ends.

When the mouth candidate group finding means 16 completes the processing, the control means 10 activates the chain structure matching means 12.

As a first step, the activated chain structure matching means 12 selects, for example, the region 50 and the region 51 illustrated in FIG. 3(C) from the region group stored in the eye candidate group storage means 11, and then For example, the center of gravity of
EH, for example, the center of gravity position of the area 51 is shown in FIG. 8(A).
As a second step, for example, the distance between the centers of gravity L(7) of the regions 50 and 51 is calculated to determine the value of the parameter L shown in FIG. 8(A), and as a third step, the For example, when the position of the center of gravity of the region group stored in the mouth candidate group storage means 16 is associated with the point PM shown in FIG. 8(A), the expression (1-1) stored in the facial structure storage means 15 is For example, the region 60 illustrated in FIG. 4B is selected, and as a fourth step, the regions 50, 51 . After calculating the area St of the triangle 80 illustrated in FIG.
, y), and the area including the holes in the first to fourth stages is Repeating the process up to step 5, for example, the first component is the right eye area number, the second component is the left eye area number, the third component is the cloth layer area number, the fourth component is the left eye area number, and the fourth component is the left eye area number. 6-dimensional vector M = (ml, m2.m3.m4.m5.m6) where the 5th component is the mouth area number and the 6th component is the score
(9), and the third and fourth components of the above vector.
After initializing the component and the sixth component, for example, the area numbers 50, 51.60 shown in FIG. m2. m5
(IQ), and as a sixth step, the area 50,
Publications (“Digital Pic
ture Processing”, 5econdE
dition, Volume2. ACADEMIC
PRESS, PP, 286-290° 1982,), for example, the ratio of each standard deviation in the major axis direction and the minor axis direction, ζ50. ζ51. ζ60
(11) is, for example, a constant al+a2ta3t determined appropriately.
When the condition using a4 is satisfied, for example, the center of gravity position is set to the 8th area group stored in the eye candidate group storage means 11.
For example, the area 53 illustrated in FIG. 4(B) that satisfies the equation (1-1) stored in the facial structure storage means 15 when correlated with the points PER and PBL shown in FIG. After substituting the region 54 into the fourth component of Equation (9) and the fifth component of Equation (9), the region 5 substituted as the component of Equation (9)
8th in each principal axis direction of 0, 51, 53, 54, 60
Parameter o1 shown in figure (B). e2. e3. θ4,
05, and the above parameters are stored in the facial structure storage means 1.
5 does not satisfy the memorized formula (1-2), the formula (9
), for example, 1 is subtracted from the sixth component of
After synthesizing a plurality of vectors shown in formula (9) by executing the formula for all combinations of regions stored by the expression (9), the facial structure storage means 15 stores the formula for the vector whose sixth component is the maximum. (1-1) and equation (1-2)
Processing similar to the sixth step is performed by narrowing the interval shown as the condition for In the eighth step, the vector of equation (9) that has been combined into one by the above processing is outputted to the display means 13, and then the display means 13 is activated and the processing is ended.

The activated display means 13 is the facial structure matching means 12.
The first component, second component, and third component of the vector output by
The fourth component and the fifth component are determined to be the right eye area, left eye area, cloth layer area, left side area, and eye area, respectively, and are stored in the eye candidate group storage means 11. For example, in the image 90 illustrated in FIG. 6(A), the number indicated by the component, the second component, the third component, and the fourth component, and the fifth of the vectors among the areas stored in the mouth candidate group storage means 17. For example, an image 91 illustrated in FIG. 6(B) with the number indicated by the component is displayed on, for example, a CRT, and all processing is thus completed.

In the above processing, the eye candidate group storage means 11, the face image storage means 14, the mouth candidate group storage means 17, and the face structure storage means 1
The display means 13 may be composed of, for example, a memory, a CRT, and current display technology, and the control means 18 may be composed of, for example, a memory and a microprocessor.

(Example 2) Hereinafter, a second example of the present invention will be described in detail using the drawings. FIG. 7 shows, for example, a second method using the fourth invention.
FIG. 12 is a block diagram showing a second example of outputting regions corresponding to eyes, eyebrows, and mouth among the regions forming the input image illustrated in FIG.

In FIG. 7, 14 is a face image storage means for storing, for example, the face image 40 illustrated in FIG. 2(A) out of the TV left camera output, and 15 is a face image storage means for storing the face image 40 illustrated in FIG. Expressions (1-1) and (1-2) of the facial structure model illustrated in
This is a facial structure storage means that stores facial structure expressions.

The operation begins, for example, when the control means 18 activates the threshold value determination means 19 using, for example, the fourth principle. As a first step, the activated threshold value determination means 19 selects, for example, the face image 4.
0 is processed to create a histogram 41 of pixel values illustrated in FIG.
In step 1, for example, a pixel value 43 that gives a peak closest to the black pixel value 42 given in advance is determined, and as a third step, in the histogram 41, for example, the pixel value 43 is determined.
The pixel value 44 indicating the minimum point of the histogram 41, which has a pixel value larger than 3 and the minimum, is found, and in the fourth step, the peak closest to the pixel value 45 of the skin color area given in advance in the histogram 41 is found. The pixel value 46 to be given is determined, and as a fifth step, the histogram 41 is
For example, a pixel value 40 indicating the minimum point of the histogram 41, which has a pixel value smaller than and maximum than the pixel value 46, is found, and as a sixth step, the histogram 41 is
For example, find a pixel value 47 indicating the minimum point of the histogram 41, which is larger and smallest than the pixel value 46, and as a seventh step, calculate the pixel values 44 and 40.
and 47 are variables TL, TSO<TSI (
13) is recorded in the threshold storage means 9, and the process ends.

When the threshold determining means 19 completes the processing, the controlling means 18 activates the eye candidate group finding means 10. The activated eye candidate group finding means 10 uses the TL shown in equation (13) stored in the threshold storage means 9 as the value of TL shown in equation (3), and performs the procedure described in the first embodiment. It operates in the same way and ends the process.

When the eye candidate group finding means 10 completes the processing, the control means 18 activates the mouth candidate group finding means 16. The activated mouth candidate group discovery means 16 converts T'so and TSI shown in equation (13) stored in the threshold value storage means 9 into equation (5).
Using the values of Tso and TSI shown in FIG.

After the mouth candidate group finding means 16 completes the processing, the operation proceeds in the same manner as the procedure described in the first embodiment, and all processing ends here.

In the above description, the threshold storage means 19, the eye candidate group storage means 11, the face image storage means 14, and the mouth candidate group discovery means 17
The facial structure storage means 15 can be constituted by, for example, a memory, the display means 13 can be constituted by, for example, a memory, a CRT, and current display technology, and the control means 18 can be constituted by, for example, a memory and a microprocessor.

(Effects of the Invention) The first invention described above has the effect of detecting the eyes, eyebrows, and mouth areas regardless of the direction of the photographed face.
In addition to the effects of the first invention, the invention can select regions corresponding to the eyes, eyebrows, and mouth more appropriately than the first invention for many input images, and as a result, improves processing accuracy. There is an effect that can be done.

[Brief explanation of the drawing]

FIG. 1 is a block diagram showing a first embodiment of the present invention;
Figures (A), (B), Figure 3 (A), (B), (C), Figure 4 (A), (B), Figure 5, Figure 6 (A), (B) are A diagram for explaining the operation of the first embodiment, FIG. 7 is a diagram for explaining the operation of the first embodiment.
FIGS. 8A and 8B are diagrams for explaining the present invention in detail, and FIGS. 9 and 10 are diagrams for explaining the prior art. In the figure, 10 is an eye candidate group discovery means, 11 is an eye candidate group storage means, 14 is a face image storage means, 12 is a face image storage means, 16 is a mouth candidate group discovery means, 17 is a mouth candidate group storage means, and 19 39 is the input image used in the example, and 41 is the histogram of the human image used in the example.

Claims (4)

[Claims] (1) A face image detection method that detects a non-skin color area surrounded by a skin color area as a region candidate corresponding to the facial features when detecting features of a face image. (2) A face image detection method in which, when detecting features of a face image, a set of candidates for facial features is selected based on the area ratio of a figure whose apex is the position of the candidate relative to the face. (3) Facial structure storage means for storing the structure of the face, face image storage means for storing the face image, and eye candidate group discovery for detecting candidates for regions corresponding to the eyes from the images stored by the face image storage means. means, eye candidate group storage means for storing region groups detected by the eye candidate group discovery means, and mouth candidate group discovery means for detecting region candidates corresponding to mouths from images stored by the face image storage means. , a mouth candidate group storage means for storing the region group detected by the mouth candidate group discovery means; a face candidate group that is a combination of the region stored by the eye candidate group storage means and the region stored by the mouth candidate group storage means; A face image detection device that detects regions corresponding to the eyes and mouth of the face image, comprising a face structure matching unit that matches the face structure stored in the face structure storage unit. (4) The facial image detection device according to claim 3, further comprising a threshold value determining means for calculating parameters that define a black area and a skin color area of the facial image.