JP2000235640A - Facial organ detecting device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a high performance facial organ detecting device which is hard to be affected by the changes in skin color and illumination conditions. SOLUTION: A center line calculating means 2 calculates the center line of a face from an input image stored in an image storing means 1. A facial color estimating means 3 estimates facial skin color from the input image stored in the means 1 on the basis of the facial center line calculated by the means 2. A template producing means 5 applies facial color estimated by the means 3 to template generation rules of facial organs stored in a dictionary 4 for template production and produces a template for facial organs. A matching means 7 compares the template produced by the means 5 with an area in the image of an inputted facial image and performs position detection of the facial organs by finding an area where similarity to the template is the highest in the inputted image.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a face organ detecting apparatus for recognizing a face image and detecting a face organ such as eyes or lips.

[0002]

2. Description of the Related Art Various techniques have been developed which use positional information of facial organs such as eyes, nose, mouth, etc. obtained from a face image. For example, there are a personal identification technique and a technique for determining the direction of a face.

[0003] Techniques for such face images include:
For example, literature: “Face image analysis based on distributed two-dimensional appearance model” IEICE Transactions D-II Vol. J77
-D-II No. 12 PP. 2342-2352, 1994.12.

In the above document (p2344, right, ninth line from the bottom), as a method for detecting a face organ, an image (template) representing each organ is created in advance, and the input image matches the template most. It uses a method to find the area. The face organ template is created by photographing several face images in advance, cutting out the face organ region, normalizing the position and luminance, and averaging each pixel.

[0005]

However, in the prior art described above, if the illumination (color, lightness, darkness, position) conditions are different from those at the time of creating the template, the color and brightness of the human skin change. Therefore, position detection by template matching may fail. Similarly, the skin color varies depending on the individual, and also changes due to makeup, sunburn, race, and the like.

Therefore, conventionally, it is necessary to prepare a plurality of templates having different skin colors in order to cope with a change in skin color, and to improve matching performance,
There is a problem that the storage capacity of the template increases and the processing amount also increases.

[0007] From such a point, it has been desired to realize a high-performance face organ detecting apparatus which is hardly affected by changes in skin color and lighting conditions.

[0008]

The present invention employs the following structure to solve the above-mentioned problems. <Structure 1> A center line calculating unit that calculates the center line of the face from the input face image, and a skin color of the face is calculated from the input face image based on the center line of the face calculated by the center line calculating unit. A face color estimating means for estimating, a template creation dictionary for storing rules for generating facial organ templates, and a face color estimated by the face color estimating means applied to the generation rules stored in the template creation dictionary. A template creating unit that creates a template of an organ, a template created by the template creating unit, and a region in the image of the input face image are compared, and a region having the highest similarity with the template is extracted from the input image. A matching means for detecting the position of the face organ by finding the face organ.

<Structure 2> In the face organ detecting apparatus according to Structure 1, the center line calculating means calculates the center line based on the left-right symmetry of the face.

<Structure 3> In the face organ detecting apparatus according to structure 1 or 2, the matching means limits the search range based on the center line information of the face calculated by the center line calculating means. Organ detection device.

[0011]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention estimates the skin color of a face from an input image and creates a template based on the estimated skin color. Then, template matching is performed using the template created in this way. Therefore, it is possible to prevent performance degradation due to a change in skin color. Hereinafter, embodiments of the present invention will be described in detail using specific examples.

<Specific Example><Configuration> FIG. 1 is a configuration diagram showing a specific example of a face organ detecting apparatus according to the present invention. The apparatus shown in the figure comprises an image storage unit 1, a center line calculation unit 2, a face color estimation unit 3, a template creation dictionary 4, a template creation unit 5, a template storage unit 6, and a matching unit 7.

The image storage means 1 is a means for storing a face image. The input of a face image is configured to A / D convert a signal obtained from an image input device such as a video or VTR, or to obtain image data via a network or the like. This image storage means 1 is composed of an auxiliary storage device such as a semiconductor memory or a magnetic disk.

The center line calculating means 2 is a means for calculating the center line of the face based on the image data stored in the image storing means 1. The face color estimating means 3 has a function of estimating the color of the face in the input image based on the pixel values on the center line of the face calculated by the center line calculating means 2.

The template creation dictionary 4 is a template storage unit that stores rules for generating facial organ templates, and is composed of an auxiliary storage device such as a semiconductor memory or a magnetic disk. The template creating means 5 has a function of creating a face organ template by applying the face color estimated by the face color estimating means 3 to the generation rules stored in the template creating dictionary 4. The template storage means 6 stores the template creation means 5
Is a means for storing the generated template, and comprises an auxiliary storage device such as a semiconductor memory or a magnetic disk.

The matching unit 7 compares the template created by the template creating unit 5 with an area in the input image stored in the image storage unit 1 and determines an area having the highest similarity with the template in the input image. It has the function of detecting the position of the face organ by finding it from inside.

The above-mentioned center line calculating means 2, face color estimating means 3, template creating means 5, and matching means 7
Is realized by a program corresponding to each function and a microprocessor that executes the program, or is configured by dedicated arithmetic hardware.

<Operation> The operation of each functional unit will be described below. [Operation of Image Storage Unit 1] First, the width and height of the input image stored in the image storage unit 1 are W and H (however, the upper left of the image is (0, 0) and the lower right is (W, H)). The red, green, and blue components of the pixel located at the coordinates (X, Y) are represented by variable names R (X, Y), G (X, Y), and B (X, Y). The R, G, B stored in the image storage unit 1
Is used by the center line calculating means 2, the face color estimating means 3, and the matching means 7.

[Operation of Center Line Calculating Unit 2] The center line calculating unit 2 performs a process of calculating the center line of the face based on the data stored in the image storage unit 1. In this processing, the X coordinate having the highest left-right symmetry in the input image is obtained.

FIG. 2 is an explanatory diagram of the center line calculating process.
The center line calculating means 2 sets a straight line of X = a with respect to the input image as shown by a thick line in FIG. 2, and considers a rectangle having a width (CH × 2) around the straight line. Within this rectangle, the RG of the pixel values of the pixels line-symmetric with respect to X = a
Find the distance in B space, and sum the distance (dist) to X
= A is a measure of symmetry at a. Then, such processing is performed by changing the value of X from CH to (W-CH), and
X in which st indicates the minimum value is obtained (in the illustrated example, X = KX is the result). The value of CH indicating the width of the region is (W /
A value smaller than 2) is set, and this value is appropriately determined according to the size and position of the face shown in the input image.

Next, the operation of the center line calculating means 2 will be further described with reference to a flowchart. FIG. 3 is an operation flowchart of the center line calculating means 2. First, as shown in FIG. 2, the width of the input image is W and the height is H. Therefore, X
The range of the coordinate and the Y coordinate is 0 ≦ X ≦ W and 0 ≦ Y, respectively.
.Ltoreq.H.

First, in step S101, the X coordinate is initialized (X = CH). Next, step S102
In, the Y coordinate is initialized (Y = 0), and the variable dist that stores the calculation result of the left-right symmetry is also initialized (dist = 0).

Next, in step S103, a variable i representing the distance (in the X direction) from the center line is initialized (i =
1). In step S104, coordinates that are line-symmetric with respect to the center line (where i is the distance from the center line) are obtained. Here, the obtained coordinate values are assumed to be X0 and X1, respectively. In step S105, how similar the pixel located at the coordinates (X0, Y) and the pixel located at the coordinates (X1, Y) are obtained by distance calculation. That is, in this step S105, red at each coordinate,
The square of the difference between the values of the green and blue components is added to take the square root. The distance obtained here is added to dist.

In step S106, the value of i is updated (i = i + 1). In the next step S107, i is CH
Is determined. If i does not exceed CH, the process returns to step S104. Step S1
At 07, when i exceeds CH, the process proceeds to step S108.

In step S108, the value of Y is updated (Y = Y + 1). Then, in step S109, Y is H
Is determined. If Y does not exceed H in step S109, the process returns to step S103,
If Y exceeds H, the process proceeds to step S110.
Thus, the above-described steps S102 to S10
By the series of processing of No. 9, the left-right symmetry dist is obtained. That is, the smaller dist is, the higher the left-right symmetry is.

Next, in step S110, the obtained dist is determined.
Is the smallest value among dist obtained before that. If it is the minimum value, step S111
Store the current X (KX = X). Meanwhile,
If dist is not the minimum value, the process moves to step S112. In step S112, X is updated (X
= X + 1).

Then, in step S113, it is determined whether X has exceeded (W-CH). If not, the process returns to step S102. Here, X is (W
If (−CH) is exceeded, the process proceeds to step S114. In step S114, the value of KX is output as a result of this processing. That is, X = KX is the center line of the face.

As described above, the center line calculating means 2 calculates the center line based on the symmetry of the face in the left and right directions. For example, when the eyes are closed due to blinking or when wearing sunglasses. Even in this case, the center line of the face can be accurately determined.

[Operation of Facial Color Estimating Means 3]
Estimates the color of the face in the input image based on the pixel values on the center line of the face calculated by the center line calculating means 2. In this specific example, the face color is determined by averaging the pixel values in a predetermined range on the center line.

FIG. 4 is an explanatory diagram of the processing of the face color estimating means 3. As shown in FIG. 4, in this specific example, Y = startY on the center line X = KX of the face obtained by the center line calculating means 2.
Pixels in the range of ~ endY are used. However, startY, end
The value of Y is determined as appropriate according to the size and position of the face in the input image. Hereinafter, the operation of the face color estimating means 3 will be further described with reference to a flowchart.

FIG. 5 is a processing flowchart of the face color estimating means 3. First, a variable total that stores the sum of pixel values
Initialize R, totalG, totalB, and N (step S2)
01). Note that N is the number of pixels. And Y is startY
(Step S202), and add the values of the pixel values R (KX, Y), G (KX, Y), and B (KX, Y) to the variables totalR, totalG, and totalB for storing the sum of the pixel values, respectively. Then, the pixel value N is incremented by 1 (step S203).

Then, the value of Y is incremented by 1 (step S2).
04), the processes of steps S203 and S204 are repeated until the condition of Y> endY is satisfied in step S205. That is, in steps S202 to S205, the sum of pixel values in a predetermined range on the center line is obtained. The total value is totalR, total for red, green, and blue respectively.
G and totalB are stored. The number of pixels is stored in N.

In step S205, if the condition of Y> endY is satisfied, the total value of the pixel values totalR, tot
alG and totalB are each divided by the number N of pixels to obtain the estimated face color (step S206). The estimated face colors are stored in faceR, faceG, and faceB, respectively, for red, green, and blue.

[Explanation of Template Creation Dictionary 4] FIG.
FIG. 4 is an explanatory diagram illustrating an example of the template creation dictionary 4. In the figure, TDIC (x, y) shown in (a) indicates the attribute of each pixel in the template. In this specific example, since it is an eye template, 0 = skin, 1 = pupil, 2
= Defined as white eye. Further, a color definition table for defining what color each number has is configured as shown in FIG. In this specific example, the number 0 is the color estimated by the face color estimating means 3, and the number 1 is a preset color (dicR
[1], dicG [1], dicB [1]), and number 2
Are preset (dicR [2], dicG [2], dicB
[2]) is defined. In this specific example, one color is assigned to each number, but a plurality of colors may be assigned according to the environment to be used.

[Operation of Template Creation Unit 5] The template creation unit 5 creates a template for template matching using the template creation dictionary 4 and the face color information estimated by the face color estimation unit 3. The created templates are TR (x, y), TG (x, y), T
B (x, y).

FIG. 7 is an operation flowchart of the template creating means 5. First, the values of Y and X are initialized (steps S301 and S302), and the attribute dictionary TDIC is set.
The value of (x, y) is checked (step S303).
In step S303, the attribute dictionary TDIC (x,
When y) is 0, the pixel value of the coordinates (x, y) in the template includes the face color (faceR, fa) estimated by the face color estimating means 3.
ceG, faceB) (step S305). On the other hand, in step S303, the attribute dictionary TDIC
If the value of (x, y) is not 0, the colors (dicR [no], dicG [no], dicB [n
o]) and assigning it as a pixel value (step S3)
04). These processes are performed for all the pixels in the template having the width TW and the height TH, and the process ends.

That is, the value of X is incremented by 1 in step S306, and it is checked in step S307 whether the value of X is equal to or larger than the width TW of the template. If X <TW, the process returns to step S303. On the other hand, if X ≧ TW,
The value of X is incremented by 1 (step S308). Then, it is checked in step S309 whether the value of Y is equal to or greater than the height TH of the template, and if Y <TH, step S30
Returning to step 2, if Y ≧ TH, the template creation processing ends.

[Processing of Template Storage Unit 6] The template storage unit 6 stores the red component TR (x, y) and the green component TG of the template generated by the template creation unit 5.
(X, y) and a blue component TB (x, y) are stored.

[Processing of Matching Means 7] The matching means 7 is a processing for finding a region in the input image that matches the template stored in the template storage means 6 most. However, since the distance between the camera and the face fluctuates, matching is performed by enlarging or reducing the template within a predetermined range. Further, in order to reduce the processing amount, the search range is limited based on the center line of the face.

FIG. 8 is an explanatory diagram of the search range of the matching means 7. As shown, in this specific example, the X coordinate of the search range is limited to (KX + SX) to (KX + EX). Here, KX is the X coordinate of the center line of the face calculated by the center line calculation means 2, and SX and EX are parameters for determining the search range. The search range in the Y direction is represented by SY to EY. These values of SX, EX, SY, EY are
It is appropriately determined according to the use environment such as the size of the input image and the type of the face organ to be searched.

Next, further explanation will be given using a flowchart. FIG. 9 and FIG. 10 are processing flowcharts (No. 1 and No. 2) of the matching means 7.

Steps S401 to S401 shown in FIG.
409 is a coordinate (x, y) to be searched and a scaling ratio Z
Is changed within a predetermined range. In this specific example, X = (KX + SX) to (KX + EX) (step width step
X), Y = SY to EY (step width stepY), Z = startZ
The processing is performed as 〜endZ (step width stepZ). Steps S501 to S511 shown in FIG.
Here is the process of actually performing template matching.

First, the value of Y, the value of X, and the scaling factor Z
As initial values (steps S401, S402, S40
3). Then, the sum dist of the distances in the RGB space between the pixel values at the time of matching is initialized (step S501), and the coordinates N and M in the template are initialized (step S5).
02, S503) to convert the coordinates (M, N) in the template into the coordinates (X1, Y1) in the input image (step S504).

Next, the RGB values TR (M, N), TG (M, N), TB of the pixel at the coordinates (M, N) of the template
(M, N) and RGB values R (X1, Y1), G (X1, Y1), B of a pixel at coordinates (X1, Y1) in the input image
The distance to (X1, Y1) in the RGB space is obtained, and the distance is added to dist (step S505).

These processes are obtained for all the pixels of the template and are added to dist. When the processing is completed for all pixels, it is determined whether dist is the minimum value so far. That is, in step S506, the coordinate M in the template is incremented by 1, and it is checked in step S507 whether the value of M is equal to or larger than the template width TW. If M <TW, the process returns to step S504. On the other hand, M ≧ TW
, The value of N is incremented by 1 (step S50).
8). In step S509, it is checked whether the value of N is equal to or greater than the height TH of the template. If N <TH, the process returns to step S503. If Y ≧ TH, dist is the minimum value so far. (Step S
510).

Here, if dist is not the minimum value, no processing is performed. On the other hand, when dist is the minimum value, X, Y and Z at that time are kept X, keepY and keep
Substitute Z. KeepX, keepY, keepZ at the time when these processes are performed on the set search range and the processes are completed.
Is the search result.

Steps S501 to S
The process of 511 is performed by enlarging the value of the enlargement / reduction ratio Z by the step width stepZ.
It is changed until it reaches the value of dZ (steps S404, S4
05) Also, the value of X is calculated by the step size stepX (KX + EX).
(Steps S406 and S407),
Further, the value of Y is changed in steps of step Y until EY is reached (steps S408 and S409). Then, keepX, keepY, and keepZ at the time when all processes are completed are search results.

In other words, when the upper left corner of the template is adjusted to the coordinates (keepX, keepY) of the input image and the template is further scaled up and down by keepZ, the template is most matched.

<Effects> As described above, according to this example,
Since the face color is estimated using the input image and the template is created using the estimated face color, a high-performance model that is not easily affected by changes in lighting or skin color such as race. A facial organ detection device can be obtained.

Further, since the matching means can limit the search range based on the center line of the face, a reduction in the processing amount can be expected.

In the above-described specific example, the case of the eye is described as an example of the template. However, the present invention is not limited to this, and the present invention can be similarly applied to a facial organ such as a mouth and a nose. Further, the face color estimating means 3 estimates the face color by searching for a straight line on the center line. However, the face color may be estimated by giving a width in the X direction and assuming an area instead of a straight line. .

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing a specific example of a face organ detection device of the present invention.

FIG. 2 is an explanatory diagram of a center line calculation process in the face organ detection device of the present invention.

FIG. 3 is an operation flowchart of a center line calculation unit in the face organ detection device of the present invention.

FIG. 4 is an explanatory diagram of processing of a face color estimating means in the facial organ detecting device of the present invention.

FIG. 5 is an operation flowchart of a face color estimating means in the face organ detecting device of the present invention.

FIG. 6 is an explanatory diagram showing an example of a template creation dictionary in the facial organ detection device of the present invention.

FIG. 7 is an operation flowchart of a template creating unit in the facial organ detection device of the present invention.

FIG. 8 is an explanatory diagram of a search range of a matching unit in the face organ detection device of the present invention.

FIG. 9 is an operation flowchart (part 1) of the matching means in the facial organ detection device of the present invention.

FIG. 10 is an operation flowchart (part 2) of the matching means in the facial organ detection device of the present invention.

[Explanation of symbols]

 2 center line calculation means 3 face color estimation means 4 template creation dictionary 5 template creation means 7 matching means

Claims (3)

[Claims] 1. A center line calculating unit that calculates a center line of a face from an input face image, and a center line of the face calculated by the center line calculating unit.
A face color estimating unit for estimating a skin color of a face from an input face image; a template creating dictionary for storing a face organ template creating rule; and a generating rule stored in the template creating dictionary. A template creating unit that creates a facial organ template by applying the face color estimated by the complexion estimating unit; and a template created by the template creating unit.
A matching unit that detects a position of a face organ by comparing a region in the image of the input face image with a region having the highest similarity to the template from the input image. A facial organ detection device characterized by the above-mentioned. 2. The face organ detecting apparatus according to claim 1, wherein the center line calculating means calculates the center line based on left-right symmetry of the face. 3. The face organ detecting apparatus according to claim 1, wherein the matching unit limits the search range based on the center line information of the face calculated by the center line calculating unit. Detection device.