JP3355068B2 - Face image processing device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a face image processing apparatus having a function of detecting and judging the inclination of a face image using image processing and correcting the face image to a position having no inclination.

[0002]

2. Description of the Related Art A conventional driver's face image processing apparatus is disclosed, for example, in Japanese Patent Application Laid-Open No. 6-227278. FIG. 29 is a block diagram showing the configuration of a conventional device. The image data taken into the apparatus by the image input unit 55 is binarized by the binarizing unit 50, and the eyeball existing region in the face image is set by the eyeball existing region setting unit 57. The driver state determination means 59 is configured to determine the driver's state based on the pattern of the open / closed eyes in the eyeball presence area detected by the open / closed eye detection means 58. The detection result of the open / closed eye detection means 58 is also sent to the erroneous detection determination means 61, and if it is determined to be erroneous detection, the resetting means 60 resets the eyeball existence area. The setting of the eyeball presence area is performed in the order of the horizontal width and the vertical width.

FIG. 30 is an explanatory diagram of image scanning in the horizontal width setting, and FIG. 31 is an explanatory diagram of image scanning in the vertical width setting. The width is determined based on the number of continuous white pixels in the left and right direction in the left and right regions bordering the scanning start line (the center of the X coordinate of the image) 62 in FIG.
64 and set the vertical width. A broken line portion 65 in FIG. 34 is an eyeball existing area in which the horizontal width and the vertical width are set by the image scanning methods in FIGS. 30 and 31. As described in the flowchart of FIG. 30, the width setting method counts the number of continuous white pixels in the lateral direction of the face upward from substantially the center of the face.

Next, the X coordinate (the horizontal coordinate of the face) at the maximum count is stored. As a result of counting the white pixels in the horizontal direction of the face in the vertical direction, if the sum of the continuous numbers of the left and right white pixels is, for example, 200 or more, the horizontal direction of the eyeball existence region is determined from the stored X coordinate at the maximum count. Set. However, when the sum of the continuous numbers is 200 or less, the continuous number of white pixels in the horizontal direction is counted further upward.

In setting the vertical width of the eyeball present area, as shown in the flowchart of FIG. 31, a vertical black area is searched from the number of black pixels. At this time, first, the maximum value of the number of pixels in the first black area is stored. Thereafter, the maximum value of the pixels in the black region is stored in the vertical direction toward the space between the eyebrows, and is compared with the previous maximum value. Therefore, when the search for a vertical black area reaches between the eyebrows, the black area disappears, and the eyeball exists between the first maximum coordinate value and the maximum coordinate value immediately before the eyebrows. It is set as the vertical direction of the area. As a result,
As shown in FIG. 34, left and right eyeball existence areas are set.

[0006]

As described above, the conventional face image processing apparatus always places the driver's face at a fixed position.
I am trying to capture the position of the eyeball from the image of the face,
The driver does not always drive in the same posture, and often drives in a posture in which the face is inclined. However, the above-described conventional driver's face image processing apparatus does not have a function of correcting the inclination of the face image. Therefore, when processing a greatly inclined face image to detect an eyeball existing area, the detected eyeball presence is not detected. There is a high possibility that the area is set at a position deviating from the actual position.

As a result, even if the eyeball image in the eyeball existing area is processed and the eye opening / closing judgment is made, accurate eye opening / closing judgment cannot be made, and the opening / closing state detection may be erroneously performed.
There has been a problem that even if dozing detection is performed based on the processing result of the eyeball image, it is erroneously detected.

Therefore, it is essential to recognize the movement of the head. When a human head is a sphere as shown in FIG. 28, the X axis is the horizontal direction of the face, and the Y axis is the Y axis. Is the vertical axis of the face, and the Z axis is the front-back axis direction of the face, the face is rotated about the X axis (for example, the degree of depression) or Y
In general, the direction of the face with respect to the rotation about the axis (turning the face left and right) is detected by image processing. (See IEICE Spring Spring, D-541, 1990)

[0009] However, there have been few methods of detecting the inclination of the face with respect to the rotation about the z-axis (tilting the face left and right), and it has not been possible to eliminate the erroneous detection of the open / closed state of the eyes.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and detects and judges the inclination of a face image before setting an eyeball existence area, and corrects the inclination to correct the inclination. An object of the present invention is to provide a face image processing apparatus capable of obtaining a non-existent face image.

[0011]

A face image processing apparatus according to the present invention comprises: a face image input means for inputting a driver's face image;
Feature candidate area extraction means for extracting candidate areas for feature points of the face, such as eyebrows, eyes, nose, and mouth, in the input driver's face image, and binarization means for binarizing the image from which the feature candidate areas have been extracted , The center of gravity (x,
y) a face area center of gravity calculating means, a face part area setting means for setting an area including a part of the face based on the calculated center of gravity of the face area, and a partial area representative point calculation for calculating a representative point for the face partial area means, tilt detecting means for detecting the face area center of gravity (x, y) and the straight line and the angle of one of the image axes passing through two points of the partial area representative point as the slope of the face, the inclination detection
Of the face image for correcting the inclination of the face detected by the
Rotation center setting means for setting a rotation center;
Face image by rotation based on the rotation center set by the
Rotating means to correct image tilt, whether face image is tilted
Until the face image is determined not to be tilted.
Tilt determining means to continue image rotation processing by rotating means, before
The face image is changed according to the angle θ detected by the inclination detecting means.
Rotation angle setting means for setting an angle for rotating an image,
Rotation limit angle setting means for setting the rotation angle limit angle
Rotating the face image with the rotating means and tilting the face image.
Is determined that the face image is not tilted by the
The rotation means when or until the rotation limit angle is reached.
Image rotation processing is continued .

[0012] In addition, the face image input means for inputting a face image of the driver, eyebrow at the input to the driver's face image in the eyes,
Feature candidate area extraction means for extracting candidate areas of feature points of the face such as the nose and mouth, binarization means for binarizing an image from which the feature candidate areas are extracted, and a center of gravity of the area including the face in the binary image A face area center of gravity calculating means for calculating (x, y), and an upper area in the face vertical direction is set based on the horizontal center of gravity x of the face area center of gravity (x, y) calculated by the face area center of gravity calculating means. A face partial area setting means, a center of gravity of the set upper area in the vertical direction of the face is calculated, and a partial area representative point calculating means using the calculated center as a representative point;
Inclination detecting means for detecting the angle between the straight line and one image axis passing the point as the slope of the face, detected by the inclination detecting means
The rotation center of the face image to correct the tilt of the
Rotation center setting means, set by the rotation center setting means
The tilt of the face image by rotation based on the rotated center of rotation
The rotation means determines whether the face image is tilted, and determines whether the face image
Image by the rotating means until it is determined that the
To the tilt determining means and the tilt detecting means for continuing the image rotation processing.
Angle to rotate the face image according to the detected angle θ
Rotation angle setting means for setting a degree, a limit angle of the rotation angle
A rotation limit angle setting means for setting a degree;
The face image is rotated at the step and the inclination determination means is used.
When it is determined that the face image is not tilted, or
Image rotation processing by the rotation means until the limit angle is reached
Is to continue .

[0013] In addition, the face image input means for inputting a face image of the driver, eyebrow at the input to the driver's face image in the eyes,
Feature candidate area extraction means for extracting candidate areas of feature points of the face such as the nose and mouth, binarization means for binarizing an image from which the feature candidate areas are extracted, and a center of gravity of the area including the face in the binary image A face area center of gravity calculating means for calculating (x, y), and an upper area in the face vertical direction is set based on the horizontal axis center of gravity x of the face area center of gravity (x, y) calculated by the face area center of gravity calculating means. A face partial area setting means for dividing the upper part of the face in the vertical direction of the face on the basis of the center of gravity y of the center of the face area (x, y) to set the left and right areas of the upper part of the face; A partial area representative point calculating means for calculating the center of gravity of each upper right area and using the middle point as a representative point, a straight line passing through the two points of the face area center of gravity (x, y) and the partial area representative point and one image inclination detecting means for detecting the angle of the axis as the inclination of the face, the inclination
Face image for correcting the inclination of the face detected by the detecting means
Rotation center setting means for setting the rotation center of the rotation center
The face is rotated based on the rotation center set by the setting means.
Rotating means to correct image tilt, whether face image is tilted
Before it is determined that the face image is not tilted.
Inclination determining means for continuing the image rotation processing by the rotating means,
Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting the angle to rotate the image, before
Rotation limit angle setting means for setting the rotation angle limit angle
And rotating the face image with the rotating means,
The face image is determined not to be tilted by the tilt determination means
The rotation means when or until the rotation limit angle is reached.
Image rotation processing is continued .

[0014] In addition, the face image input means for inputting a face image of the driver, eyebrow at the input to the driver's face image in the eyes,
Feature candidate area extracting means for extracting candidate areas of feature points of the face such as the nose and mouth, binarizing means for binarizing an image from which the feature candidate areas are extracted, and grouping of the feature candidate areas in the binary image Feature nominating area grouping means, a nostril area extracting means for extracting a nostril area within the feature nominating area group, a nostril area centroid calculating means for calculating a center of gravity of each nostril in the nostril area, and a calculated nostril area for each nostril. Inclination detecting means for detecting an angle between a straight line passing through the center of gravity and one image axis as the inclination of the face;
Rotation center to set rotation center of face image for tilt correction
During the rotation set by the setting means and the rotation center setting means
A rotating hand that corrects the inclination of the face image by rotating based on the mind
Determine whether the face image is tilted or not.
Until it is determined that there is no image
Detection by the inclination detection means and the inclination detection means
The angle to rotate the face image according to the angle θ
Rotation angle setting means to set the limit angle of the rotation angle
A rotation limit angle setting unit that performs
The image is rotated, and the face image is
When it is determined that it is not tilted, or the rotation limit angle
Until image rotation is reached .

[0015] In addition, the face image input means for inputting a face image of the driver, eyebrow at the input to the driver's face image in the eyes,
Feature candidate area extracting means for extracting candidate areas of feature points of the face such as the nose and mouth, binarizing means for binarizing an image from which the feature candidate areas are extracted, and grouping of the feature candidate areas in the binary image Feature region grouping means, a nostril region extracting means for extracting a nostril region within the feature candidate region group, a feature region center of gravity calculating means for calculating a center of gravity for each feature candidate region group, and a feature region center of gravity calculation means. A center-to-nostril center setting means having the center of gravity between the nostrils with respect to the area set by the nostril area extracting means, and a straight line passing through the center of gravity of each feature candidate area excluding the nostril area and the center between nostrils. And an inclination detecting means for detecting the average of the angles formed by one of the image axes as the inclination of the face.

Also , a face image input means for inputting a driver's face image, eyebrows, eyes, and the like in the input driver's face image.
Feature candidate area extraction means for extracting candidate areas of feature points of the face such as the nose and mouth, binarization means for binarizing an image from which the feature candidate areas are extracted, and a center of gravity of the area including the face in the binary image A face area centroid calculating means for calculating (x, y), a feature candidate area grouping means for grouping feature candidate areas in a binary image, a nostril area extracting means for extracting a nostril area in the feature candidate area group, A nostril center calculating means for calculating a center between nostrils of the extracted nostril region; an inclination of a face by determining an angle between a straight line passing through the center between the nostrils and the center of gravity of the face region (x, y) and one image axis; detecting a tilt detection means, wherein
A face for correcting the face inclination detected by the inclination detecting means
Rotation center setting means for setting a rotation center of an image, the rotation
Rotation based on the rotation center set by the center setting means
Rotation means for correcting the inclination of the face image
Until it is determined that the face image is not tilted.
The tilt determining means to continue the image rotation processing by the rotation means.
Step, according to the angle θ detected by the inclination detecting means
To set the angle to rotate the face image.
Step, a rotation limit angle setting for setting a limit angle of the rotation angle.
Setting means for rotating the face image by the rotating means.
Is determined that the face image is not tilted by the tilt determination means
Rotation or until the rotation limit angle is reached.
The image rotation processing by the rotation means is continued .

Also , a rotation center setting means for setting a rotation center of the face image for correcting the inclination of the face detected by the inclination detection means, and a rotation of the face image based on the rotation center set by the rotation center setting means. Rotating means for correcting the tilt of the face image, determining whether the face image is tilted,
Until it is determined that the image has not been rotated
The inclination is determined by the inclination determination unit and the inclination detection unit that continue the processing.
Set the angle to rotate the face image according to the output angle θ.
Rotation angle setting means for setting the rotation angle limit angle.
Rotation limit angle setting means for setting the face,
The image is rotated and the face image is determined by the inclination determining means.
Is not tilted, or the rotation limit angle
Image rotation processing by the rotation means
It is those that.

Further, the center of rotation setting means is a face area center of gravity in the binary image (x, y) is set as the rotation center.

Further, the center of rotation setting means is for setting the rotational center of the nostril between the centers.

[0020]

Further, the tilt determining means, the face horizontal direction in the binary image vertical (y) axis, and the image horizontal (x) axis Kaotate direction, the image horizontal axis in each coordinate on the image longitudinal axis An image horizontal axis direction histogram indicating the number of black pixels existing in the direction is created, and it is determined whether or not the face image is inclined based on the symmetry of the image horizontal axis direction histogram.

Further, the tilt determining means, the face horizontal direction in the binary image vertical (y) axis, and the image horizontal (x) axis Kaotate direction, the image horizontal axis in each coordinate on the image longitudinal axis An image horizontal axis direction histogram indicating the number of black pixels existing in the direction is created, and it is determined whether or not the face image is inclined based on the symmetry of the image horizontal axis direction histogram.

Further, the tilt determining means, the horizontal axis centroid x of the center of gravity for each feature candidate region groups obtained by the feature candidate area centroid calculating unit in the binary image (xi, yi)
It is determined whether or not the face image is inclined from the arrangement of i.

[0024]

[0025]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention are as follows. 1. The inclination of a straight line passing through the center of gravity of the face image area binarized by the driver and the representative point of the face image is obtained, and the face image is rotated based on the inclination to obtain a face image without inclination.

2. The center of gravity of the binarized face image region of the driver and a representative point in the face vertical direction upper region of the face image region set at the center of gravity are obtained, and the face image is determined based on the inclination of a straight line passing through the center of gravity and the representative point. To obtain a face image without inclination.

3. The center of gravity of the binarized face image area of the driver and the midpoint between the center of gravity of the upper left face area and the center of right face of the face image area set with this center of gravity as the representative point, pass through the center of gravity and the representative point. The face image is rotated based on the inclination of the straight line to obtain a face image without inclination.

4. 2 extracted from the driver's face image area
Calculate the center of gravity for each nostril in the binarized nostril region,
The face image is rotated based on the inclination of a straight line passing through each center of gravity.

5. After extracting candidate regions of each feature point of the face from the driver's face image region and binarizing the feature candidate regions into groups, the calculated center of gravity is used as a center point between the nostrils in the feature candidate regions including the nostrils. The face image is rotated based on the average of the inclination of a straight line passing through the center point and the center of gravity calculated in each of the other feature candidate areas.

6. After calculating the center of gravity of the binarized face area, extracting candidate areas for each feature point of the face from the driver's face image area, binarizing the feature candidate areas, and grouping them. In the feature candidate area including the nostrils, the calculated center of gravity is used as the center point between the nostrils, and the face image is rotated based on the inclination of a straight line passing through the center point and the calculated center of gravity.

7. The predetermined coordinate position is set as the rotation center position of the face image, and the face image is rotated in the direction opposite to the inclination direction by the inclination angle of the straight line based on the rotation center position.

8. The rotation center position is set as the center of gravity of the face area, and the face image is rotated in the direction opposite to the inclination direction by the inclination angle of the straight line based on the center of gravity.

9. The rotation center position is set as the center between the nostrils, and the face image is rotated in the direction opposite to the inclination direction by the inclination angle of the straight line based on the center.

10. It is determined whether or not the face image is tilted by the tilt determination means, and the image rotation processing by the rotation means is continued until it is determined that the face image is not tilted.

11.2 The horizontal direction of the face in the value image is defined as the image vertical (y) axis, and the vertical direction of the face is defined as the image horizontal (x) axis. An image vertical direction histogram indicating the number of pixels is created, and it is determined whether the face image is inclined based on the shape and the peak number of the image vertical direction histogram.

12. The horizontal direction of the face in the binary image is defined as the image vertical (y) axis, and the vertical direction of the face is defined as the image horizontal (x) axis. An image horizontal axis direction histogram indicating the number of pixels is created, and it is determined from the symmetry of the image horizontal axis direction histogram whether the face image is inclined.

13. Whether or not the face image is inclined from the arrangement of the barycenters (xi, yi) in the horizontal axis direction of the barycenters (xi, yi) for each feature candidate region group obtained by the feature candidate region barycenter calculation means in the binary image Is determined.

14. The face image is rotated by the rotation means, and the image rotation processing by the rotation means is continued when the inclination determination means determines that the face image is not tilted or until the rotation limit angle is reached.

Hereinafter, each embodiment will be described in detail. Embodiment 1 FIG. Hereinafter, the present embodiment will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of the face image processing apparatus according to the present embodiment. The face image processing apparatus according to the present embodiment includes a CCD camera 1 that captures a driver's face image,
An image memory 2 stores image data of a face image output from a CD camera 1, and a CPU 3 that performs image processing based on the data in the image memory 2. FIG. 2 is a flowchart showing an outline of the image processing of the CPU 3 in FIG.

First, the outline of the image processing will be described with reference to the flowchart of FIG. The CPU 3 inputs image data of the driver's face image from the image memory 2 by the face image input means 4, and extracts feature point candidate areas of the input face image by the feature candidate area extraction means 5. The candidate area of the face image extracted by the feature candidate area extracting unit 5 is binarized by the binarizing unit 6.

After the center of gravity of the candidate area including the face in the binarized face image is calculated by the face area center of gravity calculating means 7, the face partial area setting means 8 sets the center of gravity of the candidate area in the face area based on the face area center of gravity. A new face part area is set. Partial area representative point calculation means 9
Then, the inclination detecting means 10 obtains an angle between a straight line connecting the representative point and the center of gravity of the face area in the set face part area and one image axis, and corrects the inclination of the face area by rotation. The rotation center is set by the rotation center setting means 11, and the rotation is performed by the rotation means 12.

Hereinafter, each means shown in the flowchart of FIG. 2 will be described in detail with reference to each figure. FIG. 3 shows a face image of the driver. FIG. 3A shows a face image 13 of the driver captured by the CCD camera 1. FIG. 2B shows the face image 14 after the processing by the feature candidate area extracting means 5 and the binarizing means 6 has been performed, and the feature candidate area has been binarized. FIGS. 5A and 5B are explanatory diagrams of the feature candidate area extracting unit 5 and the binarizing unit 6 in the present embodiment.
FIGS. 4A, 4B, and 4C are explanatory diagrams of a max / min filter used in the feature candidate region extracting unit 5. FIG.

First, the max / min filter will be described. In FIG. 4A, the image signal 15 obtained by applying a max filter to the image signal 15 of a part of the input image is indicated by 16, and the image signal obtained by applying the min filter to the image signal 16 is indicated by 17. In FIG. 4B, reference numerals 18 to 20 denote luminance values of the image signals 15 to 17 corresponding to the respective pixels which are schematically illustrated. Here, one frame surrounding the luminance value represents one pixel, and the luminance value is set to a value from 0 to 20 for simplicity.

First, a max filter is applied to an input image. The max filter has a predetermined length (the number of pixels a, hereinafter referred to as a filter size), and the luminance value of the pixel at the center of the filter is set to the maximum luminance value in the filter. It is something to convert.
For example, assume that the number of pixels a is 5, and m
When the ax filter is applied, the maximum luminance is 10 here, so that the luminance value of the pixel 18b at the center of the filter is converted from 8 to 10.

When the pixel is shifted to the right by one pixel and subjected to the max filter in the same manner, 18 in FIG.
Is converted as follows. Next, a min filter is applied to the image after the max filter processing. The min filter is also max
It has the same filter size (number of pixels a) as the filter, and converts the luminance value of the pixel at the center of the filter into the minimum luminance value in the filter.

When the min filter is applied to the 19 square portion in FIG. 4B after the max filter processing, the minimum luminance value is 10 in this case.
The luminance value of 9c is converted from 12 to 10. FIG. 4 shows the result of shifting to the right by one pixel and applying a min filter in the same manner.
The 19 □ part in (b) is converted as 20.

However, for both the max filter and the min filter, conversion starts from the center position of the first applied filter, and ends at the center position of the last applied filter. That is, the data after the max filter processing is shown in FIG.
The data after the 19d to 19e and min filter processing in (b) are 20f to 20g in FIG.
After all, the image after the n-filter processing is smaller by a-1 pixel than the processing start position and the processing end position.

When the input image is subtracted from the image after the max / min filter processing obtained in this way, a low-luminance region of a predetermined length or less is extracted. 4 to 20 in FIG.
FIG. 4D is obtained by subtracting 8. The image signal is as shown in FIG.

The above method is used to extract a feature region. In the present embodiment, processing is performed line by line along the horizontal axis (X axis) of the image (processing direction: arrow 23 in FIG. 3A).
Direction). Reference numeral 21 in FIG. 5A denotes an image signal after the input image has been subjected to a max / min filter having a length corresponding to the width of the characteristic region. Since the luminance difference between the characteristic region of the face and the surroundings is large, the luminance value after the difference becomes large (21a).

On the other hand, shadows and clothes have a small difference in luminance from the surroundings (21b). Therefore, a threshold value is set for the result after the difference as shown at 22 in FIG.
Perform value conversion. FIG. 5B shows the signal after binarization. FIG. 3B shows an image obtained by the above processing.
It is.

FIGS. 6 (a), 6 (b) and 6 (c) are illustrations of the face area center of gravity setting means 7, the face partial area setting means 8 and the partial area representative point calculating means 9. In the face area centroid setting means 7,
First, the center of gravity 2 with respect to the entire area of the image 14 shown in FIG.
4 is calculated. The center of gravity is calculated in the black area (xj, y
j) the total number of pixels is total, and the sum of all yj pixels is Y
Assuming that the sum of the pixel numbers of t and all xj is Xt, the coordinate point (X, Y) of the center of gravity 24 is obtained by the following equation (1).

[0052]

(Equation 1)

After calculating the center of gravity 24, the face portion area setting means 8
Sets an area 25 including a face based on the coordinate points (X, Y) (hereinafter, an area including a face is referred to as a face area). Face area 2
5 indicates that the ratio of the face to the input image is large,
Considering that the center of gravity of the face is located approximately between the nose and the eyes at the center of the face, the face region 25 is a feature amount of the face (at least the eyes and the nose).
And a rectangle including The face area center of gravity 26 (coordinates (a, b)) with respect to the face area 25 is calculated using the above equation (1), similarly to the center of gravity 24.

Next, the face area setting means 8 will be described. The face area 25 is placed in the upper part in the face vertical direction (including at least the eyes) based on a of the calculated coordinates (a, b) of the face area center of gravity 26
It is divided into a region and a lower region in the face vertical direction. Here, the upper region is adopted as the face portion region 27. The processing by the face part area representative point calculation means 9 is performed on the face part area 27. In the present embodiment, the face part area representative point 28 (coordinates (c,
d)) is, as shown in FIG. 6C, the center of gravity with respect to the face part region 27 obtained by the above equation (1).

FIG. 7 is an explanatory diagram of the inclination detecting means 10.
The inclination detecting means 10 calculates the face area centroid 26 (a, b) and the face partial area representative point 28 obtained by executing the above-described processing.
A straight line 29 passing through (c, d) is obtained, and an angle θ formed with one coordinate axis, in this embodiment, the X axis, is obtained. The angle θ is obtained by the following equation (2).

[0056]

(Equation 2)

FIG. 8 shows a binary image after rotation. The rotation center setting unit 11 sets the face area center of gravity 26 as the rotation center,
The face area 25 is rotated counterclockwise by θ by the rotation means 12. In this case, the rectangular area represented by a black background is the face area 25 before rotation, and the coordinates of each pixel of the face area 25 are represented by (e, f) as a whole. The rectangular area represented by the intermediate color is the face area 25 after rotation, and the coordinates of each pixel of the face area 25 are represented by (E, F) as a whole. Expression (3) is used for the rotation process of the face area 25. However, the portion (30a in FIG. 8) where the coordinates (E, F) after rotation comes out of the image is cut, and the portion where the coordinates (e, f) before rotation does not exist (FIG. 8).
30b) has a luminance value of 0.

[0058]

(Equation 3)

Embodiment 2 In the first embodiment, a face partial area representative point is obtained in the upper area of the face partial area including the center of gravity of the face area, and the angle θ between the straight line connecting the face partial area representative point and the center of gravity and the coordinate axis is determined by the face. Although the correction rotation angle of the area is used, the face part area representative point may be displaced when noise or the like rides on the image data in which the face part area representative point is roughly determined. Therefore, the partial area setting means 8 and the partial area representative point calculating means 9 are changed as follows.

The changed face part area setting means 8 will be described. In the present embodiment, the face area centroid calculating means 7
The face part area 27 set based on a of the face area center of gravity 26 (a, b) calculated in the above is further divided into left and right face areas 31 and 32 shown in FIG. The two left and right face areas 31 and 32 are adopted as face partial areas. The processing by the face part area representative point calculation means 9 is performed on the face left and right areas 31 and 32. In the present embodiment, as shown in FIG. 9, the face part region representative points are the regions 31 and 3 obtained by the above equation (1).
2 the center point 34 of the center of gravity 33a, 33b for each
(E, f).

As described above, when the face part region representative point 34 is obtained, although not shown, a straight line connecting the face region center of gravity 26 and the face part region representative point 34 and one coordinate axis, X in the present embodiment, Axis and the angle θ to be formed. The angle θ is obtained by the above equation (2). Then, the rotation center setting means 11
, The center of gravity 26 of the face area is set as the center of rotation, and the face area 25 is rotated counterclockwise by θ by the rotation means 12.

Embodiment 3 An outline of the processing of the third embodiment will be described based on the flowchart of FIG. In the present embodiment, two nostril regions that can be converted into coordinate values with the highest accuracy among the feature candidate regions are set as feature candidate regions. C
The PU 3 inputs image data from the image memory 2 by the face image input unit 4, and extracts feature candidate regions of the input face image by the feature candidate region extraction unit 5. The image extracted from the feature candidate area is binarized by the binarization means 6. In the binarized image, the feature candidate areas are grouped into feature candidate area grouping means 35.
To group into a plurality as shown in FIG.

The nostrils are extracted by the nostril region extracting means 36 from the grouped feature candidate region groups. Then, the center of gravity of each nostril region is calculated by the nostril region center of gravity calculation means 37. An angle between a straight line passing through two points of the center of the nostril region and one of the image axes is obtained by the inclination detecting means 10, and the inclination is corrected by rotation. The rotation center is set at the center of the center of gravity of each nostril region by the rotation center setting unit 11, and the rotation unit 12 rotates the face region 25 in the direction opposite to the tilt direction of the face region based on the rotation center.

Hereinafter, each means shown in the flowchart of FIG. 10 will be described in detail. The face image input means 4, the feature candidate area extracting means 5, and the binarizing means 6 are the same as those of the first embodiment.
Therefore, the feature candidate area grouping means 35 is omitted.
It will be explained first. 11A, 11B and 11C are diagrams for explaining the operation of the feature candidate area grouping means 35, and FIGS.
14A and 14B are diagrams for explaining the operation of the nostril region extracting means 36, and FIGS.
It is explanatory drawing of FIG. FIG. 13 shows an image after processing by the feature candidate area grouping means 36.

The feature candidate area grouping means 35 counts the number of black pixels in the vertical direction of the image for each X coordinate with respect to the face image after binarization, and the vertical axis as shown in FIG. A histogram is created in the vertical axis direction of the image in which the horizontal axis is the X coordinate and the horizontal axis is the pixel number. Next, from the histogram, FIG.
As shown in the figure, a peak where the maximum number of black pixels is 38a or more is searched, and a peak start position 38c where the number of black pixels is 38e or more and the distance from the maximum point is 38b or less is found for each peak.
i, the peak end position 38di is set.

This time, the number of black pixels in the horizontal direction of the image is counted for each Y coordinate for each of the band-shaped regions 39ai from the set peak start position 38ci to the peak end position 38di, as shown in FIG. An image horizontal histogram is created in which the vertical axis is the Y coordinate and the horizontal axis is the number of black pixels. In the histogram in the horizontal direction of the image, since only the region having a predetermined length or less is extracted due to the characteristics of the feature candidate region extracting means 5, a part of the eye and a part of the eyebrow may be cut off in some cases.

Therefore, the distance 3 shown in FIG.
39c excluding the portion of 9c or more and the number of black pixels 39b or less
i and 39di and the same band-shaped region 39a
Those belonging to i are set as feature candidate area groups 40i.

In the nostril region extracting means 36, as shown in FIG. 14 (a), two groups having substantially the same width are arranged in the feature candidate region group 40i, and the width of the group between them is narrower than the other groups. The nostril area 42 is used. The nostril region center-of-gravity calculating means 37 calculates the nostril region 42 shown in FIG.
Of gravity 43a for each black area in the two groups
(A, b) and 43b (c, d) are obtained by the above equation (3).

Next, as shown in FIG.
At 0, a straight line 44 passing through the nostril region centroids 43a (a, b) and 43b (c, d) obtained by executing the above processing is obtained, and one of the coordinate axes, that is, the Y axis in the present embodiment, is determined. Obtain the angle θ. The angle θ is obtained by the above equation (2). The center of gravity 43a of the nostril region is set by the rotation center setting means 11,
The midpoint 45 of 43b is set as the rotation center, and the face area is rotated counterclockwise by θ by the rotation means 12. The rotation means 12 is the same as in the first embodiment, and utilizes the above equation (3).

Embodiment 4 In the above-described third embodiment, the inclination is corrected by rotating the face area based on the inclination angle of a single straight line passing through the center of gravity of the nostril area, which is one of the feature candidate area groups. In the present embodiment, the inclination angle of each straight line connecting the midpoint obtained in Embodiment 3 and the center of gravity of each feature candidate area is obtained, and the average value of the inclination angles is calculated to be the rotation angle θ. An outline of the processing of the present embodiment will be described based on the flowchart of FIG.

The CPU 3 inputs image data from the image memory 2 by the face image input means 4 and extracts the feature candidate area of the face image in the input image data by the feature candidate area extracting means 5. The face image extracted from the feature candidate area is binarized by the binarization means 6. The feature candidate areas are grouped by the feature candidate area grouping means 35 in the binarized face image. The nostrils are extracted by the nostril region extracting means 36 from the grouped feature candidate region groups.

The characteristic area centroid calculating means 46 calculates the centroid of each group set by the characteristic candidate area grouping means 35. The inclination detecting means 1 calculates the average of the straight line passing through the two points of the center of the nostril set by the nostril center setting means 47 and the center of gravity of each feature candidate area excluding the nostril area and the angle formed by one image axis.
Calculate with 0. The rotation center is set by the rotation center setting means 11. The face image is rotated by an average angle by the rotation means 12 based on the set center of rotation to correct the inclination.

Hereinafter, each means shown in the flowchart of FIG. 16 will be described in detail. The face image input unit 4, the feature candidate region extracting unit 5, the binarizing unit 6, the feature candidate region grouping unit 35, and the nostril region extracting unit 36 have been described in the first and third embodiments, and are omitted. The description starts with the characteristic region centroid calculation means 46. In the characteristic region centroid calculating means 46, as shown in FIG.
The center of gravity 48i (ci, di) for the black area in the group 40i existing in ai (i: 1 to 5) is obtained. The above equation (1) is used for calculating the center of gravity.

In the nostril center setting means 47, the feature candidate area centroid 48i with respect to the nostril area 42 obtained in advance by the nostril area extracting means 36 is set as the inter-nostril center 49 (a, b).

Next, as shown in FIG.
0, each feature candidate area centroid 48i (ci, di) obtained by executing the above-described processing and the center 49
An angle between the straight line passing through the two points (a, b) and one of the coordinate axes, in this embodiment, the X axis is determined, and the average value is calculated as the angle θ.
And The angle θ is obtained by the following equation (4). The rotation center setting means 11 sets the center 49 (a, b) between nostrils as the rotation center, and the rotation means 12 rotates the counterclockwise θ. The rotation means 12 is the same as in the first embodiment, and utilizes the above equation (3).

[0076]

(Equation 4)

Embodiment 5 In the fourteenth embodiment, the center of gravity of each of the grouped feature candidate regions is obtained and connected to the center of the nostrils by a straight line. However, in order to simplify the processing, the inclination of the straight line connecting the center of the nostrils and the center of gravity of the face region is simplified. The tilt correction angle may be determined by the angle. The outline of the processing of the present embodiment will be described based on the flowchart of FIG. CPU3
Input image data from the image memory 2 by the face image input means 4 and extract the feature candidate area of the face image in the input image data by the feature candidate area extracting means 5. The face image extracted from the feature candidate area is binarized by the binarization means 6. After calculating the center of gravity of the area including the face in the binarized face image by the face area centroid calculating means 7, the feature candidate areas in the binarized face image are grouped by the feature candidate area grouping means 35. I do. The nostrils are extracted by the nostril region extracting means 36 from the grouped feature candidate region groups.

The nostril center calculating means 50 calculates the nostril center for the extracted nostril region. An angle formed by a straight line passing through the center between the nostrils and the center of gravity of the face area and one image axis is obtained by the inclination detecting means 10. The face image is rotated by the rotation unit 12 based on the rotation center set by the rotation center setting unit 11 to correct the inclination of the face image.

Hereinafter, each means shown in the flowchart of FIG. 19 will be described in detail. The face image input means 4, the feature candidate area extracting means 5, the binarizing means 6, the face area centroid setting means 7, the feature candidate area grouping means 35, and the nostril area extracting means 36 are described in the first embodiment. Since the description has been given in the third embodiment, the description will be omitted, and the description will start from the nostril center calculating means 50. However, the calculated face area centroid 2
6 is (a, b) in the first embodiment, but is (c, d) in the present embodiment for explanation.

The nostril center calculating means 50 obtains the center of gravity 51 (a, b) for the black area in the nostril area group existing in the same band-like area extracted by the nostril area extracting means 36 as shown in FIG. . The above equation (1) is used for calculating the center of gravity.

Next, in the inclination detecting means 10, the face area center of gravity 26 (c,
d) and a straight line passing through two points of the nostril center 51 (a, b);
An angle θ between one coordinate axis and the X axis in the present embodiment is obtained. The angle θ is obtained by the above equation (2). The center 51 (a, b) between the nostrils is set as the rotation center by the rotation center setting means 11, and the rotation means 12 rotates counterclockwise by θ. The rotation means 12 is the same as in the first embodiment, and
Use expressions.

Embodiment 6 FIG. In the above first to fifth embodiments, the inclination determination of the face area is performed based on the result of calculating the center of gravity of the face area and the representative point. However, in order to simplify the processing, the number of black pixels of the binarized feature candidate area is calculated. The inclination of the face area is directly determined from the histogram, and if the inclination is not determined, all processing is stopped there. If the inclination is determined, the same processing as in the above embodiments is performed to perform the face image rotation processing.

FIG. 21 is a flowchart illustrating an outline of the processing according to the present embodiment. In the figure, the same step numbers as those in the flowchart of FIG. 2 indicate the same processes. In the figure,
Step 52 is an inclination determining means 52 for directly determining the presence or absence of the inclination of the face area from the histogram of the number of black pixels in the binarized feature candidate area. Then, if it is determined that there is an inclination, the process proceeds to the face area center-of-gravity calculation means 7, and if it is determined that there is no inclination, all the processing is stopped.

The inclination judging means 52 sets the image horizontal (X) axis to the face horizontal direction and the image horizontal (X) axis to the face vertical direction with respect to the binarized image, and calculates the number of black pixels in the image vertical direction. Counting is performed for each X coordinate, and a vertical axis image histogram is created in which the vertical axis represents the number of black pixels and the horizontal axis represents the X coordinate. FIG. 22 (a) is an image vertical direction histogram of a face image that is not tilted, and FIG. 22 (b) is an image vertical direction histogram of a tilted face image.

As can be seen from the figure, a face image that is not tilted has a clear histogram peak, and the number of black pixels is almost the same as the number of characteristic regions of the face (normally, four of eyebrows, eyes, nose, and mouth). peak). In contrast, the histogram of a tilted image does not have clear peaks, and the number of peaks does not match the number of characteristic regions of the face. The histogram shape,
Then, it is determined from the number of peaks whether or not it is inclined.

Embodiment 7 In the above-described sixth embodiment, a histogram is created by counting the number of black pixels in the vertical direction of the image for each X coordinate, and non-inclination of the face image is determined from the clarity of the peak. Since it is symmetrical about the line, the number of black pixels in the image
A histogram is created by counting the coordinates, and non-inclination of the face image can be determined based on the appearance of symmetrical peaks having the same peak value.

FIGS. 23 (a) and 23 (b) are explanatory diagrams of the inclination judging means 52 in which the judgment criteria are changed. Tilt determining means 52
In the image, the horizontal direction of the face is set to the image vertical (y) with respect to the binarized image.
The horizontal and vertical directions of the face are defined as the horizontal (x) axis of the image, and the number of black pixels in the horizontal direction of the image is counted for each Y coordinate. I do. FIG. 23A is an image horizontal axis histogram of an image that is not tilted, and FIG. 23B is an image horizontal axis histogram of an image that is tilted.

As can be seen from the figure, an image that is not tilted has a clear histogram peak and is almost symmetrical (normally, symmetrical at the center of the face). On the other hand,
The histogram of the skewed image has no sharp peaks and is not symmetrical. As described above, whether or not the face image is inclined is determined from the shape and the symmetry of the histogram.

Embodiment 8 FIG. FIG. 24 is a flowchart illustrating the face image processing according to the present embodiment. FIG.
(A) and (b) show the inclination determining means 52 according to the present embodiment.
FIG. As shown in FIG. 24, the inclination determining means 52 is added after the characteristic area center-of-gravity calculating means 46 of the fourth embodiment, and if the result of the determination is no inclination, all the processing is stopped. After the processing up to the rotation means 12, the process returns to the feature candidate area extracting means 5 again, and the same processing is repeated thereafter.

The inclination determining means 52 determines whether or not the center of gravity calculated by the characteristic area gravity center calculating means 46 is more inclined. FIG. 25A shows the arrangement of the centers of gravity of the images which are not inclined, and FIG. 25B shows the arrangement of the centers of gravity of the images which are inclined. As can be seen from the figure, the center of gravity of the image that is not tilted is substantially aligned with the vertical center line of the face. On the other hand, the arrangement of the center of gravity of the tilted image is not constant. As described above, it is determined whether or not the line is inclined from the arrangement of the feature region centroids.

Embodiment 9 FIG. FIG. 27 shows the rotation angle setting means 53a and the rotation limit angle setting means 5 in the present embodiment.
It is explanatory drawing of 3b. Embodiment 6 As shown in FIG.
The rotation angle setting means 53a and the rotation limit angle setting means 53b are added before the rotation means 12 to 8 to rotate the minimum rotation angle β and the maximum rotation angle (θ + α) according to the detection angle θ. ) Is set.

However, as shown in FIG. 27, the rotation angle β does not exceed the rotation limit angle (θ + α), and the rotation limit angle (θ + α) is set based on the detected angle θ. After the processing up to the rotation means 12, the process returns to the feature candidate area extracting means 5 again, and the same processing is repeated thereafter. When repeating, the rotation limit angle (θ + α) is always kept constant. The rotation angle β need not be constant, but it is considered that the closer to the detection angle θ, the smaller the rotation angle and the greater the number of rotations. The above processing is continued until the inclination determining means 52 determines that the image is not inclined or until the rotation limit angle (θ + α) is reached.

[0093]

According to the present invention, the following effects can be obtained. 1. Face image input means for inputting a driver's face image; feature candidate area extracting means for extracting candidate feature points of a face such as eyebrows, eyes, nose, mouth, etc. in the input driver's face image; Means for binarizing an image from which is extracted, a face area center of gravity calculating means for calculating a center of gravity (x, y) with respect to an area including a face in the binary image, based on the calculated center of gravity of the face area A face partial area setting means for setting an area including a part of the face; a partial area representative point calculating means for calculating a representative point for the face partial area; and two points of the face area centroid (x, y) and the partial area representative point. The inclination detecting means for detecting the angle between the passing straight line and one image axis as the inclination of the face ,
Of image rotation for correcting face tilt detected
Rotation center setting means for setting the rotation center setting means
The tilt of the face image by rotation based on the rotation center set from
Rotation means to correct the face, determine whether the face image is tilted
The rotating hand until it is determined that the face image is not tilted.
Tilt determining means for continuing the image rotation processing by the step, the tilt
The face image is rotated according to the angle θ detected by the detection means.
Rotation angle setting means for setting an angle to be rotated, the rotation angle
Equipped with a rotation limit angle setting means for setting a limit angle of the degree,
Rotating the face image by the rotation means and determining the inclination;
When it is determined by the means that the face image is not tilted,
Is the image by the rotation means until the rotation limit angle is reached.
Since the image rotation process is continued , the tilt angle can be easily detected with high accuracy and at high speed, and the fine tilt correction process can be performed.
Management is performed to improve the accuracy of correcting the inclination of the facial image, also, the face
Image processing to correct and process the face image without tilt
Improves the processing quality and limits the rotation range for tilt correction
It can suppress unnecessary rotation and prevent large erroneous rotation.
There is an effect that it can be avoided .

2. Face image input means for inputting a driver's face image, eyebrows, eyes, nose, and the like in the input driver's face image
Feature candidate area extracting means for extracting a candidate area of a feature point of a face such as a mouth, binarizing means for binarizing an image from which the feature candidate area is extracted, and a center of gravity (x , Y), and a face portion for setting an upper region in the vertical direction of the face based on the center x in the horizontal axis of the center of gravity (x, y) of the face region calculated by the center of gravity of the face region calculated by the center of gravity of the face region. An area setting means for calculating a center of gravity of the set upper area in the vertical direction of the face and a partial area representative point calculating means using the calculated center as a representative point;
Inclination detecting means for detecting the angle between the straight line and one image axis passing the point as the slope of the face, detected by the inclination detecting means
The rotation center of the face image to correct the tilt of the
Rotation center setting means, set by the rotation center setting means
The tilt of the face image by rotation based on the rotated center of rotation
The rotation means determines whether the face image is tilted, and determines whether the face image
Image by the rotating means until it is determined that the
To the tilt determining means and the tilt detecting means for continuing the image rotation processing.
Angle to rotate the face image according to the detected angle θ
Rotation angle setting means for setting a degree, a limit angle of the rotation angle
A rotation limit angle setting means for setting a degree;
The face image is rotated at the step and the inclination determination means is used.
When it is determined that the face image is not tilted, or
Image rotation processing by the rotation means until the limit angle is reached
Since continued, accurately partial area representative points through a straight line for detecting the inclination angle with the face area center of gravity, yet can be obtained in easy and fast, fine granularity inclination correction processing
Rotation range for tilt correction.
Because the enclosure can be limited, useless rotation is suppressed and large
There is an effect that erroneous rotation can be avoided .

3. Face image input means for inputting a driver's face image, eyebrows, eyes, nose, and the like in the input driver's face image
Feature candidate area extracting means for extracting a candidate area of a feature point of a face such as a mouth, binarizing means for binarizing an image from which the feature candidate area is extracted, and a center of gravity (x , Y), an upper area in the vertical direction of the face is set based on the center of gravity x in the horizontal axis of the center of gravity (x, y) of the face area calculated by the center of gravity of the face area calculated by the center of gravity of the face area. Face partial area setting means for dividing the upper part of the face in the vertical direction of the face based on the center of gravity y of the vertical axis of the face area (x, y) to set the left and right areas of the upper face of the face, the set upper left area of the face, and the upper right of the face A partial area representative point calculating means for calculating the center of gravity of each area and using the middle point as a representative point; a straight line passing through the two points of the face area center of gravity (x, y) and the partial area representative point; Tilt detecting means for detecting an angle formed as a face tilt , the tilt
Face image for correcting the inclination of the face detected by the detecting means
Rotation center setting means for setting the rotation center of the rotation center
The face is rotated based on the rotation center set by the setting means.
Rotating means to correct image tilt, whether face image is tilted
Before it is determined that the face image is not tilted.
Inclination determining means for continuing the image rotation processing by the rotating means,
Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting the angle to rotate the image, before
Rotation limit angle setting means for setting the rotation angle limit angle
And rotating the face image with the rotating means,
The face image is determined not to be tilted by the tilt determination means
The rotation means when or until the rotation limit angle is reached.
Since continued image rotation processing by, it is possible to obtain the straight line for detecting the inclination angle may more accurately partial area representative point through with the face area center of gravity, yet easy and fast, key
Fine tilt correction processing can be performed, correction accuracy is improved, tilt
Since the rotation range for correction can be limited,
There is an effect that rolling can be suppressed and a large erroneous rotation can be avoided .

4. Face image input means for inputting a driver's face image, eyebrows, eyes, nose, and the like in the input driver's face image
Feature candidate region extraction means for extracting candidate regions for feature points of a face such as a mouth, binarization means for binarizing an image from which feature candidate regions have been extracted, and feature for grouping feature candidate regions in a binary image Candidate area grouping means, a nostril area extracting means for extracting a nostril area in the feature candidate area group, a nostril area centroid calculating means for calculating a center of gravity for each nostril in the nostril area,
Inclination detecting means for detecting the angle between the straight line and one image axis through the center of gravity for each nostril of the calculated nostril area as the inclination of the face, the inclination of the detected face by the inclination detecting means
Rotation center setting to set the rotation center of the face image for correction
Means, the rotation center set by the rotation center setting means
Rotation means for correcting the inclination of the face image by rotation based on the reference, face
Determines whether the image is tilted and the face image is not tilted
The image rotation processing by the rotation means is continued until it is determined that
The inclination detecting means, which is detected by the inclination detecting means.
Time to set the angle to rotate the face image according to the angle θ
Rotation angle setting means for setting a rotation angle limit angle;
A rotation limit angle setting means is provided, and the face image is rotated by the rotation means.
And tilt the face image by the tilt determination means.
When it is determined that there is no rotation, or reaches the rotation limit angle
Image rotation processing by the rotation means until
Since the face image is rotated and image-corrected based on the inclination angle of the straight line passing through the two nostrils, the position of which does not change in the face image, the correction accuracy is improved, and the fine inclination correction process is performed.
Control the rotation range for tilt correction
Therefore, useless rotation can be suppressed and large wrong rotation can be avoided.
There is an effect that can be cut .

[0097] 5. Face image input means for inputting a driver's face image, eyebrows, eyes, nose, and the like in the input driver's face image
Feature candidate region extraction means for extracting candidate regions for feature points of a face such as a mouth, binarization means for binarizing an image from which feature candidate regions have been extracted, and feature for grouping feature candidate regions in a binary image Candidate area grouping means, a nostril area extracting means for extracting a nostril area in the feature candidate area group, a feature area centroid calculating means for calculating a centroid for each feature candidate area group, and a centroid calculated by the feature area centroid calculating means. And a nostril center setting means having a center of gravity between the nostrils with respect to the area set by the nostril area extracting means, and a straight line passing through two points of the center of gravity of each feature candidate area excluding the nostril area and the center between the nostrils. Since the inclination detecting means for detecting the average of the angles formed by the image axes as the inclination of the face is provided, there is an effect that the inclination angle detection accuracy is improved.

6. Face image input means for inputting a driver's face image, eyebrows, eyes, nose, and the like in the input driver's face image
Feature candidate area extracting means for extracting a candidate area of a feature point of a face such as a mouth, binarizing means for binarizing an image from which the feature candidate area is extracted, and a center of gravity (x , Y), a feature area grouping means for grouping feature candidate areas in a binary image, a nostril area extracting means for extracting a nostril area in the feature candidate area group, A nostril center calculating means for calculating a center between nostrils of the nostril region, and detecting an angle between a straight line passing through the center between the nostrils and the center of gravity of the face region (x, y) and one image axis as a face inclination. Tilt detecting means ,
A face for correcting the face inclination detected by the inclination detecting means
Rotation center setting means for setting a rotation center of an image, the rotation
Rotation based on the rotation center set by the center setting means
Rotation means for correcting the inclination of the face image
Until it is determined that the face image is not tilted.
The tilt determining means to continue the image rotation processing by the rotation means.
Step, according to the angle θ detected by the inclination detecting means
To set the angle to rotate the face image.
Step, a rotation limit angle setting for setting a limit angle of the rotation angle.
Setting means for rotating the face image by the rotating means.
Is determined that the face image is not tilted by the tilt determination means
Rotation or until the rotation limit angle is reached.
Since the image rotation processing by the rotation means is continued , the inclination angle can be detected accurately and easily, and the fine inclination correction processing
Rotation for improved correction accuracy and tilt correction
Since the range can be limited, unnecessary rotation is suppressed and the size is large.
This has the effect of avoiding erroneous rotation .

[0099] 7. To correct the inclination of the rotation center setting section, a face image by rotating the rotation center set by the center of rotation setting means as a reference for setting the rotational center of the face image for the detected face of the tilt correction by the inclination detecting means The rotation means determines whether the face image is tilted, and determines whether the face image is tilted.
Until it is determined that there is no image
Detection by the inclination detection means and the inclination detection means
The angle to rotate the face image according to the angle θ
Rotation angle setting means to set the limit angle of the rotation angle
A rotation limit angle setting unit that performs
The image is rotated, and the face image is
When it is determined that it is not tilted, or the rotation limit angle
Image rotation processing by the rotation means is continued until
So, I corrected the face image to a face image without tilt
Processing, the image processing quality is improved,
Rotation range can be limited, so that unnecessary rotation is suppressed and
In addition, there is an effect that large erroneous rotation can be avoided .

[0100] 8. Since the rotation center setting means sets the center of gravity (x, y) of the face area in the binary image as the rotation center, there is an effect that the rotation center can be easily set.

9. The rotation center setting means sets the center between nostrils as the rotation center, so that the rotation center can be easily set.
And the accuracy of center setting is improved.

[0102]

11. The inclination determining means sets the horizontal direction of the face in the binary image to the image vertical (y) axis, sets the vertical direction of the face to the image horizontal (x) axis, and uses the black color existing in the image vertical direction at each coordinate in the image horizontal direction. An image vertical direction histogram indicating the number of pixels is created, and it is determined whether or not the face image is tilted based on the shape and peak number of the image vertical direction histogram, so that the tilt determination process is simplified. effective.

12. The inclination determination means uses the horizontal direction of the face in the binary image as the image vertical (y) axis and the vertical direction of the face as the image horizontal (x) axis. An image horizontal axis direction histogram indicating the number of pixels is created, and it is determined whether or not the face image is tilted based on the symmetry of the image horizontal axis direction histogram. Is further simplified.

13. The inclination determining means is a horizontal axis x of the center of gravity (xi, yi) for each feature candidate area group obtained by the feature candidate area center of gravity calculation means in the binary image.
Since it is determined whether or not the face image is tilted from the arrangement of i, there is an effect that the tilt determination processing is simplified.

[0106]

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a flowchart illustrating a flow of processing according to the first and second embodiments.

FIG. 3 is a diagram of a face subjected to image processing.

FIG. 4 is an explanatory diagram of a max / min filter.

FIG. 5 is an explanatory diagram of a binarization process.

FIG. 6 is a diagram showing a part of a processing flow according to the first embodiment;

FIG. 7 is an explanatory diagram of a tilt detection unit, a rotation center setting unit, and a rotation unit;

FIG. 8 is a face image after inclination correction.

FIG. 9 is a diagram showing a part of a processing flow according to the second embodiment;

FIG. 10 is a flowchart illustrating a flow of a process according to the third embodiment;

FIG. 11 is an explanatory diagram of an image vertical axis direction histogram.

FIG. 12 is an explanatory diagram of an image horizontal axis direction histogram.

FIG. 13 is a face image after a feature candidate area grouping process.

FIG. 14 is an explanatory diagram of a nostril region extracting unit.

FIG. 15 is an explanatory diagram of a tilt detection unit, a rotation center setting unit, and a rotation unit according to the third embodiment.

FIG. 16 is a flowchart illustrating a flow of a process according to the fourth embodiment.

FIG. 17 is an explanatory diagram of a characteristic region centroid calculation unit.

FIG. 18 is an explanatory diagram of a tilt detection unit, a rotation center setting unit, and a rotation unit according to the fourth embodiment.

FIG. 19 is a flowchart showing a flow of processing according to the fifth embodiment.

FIG. 20 is an explanatory diagram of a nostril center calculating means.

FIG. 21 is a flowchart showing a flow of processing according to the sixth and seventh embodiments.

FIG. 22 is an image vertical direction histogram of an image.

FIG. 23 is an image horizontal axis histogram of an image.

FIG. 24 is a flowchart showing the flow of a process according to the eighth embodiment.

FIG. 25 is a diagram showing the arrangement of the centers of gravity of images.

FIG. 26 is a flowchart showing a flow of processing according to the ninth embodiment.

FIG. 27 is an explanatory view of a rotation angle setting means.

FIG. 28 is a diagram expressing a direction in which a face is inclined by coordinates.

FIG. 29 is a block diagram showing a configuration of a conventional example.

FIG. 30 is a flowchart showing a flow of setting an eyeball existing area in a horizontal direction in a conventional example.

FIG. 31 is a flowchart illustrating a flow of setting an eyeball existing region in a vertical direction in a conventional example.

FIG. 32 is a diagram showing a part of a flow of processing of an eyeball existing area in a conventional example.

FIG. 33 is a diagram showing a part of a flow of processing of an eyeball existing area in a conventional example.

FIG. 34 is a diagram showing a part of a flow of processing of an eyeball existing area in a conventional example.

[Explanation of symbols]

1 camera, 2 image memories, 3 CPUs, 4 face image input means, 5 feature candidate area extraction means, 6 binarization means, 7 face area centroid calculation means, 8 face partial area setting means, 9 partial area representative point calculation means , 10 tilt detection means, 11 rotation center setting means, 12 rotation means.

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-4-101280 (JP, A) JP-A-4-281578 (JP, A) JP-A 8-83344 (JP, A) Kenji Mase , Head Reader: Real-time Detection of Head Movement Using Images, IEICE Transactions D-II, March 25, 1991, Vol. J74-D-II, No. 3 pp. 398-406 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T ⁷ /00-7/60 B60K 25/00-28/16 G06T 1/00 G08G 1/00-9/02 G08B 19 / 00-21/24 JICST file (JOIS)

Claims (12)

(57) [Claims] 1. A hand for inputting a face image of a driver
Steps, eyebrows, eyes, nose, mouth, etc. in the input driver's face image
Feature candidate region extraction method for extracting candidate regions for facial feature points
A binarizing means for binarizing an image from which the feature candidate region is extracted;
Means for calculating the center of gravity of the face area to be calculated;
A face partial area setting means for setting, a partial area representative point calculation for calculating a representative point for the face partial area
Output means, the two points of the face area centroid (x, y) and the partial area representative point
The angle between the passing straight line and one of the image axes is detected as the inclination of the face.
Outgoing tilt detection means, For correcting the inclination of the face detected by the inclination detecting means
Rotation center setting means for setting the rotation center of the face image, Based on the rotation center set by the rotation center setting means
Rotation means for correcting the inclination of the face image by rotation, Determines whether the face image is tilted and determines whether the face image is tilted.
Image rotation processing by the rotation means until it is determined that
Continued inclination determination means, Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting an angle for rotating the image, A rotation limit angle setting means for setting a limit angle of the rotation angle
With steps, Rotating the face image by the rotation means and determining the inclination;
When it is determined by the means that the face image is not tilted,
Is the image by the rotation means until the rotation limit angle is reached.
Continue image rotation A face image processing apparatus characterized by the above-mentioned. 2. A face image input hand for inputting a driver's face image.
Steps, eyebrows, eyes, nose, mouth, etc. in the input driver's face image
Feature candidate region extraction method for extracting candidate regions for facial feature points
A binarizing means for binarizing an image from which the feature candidate region is extracted;
Means for calculating a face area center of gravity to be calculated, and a face area center of gravity calculated by the face area center of gravity calculation means
Based on the horizontal center of gravity x of (x, y), the upper region in the vertical direction of the face
A face partial area setting means for setting the center of gravity of the set upper area in the vertical direction of the face;
Means for calculating a partial area representative point as a table point; two points of the face area centroid (x, y) and the partial area representative point
The angle between the passing straight line and one of the image axes is detected as the inclination of the face.
Outgoing tilt detection means, For correcting the inclination of the face detected by the inclination detecting means
Rotation center setting means for setting the rotation center of the face image, Based on the rotation center set by the rotation center setting means
Rotation means for correcting the inclination of the face image by rotation, Determines whether the face image is tilted and determines whether the face image is tilted.
Image rotation processing by the rotation means until it is determined that
Continued inclination determination means, Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting an angle for rotating the image, A rotation limit angle setting means for setting a limit angle of the rotation angle
With steps, Rotating the face image by the rotation means and determining the inclination;
When it is determined by the means that the face image is not tilted,
Is the image by the rotation means until the rotation limit angle is reached.
Continue image rotation A face image processing apparatus characterized by the above-mentioned. 3. A face image input hand for inputting a driver's face image.
Steps, eyebrows, eyes, nose, mouth, etc. in the input driver's face image
Feature candidate region extraction method for extracting candidate regions for facial feature points
A binarizing means for binarizing an image from which the feature candidate region is extracted;
Means for calculating a face area center of gravity to be calculated, and a face area center of gravity calculated by the face area center of gravity calculation means
Based on the horizontal center of gravity x of (x, y), the upper region in the vertical direction of the face
Is set, and the upper part of the face in the vertical direction is set as the center of the face area (x,
y) to divide the upper right and left areas of the face
Face part area setting means to be set, center of gravity of the set upper left area and upper right face area
Is calculated, and a partial region representative point is calculated using the midpoint as a representative point.
Means, two points of the face area centroid (x, y) and the partial area representative point
The angle between the passing straight line and one of the image axes is detected as the inclination of the face.
Outgoing tilt detection means, For correcting the inclination of the face detected by the inclination detecting means
Rotation center setting means for setting the rotation center of the face image, Based on the rotation center set by the rotation center setting means
Rotation means for correcting the inclination of the face image by rotation, Determines whether the face image is tilted and determines whether the face image is tilted.
Image rotation processing by the rotation means until it is determined that
Continued inclination determination means, Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting an angle for rotating the image, A rotation limit angle setting means for setting a limit angle of the rotation angle
With steps, Rotating the face image by the rotation means and determining the inclination;
When it is determined by the means that the face image is not tilted,
Is the image by the rotation means until the rotation limit angle is reached.
Continue image rotation A face image processing apparatus characterized by the above-mentioned. 4. A face image input hand for inputting a driver's face image.
Steps, eyebrows, eyes, nose, mouth, etc. in the input driver's face image
Feature candidate region extraction method for extracting candidate regions for facial feature points
A binarizing means for binarizing an image from which a feature candidate region is extracted; a feature candidate for grouping feature candidate regions in the binary image;
Region grouping means, a nostril region for extracting a nostril region from the feature candidate region group
Extraction means, Nostril region weight for calculating the center of gravity for each nostril in the nostril region
Heart calculating means, a straight line passing through the center of gravity of each nostril of the calculated nostril region and
Tilt that detects the angle formed by one image axis as the tilt of the face
Detection means, For correcting the inclination of the face detected by the inclination detecting means
Rotation center setting means for setting the rotation center of the face image, Based on the rotation center set by the rotation center setting means
Rotation means for correcting the inclination of the face image by rotation, Determines whether the face image is tilted and determines whether the face image is tilted.
Image rotation processing by the rotation means until it is determined that
Continued inclination determination means, Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting an angle for rotating the image, A rotation limit angle setting means for setting a limit angle of the rotation angle
With steps, Rotating the face image by the rotation means and determining the inclination;
When it is determined by the means that the face image is not tilted,
Is the image by the rotation means until the rotation limit angle is reached.
Continue image rotation A face image processing apparatus characterized by the above-mentioned. 5. A face image input means for inputting a face image of a driver, a feature candidate area extraction for extracting candidate areas of face feature points such as eyebrows, eyes, nose and mouth in the input face image of the driver. Means, binarizing means for binarizing an image from which the feature candidate area has been extracted, feature candidate area grouping means for grouping feature candidate areas in the binary image, and extracting a nostril area from the feature candidate area group A nostril region extracting unit, a characteristic region centroid calculating unit that calculates a centroid for each characteristic candidate region group, and a centroid calculated by the characteristic region centroid calculating unit
An inter-nostril center setting means having a center of gravity between nostrils with respect to an area set by the nostril area extracting means; a straight line passing through two points of a center of gravity of each feature candidate area excluding the nostril area and a center between nostrils and one image A face image processing apparatus comprising: a tilt detection unit that detects an average of angles formed by axes as a tilt of a face. 6. A face image input hand for inputting a driver's face image.
Steps, eyebrows, eyes, nose, mouth, etc. in the input driver's face image
Feature candidate region extraction method for extracting candidate regions for facial feature points
A binarizing means for binarizing an image from which the feature candidate region is extracted;
Face area centroid calculating means for calculating, feature candidates for grouping feature candidate areas in a binary image
Region grouping means, a nostril region for extracting a nostril region from the feature candidate region group
Extraction means, between the nostrils to calculate the center between the nostrils in the extracted nostril area
Heart calculating means, a straight line passing through two points of the center between the nostrils and the center of gravity of the face region (x, y)
Detects the angle between a line and one of the image axes as face inclination
Tilt detection means, For correcting the inclination of the face detected by the inclination detecting means
Rotation center setting means for setting the rotation center of the face image, Based on the rotation center set by the rotation center setting means
Rotation means for correcting the inclination of the face image by rotation, Determines whether the face image is tilted and determines whether the face image is tilted.
Image rotation processing by the rotation means until it is determined that
Continued inclination determination means, Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting an angle for rotating the image, A rotation limit angle setting means for setting a limit angle of the rotation angle
With steps, Rotating the face image by the rotation means and determining the inclination;
When it is determined by the means that the face image is not tilted,
Is the image by the rotation means until the rotation limit angle is reached.
Continue image rotation A face image processing apparatus characterized by the above-mentioned. 7. The face of the face detected by the tilt detecting means.
Rotation center to set rotation center of face image for tilt correction
Setting means, based on the rotation center set by the rotation center setting means.
Rotation means for correcting the inclination of the face image by rotation, Determines whether the face image is tilted and determines whether the face image is tilted.
Image rotation processing by the rotation means until it is determined that
Continued inclination determination means, Depending on the angle θ detected by the inclination detecting means, the face
Rotation angle setting means for setting an angle for rotating the image, A rotation limit angle setting means for setting a limit angle of the rotation angle
With steps, Rotating the face image by the rotation means and determining the inclination;
When it is determined by the means that the face image is not tilted,
Is the image by the rotation means until the rotation limit angle is reached.
6. The image rotation processing according to claim 5, wherein the image rotation processing is continued.
 Face image processing device. 8. The face image processing apparatus according to claim 7, wherein the rotation center setting unit sets the center of gravity (x, y) of the face area in the binary image as the rotation center. 9. The face image processing apparatus according to claim 7, wherein the rotation center setting means sets a center between nostrils as a rotation center. 10. The image forming apparatus according to claim 1, wherein said inclination determining means includes:
Face horizontal direction is image vertical (y) axis, Face vertical direction is image horizontal (x) axis
In the image vertical axis direction at each coordinate in the image horizontal axis direction.
Creates a vertical histogram of the image indicating the number of black pixels
The shape and peak of the histogram
The feature is to determine whether the face image is tilted from the number of clicks
The face image processing device according to claim 1 . 11. The apparatus according to claim 1, wherein said inclination determining means includes:
Face horizontal direction is image vertical (y) axis, Face vertical direction is image horizontal (x) axis
And exists in the horizontal direction of the image at each coordinate on the vertical axis of the image
Creates a histogram along the horizontal axis indicating the number of black pixels
And the face image is determined from the symmetry of the histogram
Determining whether or not is tilted.
The face image processing device according to any one of claims 1 to 9 . 12. The image forming apparatus according to claim 1, wherein said inclination determining means includes:
Each feature candidate area obtained by the feature candidate area centroid calculation means
Center of gravity in the horizontal direction of the center of gravity (xi, yi) for the region group
Determining whether or not a face image is tilted from the row of xi
The face image processing device according to any one of claims 1 to 9, wherein