JP3451832B2 - Face image feature recognition device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION [0001] 1. Field of the Invention [0002] The present invention relates to a face image feature quantity recognition apparatus for forming a face feature quantity by performing arithmetic processing on face image data. 2. Description of the Related Art Face image data is subjected to arithmetic processing to extract facial features, thereby identifying a person, detecting the direction of a line of sight, and determining whether to fall asleep or asleep. Various devices have been developed. Since image data is a huge amount of data expressing the brightness of each pixel in a number of gradation levels (density values, luminances), such a device narrows down the image data early to reduce the number of data, The processing speed is secured by saving the number of operations. For example, Japanese Patent Application Laid-Open No. 7-1
In the dozing alarm device disclosed in Japanese Patent Application Laid-Open No. 81012, an eye opening index, which is one of the facial features, is obtained from the face image data. The number of data is reduced by converting to a value, setting a black mask area in an unnecessary portion of the screen to reduce the number of black portions, and treating each island-shaped black portion as a lump and performing labeling. [0003] Japanese Patent Application Laid-Open No. 7-18101
In the processing disclosed in Japanese Patent Laid-Open Publication No. 2 (1993) -1995, the following three steps need to be performed in order to detect the position of the eye, so that considerable time is required for image processing and calculation. (1) Pre-processing of image data mainly on binarization of density values and noise removal (2) Face width detection (labeling, particle removal, continuity of white pixels) for specifying the horizontal position of eyes (Determination etc.) (3) Vertical position detection of eyes (judgment of barycentric coordinates, area value and fillet diameter of each label etc.) In other words, although density values are binarized, individual island-shaped black Since each part includes several hundreds to several tens of thousands of pixels, each of these processes takes a considerable amount of time to calculate. Therefore, it is difficult to shorten the processing cycle to increase the follow-up performance and reliability of the determination, and it is also difficult to reduce the size of the arithmetic unit and interrupt processing of other arithmetic processing. In addition, since labeling is performed in a state where the black portions on the face are individually separated, when a black portion corresponding to the eye is connected to another black portion, for example, when long hair is applied to the outer corner of the eye, On the black and white binarized image, the black portion corresponding to the eye is integrated with other black portions, and the black portion corresponding to the eye cannot be specified. The present invention provides an apparatus for recognizing a feature quantity of a face image, which can reduce the amount of data to be calculated, increase the processing speed, and can accurately specify the feature quantity corresponding to the eyes even when the long hair touches the outside of the eye. It is intended to provide. [0006] Means for Solving the Problems The present invention provides a feature value recognition device of the face image to extract a feature amount of the face by processing image data of a face, in the longitudinal direction of the pixel rows of the face A pixel density is detected along the pixel row, one pixel is determined for each local increase in density in the pixel row, and an extraction point is formed. The extraction points adjacent to each other in the pixel row direction of the adjacent pixel row are connected. A group of curves extending in the horizontal direction of the face is extracted. Namely, have a storage means for storing an image input means for digitizing the concentrations for each pixel on the screen by picking up the face, the digitized density value for each said pixel, the vertical direction of the face The density value is fetched from the storage means along a pixel row in the (Y direction), and a pixel representing a one-way peak of the density value on the pixel row in the vertical direction (Y direction) is specified and extracted. provided with extraction means for, by comparing the extracted points in the vertical direction (Y-direction) pixel column adjacent, and formation means for grouping into a single extraction point adjacent to the pixel column direction (X direction)
You . [0008] extraction means obtains the arithmetic mean value as the density value of one pixel by averaging the density values of a plurality of pixels continuous comprise one pixel, prior to averaged arithmetic mean density value
The routine fine <br/> content value for the position along the pixel columns of Kitate direction (Y direction) and extraction point pixels reversed in one direction
And repeat the routine to create a dark peak.
Only the Y coordinate value of the extraction point corresponding to the mark is stored. [0009] The forming means is provided with an image in a vertical direction (Y direction).
Extraction in which the vertical difference between the extraction points obtained for each
Connect the output points in the horizontal direction (X direction) to group
Group numbers in order from the pixel row at the end in the horizontal direction.
Check if there is a numbered pixel,
If present, the group number, its vertical and horizontal coordinates, and the end
It remembers how many times it is from the beginning. [0010] Further, the forming means may further comprise :
Check if a common group exists between columns and
If it exists, the extraction point
The number and the Y coordinate value are stored in the memory as linked data,
The number of linked extraction points in the linked data exceeds a specified value
Select only the group, the average value of the ordinate and the left of the group
The position of the candidate point is determined from the abscissa value at the center of the pixel row at the right
To set . In the facial image feature quantity recognition apparatus of the present invention , an image
The density value of each pixel of the face is stored in the storage
The stage retrieves the required pixel array from the storage means and selects the extraction point.
Ask. The extraction point is the density value station such as eye, eyebrow, mouth, mole, etc.
For localized heightening (local dent in reverse image)
It is specified one by one. Then, connect the extraction points in the horizontal direction.
Moles (dots) and nose streaks (vertical lines)
Extraction points that are not continuous in the horizontal direction, such as
You. More specifically, the local increase of the density value
The center point is selected as the extraction point. Pixel by averaging
Small changes in density values along the column are smoothed to
Changes remain. Differential calculation of large changes in density value
Are identified. If the derivative is in one direction (positive to negative or
Pixels (positions) that reverse from negative to positive)
In the center of the eyes, eyebrows, mouth, etc.
Respond. The pixels whose differential values are reversed in the opposite direction are the eyes, eyebrows, and mouth.
Corresponding to the brightness peak located in the middle of the dark part
And is discarded without being considered an extraction point. [0013] In the forming means, adjacent pixel columns are common to each other.
If the group of exists, concatenate as consolidated data,
Only the groups where the number of connected extraction points exceeds the specified value
, The continuous length corresponding to the nostrils etc. is short.
Group is shaken down and beside the face like eyes, eyebrows and mouth
Only groups that are long and continuous in the direction remain. Then, for each extracted point group forming the group, the average value of the ordinate and the abscissa value corresponding to the center of the pixel row at the start of connection and the pixel row at the end of the connection are obtained. Are represented by one candidate point on the screen. From the relative positional relationship of the candidate points on the screen, it can be determined whether each group corresponds to the eye, eyebrows, or mouth. Thus, the facial image feature quantity recognition apparatus of the present invention is provided.
Of the face image by multiple curves that cross the face in the horizontal direction.
The charge is expressed. Each curve represents a bright part of the image
Represents the dark part that traverses in the horizontal direction, such as eyes, eyebrows, mouth
And so on. Nose shadows, forehead hair, cheek lines
Dark areas that traverse the face image in the vertical direction, such as
Local peaks are hardly formed, and if peaks are formed
Other extraction points connected in the horizontal direction
It is shaken off because it does not exist. On the lines and cheeks on both sides of the face
Similarly, it is difficult for the hair to become an extraction point,
Since there is no connected extraction point, it is shaken off. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Referring to FIGS. 1 to 13, an embodiment applied to driver's seat control will be described. In the present embodiment, a plurality of features such as eyes, eyebrows, and mouth are extracted from the image of the driver's face, and the feature of one eye is specified from these features. FIG.
As shown in FIG. 1, a camera 11 using a CCD image element is installed on an instrument panel of an automobile, and a driver's face during driving is continuously imaged in a certain direction and at a certain angle. An image of one frame every 0.1 second among the images captured by the camera 11 is sent to the AD converter 12. The AD converter 12 converts an analog voltage for each pixel into a digital value of 256 levels and converts the analog voltage into an image memory 13.
To accumulate. As shown in FIG. 6, the image memory 13 stores and holds a density value of 8 bits for each of 480 pixels vertically and 512 pixels horizontally. Image data operation circuit 1
Reference numeral 4 extracts the feature amount of the face using the image of one frame stored in the image memory 13. Facial feature recognition circuit 15
Specifies the feature value of one eye from the extracted feature values of the plurality of faces. The mirror / sheet control circuit 16 identifies the eye height from the specified characteristic amount of one eye, and adjusts the angle of the mirror and the height of the sheet. The image processing in the embodiment is executed according to the flowchart of FIG. In step 110, an image of the vehicle interior including the driver's face is captured by the camera 11, and the AD converter 12 converts the pixel density into a digital value. In step 120, image data corresponding to one frame image is stored in the image memory 13.
In step 130, a vertical pixel row of the face is selected from the huge amount of image data stored in the image memory 13 as shown in FIG. Several tens at equal intervals in the center of the screen
One pixel row is set for every 10 pixels in the row and X direction. One pixel row has 480 pixels in the Y direction, and the density values of all pixels can be sent to the image data calculation circuit 14 for each pixel row. In step 140, the density value of each pixel row is arithmetically processed to specify a pixel (Y coordinate location) to be an extraction point. A large change in the density value on the pixel row is obtained, and a local peak thereof is set as an extraction point. FIG. 3 shows the processing procedure. In step 141, the density values of all the pixels in each pixel column are called in order from the left side of FIG. In step 142, the arithmetic values of the density values on the pixel row are arithmetically averaged, and an arithmetic average density value obtained by removing high frequency components from a density value change curve on the pixel row is obtained. For example, the pixel column Xa shown in FIG.
8A, and an arithmetic average density value as shown in FIG. 9A is extracted from the pixel row Xb. Step 14
In 3, the arithmetic mean density value is differentiated. In the arithmetic mean density value shown in FIG. 8A, the differential value shown in FIG.
In the arithmetic mean density value shown in (a), the differential value shown in (b) is obtained. In step 144, the Y coordinate value corresponding to the dark peak of the arithmetic average density value is obtained from the differential value.
The Y coordinate value at which the differential value is inverted from negative to positive becomes one Y coordinate value representing each of the eyes, eyebrows, mouth, etc., corresponding to the peak of darkness. The Y coordinate value at which the differential value is inverted from positive to negative is a peak position of brightness corresponding to an intermediate position such as an eye or an eyebrow, and is not used in the embodiment. In the arithmetic mean density value shown in FIG. 8A, the differential value shown in FIG. 8B is obtained, and the Y coordinate values p1, p2, p3, and p at which the differential value is inverted from negative to positive.
4. p5 is extracted. Among these, the Y coordinate values p1 and p1 in which the peak of the preceding differential value is equal to or less than the threshold value (broken line)
2, p5 is the extraction point. By determining the peak of the differential value at the near side, wrinkles and irregularities on the face do not become extraction points. In the arithmetic mean density value shown in FIG. 9A, Y coordinate values p1, p2, and p3 in which the differential value shown in FIG. 9B is inverted from negative to positive are extracted, but all are extracted from the preceding differential value. The peak does not fall below the threshold (broken line), and no extraction point is formed. In step 145, the last pixel row, FIG.
It is determined whether or not the pixel row is at the right end. If it is the last pixel row, the extraction point has been specified in all the pixel rows, so the process proceeds to step 150. If it is not the last pixel row, the process proceeds to step 141, and the extraction point is started to be specified by the density value data of the next pixel row. In this way, as shown in FIG. 10, about 0 to 4 extraction points A1,
A2, A3, and A4 are specified. In the pixel row Xc, the extraction points A1, A2, and in the pixel row Xd, the extraction points A1, A2, A
3. A4 is specified. In step 150, the extracted points obtained for each pixel column are connected in the X direction and grouped. FIG. 4 shows the processing procedure. In step 151, the extraction point data of the pixel row is called in order. In steps 152 and 153, the Y coordinate values are compared for the extraction points of the adjacent pixel rows.
If the difference between the Y coordinate values of the extraction points of the adjacent pixel columns is 10 pixels or less, the process proceeds to step 154, where connected data is formed. If it is determined that the difference exceeds the predetermined range (grouping cannot be performed), the process proceeds to step 155, and no linked data is formed. The linked data is classified by the group number sequentially assigned to each generated group and the number of the pixel column (corresponding to the X coordinate value) to which the extracted point at the left end of the group belongs, and the number of linked extracted points (pixels) (The number of columns) and the Y coordinate value are stored in a memory, and each time a pixel column is processed, the stored contents are rewritten to form. In step 155, it is determined whether the pixel row is the last pixel row. If it is the last pixel column, the grouping has been determined for all the pixel columns, and the process proceeds to step 160. If it is not the last pixel row, step 1
Proceeding to 51, determination of grouping based on the extraction point of the next pixel row is started. The processing in step 150 will be briefly described with reference to FIG. Here, the description is made on the assumption that the number of pixel columns is 11. As shown in FIG. 11, the extraction points specified in the pixel columns 1 to 11 are group numbers G,
The connection data is formed by adding the connection numbers N and Y coordinate values. The 34 extraction points shown in FIG. 11 are grouped into 10 groups. In step 160, the image data arithmetic circuit 14 shown in FIG. 1 removes unnecessary connection data according to the procedure shown in FIG. 5, and finally forms candidate point data only for the necessary connection data. In step 161, the connected data is called one by one from the memory in order of the group number. In steps 162 and 163, the number of linked data of the linked data is evaluated. If the number of connected data is 5 or more, the process proceeds to the next step 164 to form candidate point data. If the number of connected data is less than 5 (nostrils, etc.), the process proceeds to step 165, and no candidate point data is formed. For example, from the extraction point group shown in FIG. 10, as shown in FIG. 12, groups G1 to G6 of six extraction points corresponding to both eyes and eyebrows, a shadow of a nose, and a mouth are left. Here, the remaining six groups are renumbered in order from left to right and from top to bottom. The candidate point data represents the group of the remaining extraction points by one candidate point on the screen.
A set of coordinate values, that is, an average value (height) of Y coordinate values of the extraction points belonging to the group, and an X coordinate value at the center of the pixel rows at both left and right ends of the group. For example, FIG.
Are represented by six candidate points K1 to K6 on the screen shown in FIG. In step 165, it is determined whether or not the data is the last linked data. If it is the last linked data, the process proceeds to step 170 because all the linked data have been evaluated. If it is not the last linked data, the process proceeds to step 161 to start evaluation of the next linked data. In step 170, the facial feature quantity recognition circuit 15 shown in FIG. 1 evaluates the candidate point data and specifies one eye candidate point from a plurality of candidate points. For example, FIG.
In the case of six candidate points K1 to K6 shown in FIG. 3, a central detection zone ZC is formed around the candidate points K5 and K6 having large (low) Y coordinate values. Outside the detection zone ZC, Y
Candidate points K1, K3 with small (high) coordinate values and candidate point K
2, K4 as the center, left and right detection zones ZL,
Set ZR. Then, it corresponds to one eye based on the height difference between the two candidate points K1 and K3 included in the left detection zone ZL and the candidate point K6 included in the central detection zone ZC, and the positional relationship between the candidate points K1 and K3. Is specified. In step 180, the mirror and seat control circuit 16 shown in FIG. 1 adjusts the mirror angle and the seat height in conformity with the specified characteristic amount of one eye, that is, the height of the driver's eye. According to the image processing of the embodiment, the density value of the pixel is called for each pixel column, the same arithmetic processing is repeated in one direction from the left side to the right side of the screen, and the limited size of each pixel column is obtained. Japanese Patent Laid-Open No. 7-18 / 1994
The processing program becomes simpler than the processing disclosed in Japanese Patent No. 1012. Then, even if the number of pixel rows increases, the number of repetitions of the same processing only increases, and the processing itself does not change. In addition, the process of forming a connected group from the calculated extraction points, narrowing down the number of connected groups, forming small data called candidate point data, and identifying the feature amount of the eye,
Even smaller data and simple calculations / comparisons are sufficient. In addition, it is necessary to perform any of the pre-processing of image data mainly on binarization of density values and noise removal, face width detection for specifying a horizontal position of an eye, and vertical position detection of an eye. There is no. Therefore, the number of operations and the memory capacity required for image processing are small, and the performance of the processing element may be slightly lower. It is also possible to do. Further, since a pixel row is selected in the vertical direction of the face, and the Y coordinate value which is a local peak of darkness is used as an extraction point, the hairline, both ends of the face, the hair vertically applied to the face, etc. Is unlikely to be an extraction point. Even if it is set as an extraction point, extraction points that do not satisfy the condition that there are extraction points connected in the horizontal direction and the number of connected points is 5 or more are automatically excluded. Unwanted candidate points are not formed due to wrinkles, irregularities, shadows due to direct sunlight, and the like, and there is no fear of erroneous eye identification. Therefore, even when the eye is covered with hair, at least the connection data of the eye line and the candidate point data up to the point where the hair is covered are formed without any problem, and the eye opening index is calculated in the same procedure as when the hair is not covered. Can be calculated. In the flowchart of FIG. 2, step 110 is the image input means of the invention, step 120 is the storage means of the invention, steps 130 and 140 are the extraction means of the invention,
Steps 150 and 160 correspond to the forming means of the present invention, respectively. Further, in the embodiment, the use of adjusting the seat height or the like based on the detection height of the eyes has been described. However, the present invention is directed to specifying a person, detecting a direction of a line of sight,
As shown in JP-A-181012, the present invention can also be applied to a use for determining the distinction between dozing / sleep. In the embodiment, processing is performed on an image in which the eyes, eyebrows, and the like have a high density value (dark gradation level). However, an image in which the gradation is inverted (the eyes, eyebrows, and the like have low density values). May be used. In the embodiment, all the processes after the AD converter 12 are performed by digital operation. However, the processes up to obtaining the arithmetic mean of the density values along the pixel row are performed by the analog signal circuit. It may be. For example, after setting scanning lines in the vertical direction with a camera using an image pickup tube, passing a luminance signal along a scanning line that crosses the face in the vertical direction through a low-pass filter to remove high-frequency components, and then removing every 10 scanning lines Then, a luminance signal for one scanning line is extracted and A / D converted by sampling. Further, in the embodiment, one pixel row is selected for every 10 pixels in the X direction as shown in FIG. 7, but a plurality of pixel rows may be selected in finer increments. Even if the number of pixel rows increases, only the number of repetitions of the same processing increases, but the basic processing content does not change. By increasing the number of pixel rows, the curved state of the curve of the feature amount can be more accurately captured, and a smaller allowable connection width can be set to avoid incorrect connection of the extraction points. After the detection zone ZL is set as shown in FIG. 13, pixel rows are set with high density only in the detection zone ZL,
The number of extraction points forming the curve of the left eye may be increased to detect the bow shape in more detail. When the driver wears glasses as shown in FIG. 14, the detection zone ZL shown in FIG. 13 includes a group G3 of extraction points corresponding to eyes and eyebrows as shown in FIG. , G5, extraction point groups G1, G7 corresponding to the upper and lower edges of the glasses remain. In FIG. 15, the remaining nine groups are renumbered in order from left to right and from top to bottom, and the relationship between the eyes, eyebrows, and symbols does not match that in FIG. However, even in such a case, in the present embodiment, as described above, the detection zone ZL
Since the height difference between the plurality of candidate points in the detection zone ZL and the candidate point of the mouth in the detection zone ZC is evaluated in addition to the vertical relationship of the candidate points in the above, the group G5 corresponding to the eye can be specified without error. When four candidate points are formed in the detection zone ZL, the second candidate point from the bottom is determined as an eye, and candidate point data corresponding to the edge of the glasses is shaken off, and the driver does not wear glasses. The feature amount of the eye is specified in the same procedure as in the case. According to the present invention, for example, since the shape of the arch of the eye can be specified by the same number of coordinate values as the number of pixel rows related to the eye, all pixels included in the image of the eye are handled. The number of data is greatly reduced as compared with. Compared to the process of calculating the center of gravity by treating all the pixels included in the eye image, the process of specifying the eye curve from the plurality of curves is simpler. Therefore, even with a small arithmetic device having a low arithmetic speed and a small memory allocation, image processing for obtaining a face feature can be executed at high speed. [0034] Then, and the peak of the density values appearing in the lateral direction of the face, because the peak of the laterally linked not the density value of the face is eliminated by hair according to shadows and eye area across the face in the vertical direction, the eye etc. This does not hinder the specification of the required feature amount. In the vertical pixel row, continuous multiple
Whether to average the density values of several pixels into one density value
Therefore, there is no need to worry about grouping wrong extraction points in adjacent pixel columns. In addition, since the peak of the density value formed by the averaging is used as an extraction point, a curve of a feature amount that accurately follows the shape of an eye or an eyebrow can be formed, and an error in obtaining an eye opening index or the like is suppressed. Is done. In the selection of a group, a group of extraction points having a small number of connections is shaken off. Therefore, it is sufficient to specify the feature amount of the eyes and the mouth in the extraction points and the groups narrowed to the minimum necessary. It is not necessary to select a feature amount. [0037] Then, the group narrowed from be represented by one of the candidate points, respectively, can be efficiently perform the determination of the relative position as compared to when working curve of the feature as it is.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an explanatory diagram of an alarm device according to an embodiment. FIG. 2 is a flowchart of an operation of the alarm device. FIG. 3 is a flowchart of a process for obtaining an extraction point. FIG. 4 is a flowchart of a process for forming linked data. FIG. 5 is a flowchart of a process of selecting a candidate point. FIG. 6 is an explanatory diagram of a face image. FIG. 7 is an explanatory diagram of setting of a pixel column. FIG. 8 is an explanatory diagram of density values along a pixel column. FIG. 9 is an explanatory diagram of density values along a pixel column. FIG. 10 is an explanatory diagram of a distribution of extraction points. FIG. 11 is an explanatory diagram of a process of grouping extraction points. FIG. 12 is an explanatory diagram of a distribution of a group of extraction points. FIG. 13 is an explanatory diagram of a distribution of candidate points. FIG. 14 is an explanatory diagram of a face image. FIG. 15 is an explanatory diagram of a distribution of extraction point groups. [Description of Signs] 11 Camera 12 A / D converter 13 Image memory 14 Image data calculation circuit 15 Face feature recognition circuit 16 Mirror and sheet control circuit

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-4-1863 (JP, A) JP-A-59-140589 (JP, A) JP-A-7-311833 (JP, A) , Driver's blink measurement in a car driving environment, IEICE Transactions C, November 20, 1995, Vol. 115-C, no. 12, pp. 1411-1416 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T ^7/ 00-7/60 G06T 1/00 H04N 7/18

Claims (1)

(57) [Claims] [Claim 1] An image of a face is taken and the density is determined for each pixel on the screen.
Image input means for digitizing, and storage means for storing a density value digitized for each pixel
In the facial image feature quantity recognition apparatus having the following expression, the density value is set in advance along the pixel row in the vertical direction (Y direction) of the face.
The pixel in the vertical direction (Y direction)
The pixel that represents the one-way peak of the density value on the column is identified.
By comparing the extraction means that are defined as extraction points with extraction points of adjacent vertical (Y-direction) pixel columns,
Extraction points close to the pixel column direction (X direction) are grouped into one.
Forming means for converting the image into a plurality of continuous images including one pixel.
Average the elementary density values to obtain the density value of one pixel.
The average value is calculated, and the averaged arithmetic average density value in the vertical direction is calculated.
The differential value at a position along the pixel row in the (Y direction) is one.
Create a routine that uses pixels that are reversed in the direction as extraction points
And repeat the routine to deal with dark peaks
Only the Y coordinate values of the extracted points are stored, and the forming means obtains the values for each pixel row in the vertical direction (Y direction).
Extraction points whose difference in the vertical direction of the extraction points is less than the specified range
(X direction) connected and grouped and numbered
Group numbers are assigned sequentially from the pixel row at the horizontal end.
Check if there is a pixel
Is the group number, its vertical and horizontal coordinates, and what
It remembers whether it was the first appearance, and shared between adjacent pixel rows.
Check if a group exists and if so
In the case, the number of connected extraction points and Y
The standard value is stored in a memory as linked data, and the linked data
Is a group where the number of connected extraction points exceeds
And select the average value of the ordinate and the left and right ends of the group.
An apparatus for recognizing a feature amount of a face image, wherein a position of a candidate point is set from an abscissa value at a center of a pixel row .