JP4825473B2 - Face orientation discrimination device - Google Patents

Description

  The present invention relates to a face orientation discriminating apparatus capable of discriminating a face orientation (lateral angle) from a face image and discriminating its reliability (accuracy).

  In order to identify a person or read a facial expression based on a face image, it is important to detect the face position, the face center position, the orientation, and the like.

  For example, Patent Document 1 discloses a technique for processing an image obtained by photographing a human face and detecting the position of the face with high accuracy without being affected by the movement or background of the person.

Further, Patent Document 2 captures an image obtained by photographing an area where a driver can exist, detects a lateral edge image (both side edges of the face) from the captured image, and based on the detected lateral edge image. In addition to setting a plurality of face center candidate lines, for each face center candidate line, the accuracy of determining the presence or absence of a face is improved based on a value representing the likelihood that the face center candidate line is the face center line. Techniques to do this are disclosed.
JP 2004-310396 A JP 2005-011097 A

The driving environment of the vehicle changes variously, and the driver's image captured by the camera also changes variously. For example, at a certain timing, the face is exposed to direct sunlight and the entire image becomes bright, and as the direction of the vehicle changes, the surface facing the camera becomes shaded and the entire image becomes dark or A situation occurs such that the part is bright and the rest is dark. For this reason, it is difficult to detect the positions of both sides of the face, and depending on the environment and conditions at the time of image acquisition, for example, how the light strikes (direction of light, intensity of outside light), etc. Detection may be wrong.
If an error occurs in the detection of these pieces of information, there will be a shift in subsequent determinations such as determination of the lateral angle.
In such a case, it is effective if an index is given to the accuracy of the detection position depending on the environment and conditions, and it can be used for the subsequent processing.

  The present invention has been made in view of the above problems, and an object of the present invention is to make it possible to determine the reliability when determining the orientation of a face.

In order to achieve the above object, a face orientation determination device according to a first aspect of the present invention is stored in a face image storage means for storing a face image obtained by photographing a face, and the face image storage means. Based on the face image, a face position determining means for determining a face position and a center position of the face, and a face position based on the face position and the face center position obtained by the face position determining means. A face orientation determining means for outputting the orientation, a reliability determining means for obtaining a reliability indicating the certainty of the face orientation determined by the face orientation determining means, and a face orientation determined by the face orientation determining means If, Bei example and output means for outputting in association with the confidence that has been judged by the reliability judging means,
The reliability determination unit obtains the reliability of the face orientation based on the magnitude of the deviation between the previously determined face position and the currently detected face position.

In order to achieve the above object, a face orientation determination device according to the second aspect of the present invention is provided:
  Face image storage means for storing a face image obtained by photographing a face;
  Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
  A face orientation determining means for outputting the orientation of the face based on the face position and the center position of the face determined by the face position determining means;
  Reliability determination means for obtaining a reliability indicating the certainty of the face orientation determined by the face orientation determination means;
  An output means for outputting the orientation of the face determined by the face orientation determining means and the reliability determined by the reliability determining means;
  With
  The reliability determination unit divides the face image into a plurality of regions, obtains the center of gravity of each region, and obtains the reliability of the face orientation based on the obtained center of gravity position.

In order to achieve the above object, a face orientation determination device according to a third aspect of the present invention is:
Face image storage means for storing a face image obtained by photographing a face;
  Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
  A face orientation determining means for outputting the orientation of the face based on the face position and the center position of the face determined by the face position determining means;
  Reliability determination means for obtaining a reliability indicating the certainty of the face orientation determined by the face orientation determination means;
  An output means for outputting the orientation of the face determined by the face orientation determining means and the reliability determined by the reliability determining means;
  With
  The reliability determination unit divides the face image into a plurality of regions, calculates the luminance of each region, and calculates the reliability of the face position based on the calculated luminance.

The face position determination unit, the face orientation determination unit, and the reliability determination unit are, for example, a process of determining a horizontal position and orientation of a face based on the face image stored in the face image storage unit And repeatedly execute a unit of processing for determining the reliability of the determined horizontal direction,
  By dividing the process of determining the position of the face in the vertical direction into a plurality of processing steps, executing one of the dividing steps within each unit of processing, and repeating the execution of the dividing step a plurality of times, Find the vertical position.

In order to achieve the above object, a computer program according to the fourth aspect of the present invention provides:
Computer
  Face image storage means for storing a face image obtained by photographing a face;
  Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
  Face orientation determining means for outputting the orientation of the face based on the position of the face determined by the face position determining means and the position of the center of the face;
  Reliability determination means for obtaining reliability indicating the certainty of the face orientation determined by the face orientation determination means;
  An output means for outputting the face orientation determined by the face orientation determination means and the reliability determined by the reliability determination means in association with each other;
  A computer program that functions as
The computer causes the reliability determination means to execute a process for determining the reliability of the face orientation based on the magnitude of the deviation between the previously determined face position and the currently detected face position.

  In order to achieve the above object, a computer program according to the fifth aspect of the present invention provides:
  Computer
  Face image storage means for storing a face image obtained by photographing a face;
  Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
  Face orientation determining means for outputting the orientation of the face based on the position of the face determined by the face position determining means and the position of the center of the face;
  Reliability determination means for obtaining reliability indicating the certainty of the face orientation determined by the face orientation determination means;
  An output means for outputting the face orientation determined by the face orientation determination means and the reliability determined by the reliability determination means in association with each other;
  A computer program that functions as
  The computer causes the face image to be divided into a plurality of areas as the reliability determination means, obtains a center of gravity of each area, and executes a process of obtaining a face orientation reliability based on the obtained center of gravity position.

In order to achieve the above object, a computer program according to a sixth aspect of the present invention is stored in a face image storage means for storing a face image obtained by photographing a face of a computer, the face image storage means. Face position determining means for determining the position of the face and the position of the center of the face based on the face image, the direction of the face based on the position of the face and the position of the center of the face obtained by the face position determining means For the face orientation determined by the face orientation determining means, a reliability determining means for obtaining a reliability indicating the certainty of the face orientation determined by the face orientation determining means, the face orientation determined by the face orientation determining means, A computer program that functions as an output means that outputs the reliability determined in association with the reliability determined by the reliability determination means ,
The computer causes the face image to be divided into a plurality of areas as the reliability determination means, obtains the brightness of each area, and executes a process of obtaining the reliability of the face position based on the obtained brightness.

  According to the present invention, it is possible to determine the face orientation from the face image and to determine its reliability.

  Hereinafter, a face orientation determination device according to an embodiment of the present invention will be described.

  As shown in FIG. 1, the face orientation determination device of the present embodiment captures a driver's face and generates a face image, a camera 10, an illumination light source 12 that illuminates the driver's face, and the driver's face center position. The computer 14 to detect and the display apparatus 16 connected to the computer 14 are provided.

  The camera 10 is composed of a CCD camera, for example, and acquires a gradation image of the driver's face. The face image generated by the camera 10 includes not only the driver's face but also its background.

  The display device 16 includes an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), or the like, and displays a binarized image extracted from a face image taken by the camera 10.

  The computer 14 processes the face image acquired by the camera 10 to detect the positions of both ends of the face, obtains the lateral accuracy of the face based on the detected both ends, and further detects the detection positions of the both ends of the face. This is a device for obtaining the reliability of the detected lateral angle by digitizing the deviation. As shown in FIG. 2, the computer 14 includes an A / D converter 21, an image memory 22, a ROM 23, a CPU 24, a RAM 25, a display control device 27, and a light source control device 28.

An A / D (analog / digital) converter 21 converts an analog image signal photographed by the camera 10 into a digital signal.
The image memory 22 stores image data generated by the camera 10 and digitized by the A / D converter 21.

  The ROM 23 stores a program for controlling the operation of the CPU. Further, the ROM 23 stores various fixed data for executing image processing to be described later.

  The CPU 24 executes the program stored in the ROM 23 to process the face image acquired by the camera 10 to detect the positions of both ends of the face, and based on the detected both ends positions, the lateral angle of the face Is obtained, and the reliability of the detected direction (lateral angle) is obtained by digitizing the deviation of the detection positions at both ends of the face.

The RAM 25 functions as a work rear for the CPU 24.
The display control device 27 controls the display device 16 under the control of the CPU 24.
The light source control device 28 controls turning on / off of the illumination light source 12.

  Next, an example of fixed data stored in the ROM 23 will be described with reference to FIG. First, the ROM 23 stores operators of vertical edge detection and horizontal edge detection Sobel filters as shown in FIGS. The vertical edge detection Sobel filter and the horizontal edge detection Sobel filter shown in FIGS. 3 (a) and 3 (b) are respectively shades in the vertical direction as shown in FIGS. 3 (c) and 3 (d). This is an operator for emphasizing the difference and the contrast in the horizontal direction.

Further, as shown in FIG. 3E, the ROM 23 stores data for specifying a position where it is assumed that an image of an eye or an eyebrow exists among the face images stored in the RAM 25.
Further, as shown in FIG. 3F, the ROM 23 stores the vertical and horizontal sizes of the sunglasses in order to detect the sunglasses.
The ROM 23 stores a standard eye image as shown in FIG.

Next, the operation of the face orientation determination apparatus having the above configuration will be described.
When the power is turned on, the CPU 24 in the computer 14 repeatedly executes the process shown in FIG.

  That is, the CPU 24 executes an initial process including a preprocess (step S01) and a face position determination (detection) process (step S02), and then repeatedly executes the first process to the nth process. To do. n is a natural number of 2 or more.

  The pre-processing (step S01) and the face position determination process (step S02) constituting the initial process are processes for obtaining initial values of the driver's left and right and vertical face positions.

  The first process to the n-th process are repeatedly executed while the power is turned on, and the position of the driver's face in the left-right direction is obtained and output together with the reliability. In the first process to the n-th process, the face position in the vertical direction of the driver is obtained once as a whole. As a result, in the steady operation state, the vertical position of the driver's face is obtained once while the horizontal position of the driver's face is obtained n times. In general, since the vertical movement of a human face is small compared to the horizontal movement, it is possible to achieve both the detection accuracy of the face position and the processing efficiency by adopting such a detection method. it can.

  As shown in FIG. 5, the pre-process S01 that constitutes the initial process includes a capture process (step S011), a coordinate conversion process (step S012), and a Sobel filter process (step S013).

  The capture process (step S011) is a process of capturing a face image of a one-frame sentence of the driver imaged by the camera 10 via the A / D converter 21 and storing it in the image memory 22.

  The coordinate conversion process (step S012) is a process of thinning out pixels to the extent that they can be processed.

  The Sobel filter processing (step S013) processes the face image after coordinate conversion using the vertical edge detection Sobel filter (FIG. 3A) stored in the ROM 23, and the vertical edge in the face image is processed. And a process of processing the face image after the coordinate conversion using the horizontal edge detection Sobel filter (FIG. 3B) and enhancing the horizontal edge in the face image. It is.

  The face position determination process (step S02) in FIG. 4 is a process for detecting both end positions and top and bottom positions of a face using a preprocessed face image. As shown in FIG. 6, face both end detection process (step S021). ) And face vertical position detection processing (step S022).

  The face both ends detection process (step S021) is a process for identifying lines constituting both ends of the face of the face image operated by the operator for detecting the vertical edge, and any known method can be adopted. it can.

  For example, as shown in FIG. 10, a histogram creation process for detecting both ends of the face is performed (step S0211), and then a predetermined number of high histogram peak values are extracted and sorted (step S0212). Then, end points are extracted based on the histogram value (step S0213). For example, if the top one or two of the histogram values are extremely large compared to the other, that point is set as the end point.

  Next, it is determined whether or not two end points (both ends) have been extracted (step S0214). If they have been determined (step S0214; Yes), the process ends, and if two end points have not been extracted ( Step S0214; No), a process in which the endpoints are determined by extracting a combination in which the distance between the histogram values is likely to be a human face width (step S0215), and finally the both ends of the face are determined Is performed (step S0216).

  Further, as disclosed in Patent Document 1, the time derivative of the captured image is calculated, and the pixel value of the pixel value time derivative image is projected in the vertical direction to create a histogram. The histogram and the histogram of the pixel value time-differential image may be summed to extract the one having a high peak value of the histogram, determine a plausible point as a human face width, and detect the face edge position.

  Next, the face vertical position detection process in step S022 of FIG. 6 is a process for detecting the position of the eyes (upper end) and the position of the mouth (lower end) of the face by performing the same process as described above for the horizontal edge. For example, as shown in FIG. 11, the process includes a histogram creation process (step S0221), a current candidate detection process (step S0222), and a face vertical position calculation process (step S0233).

  The histogram creation process (step S0221) is a process of creating a histogram by projecting the value of each pixel after the Sobel filter process using the horizontal edge detection Sobel filter in the horizontal direction.

The current candidate detection process (step S0222) is a process of selecting a histogram value candidate corresponding to the eyes, eyebrows, mouth, and the like based on the histogram value.
The face vertical position calculation process (step S0233) is a process for detecting the upper and lower end positions of the face (for example, the positions of eyes and eyebrows) from the selected candidates. For example, the upper end position of the face is set above the eyebrows, and the lower end position is set between the mouth and the chin.

  The CPU 24 stores the face end (left and right side end) positions and the face vertical position obtained in steps S021 and S022 in this manner in the RAM 25.

  Thus, the initial process in FIG. 4 is completed, and then the CPU 24 starts a loop process from the first process to the n-th process.

  The first to n-th periodic processes are respectively pre-process (steps S11 to Sn1), basic process (steps S12 to Sn2), post-process (steps S13 to Sn3), and split process (steps S14 to Sn4). And.

  The preprocessing (steps S11 to Sn1) is the same as the preprocessing of step S01, and as shown in FIG. 5, a capture process for capturing a face image (step S011), a coordinate conversion process for thinning out pixels (step S012), It includes Sobel filter processing (Step S013) for processing the face image with the Sobel filter. However, the Sobel filter processing here is only the Sobel filter processing for vertical edge detection.

  The basic processes (steps S12 to Sn2) constituting the first process to the nth process in FIG. 4 are processes for detecting both ends of the face and detecting the center of the face. As shown in FIG. It comprises a detection process (step S121) and a face center detection process (step S122).

  The face both ends detection process (step S121) can be realized by the same process as the process shown in FIG. That is, as described above, the detection histograms at both ends of the face are created (step S0211). Subsequently, the candidate data at the positions of both ends of the face are rearranged in the order of the histogram (step S0212). Next, it is determined whether there is a value extracted from the histogram, and if there is a value extracted, it is adopted as a candidate for both ends of the face (step S0213). In step S0213, it is determined whether or not both ends of the face have been detected (step S0214). If both ends of the face cannot be detected (step S0214; No), a plurality of end candidates are selected based on the histogram, and a relatively matching one is detected based on the distance between the ends of the face (step S0215). ) Finally, a process for determining both ends of the face is performed (step S0216).

  On the other hand, the face center detection process (step S122) in FIG. 7 has, for example, the configuration shown in FIG.

  First, the CPU 24 binarizes the gradation of each pixel constituting the face image stored in the RAM 25 with an arbitrary threshold as a reference (step S1221). As the threshold value, for example, an average value of all the pixels forming the face image can be used. Thereby, for example, a binary image of the driver as illustrated in FIG. 18A is generated.

Subsequently, the CPU 24 executes a monochrome white image generation process on the binarized image (step S1222).
In this monochrome white image generation process, the CPU 24 first reads from the ROM 23 area designation data (y1, y2 shown in FIG. 3 (e)) that defines the area where the eye is expected to be present in the binary image. Then, the eye region of the binarized image is cut out as shown in FIG.

Subsequently, a black-and-white white horizontal edge is detected for the cut binary image.
Here, the black-and-white white edge is not an edge that merely becomes a boundary between white and black, but an edge that changes from white to black and from black to white and has a predetermined width. Among them, the black and white white horizontal edge means a boundary extending in the horizontal direction, and among these, the black and white white vertical edge means a boundary extending in the vertical direction.

  First, the CPU 24 starts black and white white horizontal edge detection processing, and according to the setting shown in FIG. 3F, in the eye area image, 6 to 12 pixels in the vertical direction according to the size of sunglasses (normal eyes are 2 to 2). A black pixel row having a width of (4 pixels) is detected.

  The technique itself for detecting the black-and-white white horizontal edge is arbitrary. For example, while sequentially updating the coordinate value y, the luminance of the pixel (x, y) is discriminated, and when the luminance changes from white to black, the number of consecutive blacks is counted, and when black changes to white, black It is determined whether or not the continuous number is 6 to 12, and if the continuous number is 6 to 12, the pixel is maintained, otherwise, the pixel is converted to white. it can.

  Next, a monochrome white vertical edge is detected by the same operation as the monochrome white horizontal edge detection process. The black-and-white white vertical edge detection process is performed by applying a black pixel column having a width of 10 to 18 (6 to 10 for normal eyes) in the horizontal direction from the processed image of the eye area in accordance with the setting of FIG. It is a process to detect.

  By adopting such a configuration, for example, when the extracted image of the eye region is the image illustrated in FIG. 18B, the continuous number is 1 to 9 and 19 or more in the x direction, and 1 in the y direction. ˜5 and 13 or more consecutive black pixels are converted into white pixels. As a result, the black pixel rows that are short in the lateral direction and the black pixel rows that are not too long, which do not constitute sunglasses lenses, are removed as shown in FIG.

  Next, CPU24 performs the process which discriminate | determines the presence or absence of sunglasses by step S1223 of FIG. The CPU 24 determines that sunglasses have been detected when two black regions of 6 to 12 pixels in the vertical direction and 10 to 18 pixels in the horizontal direction are obtained in the processed eye region image.

  When it is determined that there is sunglasses (step S1224; Yes), an image of a standard eye area prepared in advance, for example, an image of the eye area after processing shown in FIG. Is synthesized with an image obtained by removing the eye region from the original binary image shown in FIG. 18 to generate a face image for center position measurement shown in FIG. 18F (step S1225). The position of the eye may be adjusted by determining the position of the sunglasses.

Subsequently, using the face image, the center position of the face (face position, barycentric position) is obtained (step S1226). Although the center measurement method itself is arbitrary, for example, the coordinates of the center (center of gravity) position of the face can be obtained from the following equation.
X-coordinate of the center of the face = Σxi / n xi: x-coordinate value of the i-th black pixel y-coordinate of the center of the face = Σyi / n yi: y-coordinate value of the i-th black pixel i: 1 to n n Is the total number of black pixels

  Note that the range for obtaining the center of gravity may be limited to, for example, between both side edges of the face and the upper and lower positions of the face obtained at that time. Or you may obtain | require a gravity center only from the image of the main parts (eyebrows, eyes, nose, mouth) of the face.

  In addition, when sunglasses are not detected by step S1224 (step S1224; No), the synthetic | combination process of step S1225 is skipped.

  Thus, the center position of the face is obtained, and the face center detection process (step S122) in FIG. 7 ends. That is, by this process, the horizontal position of the face and the center position of the face are obtained, and the basic process (steps S12, S22,... Sn2) in FIG.

  Thereafter, the processing proceeds to post-processing (steps S13, S23,..., Sn3) in FIG.

  Post-processing (steps S13, S23,..., Sn3) determines the face orientation based on both end positions and the face center position obtained in the immediately preceding basic processing (steps S12, S22,..., Sn2). And the process of obtaining the reliability.

  As shown in FIG. 8, the post-processing (steps S13, S23,..., Sn3 is performed at both end positions and the face center position obtained in the immediately preceding basic processing (steps S12, S22,..., Sn2). A process of calculating a face orientation angle (orientation; a lateral angle) based on the process (step S131), a process for obtaining the reliability (step S132), and a process of outputting the face orientation angle and the reliability in association with each other (step S132). Step S133) is provided.

  The face orientation angle calculation process (step S131) includes both end positions of the face obtained in the immediately preceding basic process (steps S12, S22,..., Sn2) and the immediately preceding basic process (steps S12, S22,. The face orientation (angle) is determined based on the center position of the face obtained in Sn2). It should be noted that any conventionally known method can be used as the calculation method for obtaining the face orientation. For example, if the center position of the face of the face exists at the center of both end positions of the face, it can be determined that the face is facing the front (the direction facing the camera 10). If the center position of the face exists to the left of the center of both end positions of the face, it can be determined that the face is facing left as viewed from the camera 10, and the center position of the face is more than the center of both end positions of the face. If it is present on the right, it can be determined that the camera 10 is facing right when viewed from the camera 10.

Subsequently, the CPU 24 executes a reliability calculation process for obtaining the reliability of the calculated face orientation angle (step S132).
As shown in FIG. 13, the reliability calculation process (step S132) includes a face end position displacement reliability calculation process (step S1321), a histogram reliability calculation process (step S1322), and a centroid balance reliability calculation process (step S1322). S1323) and luminance balance reliability calculation processing (step S1324).

FIG. 14 shows details of the face both-end position deviation reliability calculation process (step S1321).
As shown in the figure, the CPU 24 calculates reliability = {t1− | R1−R2 |) + (t1− | L1−L2 |) + (t1− | W1−W2 |)} / 3 (step S13211). .
Note that the minimum values of (t1− | R1−R2 |), (t1− | L1−L2 |), and (t1− | W1−W2 |)} are set to 0, respectively.

t1 is a coefficient,
R1 is the position of the right edge of the face obtained this time, R2 is the reference value of the right edge of the face (the initial value is the value obtained in the face position determination process (step S02) of the initial process),
L1 is the position of the left edge of the face obtained this time, L2 is the reference value of the left edge of the face (the initial value is the value obtained in the face position determination process (step S02) of the initial process),
W1 is the face width obtained this time = | R1-L1 |, and W2 is a reference value of the face width (the initial value is the distance between the right end and the left end obtained in the face position determination process (step S02) of the initial process). Value).

The technical meaning of this reliability is schematically shown in FIG.
Both end positions (R2, L2) of a face image (usually a face image obtained by the previous process) whose reference is the both end positions (R1, L1) and width (W1) of the face image obtained this time If the value is relatively close to the width (W2), a high evaluation is given, and if the value is relatively far, a low evaluation is given. Since the execution interval of the iterative process is short, these evaluation values are based on the assumption that there is usually no significant change in these values.

  The CPU 24 determines whether or not the evaluation value is greater than or equal to a predetermined threshold (step S13212). If the evaluation value is greater than or equal to the reference value (step S13212; Yes), that is, the current position detection is performed relatively accurately. If it can be evaluated, the R1, L2, and W1 acquired this time are stored as reference values R2, L2, and W2 for the next evaluation as schematically shown in FIG. 19B (step S13213). On the other hand, when the evaluation value is less than the threshold value (step S13212; No), the reference values R2, L2, and W2 are not changed as shown in FIG. 19C (step S13214).

Next, details of the histogram reliability calculation process (step S1322) will be described with reference to FIG.
As shown in the figure, the CPU 24 determines the reliability = {P1 / | P1 + P2 + P3 +. . . Pm)} · t2 is calculated (step 13221).

t2 is a coefficient,
P1 to Pm are equal to or higher than the threshold value among the histogram values, and are in descending order with P1 as the top. m is preferably about 4 to 7. Note that the top m histogram values (in this case, m is a fixed number) may be selected.

  The technical meaning of this reliability is schematically shown in FIGS. 20 (a) and 20 (b) are diagrams showing examples of detected face edges and histogram distributions when the left and right edges of the face can be accurately detected, and FIG. 20 (c) and FIG. FIG. 20D is a diagram showing an example of detected face edges and histogram distribution when the left edge and right edge of the face cannot be accurately detected, respectively. Note that five peaks are extracted from the histogram.

  As shown in FIGS. 20A and 20B, when the peak value when the left edge and the right edge of the face pixel are determined is extremely larger than the other peak values, the left edge or the right edge is relatively High reliability. On the other hand, as shown in FIGS. 20C and 20D, when the face position is determined without obtaining a clear peak value, the reliability is low.

Next, details of the gravity center balance reliability calculation process (step S1323) are shown in FIG.
First, as schematically shown in FIG. 21, the CPU 24 has a center-of-gravity detection area as an upper area composed of the upper face face (FU) and the center of the face (FC), and a lower area composed of the face center (FC) and the lower face (FD). The area is divided (step S13231).

  Next, the CPU 24 reads the gravity center position (GL) obtained by the face center detection process of the immediately preceding basic process, and calculates the gravity center position (UG) in the upper region and the gravity center position (DG) in the lower region (step). S13232).

Next, the CPU 24 calculates the gravity center balance reliability = {t3- (| GL-UG | + | GL-DG |)} × t4 (step S13233).
It should be noted that all the calculation results are 0 or less.
here,
t3: coefficient, t4: coefficient

The technical meaning of this reliability is schematically shown in FIG.
The human face is bilaterally symmetric as shown in FIG. 21 (a), and normally the center of gravity position (GL) of the center detection area obtained by the basic processing, and the center of gravity position (UG) of the image of the upper area, The barycentric position (DG) of the image in the lower region matches with the x-axis direction.

  However, as illustrated in FIG. 21B, when the balance of the face is lost due to the wearing of the eye patch or the like, the X coordinate values of the centroids GL, UG, and DG are shifted from each other. Therefore, when the deviation is large, it is determined that the reliability of the face direction angle is low.

Next, details of the luminance balance reliability calculation process (step S1324) are shown in FIG.
First, as schematically shown in FIG. 22A, the CPU 24 determines the face center detection area as the upper left face of the face (UL), the upper right edge of the face (UR), the lower left face of the face (DL), and the right face of the face. It divides | segments into a lower end (DR) (step S13241).

  Next, the CPU 24 calculates an average luminance (IUL, IUR, IDL, IDR) in each region (UL, UR, DL, DR) (step S13242).

Next, the CPU 24 calculates the luminance balance reliability according to the following equation (step S13243).
Luminance balance reliability = {t5− (| IUL-IUR | + | IUL-IDL |
+ | IUL-IDR | + | IUR-IDL |
+ | IUR-IDR | + | IDL-IDR |} × t6
t5: Coefficient, t6: Coefficient calculation result is 0 or less for all results.

The technical meaning of this reliability is schematically shown in FIG.
In general, as shown in FIG. 22A, a human face is symmetric and is illuminated symmetrically. However, depending on the orientation of the face and the position of the light source, as shown in FIGS. 22 (b) and 22 (c), illumination is performed in a biased manner, and a portion and a bright portion are generated on the face.

  Among these, when the brightness of the face in the left-right direction as shown in FIG. 22C is different, the determination of the position and orientation of the face in the left-right direction is adversely affected. Therefore, the evaluation formula is configured so that the evaluation decreases as the balance in the left-right direction becomes worse.

  As described above, the reliability of the face orientation obtained in the immediately preceding basic process (steps S1 to Sn2) is obtained by the evaluation process.

  Next, the CPU 24 outputs the face orientation angle obtained in step S131 and the reliability obtained in step S132 in association with each other (step S133).

  The pre-processing, basic processing, and post-processing executed in each of the first process to the n-th process are thus completed, and the angle in the horizontal direction of the face and the reliability thereof are output in association with each other.

Subsequently, the CPU 24 starts a dividing process (Steps S14 to Sn4).
As shown in FIG. 9, the division processing (steps S14 to Sn4) generally includes an image capture process (step S141), a coordinate conversion process (step S142), a Sobel filter process (step S143), a top and bottom face. And position detection processing (step S144). Image capture processing (step S141), coordinate conversion processing (step S142), and Sobel filter processing (step S143) are the same as the processing (steps S011 to S013) in the preprocessing (step S01) described above. However, for the Sobel filter processing here, only the Sobel filter processing for detecting the horizontal edge is sufficient.

  The face vertical position detection process (step S144) is the same as the face vertical position detection process (step S022) shown in FIG. 6, and is the first division process (image capture process in step SS14 (step S141)). The vertical position is determined based on the acquired face image.

  When executing specific processing, the CPU 24 saved various pointers and register values in the RAM 25 and saved in the RAM 25 when starting the division processing (steps S14 to Sn4). Restore (reset) the processing result of the previous split processing. Subsequently, the CPU 24 continues the operation according to the restored pointer and register, performs a part of the process for determining the vertical position of the face using the data stored in the RAM 25, and when the process is completed, the pointer and the register The register value is saved in the RAM 25, and the pointer value and register value of the normal process saved in the RAM 25 (the value immediately before starting the division process) are restored, and the process is continued.

  Each division process (steps S14 to Sn4) may be switched when a certain amount of processing is executed, or may be switched when a certain time has passed.

  As described above, according to this face direction detection device, it is possible to determine the discrimination value of the face position and orientation and its reliability.

  The form of use of the output signal for the face orientation determination is arbitrary. For example, the driver can emit a warning for looking aside, based on the output from the face orientation detection device. At this time, the notification sound can be updated according to the reliability.

  In addition, this invention is not limited to the said embodiment, A various deformation | transformation and application are possible.

  In the above embodiment, four types of confidence values have been obtained, but the average of these values is obtained, the remaining two degrees of reliability excluding the reliability of the highest point and the lowest point are output targets, Alternatively, the average value may be output.

  Further, in the face both-end positional deviation reliability calculation process of step S1321 of FIG. 13, the calculation formula of the reliability of step S13211 of FIG. 14 can be changed and corrected as appropriate. For example, the reliability was calculated using the deviation of the left and right edge positions of the face from the reference position, but the difference between the detected face center (center of gravity) and the reference center position, the face center and both end positions, and each reference As long as the positional deviation in the left-right direction can be detected based on the deviation from the position, such as calculating the reliability, and the reliability can be detected from the deviation, the detection and comparison target and the arithmetic expression are arbitrary. In addition, in the vertical direction (upper position, lower position, center), the deviation between the detection position and the reference position may be detected to detect (determine) the reliability. Further, the face image area may be divided into a plurality of areas, and the amount of deviation may be obtained for each area to obtain the reliability.

  Similarly, in the histogram reliability calculation process in step S1322 in FIG. 13, the calculation formula in step S13221 in FIG. 15 can be changed or modified as appropriate. For example, the reliability may be calculated based on the magnitude of the difference (P1−P2) between the first and second peaks of the histogram and the ratio to the whole. In addition, if the reliability can be measured from the superiority of the peak of the histogram used for detecting the side edge of the face with respect to the other peak, the method is arbitrary. Further, the face area may be divided into a plurality of areas, and a histogram may be obtained for each area to obtain the reliability, or the obtained plurality of reliability may be integrated into one reliability.

  Further, in the center-of-gravity balance reliability calculation process in step S1323 in FIG. 13, the region division number division method in step S13231 in FIG. 16 and the calculation formula in step S13233 can be changed or modified as appropriate. For example, the reliability may be a function of | UG-DG |. Alternatively, the face image is divided into five upper and lower regions, the center of gravity position is obtained for each region, and the deviation of the center of gravity between the five regions and / or the difference from the overall center of gravity is obtained, and the reliability is calculated from these deviations. May be equal.

  In the brightness balance reliability calculation process in step S1324 in FIG. 13, the number of area divisions (vertical division number and horizontal division number) in step S13241 in FIG. 17 and the calculation formula in step S13243 are appropriately changed.・ Can be modified. For example, the face image may be divided into upper and lower three regions and right and left four regions, the luminance is obtained for each, and the reliability may be calculated from the luminance variation among the twelve regions. Further, the reliability may be determined based on the difference or balance between the average luminance of each divided region and the overall average luminance.

  Further, reliability other than the above four types of reliability may be introduced. For example, the reliability may be set such that the reliability is lowered when a change in the histogram distribution on the time axis, a change in the center of gravity position on the time axis, or a change in luminance on the time axis is too rapid.

  Further, in the above embodiment, the four reliability levels are used as the reliability levels of the face orientation angle (lateral angle) determined in step S131. The present invention is not limited to this, and the above four reliability levels and other arbitrary reliability levels are determined based on the reliability level of the face position (for example, the reliability level of the both end positions of the face determined in step S121, step S122). The reliability of the center position of the detected face, the reliability of the vertical position of the face determined in step S144), the presence or absence of the face (exists if both ends and center of the face are detected, must detect both ends and center of the face) Such as non-existent) reliability.

  For example, in the above embodiment, the eye area is fixed at a specific position in the image, but the position of the eye area may be appropriately set according to the size and position of the face image. In this case, for example, a horizontal edge Sobel filter is applied, the positions of the eyes and eyebrows of the face are discriminated, and an area of a predetermined size including the area may be set as the eye area.

  In the above embodiment, the center (center of gravity) position of the face is obtained from the gradation images of the eyes, eyebrows (or eyes), nose, and mouth, but the face part (parts) used to determine the center position Is optional. For example, the center may be obtained by adding ears, cheeks, hair, and the like.

  Moreover, in the said embodiment, when sunglasses were detected, the image of eyes was combined and the center position of the face was calculated | required based on the synthesized image, but the method of calculating | requiring a gravity center is arbitrary. For example, when sunglasses are detected, a center is obtained by combining the images of “eyes and eyebrows”, or a weight of about 0.3 to 0.5 times is applied to each pixel constituting the lens image of sunglasses. In addition, the center of gravity of each face (lens, eyes, nose, mouth,...) May be obtained to obtain the center of gravity of the face image. The centroid may be obtained by adding an offset value in consideration of the influence of the eye image in advance to the centroid obtained from the face image of the portion excluding the eye region.

  In the above embodiment, the present invention has shown the embodiment in which the presence / absence of sunglasses is detected in order to detect the center position of the face. However, after detecting the sunglasses, what kind of processing is used for the result? Is optional. For example, it is possible to detect sunglasses, thereby changing the intensity of illumination and adjusting the brightness of the backlight of the screen.

  The system configuration described with reference to FIGS. 1 and 2 is also an example, and can be arbitrarily changed. For example, if the camera 10 is an infrared camera that captures an image with far infrared rays or the like, it is possible to acquire the image of each part of the face relatively accurately without being affected by the race, skin, or hair color.

The above-described flowchart can be arbitrarily changed as long as the same function can be realized.
For example, the sunglasses handling process may be performed when images of sunglasses are obtained more than a predetermined number of times.

  Further, in the present invention, the black and white white edge is for expressing a difference in gradation, and is not limited to white or black as a color, and any hue may be used. Regarding color images, the presence or absence of sunglasses may be determined in consideration of the hue of each pixel.

  In each of the above embodiments, the present invention is applied to the case where sunglasses are detected by photographing a driver. However, the present invention is not limited to this, and humans, animals, dolls, robots, etc. wear sunglasses in any scene. The present invention can be widely applied to processing for determining whether or not it is applied.

  The present invention is not limited to processing while acquiring an image with a camera. For example, for each of one or a plurality of face images photographed elsewhere, the presence or absence of sunglasses, the position of the center of the face, the orientation of the face, etc. It can be used to determine.

  Further, a computer program for causing a computer to execute the above-described processing may be stored in the ROM via an arbitrary recording medium or a network.

1 is a block diagram of a face orientation determination device according to an embodiment of the present invention. It is a block diagram which shows the structure of the computer shown in FIG. It is a figure for demonstrating the various data stored in ROM. 3 is a flowchart for explaining the operation of the face orientation determination device shown in FIG. 1. It is a flowchart for demonstrating the specific example of the pre-processing in the flowchart of FIG. 5 is a flowchart for explaining a specific example of face position determination processing in the flowchart of FIG. 4. It is a flowchart for demonstrating the specific example of the basic process in the flowchart of FIG. It is a flowchart for demonstrating the specific example of the post-process in the flowchart of FIG. FIG. 5 is a flowchart for explaining a specific example of first to n-th division processes in the flowchart of FIG. 4. FIG. It is a flowchart for demonstrating the specific example of the face both ends detection process of FIG. It is a flowchart for demonstrating the specific example of the face up-and-down position detection process of FIG. It is a flowchart for demonstrating the specific example of the face center detection process of FIG. It is a flowchart for demonstrating the specific example of the reliability calculation process of FIG. FIG. 9 is a flowchart for explaining a specific example of face both-end positional deviation reliability calculation processing of FIG. 8. FIG. It is a flowchart for demonstrating the specific example of the histogram reliability calculation process of FIG. It is a flowchart for demonstrating the specific example of the gravity center balance reliability calculation process of FIG. It is a flowchart for demonstrating the specific example of the brightness | luminance balance reliability calculation process of FIG. It is a figure for demonstrating a sunglasses detection process. It is a figure for demonstrating the technical meaning of a face both-ends position shift reliability. It is a figure for demonstrating the technical meaning of a histogram reliability. It is a figure for demonstrating the technical meaning of a gravity center balance reliability. It is a figure for demonstrating the technical meaning of brightness | luminance balance reliability.

Explanation of symbols

10 Camera 22 Image memory (Face image storage means)
23 ROM (face position determination means, face orientation determination means, reliability determination means, output means)
24 CPU (face position determination means, face orientation determination means, reliability determination means, output means)

Claims (7)

Face image storage means for storing a face image obtained by photographing a face;
Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
A face orientation determining means for outputting the orientation of the face based on the face position and the center position of the face determined by the face position determining means;
Reliability determination means for obtaining a reliability indicating the certainty of the face orientation determined by the face orientation determination means;
An output means for outputting the orientation of the face determined by the face orientation determining means and the reliability determined by the reliability determining means;
With
The reliability determination means obtains the reliability of the face orientation based on the magnitude of the deviation between the previously determined face position and the currently detected face position. . Face image storage means for storing a face image obtained by photographing a face;
Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
A face orientation determining means for outputting the orientation of the face based on the face position and the center position of the face determined by the face position determining means;
Reliability determination means for obtaining a reliability indicating the certainty of the face orientation determined by the face orientation determination means;
An output means for outputting the orientation of the face determined by the face orientation determining means and the reliability determined by the reliability determining means;
With
The reliability determination unit divides a face image into a plurality of areas, calculates a centroid of each area, and calculates a reliability of the face direction based on the calculated centroid position. Face image storage means for storing a face image obtained by photographing a face;
Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
A face orientation determining means for outputting the orientation of the face based on the face position and the center position of the face determined by the face position determining means;
Reliability determination means for obtaining a reliability indicating the certainty of the face orientation determined by the face orientation determination means;
An output means for outputting the orientation of the face determined by the face orientation determining means and the reliability determined by the reliability determining means;
With
The reliability determination unit divides a face image into a plurality of areas, calculates the luminance of each area, and calculates the reliability of the face position based on the calculated luminance. The face position determining means, the face orientation determining means, and the reliability determining means determine a horizontal position and orientation of a face based on the face image stored in the face image storage means; Repeat the process of 1 unit to obtain the reliability of the determined horizontal direction,
By dividing the process of determining the position of the face in the vertical direction into a plurality of processing steps, executing one of the dividing steps within each unit of processing, and repeating the execution of the dividing step a plurality of times, Find the vertical position,
The face orientation discrimination device according to any one of claims 1 to 3 . Computer
Face image storage means for storing a face image obtained by photographing a face;
Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
Face orientation determining means for outputting the orientation of the face based on the position of the face determined by the face position determining means and the position of the center of the face;
Reliability determination means for obtaining reliability indicating the certainty of the face orientation determined by the face orientation determination means;
An output means for outputting the face orientation determined by the face orientation determination means and the reliability determined by the reliability determination means in association with each other;
A computer program that functions as
A computer program for causing the computer to execute a process for determining the reliability of the face orientation based on the degree of deviation between the previously determined face position and the currently detected face position as the reliability determination means. . Computer
Face image storage means for storing a face image obtained by photographing a face;
Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
Face orientation determining means for outputting the orientation of the face based on the position of the face determined by the face position determining means and the position of the center of the face;
Reliability determination means for obtaining reliability indicating the certainty of the face orientation determined by the face orientation determination means;
An output means for outputting the face orientation determined by the face orientation determination means and the reliability determined by the reliability determination means in association with each other;
A computer program that functions as
A computer program for causing the computer to execute a process of dividing a face image into a plurality of areas, obtaining a centroid of each area, and obtaining a face orientation reliability based on the obtained centroid position as the reliability determination unit. Computer
Face image storage means for storing a face image obtained by photographing a face;
Face position determination means for determining the position of the face and the position of the center of the face based on the face image stored in the face image storage means;
Face orientation determining means for outputting the orientation of the face based on the position of the face determined by the face position determining means and the position of the center of the face;
Reliability determination means for obtaining reliability indicating the certainty of the face orientation determined by the face orientation determination means;
An output means for outputting the face orientation determined by the face orientation determination means and the reliability determined by the reliability determination means in association with each other;
A computer program that functions as
A computer program for causing the computer to execute a process of dividing a face image into a plurality of areas, obtaining the brightness of each area, and obtaining the reliability of the face position based on the obtained brightness, as the reliability determination means.

Images