JP5098982B2 - Face detection device - Google Patents

Description

  The present invention relates to a face part detection device.

Various devices have been developed to assist the driver of the vehicle, for example, a side-view determination device and a collision prevention device. In the case of an aside look determination device, a face is detected from a captured image including a driver's face imaged by a camera, a face direction and a line-of-sight direction are detected from the face image, and whether or not a person is looking aside from the face direction or the line-of-sight direction Is judged. In such an apparatus, it is important to detect the face direction and the line of sight with high accuracy. As a method for detecting the face direction, for example, a specific part (such as a nose) of the face is detected, and the position and angle of the face are detected based on the position of the part. In the apparatus described in Patent Document 1, the face orientation is estimated using the positions of facial feature points (left and right nasal cavities) detected from the face image. In particular, with this device, the brightness of the face varies depending on the direction of the illumination light source and the intensity of the light source. The orientation of the face is estimated using the obtained image (that is, the image excluding the influence of the shadow on the face).
JP 2004-94491 A Japanese Patent Laid-Open No. 2003-242491 JP 2002-24811 A JP 2007-109229 A

  However, in the case of a driver of a vehicle moving in an outdoor environment, light hits the face from various directions. For this reason, the shadow changes depending on the nose depending on the direction of light. As a result, face parts such as the nose cannot be detected with high accuracy, and the accuracy of face orientation estimation also decreases.

  Therefore, an object of the present invention is to provide a face part detection device that can detect a face part from an image with high accuracy regardless of the direction of light hitting the face.

A face part detection apparatus according to the present invention is a face part detection apparatus that detects a face part using a haar-like feature, and includes a face detection unit that detects a face from an image captured by an imaging unit, and a face detection The brightness detection means for calculating the brightness information in each of the divided areas is compared with the brightness information of each area calculated by the brightness calculation means, and the low brightness area Correction means for increasing the weight of the haar-like feature on the side or reducing the threshold value of the haar-like feature , and the luminance calculation means divides the face image into left and right regions with reference to the center line in the left-right direction. It is characterized by.

  In this face part detection device, an image is picked up by an image pickup means, and a face is detected from the picked-up image by the face detection means. Then, in the face part detection device, the luminance information is calculated for each area obtained by dividing the face image by the luminance calculating means (for example, the average value of the luminance of all the pixels in the area and the sum of the luminances of all the images in the area). To do. In the face image, the luminance is high in a region where light is applied to one side of the face, and the luminance is low in a region where light is not applied to the other side (particularly, a shadowed area such as a nose). For this reason, the haar-like feature value calculated in the other side region is relatively smaller than the haar-like feature value calculated in the one side region. Therefore, in the face part detection device, the luminance information of each region is compared by the correction means, and the weight used when calculating the haar-like feature amount on the low-luminance region side is increased or the haar-like feature amount is determined. Reduce the threshold when The face part detection apparatus detects the face part using the haar-like feature. The haar-like feature here is for detecting a facial part. In this way, the face part detection device detects the face part by changing the weight or threshold value of the haar-like feature according to the luminance information of each area in the face image, thereby detecting the face part in the direction of light hitting the face. The influence of the corresponding shadow can be eliminated, and the part of the face can be detected from the image with high accuracy regardless of the direction of the light hitting the face. By using the face part detected with high accuracy, the face orientation, line of sight, and the like can be estimated with high accuracy.

  When light hits the face from various directions, the shadow is most affected by light from the left and right directions on the nose. Therefore, in the face part detection device, the luminance calculation unit divides the face image into left and right regions with reference to the center line in the left and right direction, and calculates luminance information of the left and right regions, respectively. In the face part detection device, the luminance information of the left region and the right region is compared by the correcting means, and the weight of the haar-like feature on the lower luminance side of the left region and the right region is increased or the haar-like feature is increased. Reduce the feature threshold. As described above, the face part detection device detects the face part by changing the weight or threshold value of the haar-like feature according to the luminance information of the left and right regions in the face image, thereby causing the most influence of the shadow on the face. The effect of shadows in the vicinity of the nose can be eliminated, and the facial part can be detected with high accuracy from the image.

  In the face part detection apparatus according to the present invention, the haar-like feature value is calculated using the increased weight in the one area where the weight is increased by the correcting means, and the haar-like is calculated using the weight in the other area. It is good also as a structure which calculates a feature-value and detects a nose part based on the haar-like feature-value calculated in each area | region on either side. In this face part detection device, the nose part is detected from the image with high accuracy by detecting the nose part by increasing the weight of the haar-like feature on the low brightness side of the left and right regions in the face image. It can be detected.

  In the face part detection device of the present invention, the haar-like feature amount is determined using the reduced threshold value in the one side area where the threshold value is reduced by the correction unit, and the haar-like is used using the threshold value in the other side area. It is good also as a structure which determines a feature-value and detects a nose part based on the determination result in each area | region on either side. In this face part detection device, the nose part is detected with high accuracy from the image by detecting the nose part by reducing the threshold value of the haar-like feature on the low brightness side of the left and right regions in the face image. It can be detected.

  The present invention detects the part of the face by changing the weight or threshold value of the haar-like feature according to the luminance information of each area in the face image, thereby reducing the influence of the shadow according to the direction of light hitting the face. Therefore, it is possible to detect the part of the face with high accuracy from the image regardless of the direction of light hitting the face.

  Hereinafter, an embodiment of a face part detection device according to the present invention will be described with reference to the drawings.

  In the present embodiment, the face part detection device according to the present invention is applied to a face orientation detection device that is mounted on a vehicle and detects the orientation of a driver's face. The face orientation detection apparatus according to the present embodiment detects the nose (particularly the nasal cavity) using the haar-like feature, and detects the face orientation based on the position of the nose. Then, the face direction detection device according to the present embodiment provides the face direction information to a driving support device (for example, an aside look determination device or a collision prevention device) that requires the driver's face direction information.

  The haar-like feature does not directly use the luminance value of each pixel that is easily affected by fluctuations in illumination conditions or noise as a feature amount in the image, but instead of using the luminance value of all pixels in a plurality of adjacent rectangular areas. A difference is used, and a value obtained by multiplying the difference in luminance sum by a weight is defined as a haar-like feature value. If the haar-like feature amount of a certain area in the image is equal to or greater than the threshold value, this indicates that the area has a haar-like feature pattern. The haar-like feature is composed of a combination of a plurality of rectangular areas, and various shapes and sizes are set as the rectangular areas depending on the detection target. FIG. 4 shows an example of haar-like features, which are composed of combinations of black rectangular areas (low luminance areas) and white rectangular areas (low luminance areas) of various sizes and shapes. A weight and a threshold are set for each feature. Note that the haar-like feature provided in the face direction detection device according to the present embodiment is an image picked up when a human face is facing directly in front of the camera and the face is uniformly illuminated. It is generated from an image.

  With reference to FIG. 1, a face orientation detection apparatus 1 according to the present embodiment will be described. FIG. 1 is a configuration diagram of a face orientation detection apparatus according to the present embodiment.

  The face orientation detection device 1 detects a face from a captured image, and searches the face image for a region that matches a haar-like feature for nasal cavity detection. In particular, the face orientation detection device 1 determines the weight of the haar-like feature according to the luminance difference between the left and right regions of the face image in order to detect the nasal cavity with high accuracy regardless of the direction of the light hitting the driver. Change. The face orientation detection device 1 includes a camera 2, a near-infrared projector 3, and an ECU [Electronic Control Unit] 4.

  In the present embodiment, the camera 2 corresponds to an imaging unit described in the claims, and each process in the ECU 4 corresponds to a face detection unit, a luminance calculation unit, and a correction unit described in the claims.

  The camera 2 is a camera for imaging the periphery of the driver's face, and is a camera that can detect a region from visible light to near-infrared light. The camera 2 may be a general camera (for example, monochrome, 30 fps, NTSC system). The camera 2 is provided in front of the driver (such as a steering column), and is arranged in a direction that allows the periphery of the driver's face to be imaged. The camera 2 transmits the captured image of each frame to the ECU 4 as an image signal every time the periphery of the driver's face is imaged. The camera may be a near-infrared camera that can detect only near-infrared light.

  The near-infrared projector 3 is a projector for applying a near-infrared ray around the driver's face when the interior of the vehicle is at least dark (for example, at night). The near-infrared projector 3 is provided in front of the driver (such as a steering column) and is disposed in a direction in which near-infrared rays are applied to the periphery of the driver's face. The near-infrared projector 3 is turned on by the ECU 4 when the interior of the vehicle is at least dark. If the camera is a near-infrared camera that can detect only near-infrared light, the near-infrared projector is always turned on.

  The ECU 4 is an electronic control unit that includes a CPU [Central Processing Unit], a ROM [Read Only Memory], a RAM [Random Access Memory], and the like, and performs overall control of the face orientation detection device 1. The ECU 4 receives an image signal from the camera 2 at regular intervals, and estimates the driver's face orientation based on the image signal.

  Every time an image (frame) captured by the camera 2 is input, the ECU 4 creates an integral image (integrated image) from the captured image. In the integral image, pixels on the diagonal line from the upper left corner to the lower right corner of the image are sequentially selected, and the selected pixel is the lower right vertex and the upper left pixel of the image is the upper left vertex. The sum of the luminances of the pixels is the value of the selected pixel. For example, in the example shown in FIG. 2, when the pixel P1 of the captured image PI is selected, the sum of the luminances of all the pixels in the area A1 having the pixel P1 as the lower right vertex is the pixel P1 of the integral image II. When the pixel P2 of the captured image PI is selected, the sum of the luminances of all the pixels in the area A2 having the pixel P2 as the lower right vertex is set as the value of the pixel P2 of the integral image II. The integral image may be created only for the face image.

  By using this integral image, the sum of all pixels in an arbitrary rectangular area of the image is obtained by calculating the value of the lower right vertex of the corresponding rectangular area in the integral image−the value of the upper right vertex−the value of the lower left vertex + the upper left vertex. It is possible to calculate with the value of For example, in the example shown in FIG. 3, the sum of the luminances of the area A3 having the vertices of the pixels P3, P4, P5, and P6 of the captured image PI is the value of the pixel P6 of the integral image II−the value of the pixel P5−pixel. It can be calculated by the value of P4 + the value of the pixel P3. Thus, by obtaining the integral image, the sum of the luminances of all the pixels in an arbitrary rectangular area can be easily obtained using the values of the four points of the integral image.

  The ECU 4 detects a face from the captured image, and extracts a rectangular area (face image) including the face from the captured image. As this face detection method, a conventional method is applied, for example, pattern matching is used.

  The ECU 4 calculates the sum of the luminances of all the pixels in the left region and the sum of the luminances of all the pixels in the right region with reference to the center line in the horizontal direction of the face image. The luminance information of each region may be luminance information other than the luminance sum, for example, an average value of luminance of all pixels.

  The ECU 4 calculates the difference between the luminance sum of the left region and the luminance sum of the right region. Then, the ECU 4 determines whether the left area is bright, the right area is bright, or the left area and the right area have the same brightness based on the difference in luminance sum. Here, for example, the determination is made using a threshold value set in advance by an experiment or the like. This threshold value is set in advance by an experiment or the like.

  When it is determined that the left and right areas have the same brightness (that is, the driver's face is exposed to light from the front or the driver's face is not exposed to light from a specific direction (for example, In the cloudy day and night)), the ECU 4 sequentially cuts out rectangular regions corresponding to the haar-like feature from the face image using the haar-like feature for detecting the nasal cavity. Here, the rectangular area is cut out while shifting a predetermined number of pixels in the vertical and horizontal directions in the image. Then, the ECU 4 calculates a haar-like feature value using a pre-set weight using an integral image for each cut-out rectangular area.

  FIG. 5 shows an example of an image in the vicinity of the nose when the driver's face is uniformly illuminated. FIG. 5 (a) shows a haar− for detecting the left nasal cavity portion. Like feature h1 is shown, and FIG. 5B shows a haar-like feature h2 for detecting the right nasal cavity portion. The haar-like feature h1 is for detecting a region having a luminance pattern in which the luminance of the left black rectangular region is low and the luminance of the right white rectangular region is high. The haar-like feature h2 is for detecting a region having a luminance pattern in which the luminance of the left white rectangular region is high and the luminance of the right black rectangular region is low. As a haar-like feature for detecting the nasal cavity portion, there are black rectangular regions on the left and right of the two haar-like features h1 and h2, and a white rectangular region is present between the two black rectangular regions. It may consist of three rectangular areas.

  In FIG. 6, the horizontal axis is the horizontal position of the image, the vertical axis is the luminance value, and the luminance change around the nasal cavity shown in FIG. 5 is shown. It shows the brightness (high brightness). As shown in FIG. 6, when the driver's face is uniformly illuminated, the luminance is high on the cheeks on both the left and right sides, the luminance is low near the left and right outer edges of the nose, and the luminance is on the outer edges of the left and right nasal cavities. Is high, the luminance is low in the right and left nasal cavities, and the luminance is high between the right and left nasal cavities.

  In FIG. 7, the horizontal axis represents the horizontal position of the image, the vertical axis represents the luminance value, and the haar-like feature h1 in FIG. 5A and the haar-like feature h2 in FIG. 5B are represented. Changes in brightness. As shown in FIG. 7, the two haar-like features h1 and haar-like features h2 indicate a luminance pattern in which there is a portion with low luminance on the left and right and a portion with high luminance between them. The luminance pattern shown in FIG. 7 and the luminance pattern in which the luminance is low in the left and right nasal cavities shown in FIG.

  When it is determined that the left region is bright (that is, when light is shining from the left direction toward the driver's face (to the right side of the face)), the ECU 4 uses the haar-like feature for detecting the nasal cavity. Then, rectangular areas corresponding to the haar-like features are sequentially cut out from the face image. When a rectangular area is cut out from the left side area (bright side area) of the face image, the ECU 4 uses the integral image for each cut out rectangular area and uses a preset weight to use haar-like. The feature amount is calculated. On the other hand, when a rectangular area is cut out from the right area (dark area) of the face image, the ECU 4 increases the preset weight by a predetermined amount. Then, the ECU 4 uses the integral image for each cut-out rectangular area, and calculates the haar-like feature value using the increased weight.

  When it is determined that the right region is bright (that is, when light strikes the driver's face from the right direction (to the left side of the face)), the ECU 4 uses the haar-like feature for detecting the nasal cavity. Then, rectangular areas corresponding to the haar-like features are sequentially cut out from the face image. When a rectangular area is cut out from the right side area (bright side area) of the face image, the ECU 4 uses an integral image for each cut out rectangular area and uses a preset weight to use haar-like. The feature amount is calculated. On the other hand, when a rectangular area is cut out from the left area (dark area) of the face image, the ECU 4 increases a preset weight by a predetermined amount. Then, the ECU 4 uses the integral image for each cut-out rectangular area, and calculates the haar-like feature value using the increased weight.

  The predetermined amount when increasing the weight may be a fixed amount, or may be an amount that increases as the difference between the luminance sum of the left region and the luminance sum of the right region increases. The predetermined amount is set in advance by experiments or the like, and a map or the like is prepared when the predetermined amount is variable.

  FIG. 8 shows an example of an image in the vicinity of the nose when light is directed from the right direction toward the driver's face. The right area is bright overall and the left area is dark overall. In FIG. 9, the horizontal axis is the horizontal position of the image, the vertical axis is the luminance value, and the luminance change in the vicinity of the nasal cavity shown in FIG. 8 is shown. It shows the same brightness. As shown in FIG. 9, when light is shining on the driver's face from the right direction, the overall luminance change is as shown in FIG. The brightness is high, and the left cheek is a shadow of the nose and the brightness is low. Therefore, a difference in luminance occurs between the left and right nasal cavities, and the difference between the luminance of the left nasal cavity and the luminance of the portion between the left and right nasal cavities is very large. Since this luminance pattern is different from the luminance pattern by the two haar-like features h1 and h2 shown in FIG. 7, it cannot be detected even if the haar-like features h1 and h2 are used as they are.

  In FIG. 10, the horizontal axis is the horizontal position of the image, the vertical axis is the luminance value, and the haar-like feature h1 in which the weight is increased in the left region and the haar-like feature h2 in which the weight is not changed in the right region. The change in luminance to be expressed is shown. As shown in FIG. 10, the haar-like feature h1 with the increased weight and the haar-like feature h2 with the weight not changed change the luminance between the left portion and the left portion of the luminance pattern lower than the luminance pattern of FIG. A luminance pattern in which the luminance difference from the high portion is large and the luminance of the low luminance portion on the right side is high is shown. The luminance pattern shown in FIG. 10 and the luminance pattern in which the luminance is low in the right and left nasal cavities shown in FIG.

  The ECU 4 determines whether or not the haar-like feature value is equal to or greater than a threshold value for each cut-out rectangular area. When it is determined that the haar-like feature amount of a certain rectangular area is equal to or greater than the threshold, the ECU 4 sets the rectangular area as the nasal cavity.

  The ECU 4 detects the orientation of the face from the detected position of the nasal cavity (nose). As this detection method, a conventional method is applied.

  With reference to FIG. 1, the operation of the face orientation detection device 1 will be described. In particular, the processing in the ECU 4 will be described along the flowchart of FIG. FIG. 11 is a flowchart showing a flow of processing in the ECU of FIG.

  When the vehicle interior is dark, the near-infrared projector 3 emits near-infrared rays around the driver's face. The camera 2 captures the image of the driver's face and the surrounding area at regular intervals, and transmits image signals of each frame to the ECU 4. The ECU 4 receives an image signal at regular intervals and inputs a captured image of each frame (S1).

  Every time a captured image is input, the ECU 4 creates an integral image from the captured image (S2). Further, the ECU 4 detects a face from the captured image and extracts a face image from the captured image (S3). Then, the ECU 4 calculates the luminance sum of the left area and the right area of the face image, and calculates the luminance difference between the left and right areas (S4).

  The ECU 4 determines whether one of the left and right regions is bright based on the luminance difference between the left and right regions (S5). If it is determined in S5 that the left area is bright, the ECU 4 increases the weight of the haar-like feature in the right area (S6). If it is determined in S5 that the right area is bright, the ECU 4 increases the weight of the haar-like feature in the left area (S7). When it is determined in S5 that the brightness is the same in the left area and the right area, the ECU 4 does not change the weight.

  The ECU 4 sequentially cuts out rectangular areas corresponding to haar-like features from the face image, and uses an integral image for the cut-out rectangular areas based on haar-like features with changed weights or haar-like features without changing weights. Then, haar-like feature values are sequentially calculated (S8). Then, the ECU 4 determines whether or not the haar-like feature amount for each rectangular area is equal to or greater than a threshold value, and detects the nose (particularly the nasal cavity) from the determination result (S9). Further, the ECU 4 detects the face orientation based on the detected nose (S10). Then, the ECU 4 provides the detected face orientation information to the driving support device.

  According to the face direction detection device 1, even when light is shining from one side of the driver's left and right (when affected by shadows from the nose, etc.), the haar-like feature in the opposite area where no light is shining By increasing the weight, it is possible to eliminate the influence of the shadow (particularly the shadow of the nose) according to the direction of light, and to detect the nose (particularly the nasal cavity) from the image with high accuracy regardless of the direction of the light hitting the face. By using the nose detected with high accuracy, the orientation of the face can be detected with high accuracy.

  In addition, according to the face orientation detection device 1, a dedicated camera or DSP is not required to eliminate the influence of shadows, and there is no increase in hardware costs. Further, according to the face orientation detection device 1, processing is performed for each frame imaged by the camera 2, so that the nose can be detected with high accuracy regardless of the movement of the face and each part and the fluctuation of light. . Further, according to the face orientation detection device 1, since an algorithm for learning is not used, a corresponding image does not depend on learning data.

  Further, according to the face orientation detection device 1, by creating an integral image in advance for calculating the haar-like feature value, the processing load for calculating the haar-like feature value can be reduced, and the image can be reduced. Even when the face size is large, there is no difference in the processing load for each frame compared to the case where the face size is small (the luminance sum of the rectangular area can be calculated from the values of the four pixels).

  As mentioned above, although embodiment which concerns on this invention was described, this invention is implemented in various forms, without being limited to the said embodiment.

  For example, in the present embodiment, the present invention is applied to a face direction detecting device that detects the face direction of a driver of a vehicle, but may be applied to a device that detects a line of sight based on each part of the face. The present invention may be applied to a driving assistance device to be used (for example, an aside look determination device or a collision prevention device), or may be applied to a face part detection device that detects a face part such as a nose. In addition to the driver of the vehicle, the present invention may be applied to another person (for example, a train driver) whose direction of light hitting the face changes.

  In this embodiment, the nose (particularly the nasal cavity) is detected as the face part, but other parts (particularly, a part where shadows can be produced) may be detected.

  In this embodiment, the weight of the darker haar-like feature is increased when there is a luminance difference between the left and right regions. However, the threshold for determining the haar-like feature amount is decreased. Also good.

  In this embodiment, the integral image is created and the haar-like feature value is calculated using the integral image. However, the haar-like feature value is calculated without creating the integral image. Good.

It is a block diagram of the face direction detection apparatus which concerns on this Embodiment. It is explanatory drawing of the creation method of an integral image. It is explanatory drawing of the calculation method of the luminance sum using an integral image. It is an example of a haar-like feature. It is an example of an image near the nose when the face is uniformly illuminated, (a) is an image that also shows a haar-like feature for detecting the nasal cavity on the left side, and (b) It is the image which also showed the haar-like feature for detecting the nasal cavity on the right side. It is a graph which shows the luminance change of the nasal cavity vicinity of FIG. It is a graph which shows the change of the brightness | luminance which the haar-like feature of Fig.5 (a) and the haar-like feature of (b) express. It is an example of the image of the vicinity of the nose when the light hits from the right side toward the face. It is a graph which shows the luminance change of the nasal cavity vicinity of FIG. It is a graph which shows the change of the brightness | luminance which the haar-like feature which increased the weight, and the haar-like feature which does not change a weight represent. It is a flowchart which shows the flow of a process in ECU of FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Face direction detection apparatus, 2 ... Camera, 3 ... Near-infrared projector, 4 ... ECU

Claims (3)

A face part detection device that detects a face part using a haar-like feature,
Face detection means for detecting a face from an image captured by the imaging means;
Luminance calculation means for dividing the face image detected by the face detection means into a plurality of areas and calculating luminance information in each divided area;
Compensating means for comparing the luminance information of each area calculated by the luminance calculating means, increasing the weight of the haar-like feature on the low luminance side, or reducing the threshold of the haar-like feature ,
The brightness calculation means divides a face image into left and right regions with a center line in the left and right direction as a reference . The haar-like feature quantity is calculated using the increased weight in the one side area where the weight is increased by the correction means, and the haar-like feature quantity is calculated using the weight in the other side area. The face part detection device according to claim 1 , wherein a nose part is detected based on a haar-like feature amount calculated in a region. The haar-like feature quantity is determined using the reduced threshold value in the one side area where the threshold value is reduced by the correction means, and the haar-like feature quantity is determined using the threshold value in the other side area. The face part detection device according to claim 1 , wherein a nose part is detected based on a determination result in the region.