JP2023012283A - face detection device - Google Patents

Abstract

To provide a face detection device which can obtain with good accuracy the likelihood of a face being expressed in a prescribed region of an image, even when a portion of the face is not expressed in the image.SOLUTION: A face detection device comprises: a segmentation unit 33 for classifying pixels in a prescribed region of an image into a face pixel that expresses a face and a non-face pixel that does not expresses a face; a feature point detection unit 34 for detecting, for each organ of the face, a plurality of feature points of the organ from the prescribed region; and a confidence calculation unit 35 for setting, for each organ of the face, reliability to each of the plurality of feature points so that the reliability of a feature point which is a face pixel among the plurality of feature points having been detected with regard to the organ is higher than the reliability of a feature point which is the non-face pixel, obtaining the sum total of reliability set with regard to each of the plurality of feature points, and calculating confidence so that confidence representing the likelihood of the face being expressed in the prescribed region is progressively higher as the obtained sum total for each organ increases.SELECTED DRAWING: Figure 3

Description

The present invention relates to a face detection device for detecting a face represented in an image.

A technology for monitoring a person by detecting the person's face from a series of time-series images obtained by continuously photographing the person's face using a driver monitor camera, web camera, etc. are being studied. In such technology, it has been proposed to use tracking processing to detect a person's face from each of a series of images (see, for example, Japanese Laid-Open Patent Application Publication No. 2002-100003).

In the image analysis apparatus disclosed in Patent Document 1, in a state where the tracking flag is turned on, the search control unit calculates the amount of change in the position coordinates of the facial feature points in the current frame and the face orientation in the current frame with respect to the previous frame. It is determined whether each of the amount of change and the amount of change in the line-of-sight direction is within a predetermined range. If all these determination conditions are satisfied, the image analysis device considers that the change in the detection result of the current frame with respect to the previous frame is within the allowable range, and continues to detect faces stored in the tracking information storage unit in subsequent frames. A face image is detected according to an image area.

JP 2019-185557 A

If the tracking of an area representing a person's face (hereinafter referred to as a face area) detected from an image obtained at a certain point in time fails, the face area in the subsequent image will not actually contain the person's face. It may not be pictured. In particular, depending on the orientation of the person's face with respect to the camera, or the positional relationship between the camera and the person's face, part of the person's face may not be captured in the image. If a part of the face is not captured in any of the subsequent images, it becomes difficult to accurately estimate the position of the feature points of the person's face, the face orientation, and the line-of-sight direction. It becomes difficult to correctly obtain the amount of change in the position, face orientation, and line-of-sight direction. Therefore, with the above technique, it may not be possible to accurately determine whether or not to apply the tracking process, and there is a risk of failing to detect a person's face.

SUMMARY OF THE INVENTION Therefore, the present invention provides a face detection apparatus capable of accurately determining the likelihood that a face is represented in a predetermined area on an image even if part of the face is not represented in the image. With the goal.

According to one embodiment, a face detection device is provided. This face detection apparatus includes a segmentation unit that classifies each pixel in a predetermined region on an image into a face pixel representing a face or a non-face pixel that does not represent a face, and a segmentation unit that classifies each pixel in a predetermined region on an image into a face pixel representing a face or a non-face pixel that does not represent a face. a feature point detection unit that detects a plurality of feature points of the organ for each individual organ of the face; A reliability level is set for each of a plurality of feature points so that the reliability level is higher than that of feature points that are non-face pixels. and a certainty calculation unit for calculating the certainty so that the larger the sum of each organ, the higher the certainty representing the certainty that a face is represented in the predetermined area.

ADVANTAGE OF THE INVENTION The face detection apparatus according to the present invention has the effect of being able to accurately obtain the likelihood that a face is represented in a predetermined area on an image even if part of the face is not represented in the image.

1 is a schematic configuration diagram of a vehicle control system in which a face detection device is mounted; FIG. 1 is a hardware configuration diagram of an electronic control device that is one embodiment of a face detection device; FIG. FIG. 4 is a functional block diagram of the processor of the electronic control unit for face detection processing; FIG. 4 is a diagram showing an example of the relationship between face pixels, non-face pixels, and reliability of detected feature points; FIG. 10 is a diagram showing an example of the relationship between the sum of reliability of feature points and the confidence shown in the reference table; 4 is an operation flowchart of face detection processing;

A face detection device, and a face detection method and a face detection computer program executed on the face detection device will be described below with reference to the drawings. This face detection device detects, in an image obtained by photographing the face of a person to be photographed, each pixel included in a predetermined area in which the person's face is supposed to be represented. Pixels are classified as non-face pixels that do not represent a face. In addition, this face detection device detects a plurality of feature points of individual facial organs (eg, eyes, nose, mouth, etc.) from the predetermined area, and detects a corresponding feature point for each detected feature point. Set the confidence level that indicates the likelihood that the organ to be represented is represented. At this time, the face detection apparatus sets the reliability of each feature point so that the reliability of the feature points that are face pixels is higher than the reliability of the feature points that are non-face pixels. This face detection apparatus obtains the sum of the degrees of reliability detected for each individual organ of the face, and the greater the sum of the degrees of reliability obtained for each organ, the more the face is located in the predetermined area. The certainty factor is calculated so that the certainty factor representing the represented certainty is high.

An example in which the face detection device is applied to a driver monitoring device for monitoring a driver based on a series of time-series images obtained by continuously photographing the face of a driver of a vehicle will be described below. In this driver monitoring device, a face area representing the driver's face is detected from an image obtained at a predetermined timing, such as when the ignition switch of the vehicle is turned on, and the subsequent images obtained after that are detected. The face area is tracked by applying tracking processing to it. Then, the driver monitor device performs the above-described face detection processing on the face area on the subsequent image, thereby calculating the degree of certainty that the driver's face is represented in the face area on the subsequent image. Based on the certainty, the driver monitoring device determines whether re-detection of the driver's face can be executed.

The face detection device according to the present embodiment is not limited to the driver monitor device, and detects the face of the person to be photographed from an image obtained by a camera that photographs the face of the person to be photographed, such as a web camera or other surveillance camera. It is suitably used for various uses that require

FIG. 1 is a schematic configuration diagram of a vehicle control system in which a face detection device is installed. Also, FIG. 2 is a hardware configuration diagram of an electronic control unit, which is one embodiment of the face detection apparatus. In this embodiment, a vehicle control system 1 mounted on a vehicle 10 and controlling the vehicle 10 includes a driver monitor camera 2, a user interface 3, and an electronic control unit (ECU) 4 which is an example of a face detection device. have The driver monitor camera 2, the user interface 3, and the ECU 4 are communicably connected via an in-vehicle network conforming to a standard such as a controller area network. The vehicle control system 1 may further include a GPS receiver (not shown) for positioning the vehicle 10 itself. In addition, the vehicle control system 1 includes a camera (not shown) for photographing the surroundings of the vehicle 10, or a distance sensor such as LiDAR or radar that measures the distance from the vehicle 10 to objects existing around the vehicle 10. (not shown). Furthermore, the vehicle control system 1 may have a wireless communication terminal (not shown) for wireless communication with other devices. Furthermore, the vehicle control system 1 may have a navigation device (not shown) for searching the travel route of the vehicle 10 .

The driver monitor camera 2 is an example of a camera or an in-vehicle imaging unit, and includes a two-dimensional detector configured by an array of photoelectric conversion elements sensitive to visible light or infrared light, such as a CCD or C-MOS; It has an imaging optical system that forms an image of an area to be photographed on the dimensional detector. The driver monitor camera 2 may further have a light source for illuminating the driver, such as an infrared LED. Then, the driver monitor camera 2 is mounted on the instrument panel or the like so that the head of the driver seated in the driver's seat of the vehicle 10 is included in the photographing target area, that is, the head of the driver can be photographed. It is mounted in close proximity to the driver. Then, the driver monitor camera 2 takes an image of the driver's head at predetermined shooting intervals (for example, 1/30 second to 1/10 second), and an image representing the driver's face (hereinafter referred to as a face image for convenience of explanation). ). The face image obtained by the driver monitor camera 2 may be a color image or a gray image. Each time the driver monitor camera 2 generates a face image, it outputs the generated face image to the ECU 4 via the in-vehicle network.

The user interface 3 is an example of a notification unit, and has a display device such as a liquid crystal display or an organic EL display. The user interface 3 is installed in the interior of the vehicle 10, for example, on an instrument panel, facing the driver. The user interface 3 notifies the driver of the information by displaying various information received from the ECU 4 via the in-vehicle network. The user interface 3 may also have a speaker installed inside the vehicle. In this case, the user interface 3 notifies the driver of various information received from the ECU 4 via the in-vehicle network by outputting the information as an audio signal.

The ECU 4 detects the orientation of the driver's face based on the face image, and determines the driver's state based on the orientation of the face. Then, the ECU 4 warns the driver via the user interface 3 when the driver is in a state unsuitable for driving such as looking away.

As shown in FIG. 2, the ECU 4 has a communication interface 21, a memory 22, and a processor . The communication interface 21, memory 22 and processor 23 may each be configured as separate circuits, or may be integrally configured as one integrated circuit.

The communication interface 21 has an interface circuit for connecting the ECU 4 to the in-vehicle network. Then, the communication interface 21 passes the received face image to the processor 23 each time it receives a face image from the driver monitor camera 2 . Further, when receiving information to be displayed on the user interface 3 from the processor 23 , the communication interface 21 outputs the information to the user interface 3 .

The memory 22 is an example of a storage unit, and has, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 stores various algorithms and various data used in driver monitor processing including face detection processing executed by the processor 23 of the ECU 4 . For example, the memory 22 stores various parameters used for detection of face regions and feature points, determination of face orientation, and a reference table representing the relationship between the sum of confidence levels and confidence levels. Furthermore, the memory 22 temporarily stores the face image received from the driver monitor camera 2 and various data generated during the driver monitor process.

The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. Processor 23 may further comprise other arithmetic circuitry such as a logic arithmetic unit, a math unit or a graphics processing unit. The processor 23 then executes driver monitor processing including face detection processing.

FIG. 3 is a functional block diagram of processor 23 relating to driver monitor processing. The processor 23 has a face detection unit 31 , a tracking unit 32 , a segmentation unit 33 , a feature point detection unit 34 , a certainty calculation unit 35 , a redetection determination unit 36 and a state determination unit 37 . These units of the processor 23 are, for example, functional modules implemented by computer programs running on the processor 23 . Alternatively, each of these units of processor 23 may be a dedicated arithmetic circuit provided in processor 23 . Among these units of the processor 23, the face detection unit 31, the tracking unit 32, the segmentation unit 33, the feature point detection unit 34, the certainty calculation unit 35, and the re-detection determination unit 36 are related to face detection processing.

The face detection unit 31 detects a face area representing the driver's face from the face image received by the ECU 4 from the driver monitor camera 2 at a predetermined timing.

The predetermined timing can be, for example, the timing at which the ignition switch of the vehicle 10 is turned on, or the timing at which the face initial detection sequence is performed immediately after the driver's face is photographed. Alternatively, the predetermined timing may be timing at predetermined intervals longer than the imaging interval of the driver monitor camera 2 (for example, several tens of seconds to several minutes). Furthermore, the predetermined timing may be the timing at which the re-detection determination unit 36 instructs to detect the face area.

The face detection unit 31 detects a face area by, for example, inputting a face image into a discriminator that has been trained in advance to detect the driver's face from the image. The face detection unit 31 uses a deep neural network (DNN) having a convolutional neural network (CNN) architecture such as a Single Shot MultiBox Detector (SSD) or Faster R-CNN as such a classifier. can be used. Alternatively, the face detection unit 31 may use an AdaBoost classifier as such a classifier. In this case, the face detection unit 31 sets a window on the face image, and calculates a feature amount useful for determining whether or not there is a face, such as a Haar-like feature amount, from the window. Then, the face detection unit 31 inputs the calculated feature amount to the discriminator, thereby determining whether or not the driver's face is displayed in the window. The face detection unit 31 performs the above processing while variously changing the position, size, aspect ratio, orientation, etc. of the window on the face image, and sets the window in which the driver's face is detected as the face region. . The discriminator is trained in advance according to a predetermined learning method using teacher data including images showing faces and images not showing faces. A parameter set that defines the discriminator may be stored in the memory 22 in advance. Moreover, the face detection section 31 may detect a face area from a face image according to another method for detecting a face area from an image.

Each time a face area is detected from a face image, the face detection unit 31 obtains information indicating the detected face area (for example, upper left corner coordinates, horizontal width, and vertical height of the face area on the face image). ) is notified to the tracking unit 32 and the state determination unit 37 .

The tracking unit 32 tracks the face area in each of a series of face images following the face image from which the face detection unit 31 detected the face area. For example, the tracking unit 32 applies a tracking method based on optical flow, such as the KLT method, or a tracking method based on a prediction filter, such as a Kalman filter or a particle filter, to estimate the position and range of the facial region in the subsequent facial image. do. Each time the tracking unit 32 estimates a face region in a subsequent face image, the tracking unit 32 notifies the segmentation unit 33 and feature point detection unit 34 of information representing the position and range of the estimated face region.

In general, the calculation load of the face area tracking process by the tracking unit 32 is less than the calculation load of the face detection process by the face detection unit 31 . Therefore, once a face region is detected by the face detection unit 31, the tracking unit 32 executes face region tracking processing for each subsequent face image. Calculation load can be reduced.

The segmentation unit 33 identifies each pixel included in the face region estimated by the tracking unit 32 in each face image subsequent to the face image in which the face region is detected by the face detection unit 31. classified as non-face pixels that are not represented. Note that the face area estimated by the tracking unit 32 is an example of the predetermined area.

The segmentation unit 33, for example, inputs the estimated face region to a discriminator trained in advance so as to classify each pixel into a face pixel or a non-face pixel, thereby classifying each pixel in the face region as a face. Categorize as pixels or non-face pixels. The segmentation unit 33 can use a DNN for semantic segmentation such as Fully Convolutional Network, U-Net, or SegNet as such a classifier. Alternatively, the segmentation unit 33 may use classifiers according to other segmentation methods such as random forest.

The segmentation unit 33 outputs information representing the classification result of each pixel to the certainty calculation unit 35 for each estimated face region. The information representing the classification result of each pixel in the estimated face area is, for example, a binary image having the same size as the face area and having different values for face pixels and non-face pixels. be able to.

The feature point detection unit 34 detects each organ of the face from the face region estimated by the tracking unit 32 in each face image subsequent to the face image from which the face region is detected by the face detection unit 31. Detect multiple feature points.

A feature point detection unit 34 detects a plurality of feature points of individual organs of the face by applying a detector designed to detect the feature points to the face region to detect the features of individual organs. A point can be detected. As such a detector, the feature point detection unit 34 can use, for example, a detector that uses information of the entire face, such as Active Shape Model (ASM) or Active Appearance Model (AAM). Alternatively, the feature point detection unit 34 may use, as a detector, a DNN that has been pre-trained to detect feature points of individual facial organs.

When the driver's face is not facing the driver monitor camera 2 and faces obliquely, the face image may not capture the entire face of the driver. Also, depending on the posture of the driver, part of the driver's face may be hidden from the driver monitor camera 2 by the driver's hand or the like. In such cases, some organs may not be visible in the face image. However, since the detector uses the information of the entire face to detect the feature points of each organ, the position where the face is not actually shown corresponds to the facial organ not shown in the face image. Feature points may be detected incorrectly. In such a case, it may be difficult to accurately determine the state of the driver's face.

The feature point detection unit 34 outputs, for each estimated face region, information indicating a plurality of feature points for each organ detected from the face region to the certainty calculation unit 35 . The information representing a plurality of feature points for each organ includes, for example, the position coordinates of each feature point and an identification number indicating the facial organ represented by the feature point.

The certainty calculation unit 35 calculates the certainty of each feature point detected from the estimated face area in each face image subsequent to the face image from which the face area is detected by the face detection unit 31. Set the confidence level that represents the likeness. Further, the certainty calculation unit 35 obtains the sum of the reliability of the feature points of each individual organ of the face. Then, the certainty calculation unit 35 calculates the certainty so that the larger the sum of the reliability for each individual organ obtained, the higher the certainty that indicates the probability that the driver's face is represented in the face region. do.

In this embodiment, for each organ of the face, the certainty calculation unit 35 determines that, among a plurality of feature points detected for that organ, feature points that are face pixels have a reliability of non-face pixels. Set the reliability to each feature point so that it is higher than the reliability of . For this purpose, the certainty calculation unit 35 refers to the information indicating the classification result of each pixel in the face region received from the segmentation unit 33 and the position of each feature point received from the feature point detection unit 34 to determine the feature. For each point, it is determined whether the feature point is located in a face pixel or a non-face pixel. Then, the certainty calculation unit 35 sets the first reliability (for example, 1.0) to the feature points located in the face pixels, that is, the feature points that are the face pixels. On the other hand, the certainty calculation unit 35 assigns a second reliability lower than the first reliability (for example, 0.5) to feature points located in non-face pixels, that is, feature points that are non-face pixels. set.

FIG. 4 is a diagram showing an example of the relationship between face pixels and non-face pixels and the reliability of detected feature points. In this example, in the estimated face region 400, each pixel included in the partial region 401 is a face pixel, while each pixel included in the partial region 402 is a non-face pixel. Therefore, among the plurality of feature points 411 of individual organs of the face detected by the feature point detection unit 34, each feature point 411a located within the partial region 401, that is, a face pixel, is given a first confidence. degree (1.0 in this example) is set. On the other hand, among the plurality of feature points 411, a second reliability (0.5 in this example) is set for each feature point 411b located within the partial area 402, that is, each feature point 411b that is a non-face pixel.

After setting the reliability for each feature point, the reliability calculation unit 35 calculates the sum of the reliability set for each feature point detected for each organ of the face. Then, for each organ of the face, the certainty calculation unit 35 refers to a reference table that represents the relationship between the sum of the reliability of the organ and the certainty that represents the certainty of the organ, thereby calculating the probability of the organ. Confidence of Reference tables for individual organs are pre-stored in the memory 22 .

FIG. 5 is a diagram showing an example of the relationship between the total reliability of feature points and the certainty, which is shown in the reference table. In FIG. 5, the horizontal axis represents the total reliability, and the vertical axis represents the certainty. A curve 500 represents the relationship between the sum of reliability and confidence. In this example, the relationship between the sum of the reliability of the feature points and the confidence is calculated so that the confidence increases monotonically with respect to the sum of the reliability, and the confidence increases sharply as the sum of the confidence increases. is set.

Note that the relationship between the sum of reliability and confidence is not limited to the example shown in FIG. For example, the relationship between the sum of reliability and confidence may be set such that the confidence increases linearly as the sum of confidence increases. Alternatively, the relationship between the total reliability and the confidence may be set such that the greater the total confidence, the more moderate the degree of increase in the confidence. Furthermore, the relationship between the sum of confidence levels and the confidence level may be different for each facial organ. For example, with respect to the eye, the sum of the reliability of the feature points and the confidence are calculated so that the confidence monotonically increases with respect to the sum of the confidence and that the confidence increases sharply as the sum of the confidence increases. relationship is set. On the other hand, for the nose or mouth, the relationship between the total reliability and the confidence may be set such that the greater the total reliability, the more moderate the degree of increase in the confidence.

The certainty calculation unit 35 uses the statistical representative value of the certainty for each organ of the face as the certainty representing the certainty that the driver's face is actually represented in the estimated facial region. Note that the certainty calculation unit 35 calculates, for example, the average value of the certainty for each organ of the face, or the minimum value of the certainty for each organ of the face, as the confidence for each organ of the face. It can be a statistical representative value.

Every time the certainty factor calculation unit 35 calculates the certainty factor for the estimated face area, it notifies the re-detection determining unit 36 and the state determining unit 37 of the calculated certainty factor for the face area.

A re-detection determination unit 36 determines whether or not to re-detect the face region from the face image based on the confidence factor calculated for the estimated face region. For example, when the certainty is lower than the detection determination threshold, the redetection determination unit 36 determines that there is a high possibility that the estimated face area does not actually represent the driver's face. Therefore, when the certainty is equal to or less than the detection determination threshold, the re-detection determination unit 36 performs processing for detecting a face region in a face image including the estimated face region or a face image to be obtained next. The face detection unit 31 is instructed to execute. Thereby, the processor 23 can re-detect the driver's face from the face image when tracking of the face region fails.

The state determination unit 37 determines the state of the driver based on the facial area represented in each facial image or the estimated facial area. However, when the certainty calculated by the certainty calculating unit 35 is lower than the detection determination threshold, there is a high possibility that the estimated face area does not actually represent the driver's face. Therefore, in this case, the state determination unit 37 does not have to execute the process of determining the state of the driver. Then, the state determination unit 37 outputs the result of the state determination of the past face image obtained immediately before the latest face image including the face region whose certainty is lower than the detection determination threshold, when the latest face image is acquired. , may be the state of the driver.

In the present embodiment, the state determination unit 37 compares the direction of the driver's face shown in the face area with the reference direction of the driver's face to determine whether the driver's state is suitable for driving the vehicle 10 or not. determine whether Note that the reference direction of the face is stored in the memory 22 in advance.

The state determination unit 37 fits the feature points of the face detected by the feature point detection unit 34 to a three-dimensional face model representing the three-dimensional shape of the face. Then, the state determination unit 37 detects the face orientation of the three-dimensional face model when each feature point is best fitted to the three-dimensional face model as the face orientation of the driver. Alternatively, the state determination unit 37 may detect the orientation of the driver's face based on the face image according to another technique for determining the orientation of the face shown in the image. The orientation of the driver's face is represented by, for example, a combination of the pitch angle, yaw angle, and roll angle with reference to the direction facing the driver monitor camera 2 .

The state determination unit 37 calculates the absolute value of the difference between the orientation of the driver's face represented in the face area and the reference direction of the driver's face, and compares the absolute value of the difference with a predetermined permissible face orientation range. Then, when the absolute value of the difference is out of the allowable face orientation range, the state determination unit 37 determines that the driver is looking away, that is, the driver is not in a state suitable for driving the vehicle 10 .

It should be noted that the driver may face a direction other than the front of the vehicle 10 in order to check the situation around the vehicle 10 . However, even in such a case, if the driver is concentrating on driving the vehicle 10, the driver will not continue to face the vehicle 10 in any direction other than the front. Therefore, according to the modified example, the state determination unit 37 determines that the period in which the absolute value of the difference between the orientation of the driver's face and the reference direction of the driver's face is outside the allowable face orientation range is a predetermined time (for example, several seconds). If this continues, it may be determined that the driver's condition is not suitable for driving the vehicle 10 .

When the state determination unit 37 determines that the driver's state is not suitable for driving the vehicle 10 , the state determination unit 37 generates warning information including a warning message to warn the driver to face the front of the vehicle 10 . The state determination unit 37 then outputs the generated warning information to the user interface 3 via the communication interface 21, thereby causing the user interface 3 to display the warning message. Alternatively, the state determination unit 37 causes the speaker included in the user interface 3 to output a sound warning the driver to face the front of the vehicle 10 .

FIG. 6 is an operation flowchart of driver monitor processing executed by the processor 23 . The processor 23 may execute driver monitor processing including face detection processing according to the following operation flowchart. In the operation flowchart shown below, the processing of steps S101 to S109 corresponds to the face detection processing.

The face detection unit 31 of the processor 23 detects a face area from a face image obtained at a predetermined timing by the driver monitor camera 2 (step S101). In addition, the tracking unit 32 of the processor 23 performs facial area tracking processing on the facial image that follows the facial image obtained at a predetermined timing, thereby estimating the facial area in the subsequent facial image ( step S102).

The segmentation unit 33 of the processor 23 classifies each pixel included in the estimated face region in the subsequent face image as a face pixel or a non-face pixel (step S103). Further, the feature point detection unit 34 of the processor 23 detects a plurality of feature points for each organ of the face from the face area estimated in the subsequent face image (step S104).

The certainty calculation unit 35 of the processor 23 sets the first reliability to the feature points that are face pixels, and sets the second reliability to the feature points that are non-face pixels, with respect to the face area estimated in the subsequent face image. degree is set (step S105). Note that, as described above, the second reliability is set lower than the first reliability. Furthermore, the certainty calculation unit 35 calculates the sum of the reliability of each feature point detected for each organ of the face with respect to the facial region estimated in the subsequent facial image, and calculates the sum of the reliability. The degree of certainty of the organ is calculated so that it becomes higher as the is larger (step S106). Then, the confidence calculation unit 35 calculates the statistical representative value of the confidence calculated for each facial organ as the confidence that the driver's face is actually represented in the estimated facial region (step S107). .

The redetection determination unit 36 of the processor 23 determines whether or not the certainty is less than the detection determination threshold (step S108). If the certainty is less than the detection determination threshold (step S108-Yes), the re-detection determination unit 36 instructs the face detection unit 31 to detect a face area from the latest face image or the next obtained face image. (step S109).

On the other hand, if the certainty is equal to or greater than the detection determination threshold (step S108-No), the state determination unit 37 of the processor 23 detects the direction of the driver's face from the estimated face area, and determines that the driver's state is the vehicle 10. (step S110). Then, the state determination unit 37 executes warning processing or the like according to the determination result. After step S109 or S110, processor 23 terminates the driver monitor process.

As described above, the face detection apparatus is designed so that, in an image obtained by photographing the face of a person to be photographed, the face of the person is assumed to be included in a predetermined area. Each pixel is classified as a face pixel that represents a face or a non-face pixel that does not represent a face. Also, this face detection apparatus detects a plurality of feature points of each organ of the face from the predetermined area, and sets reliability for each of the detected feature points. At this time, the face detection apparatus sets the reliability of each feature point so that the reliability of the feature points that are face pixels is higher than the reliability of the feature points that are non-face pixels. This face detection apparatus obtains the sum of the reliability of the feature points detected for each organ of the face, and the larger the sum of the obtained reliability of each organ, the greater the predetermined region. The degree of certainty is calculated so that the degree of certainty representing the certainty that the face is represented in is high. As a result, the face detection device can accurately determine the likelihood that a face is represented in a predetermined area on the image even if a part of the face is not represented in the image.

Depending on the assumed positional relationship between the driver monitor camera 2 and the driver's face, there are facial organs whose reliability tends to be high and facial organs whose reliability tends to be low. For example, when the driver is facing the front of the vehicle 10 and the driver monitor camera 2 is installed so as to photograph the driver's face from the right side, the organs located on the right half of the driver's face are captured by the driver. Since it faces the monitor camera 2, the reliability tends to be high. Conversely, an organ located on the left half of the driver's face may not be partially represented on the face image because, for example, it is hidden by another part of the face. As a result, reliability tends to be low. Therefore, the certainty calculation unit 35 obtains the distribution of the sum of reliability for each organ based on a plurality of face images, and based on the distribution, for organs whose reliability tends to be low, the set reliability is calculated. A bias may be applied or an offset may be added. Conversely, for an organ whose reliability tends to be high, the certainty calculation unit 35 may subtract the offset from the reliability set for the feature point of that organ. In addition, the certainty calculation unit 35 determines the offset value of the reliability for each organ so that the average value or the median value of the reliability for each organ is constant according to the distribution of the sum of the reliability. A table may be generated and the confidence of each feature point corrected according to the lookup table.

In addition, the state determination unit 37 selects an index representing the state of the driver other than the orientation of the driver's face from the face region having the degree of certainty greater than or equal to the detection determination threshold, that is, the face region that is highly likely to represent a face. You may ask. For example, the state determination unit 37 may estimate the line-of-sight direction or wakefulness of the driver. When estimating the line-of-sight direction, the state determination unit 37 detects the pupil of one of the eyes of the driver and the corneal reflection image (Purkinje image) of the light source provided in the driver monitor camera 2 from the face area. At that time, the state determination unit 37 detects the pupil and the Purkinje image by template matching, for example. Then, the state determination unit 37 refers to a reference table representing the positional relationship between the center of gravity of the pupil and the Purkinje image, and the relationship between the line-of-sight direction and the position of the driver's position based on the detected position of the center of gravity of the pupil and the position of the Purkinje image. What is necessary is just to estimate a line-of-sight direction. In this case, the state determination unit 37 calculates the absolute value of the difference between the line-of-sight direction of the driver and the reference direction of the line-of-sight direction of the driver, and determines the absolute value of the difference as the predetermined line-of-sight direction tolerance. Compare with range. If the absolute value of the difference is outside the permissible line-of-sight direction range, the state determination unit 37 determines that the driver is looking away, that is, the driver is not in a state suitable for driving the vehicle 10. good.

Further, when estimating the driver's arousal level, the state determination unit 37 detects the distance between the feature point corresponding to the upper eyelid and the feature point corresponding to the lower eyelid detected by the feature point detection unit 34. , to calculate the degree of eye opening. Then, the state determination unit 37 may estimate the degree of wakefulness of the driver based on the temporal change in the eye opening degree.

In addition, when the face detection device is used for purposes other than the driver monitor device, the state determination unit 37 determines that the face region whose certainty is equal to or higher than the detection determination threshold, that is, the face of the person to be photographed is represented. Different processing from the above embodiment may be performed based on the likely face region. For example, the state determination unit 37 may perform processing such as personal authentication, skin quality determination, gender, attribute determination of a person to be photographed, body temperature estimation, face-based autofocus control, exposure control, or color correction. good. However, in this case as well, the state determination unit 37 may skip the above processing for face regions whose confidence is less than the detection determination threshold.

According to another variation, the face detector 31 may be capable of applying multiple face detection techniques. For example, the face detection unit 31 may use a face detection method using a DNN-based classifier and a face detection method using an AdaBoost-based classifier. Then, the re-detection determination unit 36, when the reliability of a predetermined number (2 or more, for example, 3 to 10) of continuously obtained face images is less than the re-detection determination threshold value, the face detection unit 31 It may be instructed to use a face detection method different from the face detection method that has been applied before. Alternatively, the redetection determination unit 36 determines that the facial images do not represent the face of the person to be photographed when the certainty factor is less than the redetection determination threshold for a predetermined number of consecutively obtained face images. , the face detection process may be stopped. Furthermore, the re-detection determination unit 36 may turn off the power of the face detection device, or put the face detection device into a standby state.

Further, the computer program that implements the functions of the processor 23 of the ECU 4 according to the above embodiment or modification is recorded in a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. may be provided in

As described above, those skilled in the art can make various modifications within the scope of the present invention according to the embodiment.

1 vehicle control system 10 vehicle 2 driver monitor camera 3 user interface 4 electronic control unit (ECU)
21 communication interface 22 memory 23 processor 31 face detection unit 32 tracking unit 33 segmentation unit 34 feature point detection unit 35 confidence calculation unit 36 redetection determination unit 37 state determination unit

Claims (1)

a segmentation unit that classifies each pixel in a predetermined region on the image into a face pixel representing a face or a non-face pixel not representing a face;
a feature point detection unit that detects a plurality of feature points of each individual organ of the face from the predetermined region;
For each organ of the face, among a plurality of feature points detected for the organ, the reliability of the feature points that are the facial pixels is higher than the reliability of the feature points that are the non-facial pixels, A reliability is set for each of the plurality of feature points, a sum of the reliability set for each of the plurality of feature points is obtained, and the larger the sum of the obtained reliability for each of the individual organs is, the a certainty calculation unit that calculates the certainty so as to increase the certainty that indicates the likelihood that a face is represented in a predetermined area;
A face detection device having

Images