JP2009296355A - Facial image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a facial image processing apparatus capable of notifying a driver about the state of the apparatus. <P>SOLUTION: The facial image processing apparatus includes a picked-up image capturing means, a featured portion detecting means and a notification means. The picked-up image capturing means captures a picked-up image from an image-pickup means which images the face of a driver of a vehicle. The featured portion detecting means detects the featured portion of the face which is being imaged in the picked-up image. When the featured portion detecting means of the facial image processing apparatus cannot detect any featured portion, the notification means notifies the driver of the featured portion. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a face image processing apparatus, and more particularly to a face image processing apparatus that captures an image of the driver from the front of the driver who gets on the vehicle.

  2. Description of the Related Art Conventionally, a system has been developed in which a driver who drives a vehicle is imaged and the state of the driver is monitored using the captured image. For example, a camera that captures images from the front of the driver is installed near the steering wheel of the vehicle, and the state of the driver is analyzed using images captured by the camera.

For example, a face state detection device that determines the orientation of a driver's face using a captured image obtained by imaging the driver is disclosed (see, for example, Patent Document 1). The face state detection device disclosed in Patent Document 1 counts the number of times a driver's facial feature is recognized, and selects a feature for detecting the orientation of the face based on the count. When the driver wears wearing items such as sunglasses or a mask, any of the features of the driver's face cannot be detected. Is recognized as a feature part candidate and the face orientation is detected.
JP 2003-296712 A

  However, in the face state detection device disclosed in Patent Document 1, the face state detection device does not normally notify a driver or the like of a feature portion of the driver's face that is not recognized by the face state detection device. The driver cannot grasp whether or not it is operating. Therefore, even if the face state detection device is not operating normally, a state in which the driver performs driving depending on the operation of the device can be considered. Further, since the driver cannot grasp whether or not the face state detection device is operating normally, there is a countermeasure that the driver himself can take in order to operate the face state detection device normally. However, since the countermeasure cannot be recognized, as a result, the state where the face state detection device is not normally operated continues.

  Therefore, an object of the present invention is to provide a face image processing apparatus that can appropriately notify the driver of the state of the apparatus.

In order to achieve the above object, the present invention has the following features.
1st invention is a face image processing apparatus provided with a captured image acquisition means, a characteristic part detection means, and an alerting | reporting means. The captured image acquisition unit acquires the captured image from the imaging unit that captures the face of the driver of the vehicle. The feature detection unit detects the feature of the face captured in the captured image. When the face image processing device feature detection unit cannot detect any feature, the notification unit notifies the driver of the feature.

  In a second aspect based on the first aspect, the informing means informs the driver of the presumed cause that the feature portion could not be detected, and informs the driver of the feature portion that could not be detected by the feature portion detecting means.

  In a third aspect based on the second aspect, the notifying means notifies the driver of the characteristic part and the estimated cause that the characteristic part detecting means could not detect, and also when the driver eliminates the estimated cause. Inform the driver of the precautions.

  In a fourth aspect based on the second aspect, the presumed cause is that the driver is wearing a wear that is supposed to block the feature as viewed from the imaging means.

  In a fifth aspect based on the second aspect, a storage means is further provided. The storage means stores in advance presumed cause information that cannot be detected for each feature. When the feature detection unit cannot detect any feature, the notification unit notifies the driver of the estimated cause information stored in the storage unit corresponding to the feature as the estimated cause.

  In a sixth aspect based on the second aspect, a storage means is further provided. The storage unit stores in advance presumed cause information that cannot be detected for each feature and precautionary information for eliminating the presumed cause. The notification means, when the feature detection means cannot detect any feature, the estimated cause information and the caution information stored in the storage means corresponding to the feature as the estimated cause and caution, respectively. Notify the driver.

  In a seventh aspect based on the first or second aspect, the captured image acquisition means acquires captured images from the imaging means at predetermined intervals. The feature detection unit detects the feature captured in the captured image every time the captured image acquisition unit acquires the captured image. The notification means notifies the driver of the feature portion when the feature portion detection means cannot continuously detect the same feature portion for the first time or more.

  In an eighth aspect based on the first or second aspect, the captured image acquisition unit acquires the captured image from the imaging unit at predetermined intervals. The feature detection unit detects the feature captured in the captured image every time the captured image acquisition unit acquires the captured image. The informing means stops informing the driver of the feature section after the feature section has been informed to the driver, and the feature section detection means has not been able to detect the feature section continuously for a second time or longer. To do.

  In a ninth aspect based on the eighth aspect, after the notification means stops notifying the driver of the feature portion, the driver displays abnormality information indicating that the face image processing device is not operating normally. To inform.

  In a tenth aspect based on the first or second aspect, the captured image acquisition means acquires captured images from the imaging means at predetermined intervals. The feature detection unit detects the feature captured in the captured image every time the captured image acquisition unit acquires the captured image. The notifying means stops notifying the driver of the feature portion when the feature portion detecting means is able to detect the feature portion after notifying the driver of the feature portion.

  According to the first aspect of the present invention, when the feature portion of the driver's face cannot be detected from the captured image, the operation state of the face image processing device is driven to notify the driver of the feature portion that could not be detected. Can confirm.

  According to the second aspect, when the feature portion of the driver's face cannot be detected from the captured image, the estimated cause that cannot be detected together with the feature portion is notified. Can be easily resolved.

  According to the third aspect of the invention, since the precautions when eliminating the estimated cause that could not be detected are further notified, it is possible to avoid a dangerous state that may occur when the estimated cause is resolved, It becomes possible to promote safer driving.

  According to the fourth aspect of the present invention, when the feature portion of the driver's face cannot be detected from the captured image, the driver cannot find the cause together with the feature portion because the driver's own clothing is known. it can.

  According to the fifth aspect, the cause of estimation for each feature can be easily notified by storing the cause of estimation for each feature.

  According to the sixth aspect of the invention, the estimated cause and the estimated cause for each feature are easily resolved by storing in advance the estimated cause and the precautions for eliminating the estimated cause for each feature. It is possible to notify the precautions.

  According to the seventh aspect, it is possible to suppress notification of undetected feature portions that are temporarily generated by the driver or the driving environment, and to reduce the driver's troublesomeness due to frequent notification. can do.

  According to the eighth aspect of the present invention, it is possible to reduce the driver's troublesomeness caused by continuously informing the feature that could not be detected.

  According to the ninth aspect, it is possible to alleviate the driver's troublesomeness caused by continuously informing the feature that could not be detected and to promote safer driving by displaying the abnormality information. .

  According to the tenth aspect of the present invention, when the cause that the feature portion cannot be detected from the captured image is resolved, the notification to the feature portion can be stopped and the normal processing can be resumed.

  Hereinafter, a face image processing apparatus according to an embodiment of the present invention will be described with reference to FIG. In the present embodiment, an example in which a driver support system including the face image processing apparatus is mounted on a vehicle will be described. As an example, the driver support system images the face of the driver driving the vehicle, determines the driver's state (face orientation, eye open / closed degree, etc.) based on the captured image, and responds to the determination result. Control the vehicle. The driver support system also recognizes other vehicles and obstacles around the vehicle, determines the risk of collision, and controls the vehicle according to the determination result. FIG. 1 is a block diagram illustrating an example of a functional configuration of a driver support system including the face image processing apparatus.

  In FIG. 1, the driver support system includes a driver monitor camera 1, a driver monitor ECU (Electrical Control Unit) 2, a driver support system ECU 3, a millimeter wave radar 4, a display unit 5, an audio amplifier 6, a speaker 61, a brake control ECU 7, an alarm. A buzzer 71 and a brake ACT (actuator) 72 are provided. The driver monitor ECU 2, the driver support system ECU 3, and the millimeter wave radar 4 are connected to each other via a CAN (Controller Area Network) 1 or the like. The driver support system ECU 3, the display unit 5, the audio amplifier 6, and the brake control ECU 7 are connected via the CAN 2 and the like. The driver monitor ECU 2 corresponds to an example of the face image processing apparatus of the present invention.

  The driver monitor camera 1 includes a CCD (Charge Coupled Device) 11 and an LED (Light Emitting Diode) 12 that are operated by power supplied from the driver monitor ECU 2. For example, LED12 is an illuminating device which irradiates near infrared light toward the driver | operator from the driver | operator's front. In this case, the CCD 11 is an imaging device having high sensitivity characteristics in the near infrared region, and is typically a near infrared CCD camera. In this way, by using the CCD 11 having high sensitivity characteristics in the near-infrared region, it is possible to image the driver illuminated by the LED 12 with high sensitivity even when the vehicle is dark at night or in a tunnel. It is. The CCD 11 is disposed at a position where the driver's face can be imaged from the front. Then, the CCD 11 captures an image of the driver's face and its surroundings at predetermined intervals, and outputs the captured image to the driver monitor ECU 2. Details of the installation position of the driver monitor camera 1 will be described later.

  The driver monitor ECU 2 includes an information processing device such as a plurality of microcomputers (hereinafter referred to as microcomputers), a memory for storing various data used for processing, a power supply circuit for supplying power to the driver monitor camera 1, and an interface. A processing apparatus including a circuit and the like. For example, the driver monitor ECU 2 uses the captured image captured by the driver monitor camera 1 to determine the driver's face orientation, the driver's eye open / closed degree, and the result of the determination. Output to the support system ECU3. Further, the driver monitor ECU 2 cannot detect a characteristic portion of the driver's face (for example, the entire face, nostril, eyelid, etc.) necessary for determining the driver's face orientation, eye open / closed degree, or the like from the captured image. In this case, information for notifying the status of the device resulting from the non-detection is output to the driver support system ECU 3.

  The millimeter wave radar 4 emits a millimeter wave toward the front, rear, obliquely forward direction, etc. of the vehicle, and receives the radio wave reflected from the object. Then, based on the received radio wave, the millimeter wave radar 4 measures the position of other vehicles and obstacles existing around the vehicle, the relative speed with the own vehicle, and the like, and sends the result to the driver support system ECU 3. Output.

  The driver support system ECU 3 is based on information such as the driver's face orientation and eye open / closed degree output from the driver monitor ECU 2, and recognition results of vehicles and obstacles around the vehicle output from the millimeter wave radar 4. Thus, the characteristics of the occupant protection device mounted on the vehicle are appropriately adjusted, the collision condition mitigation system is operated, and an appropriate warning is given to the driver. Further, the driver support system ECU 3 notifies the driver of the information based on the information for notifying the state of the device resulting from the undetected feature portion. In FIG. 1, a display unit 5, an audio amplifier 6, and a brake control ECU 7 are described as examples of devices controlled by the driver support system ECU 3.

  The display unit 5 is provided at a position that is visible to a driver who sits in the driver's seat of the vehicle and drives the vehicle. For example, the display unit 5 is provided on an instrument panel (instrument panel) in front of the driver's seat, and displays alarms and information corresponding to instructions from the driver support system ECU 3 to the driver. For example, when it is determined that the driver is not facing the front or eyes are closed and there is a risk of collision between the vehicle and another object, the driver support system ECU 3 displays the driving on the display unit 5. Display prompting the user to avoid collision (early warning). Moreover, the display part 5 displays the information for notifying a driver | operator of the condition, when the apparatus is not operating normally by the non-detection of the said characteristic part. Typically, the display unit 5 is composed of a combination meter in which several main instruments, indicator lights, warning lights, and a multi-information display for displaying various information are combined in one panel. The display unit 5 includes a half-mirror (reflective glass) provided on a part of the windshield in front of the driver's seat, and a head-up display that displays a virtual image such as information on the half-mirror. You may comprise with the display device of.

  The audio amplifier 6 outputs an audio signal amplified according to the audio information output from the driver support system ECU 3 to the speaker 61, and notifies the driver of the audio information via the speaker 61.

  The brake control ECU 7 controls the operation of the alarm buzzer 71 and the brake ACT 72 mounted on the vehicle. For example, if the driver support system ECU 3 determines that the driver is not facing the front or eyes are closed, and the driver support system ECU 3 determines that there is a risk of a collision between the vehicle and another object, the brake control ECU 7 Activates the alarm buzzer 71 at an early stage and prompts the driver to perform a collision avoidance operation (pre-warning of the alarm). As a result, the driver can quickly perform the collision avoidance operation by canceling the driving state such as the side look. The brake control ECU 7 controls the operation of the brake ACT 72 so as to pressurize and apply the brake hydraulic pressure in accordance with the force with which the driver steps on the brake pedal (alarm brake). As a result, the hydraulic response of the brake ACT 72 is improved, and the speed of the vehicle can be reduced.

  Next, the driver monitor camera 1 will be described with reference to FIGS. FIG. 2 is a schematic side view showing an example of the driver monitor camera 1 installed in the vehicle as viewed from the side of the driver's seat. FIG. 3 is a schematic front view showing an example of the driver monitor camera 1 installed in the vehicle as viewed from the steering wheel 91 side.

  2 and 3, the driver monitor camera 1 is installed on the upper surface of the steering column 92. Then, the driver monitor camera 1 passes through the opening of the steering wheel 91 (for example, an opening in which the spokes 912 and the like are not provided inside the rim 911), and performs the driving from the front of the driver who operates the steering wheel 91. The imaging direction is adjusted so as to image the person's face. In FIG. 2, a part of the viewing volume (part of the camera angle of view) indicating the visual field of the driver monitor camera 1 is indicated by a broken line, and the driver's head is included in the viewing volume. In addition, the imaging direction of the driver monitor camera 1 is adjusted. Here, as described above, the driver monitor camera 1 is provided with the LED 12, but the LED 12 also emits near-infrared light from the front of the driver toward the driver, similarly to the imaging direction. The LED 12 emits near-infrared light in a range wider than at least the camera angle of view of the driver monitor camera 1. As described above, since the driver monitor camera 1 is installed on the steering column 92, the driver monitor camera 1 can take an image of the driver from a very close distance in front of the driver, and the driver's face on the vehicle can be accurately captured. Imaging can be performed.

  Next, before describing a specific operation of the driver monitor ECU 2, an example of main data used in the operation will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of main data stored in the memory of the driver monitor ECU 2.

  In FIG. 4, in the memory of the driver monitor ECU 2, captured image data Da, face undetected flag Db, nostril undetected flag Dc, wrinkle undetected flag Dd, notification display flag De, face undetected count data Df, nostril undetected Count data Dg, wrinkle detection count data Dh, notification display count data Di, eye open / closed degree data Dj, notification information data Dk, abnormality information data Dl, and the like are stored.

  The captured image data Da stores data indicating an image captured by the driver monitor camera 1 and is updated at predetermined processing cycles.

  The face undetected flag Db stores data indicating the state of the face undetected flag Ff that is set to ON when the driver's face cannot be detected in the captured image. The nostril undetected flag Dc stores data indicating the state of the nostril undetected flag Fn which is set to ON when the driver's nostril cannot be detected in the captured image. The heel undetected flag Dd stores data indicating the state of the heel undetected flag Fe that is set to ON when a driver's heel is not detected in the captured image. The notification display flag De stores data indicating the state of the notification display flag Fd set to ON when a notification indicating that the driver support system is not operating normally is displayed on the display unit 5.

  The face undetected count data Df stores data indicating the time during which the driver's face cannot be detected in the captured image (face undetected count value Cf). The nostril undetected count data Dg stores data indicating the time during which the driver's nostril cannot be detected in the captured image (nostril undetected count value Cn). As the wrinkle undetected count data Dh, data indicating a time during which a state in which a driver's wrinkle cannot be detected in the captured image continues (wrinkle undetected count value Ce) is stored. The notification display count data Di stores data indicating the time (notification display count value Cd) during which the notification indicating that the driver support system is not operating normally is displayed on the display unit 5.

  The eye open / closed degree data Dj stores data indicating the open / closed degree of the driver's eyes calculated using the captured image.

  In the notification information data Dk, data indicating notification information indicating that the driver support system has not detected a feature in the captured image is stored. Specifically, the notification information data Dk is a notification corresponding to the feature portion that could not be detected in case the feature portion of the driver's face (for example, the entire face, nostril, eyelid, etc.) cannot be detected from the captured image. Information is stored. Hereinafter, an example of the notification information data Dk will be described with reference to FIG.

  In FIG. 5, for example, a notification information table in which notification information to be notified to the driver is described in accordance with a feature portion that has not been detected by the driver monitor ECU 2 is recorded in the notification information data Dk. In the notification information table, notification information Ia to Ic is described for undetected features that could not be detected in the captured image. Here, the notification information Ia is information for notifying the driver of the feature portion of the driver's face that the driver monitor ECU 2 could not detect from the captured image. The notification information Ib is information for notifying the driver of the estimated cause that the characteristic portion has not been detected. The notification information Ic is information for notifying the driver of precautions when the driver eliminates the estimated cause. Hereinafter, examples of the notification information Ia to Ic for each undetected feature will be listed and described.

  In the notification information table, “face could not be detected” is described as the notification information Ia for the undetected feature “face”. Also, in the notification information table, “is there anything to block with the camera?” Is described as the notification information Ib for the undetected feature “face”. Further, the notification information table describes “Please stop the vehicle at a place where there is no danger and remove the obstacle” as the notification information Ic for the undetected feature “face”.

  In the notification information table, “not detected nostril” is described as notification information Ia for the undetected feature “nasal nostril”. In the notification information table, “Do you wear a mask?” Is described as notification information Ib for the undetected feature “nasal nostril”. Further, in the above notification information table, the notification information Ic for the undetected feature “nasal nose” is described as “If the mask can be removed, stop the vehicle at a place where there is no danger and remove the mask”. Is done.

  In the notification information table, “not detected 瞼” is described as the notification information Ia for the undetected feature “瞼”. In the notification information table, “is sunglasses, glasses, or hat worn? Are bangs not visible?” Is described as notification information Ib for the undetected feature “瞼”. . Furthermore, in the above notification information table, “if the wearable item can be removed, stop the vehicle and remove it when there is no danger” is described as the notification information Ic for the undetected feature “瞼”. The

  In the notification information table, “brows could not be detected.” Is described as the notification information Ia for the undetected feature “brow”. Also, in the notification information table, “is sunglasses, glasses, or hat worn? Is bangs not on the eyebrows?” Is described as the notification information Ib for the undetected feature “eyebrow”. . Further, in the notification information table described above, “if you can remove the wear, please stop and remove the car if there is no danger” as the notification information Ic for the undetected feature “brow”. The

  In the notification information table, “Mouth could not be detected.” Is described as the notification information Ia for the undetected feature “mouth”. In the notification information table, “is a mask worn?” Is described as the notification information Ib for the undetected feature “mouth”. Further, in the notification information table, the notification information Ic for the undetected feature “mouth” is described as “If the mask can be removed, stop the vehicle at a place where there is no danger and remove the mask”. Is done.

  In the notification information table, “nose could not be detected” is described as the notification information Ia for the undetected feature “nose”. In the notification information table, “is a mask worn?” Is described as notification information Ib for the undetected feature “nose”. Further, in the notification information table, the notification information Ic for the undetected feature “nose” is described as “If the mask can be removed, stop the car at a place where there is no danger and remove the mask”. Is done.

  In the notification information table, “ears could not be detected” is described as the notification information Ia for the undetected feature “ear”. Also, in the notification information table, “not wearing a hat or earpiece? Is your hair applied to the ear?” Is described as the notification information Ib for the undetected feature “ear”. Further, in the above notification information table, “if the wearable item can be removed, please stop and remove the car if there is no danger” as the notification information Ic for the undetected feature “ear”. The

  Returning to FIG. 4, the abnormality information data Dl stores data indicating abnormality information indicating that the driver support system is not operating normally. For example, in the abnormality information data Dl, data indicating information prompting the driver to drive safely is recorded because the driver support system is not operating normally.

  Next, an example of the operation of the driver monitor ECU 2 will be described with reference to FIGS. 6 and 7. FIG. 6 is a first half flowchart showing an example of processing executed by the driver monitor ECU 2. FIG. 7 is a second half flowchart showing an example of processing executed by the driver monitor ECU 2. In the flowcharts shown in FIGS. 6 and 7, the driver's face, nostril, and wrinkle will be described as an example of the characteristic parts to be notified. Each step of the flowcharts shown in FIGS. 6 and 7 is performed by the driver monitor ECU 2 executing a predetermined program. Note that a program for executing these processes is stored in advance in a storage area (for example, a memory, a hard disk, an optical disk, etc.) provided in the driver monitor ECU 2, and the driver monitor ECU 2 is turned on. When this happens, it is executed by the driver monitor ECU 2.

  In FIG. 6, the driver monitor ECU 2 performs initial setting (step S51), and proceeds to the next step. Specifically, the driver monitor ECU 2 initializes each parameter and the like stored in the memory in step S51. For example, the driver monitor ECU 2 initializes the states of the face undetected flag Ff, the nostril undetected flag Fn, the eyelid not detected flag Fe, and the notification display flag Fd to OFF. Further, the driver monitor ECU 2 initializes the face undetected count value Cf, the nostril undetected count value Cn, the wrinkle undetected count value Ce, and the notification display count value Cd to 0.

  Next, the driver monitor ECU 2 acquires a captured image from the driver monitor camera 1, updates the captured image data Da (step S52), and proceeds to the next step.

  Next, the driver monitor ECU 2 determines whether or not the driver's face can be detected in the captured image stored in the captured image data Da in step S52 (step S53). The driver monitor ECU 2 recognizes the contour of the driver's face, the center line, the position of the facial feature, the situation of the feature, etc., from the captured image stored in the captured image data Da, as will be apparent from the following. Yes. For example, in step S53, the driver monitor ECU 2 determines that the face cannot be detected when the driver's face is not captured in the captured image or when the driver's face cannot be recognized. Then, if the driver's face can be detected, the driver monitor ECU 2 advances the process to the next step S54. On the other hand, if the driver's face cannot be detected, the driver monitor ECU 2 advances the process to the next step S57.

  In step S54, the driver monitor ECU 2 refers to the face non-detection flag Db and determines whether or not the state of the face non-detection flag Ff is ON. Then, when the face undetected flag Ff is ON, the driver monitor ECU 2 changes the state of the face undetected flag Ff to OFF and updates the face undetected flag Db (step S55). Is displayed normally (step S56), and the process proceeds to the next step S62. On the other hand, when the state of the face non-detection flag Ff is OFF, the driver monitor ECU 2 proceeds to the next step S62 as it is.

  Here, as will be apparent from the description below, if the face non-detection flag Ff is ON at the time of determination in step S54, it indicates that the driver monitor ECU 2 has not been able to detect the driver's face from the captured image until then. ing. That is, at the time of the determination, the driver monitor ECU 2 has changed from a state where the driver's face cannot be detected from the captured image to a state where the face can be detected. Therefore, in step S55, the state of the face non-detection flag Ff is changed to a state indicating that face detection is possible (that is, OFF), and in step S56, the notification indicating that no face has been detected so far. Is displayed on the display unit 5, a return process for returning to the normal display is enabled. In the process of step S56, the display unit 5 may already be in the normal display state. In this case, the driver monitor ECU 2 proceeds to the next step S62 while maintaining the normal display.

  On the other hand, when the driver's face cannot be detected from the captured image (No in step S53), the driver monitor ECU 2 refers to the face non-detection flag Db and determines whether or not the state of the face non-detection flag Ff is ON. (Step S57). When the face undetected flag Ff is ON, the driver monitor ECU 2 refers to the face undetected count data Df, adds 1 to the face undetected count value Cf, and updates the face undetected count data Df. (Step S58), the process proceeds to the next step S61. On the other hand, when the state of the face undetected flag Ff is OFF, the driver monitor ECU 2 changes the state of the face undetected flag Ff to ON and updates the face undetected flag Db (step S59), and the face undetected count data The face undetected count value Cf for Df is set to 1 (step S60), and the process proceeds to the next step S61.

  In step S61, the driver monitor ECU 2 determines whether or not the face undetected count value Cf of the face undetected count data Df is greater than or equal to a predetermined threshold C1. Here, the threshold value C1 is a value for determining whether the situation in which the driver's face cannot be detected from the captured image is a temporary phenomenon or a continuous phenomenon, and is a value obtained from various experiments and empirical rules. Is set. Then, if the face undetected count value Cf is equal to or greater than the threshold value C1, the driver monitor ECU 2 advances the process to the next step S91 (FIG. 7). On the other hand, when the face undetected count value Cf is less than the threshold value C1, the driver monitor ECU 2 returns to step S52 and repeats the process.

  Here, in the processing of steps S57 to S61, when the driver monitor ECU 2 changes from a state where the driver's face can be detected from the captured image to a state where the face cannot be detected, the state of the face undetected flag Ff is turned ON (step S59), the elapsed time after the face undetected flag Ff is turned on is counted as the face undetected count value Cf (steps S58 and S60), and whether or not the face undetected count value Cf has reached the threshold value C1. This is a process for determining (step S61). That is, in the processing of steps S57 to S61, when the driver monitor ECU 2 continues to be in a state where the driver's face cannot be detected from the captured image, and when the time reaches the threshold C1, step S91 and subsequent steps. Will be processed.

  In step S62, the driver monitor ECU 2 determines whether or not the driver's nostril can be detected in the captured image stored in the captured image data Da in step S52. For example, in step S62, the driver monitor ECU 2 determines that the nostril cannot be detected when the driver's nostril is not captured in the captured image or when the driver's nostril cannot be recognized. Then, when the driver's nostril can be detected, the driver monitor ECU 2 advances the process to the next step S63. On the other hand, if the driver's nostril cannot be detected, the driver monitor ECU 2 advances the process to the next step S66.

  In step S63, the driver monitor ECU 2 refers to the nostril non-detection flag Dc to determine whether or not the nostril non-detection flag Fn is ON. Then, when the state of the nostril undetected flag Fn is ON, the driver monitor ECU 2 changes the state of the nostril undetected flag Fn to OFF and updates the nostril undetected flag Dc (step S64). Is displayed normally (step S65), and the process proceeds to the next step S71. On the other hand, when the state of the nostril non-detection flag Fn is OFF, the driver monitor ECU 2 proceeds to the next step S71 as it is.

  Here, as will be apparent later, if the nostril detection flag Fn is ON at the time of the determination in step S63, it indicates that the driver monitor ECU 2 has not been able to detect the driver's nostril from the captured image until then. ing. That is, at the time of the determination, the driver monitor ECU 2 has changed from a state in which the driver's nostril cannot be detected from the captured image to a state in which the nostril can be detected. Therefore, in step S64, the state of the nostril undetected flag Fn is changed to a state indicating that the nostril can be detected (that is, OFF), and in step S65, the notification indicating that the nostril has not been detected so far. Is displayed on the display unit 5, a return process for returning to the normal display is enabled. In the process of step S65, the display unit 5 may already be in the normal display state. In this case, the driver monitor ECU 2 proceeds to the next step S71 while maintaining the normal display.

  On the other hand, when the driver's nostril cannot be detected from the captured image (No in step S62), the driver monitor ECU 2 refers to the nostril non-detection flag Dc to determine whether or not the nostril non-detection flag Fn is ON. (Step S66). Then, when the state of the nostril undetected flag Fn is ON, the driver monitor ECU 2 refers to the nostril undetected count data Dg and adds 1 to the nostril undetected count value Cn to update the nostril undetected count data Dg. (Step S67), the process proceeds to the next step S70. On the other hand, when the state of the nostril undetected flag Fn is OFF, the driver monitor ECU 2 changes the state of the nostril undetected flag Fn to ON and updates the nostril undetected flag Dc (step S68), and the nostril undetected count data The nostril undetected count value Cn of Dg is set to 1 (step S69), and the process proceeds to the next step S70.

  In step S70, the driver monitor ECU 2 determines whether or not the nostril undetected count value Cn of the nostril undetected count data Dg is greater than or equal to the threshold C1. Here, the threshold C1 is a value for determining whether the situation in which the driver's nostril cannot be detected from the captured image is a temporary phenomenon or a continuous phenomenon, as in step S61. The value obtained from the rule of thumb is set. And driver monitor ECU2 advances a process to following step S91 (FIG. 7), when nostril undetected count value Cn is more than threshold value C1. On the other hand, if the nostril undetected count value Cn is less than the threshold value C1, the driver monitor ECU 2 returns to step S52 and repeats the process.

  Here, in the processing of steps S66 to S70, when the driver monitor ECU 2 changes from the state in which the driver's nostril can be detected from the captured image to the state in which the nostril cannot be detected, the state of the nostril non-detection flag Fn is turned ON (step S68), the elapsed time after turning on the state of the nostril undetected flag Fn is counted as a nostril undetected count value Cn (steps S67, S69), and whether or not the nostril undetected count value Cn has reached the threshold value C1. This is a process for determining (step S70). In other words, in the processing of steps S66 to S70, the driver monitor ECU 2 measures the time during which the driver's nostril cannot be detected from the captured image, and when the time reaches the threshold value C1, the steps after step S91. Will be processed.

  In step S71, the driver monitor ECU 2 determines whether or not the driver's eyelid can be detected in the captured image stored in the captured image data Da in step S52. For example, in step S71, the driver monitor ECU 2 determines that the wrinkle cannot be detected when the driver's wrinkle itself is not captured in the captured image or when the driver's wrinkle cannot be recognized. And driver monitor ECU2 advances a process to following step S72, when a driver | operator's wrinkle is detectable. On the other hand, if the driver's wrinkle cannot be detected, the driver monitor ECU 2 advances the process to the next step S75.

  In step S72, the driver monitor ECU 2 refers to the wrinkle non-detection flag Dd to determine whether or not the state of the wrinkle non-detection flag Fe is ON. Then, when the state of the wrinkle detection flag Fe is ON, the driver monitor ECU 2 changes the state of the wrinkle detection flag Fe to OFF and updates the wrinkle detection flag Dd (step S73). Is displayed normally (step S74), and the process proceeds to the next step S80. On the other hand, if the state of the wrinkle detection flag Fe is OFF, the driver monitor ECU 2 proceeds to the next step S80 as it is.

  Here, as will be apparent later, if the eyelid undetected flag Fe is ON at the time of the determination in step S72, it indicates that the driver monitor ECU 2 has not been able to detect the driver's eyelid from the captured image until then. ing. That is, at the time of the determination, the driver monitor ECU 2 has changed from a state in which the driver's eyelid cannot be detected from the captured image to a state in which the eyelid can be detected. Therefore, in step S73, the state of the wrinkle non-detection flag Fe is changed to a state (that is, OFF) indicating that wrinkle detection is possible, and in step S74, notification indicating that the wrinkle has not been detected so far. Is displayed on the display unit 5, a return process for returning to the normal display is enabled. In the process of step S74, the display unit 5 may already be in the normal display state. In this case, the driver monitor ECU 2 proceeds to the next step S80 while maintaining the normal display.

  On the other hand, when the driver's eyelid cannot be detected from the captured image (No in step S71), the driver monitor ECU 2 refers to the eyelid undetected flag Dd to determine whether the state of the eyelid undetected flag Fe is ON. (Step S75). Then, when the state of the undetected flag Fe is ON, the driver monitor ECU 2 refers to the undetected count data Dh, adds 1 to the undetected count value Ce, and updates the undetected count data Dh. (Step S76), the process proceeds to the next step S79. On the other hand, when the state of the undetected flag Fe is OFF, the driver monitor ECU 2 changes the state of the undetected flag Fe to ON and updates the undetected flag Dd (step S77), and the undetected count data The undetected count value Ce of Dh is set to 1 (step S78), and the process proceeds to the next step S79.

  In step S79, the driver monitor ECU 2 determines whether the wrinkle detection count value Ce of the wrinkle non-detection count data Dh is greater than or equal to the threshold value C1. Here, the threshold value C1 is a value for determining whether the situation where the driver's wrinkle cannot be detected in the captured image is a temporary phenomenon or a continuous phenomenon, as in step S61. The value obtained from the rule of thumb is set. And driver monitor ECU2 advances a process to following step S91 (FIG. 7), when the wrinkle non-detection count value Ce is more than threshold value C1. On the other hand, if the wrinkle detection count value Ce is less than the threshold value C1, the driver monitor ECU 2 returns to step S52 and repeats the process.

  Here, in the processing of steps S75 to S79, when the driver monitor ECU 2 changes from a state where the driver's eyelid can be detected from the captured image to a state where the eyelid cannot be detected, the state of the eyelid undetected flag Fe is set to ON (step S77), the elapsed time after the state of the wrinkle undetected flag Fe is turned on is counted as a wrinkle undetected count value Ce (steps S76, S78), and whether or not the wrinkle undetected count value Ce has reached the threshold value C1. This is a process for determining (step S79). That is, in the processing of steps S75 to S79, when the driver monitor ECU 2 continues to be in a state where the driver's eyelid cannot be detected from the captured image, and when the time reaches the threshold value C1, the steps after step S91 are performed. Will be processed.

  In step S80, the driver monitor ECU 2 performs a process of calculating the degree of opening and closing of the driver's eyes using the captured image. Next, the driver monitor ECU 2 determines whether or not to end the process (step S81). For example, the driver monitor ECU 2 determines that the process is to be terminated when the driver of the vehicle performs an operation for terminating the process (for example, an operation for turning off the ignition switch). And driver monitor ECU2 returns to said step S52, and repeats a process, when continuing a process. On the other hand, when ending the process, the driver monitor ECU 2 ends the process according to the flowchart.

  Hereinafter, an example of the process for calculating the degree of eye opening / closing performed in step S81 will be described.

  For example, the driver monitor ECU 2 searches the captured image for a vertical center line that is the center of the face outline and the left-right direction in the captured image by edge extraction processing using a Sobel operator or the like, and is captured in the captured image. The left edge of the contour of the driver's face, the right edge of the contour, and the vertical center line of the face are detected. As an example, the driver monitor ECU 2 creates a luminance image of the captured image, performs edge extraction processing on the luminance image, and detects the left edge of the face contour, the right edge of the contour, and face parts (eyebrow, eye, nose, mouth, etc.). Extract position. Next, the driver monitor ECU 2 calculates the driver's face width and vertical centerline captured in the captured image based on the extracted face contour left end, contour right end, and face part positions. Then, the driver monitor ECU 2 calculates the left face width from the contour left end to the vertical center line, and the right face width from the contour right end to the vertical center line, respectively.

  Then, the driver monitor ECU 2 performs an eye position estimation process for estimating the range in which the driver's eye position is imaged using the captured image. The driver monitor ECU 2 uses the face width information and the like to set a range in which the driver's nose is estimated to be captured in the captured image. Then, the driver monitor ECU 2 searches the driver's nose portion within the above range and detects the position of the nose in the captured image. Specifically, the driver monitor ECU 2 detects the positions of the pair of nostrils within the above range. Then, the driver monitor ECU 2 sets an intermediate position between the pair of nostrils as a nose position in the captured image. In general, the shape of the nostril has less individual differences than the shape of the eye. Therefore, the position of the nostril can be detected more accurately than the position of the eye or the like.

  Next, the driver monitor ECU 2 sets the optimum left eye search range and right eye search range for the captured image based on the position of the nose. For example, the driver monitor ECU 2 sets the left-eye search range and the right-eye search range at positions on the captured image that are separated from the nose position by a predetermined distance in a predetermined direction. Here, the direction and distance for setting the left-eye search range and the right-eye search range based on the position of the nose may be set based on predetermined standard information. Note that the direction and distance for setting the left-eye search range and the right-eye search range, respectively, and the sizes of the left-eye search range and the right-eye search range may be changed according to the face width information. As a result, even if the size of the driver's face captured in the captured image changes (typically, the distance between the driver's face and the driver monitor camera 1 changes), the size of the face is reduced. Following the change, the positions and sizes of the left-eye search range and the right-eye search range can be set appropriately.

  And driver monitor ECU2 searches and detects a driver | operator's left eye within the left-eye search range set to the captured image. Further, the driver monitor ECU 2 searches and detects the driver's right eye within the right eye search range set in the captured image. For example, eye detection within the search range can be performed using various pattern matching techniques using a preset eye template image.

  Next, the driver monitor ECU 2 calculates the degree of opening and closing of both eyes of the driver. For example, the driver monitor ECU 2 extracts the contours of the upper eyelid and the lower eyelid for each of the left eye and the right eye of the driver in the captured image. As an example, detection of upper eyelids and lower eyelids in the left eye and the right eye can be performed using various pattern matching techniques using preset upper eyelid and lower eyelid template images. Then, the driver monitor ECU 2 calculates the distance between each upper eyelid and lower eyelid as the degree of opening / closing of each eye, and updates the eye opening / closing degree data Dj using the data indicating the degree of opening / closing.

  Thereafter, the driver monitor ECU 2 outputs data indicating the eye open / closed degree stored in the eye open / closed degree data Dj to the driver support system ECU 3. In response to this data output, the driver support system ECU 3 is mounted on the vehicle based on the data received from the driver monitor ECU 2 and the recognition result of the vehicles and obstacles around the vehicle output from the millimeter wave radar 4. To properly adjust the characteristics of the occupant protection system, activate the collision condition mitigation system, and give the driver appropriate warnings.

  In the opening / closing degree calculation process in step S80, not only the opening / closing degree of the driver's eyes but also the driver's face direction and line-of-sight direction may be calculated.

  For example, the driver monitor ECU 2 can calculate the left face width from the left edge of the contour of the driver to the vertical center line and the right face width from the right edge of the contour to the vertical center line in the captured image, respectively. Then, the driver monitor ECU 2 can calculate the face orientation angle α based on the ratio of the left face width value and the right face width value. Here, the value of the face angle α is α = 0 when the driver monitor camera 1 is facing the front (that is, the front of the vehicle). The value of the face orientation angle α is assumed to increase as the face turns to the right from the state of facing the front with respect to the driver monitor camera 1, and to decrease toward the left. That is, the value of the face orientation angle α is a positive value when the face is directed rightward with respect to the driver monitor camera 1, and is a negative value when the face is directed leftward.

  Further, the driver monitor ECU 2 can calculate the line-of-sight direction by extracting the position of the iris for each of the driver's left eye and right eye in the captured image. For example, for detection of irises in the left eye and right eye, a technique using an elliptic filter can be used. In general, the shape of the iris is often not elliptical on the captured image because it is blocked by the face direction or eyelids, but rather is elliptical. Therefore, a plurality of elliptic filters having different sizes are applied to each of the driver's left eye and right eye in the captured image, and luminances for the outer region and the inner region of the elliptic filter are obtained. At this time, if there is a difference in luminance between the luminance of the outer region and the luminance of the inner region, the position of the iris is obtained from the position of the used elliptic filter with the inner range of the elliptic filter as the iris. Note that the position detection of the iris in the left eye and the right eye is not limited to this method, and various pattern matching methods using a preset iris template image and the binarization of the left eye and the right eye are performed. Other detection methods such as a method of extracting the position of the black region may be used. Then, the driver monitor ECU 2 calculates the driver's line-of-sight direction based on the iris position in each of the driver's left eye and right eye and the face direction α. For example, the driver monitor ECU 2 calculates the angle of the driver's line of sight with respect to the front direction of the vehicle as the line of sight direction.

  Proceeding to FIG. 7, in step S91, the driver monitor ECU 2 performs notification display processing in accordance with the undetected feature in the captured image stored in the captured image data Da, and proceeds to the next step. For example, the driver monitor ECU 2 refers to the notification information data Dk, extracts the notification information Ia to Ic corresponding to the current undetected feature, and sends the character information indicating the notification information Ia to Ic to the driver support system ECU 3. The character information is output and displayed on the display unit 5.

  For example, as shown in FIG. 8, when the undetected feature is “face”, the notification information Ia to Ic described in the notification information table as notification information for the undetected feature is “face”. To display. Specifically, when the undetected feature portion is “face”, the display portion 5 displays “Face could not be detected” (notification information Ia), “Is there anything blocking between the camera and the camera? "(Notification information Ib)" and "Please stop the vehicle in a place without danger and remove the obstacle." (Notification information Ic) are displayed and notified to the driver. Thereby, when there is something that blocks between the driver monitor camera 1 and the driver, such as when the driver monitor camera 1 is covered with a cloth such as a towel, It can be predicted that the operation of the driver support system will return.

  For example, as shown in FIG. 9, when the undetected feature portion is “nasal nose”, the notification information Ia to Ic described in the notification information table as notification information for the undetected feature portion “nostril” is displayed on the display unit 5. To display. Specifically, when the undetected feature portion is “nasal nostrils”, the display unit 5 indicates “No nostrils could be detected” (notification information Ia), “Are you wearing a mask?” ( Announcement information Ib), “If you can remove the mask, stop the car where there is no danger and remove the mask.” (Notification information Ic) is displayed and notified to the driver.

  For example, as shown in FIG. 10, when the undetected feature is “部”, the notification information Ia to Ic described in the notification information table as notification information for the undetected feature is “瞼” is displayed on the display unit 5. To display. Specifically, when the undetected feature portion is “瞼”, the display portion 5 displays “No heel was detected.” (Notification information Ia), “I am not wearing sunglasses, glasses, or a hat. "Is your bangs in your eyes?" (Notification Information Ib), "If you can remove your clothes, stop and remove the car in a safe place." (Notification Information Ic) Is displayed and notified to the driver.

  Here, the display of the notification information described above is performed when the state where the characteristic portion is not detected is not a temporary phenomenon (that is, when the undetected duration is equal to or greater than the threshold value C1). For example, if the driver temporarily touches his / her face, eyes, nose, etc. with his / her hand, or temporary external light is incident on a part of the driver's face, the feature is temporarily detected from the captured image. It becomes a situation that can not be. However, if the notification display is performed according to such a temporary phenomenon, the information display becomes annoying for the driver. Therefore, the notification display is started when the undetected time reaches the threshold value C1. As a result, it is possible to reduce the troublesomeness of the driver due to the notification display.

  Returning to FIG. 7, after the process of step S91, the driver monitor ECU 2 refers to the notification display flag De and determines whether or not the state of the notification display flag Fd is ON (step S92). Then, when the state of the notification display flag Fd is ON, the driver monitor ECU 2 refers to the notification display count data Di and adds 1 to the notification display count value Cd to update the notification display count data Di (step S93). Then, the process proceeds to the next step S96. On the other hand, when the state of the notification display flag Fd is OFF, the driver monitor ECU 2 changes the state of the notification display flag Fd to ON and updates the notification display flag De (step S94), and displays the notification display count data Di. The count value Cd is set to 1 (step S95), and the process proceeds to the next step S96.

  In step S96, the driver monitor ECU 2 determines whether or not the notification display count value Cd of the notification display count data Di is greater than or equal to a predetermined threshold C2. Here, the threshold value C2 is a value for judging by time whether there is a possibility that it can be eliminated by giving a notification to the driver that the characteristic part cannot be detected in step S91, or whether it can not be eliminated. Yes, values obtained from various experiments and empirical rules are set. And driver monitor ECU2 advances a process to the following step S97, when the notification display count value Cd is more than threshold value C2. On the other hand, when the notification display count value Cd is less than the threshold value C2, the driver monitor ECU 2 returns to step S52 and repeats the process.

  In step S97, the driver monitor ECU 2 performs an abnormality display process and ends the process according to the flowchart. For example, the driver monitor ECU 2 refers to the abnormality information data Dl, outputs character information indicating the abnormality information to the driver support system ECU 3, and displays the character information on the display unit 5. Displayed on the display unit 5.

  For example, as shown in FIG. 11, the display unit 5 displays the abnormality information Id, “The driver support system is not operating normally. Please drive carefully in consideration of safety.” And notified to the driver. Thus, in step S97, the driver is notified that the driver support system is not operating normally, and information that further encourages the driver to drive safely is notified.

  Here, in the processing of steps S91 to S97, when the notification display indicating that the characteristic portion cannot be detected is performed on the display unit 5, the state of the notification display flag Fd is turned ON (step S94), and the notification display flag Fd The elapsed time after turning on the state is counted as the notification display count value Cd (steps S93 and S95), and it is determined whether or not the notification display count value Cd has reached the threshold value C2 (step S96). That is, the time during which the notification display is continued is measured, and when the time reaches the threshold value C2, the process of step S97 is performed. For example, if a feature point cannot be detected by glasses that the driver needs to wear when driving, or if a feature point cannot be detected by an attachment that cannot be removed depending on the driving environment (such as sunglasses), When a feature point cannot be detected due to a cause other than the notification information that cannot be resolved, it is often not resolved even if the above-mentioned notification display is given to the driver. In such a case, if the notification display is continued for a long time, the information display becomes troublesome for the driver. Therefore, when the notification display time reaches the threshold value C2, the display content is switched to the abnormal information. As a result, the driver's troublesomeness due to the notification display is reduced, and the driver's awareness of safe driving can be further improved.

  In the operation of step S97 described above, the display of the notification display on the display unit 5 may be stopped without displaying the abnormality information on the display unit 5. Thus, the troublesomeness felt by the driver can be reduced by simply stopping the notification display for a long time.

  As described above, according to the face image processing apparatus according to the present embodiment, when the feature portion of the driver's face cannot be detected from the captured image, the driver is notified of the feature portion that cannot be detected. The driver can check the operating status of the support system.

  In the above description, an example in which the notification information Ia to Ic is displayed on the display unit 5 according to the feature portion when the feature portion of the driver's face cannot be detected from the captured image is used. Thereby, the driver can know the undetected feature part, the estimated cause, and the precautions when the estimated cause is resolved by visually recognizing the display unit 5. However, when such an effect is not expected, the notification information displayed on the display unit 5 may be in another form. For example, even if only the notification information Ia is displayed on the display unit 5, the driver can know his / her feature that the driver support system has not been able to detect, which may lead to the fact that the feature cannot be detected. It is possible to consider whether it has occurred in the driver's condition or environment. That is, even if only the notification information Ia is displayed on the display unit 5, the driver can estimate the estimated cause, and if the driver himself / herself has the cause, the cause is eliminated and the resulting system status It is also possible to confirm.

  In the above description, an example in which the notification information Ia to Ic and the abnormality information Id is displayed on the display unit 5 is used. However, the notification information and abnormality information may be notified to the driver in other modes. As a first example, the notification information Ia to Ic and the abnormality information Id may be notified to the driver by voice via the audio amplifier 6 and the speaker 61. In this case, the driver monitor ECU 2 refers to the notification information data Dk and the abnormality information data Dl, outputs audio information indicating the notification information and abnormality information to the driver support system ECU 3, and outputs the audio based on the audio information to the speaker. 61 to output.

  As a second example, when only the notification information Ia is notified to the driver, the driver is notified by turning on / off a plurality of lamps. For example, a lamp for each feature point is provided on the instrument panel (instrument panel) in front of the driver's seat, and when the feature portion of the driver's face cannot be detected from the captured image, the lamp corresponding to the feature point is turned on. . Accordingly, the driver can know that the system in which the lamp is lit is a feature that could not be detected, and the driver can check the operation status of the driver support system.

  As a third example, when only the notification information Ia is notified to the driver, the driver is notified by the color of the lamp. For example, if the instrument panel (instrument panel) in front of the driver's seat is provided with a lamp that can be lit in multiple colors for each feature point, the feature point of the driver's face cannot be detected from the captured image The lamp is lit in a color according to. Thus, the driver can know the feature that the system could not detect due to the color of the lamp being lit, and the driver can check the operation status of the driver support system.

  As a fourth example, when only the notification information Ia is notified to the driver, the driver is notified by a pattern of blinking lamps. For example, if the instrument panel (instrument panel) on the front of the driver's seat is provided with a lamp that can blink in multiple patterns for each feature point, the feature point of the driver's face cannot be detected from the captured image The lamp blinks in a pattern according to. Thus, the driver can know the feature that the system could not detect by the pattern of the blinking lamp, and the driver can check the operation status of the driver support system.

  As a fifth example, when only the notification information Ia is notified to the driver, the driver is notified by a numerical value or a symbol displayed on the display unit 5. For example, if a display area for displaying numerical values and symbols for each feature point is provided on the instrument panel (instrument panel) in front of the driver's seat, the feature of the driver's face cannot be detected from the captured image. Display numerical values and symbols according to points. Thereby, the driver can know the characteristic part that the system could not detect from the displayed numerical values, symbols, and the like, and the driver can check the operation status of the driver support system.

  In the above description, an example in which notification information and abnormality information are displayed on the display unit 5 via the driver support system ECU 3 is used. However, the driver monitor ECU 2 controls the contents displayed directly on the display unit 5. It doesn't matter.

  In the processing operation of the driver monitor ECU 2 described above, the driver's face, nostril, and eyelid are used as an example of the feature portion to be notified, but the same processing can be performed for other feature portions. For example, it goes without saying that the same processing is possible even if the driver's eyebrows, mouth, nose, ears, etc. shown in FIG.

  In the above-described embodiment, the example in which the driver monitor camera 1 is installed on the upper surface of the steering column 92 is used. However, the driver monitor camera 1 may be installed at another position. The driver monitor camera 1 may be installed in any position as long as it can capture the face of the driver who gets on the vehicle. For example, in the instrument panel (instrument panel) in front of the driver's seat, instrument panel It may be installed at the top, in the steering wheel, and in the upper part of the passenger compartment.

  In addition, the processing order of the driver monitor ECU 2 described above, the time counting method, the state determination method using various flag data, and the like are merely examples, and it goes without saying that the present invention can be realized with other orders and methods.

  The program executed by the driver monitor ECU 2 is not only stored in advance in a storage area provided in the driver monitor ECU 2, but also supplied to the driver monitor ECU 2 through an external storage medium, or through a wired or wireless communication line. It may be supplied to the ECU 2.

  Although the present invention has been described in detail above, the above description is merely illustrative of the present invention in all respects and is not intended to limit the scope thereof. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention.

  The face image processing apparatus according to the present invention can appropriately notify the driver of the state of the apparatus, and is useful for a system for determining the state of the driver seated in the vehicle.

The block diagram which shows an example of a function structure of the driver support system containing the face image processing apparatus which concerns on one Embodiment of this invention Side surface outline figure which shows an example which looked at the driver monitor camera 1 installed in the vehicle from the lateral direction of the driver's seat Front schematic diagram showing an example of a driver monitor camera 1 installed in a vehicle viewed from the steering wheel 91 side The figure which shows an example of the main data memorize | stored in the memory of driver monitor ECU2 of FIG. The figure which shows an example of the notification information data Dk of FIG. The first half flowchart showing an example of processing executed by the driver monitor ECU 2 of FIG. The latter half flowchart which shows an example of the process performed by driver monitor ECU2 of FIG. The figure which shows an example of the notification information displayed on the display part 5 when an undetected feature part is a "face" The figure which shows an example of the notification information displayed on the display part 5 when an undetected feature part is a "nasal cavity" The figure which shows an example of the notification information displayed on the display part 5 when an undetected feature part is "瞼" The figure which shows an example of the abnormality information displayed on the display part 5

Explanation of symbols

1 ... Driver monitor camera 11 ... CCD
12 ... LED
2 ... Driver monitor ECU
3. Driver support system ECU
4 ... millimeter wave radar 5 ... display unit 6 ... audio amplifier 61 ... speaker 7 ... brake control ECU
71 ... Alarm buzzer 72 ... Brake ACT
91 ... Steering wheel 911 ... Rim 912 ... Spoke 92 ... Steering column

Claims (10)

Captured image acquisition means for acquiring a captured image from an imaging means for capturing the face of the driver of the vehicle;
Feature detection means for detecting a feature of the face imaged in the captured image;
A face image processing apparatus comprising: an informing means for informing a driver of the feature section when the feature section detection section cannot detect any of the feature sections.   The face image processing apparatus according to claim 1, wherein the notifying unit notifies the driver of an estimated cause that the feature unit could not be detected, and notifies the driver of the estimated cause that the feature unit could not be detected.   The informing means informs the driver of the characteristic part that the characteristic part detecting means could not detect and the estimated cause, and informs the driver of precautions when the driver eliminates the estimated cause. Item 3. The face image processing apparatus according to Item 2.   The face image processing apparatus according to claim 2, wherein the presumed cause is that a driver wears a wear that is supposed to block the characteristic portion when viewed from the imaging unit. Storage means for storing in advance presumed cause information that cannot be detected for each feature,
In the case where the feature detection unit cannot detect any of the feature units, the notification unit uses the estimated cause information stored in the storage unit corresponding to the feature unit as the estimated cause. The face image processing apparatus according to claim 2, which notifies to Storage means for preliminarily storing each presumed cause information that cannot be detected for each feature and precautionary information when the estimated cause is resolved,
The notification means, when the feature detection means could not detect any of the features, the estimated cause information and the notes information stored in the storage means corresponding to the feature, respectively, The face image processing apparatus according to claim 3, wherein a driver is notified of the estimated cause and the precautions. The captured image acquisition means acquires the captured image for each predetermined period from the imaging means,
The feature detection unit detects the feature captured in the captured image every time the captured image acquisition unit acquires the captured image.
The face image according to claim 1, wherein the notification unit notifies the driver of the feature portion when the feature portion detection unit cannot continuously detect the same feature portion for a first time or more. Processing equipment. The captured image acquisition means acquires the captured image for each predetermined period from the imaging means,
The feature detection unit detects the feature captured in the captured image every time the captured image acquisition unit acquires the captured image.
The notification means notifies the driver of the feature section when the feature section detection section cannot detect the feature section for a second time or more after notifying the feature section to the driver. The face image processing apparatus according to claim 1, wherein the processing is stopped.   9. The notification unit according to claim 8, wherein the notification unit notifies the driver of abnormality information indicating that the face image processing device is not operating normally after stopping reporting the feature to the driver. Face image processing device. The captured image acquisition means acquires the captured image for each predetermined period from the imaging means,
The feature detection unit detects the feature captured in the captured image every time the captured image acquisition unit acquires the captured image.
The informing means stops informing the driver of the feature section when the feature section detection section becomes capable of detecting the feature section after informing the driver of the feature section. 3. The face image processing apparatus according to 1 or 2.

Images