JP4858516B2 - Face orientation detection device - Google Patents

Description

  The present invention relates to a face orientation detection device that detects the orientation of a face of a driver of a vehicle.

2. Description of the Related Art Conventionally, for the purpose of monitoring a driver's side-view driving, a driver's face is photographed with a camera, and the direction of the driver's face is detected from the photographed image. For example, in Patent Document 1, a face is photographed when the driver is facing the front, a face model is created from the face image at that time, and the face orientation is determined using the created face model. Is described.
JP 2003-308533 A

  Here, the positional relationship between the camera and the driver changes depending on the posture or the like when the driver sits in the driver's seat, and the camera is not necessarily positioned in the front direction with respect to the driver. The driving position varies depending on the driver, and the direction in which the driver feels front is different for each driver. Therefore, even if a driver facing the front is shown in the image taken by the camera (even if the driver is facing the camera), the driver at that time is not necessarily facing the front of the vehicle, When the driver is facing the front of the vehicle, the camera may appear to face diagonally. Therefore, if the orientation of the driver's face is determined based only on the image captured by the camera, it may not be possible to accurately determine whether the driver is facing the front. As a result, the driver's face orientation may not be detected accurately.

  SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a face orientation detection device that can accurately detect the orientation of a driver's face.

  In order to solve the above problems, the present invention employs the following configuration. That is, the present invention is a face orientation detection device that detects the face orientation of a driver of a vehicle. The face orientation detection means includes a determination processing means, an estimation means, and a learning means. The determination processing means performs a determination process for determining the driver's face orientation using an image taken by the photographing means for photographing the driver. The estimation means estimates whether or not the driver is facing the front based on the driving state of the driver. The learning means learns about the determination process so that the face direction detected from the image captured by the image capturing means is determined to be the front direction when it is estimated that the driver is facing the front.

  According to the above, the face orientation detection device can perform learning so that the driver's face orientation when the driver is actually facing the front is determined to be the front orientation. Therefore, even if the state where the driver is facing the front is not a front direction with respect to the imaging means, the actual front direction for the driver can be detected, and the driver's face direction can be accurately determined. Can be detected.

  Further, the determination process may be a process of detecting whether or not the driver's face is detected and determining whether or not the driver is facing the front. At this time, the learning means changes, by learning, an angle range in which it is determined that the driver is facing the front in the determination process.

  According to the above, the angle range can be set by learning according to the actual front direction for the driver. Therefore, it is possible to accurately determine whether or not the driver is facing the front. According to this, for example, it is possible to accurately determine whether or not the driver is driving aside.

  The learning means may perform learning based on an offset amount between the center position in the image photographed by the photographing means and the position of the driver's face in the image.

  According to the above, learning is performed in consideration of the offset amount. Here, it is considered that the front direction for the driver shifts from the front direction when the driver is located at the front position of the imaging unit as the offset amount increases. Therefore, by performing learning in consideration of the offset amount, such a shift can be corrected, and the face orientation can be accurately calculated.

  In addition, the determination processing unit may calculate the position of the center line of the face from the position of a predetermined part of the driver's face as the position of the driver's face in the image photographed by the photographing unit.

  According to the above, the position of the driver's face in the captured image can be easily calculated from the center line of the face.

  In addition, the determination processing unit may calculate the center positions of both ends of the driver's face as the position of the driver's face in the image captured by the imaging unit.

  According to the above, the position of the driver's face in the captured image can be easily calculated from the positions of both ends of the face.

  Further, the estimation means may estimate that the driver is facing the front at least on the condition that the speed of the vehicle is equal to or higher than a predetermined speed. Further, the estimation means may estimate that the driver is facing the front, at least on the condition that the vehicle is traveling straight ahead.

  According to the above, since it is possible to learn the face direction when the driver is likely to be facing the front as the front direction, the driver's face direction can be accurately detected.

  Further, the estimation means may estimate that the driver is facing the front on the condition that the amount of change in the face direction detected from the image captured by the imaging means is not more than a predetermined amount. Further, the estimation means further determines that the face direction detected from the image photographed by the photographing means is within a predetermined range centered on the direction from the driver to the photographing means. You may guess that you are facing. The estimation means may estimate that the driver is facing the front at least on the condition that the brake of the vehicle is being operated.

  According to the above, since it is possible to learn the face direction when the driver is more likely to be facing the front as the front direction, it is possible to more accurately detect the driver's face direction. it can.

  According to the present invention, by performing learning so that the driver's face direction when the driver is actually facing the front is determined to be the front direction, the actual front for the driver is determined. Since the direction can be detected, the direction of the driver's face can be accurately detected.

  Hereinafter, a face orientation detection apparatus according to an embodiment of the present invention will be described. FIG. 1 is a block diagram showing the configuration of the face orientation detection device in the present embodiment. As shown in FIG. 1, the face direction detection device 1 includes a camera 2, a driver monitor ECU (Electronic Control Unit) 3, a driver support system ECU (hereinafter referred to as “DSSECU”) 4, a steering sensor 5, a brake ECU 6, A wheel speed sensor 7 is provided. In the present embodiment, the driver monitor ECU 3 and the DSSECU 4 are connected to a first CAN (Controller Area Network) 8, and the DSSECU 4, the steering sensor 5, and the brake ECU 6 are connected to a second CAN 9. However, the communication form between the components may be any form.

  The face orientation detection device 1 is mounted on a vehicle and detects the face orientation (face orientation) of the driver of the vehicle. In the present embodiment, the face orientation detection device 1 determines whether or not the driver is facing the front, and performs an operation (brake or the like) for ensuring the safety of the driver according to the determination result. Is what you do. Hereinafter, each part of the face direction detection apparatus 1 will be described.

  The camera 2 is a photographing means for photographing the driver's face, and is, for example, a CCD camera or an infrared camera. The camera 2 is installed at a position where the face of the driver seated in the driver's seat can be photographed. As an example of the installation location of the camera 2, the inside of the meter hood of a driver's seat, the inside of a meter panel, the steering column, etc. are mentioned. For example, the camera 2 captures the driver's face at predetermined time intervals, and sequentially outputs the captured images to the driver monitor ECU 3.

  The driver monitor ECU 3 performs image recognition using an image captured by the camera 2 and detects the driver's face orientation. Further, in the present embodiment, the driver monitor ECU 3 performs image recognition using the captured image, and detects the state of the driver's eyes (such as eyelid opening (opening degree)). The driver monitor ECU 3 sequentially outputs face direction information indicating the driver's face direction and state information indicating the driver's eye state to the DSSECU 4 via the first CAN 8.

  In the present embodiment, the driver monitor ECU 3 has two microcomputers: a main microcomputer (hereinafter referred to as a main microcomputer) and a sub microcomputer (hereinafter referred to as a sub microcomputer). The main microcomputer mainly detects the driver's face orientation. The sub-microcomputer detects the eye state of the driver. That is, the driver monitor ECU 3 can process the detection of the driver's face direction and the detection of the driver's eye state in parallel by the two microcomputers, and can perform detection at high speed. The driver monitor ECU 3 has a storage unit that stores data representing a human face image pattern.

  The DSSECU 4 determines the state of the driver based on information from the driver monitor ECU 3. Specifically, the DSSECU 4 determines whether the driver is facing the front of the vehicle (whether the driver is not looking aside) using the information on the driver's face orientation. Furthermore, in this embodiment, DSSECU4 uses the face information regarding the driver's eyes to determine whether or not the driver is asleep. In addition, the DSSECU 4 controls the operation of various devices installed in the vehicle based on these determination results to ensure the safety of the driver and the occupant. For example, if it is predicted that the driver's condition will lead to an accident based on the above judgment result, be careful of the driver by turning on the warning light of the vehicle or sounding the alarm buzzer. Encourage arousal. Further, for example, the DSSECU 4 may perform a collision prediction between the own vehicle and the other vehicle based on information from a radar device provided in the vehicle (information such as the position and speed of the other vehicle). At this time, the DSSECU 4 urges the driver to be alerted based on the determination result and the collision prediction result, performs an operation for avoiding the collision (actuating a brake, etc.), and damages at the time of the collision. A mitigating action (such as preparing an airbag operation) may be performed.

  The steering sensor 5, the brake ECU 6, and the wheel speed sensor 7 detect the driving state of the driver. Specifically, the steering sensor 5 detects the steering angle of the vehicle steering. Steering angle information indicating the detected steering angle is output to the DSSECU 4 via the second CAN 9. The brake ECU 6 detects whether or not the vehicle brake is operated by the driver. Brake information indicating whether or not the brake is operated is output to the DSSECU 4 via the second CAN 9. The wheel speed sensor 7 detects the rotational speed of the vehicle wheel. Speed information indicating the detected rotation speed is output to the DSSECU 4 via the brake ECU 6 and the second CAN. The brake ECU 6 controls the operation of the brakes of the vehicle. In the present embodiment, the brake ECU 6 operates the brake according to an instruction from the DSSECU 4.

  Hereinafter, the operation of the face orientation detection apparatus 1 will be described. FIG. 2 is a flowchart showing the flow of the operation of the face direction detection apparatus in the present embodiment. In the present embodiment, the operation shown in FIG. 2 is performed by the driver monitor ECU 3 and the DSSECU 4. 2 is executed, for example, while the vehicle is running or while the ignition switch of the vehicle is on.

  In FIG. 2, first, in step S <b> 1, the driver monitor ECU 3 acquires an image captured by the camera 2. In the subsequent step S2, the driver monitor ECU 3 (the main microcomputer) executes a face orientation detection process. The face orientation detection process is a process of detecting the driver's face orientation (face orientation angle) based on the captured image. In the present embodiment, as a determination process for determining the driver's face direction, the face direction is detected in step S2, and it is determined whether or not the face direction detected in step S8 described later is the front direction. Therefore, in the present embodiment, the driver monitor ECU 3 that executes step S2 and the DSSECU 4 that executes step S7 correspond to the determination processing means described in the claims. The details of the face orientation detection process will be described below with reference to FIG.

  FIG. 3 is a flowchart showing details of step S2 (face orientation detection processing) shown in FIG. In step S11 in FIG. 3, the driver monitor ECU 3 calculates the positions of the left and right ends of the driver's face in the captured image acquired in step S1. In subsequent step S12, the driver monitor ECU 3 calculates the position of a predetermined part (here, eyes, nose, and mouth) of the driver in the captured image. Note that the content of the image recognition processing in steps S11 and S12 may be anything. Following step S12, the process of step S13 is executed.

  In step S13, the driver monitor ECU 3 calculates the center line of the driver's face. The center line of the face is not the center of the face area on the photographed image (that is, the center of the left and right sides of the driver) but the line passing through the center of the actual face. It is a line that passes through. The driver monitor ECU 3 calculates the center line using the position of each predetermined part calculated in step S12. Following step S13, the process of step S14 is executed.

  In step S14, the driver monitor ECU 3 calculates the driver's face orientation. The face orientation is calculated using the positions of the left and right ends of the face calculated in step S11 and the position of the center line of the face. Specifically, the driver monitor ECU 3 calculates the face orientation based on the ratio of the length from the left end of the face to the center line and the length from the center line to the right end of the face. For example, if the lengths of both are equal, the face direction can be determined to be the front direction (0 °) with respect to the camera 2. Here, the face orientation is represented by an angle (face orientation angle) where the direction of the camera 2 is 0 °. After completion of the above step S14, the driver monitor ECU 3 ends the face orientation detection process. Note that the face orientation detection method in the face orientation detection process described above is an example, and the driver monitor ECU 3 may detect the face orientation by any method.

  Returning to FIG. 2, in step S <b> 3, the driver monitor ECU 3 (the sub-microcomputer) calculates the eyelid opening degree (the degree to which the eyes are open) of the driver. The eyelid opening degree may be calculated by any method, but may be calculated using the information obtained in step S2 when the eye part is extracted in step S2. . Following step S3, the process of step S4 is executed.

  In step S4, the driver monitor ECU 3 outputs the processing results in steps S2 and S3 to the DSSECU 4. That is, face direction information indicating the driver's face direction and state information indicating the driver's eye state are output to the DSSECU 4 via the first CAN 8. Following step S4, the process of step S5 is executed.

  In step S5, the DSSECU 4 acquires various types of information (used to determine the driving state of the driver) representing the driving state of the driver. In the present embodiment, the DSSECU 4 acquires steering angle information from the steering sensor 5, acquires brake information from the brake ECU 6, and acquires speed information from the wheel speed sensor 7. Following step S5, the process of step S6 is executed.

In step S6, the DSSECU 4 estimates whether the driver is facing the front based on the driving state of the driver. In this embodiment, DSSECU4 which performs step S6 is equivalent to the estimation means described in the claims. In the present embodiment, the estimation in step S6 is performed using the information acquired in step S5 and the face orientation information input from the driver monitor ECU 3. Specifically, the DSSECU 4 estimates that the driver is facing the front when all the following conditions [Condition A] to [Condition D] are satisfied.
[Condition A] The speed of the vehicle is a predetermined speed or more [Condition B] The vehicle is traveling straight [Condition C] The brake of the vehicle is being operated [Condition D] The current face direction is [Condition A] can be determined from the speed information, and [Condition B] can be determined from the steering angle information. [Condition C] can be determined based on the brake information, and [Condition D] can be determined based on the face orientation obtained in the last step S2. The “current face orientation” in [Condition D] is a face orientation detected from an image photographed by the camera 2 when it is estimated that the driver is facing the front. DSSECU4 performs the process of step S7, when it is estimated that the driver | operator is facing the front. On the other hand, when it is not estimated that the driver is facing the front, the DSSECU 4 skips the process of step S7 and executes the process of step S8 described later.

  When the above [Condition A] to [Condition C] are satisfied, the possibility that the driver is facing the front is considered to be quite high. Furthermore, it can be excluded from [Condition D] that the driver is presumed to be facing the front when facing a direction greatly deviating from the front. Therefore, according to the present embodiment, it is possible to accurately estimate that the driver is facing the front by satisfying all the above [Condition A] to [Condition D]. In other embodiments, the condition for estimating that the driver is facing the front is not limited to the above, and may be any condition. For example, only a part of [Condition A] to [Condition D] may be used, or another condition may be used in addition to [Condition A] to [Condition D]. Further, for example, instead of [Condition D] (or together with [Condition D]), DSSECU4 uses “the amount of change in the driver's face direction (per unit time) is a predetermined amount or less” as a condition. May be.

  In step S7, the DSSECU 4 performs learning so that the current face direction is determined to be the front direction with respect to the determination process in step S8 described later. In this embodiment, DSSECU4 which performs step S7 is equivalent to the learning means described in the claims. Hereinafter, the learning process in step S7 will be described with reference to FIGS.

  First, the necessity of performing learning processing will be described with reference to FIGS. FIG. 4 is a diagram illustrating an example of the positional relationship between the driver D and the camera 2, and FIG. 5 is a diagram illustrating an example of an image photographed by the camera 2 in the case illustrated in FIG. 4. As shown in FIG. 4, when the driver D is located in front of the camera 2, as shown in FIG. 5, the face of the driver D in the image taken by the camera 2 is viewed from the front. Become. At this time, the face of the driver D is located at the center in the left-right direction of the captured image. In the case shown in FIG. 4, the front direction for the driver D is the direction facing the direction of the camera 2. If the face direction of the driver D is the front direction in the photographed image, the driver D is actually It can be determined that it is facing the front.

  6 is a diagram illustrating another example of the positional relationship between the driver D and the camera 2, and FIG. 7 is a diagram illustrating an example of an image captured by the camera 2 in the case illustrated in FIG. . As shown in FIG. 6, when the driver D is located at a position deviated from the front position of the camera 2, as shown in FIG. 7, the face of the driver D in the image taken by the camera 2 is It may not be seen from the front. In the case of FIG. 6, the front direction for the driver (direction in which the driver D perceives the front) may be different from the front direction in the case of FIG. 4. As described above, if the face orientation in the captured image shown in FIG. 7 is determined as it is as the driver's face orientation, the front direction for the driver may not be accurately determined. For example, it is not accurate to determine from the captured image shown in FIG. 7 that “the face direction of the driver D is slightly outward from the front direction”. As described above, it is not appropriate to set the face direction in the image taken by the camera 2 as it is as the actual driver's face direction, and it is necessary to learn the front direction of the actual driver through learning processing.

  Therefore, in the present embodiment, the DSSECU 4 corrects the range determined to be facing the front (front range) in the learning process in step S7. The front range is information used in the process of determining whether or not the driver is facing the front (step S8 to be described later). In this process, the face orientation (face orientation) acquired from the driver monitor ECU 3 is used. If the angle) is a value within the front range, the driver is determined to be facing the front. FIG. 8 is a diagram illustrating an example of the front range. The front range is represented by an angle with the direction from the driver D to the camera 2 as a reference (0 °), specifically, the first boundary value S on one side with respect to the center of the front range, and the other The second boundary value T on the side of Here, the angle on the one side with respect to the reference is represented by a positive value, and the angle on the other side with respect to the reference is represented with a negative value. Therefore, in FIG. 8, the first boundary value S is a positive value, and the second boundary value T is a negative value. In the present embodiment, the range centered on the reference direction is set as the initial value of the front range.

In the learning process in step S7, the face orientation detected from the image captured by the camera 2 when the driver is assumed to be facing the front (the face orientation detected in step S2) is the front orientation. The front range is corrected so as to be determined as follows. FIG. 9 is a diagram for explaining the learning process in step S7. In FIG. 9, the angle A is the current face angle detected in step S2. At this time, the DSSECU 4 calculates the corrected front range according to the following equation (1).
N = S + A
M = TA (1)
In the above equation (1), the variable N is the first boundary value after correction, and the variable M is the second boundary value after correction. By the above equation (1), as shown in FIG. 9, the corrected front range is corrected to a range centered on the current face angle A.

The DSSECU 4 may calculate the corrected front range according to the following equation (2) instead of the above equation (1).
N = S + L × P + A
M = TL × Q−A (2)
In the above equation (2), the variable L is an offset amount (see FIG. 7) from the center position in the captured image to the position of the driver's face. The constants P and Q are determined in advance. Note that the position of the driver's face in the captured image may be, for example, the position of the centerline of the face (see step S13), the center position of the left and right ends of the face, or a predetermined part of the face (For example, the position of the nose). Here, the front direction for the driver is the front in the case where the driver is located at the front position of the camera 2 as the offset amount L becomes larger (that is, the position of the driver deviates from the front position of the camera 2). It is thought that it deviates from the direction. In consideration of this, the above equation (2) corrects the front range by an amount corresponding to the offset amount.

  In step S7, the front range is corrected to a range around the current face orientation angle A by one correction. Here, in another embodiment, the DSSECU 4 may gradually correct the front range so as to be a range centered on the current face orientation angle A by a plurality of corrections. At this time, the amount by which the correction range is changed by one correction may be determined according to the offset amount L.

  Returning to FIG. 2, in step S8, the DSSECU 4 determines whether or not the driver is facing the front. That is, the DSSECU 4 determines whether or not the face orientation angle detected in step S2 is a value within the front range. Further, the DSSECU 4 controls the operation of various devices installed in the vehicle based on the determination result to ensure the safety of the driver and the occupant. After step S8, the process of step S1 is executed again. Thereafter, the processes of steps S1 to S8 are repeatedly executed.

  As described above, according to the present embodiment, it is estimated whether or not the driver is actually facing the front (step S6), and the face orientation angle when it is estimated that the driver is actually facing the front is Learning is performed so that the face orientation angle when facing the front direction is obtained (step S7). Therefore, even when the state where the driver is facing the front is not the front direction with respect to the imaging means, the actual front direction for the driver can be accurately detected, and the driver is facing the front. It is possible to accurately determine whether or not.

  In the above-described embodiment, the face direction detection device has learned the determination process for detecting the driver's face direction and determining whether or not the driver is facing the front. Here, in another embodiment, the determination process may be a process that determines the driver's face orientation using an image captured by the camera 2 and determines whether or not the driver is facing the front. Not necessarily. For example, the determination process may be a process of calculating a face orientation angle with the front direction for the driver as a reference (0 °). At this time, the face direction angle based on the front direction can be accurately calculated by correcting the reference direction by learning.

  The present invention can be used in, for example, a vehicle safety system that determines whether or not the driver is looking aside, for the purpose of accurately detecting the direction of the driver's face.

The block diagram which shows the structure of the face direction detection apparatus in this embodiment The flowchart which shows the flow of operation | movement of the face direction detection apparatus in this embodiment. The flowchart which shows the detail of step S2 (face direction detection process) shown in FIG. The figure which shows the example of the positional relationship of the driver | operator D and the camera 2 The figure which shows the example of the image image | photographed with the camera 2 in the case shown in FIG. The figure which shows the other example of the positional relationship of the driver | operator D and the camera 2 The figure which shows the example of the image image | photographed with the camera 2 in the case shown in FIG. Diagram showing an example of the front range The figure for demonstrating the learning process of step S7

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Face direction detection apparatus 2 Camera 3 Driver monitor ECU
4 Driver support system ECU
5 Steering sensor 6 Brake ECU
7 Wheel speed sensor

Claims (10)

A face direction detection device for detecting a face direction of a driver of a vehicle,
Determination processing means for performing determination processing for determining the driver's face orientation using an image photographed by the photographing means for photographing the driver;
Based on the driving state of the driver, an estimation means for estimating whether the driver is facing the front,
Learning means for learning about the determination processing so that when the driver is assumed to be facing the front, the face orientation detected from the image photographed by the photographing means is judged to be the front orientation. A face orientation detection device. The determination process is a process of detecting whether or not the driver is facing and determining whether or not the driver is facing the front.
The face orientation detection device according to claim 1, wherein the learning unit changes, by learning, an angle range in which it is determined in the determination process that the driver is facing the front.   3. The learning device according to claim 1, wherein the learning unit performs learning based on an offset amount between a center position in the image photographed by the photographing unit and a position of the driver's face in the image. The face orientation detection device according to Item.   The said determination processing means calculates the position of the centerline of a face from the position of the predetermined part of the said driver | operator's face as a position of the said driver | operator's face in the image image | photographed by the said imaging | photography means. Face orientation detection device.   The face orientation detection device according to claim 3, wherein the determination processing unit calculates the positions of the centers of both ends of the driver's face as the position of the driver's face in the image photographed by the photographing unit.   The said estimation means presumes that the said driver is facing the front at least on the condition that the speed of the said vehicle is more than predetermined | prescribed speed of any one of Claims 1-5. Face orientation detection device.   The face direction detection according to any one of claims 1 to 6, wherein the estimation means estimates that the driver is facing the front at least on the condition that the vehicle is traveling straight ahead. apparatus.   The said estimation means estimates that the said driver is facing the front on the condition that the variation | change_quantity of the face direction detected from the image image | photographed by the said imaging | photography means is below a predetermined amount further. Or the face direction detection apparatus of any one of Claim 7.   The estimation means is further provided that the face direction detected from the image photographed by the photographing means is a direction within a predetermined range centered on a direction from the driver to the photographing means. The face direction detection device according to claim 6, wherein the person is assumed to be facing the front.   The face direction according to any one of claims 6 and 7, wherein the estimation means estimates that the driver is facing the front, at least on the condition that the brake of the vehicle is being operated. Detection device.

Images