JP2010009093A - Person state determination device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a person state determination device, capable of accurately determining a wakeful state of a person according to whether the person is smiling. <P>SOLUTION: The device includes an imaging means which photographs a person's face to obtain a face image; a closed eye estimation means which estimates whether the person is closing his/her eyes based on the face image; a smile estimation means which estimates whether the person is smiling, based on an image of the person's mouth contained in the face image; and an awakening determination means which determines that the person is awakened when opening of the person's eyes is estimated or when smiling of the person is estimated and closing of the person's eyes is estimated, and determines that the person is unawakened when non-smiling of the person is estimated and closing of the person's eyes is estimated. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a person state determination apparatus, and more particularly to a person state determination apparatus that determines a person's arousal state.

  2. Description of the Related Art In recent years, a vehicle that includes a camera that captures a driver on a vehicle, detects a driver's state based on a driver's face image captured by the camera, and controls the vehicle according to the detected driver's state Control systems have been developed. Such a system includes a device that detects the open / closed state of the driver's eyes and determines the driver's arousal state (hereinafter referred to as an arousal determination device).

  By the way, when the driver's eyes are irradiated with direct light, the driver may feel dazzled by the direct light and may have a narrowed eye. When the driver is in a state of narrowing his eyes, the awakening determination device as described above erroneously determines that the driver is not in an awakening state even though the driver is actually awakening.

  A direct light detection device developed in consideration of the above problems is disclosed in Patent Document 1. The direct light detection device measures the degree of opening of the driver's eyes (eye opening described in Citation 1) and the amount of light in the vicinity of the eyes based on an image taken by the camera. Then, it is determined whether or not the driver's eyes are exposed to direct light based on the eye opening and the light amount.

According to the direct light detection device disclosed in Patent Document 1, it is possible to determine whether or not the driver feels dazzle by direct light. Therefore, when the driver's arousal state is determined by the above-described arousal determination device, even if the device detects that the driver's eyes are closed, the driver's eyes are exposed to direct light. In other words, if it is estimated that the eyes are narrowed due to direct light hitting the driver's eyes, it can be determined that the driver is awake.
JP 2006-251926 A

  However, the condition for the driver to narrow his eyes is not limited to the case where the driver's eyes are exposed to direct light. For example, when the driver laughs, the driver's eyes are narrowed even if they are not exposed to direct light. When the driver laughs and narrows his eyes, even if the invention of Patent Document 1 is used, the awakening determination device as described above may determine that the driver is not in an awake state.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a person state determination device that can correctly determine a person's arousal state according to whether or not a person is smiling.

  The present invention has been made in view of the above, and the first invention is based on an image capturing unit that captures a face of a person and obtains a face image, and whether the person closes his eyes based on the face image. Closed eye estimation means to estimate, smile estimation means to estimate whether a person is laughing based on the image of the person's mouth included in the face image, and if the person's eyes are estimated to be open, or a person If it is estimated that the person's eyes are closed, it is determined that the person's eyes are awake, the person is not smiling, and the person's eyes are closed. It is a person state determination apparatus provided with the awakening determination means which determines that the person is not awake when it estimates that the person is waking up.

  According to a second aspect, in the first aspect, the human state determination device further includes a lip boundary detection unit that detects a lip boundary line that is a boundary line between the upper lip and the lower lip in the face image, and smile estimation The means estimates whether or not the person is laughing based on the shape of the lip boundary line.

  According to a third aspect, in the second aspect, the human state determination device includes the Mth from the left and right ends of the line segment obtained by dividing the lip boundary line into N equal parts (N is an integer equal to or greater than 2). M is further provided with line segment pair extraction means for extracting a line segment pair positioned at an integer of 1 or more, and the smile estimation means estimates whether or not a person is laughing based on the extracted line segment pair.

  In a fourth aspect based on the third aspect, the human state determination device includes: an angle calculating unit that calculates an angle between the extracted line segment and each of the two line segments that form the line segment pair and a horizontal line; The smile estimation means determines that the person is smiling when the total value is larger than the threshold value.

  In a fourth aspect based on the third aspect, the human state determination device includes a line segment approximating means for linearly approximating each line segment forming a line segment pair to form a straight line segment, and a straight line segment forming a line segment pair. Angle calculating means for calculating an angle formed by each horizontal line, and the smile estimating means determines that the person is laughing based on the angle.

  In a fifth aspect based on the third aspect, the human state determination device approximates each line segment of the line segment pair to a circular arc, and approximates each line segment forming the line segment pair as a circular arc segment by circular arc approximation. Means and a diameter calculation means for calculating the radius or diameter of each arc line segment forming a line segment pair, and the smile estimation means determines that the person is laughing based on the angle.

  In a sixth aspect based on the second aspect, the human state determination device further includes learning means for learning information related to the shape of the lip boundary line only when it is estimated by the closed eye estimation means that the person is open. The smile estimation means estimates whether or not the person is laughing based on the learning result.

  In a seventh aspect based on the first aspect, the person is a driver of the vehicle, and the person state determination device is mounted on the vehicle.

  According to the first invention, when a person is laughing, whether or not the person closes his eyes even if it is difficult to determine whether or not the person closes his eyes by narrowing his eyes. Can be determined correctly.

  According to the second invention, in the first invention, whether or not a person is laughing can be estimated based on the boundary line between the upper lip and the lower lip of the person. Since the boundary line between the upper lip and the lower lip of the person can be easily recognized in the face image, according to the second aspect, it is easily estimated based on the face image whether or not the person is laughing. be able to.

  According to the third invention, in the second invention, it is possible to estimate whether or not a person is laughing based on the shape of the end of the mouth. Since the shape of the edge of the mouth changes significantly when a person is laughing, it can be easily and correctly estimated whether or not the person is laughing.

  According to the fourth invention, in the third invention, whether or not a person is laughing can be estimated by a simple process based on the shape of the end of the mouth.

  According to the fifth aspect, in the third aspect, it is possible to estimate whether or not a person is laughing based on the shape of the end of the mouth. When the shape of the mouth end is approximated to a circular arc, even if the shape change of the mouth end is slight, the radius or diameter of the arc changes greatly, so that the change in the shape can be detected with good sensitivity. Even if the face of a person is tilted in the roll direction in the face image, it can be accurately estimated whether or not the person is laughing.

  According to the sixth invention, since the shape of the lip boundary line when the person is open is learned in advance and whether or not the person is smiling can be estimated based on the learning result, whether or not the person is smiling Can be estimated more accurately.

  According to the seventh aspect, the person state determination device according to the first aspect can be mounted on a vehicle, and the state of the driver of the vehicle can be determined.

(First embodiment)
Hereinafter, the human state determination device 132 according to the first embodiment of the present invention will be described with reference to FIGS. 1 to 12. In the present embodiment, an example in which the driver monitoring system 1 including the person state determination device 132 is mounted on a vehicle will be described. The driver monitoring system 1 is a system that takes an image of the face of a driver of a vehicle on which the driver is mounted and monitors the state of the driver based on the image. The person state determination device 132 estimates whether or not the driver is closing his eyes and whether or not the driver is laughing based on the above image, and the driver is awakened based on each estimation result. It is determined whether or not.

  First, the functional configuration of the driver monitoring system 1 will be described with reference to FIG. FIG. 1 is a block diagram illustrating a functional configuration of the driver monitoring system 1 including the person state determination device 132.

  The driver monitoring system 1 includes a near infrared illumination 11, a near infrared camera 12, and an ECU (Electrical Control Unit) 13. The ECU 13 includes a face orientation detection microcomputer 131 and a person state determination device 132. The driver monitoring system 1 is connected to the vehicle control system 2 via a CAN (Controller Area Network) or the like. The face orientation detection microcomputer 131 and the person state determination device 132 each output data indicating the driver's state to the vehicle control system 2.

  The near-infrared illumination 11 is an illumination device that emits near-infrared light, and is typically an LED (Light Emitting Diode) that emits near-infrared light. The near-infrared illumination 11 is disposed in the vehicle and irradiates the driver with near-infrared light.

  The near-infrared camera 12 is an example of an imaging unit described in the claims. The near-infrared camera 12 is an imaging device having high sensitivity characteristics in the near-infrared region, and is typically a near-infrared CCD camera. By using a camera with high sensitivity characteristics in the near-infrared region, it is possible to capture an image of the object illuminated by the near-infrared illumination 11 with good sensitivity even when the vehicle is dark at night or in a tunnel. It is. The near-infrared camera 12 is disposed at a position where the driver's face can be imaged. The near-infrared camera 12 captures images of the driver's face and surroundings, and outputs the captured images to the ECU 13. Hereinafter, an image photographed by the near-infrared camera 12 is referred to as a camera image. For the sake of explanation, position information in the camera image is represented by a coordinate system in which the lower left corner is the origin, the vertical direction is the Y coordinate, and the horizontal direction is the X coordinate.

  The ECU 13 is a processing device including an information processing device such as the face orientation detection microcomputer 131 and the person state determination device 132, an interface circuit, and the like. The ECU 13 processes the camera image with an information processing device such as the face orientation detection microcomputer 131 and the person state determination device 132. The ECU 13 outputs data indicating the driver's state obtained as a result of the above processing to the vehicle control system 2.

  The face orientation detection microcomputer 131 is typically an information processing device such as a microcomputer. The face orientation detection microcomputer 131 detects the angle of the driver's face orientation in the camera image. In the present embodiment, for the sake of simplicity, an example in which the face orientation detection microcomputer 131 detects the angle in the left-right direction of the driver's face (hereinafter referred to as the face orientation angle α) will be described. The face orientation detection microcomputer 131 outputs the face orientation angle α to the vehicle control system 2.

  The person state determination device 132 is typically an information processing device such as a microcomputer. The person state determination device 132 includes a storage device such as a memory for storing data used for processing to be described later. The person state determination device 132 performs a process for estimating whether or not the driver's eyes are closed (a closed eye estimation process described later) and a process for estimating whether or not the driver is smiling (a smile estimation process described later). Execute. Then, the person state determination device 132 determines whether or not the driver is awake based on the result of each estimation process. Based on the result of the determination, the person state determination device 132 is a signal indicating that the driver is awake (hereinafter referred to as an awake determination signal) or a signal indicating that the driver is not awake ( Hereinafter, the non-wake determination signal is output to the vehicle control system 2.

  The vehicle control system 2 includes a system that controls the vehicle, such as a pre-crash control system. The vehicle control system 2 determines the fatigue level of the driver based on data and signals indicating the driver's state, such as the input face orientation angle α, the arousal determination signal, and the non-awake determination signal. Then, the vehicle control system 2 optimally determines the timing for executing a warning for reporting the danger of a collision between the vehicle and an obstacle, control of collision avoidance travel, and the like according to the driver's fatigue level.

  Next, processing executed by the ECU 13 will be described with reference to FIG. FIG. 2 is a flowchart showing an example of processing executed by the ECU 13.

  In step A1, the ECU 13 determines whether or not the IG power supply is on. If the ECU 13 determines that the IG power supply is on, the ECU 13 proceeds to step A2. On the other hand, when the ECU 13 determines that the IG power supply is off, the ECU 13 ends the process.

  By the process of step A1, the following processes of step A2 to step A5 are repeated while the IG power supply is on.

  In step A2, the ECU 13 acquires a camera image. Specifically, the ECU 13 acquires a camera image output from the near-infrared camera 12 and stores it in a storage device provided in the ECU 13. When the ECU 13 completes the process of step A2, the process proceeds to step A3.

  In step A3, the ECU 13 executes face orientation detection processing. Specifically, the ECU 13 causes the face orientation detection microcomputer 131 to execute a process of detecting the face orientation angle α based on the camera image acquired in step A2.

  Here, the face orientation angle α will be described with reference to FIG. FIG. 3 is a top view of the camera image and the driver's head when the image is taken. In the present embodiment, the driver's head is considered to have a cylindrical shape with the top of the head as the upper surface. The face rotates around the center line of the cylindrical head. The value of the face orientation angle α is α = 0 in a state where it faces the near infrared camera 12. The value of the face orientation angle α increases as the face turns to the right from the state where the face faces the front of the near-infrared camera 12. On the other hand, the value of the face angle α decreases as the face turns to the left from the state where the face faces the near-infrared camera 12. Therefore, the value of the face orientation angle α is a positive value when the face is directed to the right, and is a negative value when the face is directed to the left.

  Hereinafter, an example of a method in which the face orientation detection microcomputer 131 detects the face orientation angle α will be described with reference to FIG. Note that the method of detecting the face orientation angle α by the following face orientation detection microcomputer 131 is an example, and the method of detecting the face orientation angle α is not limited to the following example, and a conventionally known method may be used.

  The face orientation detection microcomputer 131 detects the left end EL and right end ER of the driver's face and the center C of the face in the camera image. The left end EL of the driver's face is the X coordinate of the left end point of the face outline. The right end ER of the driver's face is the X coordinate of the right end point of the face outline. The center C of the face is the X coordinate of the position of the driver's nose. The face direction detection microcomputer 131 detects the left end EL and the right end ER by detecting the contour of the driver's face in the camera image. Further, the center C of the face is detected by detecting the position of the driver's nose in the camera image. Then, the width WL from the left end EL to the center C of the face is calculated. Further, a width WR from the right end ER to the center C of the face is calculated.

  The face orientation detection microcomputer 131 detects the face orientation angle α based on the ratio between the width WL and the width WR calculated as described above. As shown in FIG. 3A, when the face is facing the front with respect to the near-infrared camera 12, the width WR and the width WL are the same value. On the other hand, as shown in FIG. 3B, when the face is facing right, the width WL is larger than the width WR. That is, the ratio of the values of the width WR and the width WL changes according to the size of the face orientation angle α. Therefore, the face orientation detection microcomputer 131 stores in advance a data table that converts the ratio of the width WR and width WL values to the value of the face orientation angle α, and the face WR, the width WL, and the face based on the data table. The orientation angle α is calculated.

  When detecting the face orientation angle α, the face orientation detection microcomputer 131 outputs data indicating the face orientation angle α to the person state determination device 132 and the vehicle control system 2 to complete the process. And ECU13 will advance a process to step A4, if the process of the face direction detection microcomputer 131 is completed.

  In step A4, the ECU 13 determines whether a face orientation is detected. Specifically, in step A3, it is determined whether or not the face orientation detection microcomputer 131 has output data indicating the face orientation angle α. If the ECU 13 determines that the face orientation has been detected, the ECU 13 advances the process to step A5. On the other hand, when the ECU 13 determines that the face orientation is not detected, the ECU 13 returns the process to step A1.

  In step A5, the ECU 13 executes an arousal determination process. Specifically, the ECU 13 causes the person state determination device 132 to execute an arousal determination process shown in FIG. Although the detailed description of the arousal determination process will be described later, the person state determination device 132 determines whether or not the driver is awake in the awakening determination process, and based on the determination result, the awakening determination signal or the non-awakening A determination signal is output to the vehicle control system 2. When the awakening determination process of the human state determination device 132 is completed, the ECU 13 completes the process of step A5 and returns the process to step A1.

  According to the processing of step A4 and step A5, the ECU 13 causes the person state determination device 132 to determine whether or not the driver is awakened only when the direction of the driver's face is detected.

  Hereinafter, the process executed by the person state determination device 132 in the process of step A5 of the ECU 13 described above will be described in detail with reference to FIGS.

  First, main data used by the person state determination device 132 in the awakening determination process will be described with reference to FIG. FIG. 4 is a diagram showing main data (hereinafter referred to as processing data) used by the person state determination device 132 in the awakening determination process. The processing data shown in FIG. 4 is stored in a storage unit such as a memory included in the person state determination device 132. The processing data includes face width W, face angle α, nose position (Nx, Ny), left eyelid opening EL, left eyelid opening threshold ELth, right eyelid opening ER, and right eyelid opening threshold. ERth, closed eye flag, mouth search range center (Mx, My), mouth search range width MW, mouth search range length MH, mouth angle information, smile flag, timer count value C, and timer count value threshold Cth is included.

  The face width W is the width of the driver's face in the camera image. The person state determination device 132 searches for a face outline in the camera image and detects the face width W.

  The face orientation angle α is an angle in the left-right direction of the driver's face described above. The person state determination device 132 acquires the face orientation angle α output from the face orientation detection microcomputer 131.

  The nose position (Nx, Ny) is a coordinate indicating the position of the driver's nose in the camera image. The person state determination device 132 searches the nose portion in the camera image to detect the position of the nose (Nx, Ny). Although details will be described later, the person state determination device 132 sets the center (Mx, My) of the mouth search range based on the position of the nose (Nx, Ny).

  The left eyelid opening degree EL is an interval between the upper and lower eyelids of the driver's left eye in the camera image. Further, the right eyelid opening degree ER is an interval between the upper and lower eyelids of the driver's right eye in the camera image. Although details will be described later, the person state determination device 132 detects the contours of the eyelids of both eyes in the camera image, and detects the distance between the eyelids of the left and right eyes based on the contours. The threshold value ELth for the left eyelid opening is a threshold value for the left eyelid opening EL, and is a predetermined constant. Further, the threshold value ERth of the right eyelid opening is a threshold value of the right eyelid opening ER, and is a predetermined constant. Although the details will be described later, the human state determination device 132 has a value of the left eyelid opening EL that is less than the threshold value ELth of the left eyelid opening, and a value of the right eyelid opening ER that is less than the threshold value ERth of the right eyelid opening. In some cases, it is estimated that the driver has closed his eyes and the closed eye flag is turned on. The closed eye flag is a flag indicating whether or not the driver is estimated to have closed his eyes. The closed eye flag indicates that the driver is closing his eyes when the closed eye flag is on, and indicates that the driver is opening his eyes when the closed eye flag is off.

  The mouth search range is an area for searching for the driver's mouth in the camera image. In the present embodiment, an example will be described in which the mouth search range is set as a rectangle having four sides parallel to the XY coordinate axes of the camera image. The center (Mx, My) of the mouth search range is the center coordinates of the mouth search range in the camera image. The width MW of the mouth search range is the width in the X direction of the mouth search range in the camera image. The vertical length MH of the mouth search range is the length in the Y direction of the mouth search range in the camera image, and is a predetermined constant. The person state determination device 132 sets the mouth search range based on the center (Mx, My) of the mouth search range, the horizontal width MW of the mouth search range, and the vertical length MH of the mouth search range, and the region of the mouth search range 20 The driver's mouth is detected within.

  The mouth corner information is information about the mouth detected by the person state determination device 132. Details of the mouth corner information will be described later with reference to FIG. The person state determination device 132 estimates whether or not the driver is laughing based on the mouth angle information, and switches the smile flag on or off. The smile flag is a flag indicating whether or not it is estimated that the driver is laughing. The smile flag indicates that the driver is smiling when the smile flag is on, and indicates that the driver is not smiling when the smile flag is off.

  The timer count value C is a count value indicating the time measured by the person state determination device 132. The timer count value threshold Cth is a threshold of the timer count value C. Although the details will be described later, when it is estimated that the driver is in a state where the driver is closed and smiling, the person state determination device 132 measures the duration of the state and increases the timer count value C. When the timer count value C reaches the timer count value threshold Cth, the person state determination device 132 determines that the driver is awake.

  Next, the arousal determination process executed by the person state determination apparatus 132 will be described with reference to FIG. FIG. 5 is an example of a flowchart showing awakening determination processing executed by the person state determination device 132.

  In step B1, the human state determination device 132 acquires the face orientation angle α. Specifically, the person state determination device 132 acquires the value of the face orientation angle α calculated by the face orientation detection microcomputer 131 in step A3 of FIG. When the process of step B1 is completed, the person state determination device 132 proceeds with the process to step B2.

  In step B2, the human state determination device 132 detects the driver's face width W in the camera image. Specifically, for example, the face width W is detected by searching for the contour portion of the face in the camera image by edge extraction processing using a known Sobel operator. Then, the person state determination device 132 stores data indicating the detected face width W in a storage device included in the person state determination device 132. When the person state determination device 132 completes the process of step B2, the process proceeds to step B3.

  In step B3, the person state determination device 132 searches for the nose portion in the camera image and detects the position of the nose (Nx, Ny). Specifically, the person state determination device 132 detects the position of the nostril. Then, an intermediate position between the pair of nostrils is detected as a nose position (Nx, Ny). The person state determination device 132 stores data indicating the detected position (Nx, Ny) of the nose in a storage device included in the person state determination device 132. When the person state determination device 132 completes the process of step B3, the process proceeds to step B4.

  The nostril shape detected in step B3 has less individual differences than the eye and mouth shapes. Therefore, the position of the nose can be detected more accurately than the position of the eyes and mouth. On the other hand, since the eye and mouth shapes vary greatly among individuals, it is difficult to accurately detect the position and shape of the eyes and mouth unless an optimum search range is set. Therefore, in the following processing of Step B4 and Step B5, the optimal eye search range and mouth search range are set based on the position of the nose (Nx, Ny), thereby accurately determining the shape of the driver's eyes and mouth. Can be detected.

  In step B <b> 4, the person state determination device 132 performs the closed eye estimation process. The closed eye estimation process is a subroutine process for estimating whether or not the driver is closing both eyes as shown in FIG. FIG. 6 is an example of a flowchart showing a subroutine for the closed eye estimation process. Details of the closed eye estimation process shown in FIG. 6 will be described later. The person state determination device 132 executes the closed eye estimation process, and when it is estimated that the driver has closed both eyes in the process, the closed eye flag is set to ON and stored. On the other hand, in the closed eye estimation process, the person state determination device 132 sets the closed eye flag to OFF and stores it when it is estimated that the driver has opened at least one of the left and right eyes. When the process of step B4 is completed, the person state determination device 132 proceeds with the process to step B5.

  In step B5, the person state determination device 132 executes smile estimation processing. The smile estimation process is a subroutine process for estimating whether or not the driver is smiling as shown in FIG. FIG. 7 is an example of a flowchart showing a subroutine of smile estimation processing. Details of the smile estimation process shown in FIG. 7 will be described later. When the person state determination device 132 executes a smile estimation process and estimates that the driver is laughing in the process, the person state determination device 132 sets the smile flag to ON and stores it. On the other hand, when it is estimated that the driver is not laughing in the smile estimation process, the person state determination device 132 sets the smile flag to OFF and stores it. When the process of step B5 is completed, the person state determination device 132 proceeds with the process to step B6.

  In step B6, the person state determination device 132 determines whether or not the eye-closed flag is on. Specifically, the person state determination device 132 reads the state of the closed eye flag set in step B4 and determines whether it is on or off. When the closed eye flag is on, that is, when it is estimated that the driver is closing the eyes, the person state determination device 132 proceeds with the process to step B7. On the other hand, when the closed eye flag is off, that is, when it is estimated that the driver is opening his / her eyes, the person state determination device 132 proceeds with the process to step B11.

  In step B7, the person state determination device 132 determines whether the smile flag is on. Specifically, the person state determination device 132 reads the state of the smile flag set in step B5 and determines whether it is on or off. When the smile flag is on, that is, when it is estimated that the driver is laughing, the person state determination device 132 proceeds with the process to step B8. On the other hand, when the smile flag is off, that is, when it is estimated that the driver is not laughing, the person state determination device 132 advances the process to step B10.

  In step B8, the person state determination device 132 counts up the timer. Specifically, the person state determination device 132 adds a constant such as 1 to the stored timer count value C and overwrites and stores the timer count value C after the addition in the storage device. When the process of step B8 is completed, the person state determination device 132 proceeds with the process to step B9.

  In step B9, the person state determination device 132 determines whether or not the timer count value C has reached the threshold value Cth of the timer count value. Specifically, the person state determination device 132 reads the timer count value C and the threshold value Cth of the timer count value stored in the storage device, and compares the values. If the timer count value C has reached the threshold value Cth of the timer count value, the person state determination device 132 advances the process to step B13. On the other hand, if the timer count value C has not reached the timer count value threshold Cth, the person state determination device 132 advances the process to step B14.

  In step B10, the person state determination device 132 initializes a timer. Specifically, the person state determination device 132 initializes the timer count value C stored in the storage device. When the process of step B10 is completed, the person state determination device 132 proceeds with the process to step B14.

  In step B11, the person state determination device 132 initializes a timer. Specifically, the person state determination device 132 initializes the value of the timer count value C stored in the storage device, similarly to step B10. When the process of step B11 is completed, the person state determination device 132 proceeds with the process to step B12.

  In step B12, the person state determination device 132 learns the mouth angle. Although details will be described later, the mouth angle is information related to the shape of the driver's mouth in the camera image, and is calculated in the process of step B5. The person state determination device 132 stores the mouth angle as learning information. The person state determination device 132 uses the learning information when estimating whether or not the driver is smiling in the process of step B5 described above. When the process of step B12 is completed, the person state determination device 132 proceeds with the process to step B13.

  In step B13, the person state determination device 132 outputs an arousal determination signal. Specifically, the person state determination device 132 outputs an awakening determination signal to the vehicle control system 2. When the process of step B13 is completed, the person state determination device 132 completes the subroutine of the arousal determination process and returns the process to step A1 of the flowchart shown in FIG.

  In step B14, the person state determination device 132 outputs a non-wake determination signal. Specifically, the person state determination device 132 outputs a non-wake determination signal to the vehicle control system 2. When the process of step B14 is completed, the person state determination device 132 completes the subroutine of the arousal determination process and returns the process to step A1 of the flowchart shown in FIG.

  According to the processing from step B6 to step B14 described above, when the closed eye flag is on and the smile flag is on, the person state determination device 132 measures the duration of the state using a timer. Then, the person state determination device 132 outputs an arousal determination signal when the duration has reached a predetermined time determined by the threshold value Cth of the timer count value. That is, the person state determination device 132 determines that the driver is awake when a predetermined time elapses in a state where it is estimated that the driver is laughing and narrowing his eyes. Also, when the closed eye flag is off, that is, when it is estimated that the driver is in the state where both eyes are open, the person state determination device 132 determines that the driver is awake and outputs an awakening determination signal. Output.

  On the other hand, if the smile flag is off even if the closed eye flag is on, that is, if it is estimated that the driver closes his eyes and is not laughing, the person state determination device 132 causes the driver to wake up. It determines that it is not, and outputs a non-wakefulness determination signal. In addition, even when the closed eye flag is on and the smile flag is on, while the timer count C does not reach the threshold value Cth of the timer count value, the person state determination device 132 causes the driver to wake up. It is determined that there is not, and a non-wakefulness determination signal is output.

  Note that the person state determination device 132 learns the mouth angle when the closed eye flag is off. That is, the person state determination device 132 learns the mouth angle when the driver's eyes are open. Although details will be described later, the person state determination device 132 can more accurately estimate whether or not the driver is laughing by determining whether or not the driver is laughing based on the learning result. .

  According to the above arousal determination process, the person state determination device 132 can prevent erroneous determination that the driver is not awake when the driver is laughing with his eyes narrowed.

  Next, details of the closed eye estimation process executed by the person state determination apparatus 132 in step B4 of the awakening determination process described above will be described with reference to FIG. In addition, the following closed eye estimation process is an example, and the method for estimating whether or not the person state determination device 132 is closed by the driver is not limited to the following example, and a conventional known method may be used.

  In step C1, the person state determination device 132 calculates the left eyelid opening degree EL. Specifically, first, the person state determination device 132 detects the driver's left eye in the camera image. Next, the person state determination device 132 detects the contours of the upper eyelid and the lower eyelid of the left eye by, for example, an edge extraction process using a known Sobel operator. Then, the person state determination device 132 calculates the maximum value of the distance in the Y direction from the contour of the upper eyelid to the contour of the lower eyelid, and stores the maximum value in the storage device as the left eyelid opening degree EL ( (See FIG. 8). When the process of step C1 is completed, the person state determination device 132 proceeds with the process to step C2.

  In step C2, the person state determination device 132 calculates the right eyelid opening degree ER. Specifically, the person state determination device 132 calculates the right eyelid opening ER by the same method as in step C1, and stores it in the storage device (see FIG. 8). When the process of step C2 is completed, the person state determination device 132 proceeds with the process to step C3.

  In step C3, it is determined whether or not the left eyelid opening degree EL is less than a threshold value ELth. Specifically, the person state determination device 132 reads the left eyelid opening degree EL and the left eyelid opening degree threshold value ELth stored in the storage device, and compares the values. If the timer count value C is less than the threshold value Cth of the timer count value, the person state determination device 132 proceeds with the process to step C4. On the other hand, when the timer count value C is equal to or greater than the threshold value Cth of the timer count value, the person state determination device 132 proceeds with the process to step C6.

  According to the process in step C3, it can be estimated whether or not the driver's left eye is closed.

  In Step C4, it is determined whether or not the right eyelid opening degree ER is less than the threshold value ERth. Specifically, the person state determination device 132 reads the right eyelid opening ER and the right eyelid opening threshold ERth stored in the storage device, and compares the magnitudes of the values. If the timer count value C is less than the threshold value Cth of the timer count value, the person state determination device 132 proceeds with the process to step C5. On the other hand, when the timer count value C is equal to or greater than the threshold value Cth of the timer count value, the person state determination device 132 proceeds with the process to step C6.

  According to the process of step C4, it can be estimated whether or not the right eye of the driver is closed.

  In step C5, the person state determination device 132 turns on the closed eye flag. Specifically, the person state determination device 132 sets the closed eye flag to ON, and stores the state of the closed eye flag in the storage device. When the process of step C5 is completed, the person state determination device 132 completes the closed eye estimation process subroutine and advances the process to step B5 of the arousal determination process shown in FIG.

  In step C6, the person state determination device 132 turns off the closed eye flag. Specifically, the person state determination device 132 sets the closed eye flag to OFF, and stores the state of the closed eye flag in the storage device. When the process of step C6 is completed, the person state determination device 132 completes the closed eye estimation process subroutine and advances the process to step B5 of the arousal determination process shown in FIG.

  According to the processing of steps C1 to C6, when the left eyelid opening and the right eyelid opening are less than the respective threshold values, that is, when it is estimated that both the left and right eyes are closed, the closed eye flag is turned on. In addition, when either the left eyelid opening or the right eyelid opening is equal to or greater than the threshold value, that is, when it is estimated that at least one of the left and right eyes is open, the closed eye flag is turned off.

  Next, details of the smile estimation process executed by the person state determination apparatus 132 in step B5 of the awakening determination process described above will be described with reference to FIG.

  In step D1, the person state determination device 132 sets the mouth search range 20. The mouth search range setting method will be specifically described with reference to FIG. FIG. 8 is a diagram showing the mouth search range 20 set in the camera image.

  First, the person state determination device 132 calculates the width MW of the mouth search range 20. Since the size of the driver's mouth in the camera image changes according to the direction of the driver's face, it is desirable to change the size to an optimum size according to the value of the face orientation angle α. Therefore, the human state determination device 132 stores in advance a data table for converting the value of the face orientation angle α into the value of the width MW of the mouth search range 20, and based on the face orientation angle α and the data table, the mouth search range 20 The horizontal width MW of is calculated.

Next, the person state determination device 132 calculates the center (Mx, My) of the mouth search range by the following equation based on the nose position (Nx, Ny) detected in step B3 of the arousal determination process.
Mx = Nx
My = Ny−ΔM
ΔM is a constant stored in advance in the storage device of the person state determination device 132. Since the relative positional relationship between the position of the person's mouth and the position of the nose is small in individual differences, if the position of the nose is known, the approximate position of the mouth can be estimated by the above formula. The person state determination device 132 sets the range of the horizontal width MW and the vertical length MH as the mouth search range 20 around the center (Mx, My) of the mouth search range obtained as described above. When the person state determination device 132 completes the process of step D1, the process proceeds to step D2.

  In the description of the processing in step D1, the example in which the human state determination device 132 sets the mouth search range 20 to a rectangle has been described. However, the mouth search range 20 is not limited to the rectangle as described above, and the mouth search range 20 is determined by the driver. Various other shapes may be used as long as the region including the mouth portion can be set. For example, the person state determination device 132 may set the mouth search range 20 to a shape such as an ellipse.

  In step D2, the human state determination device 132 detects a lip boundary line. Specifically, the person state determination device 132 detects the lip boundary line 30 in the mouth search range 20 from the vertical direction by, for example, edge extraction processing using a known Sobel operator. FIG. 9 shows the lip boundary line 30. FIG. 9 is an enlarged view of the driver's mouth in the camera image. When the person state determination device 132 completes the process of step D2, the process proceeds to step D3.

  In step D3, the human state determination device 132 extracts a line segment pair. A process in which the human state determination device 132 extracts line segment pairs will be described with reference to FIGS. 9 and 10. First, the person state determination device 132 divides the lip boundary line 30 into N equal parts (N is an integer) as shown in FIG. In this embodiment, as an example, a case where N = 8, that is, an example in which the lip boundary line is equally divided will be described. Next, the person state determination device 132 linearly approximates each lip boundary line divided into N as shown in FIG. FIG. 10 is a diagram showing a lip boundary line approximated by a straight line. Here, for the sake of explanation, the line segments obtained by the straight line approximation are respectively denoted by reference numerals as a line segment 31 to a line segment 38 in order from the left of the image. The line segments 31 to 38 obtained by the above process are processed in the process described later with the M-th line segments (M is an integer) from both ends of the lip boundary line 30 as a pair of line segments. For example, if M = 1 is the first line segment pair, P1 is composed of the line segment 31 and 38, and M = 2 is the line pair P2, the line pair P2 is the line segment. It consists of a minute segment 32 and a line segment 37. When the process of step D3 is completed, the person state determination device 132 proceeds with the process to step D4.

  The person state determination device 132 temporarily estimates whether or not the driver is laughing for each of a plurality of designated line segment pairs by the processing of the following steps D4 to D6, and indicates a flag (temporary smile flag) indicating the result. ) Is set. Then, the person state determination device 132 comprehensively determines the smile temporary flag set for each line segment pair in the process of step D7 described later, and finally estimates whether or not the driver is laughing. Set the estimation flag.

  In step D4, the person state determination device 132 selects one line segment pair from the specified line segment pair. The designated line segment pair is a line segment pair that is a target for provisional estimation as to whether or not the driver is laughing in the processing after Step D5 described later. The person state determination device 132 stores a designation number Q in advance in order to determine which line segment pair is the designated line segment pair. The designated number Q corresponds to M in the description of step D3. For example, when the person state determination device 132 stores 1 and 2 as the designated number Q in advance, the first number from both ends of the lip boundary line is the first. The line segment pair P1 and the second line segment pair P2 are designated line segment pairs. The person state determination device 132 selects one of the specified line segment pairs determined as described above. Hereinafter, the designated line segment pair selected in step D4 is referred to as a selected line segment pair. When the process of step D4 is completed, the person state determination device 132 proceeds with the process to step D5.

  According to the processing of step D3 and step D4, the human state determination device 132 changes the rejection of the near-infrared camera 12 and the driver's face, and the size of the driver's mouth reflected in the camera image. Even when is changed, a predetermined portion of the lip boundary line can be extracted and selected as a line segment pair.

  In step D5, the person state determination device 132 executes a temporary smile flag setting process. The smile temporary flag setting process is a process of temporarily estimating whether or not the driver is laughing based on the selected line segment pair. Here, the smile temporary flag setting process will be described with reference to FIGS. 11 and 12. Note that FIG. 11 is a diagram showing mouth angle information data processed in the temporary smile flag setting process. FIG. 12 is an example of a flowchart showing a subroutine of smile temporary flag setting processing.

  First, mouth angle information data processed in the smile temporary flag setting process will be described with reference to FIG. As the mouth angle information, the person state determination device 132 sets the left mouth angle θL, the right mouth angle θR, the learning information, the sum value Σθ, the sum value threshold Σθth, the average left mouth angle AθL, and the average right mouth for each specified line segment pair. The angle AθR, the left mouth corner upper deviation σθL, the right mouth corner upper deviation σθR, and the smile temporary flag are stored and processed.

  The left exit angle θL is an angle formed by a left line segment and a horizontal line as viewed from the driver among line segments forming a line segment pair. Assuming that the left mouth angle θL of the line segment pair P1 is θL_P1, as shown in FIG. 10, the left mouth angle θL_P1 is an angle formed by the line segment 38 and the horizontal line. On the other hand, the right exit angle θR is an angle formed by a right-side line segment and a horizontal line as viewed from the driver among line-segment pairs. Therefore, if the right mouth angle θR is θR_P1, as shown in FIG. 10, the right mouth angle θR_P1 is an angle formed by the line segment 31 and the horizontal line. When the driver is smiling, the lip boundary is curved and the values of the left mouth angle θL and the right mouth angle θR are increased. The mouth angle described in step B12 of the awakening determination process is a generic name for the left mouth angle θL and the right mouth angle θR. As described above, the size of the mouth angle changes depending on whether or not the driver is laughing. Therefore, it is possible to estimate whether or not the driver is laughing using the mouth angle.

The total value Σθ is a total value obtained by adding the values of the latest left mouth angle θL and right mouth angle θR of the line segment pair, and can be expressed by the following equation.
Σθ = θL + θR
When the driver is smiling, the values of the left mouth angle θL and the right mouth angle θR are large, so the total value Σθ is large.

  The learning information is a generic name for a plurality of mouth angle data groups stored in step B12 of the awakening determination process. The person state determination device 132 stores the mouth angles in the storage device as learning information by the number determined in the newly calculated order. In the present embodiment, description will be made assuming that the person state determination device 132 stores the latest three left mouth angles θL and right mouth angles θR. The values of the left mouth angle stored as the learning information are set to θL1, θL2, and θL3 in the newly stored order. Similarly, the value of the right mouth angle stored as the learning information is set to θR1, θR2, and θR3 in the order in which they are newly stored. The process of storing the mouth angle as learning information is executed by the person state determination device 132 in step B12 of the awakening determination process. At this time, since the person state determination device 132 stores the mouth angle data up to three, the oldest mouth angle data of the stored learning information is overwritten with the latest mouth angle data.

  The average left mouth angle AθL is an average value of the left mouth angles θL of the selected line segment pairs stored as learning information. The average right mouth angle AθR is an average value of the right mouth angle θR of the pair of selected line segments stored as learning information.

  The left mouth angle upper deviation σθL is a standard deviation of the left mouth angle θL having a value larger than the average left mouth angle AθL among the left mouth angles θL of the pair of selected line segments stored as learning information. Further, the right mouth angle upper deviation σθR is a standard deviation of the right mouth angle θR having a value larger than the average right mouth angle AθR among the right mouth angles θR of the pair of selected line segments stored as learning information.

  The threshold value Σθth of the sum value is a threshold value of the sum value Σθ. Although details will be described later, the threshold value Σθth of the total value is calculated based on the stored mouth angle. The person state determination device 132 sets the temporary smile flag to ON or OFF depending on whether or not the value of the total value Σθ is equal to or greater than the threshold value Σθth of the total value.

  The temporary smile flag is a flag indicating a result of temporarily estimating whether or not the driver is laughing based on the selected line segment pair. When the smile temporary flag is on, it is indicated that the driver is estimated to be laughing based on the selected line segment pair. On the other hand, when the smile temporary flag is off, it is indicated that it is estimated that the driver is not laughing based on the selected line segment pair. Although details will be described later, the person state determination device 132 sets the smile temporary flag to ON or OFF based on the total value Σθ of the selected line segment pair and the threshold value Σθth of the total value.

  Next, the smile temporary flag setting process subroutine will be described with reference to FIG.

  In step E1, the person state determination device 132 calculates a mouth angle. Specifically, the person state determination device 132 calculates the mouth angle of the selected line segment pair, that is, the left mouth angle θL and the right mouth angle θR, and stores them in the storage device. For example, when the line segment pair P1 is selected as the selected line segment pair, the person state determination device 132 calculates and stores the left mouth angle θL_P1 and the right mouth angle θR_P1. When the process of step E1 is completed, the person state determination device 132 advances the process to step E2.

  In step E2, the person state determination device 132 calculates the sum value of the mouth angles. Specifically, the human state determination device 132 reads the latest values of the left mouth angle θL and the right mouth angle θR calculated in step E1, and calculates and stores the sum value Σθ from the above formula based on the values. To do. When the process of step E2 is completed, the person state determination device 132 advances the process to step E3.

In step E3, the person state determination device 132 calculates a threshold value Σθth of the sum value. Specifically, first, the person state determination device 132 calculates the average left mouth angle AθL based on the left mouth angle stored as learning information. Next, the person state determination device 132 calculates the left mouth angle upper deviation σθL based on the calculated average left mouth angle AθL. Similarly, the person state determination device 132 calculates the average right mouth angle AθR and the right mouth angle upper deviation σθR. Next, the person state determination device 132 calculates and stores a threshold value Σθth of the sum value by the following equation.
Σθth = 3 (σθL + σθR) × ε
In the above equation, ε is a predetermined coefficient. When the process of step E3 is completed, the person state determination device 132 advances the process to step E4.

  According to the processing of step E3 and step B12 of the arousal determination processing, the person state determination device 132 learns in advance the shape of the mouth in a state where the driver has opened his eyes, and the threshold value of the sum value based on the learned information thus learned. Σθth can be calculated.

  In step E4, the person state determination device 132 determines whether or not the sum value Σθ is equal to or greater than the sum value threshold Σθth. Specifically, the person state determination device 132 reads the sum value Σθ and the sum value threshold value Σθth from the storage device, and compares the values. If the sum value Σθ is equal to or greater than the sum value threshold Σθth, the person state determination device 132 advances the process to step E5. On the other hand, when the sum value Σθ is less than the sum value threshold Σθth, the person state determination device 132 advances the process to step E6.

  In step E5, the human state determination device 132 turns on the smile temporary flag for the selected line segment pair. Specifically, the person state determination device 132 sets the smile temporary flag for the selected line segment pair to ON, and stores the setting in the storage device. When the process of step E5 is completed, the human state determination device 132 completes the smile temporary flag setting process, and advances the process to step D6 of the smile estimation process shown in FIG.

  In step E6, the human state determination device 132 turns off the smile temporary flag for the selected line segment pair. Specifically, the person state determination device 132 sets the smile temporary flag for the selected line segment pair to OFF, and stores the setting in the storage device. When the process of step E6 is completed, the human state determination device 132 completes the smile temporary flag setting process, and advances the process to step D6 of the smile estimation process shown in FIG.

  By the processing from step E4 to step E6, the person state determination device 132 sets the smile temporary flag to ON when the sum value Σθ is equal to or greater than the sum value threshold value Σθth. On the other hand, when the sum value Σθ is less than the sum value threshold value Σθth, the person state determination device 132 sets the smile temporary flag to OFF.

  According to the processing from step E4 to step E6, the person state determination device 132 can determine whether or not the person is laughing based on the threshold value Σθth of the total value reflecting the learning result. Therefore, the person state determination device 132 can easily identify the difference between the mouth shape in the normal state of the driver and the mouth shape in the state where the driver is laughing, and whether or not the driver is laughing. Can be estimated more accurately.

  According to the smile temporary flag setting process described above, the person state determination device 132 is based on the learning result of the mouth angle of the line segment pair when the driver is not closing his eyes, that is, when he is not laughing. A temporary smile flag can be set by presuming whether the driver is laughing or not.

  In addition, in the smile temporary flag setting process described above, if the end of the lip boundary line is specified as a specified line segment pair, it can be temporarily estimated whether the driver is laughing according to the shape of the end of the mouth. . Since the shape of the end of the mouth changes significantly when the driver is laughing, it is possible to accurately determine whether or not the driver is laughing according to the smile temporary flag setting process.

  Returning to the description of the flowchart of FIG. 7, in step D <b> 6, the person state determination device 132 determines whether or not all the designated line segment pairs have been selected. If the person state determination device 132 has selected all the specified line segment pairs, the process proceeds to step D7. On the other hand, if all the designated line segment pairs have not been selected, the person state determination device 132 returns the process to step D4 and selects an unselected designated line segment pair. If the smile temporary flag can be finally set for all the specified line segment pairs, the order in which the person state determination device 132 selects the specified line segment pairs may be determined in advance or determined randomly. It doesn't matter.

  In step D7, the person state determination device 132 determines whether all the smile temporary flags are on. If all the smile temporary flags are on, the person state determination device 132 proceeds with the process to step D8. On the other hand, if any of the temporary smile flags for the designated line segment pair is OFF, the person state determination apparatus 132 proceeds with the process to step D8.

  In step D8, the person state determination device 132 sets the smile flag to ON. When the person state determination apparatus 132 completes the process of step D8, the person state determination apparatus 132 completes the smile estimation process and advances the process to step B6 of the awakening determination process.

  In step D9, the person state determination device 132 sets the smile flag to OFF. When the person state determination device 132 completes the process of step D9, the person state determination apparatus 132 completes the smile estimation process and advances the process to step B6 of the awakening determination process.

  According to the processing from step D7 to step D9 described above, when all the smile temporary flags are on, the person state determination device 132 sets the smile flag to on, and any of the smile temporary flags for the specified line segment pair is set. If it is off, the smile flag is set off.

  According to the smile estimation process described above, the person state determination device 132 extracts a line segment pair of the driver, and whether the driver is laughing for each line segment pair, that is, for each partial shape of the mouth. It is possible to accurately estimate whether or not the driver is laughing by temporarily estimating whether or not the driver is laughing.

  In the first embodiment, the example in which the person state determination device 132 uses two pairs of line segments as the specified line segment pair in the process of step D4 of the smile estimation process has been described. The number of designated line segment pairs processed by 132 is not limited to the above, and one or more pairs of line segments may be used as the designated line segment pair.

  In the first embodiment, the example in which the number of mouth angles stored as learning information by the person state determination device 132 is three is shown. However, the number of mouth angle data to be stored is not limited to the above, and more May be stored, or less data may be stored. If the threshold value Σθth of the total value is not calculated from the learning information and is set to a predetermined fixed value, the process of step B12 of the awakening determination process may be omitted and the learning information may not be stored.

  In the first embodiment, the example is shown in which the person state determination device 132 calculates the angle formed by the horizontal line and the line segment as the mouth angle in Step E1 of the smile temporary flag setting process. First, an angle formed by a predetermined reference line and a line segment in the image may be used as the mouth angle.

  In the first embodiment, in step E3 of the smile temporary flag setting process, the person state determination device 132 calculates the threshold value Σθth of the sum value based on the standard deviation of the angle larger than the average in the learning information. Although an example has been shown, the method for determining the threshold value Σθth of the total value is not limited to the above. For example, the threshold value Σθth of the total value may be calculated based on other parameters by the human state determination device 132, or a fixed value obtained experimentally may be stored in advance.

(Second Embodiment)
In step D3 of the smile estimation process of the first embodiment, an example in which the human state determination device 132 extracts a line segment pair by linearly approximating the N-divided lip boundary line has been described. 132 may extract a pair of arcs by approximating the N equally divided boundary line to a circular arc, and may estimate whether the driver is laughing based on the arc pair. Hereinafter, a process in which the human state determination device 132 approximates an N-divided boundary line to an arc to extract an arc pair and estimates whether the driver is laughing based on the arc pair will be described with reference to FIGS. Reference is made to FIG. In addition, about the structure of the driver | operator monitoring system containing the person state determination apparatus 132 and the person state determination apparatus 132 which concern on 2nd Embodiment, since it is the same as that of the structure which concerns on 1st Embodiment, description is abbreviate | omitted.

  The processing executed by the human state determination device 132 according to the second embodiment is different from the processing of the smile estimation processing from step D3 to step D5, and the other processing is the same as the processing of the first embodiment. . Hereinafter, among the processes executed by the human state determination device 132 according to the second embodiment, processes different from the first embodiment will be described.

  In step D3, the human state determination device 132 extracts a pair of arcs. A process in which the human state determination device 132 extracts line segment pairs will be described with reference to FIG. FIG. 13 is a diagram showing a lip boundary line approximated by an arc. FIG. 13 shows a state where the lip boundary line in a state where the driver is not laughing is approximated by an arc. First, the person state determination device 132 divides the lip boundary line 30 into N equal parts (N is an integer) as shown in FIG. In the second embodiment, as an example, a case where N = 4, that is, an example in which the lip boundary line is equally divided will be described. Next, as shown in FIG. 13A, the human state determination device 132 approximates each lip boundary line divided into N parts to an arc. Here, for the sake of explanation, arcs obtained by approximating arcs are respectively denoted as arcs 41 to 44 in order from the left of the image. The arcs 41 to 44 obtained by the above processing are processed with the M-th (M is an integer) arcs from both ends of the lip boundary line 30 as a set of arc pairs. For example, if the M = 1st arc pair is Pa1, the arc pair Pa1 is constituted by an arc 41 and an arc 44, and if the M = 2nd arc pair is Pa2, the arc pair Pa2 is constituted by an arc 42 and an arc 43. Is done. When the process of step D3 is completed, the person state determination device 132 proceeds with the process to step D4.

  In step D4, the person state determination device 132 selects one pair from the designated arc pair. The designated arc pair is an arc pair that is a target for provisional estimation as to whether or not the driver is laughing in the processing after Step D5 described later. The person state determination device 132 stores a designated number Q in advance in order to determine which arc pair is designated as the designated arc pair. The designation number Q corresponds to M in the description of step D3 described above. For example, when the person state determination device 132 has previously stored 1 and 2 as the designation number Q, the first designation from both ends of the lip boundary line The arc pair Pa1 and the second arc pair Pa2 are designated arc pairs. The person state determination device 132 selects one of the designated arc pairs determined as described above. Hereinafter, an example in which the designated arc pair selected in Step D4 is referred to as a selected arc pair and the person state determination device 132 selects the arc pair Pa1 will be described below. When the process of step D4 is completed, the person state determination device 132 proceeds with the process to step D5.

  In step D5, the person state determination device 132 executes a temporary smile flag setting process. The smile temporary flag setting process according to the second embodiment will be described below.

  First, data processed in the temporary smile flag setting process according to the second embodiment will be described. For each designated arc pair, the human state determination device 132 includes a left arc diameter RL, a right arc diameter RR, learning information, a sum value ΣR, a sum value threshold ΣRth, an average left arc diameter ARL, an average right arc diameter ARR, and a left arc. The diameter upper deviation σRL, the right arc diameter upper deviation σRR, and the temporary smile flag are stored and processed.

  The human state determination device 132 according to the second embodiment sets the temporary smile flag using the left arc diameter RL and the right arc diameter RR instead of the mouth angle. The left arc diameter RL is the radius of the left arc as viewed from the driver among the line segments forming the arc pair. On the other hand, the right arc diameter RR is the radius of the right arc when viewed from the driver among the line segments forming the arc pair. The left arc diameter RL and the left arc diameter RL are collectively referred to as an arc diameter.

  As shown in FIG. 13, the left arc diameter of the arc pair Pa1 is RL_Pa1, and the right arc diameter of the arc pair Pa1 is RR_Pa1. Here, when the driver laughs, the arc diameter changes as shown in FIG. FIG. 14 is a diagram illustrating a state in which the lip boundary line in a state where the driver laughs is approximated by an arc. When the driver is smiling, the lip boundary line is curved, so the left arc diameter RL_Pa1 shown in FIG. 14 is smaller than the left arc diameter RL_Pa1 shown in FIG. Similarly, the right arc diameter RR_Pa1 shown in FIG. 14 is smaller than the right arc diameter RR_Pa1 shown in FIG. As described above, since the size of the arc diameter changes depending on whether or not the driver is laughing, it is possible to estimate whether or not the driver is laughing using the arc diameter.

The total value ΣR is a total value obtained by adding the values of the latest left arc diameter RL and right arc diameter RR of the arc pair, and can be expressed by the following equation.
ΣR = RL + RR
When the driver is smiling, the values of the left arc diameter RL and the right arc diameter RR are increased, and the combined value ΣR is increased.

  Next, a smile temporary flag setting process subroutine according to the second embodiment will be described.

  In step E1, the human state determination device 132 calculates an arc diameter. Specifically, the person state determination device 132 calculates the arc diameter of the selected arc pair, that is, the left arc diameter RL and the right arc diameter RR, and stores them in the storage device. For example, if the line segment pair P1 is selected as the selected line segment pair, the person state determination device 132 calculates and stores the left arc diameter RL_Pa1 and the right arc diameter RR_P1. When the process of step E1 is completed, the person state determination device 132 advances the process to step E2.

  In step E2, the person state determination device 132 calculates the total value of the arc diameters. Specifically, the person state determination device 132 reads the latest values of the left arc diameter RL and the right arc diameter RR calculated in step E1, and calculates and stores the sum value ΣR from the above formula based on the values. To do. When the process of step E2 is completed, the person state determination device 132 advances the process to step E3.

  In step E <b> 3, the person state determination device 132 calculates a threshold value ΣRth of the sum value. The person state determination device 132 learns the arc diameter in advance as in the first embodiment, and calculates the threshold value ΣRth of the sum value. When the process of step E3 is completed, the person state determination device 132 advances the process to step E4.

  In step E4, the person state determination device 132 determines whether or not the sum value ΣR is equal to or less than the sum value threshold ΣRth. Specifically, the person state determination device 132 reads the sum value ΣR and the sum value threshold value ΣRth from the storage device, and compares the values. When the sum value ΣR is equal to or less than the sum value threshold ΣRth, the person state determination device 132 proceeds with the process to step E5. On the other hand, when the sum value ΣR is larger than the sum value threshold ΣRth, the person state determination device 132 advances the process to step E6.

  Since the processing of step E5 and step E6 is the same as that of the first embodiment, description thereof is omitted.

  Through the processing from step E4 to step E6, the person state determination device 132 sets the smile temporary flag to ON when the sum value ΣR is equal to or less than the sum value threshold ΣRth. On the other hand, when the sum value ΣR is larger than the sum value threshold ΣRth, the person state determination device 132 sets the smile temporary flag to OFF. Even when the arc diameter is used instead of the mouth angle as described above, it is possible to temporarily set whether or not the driver is smiling as in the first embodiment and set the smile temporary flag.

In the second embodiment, the example in which the person state determination device 132 sets the smile temporary flag depending on whether or not the sum ΣR is greater than the sum threshold ΣRth has been described. The setting method is not limited to the above. For example, the smile temporary flag may be turned on when at least one of the following two conditions is satisfied, and the smile temporary flag may be turned off when neither of the following two expressions is satisfied.
RL ≦ Rth−ρ
RR ≦ Rth−τ
In the above expression, ρ and τ are predetermined constants. When the smile temporary flag is set as described above, when the arc diameter of one arc of the arc pair is a large value and the arc diameter of the other arc is a small value, that is, the driver Even when the person is laughing with the edges raised, the person state determination device 132 can accurately determine whether or not the driver is laughing.

  In the second embodiment, the example in which the human state determination device 132 sets the temporary smile flag based on the radius of the arc has been described. However, instead of using the radius of the arc, the diameter of the arc may be used. Absent.

  In the first embodiment and the second embodiment described above, the example in which the human state determination device 132 divides the lip boundary into eight equal parts has been described in the process of step D3 of the smile estimation process. The number of dividing the lip boundary line in the process of D is not limited to the above, and it may be divided into two or more. The human state determination device 132 does not necessarily need to divide the lip boundary line at equal intervals, and may partially change the interval at which the lip boundary line is divided. Furthermore, the person state determination device 132 may cut out a certain distance from both ends of the lip boundary line and extract it as the line segment pair.

  Further, in the first embodiment and the second embodiment described above, in the processing from step D7 to step D9 of the smile estimation process, the person state determination device 132 smiles when all the smile temporary flags are on. Although an example has been described in which the flag is turned on and the smile flag is turned off when any of the smile temporary flags is off, the method for setting the smile flag is not limited to the above. For example, the person state determination device 132 may turn on the smile flag when 70% or more of the total smile temporary flag is on.

  In the first embodiment and the second embodiment, it is estimated that the person state determination device 132 is narrowing the eyes while the driver is laughing in the processing from Step B6 to Step B14 of the arousal determination processing. Although the example in which it is determined that the driver is awake when a predetermined time elapses in the state in which the driver is in the state of being held, the person state determination device 132 determines whether the driver is awake without waiting for the elapse of the predetermined time. It may be determined. For example, the processing of step B8, step B9, step B10, and step B11 may be omitted.

  In the first embodiment and the second embodiment described above, in the processing from Step B6 to Step B14 of the arousal determination processing, the person state determination device 132 estimates that the driver is closed, and In the above description, an example in which it is determined that the driver is awake when the state estimated to be laughing (hereinafter referred to as a smile awake state) continues for a certain period of time has been described. The frequency at which the driver is in a smile awake state may be calculated, and it may be determined whether the driver is awake based on the frequency.

  For example, it is assumed that the person state determination device 132 periodically determines whether or not the person is awake. The person state determination device 132 calculates, as the frequency, the total value of the time during which the driver is in a smiling and awakening state within one period of time. Then, when the frequency reaches a predetermined threshold, the person state determination device 132 determines that the driver is awake, and when the frequency does not reach the threshold, the driver is not awake. judge.

  In the first embodiment and the second embodiment described above, in the awakening determination process of step A5 in FIG. 2, the person state determination device 132 searches for the driver's eyes and mouth according to the face orientation angle α. Although the example of setting the range has been described, the human state determination device 132 may not necessarily change the search range according to the face orientation angle α. When the human state determination device 132 does not change the search range according to the face orientation angle α, the process of step A3 may be omitted.

  In the first and second embodiments, the example in which the person state determination device 132 determines the state of the driver of the vehicle has been described. Is not limited to a vehicle driver. For example, the state of an occupant other than the driver may be determined, or the present invention may be applied so as to determine the state of a pilot such as an aircraft.

  The person state determination apparatus according to the present invention is useful as a person state determination apparatus that can correctly determine the awakening state of a person according to whether or not the person is smiling.

The block diagram which shows the function structure of the driver | operator monitoring system 1 containing the person state determination apparatus 132. The flowchart which shows an example of the process which ECU13 performs Camera image and top view of the driver's head at the time the image was taken The figure which shows the main data which the person state determination apparatus 132 uses in an arousal determination process An example of a flowchart showing awakening determination processing executed by the person state determination device 132 An example of a flowchart showing a subroutine for eye closure estimation processing An example of a flowchart showing a subroutine for smile estimation processing Is a diagram showing the mouth search range 20 set in the camera image Enlarged view of the driver's mouth in the camera image Diagram showing lip boundary line approximated by a straight line The figure which shows the data of mouth corner information processed in a smile temporary flag setting process An example of a flowchart showing a subroutine of smile temporary flag setting processing The figure which shows a mode that the lip boundary line of the state where the driver is not laughing is approximated by an arc The figure which shows a mode that the lip boundary line of the state where the driver laughed is approximated by an arc

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Driver monitoring system 2 Vehicle control system 11 Near infrared illumination 12 Near infrared camera 13 ECU
20 mouth search range 30 lip boundary line 31, 32, 33, 34, 35, 36, 37, 38 line segment 41, 42, 43, 44 arc 131 face direction detection microcomputer 132 human state determination device

Claims (7)

Imaging means for capturing a face image by photographing a person's face;
Closed eye estimation means for estimating whether or not the person has eyes closed based on the face image;
Smile estimation means for estimating whether or not the person is laughing based on the mouth image of the person included in the face image;
(A) When the closed eye estimation means estimates that the person's eyes are open, or when the smile estimation means estimates that the person is smiling, it is determined that the person is awake. (B) If the smile estimation means estimates that the person is not laughing, and the closed eye estimation means estimates that the person's eyes are closed, it is determined that the person is not awake A person state determination device comprising awakening determination means. A lip boundary detecting means for detecting a lip boundary that is a boundary between the upper lip and the lower lip in the face image;
The person state determination apparatus according to claim 1, wherein the smile estimation unit estimates whether or not a person is laughing based on a shape of the lip boundary line. Among the line segments formed by dividing the lip boundary line into N parts in the horizontal direction (N is an integer of 2 or more), a line segment for extracting the line segment pair located at the Mth (M is an integer of 1 or more) from the left and right ends It further comprises a pair extraction means,
The person state determination device according to claim 2, wherein the smile estimation unit estimates whether or not a person is laughing based on the extracted line segment pair. Line segment approximating means for linearly approximating each line segment forming the line segment pair,
An angle calculating means for calculating an angle formed by each straight line segment forming the line pair and a reference line;
The person state determination apparatus according to claim 3, wherein the smile estimation unit estimates whether the person is smiling based on the angle. Arc approximating means for approximating each line segment forming the line segment to an arc line segment by arc approximation;
A diameter calculating means for calculating a radius or a diameter of each arc line segment forming the line segment pair;
The person state determination apparatus according to claim 3, wherein the smile estimation unit estimates whether or not the person is laughing based on a radius or a diameter of each arc line segment forming the line segment pair. Learning means for learning information related to the shape of the lip boundary only when the closed eye estimation means estimates that the person is open;
The person state determination apparatus according to claim 2, wherein the smile estimation unit estimates whether or not the person is laughing based on the learned result. The person is a driver of the vehicle;
The person state determination apparatus according to claim 1, wherein the person state determination apparatus is mounted on the vehicle.

Images