JP2009166783A - Symptom inference device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a symptom inference device that can infer a physiological symptom that gives uncomfortable feeling to a driver in a prior stage that leads to a dangerous driving state or the like. <P>SOLUTION: The symptom inference device has an image pickup device, a face data generation means, and an inference means. The image pickup device captures images in the direction of the driver seat from the front of the driver seat. The face data generation means detects the state of the driver's face using the images captured by the image pickup device and generates face data that show the state of the face. The inference device presumes the physiological symptom which gives uncomfortable feeling to the driver in a prior stage that leads to a dangerous driving state or the like. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a symptom estimation apparatus, and more particularly, to a symptom estimation apparatus that can detect a pilot's condition by detecting a state of a pilot's face.

2. Description of the Related Art Conventionally, there is a driver state determination device that images a driver's face with a camera or the like and determines a driver's symptom from the captured image (for example, see Patent Document 1). The driver state determination device disclosed in Patent Document 1 captures a driver's face with a camera, and analyzes the captured face image, so that the driver can sleep, look aside, look aside, etc. Detect symptoms that lack cognitive judgment. And when the said driver | operator state determination apparatus detects the said state, it warns a driver and controls the vehicle which a driver drives in order to avoid danger beforehand.
JP 2005-108033 A

  Here, a driver's symptom that lacks cognitive judgment when driving is, for example, a symptom in which a collision between a driving vehicle and another object is unavoidable or a collision is likely to occur. It is a dangerous driving condition that leads to connection. On the other hand, there is a dangerous driving state one step ahead where it is expected that if the driver is left as it is, such as feeling tired, drowsy or uncomfortable, the above dangerous driving state is expected. However, the driver state determination device disclosed in Patent Document 1 detects a driver symptom in a dangerous driving state that immediately leads to an accident, and a symptom of the driver in the previous stage that leads to the dangerous driving state. It was not intended to detect physiological symptoms that made the driver feel uncomfortable.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to estimate physiological symptoms that give an uncomfortable feeling to the driver, such as a driver's symptom that may be a preliminary stage leading to a dangerous driving state. The object is to provide a symptom estimation device.

  In order to achieve the above object, the present invention has the following features. 1st invention is the symptom estimation apparatus which estimates the symptom of the pilot who sat down in the cockpit of the moving body. The symptom estimation device includes an imaging unit, a face information generation unit, and an estimation unit. The imaging means images the direction of the cockpit from the front of the cockpit. The face information generation means detects the state of the driver's face using the image picked up by the image pickup means, and generates face information indicating the face state. Based on the face information, the estimating means estimates whether or not a physiological symptom that causes the driver to feel uncomfortable, such as a symptom that may be a stage before the driver leads to a dangerous driving state. In addition, the said symptom estimation apparatus is provided in moving bodies, such as a motor vehicle, an airplane, a ship. The pilot is a concept including a driver of a car, a pilot of a plane, a steering of a ship, and the like. The cockpit is a concept including a driver's seat of an automobile, an airplane's cockpit, a ship's steering seat, and the like.

  In a second aspect based on the first aspect, the face information generation means detects the state of the eyes of the driver using the image captured by the imaging means, and generates the eye state as face information.

  In a third aspect based on the second aspect, the face information generation means uses the image picked up by the image pickup means to determine whether the operator's eyes are in an open state or a closed eye state. And the number of times the pilot's eyes are opened and closed per unit time is generated as face information.

  In a fourth aspect based on the second aspect, the face information generating means detects an eye fullness state of the pilot's eyes using the image captured by the image capturing means, and uses the eye fullness state of the pilot's eyes as face information. Generate.

  In a fifth aspect based on the fourth aspect, the face information generation means extracts the pilot's white-eye portion using the image picked up by the image pickup means, and based on the edge length at which the luminance changes in the white-eye portion. And detecting the hyperemia state of the pilot's eyes.

  In a sixth aspect based on the second aspect, the face information generating means detects whether or not tears are spilled from the pilot's eyes using the image picked up by the image pickup means. The state of the person's tears is generated as face information.

  In a seventh aspect based on the sixth aspect, the image of the driver's eyes or corners is extracted from the image captured by the imaging means, and the luminance is greater than or equal to a predetermined value for the eyes or corners and surrounding images. When there is an area with a difference, it is detected that tears are spilled from the pilot's eyes.

  According to an eighth aspect of the present invention, in the first or second aspect of the present invention, based on the face information generated by the face information generating means, it is estimated whether or not a symptom causing an allergic reaction has occurred.

  In a ninth aspect based on the first or second aspect, the face image generation means determines in advance whether the eyes of the driver are in an open state or in a closed state using an image captured by the imaging means. The number of blinks that the driver's eyes open and close per unit time is generated as the face information. Then, when the number of blinks is greater than a predetermined number, the estimation means estimates that one of a symptom causing an allergic reaction and a symptom indicating a fatigue state has occurred in the pilot.

  In a tenth aspect based on the ninth aspect, the face information generating means detects an eye fullness of the driver's eyes using the image picked up by the image pick-up means, and uses the eye fullness of the pilot as face information. Generate more. Then, the estimating means estimates that a symptom causing an allergic reaction has occurred when the number of blinks is greater than a predetermined number and the eyes of the driver are in a congested state.

  In an eleventh aspect based on the ninth aspect, the face information generating means detects whether or not tears are spilled from the pilot's eyes using the image picked up by the image pickup means. A tear state is further generated as face information. Then, the estimating means estimates that there is a symptom causing an allergic reaction when the number of blinks is greater than a predetermined number and tears are spilled from the eyes of the pilot. .

  In a twelfth aspect based on the ninth aspect, the face information generation means detects an eye fullness state of the pilot's eyes using the image captured by the image pickup means, and uses the blood pressure state of the pilot's eyes as face information. Generate more. Then, the estimating means estimates that a symptom indicating a fatigue state is occurring in the driver when the number of blinks is greater than a predetermined number and the eyes of the driver are not congested.

  In a thirteenth aspect based on the ninth aspect, the face information generation means detects whether or not tears have spilled from the pilot's eyes using the image captured by the imaging means. A tear state is further generated as face information. Then, the estimation means estimates that a symptom indicating a fatigue state has occurred in the driver when the number of blinks is greater than a predetermined number and tears are not spilled from the eyes of the driver.

  In a fourteenth aspect based on the first or second aspect, the estimating means estimates whether or not a symptom indicating a fatigue state has occurred in the pilot based on the face information generated by the face information generating means. .

  In a fifteenth aspect based on the fourteenth aspect, the face image generation means detects whether or not the driver's eyes are closed using the image captured by the imaging means. Calculates an eye closure time when the eye is closed, and generates the eye closure time as face information. Then, the estimating means estimates that a symptom indicating a fatigue state has occurred in the operator when the closed eye state in which the closed eye time is equal to or longer than a predetermined time occurs at intervals equal to or less than a predetermined cycle. .

  In a sixteenth aspect based on the fourteenth aspect, the face information generation means determines whether the operator's eyes are in an open state or a closed eye state using an image captured by the imaging means. And the number of blinks that the driver's eyes open and close per unit time is generated as face information. Then, when the number of blinks is less than a predetermined value, the estimation means estimates that a symptom indicating a fatigue state is occurring in the operator.

  In a seventeenth aspect based on the first or second aspect, the driver is a driver seated in the driver's seat of the vehicle, and the symptom estimation device is installed in the vehicle. The apparatus further includes control means for controlling equipment for controlling the in-vehicle environment of the vehicle or equipment for notifying the driver of information according to the driver's symptoms estimated by the estimation means.

  In an eighteenth aspect based on the seventeenth aspect, the estimating means estimates whether or not a symptom causing an allergic reaction has occurred based on the face information generated by the face information generating means. And a control means starts the air cleaner provided in the vehicle, when an estimation means estimates that the symptom which has caused the allergic reaction has arisen in the driver | operator.

  In a nineteenth aspect based on the seventeenth aspect, the estimating means estimates whether or not a symptom causing an allergic reaction has occurred on the basis of the face information generated by the face information generating means. Then, when the estimation means estimates that the symptom causing an allergic reaction has occurred in the driver, the control means controls the power window mechanism provided in the vehicle to close the window of the vehicle.

  According to a twentieth aspect, in the seventeenth aspect, based on the face information generated by the face information generating means, it is estimated whether or not a symptom indicating a fatigue state has occurred in the driver. And a control means adjusts the preset temperature of the air conditioner provided in the vehicle, when an estimation means estimates that the symptom which shows a fatigue state has arisen in the driver | operator.

  In a twenty-first aspect based on the seventeenth aspect, the estimating means estimates whether or not a symptom indicating a fatigue state has occurred in the driver based on the face information generated by the face information generating means. The control means controls the car navigation system provided in the vehicle and guides the driver to the nearest resting facility when the estimation means estimates that a symptom indicating a fatigue condition has occurred in the driver. .

  According to the first aspect of the present invention, it is possible to determine whether the symptom that can be estimated from the state of the face of the driver is occurring in the driver, and it is possible to prevent an accident using the determination result. . For example, if the pilot feels tired, sleepy, or uncomfortable, the pilot may fall into a dangerous piloting state one step ahead of which is expected to result in a dangerous piloting state if left unattended. Physiological symptoms that give a sense of incongruity to the driver can be estimated during the maneuvering, and measures can be taken before the physiological symptoms that give the operator a sense of incongruity.

  According to the second aspect, the state of the driver is detected using the state of the driver's eyes, and it is estimated whether the symptom is occurring in the driver using the state of the eyes. Therefore, since the symptom estimation is performed by focusing on limited face parts, the image processing and the symptom estimation process are simplified.

  According to the third aspect, the symptom estimation process according to the number of blinks of the driver is possible.

  According to the fourth aspect of the invention, the symptom estimation process according to the hyperemia state of the driver's eyes can be performed.

  According to the fifth aspect of the present invention, it is possible to easily determine the hyperemia state of the operator's eyes using the luminance change of the white eye portion of the captured image.

  According to the sixth aspect, the symptom estimation process according to the state of the tears of the driver is possible.

  According to the seventh aspect, it is possible to easily determine the tear state of the driver's eyes using the brightness difference between the eyes or corners of the driver's eyes and the periphery thereof.

  According to the eighth aspect of the present invention, it is possible to determine whether or not an allergic symptom that can be estimated from face information indicating the state of the eyes of the driver, the state of sweat, the state of the face color, or the like has occurred in the driver. It is possible to prevent accidents using the judgment result.

  According to the ninth aspect, it is possible to easily determine the allergic symptom and fatigue state of the pilot based on the number of blinks of the pilot. For example, when the operator's eye mucous membrane reacts excessively with dust or plant pollen in the car and the number of blinks increases, or when the number of blinks increases due to eye fatigue, etc. Therefore, it is possible to easily perform the symptom estimation process for a symptom in which the symptom of the pilot appears in the number of blinks.

  According to the tenth and twelfth aspects of the invention, it is possible to easily distinguish between allergic symptoms and fatigue states based on the presence or absence of the driver's hyperemia.

  According to the eleventh and thirteenth inventions, it is possible to easily distinguish between allergic symptoms and fatigue states according to the state of tears of the pilot.

  According to the fourteenth aspect of the present invention, it is possible to determine whether the driver is in a fatigue state that can be estimated from face information indicating the driver's eye state, sweat state, facial color state, and the like during the operation. Therefore, it is possible to prevent accidents using the judgment result.

  According to the fifteenth aspect of the present invention, the operator's fatigue state can be easily determined based on the driver's eye-closing time and its period. For example, when the state where the upper eyelid is lowered and the thin state is continued, it can be estimated that the symptom indicating the fatigue state is generated in the operator.

  According to the sixteenth aspect, the driver's fatigue state can be easily determined based on the number of blinks of the driver. For example, when the upper eyelid of the pilot is lowered and the light eye is about to fall, and the number of blinks is reduced, it can be estimated that the pilot has a symptom indicating a fatigue state. .

  According to the seventeenth aspect, it is possible to determine whether a symptom that can be estimated from the state of the driver's face has occurred in the driver, while driving the vehicle, and vehicle accident prevention using the determination result is possible. It becomes. For example, if the driver feels tired or sleepy or feels uncomfortable, the driver may fall into a dangerous driving state one step ahead that is expected to result in a dangerous driving state if left unattended. Physiological symptoms that give a sense of incongruity to the driver can be estimated while driving the vehicle, and measures before the physiological symptoms that give the driver a sense of incongruity can be implemented.

  According to the eighteenth aspect of the invention, when it is estimated that the driver is experiencing symptoms that cause an allergic reaction, the air cleaner is operated, so that pollen and dust in the vehicle can be removed, and allergic symptoms are observed. Can be reduced.

  According to the nineteenth invention, when it is estimated that the driver is suffering from an allergic reaction, the vehicle window is closed, so that pollen and dust can be prevented from entering the vehicle. Can alleviate allergic symptoms.

  According to the twentieth aspect of the invention, when the driver is estimated to have a symptom indicating a fatigue state, the set temperature of the air conditioner provided in the vehicle is adjusted. A refreshing feeling can be obtained by the breeze coming out, and fatigue can be reduced.

  According to the twenty-first aspect, when it is estimated that the driver has a symptom indicating a fatigue state, the driver is routed to the nearest resting facility, so the driver takes rest at the nearest resting facility. Can reduce fatigue.

  Hereinafter, the configuration and operation of a symptom estimation apparatus according to an embodiment of the present invention will be described with reference to the drawings. In the present embodiment, assuming that the symptom estimation device is installed in a vehicle, it is estimated whether or not the driver driving the vehicle has a symptom that may be a previous stage leading to a dangerous driving state. To do.

  FIG. 1 is a block diagram illustrating an example of a configuration of a symptom estimation apparatus according to an embodiment of the present invention. In FIG. 1, the symptom estimation apparatus includes a camera 100, a driver monitor ECU (Electronic Control Unit) 110, and a preventive safety ECU (Electronic Control Unit) 120.

  The camera 100 is, for example, a CCD camera, an infrared camera, or the like that images a driver's face. The camera 100 may have a zoom function. For example, the camera 100 implements a zoom function by incorporating a liquid lens. In general, a liquid lens can change the refractive index of light with the voltage of the current that flows, and is effective in space saving because it does not require a motor, glass, etc., which mechanical autofocus has. Considering the use environment of the vehicle such as the service life, it is preferable. The camera 100 corresponds to an example of the imaging unit of the present invention.

  The camera 100 is installed at a position where the face of the driver seated in the driver's seat can be imaged. As an example of the installation location of the camera 100, the inside of the meter hood of the driver's seat, the inside of the meter panel, and the steering column can be cited. In addition, the camera 100 captures the driver's face at predetermined time intervals, for example. The image captured by the camera 100 is output to the driver monitor ECU 110.

  The driver monitor ECU 110 includes a main microcomputer (hereinafter referred to as a main microcomputer) 111, a sub microcomputer (hereinafter referred to as a sub microcomputer) 112, a face image pattern storage unit 113, and the like. The driver monitor ECU 110 is connected to the camera 100 that captures the driver's face image and a preventive safety ECU 120 described later. The driver monitor ECU 110 corresponds to an example of face information generating means of the present invention.

  The main microcomputer 111 includes a CPU (Central Processing Unit) and a memory. Although details will be described later, the main microcomputer 111 mainly detects the face orientation of the driver from the image captured by the camera 100.

  The sub-microcomputer 112 includes a CPU and a memory. Although details will be described later, the sub-microcomputer 112 determines the driver's eyes such as the number of blinks of the driver's eyes, closed eye time, closed eye time, presence / absence of eye redness, and presence / absence of eye tears from an image captured by the camera 100. State is detected, face information indicating the state is generated, and output to the preventive safety ECU 120.

  The face image pattern storage unit 113 is composed of a readable / writable storage medium, and stores information indicating a person's face image pattern. Specifically, the face image pattern includes various patterns such as a face size, an eye position and size, a mouth position and size, and a nose position and size. The face image pattern includes template images (for example, mouth template images) of various human face parts. Note that the face information stored in the face image pattern storage unit 113 may be appropriately updated by, for example, the rewriting operation of the user of the vehicle.

  The preventive safety ECU 120 uses the face information about the driver's eyes generated by the sub-microcomputer 112 to estimate whether or not there is a symptom that may be a previous stage that leads to a dangerous driving state for the driver. In other words, the preventive safety ECU 120 not only feels tired of the driver, such as “feeling sleepy”, “looking aside”, “sleeping”, but also “feeling tired”. It is estimated whether or not the driver has symptoms that are expected to lead to an accident if left unattended, such as “I have an allergic reaction” (with pollen, etc.). In addition, the preventive safety ECU 120 acquires vehicle information (for example, vehicle shift lever position information) of the vehicle from each device provided in the vehicle.

  Further, the preventive safety ECU 120 performs preventive safety by controlling the operation of various devices installed in the vehicle based on the estimated driver's symptoms. Although details will be described later, the control performed by the preventive safety ECU 120 for each of the above devices includes danger avoidance control and danger driving prevention control.

  The danger avoidance control is a control to be performed on each of the above devices when the preventive safety ECU 120 estimates that a dangerous driving state that immediately leads to an accident such as “the driver is driving aside”. is there. Specifically, the preventive safety ECU 120 urges the driver to be alerted by turning on a warning light provided in the vehicle or sounding an alarm buzzer. Typically, the preventive safety ECU 120 estimates the driving state of the driver based on the face orientation detection result output from the driver monitor ECU 110 (main microcomputer 111), and performs the risk avoidance control.

  On the other hand, the above dangerous driving prevention control means that the driver feels tired or that the driver is experiencing symptoms that are causing an allergic reaction (due to pollen, etc.). This control is performed on each of the above devices when the preventive safety ECU 120 estimates that a symptom that is expected to result in such a dangerous driving state occurs in the driver. As a first example of dangerous driving prevention control, the preventive safety ECU 120 controls an air conditioner (hereinafter referred to as an air conditioner) 131 to adjust the temperature in the vehicle according to the symptoms of the driver. As a second example of dangerous driving prevention control, the preventive safety ECU 120 controls the air cleaner 132 to clean the air in the vehicle. As a third example of dangerous driving prevention control, the preventive safety ECU 120 controls the power window mechanism 133 of the vehicle to close the window and prevent outside air from entering from the outside of the vehicle. As a fourth example of dangerous driving preventive control, the preventive safety ECU 120 controls the car navigation system 134 to give the driver rest guidance to the nearest rest facility, or controls the speaker 135 to issue an alarm. . In addition, the various apparatuses installed in the vehicle for performing the dangerous driving prevention control are not limited to the above-described apparatuses. The preventive safety ECU 120 corresponds to an example of an estimation unit and an example of a control unit of the present invention.

  As described above, the driver monitor ECU 110 generates information related to the driver's face from the image captured by the camera 100. Then, the preventive safety ECU 120 uses the information regarding the driver's face generated by the driver monitor ECU 110 to estimate whether or not the driver has a symptom that may be a previous stage leading to a dangerous driving state. Further, the preventive safety ECU 120 performs dangerous driving preventive control on various devices installed in the vehicle based on the estimation result. Hereinafter, an example of processing performed by each unit of the symptom estimation apparatus according to the present embodiment will be described with reference to FIGS.

  FIG. 2 is a flowchart showing an example of the flow of processing performed in the symptom estimation apparatus according to the present embodiment. 2 is performed by the main microcomputer 111, the sub-microcomputer 112, and the preventive safety ECU 120 provided in the driver monitor ECU 110 executing predetermined programs. 2 is started when the power of the symptom estimation apparatus is turned on (for example, the ignition switch of a vehicle on which the symptom estimation apparatus is mounted is turned on). In addition, the processing ends when the symptom estimation apparatus is powered off (for example, an ignition switch is turned off). The ignition switch is hereinafter abbreviated as IG.

  In step S201 of FIG. 2, the main microcomputer 111 acquires an image captured by the camera 100.

  Next, in step S202, the main microcomputer 111 calculates the face orientation of the driver. Hereinafter, the operation for calculating the driver's face orientation will be described with reference to FIG. FIG. 3 is a flowchart showing an example of the operation for calculating the driver's face orientation in step S202 of FIG.

  In step S301 in FIG. 3, the main microcomputer 111 performs a Sobel filter process in a predetermined direction (for example, the vertical direction) of the captured image acquired in step S201. Then, the main microcomputer 111 performs processing for extracting vertical edge points from the Sobel image using the luminance difference in the processed Sobel image, and proceeds to the next Step S302.

  In step S302, the main microcomputer 111 performs a process of calculating the width of the driver's face in the sobel image using the vertical edge points extracted from the sobel image in step S301. Specifically, the main microcomputer 111 generates a histogram (vertical edge histogram) for the extracted vertical edge points, and extracts the right edge and the left edge of the driver's face by detecting long contours at the left and right ends of the face. To do. Then, the main microcomputer 111 calculates the length obtained by subtracting the position of the left face of the face from the extracted position of the right face of the face as the face width of the driver. Also, the main microcomputer 111 outputs the calculated driver face width information to the sub-microcomputer 112. Then, the main microcomputer 111 proceeds to the next step S303.

  In step S303, the main microcomputer 111 determines whether or not the face width can be calculated in step S302. Then, if the face width can be calculated, the main microcomputer 111 proceeds to the next step S304. On the other hand, if the vertical edge point cannot be extracted from the captured image, or if the driver's face right end and face left end cannot be extracted, the main microcomputer 111 returns to step S201 and repeats the process. When the driver's face is not in the imaging area of the camera 100, that is, when the entire driver's face is not captured in the captured image, the main microcomputer 111 controls the imaging magnification of the camera 100 to zoom in or zoom out. Then, the driver may be imaged. Further, when the driver is facing sideways, that is, when the driver is looking aside, the face width of the driver may not be calculated. In this case, the main microcomputer 111 may instruct the preventive safety ECU 120 to perform the danger avoidance control.

  In step S304, the main microcomputer 111 estimates the positions (face part positions) of the driver's eyes, nose, and mouth in the captured image using the face width calculated in step S303. As an example of a specific method, the main microcomputer 111 performs Sobel filter processing in a predetermined direction (for example, the horizontal direction) of the captured image. Then, the main microcomputer 111 creates an image (monochrome white edge image) from which facial parts are extracted using a horizontal Sobel image, estimates the position of the facial parts (eyes, nose, mouth), and Verify the symmetry. Then, the main microcomputer 111 advances the process to step S305.

  In step S305, the main microcomputer 111 obtains the face centerline for the face image based on the estimated eye, nose, and mouth positions. In step S306, the main microcomputer 111 determines whether or not the center line of the face has been obtained. When the main microcomputer 111 finds the center line of the face, the main microcomputer 111 proceeds to the next step S307. On the other hand, when the center line of the face is not obtained, the main microcomputer 111 returns to step S201 and repeats the process.

  In step S307, the main microcomputer 111 calculates the left / right ratio of the face using the left end of the face, the right end of the face and the center line of the face detected in the above step, and converts the left / right ratio into an angle to convert the driver's face. Calculate the orientation angle. Then, the main microcomputer 111 ends the process according to the flowchart.

  Returning to FIG. 2, after the face orientation calculation process in step S202, in step S203, the main microcomputer 111 determines whether or not the driver's face orientation is front. If the driver's face is front, the process proceeds to step S205. On the other hand, if the driver's face is not in front, the process proceeds to step S204. An example of the case where the driver's face is not in front is when the driver is looking aside or looking away.

  As described above, the main microcomputer 111 can determine whether the driver is looking aside or looking away from the image captured by the camera 100 through the processing in steps S201 to S203. When the driver's face width is calculated in step S303, the main microcomputer 111 may store imaging condition information such as the current zoom ratio of the camera 100 in a memory. If the camera 100 captures images according to the imaging condition information, a captured image suitable for the processing can be acquired immediately in a situation where the processing is resumed.

  In step S204, the main microcomputer 111 instructs the preventive safety ECU 120 to execute the danger avoidance control. Then, the main microcomputer 111 proceeds to the next step S205.

  In step S205, the sub-microcomputer 112 calculates the number of blinks of the driver's eyes. Hereinafter, a method for calculating the number of blinks performed by the sub-microcomputer 112 will be described with reference to FIGS. 4 and 5. FIG. 4 is a flowchart of the first half showing an example of the operation for calculating the number of blinks in step S205 of FIG. FIG. 5 is a second half flowchart illustrating an example of the operation for calculating the number of blinks in step S205 of FIG.

  In step S401 of FIG. 4, the sub-microcomputer 112 searches for the driver's nostril from the captured image acquired in step S201. For example, when the sub-microcomputer 112 uses the data indicating the face width acquired from the main microcomputer 111 and the face image pattern stored in the face image pattern storage unit 113, the nostril is present in the captured image. An expected range, that is, a nostril search range is set. Then, the sub-microcomputer 112 detects pixels with relatively low luminance from the captured image within the set nostril search range, and calculates the circularity of the low-luminance pixel group. Then, the sub-microcomputer 112 sets the point group having the highest circularity (close to a circle) as the driver's nostril, and proceeds to step S402.

  In step S402, the sub-microcomputer 112 determines whether or not the nostril has been found. If the sub-microcomputer 112 can find the nostril, it proceeds to the next step S405. On the other hand, if the sub-microcomputer 112 cannot find the nostril, it proceeds to step S403.

  In step S403, the sub-microcomputer 112 searches for the driver's mouth from the captured image acquired in step S201, and proceeds to the next step S404. For example, if the sub-microcomputer 112 uses the data indicating the face width acquired from the main microcomputer 111 and the face image pattern stored in the face image pattern storage unit 113, the mouth is present in the captured image. An expected range, that is, a mouth search range is set. Next, the sub-microcomputer 112 performs image processing for extracting an edge line for the captured image within the set mouth search range. Then, the sub-microcomputer 112 uses the mouth template image stored in the face image pattern storage unit 113 to perform pattern matching processing on the image from which the edge line has been extracted to calculate the degree of coincidence. In step S403, pattern recognition may be performed using a classifier such as a neural network or SVM (support vector machine), and the pattern matching process described later is the same.

  In step S404, the sub-microcomputer 112 determines whether or not the mouth has been searched. If the sub-microcomputer 112 can search for a mouth (for example, if the degree of coincidence with the template image of the mouth is equal to or greater than a predetermined threshold), the process proceeds to step S405. On the other hand, if the sub-microcomputer 112 cannot find the mouth, it returns to step S201 and repeats the process. When returning to step S201 and executing the processing, the main microcomputer 111 may zoom in or out of the camera 100 to image the driver.

  In step S405, the sub-microcomputer 112 identifies the driver's eyes from the captured image acquired in step S201, and proceeds to step S406. For example, the position of the eye in the captured image is estimated based on the position of the driver's nostril or mouth detected in step S401 or step S403. Then, the sub-microcomputer 112 performs image processing for extracting an edge line on the captured image at the estimated eye position. Further, the sub-microcomputer 112 uses the eye template image stored in the face image pattern storage unit 113 to perform a pattern matching process on the image from which the edge line has been extracted to calculate the degree of coincidence.

  In step S406, the sub-microcomputer 112 determines whether eyes have been identified. Then, if the sub-microcomputer 112 has identified the eye (for example, the degree of coincidence with the template image of the eye is equal to or greater than a predetermined threshold value), the sub-microcomputer 112 proceeds to the next step S407. On the other hand, if the eye cannot be identified, the sub-microcomputer 112 returns to step S201 and repeats the process. When returning to step S201 and executing the processing, the main microcomputer 111 may zoom in or out of the camera 100 to image the driver. For example, when the driver wears sunglasses or the like, the eye edge line cannot be extracted from the captured image, and thus the eye cannot be specified. In such a case, the sub-microcomputer 112 generates a sound indicating that the symptom estimation process is impossible, such as “please remove sunglasses” or “cannot process” via a speaker or the like provided on the vehicle. You can let the driver be alerted.

  In step S407, the sub-microcomputer 112 detects a black eye portion in the driver's eyes, and the process proceeds to step S408. In general, the shape of the black eye part (iris and pupil) is often not elliptical on the captured image because it is obstructed by the face direction or eyelids, but rather is elliptical. Here, in step S405, the sub-microcomputer 112 specifies the eye in the captured image and specifies the position of the eye. In step S407, a plurality of elliptic filters having different sizes with respect to the specified eye position are used. Then, an elliptic filter is applied to the identified eye region, and luminances for the outer region and the inner region of the elliptic filter are obtained. At this time, if there is a difference in luminance between the luminance of the outer region and the luminance of the inner region, the inner range of the elliptic filter is set to black eyes. The method for detecting the black eye is not limited to the above-described example, and other processing may be used as long as the image processing can recognize the black eye portion and the white eye portion of the eye.

  Proceeding to FIG. 5, in step S <b> 408, the sub-microcomputer 112 determines whether or not a black eye has been detected. If the sub-microcomputer 112 detects a black eye, the sub-microcomputer 112 proceeds to step S416. On the other hand, if the sub-microcomputer 112 cannot detect the black eye, the process proceeds to step S409.

  In step S409, the sub-microcomputer 112 determines whether or not the eye opening flag is ON. Here, as will be apparent later, the eye opening flag is set to ON when the driver's eyes are open, and is set to OFF when the driver's eyes are closed, and stored in the memory. . If the eye opening flag is ON, the sub-microcomputer 112 advances the process to step S410. On the other hand, when the eye opening flag is OFF, the sub-microcomputer 112 advances the process to step S412.

  In step S410, the sub-microcomputer 112 detects blinks, calculates the number of blinks, and proceeds to step S411. Here, the number of blinks indicates how many times the cycle is performed per unit time (for example, 1 second), from the time when the driver's eyes are opened from the closed eyes to the closed eyes.

  In step S411, the sub-microcomputer 112 sets the eye opening flag to OFF, updates the eye opening flag in the memory, and advances the process to step S412.

  In step S412, the sub-microcomputer 112 performs a process of adding the eye closing time, and the process proceeds to step S413. Here, the eye closing time is the time from when the driver closes the eye until the driver opens the eye, and the eye closing time up to the present time is stored in the memory. In step S412, a new eye closing time is calculated by adding the processing period to the eye closing time stored at the current time, and the stored eye closing time is updated using the new eye closing time. As will become clear later, the eye-closing time stored in the memory is initialized to 0 when the driver is open.

  In step S413, the sub-microcomputer 112 determines whether or not the stored eye closure time is equal to or longer than the time T1. That is, in the above step S413, whether or not the time when the driver is in the state of closing the eyes continues to the length of time T1 or more, that is, whether or not the eye blinks for the duration of time T1 or more. Judging. Then, if the closed eye time is equal to or longer than time T1, the sub-microcomputer 112 advances the process to step S414. On the other hand, if the closed eye time is less than the time T1, the sub-microcomputer 112 ends the process according to the flowchart.

  In step S414, the sub-microcomputer 112 determines whether or not the closed eye time for the current closed eye state has been recorded. If the closing time for the current closed eye state is not recorded, the sub-microcomputer 112 describes the closed eye time for the current closed eye state in time series in the memory using the current time (step S415), and ends the processing according to the flowchart. To do. On the other hand, if the closed eye time for the current closed eye state has already been recorded, the sub-microcomputer 112 ends the process according to the flowchart.

  If the black eye can be detected in step S408, the sub-microcomputer 112 sets the eye opening flag to ON in step S416 and updates the eye opening flag in the memory. Next, in step S417, the sub-microcomputer 112 initializes the eye closing time recorded in the memory to 0, and ends the processing according to the flowchart.

  As described above, the sub-microcomputer 112 not only can calculate the number of blinks of the driver's eyes by the processing of step S401 to step S417, but also when the closed eye state of the driver's eyes continues for a length of time T1 or more. Can be recorded.

  Returning to FIG. 2, after the process of calculating the number of blinks in step S205, in step S206, the sub-microcomputer 112 determines whether or not the driver's eyes are congested. Hereinafter, with reference to FIG. 6, an example of an operation for determining the presence or absence of eye redness will be described. FIG. 6 is a flowchart illustrating an example of processing for determining whether or not the eyes are red in step S206 illustrated in FIG.

  In step S501 in FIG. 6, the sub-microcomputer 112 detects a white-eye portion on the captured image acquired in step S201, and proceeds to the next step S502. For example, the sub-microcomputer 112 detects the luminance of the captured image in a certain area around the black eye part detected in step S407. Then, when the luminance corresponding to the luminance of the white eye portion is obtained in the certain region around the black eye, the sub-microcomputer 112 identifies the certain region as the white eye portion. Note that the processing for specifying the white-eye portion in step S501 is not limited to the above-described example, and other image processing may be used as long as the image processing can specify the white-eye portion.

  In step S502, the sub-microcomputer 112 extracts the edge point from the white eye portion by performing a Sobel filter process or the like in a predetermined direction (for example, the vertical direction) for the white eye portion specified in step S501. Then, the sub-microcomputer 112 proceeds to the next step S503.

  In step S503, the sub-microcomputer 112 performs line segment detection for the edge points extracted in step S502. As an example, the sub-microcomputer 112 performs a process based on a curve Hough transform on an edge point that changes from high luminance to low luminance among the edge points extracted in step S502. Thereby, a line segment passing through the edge point can be detected.

  In step S504, the sub-microcomputer 112 determines whether or not the total length of the line segments detected in step S503 is equal to or greater than a threshold value L. Then, if the total length of the line segments is equal to or greater than the threshold value L, the sub-microcomputer 112 advances the process to step S505. On the other hand, if the total length of the line segments is less than the threshold value L, the sub-microcomputer 112 advances the process to step S506.

  In step S505, the sub-microcomputer 112 determines that the driver's eyes are congested, outputs the determination result to the preventive safety ECU 120, and ends the processing according to the flowchart. On the other hand, in step S506, the sub-microcomputer 112 determines that the driver's eyes are not congested, outputs the determination result to the preventive safety ECU 120, and ends the processing according to the flowchart. The sub-microcomputer 112 may store the determination result made in step S505 or step S506 in a memory.

  As described above, the sub-microcomputer 112 can determine whether or not the driver's eyes are congested by the processing in steps S501 to S506.

  Returning to FIG. 2, after determining the presence or absence of hyperemia in step S206, in step S207, the sub-microcomputer 112 determines the presence or absence of tears in the eyes of the driver. Hereinafter, an example of processing for determining the presence or absence of tears in the eyes will be described with reference to FIG. FIG. 7 is a flowchart illustrating an example of processing for determining the presence or absence of eye tears in step S207 illustrated in FIG. Here, the tear determined in the process shown in FIG. 7 is a secreted fluid that is secreted in a large amount from the eyes of the driver due to stimulation of the cornea, conjunctiva, nasal mucosa, and the like of the driver's eyes.

  In step S601 of FIG. 7, the sub-microcomputer 112 estimates that the left and right ends of the white eye portion are the positions of the eyes or corners of the white eye portion detected in step S501, and the process proceeds to the next step S602. To proceed. At this time, a template image of the eye may be acquired from the face image pattern storage unit 113, and the position of the top of the eye or the corner of the eye may be estimated from the region of the white eye using the template image.

  In step S602, the sub-microcomputer 112 determines whether or not there is a region having a luminance difference equal to or larger than the threshold T around the position of the eye or the corner of the eye estimated in step S601. Then, if there is a region having a luminance difference equal to or greater than the threshold value T, the sub-microcomputer 112 proceeds to the next step S603. On the other hand, if there is no region having a luminance difference equal to or greater than the threshold T, the sub-microcomputer 112 proceeds to the next step S604. For example, in step S602, the luminance in the peripheral area (for example, an area within a pixel within a distance from the position of the corner of the eye or the eye) is detected based on the position of the eye corner or the eye estimated in step S602. . Then, in the peripheral region, it is determined whether or not there is a region whose luminance is higher than the threshold value T by the threshold T and whose area is B pixels or more. This is because the tears secreted from the eyes of the driver due to stimulation to the cornea, conjunctiva, nasal mucosa, etc. of the driver's eyes adhere to the skin around the eyes and the corners of the eyes, and the brightness of the adhesion area compared to the surrounding skin. The presence or absence of tears is determined by taking advantage of the increase in

  In step S603, the sub-microcomputer 112 determines that the driver's tears have been detected, outputs the determination result to the preventive safety ECU 120, and ends the process according to the flowchart. On the other hand, in step S604, the sub-microcomputer 112 determines that no driver's tears have been detected, outputs the determination result to the preventive safety ECU 120, and ends the process according to the flowchart. Note that the determination result made in step S603 or step S604 may be stored in a memory.

  As described above, the sub-microcomputer 112 can determine the presence or absence of the driver's tears by the processing from step S601 to step S604.

  Returning to FIG. 2, after the process of determining the presence or absence of tears in step S <b> 207, in step S <b> 208, the preventive safety ECU 120 estimates whether or not a symptom that may lead to a dangerous driving state has occurred in the driver. . Specifically, as described above, the sub-microcomputer 112 acquires the state relating to the driver's eyes (number of blinks of the driver's eyes, eye closure time, presence / absence of eye redness, presence / absence of tears, etc.). Using information indicating these states, the preventive safety ECU 120 estimates whether or not the above-described symptoms occur in the driver. Hereinafter, an example of processing for estimating the occurrence of the above symptoms will be described with reference to FIG. FIG. 8 is a flowchart illustrating an example of processing for estimating whether or not the above-described symptom is occurring in the driver.

  In step S701 in FIG. 8, the preventive safety ECU 120 acquires data indicating the number of blinks of the driver's eyes from the sub-microcomputer 112, and proceeds to the next step S702. When the data indicating the number of blinks of eyes is stored in the memory provided in the sub-microcomputer 112, the preventive safety ECU 120 acquires the data indicating the number of blinks of eyes from the memory.

  In step S702, the preventive safety ECU 120 determines whether or not the acquired number of eye blinks is equal to or greater than the threshold value M1. Then, when the number of blinks is equal to or greater than the threshold value M1, the preventive safety ECU 120 advances the process to step S706. On the other hand, when the number of blinks is less than the threshold value M1, the preventive safety ECU 120 advances the process to step S703. Here, the threshold value M1 is set to be equal to or higher than the upper limit of a general value of the number of so-called periodic blinks that humans usually perform unconsciously. Generally, when the eye mucosa reacts excessively (allergic reaction) with dust in the vehicle or pollen of a plant, the number of blinks increases accordingly. Also, when the driver's eyes become tired, the number of blinks increases. That is, when the number of blinks exceeds the threshold value M1, it is estimated that the driver has allergic disease, eye fatigue, or the like. The threshold value M1 may be determined by a specific numerical value based on an evaluation experiment (for example, stratification of the number of blinks depending on various types of allergic patients or eye fatigue).

  In step S703, the preventive safety ECU 120 determines whether or not there is one or more blinks within the time T2 that the eye closure time is longer than the time T1. Then, the preventive safety ECU 120 advances the process to step S704 when the blink has occurred once or more. On the other hand, when there is no blink, the preventive safety ECU 120 advances the process to step S705.

  Here, a first example of a specific processing method in step S703 will be described. The preventive safety ECU 120 refers to the closed eye time recorded in step S415 by the sub-microcomputer 112. Then, the preventive safety ECU 120 calculates the time from the latest closed eye time recorded in step S415 to the current time for executing the process of step S703, and determines whether or not the calculated time is equal to or less than time T2. Judgment (judgment condition A). Further, the preventive safety ECU 120 determines whether or not the time difference between the latest eye closing time recorded in step S415 and the eye closing time recorded immediately before the latest eye closing time is equal to or less than the time T2 (determination condition). B). Then, the preventive safety ECU 120 determines that the blink has occurred once or more when the recorded closed eye time satisfies both the determination condition A and the determination condition B. For example, as an example of the cause of the blink, there is a state in which the upper eyelid is lowered and a light state is about to be lost, that is, a fatigue state. The time T2 may be determined by a specific numerical value by an evaluation experiment (for example, the wakefulness level is evaluated from the subject's brain wave or heart rate, and the number of blinks and the wakefulness level are associated at that time). .

  Also, a second example of the specific processing method in step S703 will be described. In step S703, the preventive safety ECU 120 determines whether or not the number of blinks acquired in step S701 is equal to or less than the threshold value M2. Then, when the number of blinks is equal to or less than the threshold value M2, the preventive safety ECU 120 advances the process to step S704. On the other hand, when the number of blinks is greater than the threshold value M2, the preventive safety ECU 120 advances the process to step S705. Here, the threshold value M2 is set to be equal to or less than the lower limit of a general value of the number of so-called periodic blinks, which is usually performed unconsciously by humans. Further, the threshold value M2 may be set to a specific numerical value capable of distinguishing the presence or absence of the fatigue state by the evaluation experiment described above. For example, when the driver is in the above-mentioned fatigue state in which the upper eyelid is lowered and is thin, the number of blinks of the driver may be reduced. That is, the fatigue state of the driver described above can also be determined by the determination according to the second example. In addition, when performing said step S703 in the said 2nd example, it does not need to perform the process of said step S412-step S415.

  In step S704, the preventive safety ECU 120 estimates that a symptom indicating the fatigue state of the whole body or eyes has occurred in the driver, and ends the process according to the flowchart. Since the driver may feel sleepy due to fatigue, the preventive safety ECU 120 may estimate that the driver feels sleepy in a broad sense. Then, the estimation result in step S704 is stored in a memory provided in the preventive safety ECU 120. Although details will be described later, the preventive safety ECU 120 performs the dangerous driving preventive control based on the estimation result. On the other hand, in step S705, the preventive safety ECU 120 estimates that the driver's state is normal, and ends the process according to the flowchart.

  In the process of step S704, the preventive safety ECU 120 estimates that a symptom indicating a fatigue state has occurred in the driver or the driver feels drowsy. It is also possible to distinguish. For example, if the driver feels sleepy, the driver may intentionally blink in an attempt to awaken, which may occasionally increase the number of blinks in the driver's eyes. If it is further determined whether or not there is an intentional blink in the process of step S704, the driver's fatigue state and sleepiness can be distinguished and estimated.

  When it is determined in step S702 that the number of blinks is equal to or greater than the threshold value M1, in step S706, the preventive safety ECU 120 acquires information regarding the presence or absence of eye redness from the sub-microcomputer 112, and proceeds to the next step S707. . In addition, when the data which shows the information regarding the presence or absence of the redness of eyes is memorize | stored in the memory of the submicrocomputer 112, the preventive safety ECU120 acquires the data which shows the information regarding the presence or absence of the redness from the said memory.

  In step S707, the preventive safety ECU 120 determines whether or not the driver's eyes are congested. If the eyes are not congested, in the next step S708, the preventive safety ECU 120 estimates that the driver's eyes are fatigued, and ends the process according to the flowchart. Note that the estimation result in step S708 is stored in a memory provided in the preventive safety ECU 120. Although details will be described later, the preventive safety ECU 120 performs the dangerous driving preventive control based on the estimation result. On the other hand, if the eyes are congested, the preventive safety ECU 120 proceeds to the next step S709. In step S708, the preventive safety ECU 120 estimates that the driver's eyes are fatigued, but may estimate that the whole body is in a fatigued state.

  In step S709, the preventive safety ECU 120 acquires information related to eye tears from the sub-microcomputer 112, and proceeds to the next step S710. In addition, when the data which shows the information regarding tears of eyes are memorize | stored in the memory of the submicrocomputer 112, the preventive safety ECU120 acquires the data which shows the information regarding tears from the said memory.

  In step S710, the preventive safety ECU 120 determines whether or not there are tears in the eyes of the driver, that is, whether or not tears have spilled from the eyes of the driver. When the tear is spilled from the eyes of the driver, the preventive safety ECU 120 estimates that a symptom causing an allergic reaction has occurred in the next step S711, and ends the processing according to the flowchart. . Note that the estimation result in step S711 is stored in a memory provided in the preventive safety ECU 120. Although details will be described later, the preventive safety ECU 120 performs the dangerous driving preventive control based on the estimation result. On the other hand, when the tear is not spilled from the driver's eyes, the preventive safety ECU 120 advances the process to step S708.

  Returning to FIG. 2, after the process of estimating whether or not the symptom is occurring in the driver in step S <b> 208, in step S <b> 209, the preventive safety ECU 120 determines whether or not dangerous driving preventive control is necessary based on the estimation result described above. To do. Then, the preventive safety ECU 120 advances the process to step S210 when the dangerous driving preventive control is necessary (that is, when some symptom occurs in the driver). On the other hand, the preventive safety ECU 120 advances the process to step S211 when the dangerous driving preventive control is unnecessary (that is, when the driver is normal).

  In step S210, the preventive safety ECU 120 performs dangerous driving preventive control for various devices installed in the vehicle. Hereinafter, an example of the dangerous driving prevention control process performed by the preventive safety ECU 120 will be described with reference to FIG. FIG. 9 is a flowchart showing an example of the dangerous driving prevention control process.

  In step S801 in FIG. 9, the preventive safety ECU 120 refers to the estimation result performed in step S208, and determines whether or not a symptom causing an allergic reaction has occurred in the driver. If the symptom causing an allergic reaction has occurred in the driver, the preventive safety ECU 120 advances the process to step S802. On the other hand, when the symptom causing the allergic reaction has not occurred in the driver, the preventive safety ECU 120 advances the process to step S803.

  In step S802, the preventive safety ECU 120 controls the operation of the power window mechanism 133 and the air purifier 132, and proceeds to step S803. For example, when a driver has symptoms that cause an allergic reaction, dust in the vehicle, plant pollen, or the like may react excessively on the mucous membrane of the driver's eyes. Therefore, in step S802, the preventive safety ECU 120 controls the operation of the power window mechanism 133 to close the window of the vehicle, thereby preventing dust and pollen from entering the vehicle from the window. In step S802, the preventive safety ECU 120 operates the air cleaner 132 to remove dust in the vehicle, plant pollen, and the like.

  In step S803, the preventive safety ECU 120 refers to the estimation result performed in step S208, and whether or not a symptom indicating a fatigue state has started to occur in the driver, and a time during which the symptom indicating the fatigue state continues to the driver is continued. Is reached at a predetermined time (for example, every 10 minutes). The preventive safety ECU 120 advances the process to step S804 when a symptom indicating a fatigue state starts to occur or when a time during which the symptom indicating the fatigue state continues reaches a predetermined time. On the other hand, the preventive safety ECU 120 performs the processing according to the flowchart when the symptom indicating the fatigue state does not occur in the driver or when the symptom indicating the fatigue state continues for the driver does not reach the predetermined time. finish.

  In step S804, the preventive safety ECU 120 controls the operation of the air conditioner 131 installed in the vehicle, and proceeds to step S805. For example, in the case where a symptom indicating a fatigue state occurs in the driver, if the environment in the vehicle is made more comfortable, the driver may feel refreshed and may recover from fatigue. Therefore, in step S804, the preventive safety ECU 120 adjusts the set temperature of the air conditioner 131 (for example, lowers the set temperature by a predetermined temperature) or adjusts the air volume from the air vent on the driver's seat side (for example, the air volume is decreased by one step). Or if the air conditioner 131 is OFF, it is controlled to be ON. In this way, the cool air coming out of the air blowing port of the air conditioner 131 can give the driver a refreshing feeling and reduce the driver's fatigue state.

  In step S805, the preventive safety ECU 120 acquires vehicle information from the vehicle and determines whether the vehicle is traveling. Here, the vehicle information acquired by the preventive safety ECU 120 includes vehicle shift lever position information. If the shift lever position of the vehicle is not in the neutral or parking position, it is determined that the vehicle is running. . Then, when the vehicle is traveling, the preventive safety ECU 120 advances the process to step S806. On the other hand, when the vehicle is not traveling, the preventive safety ECU 120 ends the process according to the flowchart.

  In step S806, the preventive safety ECU 120 controls the operation of the car navigation system 134 to start rest guidance, and ends the process according to the flowchart. For example, by the operation control of the preventive safety ECU 120 in step S806, the car navigation system 134 searches for a route to the nearest resting facility, starts voice guidance of the route to the resting facility, and prompts the driver to rest. .

  Returning to FIG. 2, in step S211, the driver monitor ECU 110 and the preventive safety ECU 120 determine whether or not the ignition switch of the vehicle is turned off. And driver monitor ECU110 and preventive safety ECU120 complete | finish the process by the said flowchart, when an ignition switch is turned off. On the other hand, when the ignition switch is not turned off, the driver monitor ECU 110 and the preventive safety ECU 120 return to step S201 and repeat the process.

  As described above, according to the symptom estimation device of the present embodiment, it is possible to estimate whether a symptom that may be a previous stage leading to a dangerous maneuvering state occurs in the driver. Furthermore, the symptom estimation device of the present embodiment controls each device provided in the vehicle according to the symptom occurring in the driver, and performs an appropriate treatment according to the symptom occurring in the driver. You can also.

  In the above-described embodiment, the example in which the processes of the main microcomputer 111, the sub-microcomputer 112, and the preventive safety ECU 120 are repeated as a series of processes that are sequentially performed in series has been described. An estimation process may be performed. For example, the main microcomputer 111, the sub microcomputer 112, and the preventive safety ECU 120 may perform the above-described processes in parallel. In this case, it can be expected that the above-described processing cycle is shortened and a series of processing cycles is shortened.

  In the above-described embodiment, the sub-microcomputer 112 acquires information on the driver's face such as the blinking frequency of the eyes, the eye closing time, the presence or absence of redness of the eyes, and the presence or absence of tears from the image captured by the camera 100. Other face information may be added to the symptom estimation process. For example, the symptom estimation process may be performed using face information indicating whether or not sweat is exposed on the driver's face and the driver's face color. For example, the driver's discomfort can be alleviated by lowering the set temperature of the air conditioner 131 or opening the vehicle window in response to detecting that the sweat is exposed on the driver's face. In addition, by detecting that the driver's complexion is bad, it is possible to prevent the driver from continuing driving in a poor physical condition by performing the same operation control as when the fatigue state occurs. It becomes possible.

  Note that the symptom estimation device according to the above-described embodiment estimates whether or not the driver has a symptom that may be a previous stage leading to a dangerous driving state, but the driver does not have a symptom that does not lead to a dangerous driving state. It may be estimated whether or not it has occurred. For example, symptoms that cause fatigue to the driver or symptoms that cause allergic reactions are very minor, and if left as it is, it does not lead to a dangerous driving state, but the driver may feel uncomfortable There is. Specifically, the driver's physical condition is different, such as when the driver feels tired and feels tired, or the driver is experiencing allergic reactions and feels itchy eyes It seems that the driver feels uncomfortable when he / she is in trouble. In this way, the symptom estimation device determines whether the driver has physiological symptoms that make the driver feel uncomfortable, such as a symptom that may lead to a dangerous driving state or a symptom that does not lead to a dangerous driving state. May be estimated.

  In the present embodiment, the camera 100 and the driver monitor ECU 110 are separately provided. However, for example, the driver monitor ECU 110 may be incorporated in the housing of the camera 100. In this way, the installation space in the vehicle is reduced, so that space can be saved. Further, it is not necessary to install the camera 100 and the driver monitor ECU 110 separately, and the work of attaching the camera 100 and the driver monitor ECU 110 to the vehicle becomes easy.

  Further, in step S805 in FIG. 9, the preventive safety ECU 120 may determine the traveling state of the vehicle by referring to information such as the parking brake of the vehicle.

  In the above-described embodiment, the vehicle has been described as an example, but the present invention is not limited to the vehicle. For example, the symptom estimation apparatus is applied to a moving body such as an aircraft, a ship, or a railroad vehicle, a camera is installed in front of the cockpit of the moving body, and the moving body is controlled from an image captured by the camera. A person's symptom can also be estimated. At this time, a treatment corresponding to the symptom of the pilot may be performed according to each device provided in the moving body.

  The aspect described in the above embodiment is merely a specific example, and does not limit the technical scope of the present invention. Therefore, it is possible to employ any configuration within the range where the effects of the present application are achieved.

  The symptom estimation apparatus according to the present invention can estimate whether or not a symptom that may be a previous stage leading to a dangerous maneuvering state occurs in a driver of a moving object such as a vehicle, and prevents an accident of the driver in advance. It can be applied to the use of a device or the like.

The block diagram which shows the structure of the symptom estimation apparatus which concerns on one Embodiment of this invention. The flowchart which showed the flow of the process performed in the symptom estimation apparatus which concerns on one Embodiment. The flowchart which shows the detail of the detection method of the driver's face direction of step S202 of FIG. Flowchart for the first half showing an example of the operation for calculating the number of blinks in step S205 in FIG. Flowchart in the second half showing an example of the operation for calculating the number of blinks in step S205 in FIG. The flowchart which shows an example of the process which judges the presence or absence of the redness of the eye in step S206 shown in FIG. The flowchart which shows an example of the process which judges the presence or absence of the tear of the eye in step S207 shown in FIG. The flowchart which shows an example of the process which estimates the symptom of the triver of step S208 shown in FIG. The flowchart which showed an example of the dangerous driving prevention control process in step S210 of FIG.

Explanation of symbols

100 camera 110 driver monitor ECU
111 Main microcomputer 112 Sub microcomputer 113 Face image pattern storage unit 120 Preventive safety ECU
131 Air Conditioner 132 Air Cleaner 133 Power Window Mechanism 134 Car Navigation System 135 Speaker

Claims (21)

A symptom estimation device for estimating a symptom of a pilot seated in a cockpit of a moving body,
Imaging means for imaging the direction of the cockpit from the front of the cockpit;
Face information generating means for detecting the state of the face of the pilot using the image picked up by the image pickup means and generating face information indicating the face state;
A symptom estimation device comprising: estimation means for estimating whether or not a physiological symptom that gives a sense of incongruity to the pilot is occurring in the pilot based on the face information.   The symptom estimation apparatus according to claim 1, wherein the face information generation unit detects an eye state of the driver using an image captured by the imaging unit, and generates the eye state as the face information.   The face information generation means detects whether the driver's eyes are in an open state or a closed eye state using an image picked up by the image pickup means at a predetermined time interval. The symptom estimation device according to claim 2, wherein the number of times of opening and closing per unit time is generated as the face information.   The said face information generation means detects the hyperemia state of the said operator's eyes using the image imaged by the said imaging means, and produces the hyperemia state of the said operator's eyes as said face information. Symptom estimation device.   The face information generation unit extracts a white eye portion of the driver using an image captured by the image capturing unit, and based on an edge length in which luminance changes in the white eye portion, the redness of the eye of the driver The symptom estimation device according to claim 4 which detects a state.   The face information generation means detects whether or not tears are spilled from the eyes of the pilot using the image captured by the imaging means, and generates the state of tears of the pilot as the face information The symptom estimation device according to claim 2.   The face information generation unit extracts an image of the driver's eyes or corners from the image captured by the imaging unit, and the eyes or corners image and surrounding images have a luminance difference equal to or greater than a predetermined value. The symptom estimation device according to claim 6, wherein when there is a region, it is detected that tears are spilled from the eyes of the pilot.   3. The symptom estimation according to claim 1, wherein the estimation unit estimates whether or not a symptom causing an allergic reaction has occurred on the pilot based on the face information generated by the face information generation unit. apparatus. The face image generation means detects whether the driver's eyes are in an open state or a closed eye state using an image captured by the imaging unit at a predetermined time interval, and the driver's eyes Generate the number of blinks that open and close per unit time as the face information,
The estimating means estimates that one of a symptom causing an allergic reaction and a symptom indicating a fatigue state occurs in the pilot when the number of blinks is greater than a predetermined number. Item 3. The symptom estimation device according to Item 1 or 2. The face information generating means detects an eye fullness state of the pilot's eyes using an image picked up by the imaging means, and further generates the blood pressure state of the pilot's eyes as the face information;
When the number of blinks is greater than the predetermined number and the driver's eyes are congested, the estimating means has a symptom of causing an allergic reaction to the driver. The symptom estimation device according to claim 9 which estimates. The face information generating means detects whether or not tears are spilled from the eyes of the pilot using the image captured by the imaging means, and further uses the state of the tears of the pilot as the face information. Generate
The estimating means has a symptom of causing an allergic reaction to the pilot when the number of blinks is greater than a predetermined number and tears are spilled from the eyes of the pilot. The symptom estimation device according to claim 9 which estimates. The face information generating means detects an eye fullness state of the pilot's eyes using an image picked up by the imaging means, and further generates the blood pressure state of the pilot's eyes as the face information;
The estimation means estimates that a symptom indicating a fatigue state occurs in the driver when the number of blinks is greater than the predetermined number of times and the eyes of the driver are not congested. The symptom estimation device according to claim 9. The face information generating means detects whether or not tears are spilled from the eyes of the pilot using the image captured by the imaging means, and further uses the state of the tears of the pilot as the face information. Generate
The estimation means presumes that the pilot has a symptom indicating a fatigue state when the number of blinks is greater than a predetermined number and no tears are spilled from the pilot's eyes. The symptom estimation device according to claim 9.   The symptom estimation device according to claim 1, wherein the estimation unit estimates whether or not a symptom indicating a fatigue state has occurred in the pilot based on the face information generated by the face information generation unit. The face image generation means calculates whether the eyes of the driver are in the closed state by detecting whether the eyes of the driver are in the closed state using the image captured by the imaging unit. Then, the closed eye time is generated as the face information,
The estimating means estimates that a symptom indicating a fatigue state is occurring in the driver when the closed eye state in which the closed eye time is equal to or longer than a predetermined time occurs at intervals equal to or less than a predetermined cycle. The symptom estimation device according to claim 14. The face information generation means detects whether the driver's eyes are in an open state or a closed eye state using an image captured by the imaging means at a predetermined time interval. Generate the number of blinks that open and close per unit time as the face information,
15. The symptom estimation device according to claim 14, wherein the estimation means estimates that a symptom indicating a fatigue state has occurred in the pilot when the number of blinks is less than a predetermined value. The pilot is a driver seated in the driver's seat of the vehicle,
The symptom estimation device is installed in the vehicle,
The control device according to claim 1, further comprising: a control device that controls equipment for controlling an in-vehicle environment of the vehicle or equipment for notifying the driver of information according to a symptom of the driver estimated by the estimation device. 2. The symptom estimation device according to 2. The estimating means estimates whether or not a symptom causing an allergic reaction has occurred based on the face information generated by the face information generating means,
The symptom according to claim 17, wherein the control unit activates an air cleaner provided in the vehicle when the estimation unit estimates that a symptom causing the allergic reaction has occurred in the driver. Estimating device. The estimating means estimates whether or not a symptom causing an allergic reaction has occurred based on the face information generated by the face information generating means,
The control means controls the power window mechanism provided in the vehicle and closes the window of the vehicle when the estimation means estimates that the symptom causing the allergic reaction has occurred in the driver. The symptom estimation device according to claim 17. The estimation means estimates whether or not a symptom indicating a fatigue condition has occurred in the driver based on the face information generated by the face information generation means,
18. The symptom estimation according to claim 17, wherein the control unit adjusts a set temperature of an air conditioner provided in the vehicle when the estimation unit estimates that a symptom indicating a fatigue state is occurring in the driver. apparatus. The estimation means estimates whether or not a symptom indicating a fatigue condition occurs in the driver based on the face information generated by the face information generation means,
The control means controls the car navigation system provided in the vehicle when the estimation means estimates that a symptom indicative of a fatigue condition has occurred in the driver, and sends the driver to the nearest resting facility. The symptom estimation device according to claim 17, which performs route guidance.

Images