WO2023017595A1 - 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム - Google Patents

乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム Download PDF

Info

Publication number
WO2023017595A1
WO2023017595A1 PCT/JP2021/029705 JP2021029705W WO2023017595A1 WO 2023017595 A1 WO2023017595 A1 WO 2023017595A1 JP 2021029705 W JP2021029705 W JP 2021029705W WO 2023017595 A1 WO2023017595 A1 WO 2023017595A1
Authority
WO
WIPO (PCT)
Prior art keywords
occupant
eye
state
line
determination
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/029705
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
和謙 中村
太郎 熊谷
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to PCT/JP2021/029705 priority Critical patent/WO2023017595A1/ja
Priority to DE112021007703.1T priority patent/DE112021007703B4/de
Priority to JP2023541180A priority patent/JP7378681B2/ja
Publication of WO2023017595A1 publication Critical patent/WO2023017595A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08B—SIGNALLING SYSTEMS, e.g. PERSONAL CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
    • G08B21/00—Alarms responsive to a single specified undesired or abnormal condition and not otherwise provided for
    • G08B21/02—Alarms for ensuring the safety of persons
    • G08B21/06—Alarms for ensuring the safety of persons indicating a condition of sleep, e.g. anti-dozing alarms
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60K—ARRANGEMENT OR MOUNTING OF PROPULSION UNITS OR OF TRANSMISSIONS IN VEHICLES; ARRANGEMENT OR MOUNTING OF PLURAL DIVERSE PRIME-MOVERS IN VEHICLES; AUXILIARY DRIVES FOR VEHICLES; INSTRUMENTATION OR DASHBOARDS FOR VEHICLES; ARRANGEMENTS IN CONNECTION WITH COOLING, AIR INTAKE, GAS EXHAUST OR FUEL SUPPLY OF PROPULSION UNITS IN VEHICLES
    • B60K28/00—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions
    • B60K28/02—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver
    • B60K28/06—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver responsive to incapacity of driver
    • B60K28/066—Safety devices for propulsion-unit control, specially adapted for, or arranged in, vehicles, e.g. preventing fuel supply or ignition in the event of potentially dangerous conditions responsive to conditions relating to the driver responsive to incapacity of driver actuating a signalling device
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/0818—Inactivity or incapacity of driver
    • B60W2040/0827—Inactivity or incapacity of driver due to sleepiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/225—Direction of gaze
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/229—Attention level, e.g. attentive to driving, reading or sleeping

Definitions

  • the present disclosure relates to an occupant state determination device, an occupant state determination method, and an occupant state determination system that determine the state of a vehicle occupant.
  • the degree of eye opening of the occupant is determined on the captured image, and based on the determined degree of eye opening, the eye opening of the occupant is determined.
  • the driver's line of sight can be obtained from the image captured by the main camera.
  • an inattentive driving detection device that makes a clear distinction between inattentive driving and non-inattentive driving, as compared with a detection method based on the line of sight that is used (for example, Patent Literature 1).
  • the occupant In the conventional technology for determining whether the occupant is in the eye-open state or the eye-closed state based on the captured image, in order to determine that the occupant is in the eye-open state when the occupant actually has their eyes open, the occupant It is necessary to determine that the degree of eye opening of the eye is a certain size or more.
  • an imaging device that images an occupant is often installed near or above the front of the occupant. Therefore, when the occupant directs his or her line of sight downward from the line of sight in the normal riding state, the occupant is in a downcast state as viewed from the imaging device. That is, on the captured image, the upper eyelid and lower eyelid of the occupant appear closer together.
  • the degree of eye opening based on the captured image is determined to be small. That is, based on the captured image, there is a possibility that the occupant's downcast eyes state is determined to be the eye-closed state. In the prior art, the possibility that the occupant may be in the downcast state as seen from the imaging device is not taken into consideration, and there is the problem that the actual closed-eyes state and the downcast-eyed state cannot be distinguished.
  • the present disclosure has been made in order to solve the above-described problems. It is an object of the present invention to provide an occupant state determination device that determines whether there is a passenger.
  • An occupant state determination device includes an image acquisition unit that acquires a captured image in which the face of an occupant in the vehicle is captured by an imaging device provided in the vehicle, and based on the captured image acquired by the image acquisition unit.
  • an eye opening degree determination unit that determines the degree of eye openness of the occupant
  • a line of sight detection unit that detects the line of sight of the occupant based on the captured image acquired by the image acquisition unit; When it is less than the threshold, if the degree of eye opening becomes less than the first threshold after the line-of-sight direction of the occupant detected by the line-of-sight detection unit is directed downward, it is determined that the occupant is in the downcast state and the line of sight is determined.
  • an eye state determination unit for determining that the occupant is in a closed eye state unless the degree of eye opening becomes less than a first threshold value after the line of sight direction of the occupant detected by the detection unit has turned downward. be.
  • the present disclosure it is possible to determine whether the occupant is in the closed-eyes state or the downcast state, taking into consideration the case where the occupant's line of sight is directed downward from the line of sight in the normal riding state.
  • FIG. 1 is a diagram showing a configuration example of an occupant state determination device according to Embodiment 1;
  • FIG. FIG. 2 is a diagram for explaining an installation example of an imaging device and an example of how the imaging device captures an image of an occupant in Embodiment 1;
  • FIG. 4 is another diagram for explaining an example of installation of the imaging device and an example of how the imaging device captures an image of an occupant in the first embodiment;
  • 4A and 4B show an example of the state of an occupant on a captured image in which the imaging device captures the face of the occupant in a normal riding state, and an example of the state of the occupant in the face of the occupant when viewed from the imaging device when the occupant looks down in the first embodiment.
  • FIG. 4 is a flowchart for explaining the operation of the occupant state determination device according to Embodiment 1;
  • FIG. 6 is a flow chart explaining details of a bind-off eye determination process by an eye state determination unit in step ST7 of FIG. 5;
  • FIG. 6 is a flowchart for explaining details of inattentiveness determination processing by an inattentiveness determination unit in step ST9 of FIG. 5;
  • FIG. 6 is a flow chart illustrating details of dozing determination processing by a dozing determination unit in step ST10 of FIG. 5;
  • FIG. 9A and 9B are diagrams showing an example of the hardware configuration of the occupant state determination device according to Embodiment 1.
  • FIG. 1 is a flow chart explaining details of a bind-off eye determination process by an eye state determination unit in step ST7 of FIG. 5;
  • FIG. 6 is a flowchart for explaining details of inattentiveness determination processing by an inattentiveness determination unit in step ST9 of FIG. 5;
  • FIG. 6
  • Embodiment 1 determines the state of the occupant based on an image of the face of the occupant present in the vehicle (hereinafter referred to as "captured image"). Specifically, the occupant state determination apparatus according to Embodiment 1 determines whether the occupant is in an eye-open state in which the occupant has his/her eyes open, or in the case where the occupant is not in the eye-open state, the occupant's eyes are closed. It is determined whether the eyes should be closed or the gaze should be directed downward.
  • the "downcast eyes state” is assumed to be a state in which the passenger's line of sight is directed downward from the line of sight in the normal riding state.
  • the occupant's eyes are not closed when the occupant is actually in the downcast state.
  • the occupant's upper eyelid and lower eyelid are captured closer to each other on the captured image, and when the occupant's eye openness is determined from the captured image, the eye openness is determined to be small.
  • the occupant state determination device is characterized in that, in such a case, it is determined whether the occupant is in the closed-eyes state or in the downcast state. Furthermore, the occupant state determination device according to Embodiment 1 can determine whether the occupant is looking aside and is in a dozing state. In the first embodiment, inattentiveness determined by the occupant state determination device is assumed to be downward looking inattentive due to downward gaze. In Embodiment 1 below, an occupant present in the vehicle is referred to as an occupant in the vehicle, or simply as an "occupant.” In the first embodiment, the occupant whose state is to be determined by the occupant state determination device is assumed to be the driver of the vehicle.
  • FIG. 1 is a diagram showing a configuration example of an occupant state determination device 1 according to Embodiment 1.
  • the occupant state determination device 1 is mounted on a vehicle 30 (see FIG. 2 described later) and connected to the imaging device 2 .
  • An occupant condition determination system 100 is configured by the occupant condition determination device 1 and the imaging device 2 .
  • the imaging device 2 is mounted on the vehicle 30 and installed so as to be capable of imaging at least a range in which the face of the passenger 4 (see FIG. 2, which will be described later) should exist.
  • the imaging device 2 is provided near the front of the head of the occupant 4 or above the front of the head of the occupant 4 in the vehicle 30 .
  • the imaging device 2 is mounted in a vehicle 30 by a center visor 31 (see FIG. 2 described later) of an instrument panel (hereinafter referred to as "instrument panel") 32 (see FIG. 2 described later). Assuming it is on top. Note that this is only an example, and the imaging device 2 may be provided above a headrest 33a (see FIG. 2 described later) or above a steering wheel in the vehicle 30, for example. Above the headrest 33a or above the steering wheel in the vehicle 30 is, for example, the overhead console, the inner mirror 36 (see FIG. 3 described later), the upper part of the windshield W (see FIG. 3 described later), or the ceiling. is. Above the handle also includes, for example, pillars. In FIG.
  • the imaging device 2 is assumed to be shared with a so-called DMS (Driver Monitoring System) installed for the purpose of monitoring the interior of the vehicle, for example.
  • the imaging device 2 is a visible light camera or an infrared camera.
  • the imaging device 2 outputs the captured image to the occupant condition determination device 1 .
  • the image may be a still image or a moving image.
  • the occupant state determination device 1 determines the degree of eye opening of the occupant 4 based on the captured image acquired from the imaging device 2, and the determined degree of eye opening of the occupant 4 is less than a preset threshold (hereinafter referred to as "first threshold").
  • first threshold a preset threshold
  • the eye-opening degree of the occupant 4 is less than the first threshold, which occurs after the line of sight of the occupant 4 is directed downward and the face direction of the occupant 4 is directed downward. It is determined whether the occupant 4 is in the downcast state or in the closed-eye state depending on whether or not there is.
  • the occupant state determination device 1 determines that the occupant 4 is in the eye open state when the degree of eye opening of the occupant 4 is equal to or greater than the first threshold.
  • the first threshold value is set in advance to a value at which it can be determined that the occupant 4 has his or her eyes fully open on the captured image.
  • the occupant state determination device 1 determines whether the occupant 4 is looking aside based on whether the occupant 4 is determined to be in the downcast state or the occupant 4 is in the closed-eye state, or whether the occupant 4 is dozing off. It is determined whether or not it is in the state. Details of the occupant state determination device 1 will be described later.
  • FIGS. 2 and 3 are diagrams for explaining an installation example of the imaging device 2 and an example of how the imaging device 2 images the passenger 4 in the first embodiment.
  • FIG. 2 shows the interior of the vehicle when the driver's seat 33 of the vehicle 30 is viewed from the side of the vehicle 30 .
  • FIG. 3 shows the inside of the vehicle when the windshield W and the instrument panel 32 are viewed from the inside of the vehicle in the traveling direction of the vehicle 30 .
  • the imaging device 2 provided on the upper part of the center visor 31 of the instrument panel 32 images the face of the occupant 4 from near the front of the occupant's 4 head position.
  • the instrument panel 32 is provided with a display 34 and an air conditioner operating section 35 .
  • the display 34 is, for example, a display device provided in a car navigation device (not shown), and displays route information, television images, or the like.
  • the air conditioner operation unit 35 is, for example, an operation panel for adjusting the air volume or temperature of the air conditioner.
  • the occupant 4 looks straight ahead. That is, the line of sight of the occupant 4 is the traveling direction of the vehicle 30 (see D1 in FIG. 2).
  • the passenger 4 may look in the direction of the display 34 in order to operate the car navigation device.
  • the occupant 4 may look in the direction of the air conditioner operation unit 35 in order to adjust the temperature of the air conditioner.
  • the line of sight of the occupant 4 moves from the traveling direction of the vehicle 30 to the direction in which the display 34 or the air conditioner operation unit 35 is provided (see D2 in FIG. 2 ).
  • the imaging device 2 captures an image of the face of the occupant 4 from near the front of the occupant 4 head position, when the occupant 4 moves the line of sight downward from the normal riding state, more specifically, the occupant 4 moves the line of sight downward from the normal riding state and the direction of the line of sight becomes lower than the imaging device 2, the state of the occupant 4 becomes a downcast state as seen from the imaging device 2. ⁇ When the occupant 4 is in a downcast state as viewed from the imaging device 2 , the upper eyelid and the lower eyelid of the occupant 4 approach each other on the captured image captured by the imaging device 2 .
  • the occupant 4 directing his or her line of sight more downward than in a normal riding state is also referred to as a "downward look.”
  • a region indicated by 500 in FIG. 3 (hereinafter referred to as a “downward viewing region”) will be described later.
  • FIG. 4A and FIG. 4B show an example of the state of the occupant 4 on the captured image in which the imaging device 2 captures the face of the occupant 4 in the normal riding state, and the state of the occupant 4 looking downward in the first embodiment.
  • 4 is a diagram showing an example of a state of an occupant 4 on a captured image in which the face of the occupant 4 is captured by the imaging device 2 when the eyes are downcast as viewed from the imaging device 2.
  • FIG. 4 is a diagram showing an example of a state of an occupant 4 on a captured image in which the face of the occupant 4 is captured by the imaging device 2 when the eyes are downcast as viewed from the imaging device 2.
  • the occupant 4 When the occupant 4 is in a normal riding state, that is, when the occupant 4 is looking at the traveling direction of the vehicle 30, in other words, the front, the occupant's 4 eyes are visible in the captured image, as in the captured image shown in FIG. 4A. It spreads open enough to spread. In this case, the degree of eye openness determined from the captured image is equal to or greater than the first threshold. On the other hand, when the occupant 4 looks downward, the eyes of the occupant 4 are fully open in the captured image, as shown in FIG. 4B, although the occupant 4 does not actually close their eyes. No, in other words, it looks like it's closed. In this case, the degree of eye openness determined from the captured image is less than the first threshold.
  • the occupant state determination device 1 As in the captured image shown in FIG. , it is determined that the occupant 4 is in the downcast state.
  • the occupant state determination device 1 according to Embodiment 1 includes an image acquisition unit 11, an eye opening degree determination unit 12, a line of sight detection unit 13, a face direction detection unit 14, a drowsiness detection unit 15, an eye state determination unit 16, and an inattentiveness determination unit 17. , doze determination unit 18 and output unit 19 .
  • the image acquisition unit 11 acquires a captured image of the face of the occupant 4 in the vehicle 30 captured by the imaging device 2 provided in the vehicle 30 .
  • the image acquisition unit 11 outputs the acquired captured image to the degree of eye opening determination unit 12 , line of sight detection unit 13 , face orientation detection unit 14 , and drowsiness detection unit 15 .
  • the degree of eye openness determination unit 12 determines the degree of eye openness of the occupant 4 based on the captured image acquired by the image acquisition unit 11 . Specifically, the eye opening degree determination unit 12 detects the upper eyelid and the lower eyelid of the occupant 4 using a known image recognition technique for the captured image. For example, in the captured image, a region in which the face of the occupant 4 may exist (hereinafter referred to as a “face detection region”) is set in advance, and the degree of eye opening determination section 12 determines a known face detection region of the captured image. Edge detection is performed to extract a feature point indicating the upper eyelid of the occupant 4 and a feature point indicating the lower eyelid.
  • face detection region a region in which the face of the occupant 4 may exist
  • the degree-of-open-eyes determination unit 12 calculates, for example, the vertical distance between the upper eyelid and the lower eyelid of the occupant 4 in the captured image (hereinafter referred to as "eyelid distance") in order from the top of the captured image, from the left. Calculate the eyelid distance. Then, the degree of eye openness determination unit 12 determines the degree of eye openness based on the ratio between the calculated maximum eyelid distance and the reference value of the degree of eye openness. That is, the unit of eye openness is percent. As the reference value of the degree of eye openness, for example, an average value of the degree of eye openness at the time of awakening of adults of general physique, which is calculated by conducting tests, is set in advance.
  • the reference value of the degree of eye opening may be, for example, the average value of the degree of eye opening of the occupant 4 during a preset time after the passenger 4 boarded the vehicle 30. It may be the average value of all eye opening degrees determined so far by the unit 12 . It should be noted that the method for determining the degree of eye opening described above is merely an example.
  • the degree of eye openness determination section 12 may determine the degree of eye openness using various known algorithms.
  • the degree-of-eye-openness determination unit 12 outputs information regarding the determination result of the degree of eye-openness of the occupant 4 (hereinafter referred to as “eye-openness information”) to the eye state determination unit 16 .
  • the degree of eye opening information is, for example, information in which the degree of eye opening of the occupant 4 is associated with the captured image acquired by the image acquisition unit 11 .
  • the line-of-sight detection unit 13 detects the line of sight of the occupant 4 based on the captured image acquired by the image acquisition unit 11 . Specifically, the line-of-sight detection unit 13 detects the line-of-sight direction of the occupant 4 based on the captured image. The line-of-sight direction of the occupant 4 is the line-of-sight direction of the occupant 4 . The line-of-sight detection unit 13 may detect the line-of-sight direction of the occupant 4 using a known image recognition technique for the captured image.
  • the line-of-sight detection unit 13 can detect the inner corners and iris of the occupant 4 on the captured image, and detect the line-of-sight direction of the occupant 4 from the positional relationship between the inner corners of the eyes and the iris. Further, for example, if the imaging device 2 is an infrared camera, the line-of-sight detection unit 13 detects the positional relationship between the pupil and the Purkinje image obtained by the reflection from the cornea when the infrared camera irradiates a near-infrared point light source. The line-of-sight direction of the occupant 4 can also be detected.
  • the line-of-sight detection unit 13 may detect the line-of-sight direction of the occupant 4 using various known algorithms.
  • the line-of-sight direction of the occupant 4 is represented by an angle with respect to a predetermined reference direction.
  • the line-of-sight direction of the occupant 4 is a line-of-sight direction that represents how many degrees the line-of-sight direction has rotated in the horizontal direction and the vertical direction from the state when the occupant 4 is looking straight ahead in the seat. expressed as an angle.
  • the line-of-sight detection unit 13 detects information about the line of sight of the occupant 4 (hereinafter referred to as "line-of-sight information"). is output to the eye state determination unit 16 .
  • the line-of-sight information is, for example, information in which information indicating the line-of-sight direction of the occupant 4 is associated with a captured image.
  • the line of sight detection unit 13 when the line of sight of the occupant 4 cannot be detected, the line of sight detection unit 13 outputs line of sight information indicating that the line of sight of the occupant 4 has not been detected to the eye condition determination unit 16 .
  • the face direction detection unit 14 detects the face direction of the occupant 4 based on the captured image acquired by the image acquisition unit 11 .
  • the face direction detection unit 14 may detect the face direction of the occupant 4 using a known image recognition technique for the captured image.
  • the face direction detection unit 14 detects the face direction of the occupant 4 by general pattern recognition based on the captured image. This is merely an example, and the face direction detection unit 14 may detect the face direction of the occupant 4 using various known algorithms.
  • the orientation of the face of the occupant 4 is represented by an angle with respect to a predetermined reference direction.
  • the face orientation of the occupant 4 represents how many degrees the occupant 4 has rotated in the horizontal direction and the vertical direction from the state when the occupant 4 is looking straight ahead in the seat. is represented by the angle of
  • the face orientation detection unit 14 outputs information regarding the detected face orientation of the occupant 4 (hereinafter referred to as “face orientation information”) to the eye condition determination unit 16 .
  • the face direction information is, for example, information in which the face direction of the occupant 4 and the captured image are associated with each other.
  • the drowsiness detection unit 15 detects the drowsiness level of the occupant 4 based on the captured image acquired by the image acquisition unit 11 .
  • the drowsiness detection unit 15 may detect the drowsiness level of the crew member 4 using a known image recognition technique for the captured image. For example, the drowsiness detection unit 15 determines the number of blinks of the occupant 4 in a preset unit time based on the captured image.
  • the drowsiness detection unit 15 detects the drowsiness level of the occupant 4 based on how many times the occupant 4 blinks within a unit time. It is determined in advance which level of drowsiness is to be set when how many times blinking is performed within a unit time.
  • the sleepiness level is set to a value that indicates that the higher the sleepiness level, the stronger the sleepiness of the passenger 4, and the lower the sleepiness level, the more wakeful the passenger 4 is.
  • the drowsiness detection unit 15 may detect the drowsiness level of the crew member 4 using various known algorithms. Here, the drowsiness detection unit 15 detects the drowsiness level of the crew member 4 based on the captured image, but this is only an example. For example, the drowsiness detection unit 15 may acquire biological information of the occupant 4 such as body temperature from a biological sensor provided in the vehicle 30 and detect the drowsiness level of the occupant 4 from the biological information.
  • the drowsiness detection unit 15 outputs information regarding the detected drowsiness level of the passenger 4 (hereinafter referred to as “drowsiness level information”) to the dozing determination unit 18 .
  • the drowsiness level information is, for example, information in which the drowsiness level of the crew member 4 and the captured image are associated with each other.
  • the eye state determination unit 16 determines that the degree of eye openness of the occupant 4 is less than the first threshold. Based on the line-of-sight direction of the occupant 4 and the face orientation of the occupant 4 detected by the face orientation detection unit 14, it is determined whether the occupant 4 is in the downcast state or the closed eye state. Specifically, when the degree of eye openness determined by the degree of eye openness determination unit 12 is less than the first threshold, the eye state determination unit 16 determines that the direction of the line of sight of the occupant 4 detected by the line of sight detection unit 13 is directed downward.
  • the eye state determination unit 16 determines that the line-of-sight direction of the occupant 4 detected by the line-of-sight detection unit 13 is downward, and the face direction of the occupant 4 detected by the face direction detection unit 14 is downward. If the degree of eye opening does not later become less than the first threshold, it is determined that the occupant 4 is in the eye-closed state.
  • the determination of whether the occupant 4 is in the downcast state or in the closed-eye state, which is performed by the eye state determination unit 16, is also referred to as "downcast eyes determination".
  • the eye state determination unit 16 determines whether or not the eye openness of the occupant 4 is less than the first threshold based on the eye openness information output from the eye openness determination unit 12 . Note that, when the degree of eye openness of the occupant 4 determined by the degree of eye openness determination unit 12 is equal to or greater than the first threshold value, the eye state determination unit 16 determines that the occupant 4 is in an open-eyed state without performing the blinded eye determination.
  • the bind-off determination process by the eye state determination unit 16 will be described in detail below.
  • the eye state determination unit 16 time-sequentially displays the degree of eye openness information output from the degree of eye openness determination unit 12, the line of sight information output from the line of sight detection unit 13, and the face direction information output from the face direction detection unit 14. It is associated and stored.
  • the information in which the degree of eye openness information, the gaze information, and the face direction information are stored in chronological order by the eye state determination unit 16 is referred to as "eye-related information".
  • the eye state determination unit 16 When determining that the degree of eye opening of the occupant 4 is less than the first threshold, the eye state determination unit 16 refers to the eye-related information to obtain the line-of-sight information of the occupant 4 before the degree of eye opening of the occupant 4 is less than the first threshold. to get Next, based on the obtained line-of-sight information of the occupant 4, the eye state determination unit 16 determines whether the direction of the line-of-sight of the occupant 4 detected by the line-of-sight detection unit 13 before it is determined that the degree of eye opening is less than the first threshold value. , the trajectory of the line-of-sight direction is calculated.
  • the eye state determination unit 16 refers to the eye-related information and acquires face orientation information of the occupant 4 before the degree of eye opening of the occupant 4 becomes less than the first threshold. Based on the obtained face orientation information of the occupant 4, the eye state determination unit 16 determines the face orientation of the occupant 4 from the face orientation of the occupant 4 detected by the face orientation detection unit 14 before it is determined that the degree of eye opening is less than the first threshold. Calculate the trajectory of the face direction.
  • the length of time for calculating the trajectory of the line-of-sight direction and the length of time for calculating the trajectory of the face orientation be the same.
  • "before it is determined that the degree of eye openness is less than the first threshold” refers to a set time before the time when the degree of eye openness is determined to be less than the first threshold. The set time can be set appropriately as described above.
  • the eye state determination unit 16 determines whether the calculated trajectory of the line-of-sight direction of the occupant 4 is downward and whether or not the calculated trajectory of the face orientation of the occupant 4 is downward. More specifically, the eye condition determining unit 16 determines whether the calculated trajectory of the line-of-sight direction of the occupant 4 is directed in the direction of the preset downward viewing region, and determines whether the calculated trajectory of the occupant's 4 face direction is directed. is directed toward the downward viewing area. The eye state determination unit 16 determines that the line-of-sight direction of the occupant 4 is downward when the locus of the line-of-sight direction of the occupant 4 is directed toward the downward viewing region.
  • the eye state determination unit 16 determines that the face direction of the occupant 4 is directed downward.
  • the downward viewing region is, for example, the region indicated by 500 in FIG. 3, and the eye condition determining section 16 stores information regarding the downward viewing region.
  • the downward viewing area is an area set below the imaging device 2 within the vehicle 30 . For example, when the line-of-sight direction of the downward-looking occupant 4 continues in the downward-looking region for a predetermined period of time, it is assumed that the occupant 4 may be looking aside. area is set.
  • the downward viewing region is set to include a region in which there is a device that can be operated by the passenger 4 below the imaging device 2, such as the display 34 of the car navigation device.
  • the downward viewing region is set in advance by the manufacturer or the like according to the installation position of the imaging device 2, but this is merely an example.
  • the eye state determination unit 16 has a function of setting the downward viewing region each time according to the positional relationship between the installation position of the imaging device 2 and the head position of the occupant 4. may Note that the eye condition determination unit 16 may estimate the head position of the occupant 4 using a known image recognition technique.
  • the installation position of the imaging device 2 is known in advance.
  • the eye state determination unit 16 determines that the occupant 4 is in the downcast state when the trajectory of the line-of-sight direction of the occupant 4 and the trajectory of the face orientation of the occupant 4 are directed downward. In other words, the eye state determination unit 16 determines that the occupant 4 is in the downcast state when the degree of eye opening of the occupant 4 is less than the first threshold while the direction of the line of sight of the occupant 4 and the orientation of the face of the occupant 4 tend to be downward. It is determined that The eye state determination unit 16 determines that the occupant 4 is in the eye-closed state when the trajectory of the line-of-sight direction of the occupant 4 and the trajectory of the face orientation of the occupant 4 do not point downward.
  • the eye state determination unit 16 determines that unless the degree of eye opening of the occupant 4 is less than the first threshold while the line-of-sight direction of the occupant 4 and the face direction of the occupant 4 tend to face downward, the occupant 4 It is determined that the eyes are closed.
  • the eye state determination unit 16 stores information (hereinafter referred to as "eye state information") indicating whether the occupant 4 has determined that the eyes are downcast, whether the occupant 4 has closed the eyes, or whether the occupant 4 has the eyes open. , looking aside determination unit 17 , doze determination unit 18 , or output unit 19 . Specifically, when the eye state determination unit 16 determines that the occupant 4 is in the bound-down state as a result of performing the blind-eyed determination, the eye state information indicating that the occupant 4 is in the bound-down state is sent to the inattentive-looking determination unit 17. Output.
  • eye state information information indicating whether the occupant 4 has determined that the eyes are downcast, whether the occupant 4 has closed the eyes, or whether the occupant 4 has the eyes open.
  • the eye state determination unit 16 determines that the occupant 4 is in the closed-eye state as a result of the down-eyed determination, the eye state determination unit 16 outputs eye state information indicating that the occupant 4 is in the closed-eye state to the doze determination unit 18 .
  • the eye state determination unit 16 determines that the occupant 4 is in the eye open state, the eye state determination unit 16 outputs eye state information indicating that the occupant 4 is in the eye open state to the output unit 19 .
  • the inattentive determination unit 17 outputs eye state information indicating that the occupant 4 is in the downcast state from the eye state determination unit 16, in other words, when the eye state determination unit 16 determines that the occupant 4 is in the downcast state. , it is determined whether or not the occupant 4 is looking aside based on whether or not a preset condition (hereinafter referred to as "condition for determination of inattentiveness") is satisfied.
  • condition for determination of inattentiveness a preset condition
  • the determination of whether or not the occupant 4 is looking aside is performed by the looking-aside determining unit 17 when the eye state information indicating that the occupant 4 is in the downcast state is output from the eye state determining unit 16. , is also called “aside judgment”.
  • condition (1) is set as the condition for inattentiveness determination.
  • the inattentive determination unit 17 determines that the occupant 4 is looking aside when the following condition (1) is satisfied.
  • ⁇ Condition (1)> The occupant's downcast eyes state continues for a preset time (hereinafter referred to as "time for judging inattentiveness") or more.
  • the inattentive determination unit 17 When eye state information indicating that the occupant 4 is in the downcast state is output from the eye state determination unit 16, the inattentive determination unit 17 counts up the time (hereinafter referred to as "time after determination of downcast eyes"). Note that the inattentive determination unit 17 stores the counted-up time after determination of downcast eyes. Then, when the counted-up post-disturbance determination time is equal to or longer than the inattentive-sight determination time, the inattentive-sight determination unit 17 determines that the occupant 4's downcast state continues for the inattentive-sight determination time or longer, and thus satisfies the inattentive-sight determination condition. judge.
  • the inattentive-looking determination unit 17 determines that the occupant 4 is looking aside. If the inattentive-looking determining unit 17 determines that the counted-up after-judgment time for looking down eyes is equal to or longer than the time for inattentive-looking determination and determines that the occupant 4 is looking aside, the inattentive-looking determination unit 17 starts counting up the time after determining the inattentive eyes, and then the occupant 4 looks down. When the state disappears, or when the engine of the vehicle 30 is turned off, the time after the blind eye determination is cleared.
  • the inattentive-looking determining unit 17 determines that the downcast state of the occupant 4 has not continued for the time for inattentive-looking determination or longer, it determines that the condition for inattentive-looking determination is not satisfied. That is, the inattentive-looking determination unit 17 determines that the occupant 4 is not looking aside.
  • the inattentive determination unit 17 determines that the occupant 4 is looking aside as a result of the inattentive determination, it outputs information to the output unit 19 that the occupant 4 is in a state of inattentive looking.
  • the doze determination unit 18 determines that the occupant 4 has the eyes closed. , based on the drowsiness level of the occupant 4 detected by the drowsiness detector 15, it is determined whether the occupant 4 is dozing off. In the first embodiment, the dozing determination unit 18 determines whether or not the occupant 4 is dozing off when the eye state determination unit 16 outputs the eye state information indicating that the occupant 4 is in a closed eye state. , is also called "drowsy determination". The dozing determination unit 18 may identify the drowsiness level of the crew member 4 from the drowsiness level information output from the drowsiness detection unit 15 .
  • the drowsiness determination unit 18 presets a state in which the drowsiness level of the occupant 4 is equal to or higher than a preset threshold value (hereinafter referred to as a “threshold value for determining drowsiness”) (hereinafter referred to as a “drowsiness state"). If it continues longer than the set time (hereinafter referred to as “drowsiness determination time”), it is determined that the occupant 4 is dozing off.
  • a preset threshold value hereinafter referred to as a "threshold value for determining drowsiness”
  • the dozing determination unit 18 outputs the eye state information indicating that the occupant 4 has closed eyes from the eye state determination unit 16, and the drowsiness level of the occupant 4 detected by the drowsiness detection unit 15 is equal to or higher than the drowsiness determination threshold. In this case, it is determined that the occupant 4 is drowsy.
  • the drowsiness determining unit 18 determines that the occupant 4 is in a state of drowsiness, it counts up the time (hereinafter referred to as "time after drowsiness determination"). Note that the dozing determination unit 18 stores the counted-up post-drowsiness determination time.
  • the dozing determination unit 18 determines that the drowsiness state of the occupant 4 has continued for the dozing determination time or longer, and the occupant 4 has fallen asleep.
  • the dozing determination unit 18 starts counting up the post-drowsiness determination time and determines whether the drowsiness of the occupant 4 is When the level is no longer equal to or greater than the threshold for drowsiness determination, when the occupant 4 is no longer in a state of closed eyes after the start of counting up the time after drowsiness determination, or when the engine of the vehicle 30 is turned off, the time after drowsiness determination clear.
  • the dozing determination unit 18 determines that the occupant 4 is dozing off as a result of the dozing determination, it outputs information to the output unit 19 that the occupant 4 is in a dozing state.
  • the output unit 19 outputs various alarms. Specifically, when the inattentive determination unit 17 outputs information indicating that the occupant 4 is in the inattentive state, the output unit 19 determines that the occupant 4 is looking aside. In this case, information (hereinafter referred to as "first warning information”) for warning the occupant 4 that he or she is careless ahead is output.
  • the output unit 19 outputs information indicating that the occupant 4 is dozing off from the dozing determining unit 18, in other words, when the dozing determining unit 18 determines that the occupant 4 is dozing off, the occupant 4 information (hereinafter referred to as "second warning information”) for warning that the user is dozing off.
  • the output unit 19 outputs the first warning information or the second warning information to an output device (not shown) mounted on the vehicle 30, for example.
  • an output device (not shown) mounted on the vehicle 30, for example.
  • the output device is, for example, the display 34 or an audio output device.
  • the display 34 displays according to the first warning information or the second warning information.
  • the display 34 displays the message "Caution about falling asleep.
  • the display 34 may display an icon such as a coffee cup icon that prompts a break.
  • the audio output device outputs audio according to the first warning information or the second warning information.
  • the audio output device outputs the audio "Caution ahead”.
  • the audio output device outputs audio such as "Beware of falling asleep. Please take a break.”
  • the output unit 19 does not output anything when the eye state determination unit 16 outputs the eye state information indicating that the occupant 4 has his or her eyes open. Note that this is only an example, and the output unit 19, for example, when the eye state determination unit 16 outputs the eye state information indicating that the occupant 4 is in a state where the eyes are open, is used to notify that the driving state is good. may be output to an output device.
  • FIG. 5 is a flow chart for explaining the operation of the occupant state determination device 1 according to the first embodiment.
  • the occupant condition determination device 1 repeats the operation shown in the flowchart of FIG. 5 while the vehicle 30 is running.
  • the image acquisition unit 11 acquires a captured image of the face of the occupant 4 in the vehicle 30 captured by the imaging device 2 provided in the vehicle 30 (step ST1).
  • the image acquisition unit 11 outputs the acquired captured image to the degree of eye opening determination unit 12 , line of sight detection unit 13 , face orientation detection unit 14 , and drowsiness detection unit 15 .
  • the degree of eye openness determination unit 12 determines the degree of eye openness of the occupant 4 based on the captured image acquired by the image acquisition unit 11 in step ST1 (step ST2). Eye openness determination section 12 outputs eye openness information to eye state determination section 16 .
  • the line-of-sight detection unit 13 detects the line of sight of the occupant 4 based on the captured image acquired by the image acquisition unit 11 in step ST1 (step ST3).
  • the line-of-sight detection unit 13 outputs the line-of-sight information to the eye state determination unit 16 .
  • the face direction detection unit 14 detects the face direction of the occupant 4 based on the captured image acquired by the image acquisition unit 11 in step ST1 (step ST4).
  • the face orientation detection unit 14 outputs face orientation information to the eye state determination unit 16 .
  • the drowsiness detection unit 15 detects the drowsiness level of the passenger 4 based on the captured image acquired by the image acquisition unit 11 in step ST1 (step ST5).
  • the drowsiness detection unit 15 outputs the drowsiness level information to the dozing determination unit 18 .
  • the eye state determination section 16 determines whether the eyes are down (step ST7). On the other hand, when the degree of eye openness of the occupant 4 determined by the degree of eye openness determination unit 12 is equal to or greater than the first threshold value (“NO” in step ST6), the eye state determination unit 16 determines that the eye state of the occupant 4 is open. judge. Then, the eye state determination unit 16 outputs eye state information indicating that the occupant 4 is in the open state to the output unit 19, and the operation of the occupant state determination device 1 ends the processing.
  • the inattentive looking determination unit 17 determines whether or not the conditions for inattentive determination are satisfied. Inattentive determination is performed to determine whether or not the occupant 4 is looking aside (step ST9).
  • the dozing determination unit 18 determines whether or not the occupant 4 is dozing off based on the drowsiness level of the occupant 4 detected by the drowsiness detection unit 15 in step ST5 (step ST10).
  • the output unit 19 outputs information indicating that the occupant 4 is looking aside from the inattentive determination unit 17 in step ST9, in other words, the inattentive determination unit 17 determines that the occupant 4 is looking aside. In this case, the first warning information is output. Further, the output unit 19 outputs information indicating that the occupant 4 is dozing off from the dozing determination unit 18 in step ST10. When determined, the second warning information is output (step ST11).
  • steps ST1 to ST5 do not necessarily have to be performed in this order.
  • the processes of steps ST1 to ST5 may be performed in parallel.
  • FIG. 6 is a flow chart for explaining the details of the bind-off determination process by the eye condition determining section 16 in step ST7 of FIG.
  • the eye state determination unit 16 determines whether or not the direction of the line of sight of the occupant 4 detected by the line of sight detection unit 13 before it is determined that the degree of eye opening is less than the first threshold (step ST71). Specifically, the eye condition determination unit 16 first refers to the eye-related information and acquires the line-of-sight information of the occupant 4 before the degree of eye opening of the occupant 4 becomes less than the first threshold.
  • the eye state determination unit 16 determines whether the direction of the line-of-sight of the occupant 4 detected by the line-of-sight detection unit 13 before it is determined that the degree of eye opening is less than the first threshold value. , the trajectory of the line-of-sight direction is calculated. Then, the eye condition determination unit 16 determines whether or not the calculated trajectory of the line-of-sight direction of the occupant 4 is directed downward. More specifically, the eye condition determination unit 16 determines whether or not the calculated trajectory of the line-of-sight direction of the occupant 4 is directed toward the downward viewing area.
  • the eye state determination unit 16 detects the face orientation before determining that the degree of eye opening is less than the first threshold. It is determined whether or not the face direction of the occupant 4 detected by the unit 14 is downward (step ST72). Specifically, the eye state determination unit 16 refers to the eye-related information and acquires the face orientation information of the occupant 4 before the degree of eye opening of the occupant 4 becomes less than the first threshold. Next, based on the acquired face orientation information of the occupant 4, the eye state determination unit 16 determines whether the face orientation of the occupant 4 detected by the face orientation detection unit 14 before the degree of eye opening is determined to be less than the first threshold value.
  • the trajectory of the face direction of the passenger 4 is calculated. Then, the eye state determination unit 16 determines whether or not the calculated trajectory of the face orientation of the occupant 4 is directed downward. More specifically, the eye condition determination unit 16 determines whether or not the calculated trajectory of the face orientation of the occupant 4 is directed toward the downward viewing area.
  • step ST72 If it is determined in step ST72 that the face direction of the occupant 4 detected by the face direction detection unit 14 before it is determined that the degree of eye opening is less than the first threshold is downward ("YES" in step ST72) case), the eye state determination unit 16 determines that the occupant 4 is in the downcast state (step ST73). In other words, the eye state determination unit 16 determines that the occupant 4 is in the downcast state when the degree of eye opening of the occupant 4 is less than the first threshold while the direction of the line of sight of the occupant 4 and the orientation of the face of the occupant 4 tend to be downward. It is determined that Then, the eye state determining section 16 outputs eye state information indicating that the occupant 4 is in the downcast state to the looking-aside determining section 17 .
  • step ST71 if the line of sight of the occupant 4 is not downward ("NO” in step ST71), or if the face of the occupant 4 is not downward (“NO” in step ST72) ”), that is, when the degree of eye opening of the occupant 4 becomes less than the first threshold value after the direction of the line of sight of the occupant 4 and the direction of the face of the occupant 4 are directed downward, the eye state determination unit 16 determines that the occupant 4 determines that the eyes are closed (step ST74).
  • the eye state determination unit 16 determines that unless the degree of eye opening of the occupant 4 is less than the first threshold while the line-of-sight direction of the occupant 4 and the face direction of the occupant 4 tend to face downward, the occupant 4 It is determined that the eyes are closed. Then, the eye state determination section 16 outputs eye state information indicating that the occupant 4 is in a closed eye state to the dozing determination section 18 .
  • FIG. 7 is a flowchart for explaining the details of the inattentiveness determination process by the inattentiveness determination unit 17 in step ST9 of FIG.
  • the condition (1) described above is set as the condition for inattentiveness determination.
  • the inattentive determination unit 17 counts up the time after determination of downcast eyes (step ST91). Note that the inattentive determination unit 17 stores the counted-up time after determination of downcast eyes.
  • the inattentive-looking determination unit 17 determines whether or not the occupant 4's downcast eyes state has continued for the inattentive-looking determination time or longer (step ST92). Specifically, the inattentive-looking determination unit 17 determines whether or not the counted-up post-determining time for depressed eyes has reached or exceeded the time for inattentive-looking determination. If it is determined in step ST92 that the occupant 4's blinded eyes state has continued for the inattentive determination time or longer ("YES" in step ST92), the inattentive-looking determination section 17 determines that the inattentive-sight determination condition is satisfied, and the occupant 4 is in the inattentive state (step ST93).
  • the inattentive determination unit 17 outputs to the output unit 19 information indicating that the occupant 4 is in the inattentive state.
  • the inattentive-looking determining section 17 determines that the condition for inattentive-looking determination is not satisfied. If so, it is determined that the occupant 4 is not looking aside, and the inattentive-looking determination process ends.
  • FIG. 8 is a flowchart for explaining the details of the dozing determination process by the dozing determining section 18 in step ST10 of FIG.
  • the dozing determination unit 18 identifies the drowsiness level of the passenger 4 detected by the drowsiness detection unit 15 in step ST5 of FIG. Then, the dozing determination unit 18 determines whether or not the occupant 4 is in a drowsiness state in which the drowsiness level is equal to or higher than the drowsiness determination threshold (step ST101). In step ST101, when the occupant 4 determines that the drowsiness level is equal to or higher than the drowsiness determination threshold value ("YES" in step ST101), the dozing determination unit 18 determines the post-drowsiness determination time.
  • the doze determination unit 18 determines whether or not the drowsy state of the passenger 4 has continued for the doze determination time or longer (step ST103). Specifically, the doze determination unit 18 determines whether or not the counted-up post-drowsy determination time has reached or exceeded the doze determination time. When it is determined in step ST103 that the drowsiness state of the occupant 4 has continued for the dozing determination time or longer ("YES" in step ST103), the dozing determination section 18 determines that the occupant 4 is dozing off. (step ST104). Then, the dozing determination unit 18 outputs information to the output unit 19 indicating that the passenger 4 is in a dozing state.
  • step ST101 if the occupant 4 determines that the drowsiness level of the occupant 4 is equal to or higher than the drowsiness determination threshold value ("NO" in step ST101), and in step ST103, the occupant If it is determined that the drowsy state of 4 has not continued for the dozing determination time or longer ("NO" in step ST103), the dozing determination unit 18 determines that the occupant 4 is not dozing, and performs the dozing determination process. exit.
  • the conventional technology does not take into consideration the possibility that the occupant 4 may be in the downcast state as viewed from the imaging device 2, and there is the problem that the actual closed-eye state and the downcast-eye state cannot be distinguished. there were.
  • the occupant state determination device 1 according to Embodiment 1 when the degree of eye opening of the occupant 4 in the vehicle 30 is less than the first threshold value, the line-of-sight direction of the occupant 4 and the face direction of the occupant 4 are directed downward.
  • the occupant state determination device 1 can determine whether the occupant 4 is in the closed-eyes state or the downcast state in consideration of the case where the occupant 4 directs the line of sight downward from the line of sight in the normal riding state. can.
  • the occupant state determination device 1 determines whether or not the occupant 4 is looking aside when determining that the occupant 4 is in the downcast state. 1 Output warning information.
  • the occupant state determination device 1 determines that the occupant 4 is in a closed-eye state
  • the occupant state determination device 1 determines whether the occupant 4 is dozing off, and outputs second warning information when determining that the occupant 4 is dozing off. . Since the occupant state determination device 1 can determine whether the occupant 4 is in the eye-closed state or the eyes-down state from the captured image, for example, the occupant 4 is in the closed-eyes state even though the occupant 4 does not actually close the eyes.
  • the occupant state determination device 1 prevents, for example, warning the occupant 4 that he is dozing off despite the fact that the occupant 4 does not actually close his eyes. can be done. Further, as described above, when the occupant state determination device 1 determines that the occupant 4 is looking aside, the occupant state determination device 1 outputs the first warning information for warning that the occupant is "careless ahead". When the occupant 4 looks downward, it may not be possible to strictly distinguish between the downcast state and the closed-eye state directly from the captured image.
  • the occupant state determination device 1 determines whether the occupant 4 is not in the eye-open state based on the captured image, in other words, when the eye-opening degree of the occupant 4 is less than the first threshold. It is determined that the occupant 4 should be in the downcast state if the degree of eye opening becomes less than the first threshold value after the line of sight direction and the face direction of the occupant 4 are directed downward. Then, the occupant state determination device 1 determines that the occupant 4 is in the inattentive state if the downcast state continues for the inattentive-looking determination time or longer.
  • the occupant state determination device 1 outputs a warning of ⁇ not paying attention to the front'' when the occupant 4 is determined to be looking aside, thereby outputting a warning with a wider range of meaning than the warning of ⁇ beware of looking aside'', for example. By doing so, it is possible to prevent the occupant 4 from being given an erroneous warning.
  • the eye state determination unit 16 detects the occupant detected by the line-of-sight detection unit 13 when the degree of eye openness determined by the eye openness determination unit 12 is less than the first threshold value. 4 is directed downward, and the degree of eye openness determined by the degree of eye openness determination unit 12 becomes less than the first threshold after the face direction of the occupant 4 detected by the face direction detection unit 14 is directed downward. For example, it is determined that the occupant 4 is in the downcast state, the direction of the line of sight of the occupant 4 detected by the line of sight detection unit 13 is directed downward, and the face direction of the occupant 4 detected by the face direction detection unit 14 is directed downward.
  • the degree of eye openness determined by the degree of eye openness determination unit 12 later became less than the first threshold value, it was determined that the occupant 4 was in the eye-closed state. This is only an example, and the eye state determination unit 16 does not consider the face direction of the occupant 4.
  • the line of sight detection unit 13 detects If the degree of eye openness of the occupant 4 becomes less than the first threshold value after the direction of the line of sight of the occupant 4 who has been doing so has turned downward, it is determined that the occupant 4 is in the downcast state, and the occupant 4 detected by the line of sight detection unit 13 It may be determined that the occupant 4 is in the eye-closed state unless the degree of eye opening becomes less than the first threshold after the line of sight of the occupant 4 is directed downward.
  • the occupant state determination device 1 can be configured without the face direction detection unit 14 in the configuration example of the occupant state determination device 1 shown in FIG. 1 .
  • step ST72 can be omitted for the operation of the eye state determination section 16 in the occupant state determination device 1 described using the flowchart of FIG.
  • the occupant state determination device 1 considers the direction of the face of the occupant 4, and when the degree of eye opening of the occupant 4 is less than the first threshold, the direction of the line of sight of the occupant 4 is directed downward, and If the degree of eye opening becomes less than the first threshold after the orientation is directed downward, it is determined that the occupant 4 is in the downcast state, thereby determining whether or not the line of sight of the occupant 4 is directed downward. accuracy can be improved.
  • the eye state determination unit 16 determines that the degree of eye opening of the occupant 4 is less than the first threshold.
  • the trajectory of the face orientation of the occupant 4 calculated from the above is directed toward an object existing in the downward viewing area (hereinafter referred to as the “downward viewing object”), the direction of the line of sight of the occupant 4 is directed downward.
  • the downward-viewing object is, for example, a vehicle-mounted device such as a navigation device, an air conditioner operation unit 35, or a display 34 or the like.
  • the inner mirror 36 may be used as the downward viewing object. Since the installation position of the downward-looking object is known in advance, the eye state determination unit 16 determines whether or not the line-of-sight direction is directed toward the downward-looking object if the trajectory of the line-of-sight direction of the occupant 4 is known. can. In this case, regarding the operation of the eye condition determination unit 16 in the occupant condition determination device 1 described using the flowchart of FIG.
  • the eye state determination unit 16 determines whether or not the trajectory of the face direction of the occupant 4 is directed toward the downward-viewing object. Further, the eye state determination unit 16 does not consider the face direction of the occupant 4, and when it is determined that the degree of eye opening of the occupant 4 is less than the first threshold, When the trajectory of the line-of-sight direction of the occupant 4 calculated from the line-of-sight direction of the occupant 4 detected by the line-of-sight detection unit 13 is directed toward a downward-looking object existing in the downward-looking region, the line-of-sight direction of the occupant 4 could have been directed downwards.
  • the occupant state determination device 1 can be configured without the face direction detection unit 14 in the configuration example of the occupant state determination device 1 shown in FIG. 1 .
  • the eye state determination unit 16 in the occupant state determination device 1 described using the flowchart of FIG. Determine whether or not the player is heading toward an existing downward looking object.
  • the eye condition determining section 16 can omit the process of step ST72.
  • the occupant state determination device 1 performs the inattentive determination as to whether or not the occupant 4 is looking aside.
  • the occupant state determination device 1 does not necessarily have a function of performing inattentiveness determination.
  • the occupant state determination device 1 does not include the inattentive determination unit 17 in the configuration example of the occupant state determination device 1 shown in FIG. can.
  • the processing of step ST9 can be omitted for the operation of the occupant condition determination device 1 described using the flowchart of FIG.
  • the eye state determination unit 16 determines that the occupant 4 is in the downcast state
  • the eye state information indicating that the occupant 4 is in the downcast state is output to the output unit 19 .
  • the output unit 19 may output, for example, first warning information for warning of looking aside.
  • the output unit 19 may output the first warning information, for example, when the occupant 4's downcast state continues for the time for judging inattentiveness or longer.
  • the output unit 19 receives the output from the eye state determination unit 16.
  • the eye state information indicating that the occupant 4 is in the downcast state may be output to the inattentiveness determination device.
  • the occupant state determination device 1 determines whether or not the occupant 4 is dozing off. However, this is only an example, and the occupant state determination device 1 does not necessarily have the function of determining dozing off. When the occupant state determination device 1 does not have the function of determining dozing, for example, in the configuration example of the occupant state determination device 1 shown in FIG. can be configured without Further, the operation of step ST10 can be omitted from the operation of the occupant condition determination device 1 described using the flowchart of FIG. In this case, when the eye state determination unit 16 determines that the occupant 4 is in the closed eye state, the eye state information indicating that the occupant 4 is in the closed eye state is output to the output unit 19 .
  • the output unit 19 may output, for example, second warning information for warning about falling asleep. At that time, the output unit 19 may output the second warning information, for example, when the eye-closed state of the occupant 4 continues for the dozing determination time or longer. For example, when the occupant state determination device 1 is connected to a dozing determination device (not shown) that determines dozing of the occupant 4 based on the eye state information, the output unit 19 receives the output from the eye state determination unit 16. , the eye state information indicating that the occupant 4 is in a closed eye state may be output to the dozing determination device.
  • the occupant state determination device 1 may not include the inattentive determination unit 17, the drowsiness detection unit 15, and the dozing determination unit 18, and may not have both the function of determining inattentiveness and the function of determining dozing off.
  • the inattentive-looking determination unit 17 performs the inattentive-looking determination process when the eye state determination unit 16 determines that the occupant 4 is in the downcast state.
  • the degree of eye openness determined by the degree of eye openness determination unit 12 becomes less than a threshold smaller than the first threshold (hereinafter referred to as "second threshold"), the count of the time after determination of downcast eyes You can pause the upload. At this time, the inattentive-looking determination unit 17 does not clear the time after determination of the blinded eye.
  • the second threshold can be set as appropriate, and the second threshold may be set to a value smaller than the first threshold.
  • the inattentive-looking determination unit 17 temporarily stops counting up the time after determination of the blinded eye if the degree of eye opening of the occupant 4 is less than the second threshold in the inattentive-looking determination process.
  • the occupant state determination device 1 can prevent erroneous determination that the occupant 4 is looking aside when the occupant 4 closes his/her eyes in a motion similar to a downcast eye.
  • the inattentive determination unit 17 temporarily stops counting up the time after determination of downcast eyes and notifies the dozing determination unit 18 to that effect, and the dozing determination unit 18 counts up the time after determination of drowsiness. good too.
  • the occupant 4 whose state is to be determined by the occupant state determination device 1 is assumed to be the driver of the vehicle 30 .
  • the occupant state determination device 1 can also determine the state of the occupant 4 of the vehicle 30 other than the driver.
  • FIG. 9A and 9B are diagrams showing an example of the hardware configuration of the occupant state determination device 1 according to Embodiment 1.
  • FIG. 1 the image acquisition unit 11, the degree of eye opening determination unit 12, the line of sight detection unit 13, the face direction detection unit 14, the drowsiness detection unit 15, the eye state determination unit 16, and the inattentiveness determination unit 17 , the dozing determination unit 18 and the output unit 19 are implemented by the processing circuit 1001 . That is, the occupant state determination device 1 controls the determination of the state of the occupant 4 including the determination of whether the occupant 4 is in the downcast state or in the closed-eye state when the eye opening degree of the occupant 4 is less than the first threshold.
  • a processing circuit 1001 is provided for processing.
  • the processing circuitry 1001 may be dedicated hardware as shown in FIG. 9A or a processor 1004 executing a program stored in memory as shown in FIG. 9B.
  • the processing circuit 1001 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.
  • ASIC Application Specific Integrated Circuit
  • FPGA Field-Programmable Gate Array
  • the image acquisition unit 11, the degree of eye opening determination unit 12, the line of sight detection unit 13, the face direction detection unit 14, the drowsiness detection unit 15, the eye state determination unit 16, and the inattentiveness determination unit 17, the dozing determination unit 18, and the output unit 19 are implemented by software, firmware, or a combination of software and firmware.
  • Software or firmware is written as a program and stored in memory 1005 .
  • the processor 1004 reads out and executes the programs stored in the memory 1005, thereby controlling the image acquisition unit 11, the degree of eye openness determination unit 12, the line of sight detection unit 13, the face direction detection unit 14, and the drowsiness detection unit 15.
  • the occupant condition determination device 1 includes a memory 1005 for storing a program that, when executed by the processor 1004, results in the execution of steps ST1 to ST11 in FIG.
  • the programs stored in the memory 1005 include an image acquisition unit 11, an eye opening degree determination unit 12, a line of sight detection unit 13, a face direction detection unit 14, a drowsiness detection unit 15, an eye state determination unit 16, It can also be said that a computer is caused to execute the procedures or methods of the processes of the inattentive determination unit 17 , the dozing determination unit 18 , and the output unit 19 .
  • the memory 1005 is a non-volatile or volatile memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory).
  • RAM random access memory
  • ROM Read Only Memory
  • flash memory EPROM (Erasable Programmable Read Only Memory)
  • EEPROM Electrically Erasable Programmable Read-Only Memory
  • a semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versatile Disc), or the like is applicable.
  • Image acquisition unit 11, eye opening determination unit 12, line of sight detection unit 13, face direction detection unit 14, drowsiness detection unit 15, eye state determination unit 16, looking aside determination unit 17, dozing determination unit 18 and the output unit 19 may be partly realized by dedicated hardware and partly by software or firmware.
  • the function of the image acquisition unit 11 is realized by a processing circuit 1001 as dedicated hardware.
  • the functions of the eye state determination unit 16, the inattentive determination unit 17, the doze determination unit 18, and the output unit 19 can be realized by the processor 1004 reading out and executing a program stored in the memory 1005. be.
  • the occupant state determination device 1 also includes a device such as the imaging device 2 or an output device, and an input interface device 1002 and an output interface device 1003 that perform wired or wireless communication.
  • the occupant state determination device 1 is an in-vehicle device mounted in the vehicle 30, and includes the image acquisition unit 11, the eye opening degree determination unit 12, the line of sight detection unit 13, and the face direction detection unit 14. , the drowsiness detection unit 15 , the eye state determination unit 16 , the inattentiveness determination unit 17 , the doze determination unit 18 , and the output unit 19 are provided in the occupant state determination device 1 .
  • a portion of the doze determination unit 18 and the output unit 19 is installed in an in-vehicle device of the vehicle 30, and the others are provided in a server connected to the in-vehicle device via a network.
  • the state determination system 100 may be configured.
  • an image acquisition unit 11, an eye opening degree determination unit 12, a gaze detection unit 13, a face direction detection unit 14, a drowsiness detection unit 15, an eye state determination unit 16, an inattentiveness determination unit 17, and a dozing determination unit 18 and the output unit 19 may all be provided in the server.
  • the occupant state determination device 1 includes an image acquisition unit that acquires a captured image in which the face of the occupant 4 in the vehicle 30 is captured by the imaging device 2 provided in the vehicle 30. 11, an eye opening degree determination unit 12 that determines the degree of eye opening of the occupant 4 based on the captured image acquired by the image acquisition unit 11, and the line of sight of the occupant 4 is detected based on the captured image acquired by the image acquisition unit 11.
  • the degree of eye openness determined by the line-of-sight detection unit 13 and the degree of eye-openness determination unit 12 is less than the first threshold, the degree of eye-opening is changed after the line-of-sight direction of the occupant 4 detected by the line-of-sight detection unit 13 is directed downward.
  • the occupant state determination device 1 can determine whether the occupant 4 is in the closed-eyes state or the downcast state in consideration of the case where the occupant 4 directs the line of sight downward from the line of sight in the normal riding state. can.
  • the occupant state determination device can determine whether the occupant is in a closed-eyes state or a downcast state, taking into consideration the case where the occupant's line of sight is directed downward from the line of sight in the normal riding state. .

Landscapes

  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Business, Economics & Management (AREA)
  • Combustion & Propulsion (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Theoretical Computer Science (AREA)
  • Chemical & Material Sciences (AREA)
  • Emergency Management (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)
  • Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
PCT/JP2021/029705 2021-08-12 2021-08-12 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム Ceased WO2023017595A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/JP2021/029705 WO2023017595A1 (ja) 2021-08-12 2021-08-12 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム
DE112021007703.1T DE112021007703B4 (de) 2021-08-12 2021-08-12 Insassenzustandsbeurteilungsvorrichtung, verfahren zum beurteilen des zustands eines insassen und insassenzustandsbeurteilungssystem
JP2023541180A JP7378681B2 (ja) 2021-08-12 2021-08-12 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2021/029705 WO2023017595A1 (ja) 2021-08-12 2021-08-12 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム

Publications (1)

Publication Number Publication Date
WO2023017595A1 true WO2023017595A1 (ja) 2023-02-16

Family

ID=85200152

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/029705 Ceased WO2023017595A1 (ja) 2021-08-12 2021-08-12 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム

Country Status (3)

Country Link
JP (1) JP7378681B2 (https=)
DE (1) DE112021007703B4 (https=)
WO (1) WO2023017595A1 (https=)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197880A (zh) * 2023-03-09 2023-12-08 湖南工学院 专注度监测方法和系统
JPWO2024195072A1 (https=) * 2023-03-23 2024-09-26
WO2026044858A1 (zh) * 2024-08-28 2026-03-05 杭州锐见智行科技有限公司 一种疲劳驾驶检测方法、装置、电子设备及可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008204107A (ja) * 2007-02-19 2008-09-04 Toyota Motor Corp 不注意警告装置、不注意警告装置の車両機器制御方法及び車両制御装置のプログラム
JP2011048531A (ja) * 2009-08-26 2011-03-10 Aisin Seiki Co Ltd 眠気検出装置、眠気検出方法、及びプログラム
JP2016115118A (ja) * 2014-12-15 2016-06-23 アイシン精機株式会社 下方視判定装置および下方視判定方法
JP2020024532A (ja) * 2018-08-07 2020-02-13 オムロン株式会社 脇見運転検出装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6666892B2 (ja) * 2017-11-16 2020-03-18 株式会社Ｓｕｂａｒｕ 運転支援装置及び運転支援方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008204107A (ja) * 2007-02-19 2008-09-04 Toyota Motor Corp 不注意警告装置、不注意警告装置の車両機器制御方法及び車両制御装置のプログラム
JP2011048531A (ja) * 2009-08-26 2011-03-10 Aisin Seiki Co Ltd 眠気検出装置、眠気検出方法、及びプログラム
JP2016115118A (ja) * 2014-12-15 2016-06-23 アイシン精機株式会社 下方視判定装置および下方視判定方法
JP2020024532A (ja) * 2018-08-07 2020-02-13 オムロン株式会社 脇見運転検出装置

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117197880A (zh) * 2023-03-09 2023-12-08 湖南工学院 专注度监测方法和系统
JPWO2024195072A1 (https=) * 2023-03-23 2024-09-26
WO2024195072A1 (ja) * 2023-03-23 2024-09-26 三菱電機株式会社 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム
JP7696527B2 (ja) 2023-03-23 2025-06-20 三菱電機モビリティ株式会社 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム
WO2026044858A1 (zh) * 2024-08-28 2026-03-05 杭州锐见智行科技有限公司 一种疲劳驾驶检测方法、装置、电子设备及可读存储介质

Also Published As

Publication number Publication date
DE112021007703B4 (de) 2025-11-06
DE112021007703T5 (de) 2024-03-14
JPWO2023017595A1 (https=) 2023-02-16
JP7378681B2 (ja) 2023-11-13

Similar Documents

Publication Publication Date Title
JP4370915B2 (ja) 車載アプリケーション選択システム及び車載アプリケーション選択装置
US9117358B2 (en) Method for classification of eye closures
US7835834B2 (en) Method of mitigating driver distraction
EP2908726B1 (en) Method and device for detecting decreased attentiveness of vehicle driver
JP7378681B2 (ja) 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム
US12374129B2 (en) Occupant detection device, occupant detection system, and occupant detection method
CN108639055A (zh) 车辆的显示系统及车辆的显示系统的控制方法
US20160262682A1 (en) Driver monitoring apparatus
JP4840146B2 (ja) 眠気検知装置
CN110390285A (zh) 驾驶员分神检测方法、系统及车辆
JP2010013090A (ja) 運転者状態監視システム
JP6756174B2 (ja) 車両制御装置
JP6187155B2 (ja) 注視対象物推定装置
JP7550879B2 (ja) 睡眠検出装置及び睡眠検出システム
US11881054B2 (en) Device and method for determining image data of the eyes, eye positions and/or a viewing direction of a vehicle user in a vehicle
US11945461B2 (en) System for preventing drunk driving and method for controlling the same
JP2004133749A (ja) 顔の向き検知装置
JP7046748B2 (ja) 運転者状態判定装置および運転者状態判定方法
JPWO2023017595A5 (https=)
CN115965857B (zh) 一种环境光处理方法及dms系统
CN117002387A (zh) 一种车载流媒体后视镜的显示方法及装置
US20220272269A1 (en) Occupant monitoring device for vehicle
JP7787722B2 (ja) 車両の乗員監視装置
KR20230073790A (ko) 적응형 부주의 판단 기능을 이용한 운전자 자세제어 시스템 및 제어 방법
JP7843933B2 (ja) 眠気推定装置および眠気推定方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21953490

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023541180

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 112021007703

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21953490

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 112021007703

Country of ref document: DE