WO2025115202A1 - 乗員状態検知装置、乗員状態検知プログラム及び乗員状態検知方法 - Google Patents

乗員状態検知装置、乗員状態検知プログラム及び乗員状態検知方法 Download PDF

Info

Publication number
WO2025115202A1
WO2025115202A1 PCT/JP2023/043028 JP2023043028W WO2025115202A1 WO 2025115202 A1 WO2025115202 A1 WO 2025115202A1 JP 2023043028 W JP2023043028 W JP 2023043028W WO 2025115202 A1 WO2025115202 A1 WO 2025115202A1
Authority
WO
WIPO (PCT)
Prior art keywords
occupant
alertness
unit
image information
driver
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
PCT/JP2023/043028
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
和樹 國廣
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Mobility Corp
Original Assignee
Mitsubishi Electric Mobility Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Mobility Corp filed Critical Mitsubishi Electric Mobility Corp
Priority to PCT/JP2023/043028 priority Critical patent/WO2025115202A1/ja
Priority to JP2025560498A priority patent/JPWO2025115202A1/ja
Publication of WO2025115202A1 publication Critical patent/WO2025115202A1/ja
Anticipated expiration legal-status Critical
Pending legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/0818—Inactivity or incapacity of driver
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/0818—Inactivity or incapacity of driver
    • B60W2040/0827—Inactivity or incapacity of driver due to sleepiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/225—Direction of gaze
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/229—Attention level, e.g. attentive to driving, reading or sleeping

Definitions

  • This disclosure relates to an occupant status detection device, an occupant status detection program, and an occupant status detection method.
  • an occupant status monitoring device that detects the line of sight of an occupant in a vehicle from an image obtained by photographing the eyeballs of the occupant, and determines whether the occupant is looking away based on the detected line of sight (see, for example, Patent Document 1).
  • the present disclosure aims to solve the above problem and provide an occupant status detection device, an occupant status detection program, and an occupant status detection method that can detect an occupant's inattentiveness even when the occupant's line of sight cannot be detected.
  • the occupant status detection device is characterized by including an image information acquisition unit that acquires image information obtained by imaging the interior of the vehicle cabin, an eyelid opening detection unit that detects the degree to which the eyelids of the occupant in the vehicle cabin are open based on the image information acquired by the image information acquisition unit, an alertness estimation unit that estimates the alertness of the occupant, and an inattentive detection unit that detects whether the occupant is looking away based on the eyelid opening degree of the occupant detected by the eyelid opening detection unit and the alertness of the occupant estimated by the alertness estimation unit.
  • the driver's inattentiveness is detected based on the degree to which the driver's eyelids are open and the driver's level of alertness, so that the driver's inattentiveness can be detected even when the driver's line of sight cannot be detected.
  • FIG. 1 is a side view showing a vehicle according to a first embodiment.
  • 1 is a block diagram showing a schematic configuration of an occupant condition detection device according to a first embodiment
  • 2 is a block diagram showing an example of a hardware configuration of the occupant condition detection device according to the first embodiment
  • FIG. 2 is a block diagram showing an example of a hardware configuration of the occupant condition detection device according to the first embodiment
  • FIG. 4 is a flowchart showing a process performed by the occupant state detection device according to the first embodiment.
  • FIG. 11 is a block diagram showing a schematic configuration of an occupant condition detection device according to a second embodiment.
  • 10 is a flowchart showing a process performed by an occupant condition detection device according to a second embodiment;
  • FIG. 11 is a block diagram showing a schematic configuration of an occupant condition detection device according to a third embodiment.
  • 13 is a flowchart showing a process performed by an occupant condition detection device according to a third embodiment.
  • FIG. 13 is a block diagram showing a schematic configuration of an occupant condition detection device according to a fourth embodiment.
  • 13 is a flowchart showing a process performed by an occupant state detection device according to a fourth embodiment;
  • FIG. 13 is a block diagram showing a schematic configuration of an occupant condition detection device according to a fifth embodiment.
  • 13 is a flowchart showing a process performed by an occupant state detection device according to a fifth embodiment.
  • Fig. 1 is a side view showing the vehicle 1 according to the first embodiment.
  • the vehicle 1 includes a seat 3 arranged in a vehicle compartment R defined by a main body of the vehicle 1, an imaging device 110 that images the interior of the vehicle compartment R, an occupant state detection device 100 that detects the state of an occupant J2, and an alarm device that issues an alarm based on an input signal from the occupant state detection device 100.
  • the vehicle 1 includes a plurality of seats 3 including a driver's seat 2 in which a driver J1 of the vehicle 1 as an occupant J2 is seated.
  • the direction opposite to the Z direction shown in FIG. 1, which is the direction in which the vehicle 1 moves when the steering wheel (not shown) of the vehicle 1 is in the neutral position and the gear shift position of the transmission (not shown) of the vehicle 1 is in the forward position, is defined as the front of the vehicle, and the front, rear, left and right of the vehicle are defined based on this. Also, in the first embodiment, the front of the vehicle is simply referred to as the "forward".
  • the imaging device 110 captures an image of the interior of the vehicle compartment R while the occupant J2 is seated in the seat 3, thereby acquiring image information of the interior of the vehicle compartment R.
  • the imaging device 110 has an imaging element (image sensor) and a lens (not shown), and converts light captured from the lens into a signal by the imaging element.
  • the imaging element is, for example, a solid-state imaging element such as a CCD (CHARGE COUPLED DEVICE) image sensor or a CMOS (COMPLEMENTARY METAL OXIDE SEMICONDUCTOR) image sensor.
  • the imaging device 110 captures an image of an area including the face of the passenger J2 seated in one of the seats 3 in the vehicle interior R.
  • the imaging device 110 is disposed in a position where it can capture an image of the driver J1 seated in the driver's seat 2 with his/her face facing forward and his/her eyelids open, and captures an image of an area including at least the face of the driver J1.
  • the imaging device 110 is disposed in front of the driver J1 seated in the driver's seat 2 and slightly below the face of the driver J1, or in front of the driver J1 seated in the driver's seat 2 and above the face of the driver J1, and captures an image of an area including the face of the driver J1.
  • the imaging device 110 is disposed on the dashboard or overhead console, and captures an image of an area including the face of the driver J1. Also, for example, the imaging device 110 captures an image of the vehicle interior R at a preset frame rate. The imaging device 110 converts the image information obtained by imaging into an electrical signal and outputs it.
  • the notification device 120 notifies information related to the detection result by the occupant status detection device 100 based on an input signal from the occupant status detection device 100.
  • the notification device 120 is configured to have an audio output device such as a speaker that notifies information by outputting sound based on an input signal from the occupant status detection device 100, a display device such as a liquid crystal monitor that notifies information by displaying an image based on an input signal from the occupant status detection device 100, and a light-emitting device such as an LED that notifies information by changing the light-emitting mode based on an input signal from the occupant status detection device 100.
  • the notification device 120 when the occupant status detection device 100 detects that occupant J2 is looking aside, the notification device 120 notifies a message to urge occupant J2 not to look aside based on an input signal from the occupant status detection device 100 indicating that occupant J2 is looking aside. Also, for example, the notification device 120 calculates the length of continuous inattentive time, which is the time during which the occupant J2 continues to look inattentively, based on an input signal from the occupant state detection device 100 indicating that the occupant J2 is looking inattentively, and if the calculated length of continuous inattentive time is equal to or greater than a preset threshold, the notification device 120 issues a message to urge the occupant J2 not to look inattentively.
  • the notification device 120 calculates the length of continuous inattentive time, which is the time during which the occupant J2 continues to look inattentively, based on an input signal from the occupant state detection device 100 indicating that the occupant J2 is looking inattentively, and if the
  • the occupant status detection device 100 detects the status of the occupant J2 in the vehicle compartment R based on an input signal from the imaging device 110. For example, based on an input signal from the imaging device 110, the occupant status detection device 100 detects the driver J2 looking aside as the status of the occupant J2 in the vehicle compartment R. Specifically, based on an input signal from the imaging device 110, the occupant status detection device 100 detects the driver J1 looking aside as the status of the driver J1 seated in the driver's seat 2 in the vehicle compartment R. In other words, the occupant status detection device 100 detects the driver J1 looking aside based on image information obtained by imaging the interior of the vehicle compartment R by the imaging device 110.
  • the occupant status detection device 100 will be used to detect the status of the driver J1 as the occupant J2, but the target for which the occupant status detection device 100 detects the status is not limited to the driver J1, but may be any occupant J2 aboard the vehicle 1.
  • the occupant state detection device 100 outputs the detection result to the notification device 120.
  • "looking aside” refers to a state in which the occupant J2 is looking at anything other than the surroundings of the road ahead of the vehicle 1.
  • FIG. 2 is a block diagram showing a schematic configuration of an occupant state detection device 100 according to the first embodiment.
  • the occupant state detection device 100 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an alertness estimation unit 16, and an inattentive detection unit 19, and is electrically connected to the image capture device 110 and the notification device 120 so as to be able to communicate with them.
  • the input unit 11 accepts input signals from the image capture device 110 and each device of the vehicle 1 (not shown).
  • the output unit 12 outputs a signal to the notification device 120 according to the result of processing by the occupant state detection device 100.
  • the image information acquisition unit 13 acquires image information obtained by imaging the inside of the vehicle cabin R based on an input signal from the imaging device 110. For example, the image information acquisition unit 13 acquires image information including an image of the face of the driver J1 obtained by imaging the inside of the vehicle cabin R. Specifically, the image information acquisition unit 13 acquires image information including an image of the face of the driver J1 obtained by imaging the inside of the vehicle cabin R.
  • the eyelid opening detection unit 14 detects the degree of eyelid opening of the driver J1 in the vehicle compartment R based on the image information acquired by the image information acquisition unit 13. For example, the eyelid opening detection unit 14 detects the degree of eyelid opening of the driver J1 in the vehicle compartment R based on the image information acquired by the image information acquisition unit 13. Also, for example, the eyelid opening detection unit 14 detects the upper eyelid and lower eyelid of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a known image recognition technology.
  • the eyelid opening detection unit 14 extracts feature points indicating the upper eyelid and the lower eyelid of the driver J1 present in the face detection area of the image information by using a known edge detection technology in a state where a face detection area, which is an area where the face of the driver J1 may be present, is set in advance in the image information acquired by the image information acquisition unit 13.
  • the eyelid opening degree detection unit 14 calculates, for example, the maximum eyelid distance, which is the distance between the upper and lower eyelids of the occupant in the image information.
  • the eyelid opening degree detection unit 14 detects the eyelid opening degree by calculating, for example, the ratio of the maximum eyelid distance to a preset reference value.
  • a reference value for the eyelid opening degree is a value that is set based on the average eyelid distance of an adult of average build when awake.
  • the reference value of the degree of eyelid opening is not limited to a value set based on the average eyelid distance of an adult of a general physique when awake.
  • the reference value of the degree of eyelid opening may be any value that can determine the degree of eyelid opening of the driver J1 by comparing it with the eyelid distance acquired by the eyelid opening detection unit 14, and may be, for example, a value set based on the average eyelid distance during a preset period in which the driver J1 in the vehicle 1 is awake and the driver J1 is facing forward, or may be a value set based on the eyelid distance of a passenger who has previously ridden in the vehicle 1.
  • the degree of eyelid opening is not limited to the ratio of the maximum eyelid distance to a preset reference value.
  • the degree of eyelid opening may be any value that quantifies the degree of eyelid opening of the passenger, and may be, for example, the maximum eyelid distance itself described above, the ratio of the eyelid distance to the maximum distance between the left and right eyelids, or the area of the eyeball shown in the image information.
  • the alertness estimation unit 16 estimates the alertness of the driver J1 based on various information. Specifically, the alertness estimation unit 16 estimates the alertness of the driver J1 based on the image information acquired by the image information acquisition unit 13. For example, the alertness estimation unit 16 estimates the alertness of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a learned model that learns using learning data that associates image information obtained by capturing an image of at least the occupant's face with the alertness, and outputs the alertness of the occupant when image information including an image of the occupant's face is input.
  • the alertness estimation unit 16 is configured to estimate the alertness of the driver J1 based on a learned model that learns using image information acquired by the image information acquisition unit 13 and learning data that associates image information obtained by capturing an image of a range including the occupant's face and all or a part of the upper body other than the face with the alertness, and outputs the alertness of the occupant when image information including an image of the occupant's face and all or a part of the upper body other than the face is input.
  • the alertness estimation unit 16 estimates the alertness of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a known image recognition technique. Specifically, the alertness estimation unit 16 uses a known edge detection technique in a state where a detection area in which the driver J1 may be present is set in advance in the image information acquired by the image information acquisition unit 13, to extract characteristic points of each part of the body of the driver J1 that is present in the detection area of the image information. Next, the alertness estimation unit 16 estimates the alertness of the driver J1 based on the extracted characteristic points of each part of the body of the driver J1.
  • the alertness estimation unit 16 estimates the alertness by detecting the bodily movements of the driver J1 attempting to resist drowsiness based on the extracted characteristic points of each part of the driver J1. For example, the alertness estimation unit 16 estimates the alertness based on an increase or decrease in the bodily movements of the driver J1 attempting to resist drowsiness. In addition, for example, the alertness estimation unit 16 estimates the alertness by detecting the frequency of characteristic movements of the driver J1, such as stretching, slapping the body, touching the face or head, etc. In general, when a person tries to resist drowsiness, bodily movements increase and characteristic movements such as stretching, slapping the body, touching the face or head, etc. occur.
  • the alertness estimation unit 16 is configured to estimate the alertness of the occupant based on factors other than at least the degree of eyelid opening of the occupant. In other words, it is preferable that the alertness estimation unit 16 is configured so that the estimation result of the alertness of the occupant is not linked to the detection result of the eyelid opening of the occupant by the eyelid opening detection unit 14. It is preferable that the alertness estimation unit 16 is configured to estimate the alertness of the occupant based on factors other than at least the degree of eyelid opening of the occupant does not mean that the eyelid opening degree has no influence on the estimation result of the alertness of the occupant, and the eyelid opening degree of the occupant may be included as one of the factors when estimating the alertness of the occupant.
  • the alertness estimation unit 16 estimates the alertness by detecting body movements that indicate that the driver J1 is drowsy, based on the extracted feature points of each part of the driver J1. For example, the alertness estimation unit 16 estimates the alertness by detecting the mouth opening degree of the driver J1. For example, the alertness estimation unit 16 estimates the alertness by detecting the frequency of yawning by the driver J1. For example, the alertness estimation unit 16 estimates the alertness by detecting the frequency of blinking by the driver J1. For example, the alertness estimation unit 16 estimates the alertness by detecting a change in the line of sight of the driver J1. In general, when a person is drowsy, the mouth opening degree, frequency of yawning, and frequency of blinking increase, and changes in the line of sight decrease.
  • the inattentiveness detection unit 19 detects inattentiveness of the driver J1 based on the degree of eyelid opening of the driver J1 detected by the eyelid opening degree detection unit 14 and the degree of alertness of the driver J1 estimated by the alertness degree estimation unit 16. For example, the inattentiveness detection unit 19 compares the degree of eyelid opening of the driver J1 detected by the eyelid opening degree detection unit 14 with a preset threshold value for eyelid opening, and compares the level of alertness of the driver J1 estimated by the alertness degree estimation unit 16 with a preset threshold value for alertness, and detects inattentiveness of the driver J1 based on the results of these comparisons. Details of the processing performed by the inattentiveness detection unit 19 will be described later.
  • Figure 3 is a diagram showing an example of the hardware configuration of the occupant status detection device 100
  • Figure 4 is a diagram showing an example of the hardware configuration of the occupant status detection device 100 that is different from that of Figure 3.
  • the occupant status detection device 100 is configured as a computer having a processor 100a, a memory 100b, and an I/O port 100c, and is configured so that the processor 100a reads and executes a program stored in the memory 100b.
  • the occupant status detection device 100 is configured as a computer having a processing circuit 100d, which is dedicated hardware, and an I/O port 100c.
  • the processing circuit 100d is configured, for example, by a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these.
  • Each function of the occupant status detection device 100 is realized by the processor 100a or the processing circuit 100d, which is dedicated hardware, executing a program.
  • the occupant status detection device 100 may have hardware other than the above, such as a hardware timer used for measuring time.
  • FIG. 5 is a flowchart showing the processing performed by the occupant status detection device 100.
  • the occupant status detection device 100 when the occupant status detection device 100 starts processing, it first acquires image information obtained by imaging the interior of the vehicle compartment R (step ST1). In this processing, the occupant status detection device 100 acquires image information obtained by the imaging device 110 imaging the interior of the vehicle compartment R using the image information acquisition unit 13.
  • the occupant status detection device 100 After performing the processing of step ST1, the occupant status detection device 100 next detects the degree to which the eyelids of the driver J1 are open (step ST2). In this processing, the occupant status detection device 100 detects the degree to which the eyelids of the driver J1 are open using the eyelid opening detection unit 14 based on the image information acquired by the image information acquisition unit 13.
  • the occupant status detection device 100 After performing the processing of step ST2, the occupant status detection device 100 next estimates the alertness of the driver J1 (step ST3). In this processing, the occupant status detection device 100 estimates the alertness of the driver J1 by the alertness estimation unit 16 based on, for example, the image information acquired by the image information acquisition unit 13.
  • the occupant status detection device 100 next judges whether the degree of eyelid opening of the driver J1 is equal to or greater than a preset threshold (step ST6).
  • a preset threshold In general, when detecting the degree of eyelid opening using image recognition technology based on image information obtained when the face of an occupant is photographed, the degree of eyelid opening tends to be detected as lower in a state in which the occupant's face is looking aside and is facing downward rather than forward, such as when the occupant is operating a device arranged on the dashboard of the vehicle or when the occupant is operating a mobile terminal, than when the occupant's face is looking forward.
  • the occupant status detection device 100 compares the degree of eyelid opening of the driver J1 with a threshold value of the degree of eyelid opening that is considered to be highly likely to indicate that the driver J1 is facing forward and is awake, and judges by the inattentive detection unit 19 whether the degree of eyelid opening of the driver J1 is sufficient to indicate that the driver J1 is not looking aside.
  • step ST9 the occupant status detection device 100 judges whether driver J1 is in a state of falling asleep by using the inattentive detection unit 19, for example, by comparing the alertness of driver J1 with an alertness threshold that is considered sufficient to determine that driver J1 is not falling asleep.
  • step ST9 if the degree of alertness of the driver J1 is equal to or higher than the threshold (YES in step ST9), the occupant state detection device 100 next detects that the driver J1 is looking aside (step ST12). In other words, based on the fact that the degree of alertness of the driver J1 is equal to or higher than the threshold in the process of step ST9, the occupant state detection device 100 determines that the driver J1 is looking aside. As described above, in a state in which the occupant's face is facing downward rather than forward, the degree of eyelid opening tends to be detected as a lower value than in a state in which the occupant's face is facing forward.
  • the occupant state detection device 100 detects that the driver J1 is not dozing but is looking aside with his/her head facing downward rather than forward, based on the fact that the degree of alertness is equal to or greater than a threshold when the degree of eyelid opening is lower than a threshold.
  • the occupant status detection device 100 After performing the processing of step ST12, the occupant status detection device 100 then outputs the detection result of driver J1's inattentiveness to the notification device 120 (step ST13). If the degree of eyelid opening of driver J1 is equal to or greater than the threshold value in the processing of step ST6 (YES in step ST6), if the degree of alertness of driver J1 is less than the threshold value in the processing of step ST9 (NO in step ST9), or if the processing of step ST13 is performed, the occupant status detection device 100 ends the processing. Note that the occupant status detection device 100 may be configured to output a detection result indicating that driver J1's inattentiveness was not detected to the notification device 120 when the occupant status detection device 100 does not detect driver J1's inattentiveness.
  • the occupant status detection device 100 is characterized by including an image information acquisition unit 13 that acquires image information obtained by imaging the interior of the vehicle compartment R, an eyelid opening detection unit 14 that detects the degree of eyelid opening of the driver J1 in the vehicle compartment R based on the image information acquired by the image information acquisition unit 13, an alertness estimation unit 16 that estimates the alertness of the driver J1, and an inattentive detection unit 19 that detects whether the driver J1 is looking away based on the eyelid opening degree of the driver J1 detected by the eyelid opening detection unit 14 and the alertness of the driver J1 estimated by the alertness estimation unit 16.
  • the occupant status detection device 100 is configured to detect the driver J1's inattentiveness based on the degree of eyelid opening of the driver J1 and the degree of alertness of the driver J1, and therefore can detect the driver J1's inattentiveness even when the driver J1's gaze cannot be detected.
  • the occupant status detection device 100 according to the first embodiment detects the driver J1's inattentiveness and causes the notification device 120 to notify the driver J1 of a message to urge the driver J1 not to inattentively, thereby making it possible to suppress the driver J1 from inattentive driving of the vehicle 1.
  • the degree of eyelid opening when detecting the degree of eyelid opening using image recognition technology based on the image information obtained when capturing an image of an occupant's face, when the occupant's face is facing downward rather than forward and looking aside, the degree of eyelid opening tends to be detected as a lower value than when the occupant's face is facing forward. For this reason, when judging the occupant's state based only on the degree of eyelid opening, it is difficult to determine whether the occupant is facing downward rather than forward and looking aside, or whether the occupant has their eyes closed and is dozing.
  • the occupant state detection device 100 is configured to detect inattentive behavior based on the occupant's level of alertness in addition to the degree to which the occupant's eyes are open, and therefore can improve the accuracy of inattentive behavior detection compared to, for example, detecting inattentive behavior based only on the degree to which the occupant's eyes are open.
  • the occupant state detection device 100 according to the first embodiment is configured to estimate the alertness of the occupant based on the image information acquired by the image information acquisition unit 13. Configured in this manner, the occupant state detection device 100 according to the first embodiment does not require a sensor for estimating the alertness and a communication path connecting the sensor and the occupant state detection device, making it possible to simplify the device.
  • the occupant state detection device 100 is configured to detect an occupant looking away when the degree of eyelid opening of the occupant detected by the eyelid opening detection unit 14 is lower than a preset eyelid opening threshold, and when the degree of alertness of the occupant estimated by the alertness estimation unit 16 is equal to or higher than a preset alertness threshold.
  • the occupant state detection device 100 according to the first embodiment is capable of distinguishing between a state in which the occupant is dozing and a state in which the occupant is looking away when, for example, the degree of eyelid opening of the occupant is lower than a threshold, and is able to improve the detection accuracy when detecting an occupant looking away compared to the conventional technology.
  • the alertness estimation unit 16 is configured to estimate the alertness of the driver J1 based on the image information acquired by the image information acquisition unit 13, but is not limited thereto.
  • the alertness estimation unit may be configured to estimate the alertness of any of the occupants J2 using at least information other than the eyelid opening degree.
  • the alertness estimation unit may be configured to estimate the alertness of the occupant J2 based on information acquired by a sensor (not shown) that detects the operation of the vehicle.
  • the alertness estimation unit may be configured to estimate the alertness of the driver by detecting the frequency of sudden deceleration and sudden acceleration of the vehicle based on information from a speed sensor that detects the speed of the vehicle or information from an acceleration sensor (not shown) that detects the acceleration of the vehicle. Also, for example, the alertness estimation unit may be configured to estimate the alertness of the driver by detecting the meandering of the vehicle based on information from a sensor (not shown) that detects the operation of the steering wheel of the vehicle.
  • the alertness estimation unit may be configured to estimate the alertness of the driver by detecting the frequency with which the vehicle crosses a lane marking based on information from a sensor (not shown) that detects the lane markings on the road on which the vehicle is traveling.
  • the occupant state detection device 100 is configured to detect the occupant J2 looking aside when the eyelid opening degree of the occupant J2 detected by the eyelid opening degree detection unit 14 is lower than a preset threshold value and when the alertness of the occupant J2 estimated by the alertness estimation unit 16 is equal to or higher than a preset threshold value, but is not limited to this.
  • the occupant state detection device may be configured to detect the occupant looking aside based on the eyelid opening degree of the occupant detected by the eyelid opening degree detection unit and the alertness of the occupant estimated by the alertness estimation unit, and for example, the occupant state detection device may be configured such that the inattentive detection unit detects the occupant looking aside based on an index other than the eyelid opening degree calculated based on the eyelid opening degree of the occupant detected by the eyelid opening degree detection unit and the alertness of the occupant estimated by the alertness estimation unit.
  • the occupant state detection device may be configured such that the inattentive detection unit detects the occupant's inattentive behavior based on the occupant's eyelid closing time calculated based on the degree of eyelid opening of the occupant detected by the eyelid opening detection unit and the occupant's alertness estimated by the alertness estimation unit.
  • the occupant state detection device may also be configured to calculate the frequency at which the occupant's eyelid opening degree falls below a threshold based on the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit, and detect the occupant's looking aside based on the result of comparing the frequency at which the occupant's eyelid opening degree falls below the threshold with a preset frequency threshold and the occupant's alertness estimated by the alertness degree estimation unit.
  • the occupant's eyelid opening degree falls below the threshold when the alertness degree is equal to or above the threshold, it is possible that the occupant is either blinking or looking aside, but if the frequency at which the occupant's eyelid opening degree falls below the threshold is equal to or above the threshold, it can be determined that the occupant is frequently looking aside.
  • the occupant state detection device may also be configured to estimate the occupant's gaze direction based on the image information acquired by the image information acquisition unit when the degree of eyelid opening of the occupant detected by the eyelid opening detection unit is equal to or greater than a threshold, and to detect the occupant's looking aside based on the occupant's gaze direction and the occupant's alertness estimated by the alertness estimation unit. For example, if the angle between the occupant's gaze direction and the forward direction is equal to or greater than a preset threshold angle and the occupant's alertness is equal to or greater than a threshold, it can be determined that the occupant is looking aside.
  • the occupant state detection device 100 is configured to estimate the degree of wakefulness of the driver J1, but is not limited to this.
  • the occupant state detection device may be configured to detect the state of at least one of the occupants J2 looking away from the vehicle.
  • the occupant state detection device may be configured to detect the looking away of an occupant other than the driver, or may be configured to detect an occupant state other than looking away from the vehicle.
  • an occupant state other than looking away from the vehicle may be an occupant falling asleep.
  • a monitor located forward of the seat in which the occupant is seated, and if the occupant is not gazing at a monitor, it is possible to reduce the power consumption of the monitor by turning off the monitor or lowering the volume of the monitor.
  • FIG. 6 is a block diagram showing a schematic configuration of the occupant state detection device 200 according to the second embodiment.
  • the occupant state detection device 200 according to the second embodiment is different from the occupant state detection device 100 according to the first embodiment in a configuration in which an inattentive detection unit 219 detects an inattentive state of an occupant based on a period of time the occupant has his/her eyes closed.
  • the other configurations are the same, and the same configurations as those in the first embodiment are given the same names and symbols as those in the first embodiment, and description thereof will be omitted.
  • the occupant status detection device 200 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, and an inattentive detection unit 219, and is electrically connected to the imaging device 110 and the alarm device 120 so as to be able to communicate with them.
  • the eyelid closing time determination unit 15 determines whether the length of the eyelid closing time of the driver J1 is equal to or greater than a preset threshold value based on the degree of eyelid opening of the driver J1 detected by the eyelid opening degree detection unit 14. For example, the eyelid closing time determination unit 15 calculates the length of continuous eyelid closing time, which is the time during which the degree of eyelid closure of the driver J1 is continuously maintained below a preset threshold value, based on the degree of eyelid opening of the driver J1 detected by the eyelid opening degree detection unit 14, and determines whether the calculated length of continuous eyelid closing time is equal to or greater than a preset threshold value.
  • the inattentiveness detection unit 219 detects inattentiveness of the driver J1 based on the judgment result by the eyelid closed time judgment unit 15 and the alertness of the driver J1 estimated by the alertness estimation unit 16. For example, the inattentiveness detection unit 219 detects inattentiveness of the driver J1 based on the judgment result by the eyelid closed time judgment unit 15 and the comparison result between the alertness of the driver J1 estimated by the alertness estimation unit 16 and a preset alertness threshold value. Details of the processing performed by the inattentiveness detection unit 219 will be described later.
  • FIG. 7 is a flowchart showing the processing performed by the occupant status detection device 200 according to embodiment 2. Note that the processing of steps ST1, ST2, ST3, ST9, and ST13 performed by the occupant status detection device 200 according to embodiment 2 is similar to the processing performed by the occupant status detection device 100 according to embodiment 1, and therefore a description thereof will be omitted.
  • the occupant status detection device 200 After performing the process of step ST3, the occupant status detection device 200 next judges whether the eyelid closed time of the driver J1 is equal to or greater than a preset eyelid open time threshold (step ST7).
  • the occupant status detection device 200 compares the continuous eyelid closed time of the driver J1 with a continuous eyelid closed time threshold that is considered to be highly likely to indicate that the driver J1 is facing forward and is awake, and uses the inattentive detection unit 219 to judge whether the length of the continuous eyelid closed time calculated based on the degree of eyelid opening of the driver J1 is long enough to indicate that the driver J1 is not looking aside.
  • step ST7 determines whether the eyelid-closed time of driver J1 is equal to or greater than a preset threshold value (step ST9). If the eyelid-closed time of driver J1 is less than the threshold value in the process of step ST7 (NO in step ST7), if the alertness of driver J1 is less than the threshold value in the process of step ST9 (NO in step ST9), or if the process of step ST13 is performed, the occupant status detection device 200 ends the process.
  • the occupant state detection device 200 includes an eyelid closing time determination unit 15 that determines whether the length of the occupant's continuous eyelid closing time is equal to or longer than a preset eyelid closing time threshold based on the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit 14, and the inattentive detection unit 319 is configured to detect the occupant's inattentiveness based on the determination result by the eyelid closing time determination unit 15 and the occupant's alertness estimated by the alertness estimation unit 16.
  • the occupant state detection device 200 can suppress erroneous detection of inattentiveness due to the occupant's blinking, and suppress overdetection in which a short-term movement of the eyes that does not affect driving is detected as inattentiveness, thereby improving the detection accuracy when detecting inattentiveness of the occupant compared to the conventional method.
  • the eyelid closing time determination unit 15 is configured to determine whether the length of the continuous eyelid closing time of the driver J1 is equal to or greater than a preset threshold value based on the degree of eyelid opening of the driver J1 detected by the eyelid opening degree detection unit 14, but is not limited to this.
  • the eyelid closing time determination unit may be configured to determine whether the length of the occupant's eyelid closing time is equal to or greater than a preset threshold value based on the degree of eyelid opening of the driver J1 detected by the eyelid opening degree detection unit, and for example, the eyelid closing time determination unit may be configured to calculate the total time during a specific period during which the degree of eyelid closing of the occupant is less than a preset threshold value based on the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit, and to determine whether the proportion of the total time during the specific period is equal to or greater than a preset threshold value.
  • the eyelid closing time determination unit may be configured to ignore opening and closing of the eyelids for a period of time less than a preset threshold when calculating the length of time the occupant has the eyelids continuously closed based on the degree of eyelid opening of the occupant detected by the eyelid opening degree detection unit.
  • FIG. 8 is a block diagram showing a schematic configuration of the occupant state detection device 300 according to the third embodiment.
  • the occupant state detection device 300 according to the third embodiment differs from the occupant state detection device 200 according to the second embodiment in the configuration in which the inattentive detection unit 319 detects an occupant's inattentiveness based on the facial direction of the occupant.
  • the other configurations are the same, and the same configurations as those in the second embodiment will be given the same names and symbols as those in the second embodiment, and description thereof will be omitted.
  • the occupant status detection device 300 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, a face direction estimation unit 17, and an aside-looking detection unit 319, and is electrically connected to the imaging device 110 and the alarm device 120 so as to be capable of communicating with them.
  • the face direction estimation unit 17 estimates the face direction of the driver J1 based on the image information acquired by the image information acquisition unit 13. For example, the face direction estimation unit 17 estimates the angle between the face direction of the driver J1 and the front based on the image information acquired by the image information acquisition unit 13 and a learned model that learns using learning data that associates image information obtained by imaging at least a range including the face of the occupant with the face direction of the occupant, and outputs the angle of the face direction of the occupant relative to the front when image information including an image of the occupant's face is input. Also, for example, the face direction estimation unit 17 estimates the face direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a known image recognition technology.
  • the face direction estimation unit 17 extracts the feature points of the face of the driver J1 that are present in the detection area of the image information acquired by the image information acquisition unit 13 by using a known edge detection technology in a state in which a detection area, which is an area where the face of the driver J1 may be present, is set in advance in the image information acquired by the image information acquisition unit 13.
  • the face direction estimation unit 17 estimates the face direction of the driver J1 based on the extracted feature points of each part of the face of the driver J1.
  • the inattentive detection unit 319 detects inattentive behavior of the driver J1 based on the judgment result by the eyelid closed time judgment unit 15, the alertness of the driver J1 estimated by the alertness estimation unit 16, and the facial direction of the driver J1 estimated by the facial direction estimation unit 17.
  • the inattentive behavior detection unit 219 detects inattentive behavior of the driver J1 based on the judgment result by the eyelid closed time judgment unit 15, the comparison result between the alertness of the driver J1 estimated by the alertness estimation unit 16 and a preset alertness threshold value, and the judgment result of whether the facial direction of the driver J1 estimated by the facial direction estimation unit 17 is within a preset range. Details of the processing performed by the inattentive behavior detection unit 319 will be described later.
  • FIG. 9 is a flowchart showing the processing performed by the occupant status detection device 300 according to embodiment 3. Note that the processing of steps ST1, ST2, ST3, ST7, ST9, and ST13 performed by the occupant status detection device 300 according to embodiment 3 is similar to the processing performed by the occupant status detection device 200 according to embodiment 2, and therefore a description thereof will be omitted.
  • the occupant status detection device 300 After performing the processing of step ST3, the occupant status detection device 300 next estimates the facial direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 (step ST4). In this processing, the occupant status detection device 300 estimates the facial direction of the driver J1 by the facial direction estimation unit 17 based on the image information acquired by the image information acquisition unit 13.
  • the occupant status detection device 300 next determines whether the eyelid closed time of driver J1 is equal to or greater than a preset eyelid open time threshold (step ST7). In the processing of step ST7, if the eyelid closed time of driver J1 is equal to or greater than the preset threshold (YES in step ST7), the occupant status detection device 300 next determines whether the alertness of driver J1 is equal to or greater than a preset threshold (step ST9). In the processing of step ST9, if the alertness of driver J1 is equal to or greater than the threshold (YES in step ST9), the occupant status detection device 300 next detects that driver J1 is looking away (step ST32).
  • step ST9 if the degree of wakefulness of the driver J1 is less than the threshold (NO in step ST9), the occupant state detection device 300 next judges whether the face direction of the driver J1 estimated by the face direction estimation unit 17 is lower than a preset range (step ST10). For example, in this process, the occupant state detection device 300 compares the angle between the face direction of the driver J1 estimated by the face direction estimation unit 17 and the front with a preset angle threshold, and judges whether the angle between the face direction of the driver J1 and the front is equal to or greater than the threshold by the inattentive detection unit 319.
  • the angle threshold is set as the angle between the face direction of the driver who is considered to be highly likely to be not looking aside and to be in an awake state and the front.
  • the angle threshold is set as the angle between the face direction of the occupant detected by the eyelid opening detection unit 14 as an eyelid opening degree of a threshold or greater for an occupant who is not looking aside and is in an awake state.
  • step ST10 if the driver J1's face direction is not more downward than the preset range (NO in step ST10), the occupant status detection device 300 then detects that the driver J1 is looking aside (step ST32). In this process, the occupant status detection device 300 determines, through the inattentive detection unit 319, that the driver J1 is not at least looking forward with his eyes open, based on the fact that the eyelids are closed for a period of time that is more than a threshold, the driver J1's alertness is less than a threshold, and the driver J1's face direction is not more downward than the preset range. After performing the process of step ST32, the occupant status detection device 300 then outputs the detection result of the driver J1's looking aside to the alarm device 120 (step ST13).
  • step ST7 If the time that the driver J1's eyes are closed is less than the threshold value in the process of step ST7 (NO in step ST7), if the driver J1's face is facing downward beyond the preset range in the process of step ST10 (YES in step ST10), or if the process of step ST13 is performed, the occupant status detection device 300 ends the process.
  • the occupant state detection device 300 includes a face direction estimation unit 17 that estimates the facial direction of the occupant based on the image information acquired by the image information acquisition unit 13, and the inattentive detection unit 319 is configured to detect the occupant's inattentiveness based on the facial direction of the occupant estimated by the facial direction estimation unit 17.
  • the occupant state detection device 300 is configured to detect the occupant's inattentiveness based on the facial direction of the occupant in addition to the degree to which the occupant's eyelids are open and the occupant's level of alertness, and therefore can improve the detection accuracy when detecting the occupant's inattentiveness compared to when the occupant's inattentiveness is detected based only on the degree to which the occupant's eyelids are open and the occupant's level of alertness.
  • the occupant status detection device 300 is configured to determine that the driver J1 is looking aside when the facial direction of the driver J1 estimated by the facial direction estimation unit 17 is not downward beyond a preset range, but is not limited to this.
  • the occupant status detection device may be configured to detect the occupant's looking aside based on the degree to which the occupant's eyes are open, the occupant's alertness, and the occupant's facial direction, and may be configured to determine that the occupant is looking aside when, for example, the occupant's eyelids are closed for a period of time that is less than a threshold, the occupant's alertness is equal to or greater than a threshold, and the angle between the occupant's facial direction and the front is equal to or greater than a preset threshold angle.
  • FIG. 10 is a block diagram showing a schematic configuration of the occupant state detection device 400 according to the fourth embodiment.
  • the occupant state detection device 400 according to the fourth embodiment differs from the occupant state detection device 300 according to the third embodiment in that the occupant state detection device 400 according to the fourth embodiment includes a drowsiness detection unit 20 for detecting drowsiness of an occupant, but the other configurations are similar.
  • the same configurations as those in the third embodiment are given the same names and symbols as those in the third embodiment, and descriptions thereof will be omitted.
  • the occupant status detection device 400 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, a face direction estimation unit 17, an aside-looking detection unit 319, and a drowsiness detection unit 20, and is electrically connected to the imaging device 110 and the alarm device 120 so as to be able to communicate with them.
  • the drowsiness detection unit 20 detects that the driver J1 is dozing off based on the degree of eyelid opening of the driver J1 detected by the eyelid opening detection unit 14, the degree of alertness of the driver J1 estimated by the alertness estimation unit 16, and the facial direction of the driver J1 estimated by the facial direction estimation unit 17. For example, the drowsiness detection unit 20 detects that the driver J1 is dozing off when the degree of alertness of the driver J1 estimated by the alertness estimation unit 16 is lower than a preset threshold value of alertness. Details of the process performed by the drowsiness detection unit 20 will be described later.
  • FIG. 11 is a flowchart showing the processing performed by the occupant status detection device 400 according to embodiment 4. Note that the processing of steps ST1, ST2, ST3, ST4, ST7, ST9, ST10, ST32, and ST13 performed by the occupant status detection device 400 according to embodiment 4 is similar to the processing performed by the occupant status detection device 300 according to embodiment 3, and therefore a description thereof will be omitted.
  • step ST10 if the driver J1's face is facing downward more than the preset range (YES in step ST10), the occupant status detection device 400 detects that the driver J1 is dozing off (step ST11). In this process, the occupant status detection device 400 determines that the driver J1 is dozing off by the dozing detection unit 20 based on the fact that the eyelids of the driver J1 are closed for a period of time equal to or longer than a threshold, the driver J1's alertness is less than a threshold, and the driver J1's face is facing downward more than the preset range.
  • step ST32 If the processing of step ST32 has been performed, and if the processing of step ST11 has been performed, then the occupant status detection device 400 outputs the detection result of driver J1's looking away and the detection result of driver drowsiness to the alarm device 120 (step ST13). If the time that driver J1's eyes are closed is less than the threshold value in the processing of step ST7 (NO in step ST7), and if the processing of step ST13 has been performed, the occupant status detection device 400 ends the processing.
  • the occupant status detection device 400 includes a drowsiness detection unit 20 that detects whether an occupant is dozing off based on the degree of eyelid opening of the occupant detected by the eyelid opening detection unit 14, the degree of alertness of the occupant estimated by the alertness estimation unit 16, and the facial direction of the occupant estimated by the facial direction estimation unit 17.
  • the occupant status detection device 400 according to the fourth embodiment is able to distinguish between an occupant looking away and an occupant dozing off, thereby enabling the notification device 120 to notify information according to the occupant's status.
  • the occupant state detection device 400 is configured to determine, by the drowsiness detection unit 20, that the driver J1 is dozing off based on the eyelid closing time of the driver J1 being equal to or longer than a threshold, the driver J1's alertness being less than a threshold, and the driver J1's facial orientation being more downward than a preset range, but is not limited thereto.
  • the occupant state detection device may be configured to detect the occupant's dozing off based on the eyelid opening degree of the occupant detected by the eyelid opening degree detection unit, the alertness of the occupant estimated by the alertness degree estimation unit, and the facial orientation of the occupant estimated by the facial orientation estimation unit, and may be configured to determine, for example, that the occupant is dozing off based on the eyelid opening degree of the occupant being less than a threshold, the alertness of the occupant being less than a threshold, and the facial orientation of the occupant not being more downward than a preset range.
  • Fig. 12 is a block diagram showing a schematic configuration of the occupant state detection device 500 according to embodiment 5.
  • the occupant state detection device 500 according to embodiment 5 differs from the occupant state detection device 400 according to embodiment 3 in that the occupant state detection device 500 according to embodiment 5 has a configuration in which an inattentive detection unit 519 detects an occupant's inattentiveness based on the occupant's line of sight.
  • the other configurations are the same, and the same configurations as those in embodiment 4 are given the same names and symbols as those in embodiment 4, and descriptions thereof will be omitted.
  • the occupant status detection device 500 includes an input unit 11, an output unit 12, an image information acquisition unit 13, an eyelid opening detection unit 14, an eyelid closing time determination unit 15, an alertness estimation unit 16, a face direction estimation unit 17, a gaze detection unit 18, an aside-looking detection unit 519, and a drowsiness detection unit 20, and is electrically connected to the imaging device 110 and the alarm device 120 so as to be able to communicate with them.
  • the gaze detection unit 18 detects the gaze direction of the driver J1 by estimating the facial direction of the driver J1 based on the image information acquired by the image information acquisition unit 13. For example, the gaze detection unit 18 estimates the gaze direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 and a trained model that learns using learning data that associates image information obtained by capturing an image of at least the occupant's face with the occupant's gaze direction, and outputs the occupant's gaze direction in response to input image information including an image of the occupant's face. Also, for example, the gaze detection unit 18 estimates the gaze direction of the driver J1 based on the image information acquired by the image information acquisition unit 13 and publicly known image recognition technology.
  • the gaze detection unit 18 extracts feature points indicating the upper eyelids, the lower eyelids, and the pupils of the driver J1 that are present in the detection area of the image information acquired by the image information acquisition unit 13 by using a known edge detection technique in a state where a detection area in which the face of the driver J1 may be present is preset in the image information acquired by the image information acquisition unit 13.
  • the face direction estimation unit 17 estimates the gaze direction of the driver J1 based on the positional relationship between the left and right ends of the upper and lower eyelids of the driver J1 and the center of the pupil.
  • the inattentiveness detection unit 519 detects inattentiveness of the driver J1 based on the judgment result by the eyelid closing time judgment unit 15, the alertness of the driver J1 estimated by the alertness estimation unit 16, the facial direction of the driver J1 estimated by the facial direction estimation unit 17, and the gaze direction of the driver J1 detected by the gaze detection unit 18. Details of the processing performed by the inattentiveness detection unit 519 will be described later.
  • FIG. 13 is a flowchart showing the processing performed by the occupant status detection device 500 according to embodiment 5. Note that the processing of steps ST1, ST2, ST3, ST4, ST7, ST9, ST10, ST11, and ST13 performed by the occupant status detection device 500 according to embodiment 5 is similar to the processing performed by the occupant status detection device 400 according to embodiment 4, and therefore a description thereof will be omitted.
  • the occupant status detection device 500 After performing the processing of step ST4, the occupant status detection device 500 next detects the gaze direction of the driver J1 (step ST5). In this processing, the occupant status detection device 500 detects the gaze direction of the driver J1 by calculating the gaze direction of the driver J1 using the gaze detection unit 18 based on the image information acquired by the image information acquisition unit 13.
  • the occupant status detection device 500 next judges whether the angle between the line of sight of the driver J1 and the road ahead is less than a preset threshold angle (step ST8).
  • the occupant status detection device 500 judges whether the driver J1 is looking aside by using the inattentive detection unit 519, by comparing the line of sight of the driver J1, whose eyelid opening degree detected by the inattentive detection unit 18 is equal to or greater than a threshold angle, with a threshold angle that is the angle between the line of sight and the road ahead at which the driver J1 is considered not to be looking aside.
  • step ST8 if the angle between the line of sight of the driver J1 and the front is equal to or greater than the preset angle threshold (NO in step ST8), the occupant status detection device 500 then detects that the driver J1 is looking aside (step ST52). In this process, the occupant status detection device 500 determines that the driver J1 is looking aside by using the inattentive detection unit 519 based on the fact that the angle between the line of sight of the driver J1 and the front when the eyelids are closed for a period less than the threshold is equal to or greater than the preset angle threshold. For example, when an attempt is made to detect the line of sight of the occupant based on image information when the eyelids are closed for a period greater than the threshold, sufficient detection accuracy may not be obtained.
  • the occupant status detection device 500 detects that the driver J1 is looking aside based on the time that the driver J1's eyelids are closed being less than a threshold value and the angle between the driver J1's line of sight and the front being equal to or greater than a preset threshold angle, thereby improving the accuracy of detection of inattentive behavior compared to conventional methods.
  • step ST52 If the processing of step ST52 has been performed, and if the processing of step ST11 has been performed, then the occupant status detection device 500 outputs the detection result of the driver J1 looking away and the detection result of the driver J1 falling asleep to the alarm device 120 (step ST13). If the angle between the line of sight of the driver J1 and the front is less than the threshold value in the processing of step ST8 (YES in step ST8), and if the processing of step ST13 has been performed, the occupant status detection device 500 ends the processing.
  • the occupant status detection device 500 according to embodiment 5 is configured to detect an occupant looking aside based on the determination result by the eyelid closure time determination unit 15, the occupant's alertness estimated by the alertness estimation unit 16, the occupant's facial direction estimated by the facial direction estimation unit 17, and the occupant's gaze direction detected by the gaze detection unit 18. As configured in this way, the occupant status detection device 500 according to embodiment 5 is configured to detect an occupant looking aside based on the gaze direction when sufficient detection accuracy of the gaze direction can be obtained, thereby improving the detection accuracy of looking aside compared to the conventional method.
  • the occupant status detection device 500 is configured to detect the occupant's inattentive behavior based on the determination result by the eyelid closure time determination unit 15, the occupant's alertness estimated by the alertness estimation unit 16, the occupant's facial direction estimated by the facial direction estimation unit 17, and the occupant's gaze direction detected by the gaze detection unit 18, but is not limited to this.
  • the occupant status detection device may be configured to detect the occupant's inattentive behavior based on the degree of eyelid closure, the occupant's alertness, the occupant's facial direction, and the occupant's gaze direction, and for example, the occupant status detection device may be configured to detect the occupant's inattentive behavior based on the gaze direction when the degree of eyelid closure of the occupant is equal to or greater than a threshold value.
  • the occupant status detection device may have some or all of the functions of the imaging device and the notification device, or the imaging device and the notification device may have some of the functions of the occupant status detection device.
  • this disclosure allows for free combinations of each embodiment, modifications to any of the components of each embodiment, or the omission of any of the components of each embodiment.
  • the occupant status detection device can be used, for example, in a system that detects when an occupant in a vehicle is looking away from the road and issues a warning to alert the occupant not to look away from the road.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Mathematical Physics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Traffic Control Systems (AREA)
PCT/JP2023/043028 2023-12-01 2023-12-01 乗員状態検知装置、乗員状態検知プログラム及び乗員状態検知方法 Pending WO2025115202A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2023/043028 WO2025115202A1 (ja) 2023-12-01 2023-12-01 乗員状態検知装置、乗員状態検知プログラム及び乗員状態検知方法
JP2025560498A JPWO2025115202A1 (https=) 2023-12-01 2023-12-01

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/043028 WO2025115202A1 (ja) 2023-12-01 2023-12-01 乗員状態検知装置、乗員状態検知プログラム及び乗員状態検知方法

Publications (1)

Publication Number Publication Date
WO2025115202A1 true WO2025115202A1 (ja) 2025-06-05

Family

ID=95896354

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/043028 Pending WO2025115202A1 (ja) 2023-12-01 2023-12-01 乗員状態検知装置、乗員状態検知プログラム及び乗員状態検知方法

Country Status (2)

Country Link
JP (1) JPWO2025115202A1 (https=)
WO (1) WO2025115202A1 (https=)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017146744A (ja) * 2016-02-16 2017-08-24 株式会社デンソー ドライバ状態判定装置
JP2018013811A (ja) * 2016-07-18 2018-01-25 株式会社デンソー ドライバ状態判定装置、及びドライバ状態判定プログラム

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017146744A (ja) * 2016-02-16 2017-08-24 株式会社デンソー ドライバ状態判定装置
JP2018013811A (ja) * 2016-07-18 2018-01-25 株式会社デンソー ドライバ状態判定装置、及びドライバ状態判定プログラム

Also Published As

Publication number Publication date
JPWO2025115202A1 (https=) 2025-06-05

Similar Documents

Publication Publication Date Title
KR101386823B1 (ko) 동작, 안면, 눈, 입모양 인지를 통한 2단계 졸음운전 방지 장치
US7948387B2 (en) Drowsiness determination apparatus, program, and method
JP5493593B2 (ja) 眠気検出装置、眠気検出方法、及びプログラム
US11430231B2 (en) Emotion estimation device and emotion estimation method
CN107209979B (zh) 用于对车辆的驾驶员的微睡进行识别的方法和装置
JP2017131445A (ja) 生体情報計測装置、車載器、及び生体情報計測システム
CN108639055A (zh) 车辆的显示系统及车辆的显示系统的控制方法
CN112220480A (zh) 基于毫米波雷达和相机融合的驾驶员状态检测系统及车辆
JP7589614B2 (ja) ドライバ監視装置、ドライバ監視方法及びドライバ監視用コンピュータプログラム
JP2018128974A (ja) ドライバ状態監視装置
KR101500016B1 (ko) 차선 이탈 경보 방법
Chatterjee et al. Driving fitness detection: A holistic approach for prevention of drowsy and drunk driving using computer vision techniques
JP4173083B2 (ja) 覚醒状態判定装置及び覚醒状態判定方法
JP7046748B2 (ja) 運転者状態判定装置および運転者状態判定方法
JP2009166783A (ja) 症状推定装置
JPH08290726A (ja) 居眠り警報装置
WO2023017595A1 (ja) 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム
JP3898586B2 (ja) 開閉眼モニタ装置
JP4677963B2 (ja) 居眠り検知装置、居眠り検知方法
CN114529887A (zh) 一种驾驶行为分析方法及装置
WO2008020458A2 (en) A method and system to detect drowsy state of driver
WO2025115202A1 (ja) 乗員状態検知装置、乗員状態検知プログラム及び乗員状態検知方法
KR20190044818A (ko) 운전자 모니터링 장치 및 방법
JP7704008B2 (ja) 運転者状態判定方法及びその装置
AU2021105935A4 (en) System for determining physiological condition of driver in autonomous driving and alarming the driver using machine learning model

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23960219

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025560498

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025560498

Country of ref document: JP