WO2024195072A1 - 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム - Google Patents

乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム Download PDF

Info

Publication number
WO2024195072A1
WO2024195072A1 PCT/JP2023/011366 JP2023011366W WO2024195072A1 WO 2024195072 A1 WO2024195072 A1 WO 2024195072A1 JP 2023011366 W JP2023011366 W JP 2023011366W WO 2024195072 A1 WO2024195072 A1 WO 2024195072A1
Authority
WO
WIPO (PCT)
Prior art keywords
occupant
state
determination unit
opening degree
eye
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/011366
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
翔悟 甫天
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to PCT/JP2023/011366 priority Critical patent/WO2024195072A1/ja
Priority to DE112023005629.3T priority patent/DE112023005629T5/de
Priority to JP2025508038A priority patent/JP7696527B2/ja
Publication of WO2024195072A1 publication Critical patent/WO2024195072A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/163—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state by tracking eye movement, gaze, or pupil change
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/168—Evaluating attention deficit, hyperactivity
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems

Definitions

  • This disclosure relates to an occupant state determination device, an occupant state determination system, an occupant state determination method, and an occupant state determination program.
  • a technology has been developed that captures images of the inside of a vehicle, and uses the shape of the occupants' eyes extracted from the captured images to determine whether they are open or closed, and whether they are looking downward, and detects whether the occupant is operating a mobile phone, smartphone, etc.
  • the determination device in Patent Document 1 uses the detected position of the driver's eyes to determine the degree of eye opening, which is the distance between the driver's upper and lower eyelids, and identifies shape parameters that correspond to gradual changes in the shape of the driver's eyes that accompany changes in the driver's facial expression and movements, and determines whether the driver is looking downward and whether the eyes are open based on the degree of eye opening and the shape parameters.
  • the determination device determines whether the degree of eye opening is equal to or less than a closing threshold.
  • the degree of eye opening being equal to or less than a closing threshold means that the eyes may be closed.
  • the determination device determines whether the specified shape parameter is equal to or less than a predetermined downward gaze threshold.
  • the determination device determines that the driver is looking downward. In such a case, the determination device determines that the driver has his/her eyes open, but is not looking ahead while driving, but is looking downward by operating a smartphone or the like, and outputs a warning to make the driver look ahead.
  • the determination device determines that the driver has his/her eyes closed. In such a case, the determination device outputs a warning as it is highly likely that the driver is drowsy while driving.
  • the closure threshold is the maximum distance between the upper and lower eyelids that indicates a state in which the eyes are closed
  • the downward gaze threshold is the maximum value of the shape parameters of the eye shape when the driver is looking downward.
  • the determination device in Patent Document 1 only describes that it uses the degree of eye opening and shape parameters to distinguish between looking downward due to operating a smartphone or the like and closing the eyes due to drowsy driving, and outputs an assigned warning for each determination result, but does not describe a specific method for distinguishing between looking downward due to various factors. Therefore, if the determination device in Patent Document 1 determines that an occupant is looking downward, it may determine that the occupant is inattentive regardless of the cause of the downward gaze and output a warning to the occupant. In other words, when an occupant looks downward, it is determined that the occupant is inattentive regardless of the cause of the downward gaze, resulting in a problem of reduced accuracy in detecting the occupant's state.
  • the present disclosure has been made to solve these problems, and aims to provide an occupant condition determination device that improves the accuracy of detecting occupant conditions.
  • the occupant state determination device includes a face image acquisition unit that acquires a face image of the face of a vehicle occupant, an eye opening degree calculation unit that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the face image acquired from the face image acquisition unit, a first eye opening degree determination unit that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit, a second eye opening degree determination unit that determines whether the occupant is in a downward looking state or a closed eye looking state using the eye opening degree acquired from the eye opening degree calculation unit when a determination result indicating that the occupant is not in an open eye state is acquired from the first eye opening degree determination unit, and an occupant state determination unit that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward looking state, using a downward looking state determination rate indicating the proportion of times that the second eye opening degree determination unit has determined that the occupant is in a downward looking
  • the occupant condition determination device can improve the accuracy of detecting the occupant condition.
  • FIG. 1 is a diagram illustrating an example of the configuration of an occupant state determination device according to a first embodiment
  • 4 is a flowchart showing an example of the operation of the occupant state determination device according to the first embodiment
  • 1 is a diagram showing a hardware configuration of an occupant state determination device according to a first embodiment
  • 1 is a diagram showing a hardware configuration of an occupant state determination device according to a first embodiment
  • FIG. 11 is a diagram illustrating an example of the configuration of an occupant state determination device according to a second embodiment.
  • 10 is a flowchart showing an example of the operation of the occupant state determination device according to the second embodiment
  • FIG. 13 is a diagram illustrating an example of the configuration of an occupant state determination device according to a third embodiment. 13 is a flowchart showing an example of the operation of the occupant state determination device according to the third embodiment;
  • Embodiment 1. 1 is a diagram showing an example of the configuration of an occupant state determination device 1A according to a first embodiment.
  • the occupant state determination device 1A is mounted on a vehicle.
  • the occupant state determination device 1A is not limited to being permanently installed in the vehicle, and may be implemented in, for example, a portable device that can be carried in the vehicle, such as a portable navigation device.
  • a part or the whole of the occupant state determination device 1A may be implemented on a server capable of communicating with the vehicle.
  • the occupant status determination system 100A includes an occupant status determination device 1A, a camera 2A, and a warning device 3A.
  • Camera 2A photographs the interior of the vehicle and is installed in a position that allows it to photograph the face of the occupant in the driver's seat of the vehicle.
  • the photographing range of camera 2A may be wide enough to photograph only the driver's seat, or wide enough to photograph the faces of occupants in all seats of the vehicle.
  • the occupant status determination device 1A determines the occupant status based on an image of the occupant's face captured by the camera 2A (hereinafter referred to as the "face image") and outputs the determination result to the warning device 3A. In this embodiment, the occupant status determination device 1A determines the status of the occupant in the driver's seat captured by the camera 2A.
  • the warning device 3A issues a warning to the occupant depending on the result of the judgment by the occupant status judgment unit 15A, and is equipped with a speaker that issues a warning sound or a warning message, a display that shows a warning screen, etc.
  • the occupant state determination device 1A includes a determination unit 10A, a face image acquisition unit 11A, and an eye opening degree calculation unit 12A.
  • the determination unit 10A includes a first eye opening degree determination unit 13A, a second eye opening degree determination unit 14A, an occupant state determination unit 15A, and a warning type determination unit 16A.
  • the facial image acquisition unit 11A acquires the facial image of the occupant captured by the camera 2A.
  • the facial image acquisition unit 11A uses face recognition technology to extract the facial image of the occupant from the image of the interior of the vehicle captured by the camera 2A.
  • the facial image acquisition unit 11A outputs the facial image of the occupant to the eye opening degree calculation unit 12A.
  • the eye opening degree calculation unit 12A detects the position of the highest point of the upper eyelid and the position of the lowest point of the lower eyelid using the facial image of the occupant acquired from the facial image acquisition unit 11A.
  • the eye opening degree calculation unit 12A calculates the distance between the position of the highest point of the upper eyelid and the position of the lowest point of the lower eyelid as the eye opening degree indicating the degree to which the occupant's eyes are open.
  • the eye opening degree calculation unit 12A outputs the calculated eye opening degree to the first eye opening degree determination unit 13A and the second eye opening degree determination unit 14A.
  • the eye opening degree calculation unit 12A determines the degree of eye opening as the distance between the highest point of the upper eyelid and the lowest point of the lower eyelid, but the degree of eye opening may also be determined as the distance between the highest point of the upper eyelid and the intersection of a vertical line drawn from the highest point of the upper eyelid and the contour of the lower eyelid. The degree of eye opening may also be determined as the distance between the highest point of the upper eyelid and the intersection of a vertical line drawn from the highest point of the upper eyelid toward a line connecting the inner corner and outer corner of the eye and the contour of the lower eyelid.
  • the first eye openness determination unit 13A uses the eye openness obtained from the eye openness calculation unit 12A to determine whether the occupant has their eyes open. The first eye openness determination unit 13A determines whether the eye openness is equal to or greater than a first threshold. Specifically, the first eye openness determination unit 13A determines that the occupant has their eyes open when the eye openness is equal to or greater than the first threshold, and determines that the occupant does not have their eyes open when the eye openness is less than the first threshold.
  • the first eye opening degree determination unit 13A If the degree of eye opening is equal to or greater than the first threshold, the first eye opening degree determination unit 13A outputs a determination result including information indicating that the occupant has their eyes open (hereinafter, information indicating that the occupant has their eyes open is referred to as eye open state information) to the occupant state determination unit 15A. If the degree of eye opening is less than the first threshold, the first eye opening degree determination unit 13A outputs a determination result including information indicating that the occupant does not have their eyes open to the second eye opening degree determination unit 14A.
  • the second eye opening degree determination unit 14A uses the eye opening degree obtained from the eye opening degree calculation unit 12A to determine whether the occupant is looking downward or with their eyes closed. Specifically, the second eye opening degree determination unit 14A determines whether the eye opening degree is equal to or greater than a second threshold. If the eye opening degree is equal to or greater than the second threshold, the second eye opening degree determination unit 14A determines that the occupant is looking downward, and if the eye opening degree is less than the second threshold, it determines that the occupant is in a closed eye state.
  • the first threshold is greater than the second threshold.
  • the second eye openness determination unit 14A outputs a determination result including information indicating that the occupant is looking downward (hereinafter, information indicating that the occupant is looking downward is referred to as downward looking state information), or information indicating that the occupant has their eyes closed (hereinafter, information indicating that the occupant has their eyes closed is referred to as eye closed state information) to the occupant state determination unit 15A.
  • downward looking state information information indicating that the occupant is looking downward
  • eye closed state information information indicating that the occupant has their eyes closed
  • the occupant state determination unit 15A uses the downward gaze state determination rate, which indicates the percentage of times during the first period when the second eye opening determination unit 14A determines that the occupant is in a downward gaze state, to determine whether the occupant's state corresponds to the first occupant state, which is an inattentive state caused by a downward gaze state.
  • the first period is a predetermined period.
  • the first period is a predetermined period (e.g., 20 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km/h) and the steering angle is within a predetermined range (e.g., ⁇ 20 degrees). Note that this is merely an example, and the first period may be appropriately set by a manager of the occupant state determination device 1A or may be stored in advance in the occupant state determination device 1A.
  • the occupant state judgment unit 15A obtains a judgment result including eye open state information from the first eye opening degree judgment unit 13A during the first period, a judgment result including downward gaze state information from the second eye opening degree judgment unit 14A during the first period, or a judgment result including eye closed state information from the second eye opening degree judgment unit 14A during the first period.
  • the occupant state determination unit 15A then calculates the proportion of determination results including downward gaze state information acquired in the first time period to the determination results including open eye state information acquired in the first time period, the determination results including downward gaze state information acquired in the first time period, and the determination results including closed eye state information acquired in the first time period, as the downward gaze state determination rate.
  • the occupant state determination unit 15A uses the calculated downward gaze state determination rate to determine whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward gaze state.
  • the first occupant state is, for example, a state in which a smartphone or the like is being operated.
  • the occupant state determination unit 15A determines whether the downward gaze state determination rate is equal to or greater than a third threshold value. If the downward gaze state determination rate is equal to or greater than the third threshold value, the occupant state determination unit 15A determines that the occupant's state is the first state. When the downward gaze state determination rate is less than the third threshold, the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state. Furthermore, even when the occupant state determination unit 15A obtains a determination result including eye open state information from the first eye open degree determination unit 13A, the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state.
  • the third threshold is set to, for example, 50 percent. Note that, although an example in which the third threshold is 50 percent has been shown, the third threshold is not limited to this and may be appropriately set by an administrator of the occupant state determination device 1A, for example.
  • the occupant state determination unit 15A outputs a determination result indicating whether the occupant state corresponds to the first occupant state to the warning type determination unit 16A.
  • the occupant state determination unit 15A determines that the occupant state is the first occupant state when the downward gaze state determination rate is equal to or greater than the third threshold, and determines that the occupant state is an occupant state other than the first occupant state when the downward gaze state determination rate is less than the third threshold.
  • the occupant looks downward continuously or frequently. In other words, the occupant is in a downward gaze state continuously or frequently during the first period, so the downward gaze state determination rate tends to be high. In addition, because the occupant is in a downward gaze state continuously or frequently during a predetermined period, the time that the occupant looks forward is reduced compared to normal driving, so the occupant can be considered to be in an inattentive state. For the above reasons, when the downward gaze state determination rate is equal to or greater than the third threshold, the occupant state determination unit 15A determines that the occupant state is the first occupant state, which is an inattentive state caused by a downward gaze state.
  • the occupant state determination unit 15A determines that the occupant state is the first occupant state, and when the downward gaze state determination rate is less than the third threshold, the occupant state determination unit 15A determines that the occupant state is an occupant state other than the first occupant state.
  • the warning type determination unit 16A uses the determination result acquired from the occupant state determination unit 15A to determine the type of warning to be output by the warning device 3A that prompts the occupant to issue a warning.
  • the warning type determination unit 16A causes the warning device 3A to output a first warning.
  • the warning type determination unit 16A does not cause the warning device 3A to output the first warning.
  • FIG. 2 is a flowchart showing an example of the operation of the occupant state determination device 1A according to this embodiment.
  • the processing executed by the occupant state determination device 1A according to this embodiment will be described with reference to the flowchart shown in FIG. 2.
  • the operation of the occupant state determination device 1A is started, for example, after the vehicle ignition is turned ON.
  • the facial image acquisition unit 11A acquires a facial image (step ST101).
  • the eye opening degree calculation unit 12A calculates the eye opening degree using the face image acquired from the face image acquisition unit 11A (step ST102).
  • the first eye opening degree determination unit 13A uses the eye opening degree obtained from the eye opening degree calculation unit 12A to determine whether the occupant has their eyes open (step ST103).
  • the second eye opening degree determination unit 14A uses the eye opening degree obtained from the eye opening degree calculation unit 12A to determine whether the occupant is looking downward or has their eyes closed (step ST104).
  • the occupant state determination unit 15A calculates the downward gaze state determination rate using the determination result including the open eye state information obtained from the first eye opening degree determination unit 13A, the determination result including the downward gaze state information obtained from the second eye opening degree determination unit 14A, and the determination result including the closed eye state information obtained from the second eye opening degree determination unit 14A (step ST105).
  • step ST105 the occupant state determination unit 15A determines whether the downward gaze state determination rate is equal to or greater than a third threshold (step ST106).
  • step ST106 determines that the occupant state is the first occupant state (step ST107).
  • step ST108 determines that the occupant state is an occupant state other than the first occupant state (step ST108).
  • the warning type determination unit 16A causes the warning device 3A to output a first warning (step ST109).
  • step ST108 the warning type determination unit 16A does not cause the warning device 3A to output the first warning (step ST110). Then, the occupant state determination device 1A ends the operation shown in the flowchart of FIG. 2.
  • FIGS. 3 and 4 are diagrams showing an example of the hardware configuration of the occupant state determination device 1A according to the first embodiment.
  • the functions of the face image acquisition unit 11A, eye opening degree calculation unit 12A, first eye opening degree determination unit 13A, second eye opening degree determination unit 14A, occupant state determination unit 15A, and warning type determination unit 16A in the occupant state determination device 1A are realized by a processing circuit. That is, the occupant state determination device 1A includes a processing circuit for realizing the above functions.
  • the processing circuit may be a processing circuit 100 as dedicated hardware, or may be a processor 101 that executes a program stored in a memory 102.
  • the processing circuit 100 is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination of these.
  • the functions of the face image acquisition unit 11A, the eye opening degree calculation unit 12A, the first eye opening degree determination unit 13A, the second eye opening degree determination unit 14A, the occupant state determination unit 15A, and the warning type determination unit 16A may be realized by multiple processing circuits 100, or the functions of each unit may be realized by a single processing circuit 100.
  • the processing circuit is a processor 101
  • the functions of the facial image acquisition unit 11A, eye opening degree calculation unit 12A, first eye opening degree determination unit 13A, second eye opening degree determination unit 14A, occupant state determination unit 15A, and warning type determination unit 16A are realized by software, firmware, or a combination of software and firmware.
  • the software or firmware is written as a program and stored in memory 102.
  • the processor 101 realizes the functions of each unit by reading and executing the program stored in memory 102.
  • the occupant state determination device 1A is equipped with memory 102 for storing a program which, when executed by the processor 101, will result in the steps shown in the flowchart of FIG. 2 being executed.
  • This program can also be said to cause a computer to execute the procedures or methods of the face image acquisition unit 11A, eye opening degree calculation unit 12A, first eye opening degree determination unit 13A, second eye opening degree determination unit 14A, occupant state determination unit 15A, and warning type determination unit 16A.
  • the processor 101 refers to a CPU (Central Processing Unit), a processing unit, an arithmetic unit, or a microprocessor.
  • the memory 102 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or a flash memory, or may be a magnetic disk such as a hard disk or a flexible disk, or may be an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc).
  • the functions of the face image acquisition unit 11A, eye opening degree calculation unit 12A, first eye opening degree determination unit 13A, second eye opening degree determination unit 14A, occupant state determination unit 15A, and warning type determination unit 16A may be partially realized by dedicated hardware and partially realized by software or firmware. In this way, the processing circuit in the occupant state determination device 1A can realize the above-mentioned functions by hardware, software, firmware, or a combination of these.
  • the occupant state determination device 1A of this embodiment includes a facial image acquisition unit 11A that acquires a facial image, which is an image obtained by capturing the face of a vehicle occupant; an eye opening degree calculation unit 12A that calculates the degree of eye opening indicating the degree to which the occupant's eyes are open, using the facial image acquired from the facial image acquisition unit 11A; a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12A; a second eye opening degree determination unit 14A that determines whether the occupant is in a downward looking state or a closed eye state, using the eye opening degree acquired from the eye opening degree calculation unit 12A, when a determination result indicating that the occupant is not in an open eye state is obtained from the first eye opening degree determination unit 13A; and an occupant state determination unit 15A that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward looking state, using a
  • the occupant state determination device 1A can determine whether or not an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of erroneous determination that an occupant determined to be in a downward-looking state or a closed-eye state is in the first occupant state even though the occupant is not in the first occupant state, and can improve the detection accuracy of the occupant state.
  • the occupant state determination device 1A includes a face image acquisition unit 11A that acquires a face image, which is an image obtained by capturing the face of a vehicle occupant, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree to which the occupant's eyes are open, using the face image acquired from the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is looking downward or has their eyes closed, using the eye opening degree acquired from the eye opening degree calculation unit 12A, when a determination result indicating that the occupant is not in an eye open state is acquired from the first eye opening degree determination unit 13A, and an occupant state determination unit 15A that determines whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward looking state, using a downward looking state determination
  • the occupant state determination device 1A can determine whether or not an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of erroneous determination that an occupant determined to be in a downward-looking state or a closed-eye state is in the first occupant state even though the occupant is not in the first occupant state, and can improve the detection accuracy of the occupant state.
  • the occupant state determination device 1A includes a face image acquisition unit 11A that acquires a face image, which is an image obtained by capturing the face of a vehicle occupant, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree to which the occupant's eyes are open, using the face image acquired from the face image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is looking downward or has their eyes closed, using the eye opening degree acquired from the eye opening degree calculation unit 12A, when a determination result indicating that the occupant is not in an eye open state is acquired from the first eye opening degree determination unit 13A.
  • the system further comprises an occupant state determination unit 15A that determines whether the occupant's state corresponds to a first occupant state, which is a state of inattention caused by a downward looking state, using a downward looking state determination rate indicating the proportion of times that the second eye opening degree determination unit 14A during a first period has determined that the occupant is in a downward looking state, and a warning type determination unit 16A that uses the determination result obtained from the occupant state determination unit 15A to determine the type of warning to be output by the warning device 3A to alert the occupant.
  • the warning type determination unit 16A obtains a determination result from the occupant state determination unit 15A indicating that the occupant's state is the first occupant state, it causes the warning device 3A to output a first warning.
  • the occupant state determination device 1A can avoid outputting an unnecessary warning to an occupant who has been determined to be in a downward gaze state or a closed-eye state, by determining that the occupant is in the first occupant state even though the occupant is in a state other than the first occupant state. Therefore, the occupant state determination device 1A can output a necessary warning to the occupant.
  • the occupant state determination system 100A includes a facial image acquisition unit 11A that acquires a facial image, which is an image obtained by capturing the face of a vehicle occupant, an eye opening degree calculation unit 12A that calculates an eye opening degree indicating the degree to which the occupant's eyes are open, using the facial image acquired from the facial image acquisition unit 11A, a first eye opening degree determination unit 13A that determines whether the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a second eye opening degree determination unit 14A that determines whether the occupant is looking downward or has their eyes closed, using the eye opening degree acquired from the eye opening degree calculation unit 12A, when a determination result indicating that the occupant is not in an eye open state is acquired from the first eye opening degree determination unit 13A.
  • the abnormal posture determination system includes an occupant state determination device 1A having an occupant state determination unit 15A that determines whether the occupant's state corresponds to a first occupant state, which is a state of inattention caused by a downward looking state, using a downward looking state determination rate that indicates the percentage of times that a second eye openness determination unit 14A in a first period determines that the occupant is in a downward looking state, a camera 2A installed in the vehicle so as to be able to capture an image of at least the occupant's face, and a warning device 3A that outputs a warning to the occupant, wherein the occupant state determination device 1A determines the occupant's state using the facial image captured by the camera 2A, and causes the warning device 3A to output a warning in accordance with the determination result of the occupant state determination unit 15A.
  • the occupant state determination system 100A can determine whether or not an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can suppress the occurrence of erroneous determination that an occupant determined to be in a downward-looking state or a closed-eye state is in the first occupant state even though the occupant is not in the first occupant state, and can improve the detection accuracy of the occupant state.
  • the occupant state determination method of this embodiment includes the steps of acquiring a facial image, which is an image obtained by capturing the face of a vehicle occupant; an eye opening degree calculation unit 12A calculating an eye opening degree indicating the degree of opening of the occupant's eyes, using the facial image acquired from the facial image acquisition unit 11A; a first eye opening degree determination unit 13A determining whether or not the occupant has his/her eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12A; if the second eye opening degree determination unit 14A acquires a determination result from the first eye opening degree determination unit 13A indicating that the occupant is not in an eye open state, determining whether the occupant is in a downward looking state or a closed eye state, using the eye opening degree acquired from the eye opening degree calculation unit 12A; and an occupant state determination unit 15A determining whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward looking state, using a downward looking state determination rate indicating the proportion of
  • the occupant state determination method can determine whether or not an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by a determination using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can reduce the occurrence of erroneous determination that an occupant determined to be in a downward-looking state or a closed-eye state is in the first occupant state even though the occupant is not in the first occupant state, and can improve the detection accuracy of the occupant state.
  • the occupant state determination program includes a step in which a face image acquisition unit 11A acquires a face image, which is an image obtained by capturing the face of an occupant of a vehicle, a step in which an eye opening degree calculation unit 12A calculates an eye opening degree indicating a degree of opening of the occupant's eyes, using the face image acquired from the face image acquisition unit 11A, a step in which a first eye opening degree determination unit 13A determines whether or not the occupant has his/her eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12A, and a step in which a second eye opening degree determination unit 14A determines whether or not the first eye opening degree determination unit 13A has his/her eyes open.
  • the computer executes the steps of: determining whether the occupant is in a downward looking state or a closed eye state using the eye opening degree obtained from the eye opening degree calculation unit 12A; and determining whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward looking state, using a downward looking state determination rate indicating the proportion of times during a first period that the second eye opening degree determination unit 14A determines that the occupant is in a downward looking state.
  • the occupant state determination program can determine whether or not an occupant determined to be in a downward-looking state or a closed-eye state corresponds to the first occupant state by using the downward-looking state determination rate in the occupant state determination unit 15A. Therefore, the occupant state determination device 1A can reduce the occurrence of erroneous determination that an occupant determined to be in a downward-looking state or a closed-eye state is in the first occupant state even though the occupant is not in the first occupant state, and can improve the detection accuracy of the occupant state.
  • Embodiment 2 An occupant condition determining device 1B according to the present embodiment will be described.
  • the occupant state determination device 1A in embodiment 1 is configured to determine the occupant state based on the judgment result of the eye openness degree and a downward gaze state judgment rate calculated from the judgment result of the eye openness degree, whereas the occupant state determination device 1B in the present embodiment differs from embodiment 1 in that it determines the occupant state based on the judgment result of the eye openness degree and the downward gaze state judgment rate calculated from the judgment result of the eye openness degree, in addition to the judgment result of the eye openness degree and the downward gaze state judgment rate calculated from the judgment result of the eye openness degree.
  • FIG. 5 is a diagram showing an example of the configuration of an occupant state determination device 1B according to this embodiment.
  • the occupant state determination system 100B includes a camera 2B, an occupant state determination device 1B, and a camera 2B.
  • the occupant state determination device 1B also includes a determination unit 10B, a face image acquisition unit 11B, and an eye opening degree calculation unit 12B.
  • the determination unit 10B includes a first eye opening degree determination unit 13B, a second eye opening degree determination unit 14B, an occupant state determination unit 15B, and a warning type determination unit 16B.
  • the operation of the occupant state determination unit 15B and the operation of the warning type determination unit 16B are different from those of the occupant state determination device 1A of the first embodiment.
  • the rest of the configuration and operation are the same as those of the first embodiment.
  • the occupant state determination unit 15B uses the downward gaze state determination rate and the closed eye state determination rate indicating the percentage of times during the first period when the second eye openness determination unit 14B determines that the occupant has closed eyes to determine whether the occupant's state corresponds to the first occupant state, the second occupant state which is an inattentive state caused by closed eyes, or the third occupant state which is not an inattentive state.
  • the second occupant state is, for example, a state of feeling drowsy, a state of falling asleep while driving, etc.
  • the third occupant state is, for example, a state of checking meters, etc., a state of driving normally, etc.
  • the occupant state judgment unit 15B obtains a judgment result including eye open state information from the first eye opening degree judgment unit 13B during the first period, a judgment result including downward gaze state information from the second eye opening degree judgment unit 14B during the first period, or a judgment result including eye closed state information from the second eye opening degree judgment unit 14B during the first period.
  • the occupant state determination unit 15B calculates the proportion of determination results including downward gaze state information obtained in the first time period to the determination results including open eye state information obtained in the first time period, the determination results including downward gaze state information obtained in the first time period, and the determination results including closed eye state information obtained in the first time period as the downward gaze state determination rate.
  • the occupant state determination unit 15B also calculates the proportion of determination results including closed eye state information acquired in the first time period to the determination results including open eye state information acquired in the first time period, the determination results including downward gaze state information acquired in the first time period, and the determination results including closed eye state information acquired in the first time period as the closed eye state determination rate.
  • the occupant state determination unit 15B uses the downward gaze state determination rate and the closed eyes state determination rate to determine whether the occupant state corresponds to the first occupant state, the second occupant state, or the third occupant state.
  • the occupant state determination unit 15B determines whether the downward gaze state determination rate is equal to or greater than a third threshold value. If the downward gaze state determination rate is equal to or greater than the third threshold value, the occupant state determination unit 15B determines that the occupant's state is the first state. When the downward gaze state determination rate is less than the third threshold, the occupant state determination unit 15B determines whether the eye closed state determination rate is equal to or greater than the fourth threshold. When the eye closed state determination rate is equal to or greater than the fourth threshold, the occupant state determination unit 15B determines that the occupant state is the second occupant state, and when the eye closed state determination rate is less than the fourth threshold, the occupant state determination unit 15B determines that the occupant state is the third occupant state.
  • the occupant state determination unit 15B determines that the occupant state is the third occupant state.
  • the fourth threshold value is set to, for example, 50 percent. Although the fourth threshold value is set to 50 percent in the above example, the fourth threshold value is not limited to this and may be appropriately set by a manager of the occupant state determination device 1B or the like.
  • the occupant state determination unit 15B outputs a determination result indicating whether the occupant state corresponds to the first occupant state, the second occupant state, or the third occupant state to the warning type determination unit 16B.
  • the occupant state determination unit 15B determines that the occupant state is the first occupant state when the downward gaze state determination rate is equal to or greater than the third threshold, determines that the occupant state is the second occupant state when the downward gaze state determination rate is less than the third threshold and the closed eye state determination rate is equal to or greater than the fourth threshold, and determines that the occupant state is the third occupant state when the downward gaze state determination rate is less than the third threshold and the closed eye state determination rate is less than the fourth threshold.
  • the occupant state determination unit 15B determines that the occupant's state is the second occupant state when the downward gaze state determination rate is less than the third threshold value and when the closed eye state determination rate is greater than or equal to the fourth threshold value.
  • the occupant state determination unit 15B determines that the occupant's state is the third occupant state if the downward gaze state determination rate is less than the third threshold value and the closed eye state determination rate is less than the fourth threshold value.
  • the occupant state determination unit 15B determines that the occupant state is the first occupant state when the downward gaze state determination rate is equal to or greater than the third threshold, determines that the occupant state is the second occupant state when the downward gaze state determination rate is less than the third threshold and the closed eye state determination rate is equal to or greater than the fourth threshold, and determines that the occupant state is the third occupant state when the downward gaze state determination rate is less than the third threshold and the closed eye state determination rate is less than the fourth threshold.
  • the warning type determination unit 16B uses the determination result acquired from the occupant state determination unit 15B to determine the type of warning to be output by the warning device 3B that prompts the occupant to issue a warning.
  • the warning type determination unit 16B causes the warning device 3B to output a first warning when the occupant state determination unit 15B determines that the occupant state is the first state.
  • the warning type determination unit 16B causes the warning device 3B to output a second warning different from the first warning when the occupant state determination unit 15B determines that the occupant state is the second state.
  • the warning type determination unit 16B causes the warning device 3B not to output the first warning or the second warning when the occupant state determination unit 15B determines that the occupant state is the third state.
  • FIG. 6 is a flowchart showing an example of the operation of the occupant state determination device 1B according to this embodiment.
  • the processing executed by the occupant state determination device 1B according to this embodiment will be described with reference to the flowchart shown in FIG. 6.
  • the operation of the occupant state determination device 1B is started, for example, after the vehicle ignition is turned ON.
  • the facial image acquisition unit 11B acquires a facial image (step ST201).
  • the eye opening degree calculation unit 12B calculates the eye opening degree using the face image acquired from the face image acquisition unit 11B (step ST202).
  • the first eye opening degree determination unit 13B uses the eye opening degree obtained from the eye opening degree calculation unit 12B to determine whether the occupant has their eyes open (step ST203).
  • the second eye opening degree determination unit 14B uses the eye opening degree acquired from the eye opening degree calculation unit 12B to determine whether the occupant is looking downward or has their eyes closed (step ST204).
  • the occupant state determination unit 15B calculates a downward gaze state determination rate and a closed eye state determination rate using the determination result including the open eye state information obtained from the first eye openness determination unit 13B, the determination result including the downward gaze state information obtained from the second eye openness determination unit 14B, and the determination result including the closed eye state information obtained from the second eye openness determination unit 14B (step ST205).
  • step ST205 the occupant state determination unit 15B determines whether the downward gaze state determination rate is equal to or greater than a third threshold (step ST206).
  • step ST206 determines that the occupant state is the first occupant state (step ST207).
  • step ST208 the occupant state determination unit 15B determines whether the eyes closed state determination rate is equal to or greater than the fourth threshold (step ST208).
  • step ST208 determines that the eye closed state is the second occupant state (step ST209).
  • step ST210 determines that the occupant's state is the third occupant state (step ST210).
  • the warning type determination unit 16B causes the warning device 3B to output a first warning (step ST211).
  • the warning type determination unit 16B causes the warning device 3B to output a second warning (step ST212).
  • step ST210 the warning type determination unit 16B does not cause the warning device 3B to output the first warning or the second warning (step ST213). Then, the occupant state determination device 1B ends the operation shown in the flowchart of FIG. 6.
  • the occupant state determination device 1B includes a face image acquisition unit 11B that acquires a face image, which is an image obtained by capturing the face of a vehicle occupant, an eye opening degree calculation unit 12B that calculates an eye opening degree indicating the degree to which the occupant's eyes are open, using the face image acquired from the face image acquisition unit 11B, a first eye opening degree determination unit 13B that determines whether the occupant has their eyes open, using the eye opening degree acquired from the eye opening degree calculation unit 12B, and a second eye opening degree determination unit 14B that determines whether the occupant is looking downward or has their eyes closed, using the eye opening degree acquired from the eye opening degree calculation unit 12B, when a determination result indicating that the occupant is not in an eye open state is acquired from the first eye opening degree determination unit 13B, and and an occupant state determination unit 15B that uses a downward looking state determination rate that indicates the percentage of times during the first period when the second eye opening degree determination unit 14B determines that the occupant is in a downward looking state
  • the occupant state determination device 1B can detect a greater number of occupant states compared to the occupant state determination device 1A of embodiment 1 by detecting a second occupant state and a third occupant state in addition to the first occupant state.
  • the occupant state determination device 1B includes a face image acquisition unit 11B that acquires a face image, which is an image obtained by capturing the face of a vehicle occupant, an eye opening degree calculation unit 12B that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the face image acquired from the face image acquisition unit 11B, a first eye opening degree determination unit 13B that determines whether the occupant has his/her eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12B, a second eye opening degree determination unit 14B that, when a determination result indicating that the occupant is not in an open-eyed state is acquired from the first eye opening degree determination unit 13B, determines whether the occupant is in a downward-looking state or a closed-eyed state using the eye opening degree acquired from the eye opening degree calculation unit 12B, and a percentage of times that the second eye opening degree determination unit 14B has determined that the occupant is in a downward-looking state during a first period.
  • the vehicle occupant state determination unit 15B is equipped with an occupant state determination unit that uses a downward gaze state determination rate that indicates whether the occupant state corresponds to a first occupant state, which is a state of inattention caused by a downward gaze state, and a warning type determination unit 16B that uses the determination result obtained from the occupant state determination unit 15B to determine the type of warning to be output by the warning device 3B to alert the occupant, wherein when the warning type determination unit 16B obtains a determination result from the occupant state determination unit 15B indicating that the occupant state is the first occupant state, it causes the warning device 3B to output a first warning, and when the warning type determination unit 16B obtains a determination result from the occupant state determination unit 15B indicating that the occupant state is the second occupant state, it causes the warning device 3B to output a second warning different from the first warning.
  • the occupant state determination device 1B can output more warnings corresponding to the occupant state by outputting a warning corresponding to the second occupant state in addition to the first occupant state, compared to when only a warning corresponding to the first occupant state is output.
  • Embodiment 3 An occupant condition determining device 1C according to the present embodiment will be described.
  • the occupant state determination device 1A in embodiment 1 is configured to determine the occupant state based on the judgment result of the eye openness degree and the downward gaze state judgment rate calculated from the judgment result of the eye openness degree, whereas the occupant state determination device 1C in the present embodiment differs from embodiment 1 in that it determines the occupant state based on the judgment result of the blink count difference in addition to the judgment result of the eye openness degree and the downward gaze state judgment rate calculated from the judgment result of the eye openness degree.
  • FIG. 7 is a diagram showing an example of the configuration of an occupant state determination device 1C according to an embodiment.
  • occupant state determination system 100C includes camera 2C, occupant state determination device 1C, and camera 2C.
  • Occupant state determination device 1C also includes determination unit 10C, face image acquisition unit 11C, eye opening degree calculation unit 12C, blink count calculation unit 17C, and reference blink count estimation unit 18C.
  • Determination unit 10C includes first eye opening degree determination unit 13C, second eye opening degree determination unit 14C, blink count determination unit 19C, occupant state determination unit 15C, and warning type determination unit 16C.
  • this embodiment further includes a blink count calculation unit 17C, a reference blink count estimation unit 18C, and a blink count determination unit 19C.
  • a blink count calculation unit 17C the reference blink count estimation unit 18C
  • a blink count determination unit 19C the occupant state determination unit 15C
  • the blink count calculation unit 17C calculates the blink count indicating the number of blinks of the occupant in the second period, using the degree of eye opening acquired from the eye opening degree calculation unit 12C.
  • the second period is a predetermined period.
  • the second period is a predetermined period (e.g., 5 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km/h) and the steering angle is within a predetermined range (e.g., ⁇ 20 degrees).
  • a predetermined speed e.g. 25 km/h
  • the steering angle is within a predetermined range (e.g., ⁇ 20 degrees).
  • this is merely an example, and the second period may be appropriately set by a manager of the occupant state determination device 1C or may be stored in advance in the occupant state determination device 1C.
  • the blink count calculation unit 17C determines that the eye is closed when the degree of eye opening falls below a second threshold, and determines that the eye is open when the degree of eye opening exceeds a first threshold that is greater than the second threshold.
  • the blink count calculation unit 17C determines that "blinking has occurred” when an eye open state is detected within a third period after detecting a closed eye state.
  • the blink count calculation unit 17C determines that "blinking has not occurred” when an eye open state is not detected within a third period after detecting a closed eye state. In this way, the blink count calculation unit 17C calculates the number of blinks in the second period.
  • the third period is a predetermined period.
  • the third period is a predetermined period (e.g., 0.2 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km/h) and the steering angle is within a predetermined range (e.g., ⁇ 20 degrees).
  • a predetermined speed e.g. 25 km/h
  • a predetermined range e.g., ⁇ 20 degrees
  • the blink count calculation unit 17C outputs the calculated blink count to the reference blink count estimation unit 18C and the blink count determination unit 19C.
  • the reference blink count estimator 18C estimates the reference blink count of the occupant using the blink count calculated by the blink count calculator 17C during the fourth time period.
  • the fourth period is a predetermined period.
  • the fourth period is a predetermined period (e.g., 10 seconds) when the vehicle speed is equal to or greater than a predetermined speed (e.g., 25 km/h) and the steering angle is within a predetermined range (e.g., ⁇ 20 degrees).
  • a predetermined speed e.g., 25 km/h
  • a predetermined range e.g., ⁇ 20 degrees
  • the reference blink count estimation unit 18C outputs the estimated reference blink count to the blink count determination unit 19C.
  • the blink count determination unit 19C compares the blink count acquired from the blink count calculation unit 17C with the reference blink count acquired from the reference blink count estimation unit 18C, and determines whether or not there has been a change in the blink count of the occupant using a blink count difference indicating the difference between the blink count and the reference blink count. Specifically, the blink count determination unit 19C determines whether or not the blink count difference is equal to or greater than a fifth threshold. If the blink count difference is equal to or greater than the fifth threshold, the blink count determination unit 19C determines that there has been a change in the blink count of the occupant, and if the blink count difference is less than the fifth threshold, determines that there has been no change in the blink count of the occupant.
  • the fifth threshold value may be appropriately set by an administrator of the occupant state determination device 1C, or may be stored in advance in the occupant state determination device 1C.
  • the blink count determination unit 19C If the blink count difference is less than the fifth threshold, the blink count determination unit 19C outputs a determination result including information indicating that the blink count difference is less than the fifth threshold to the occupant state determination unit 15C. If the blink count difference is equal to or greater than the fifth threshold, the blink count determination unit 19C outputs a determination result including information indicating that the blink count difference is equal to or greater than the fifth threshold to the occupant state determination unit 15C.
  • the occupant state determination device 1C of the present embodiment includes the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C.
  • the number of blinks tends to increase or decrease compared to the number of blinks during normal driving.
  • the blink count determination unit 19C further determines that there is a change in the number of blinks of the occupant, it is more likely that the occupant is operating a smartphone, etc., is drowsy, is dozing while driving, or is checking a meter, etc.
  • the blink count determination unit 19C further determines whether there is a change in the number of blinks of the occupant, it is possible to more accurately distinguish between a state in which the occupant operates a smartphone, etc., a state in which the occupant is drowsy, is dozing while driving, or a state in which the occupant is checking a meter, etc., and other occupant states.
  • the occupant state determination unit 15C When the occupant state determination unit 15C obtains a determination result from the blink count determination unit 19C indicating that there is a change in the number of blinks of the occupant, it uses the downward gaze state determination rate, which indicates the proportion of times that the second eye opening determination unit 14C determined that the occupant is in a downward gaze state during the first period, to determine whether the occupant's state corresponds to the first occupant state.
  • the occupant state determination unit 15C determines whether the downward gaze state determination rate is equal to or greater than a third threshold value. If the downward gaze state determination rate is equal to or greater than the third threshold value, the occupant state determination unit 15C determines that the occupant's state is the first state. When the downward gaze state determination rate is less than the third threshold, the occupant state determination unit 15C determines that the occupant state is other than the first occupant state, where the occupant state other than the first occupant state includes the second occupant state or the third occupant state.
  • the occupant state determination unit 15C obtains a determination result including eye open state information from the first eye opening degree determination unit 13C, or when it obtains a determination result indicating that there is no change in the number of blinks of the occupant from the blink count determination unit 19C, it determines that the occupant state is an occupant state other than the first occupant state.
  • the occupant state determination unit 15C outputs a determination result indicating whether the occupant state corresponds to the first occupant state to the warning type determination unit 16C.
  • FIG. 8 is a flowchart showing an example of the operation of the occupant state determination device 1C according to this embodiment.
  • the processing executed by the occupant state determination device 1C according to this embodiment will be described with reference to the flowchart shown in FIG. 8. Note that the operation of the occupant state determination device 1C is started, for example, after the vehicle ignition is turned ON. In addition, it is preferable to perform the occupant state determination processing each time a face image is acquired from the camera 2C, that is, for each frame.
  • the facial image acquisition unit 11C acquires a facial image (step ST301).
  • the eye opening degree calculation unit 12C calculates the eye opening degree using the face image acquired from the face image acquisition unit 11C (step ST302).
  • the blink count calculation unit 17C calculates the blink count using the degree of eye opening acquired from the eye opening degree calculation unit 12C (step ST303).
  • the reference blink count estimation unit 18C uses the blink count calculated by the blink count calculation unit 17C to estimate the reference blink count (step ST304).
  • the first eye opening degree determination unit 13C uses the eye opening degree obtained from the eye opening degree calculation unit 12C to determine whether the occupant has their eyes open (step ST305).
  • the second eye opening degree determination unit 14C uses the eye opening degree acquired from the eye opening degree calculation unit 12C to determine whether the occupant is looking downward or has their eyes closed (step ST306).
  • the blink count determination unit 19C determines whether there has been a change in the blink count of the occupant (step ST307).
  • the occupant state determination unit 15C calculates the downward gaze state determination rate using the determination result including the open eye state information obtained from the first eye opening degree determination unit 13C, the determination result including the downward gaze state information obtained from the second eye opening degree determination unit 14C, and the determination result including the closed eye state information obtained from the second eye opening degree determination unit 14C (step ST308).
  • step ST308 the occupant state determination unit 15C determines whether the downward gaze state determination rate is equal to or greater than a third threshold (step ST309).
  • step ST309 determines that the occupant state is the first occupant state (step ST310).
  • step ST305 determines whether the occupant has their eyes open (if step ST305 is YES), if there is no change in the number of times the occupant blinks (if step ST307 is NO), or if the downward gaze state determination rate is less than the third threshold (if step ST309 is NO), the occupant state determination unit 15C determines that the occupant's state is an occupant state other than the first occupant state (step ST311).
  • the warning type determination unit 16C causes the warning device 3C to output a first warning (step ST312).
  • step ST311 the warning type determination unit 16C does not cause the warning device 3C to output the first warning (step ST313). Then, the occupant state determination device 1C ends the operation shown in the flowchart of FIG. 8.
  • the occupant state determination device 1C relating to a modified example of the present embodiment includes a face image acquisition unit 11C that acquires a face image, which is an image obtained by capturing the face of a vehicle occupant, an eye opening degree calculation unit 12C that calculates an eye opening degree indicating the degree to which the occupant's eyes are open using the face image acquired from the face image acquisition unit 11C, a first eye opening degree determination unit 13C that determines whether the occupant has their eyes open using the eye opening degree acquired from the eye opening degree calculation unit 12C, a second eye opening degree determination unit 14C that, when a determination result indicating that the occupant is not in an open-eyed state is acquired from the first eye opening degree determination unit 13C, determines whether the occupant is in a downward-looking state or a closed-eyed state using the eye opening degree acquired from the eye opening degree calculation unit 12C, and a ratio during a first period in which the second eye opening degree determination unit 14C determines that the occupant is in a downward-looking state.
  • the occupant state determination unit 15C is equipped with an occupant state determination rate that indicates whether the occupant's state corresponds to a first occupant state, which is an inattentive state caused by a downward gaze state, using a downward gaze state determination rate that indicates the downward gaze state, and further includes a blink count determination unit 19C that determines whether there has been a change in the blink count, which indicates the number of times the occupant blinks.
  • a blink count determination unit 19C determines whether there has been a change in the blink count, which indicates the number of times the occupant blinks.
  • the occupant state determination device 1C can more accurately distinguish between a state in which the occupant operates a smartphone or the like, a state in which the occupant is drowsy, a state in which the occupant is dozing while driving, or a state in which the occupant checks a meter or the like, and other occupant states. As a result, the occupant state determination device 1C can further improve the detection accuracy of the occupant state compared to the occupant state determination device 1A of the first embodiment.
  • the combination of the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C, which are configurations unique to this embodiment, and embodiment 1 has been described.
  • the blink count calculation unit 17C, the reference blink count estimation unit 18C, and the blink count determination unit 19C, which are configurations unique to this embodiment may also be combined with embodiment 2.
  • 1A Occupant status determination device 2A Camera, 3A Warning device, 10A Determination unit, 11A Face image acquisition unit, 12A Eye opening degree calculation unit, 13A First eye opening degree determination unit, 14A Second eye opening degree determination unit, 15A Occupant status determination unit, 16A Warning type determination unit, 100A Occupant status determination system

Landscapes

  • Health & Medical Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Developmental Disabilities (AREA)
  • Social Psychology (AREA)
  • Biophysics (AREA)
  • Veterinary Medicine (AREA)
  • Public Health (AREA)
  • Child & Adolescent Psychology (AREA)
  • Educational Technology (AREA)
  • Hospice & Palliative Care (AREA)
  • Psychiatry (AREA)
  • Psychology (AREA)
  • General Health & Medical Sciences (AREA)
  • Animal Behavior & Ethology (AREA)
  • Surgery (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Molecular Biology (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Mathematical Physics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Traffic Control Systems (AREA)
PCT/JP2023/011366 2023-03-23 2023-03-23 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム Ceased WO2024195072A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
PCT/JP2023/011366 WO2024195072A1 (ja) 2023-03-23 2023-03-23 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム
DE112023005629.3T DE112023005629T5 (de) 2023-03-23 2023-03-23 Insassenzustand-Bestimmungsvorrichtung, Insassenzustand- Bestimmungssystem, Insassenzustand-Bestimmungsverfahren und Insassenzustand-Bestimmungsprogramm
JP2025508038A JP7696527B2 (ja) 2023-03-23 2023-03-23 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2023/011366 WO2024195072A1 (ja) 2023-03-23 2023-03-23 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム

Publications (1)

Publication Number Publication Date
WO2024195072A1 true WO2024195072A1 (ja) 2024-09-26

Family

ID=92841453

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/011366 Ceased WO2024195072A1 (ja) 2023-03-23 2023-03-23 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム

Country Status (3)

Country Link
JP (1) JP7696527B2 (https=)
DE (1) DE112023005629T5 (https=)
WO (1) WO2024195072A1 (https=)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008171065A (ja) * 2007-01-09 2008-07-24 Denso Corp 眠気検知装置
JP2011048531A (ja) * 2009-08-26 2011-03-10 Aisin Seiki Co Ltd 眠気検出装置、眠気検出方法、及びプログラム
JP2016115117A (ja) * 2014-12-15 2016-06-23 アイシン精機株式会社 判定装置および判定方法
JP2019087170A (ja) * 2017-11-10 2019-06-06 アルパイン株式会社 瞼開閉判定装置および眠気検知装置
WO2020174601A1 (ja) * 2019-02-27 2020-09-03 三菱電機株式会社 覚醒度推定装置、自動運転支援装置および覚醒度推定方法
WO2023017595A1 (ja) * 2021-08-12 2023-02-16 三菱電機株式会社 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008171065A (ja) * 2007-01-09 2008-07-24 Denso Corp 眠気検知装置
JP2011048531A (ja) * 2009-08-26 2011-03-10 Aisin Seiki Co Ltd 眠気検出装置、眠気検出方法、及びプログラム
JP2016115117A (ja) * 2014-12-15 2016-06-23 アイシン精機株式会社 判定装置および判定方法
JP2019087170A (ja) * 2017-11-10 2019-06-06 アルパイン株式会社 瞼開閉判定装置および眠気検知装置
WO2020174601A1 (ja) * 2019-02-27 2020-09-03 三菱電機株式会社 覚醒度推定装置、自動運転支援装置および覚醒度推定方法
WO2023017595A1 (ja) * 2021-08-12 2023-02-16 三菱電機株式会社 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム

Also Published As

Publication number Publication date
DE112023005629T5 (de) 2026-02-26
JPWO2024195072A1 (https=) 2024-09-26
JP7696527B2 (ja) 2025-06-20

Similar Documents

Publication Publication Date Title
US11640717B2 (en) Image analysis technologies for identifying abnormal vehicle conditions
US20160272217A1 (en) Two-step sleepy driving prevention apparatus through recognizing operation, front face, eye, and mouth shape
JP2016103249A (ja) 運転支援装置および運転支援方法
US20160159217A1 (en) System and method for determining drowsy state of driver
JP7019394B2 (ja) 視認対象検知装置、視認対象検知方法、およびプログラム
JP6973928B2 (ja) 瞼開閉判定装置および眠気検知装置
JP2018511376A (ja) 車両のドライバの疲労状態及び／又は睡眠状態を判定する方法及び装置
JP7550879B2 (ja) 睡眠検出装置及び睡眠検出システム
JP7109649B2 (ja) 覚醒度推定装置、自動運転支援装置および覚醒度推定方法
JP7696527B2 (ja) 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及び乗員状態判定プログラム
JP2020194229A (ja) 運転者状態判定装置、運転者状態判定方法及び運転者状態判定プログラム
JPWO2023017595A5 (https=)
JP7175381B2 (ja) 覚醒度推定装置、自動運転支援装置および覚醒度推定方法
JP2009159568A (ja) 異常検出装置および方法、並びに、プログラム
JP7098265B2 (ja) 運転支援装置
WO2021210041A1 (ja) 顔検出装置および顔検出方法
JP6847323B2 (ja) 視線検出装置および視線検出方法
WO2024047856A1 (ja) 眠気推定装置および眠気推定方法
US20230311899A1 (en) Driving state determination apparatus, method, and computer readable medium
US11200417B2 (en) Determination apparatus and method to enhance accuracy in determining the presence of an erroneously detected eyelid
JP7594843B2 (ja) 乗員状態検知装置および乗員状態検知方法
JP2021043637A (ja) 運転者モニタ装置
WO2019030855A1 (ja) 運転不能状態判定装置および運転不能状態判定方法
JP7505660B2 (ja) 乗員監視装置、乗員監視方法、及び乗員監視プログラム
JP7450715B2 (ja) 運転不能判定装置および運転不能判定方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23928651

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2025508038

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 112023005629

Country of ref document: DE

WWP Wipo information: published in national office

Ref document number: 112023005629

Country of ref document: DE

122 Ep: pct application non-entry in european phase

Ref document number: 23928651

Country of ref document: EP

Kind code of ref document: A1