WO2022009401A1 - 乗員状態検出装置および乗員状態検出方法 - Google Patents

乗員状態検出装置および乗員状態検出方法 Download PDF

Info

Publication number
WO2022009401A1
WO2022009401A1 PCT/JP2020/026913 JP2020026913W WO2022009401A1 WO 2022009401 A1 WO2022009401 A1 WO 2022009401A1 JP 2020026913 W JP2020026913 W JP 2020026913W WO 2022009401 A1 WO2022009401 A1 WO 2022009401A1
Authority
WO
WIPO (PCT)
Prior art keywords
occupant
temperature
detection unit
hand
movement
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/026913
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
浩隆 坂本
俊之 八田
信太郎 渡邉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to JP2022534604A priority Critical patent/JP7286022B2/ja
Priority to PCT/JP2020/026913 priority patent/WO2022009401A1/ja
Priority to DE112020007404.8T priority patent/DE112020007404T5/de
Priority to US17/928,015 priority patent/US12240469B2/en
Publication of WO2022009401A1 publication Critical patent/WO2022009401A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W40/09—Driving style or behaviour
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/01—Measuring temperature of body parts ; Diagnostic temperature sensing, e.g. for malignant or inflamed tissue
    • A61B5/015—By temperature mapping of body part
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/103—Measuring devices for testing the shape, pattern, colour, size or movement of the body or parts thereof, for diagnostic purposes
    • A61B5/11—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb
    • A61B5/1126—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb using a particular sensing technique
    • A61B5/1128—Measuring movement of the entire body or parts thereof, e.g. head or hand tremor or mobility of a limb using a particular sensing technique using image analysis
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/18—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/20—Movements or behaviour, e.g. gesture recognition
    • G06V40/28—Recognition of hand or arm movements, e.g. recognition of deaf sign language
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/09626—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/0818—Inactivity or incapacity of driver
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
    • B60W2420/403—Image sensing, e.g. optical camera
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/221—Physiology, e.g. weight, heartbeat, health or special needs
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/223—Posture, e.g. hand, foot, or seat position, turned or inclined
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/229—Attention level, e.g. attentive to driving, reading or sleeping
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/26—Incapacity

Definitions

  • This disclosure relates to an occupant condition detection device and an occupant condition detection method.
  • Patent Document 1 discloses a driver arousal degree test device that determines the arousal degree of a driver such as a vehicle based on the facial skin temperature, the finger skin temperature, and the pulse rate.
  • the driver arousal test device obtains the skin temperature of the driver's fingers from the finger temperature sensor located on the peripheral edge of the steering wheel at a position where the driver's fingers come into contact with the driver while holding the steering wheel. do.
  • This disclosure is made to solve the above-mentioned problems, and it is possible to estimate the arousal degree of the person based on the temperature of the person's hand regardless of the position where the person holds the steering wheel. It is an object of the present invention to provide a occupant condition detection device.
  • the occupant state detection device includes an image acquisition unit that acquires an image of an occupant image, and a temperature image acquisition unit that acquires a temperature image that represents the temperature of the surface of the occupant's body measured in a non-contact manner.
  • a motion detection unit that detects the movement of the occupant based on the captured image acquired by the captured image acquisition unit, and a temperature detection unit that detects the temperature of the occupant's hand based on the temperature image acquired by the temperature image acquisition unit. It is provided with an arousal degree estimation unit that estimates the arousal degree of the occupant based on the movement of the occupant detected by the motion detection unit and the temperature of the occupant's hand detected by the temperature detection unit.
  • the arousal degree of the person can be estimated based on the temperature of the person's hand regardless of the position where the person holds the steering wheel.
  • FIG. 1 shows the configuration example of the occupant state detection apparatus which concerns on Embodiment 1.
  • FIG. 1 the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face detected by the occupant detection unit on the captured image. It is a figure for demonstrating the image which showed an example.
  • it is a figure which shows an example of the image of the temperature image after the temperature detection part performs the alignment with the image taken after the position is given.
  • FIG. 8A and 8B are diagrams showing an example of the hardware configuration of the occupant state detection device according to the first embodiment.
  • FIG. 1 is a diagram showing a configuration example of the occupant state detection device 1 according to the first embodiment.
  • the occupant state detection device 1 according to the first embodiment is mounted on a vehicle and estimates the arousal degree of the occupant. In the first embodiment, it is assumed that the occupant is the driver of the vehicle.
  • the occupant state detection device 1 is connected to the image pickup device 2 and the temperature acquisition device 3.
  • the image pickup device 2 is a camera or the like installed for the purpose of monitoring the inside of the vehicle.
  • the image pickup device 2 is installed at least in a position where the upper body of the occupant can be imaged.
  • the image pickup device 2 may be shared with, for example, a so-called “driver monitoring system (DMS)".
  • DMS driver monitoring system
  • the image pickup device 2 outputs the captured image (hereinafter referred to as “captured image”) to the occupant state detection device 1.
  • the temperature acquisition device 3 is an infrared camera having a temperature measurement function, an infrared array sensor, or the like.
  • the temperature acquisition device 3 is installed at a position where the temperature of the upper body of the occupant including at least the occupant's hand can be measured in a non-contact manner.
  • the temperature acquisition device 3 outputs an image showing the measured temperature (hereinafter referred to as “temperature image”) to the occupant state detection device 1.
  • the temperature image includes temperature information for each pixel. The larger the pixel value, the higher the temperature. Further, in general, the temperature image output from the temperature acquisition device 3 has a low frame rate.
  • the occupant state detection device 1 detects the movement of the occupant based on the captured image acquired from the image pickup device 2. Further, the occupant state detection device 1 detects the temperature of the occupant based on the temperature image acquired from the temperature acquisition device 3. The occupant state detection device 1 estimates the arousal degree of the occupant based on the detected movement of the occupant and the temperature of the occupant. Details of the movement of the occupant and the temperature of the occupant detected by the occupant state detection device 1 will be described later. Further, the details of the method of estimating the arousal degree of the occupant by the occupant state detection device 1 will be described later.
  • the occupant state detection device 1 includes a captured image acquisition unit 11, a temperature image acquisition unit 12, an occupant detection unit 13, a motion detection unit 14, a temperature detection unit 15, an arousal degree estimation unit 16, and an output unit 17.
  • the captured image acquisition unit 11 acquires the captured image output from the image pickup device 2.
  • the captured image acquisition unit 11 outputs the acquired captured image to the occupant detection unit 13.
  • the temperature image acquisition unit 12 acquires the temperature image output from the temperature acquisition device 3.
  • the temperature image acquisition unit 12 outputs the acquired temperature image to the temperature detection unit 15.
  • the occupant detection unit 13 detects information about the occupant (hereinafter referred to as "occupant information") based on the captured image acquired by the captured image acquisition unit 11. Specifically, the occupant detection unit 13 detects the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face. If the occupant detection unit 13 detects the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face using known image recognition technology. good.
  • the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face detected by the occupant detection unit 13 are shown in the captured image.
  • the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face are represented by the coordinates on the captured image.
  • the occupant detection unit 13 detects the positions of the left and right ends of each of the left and right eyes of the occupant, one point on the upper eyelid, and one point on the lower eyelid as the positions of the occupant's eyes. Further, for example, the occupant detection unit 13 detects the left and right corners of the occupant's mouth, one point on the upper lip, and one point on the lower lip as the positions of the occupant's mouth. Further, for example, the occupant detection unit 13 detects the positions of the tips of the left and right shoulders of the occupant as the positions of the occupant's body.
  • the occupant detection unit 13 detects the position of one point on the base of the thumb and one point on the base of the little finger as the position of the occupant's hand. Further, for example, the occupant detection unit 13 detects the position of the tip of the occupant's jaw as the position of the occupant's face.
  • FIG. 2 shows the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, and the position of the occupant's hand detected by the occupant detection unit 13 on the captured image in the first embodiment.
  • FIG. 2 is a diagram for explaining an image showing an example of the position of the occupant's face.
  • the eight points indicated by 201 indicate the positions of the occupant's eyes detected by the occupant detection unit 13, here, the left and right ends of the eyes, one point on the upper eyelid, or one point on the lower eyelid. There is. Further, in FIG.
  • the four points shown by 202 indicate the positions of the occupant's mouth detected by the occupant detection unit 13, here, both ends of the corner of the mouth, one point on the upper lip, or one point on the lower lip. ..
  • one point shown by 203 indicates the position of the occupant's face detected by the occupant detection unit 13, here, the position of the tip of the jaw.
  • the two points shown by 204 indicate the positions of the body detected by the occupant detection unit 13, here, both ends of the shoulder.
  • the four points shown by 205 indicate the positions of the hand detected by the occupant detection unit 13, here, one point on the base of the thumb or one point on the base of the little finger.
  • any point in the area showing the occupant's eyes, the occupant's mouth, the occupant's body, the occupant's hand, or the occupant's face is the position of the occupant's eyes.
  • the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face can be appropriately set.
  • the occupant detection unit 13 provides information indicating the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face in the motion detection unit 14 and the temperature detection unit.
  • the occupant detection unit 13 has a reference to the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face. , The position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face.
  • the post-captured image is output to the motion detection unit 14 and the temperature detection unit 15.
  • the motion detection unit 14 detects the motion of the occupant based on the captured image acquired by the captured image acquisition unit 11. Specifically, the motion detection unit 14 detects the motion of the occupant based on the position-assigned captured image output from the occupant detection unit 13. In the first embodiment, the motion detection unit 14 detects the movement of the occupant's eyes, the occupant's mouth, the occupant's body, the occupant's hand, or the occupant's face as the occupant's movement. do. When the motion detection unit 14 acquires the image after the position is assigned, the motion detection unit 14 stores the acquired image after the position in the storage unit (not shown) in association with the information regarding the acquisition date and time of the image after the position is assigned. It is assumed that there is.
  • the storage unit may be provided in the occupant state detecting device 1, or may be provided in a place outside the occupant state detecting device 1 where the occupant state detecting device 1 can be referred.
  • the motion detection unit 14 determines the movement of the occupant's eyes and the occupant's mouth based on the post-positioning captured image output from the occupant detection unit 13 and the past post-positioning captured image stored in the storage unit. Detects movements, occupant body movements, occupant hand movements, or occupant face movements.
  • the motion detection unit 14 stores the captured image after the position is assigned in the storage unit, but this is only an example.
  • the occupant detection unit 13 outputs the captured image to the motion detection unit 14 and stores it in the storage unit after the position is assigned, and the motion detection unit 14 refers to the storage unit and stores the image in the occupant detection unit 13.
  • the captured image may be acquired after the position is assigned.
  • the motion detection unit 14 detects that the occupant is closing his eyes as the movement of the occupant's eyes based on the position of the occupant's eyes in the image captured after the position is assigned. Specifically, the motion detection unit 14 detects that the occupant has closed his eyes when, for example, the distance between one point on the upper eyelid and one point on the lower eyelid is within a preset threshold value. do. Further, for example, the motion detection unit 14 detects the blink of the occupant as the movement of the occupant's eyes. Specifically, the motion detection unit 14 detects the blink of an occupant based on, for example, a change in the distance between one point on the upper eyelid and one point on the lower eyelid.
  • the motion detection unit 14 uses a known technique for detecting the opening / closing or blinking of a person's eyes based on an image, and the occupant is closing his eyes or the occupant. It is only necessary to detect that the person blinks.
  • the motion detection unit 14 detects that the occupant has yawned as the movement of the occupant's mouth based on the position of the occupant's mouth in the image captured after the position is assigned. Specifically, in the motion detection unit 14, for example, the distance between one point on the upper lip of the occupant and one point on the lower lip of the occupant is equal to or larger than a preset threshold value (hereinafter referred to as “opening determination threshold value”). When separated, it detects that the occupant has necked.
  • a preset threshold value hereinafter referred to as “opening determination threshold value”.
  • a state in which the distance between one point on the upper lip of the occupant and one point on the lower lip of the occupant is separated by an opening determination threshold value or more is a preset time (hereinafter, "opening determination time"). If it continues for more than that, it may be possible to detect that the occupant has yawned.
  • opening determination time a preset time
  • the motion detection unit 14 may detect that the occupant has yawned by using a known technique for detecting a person's yawning based on an image.
  • the motion detection unit 14 detects that the occupant's body is swaying as the occupant's body movement based on the position of the occupant's body in the captured image after the position is assigned. Specifically, the motion detection unit 14 has changed, for example, to a position where the position of the occupant's body in the captured image after the position is assigned is separated by a preset threshold value (hereinafter referred to as “body movement determination threshold value”) or more. If so, it detects that the occupant's body is staggering.
  • body movement determination threshold value a preset threshold value
  • the motion detection unit 14 is, for example, when the position of the occupant's body in the captured image after the position is assigned changes by a preset threshold value (hereinafter referred to as “body movement determination delta threshold value”) or more per unit time, the occupant. You may want to detect that your body is staggering. At this time, the motion detection unit 14 also detects the degree of wobbling of the occupant's body.
  • the degree of wobbling of the occupant's body is, for example, the angle at which the origin on the image captured after the position is assigned or the line connecting the predetermined reference point on the image captured after the position is changed and the position of the occupant's body changes. (Hereinafter referred to as "body wobble angle").
  • the motion detection unit 14 is, for example, at the first shoulder position or the second shoulder position. It may be detected that the occupant's body is swaying when any of them changes to a position more than the body movement determination threshold, and both the first shoulder position and the second shoulder position are body movement determination. It may be detected that the occupant has staggered when the vehicle changes to a position farther than the threshold. Further, for example, in the movement detection unit 14, the occupant's body sways when either the first shoulder position or the second shoulder position changes by the body movement determination delta threshold value or more per unit time.
  • the motion detection unit 14 uses a known technique for detecting the wobbling of the human body based on an image to detect the wobbling of the occupant's body and the wobbling angle. It should be detected.
  • the motion detection unit 14 detects the motion of the occupant's hand based on the position of the occupant's hand in the captured image after the position is assigned. Specifically, in the motion detection unit 14, for example, the position of one point on the base of the thumb of the occupant or the position of one point on the base of the little finger of the occupant is set to a preset threshold value (hereinafter referred to as "manual threshold"). It is called “judgment threshold".) When the position changes until the position is separated by more than that, it is detected that the occupant's hand has moved.
  • manual threshold a preset threshold value
  • the amount of change per unit time at the position of one point on the base of the thumb of the occupant or the position of one point on the base of the little finger of the occupant is a preset threshold value (hereinafter,).
  • the threshold value for manual determination is exceeded
  • the movement detection unit 14 is located at one point on the base of the occupant's thumb (hereinafter referred to as "thumb point") or at one point on the base of the occupant's little finger (hereinafter referred to as "pinkie point").
  • the motion detection unit 14 moves to the occupant's hand when the amount of change per unit time of either the position of the occupant's thumb point or the position of the occupant's small finger point exceeds the manual determination delta threshold value.
  • the occupant's hand may be detected when the change amount per unit time of both the position of the occupant's thumb point and the position of the occupant's small finger point exceeds the delta threshold for manual judgment.
  • the motion detection unit 14 may detect, for example, that the occupant's hand has moved when the change in position as described above is either the right hand or the left hand.
  • the above-mentioned example is only an example, and the motion detection unit 14 may detect the motion of the occupant's hand by using a known technique for detecting the motion of the human hand based on the image.
  • the motion detection unit 14 detects that the occupant's head is swaying as the occupant's face movement based on the position of the occupant's face in the captured image after the position is assigned. Specifically, the motion detection unit 14 has changed, for example, to a position where the position of the occupant's face in the captured image after the position is assigned is separated by a preset threshold value (hereinafter referred to as “face motion determination threshold value”) or more. If so, detect that the occupant's head is swaying.
  • face motion determination threshold value a preset threshold value
  • the motion detection unit 14 is, for example, when the position of the occupant's face in the captured image after the position is assigned changes by a preset threshold value (hereinafter referred to as “face motion determination delta threshold value”) or more per unit time, the occupant. It may be possible to detect that the face of the person is wobbling. At this time, the motion detection unit 14 also detects the degree of wobbling of the occupant's head.
  • the degree of wobbling of the occupant's head is, for example, the origin of the image captured after the position is assigned, or the angle at which the line connecting the predetermined reference point on the image captured after the position is assigned and the position of the occupant's face changes ( Hereinafter, it is referred to as "head wobbling angle").
  • the motion detection unit 14 uses a known technique for detecting the motion of a person's face based on an image to determine that the occupant's head is swaying and the head swaying angle. It should be detected.
  • the motion detection unit 14 provides the arousal degree estimation unit 16 with information indicating whether or not the motion of the occupant has been detected based on the captured image acquired by the captured image acquisition unit 11 (hereinafter referred to as “motion detection notification information”). Output. At this time, the motion detection unit 14 outputs motion detection notification information in association with the information regarding the acquisition date and time of the captured image.
  • the motion detection unit 14 may set the acquisition date and time of the captured image to, for example, the imaging date and time of the captured image attached to the captured image.
  • the movement detection notification information detects whether or not the movement of the occupant's eyes is detected, whether or not the movement of the occupant's mouth is detected, whether or not the movement of the occupant's body is detected, and whether or not the movement of the occupant's hand is detected. It contains information on whether or not the vehicle has been used and whether or not the movement of the occupant's face has been detected.
  • the information on whether or not the occupant's eye movement was detected is, for example, whether or not the occupant detected that the occupant was closing his eyes and whether or not the occupant blinked. It is information on whether or not.
  • the information on whether or not the movement of the occupant's mouth is detected is, for example, information on whether or not the occupant has detected yawning.
  • the information on whether or not the movement of the occupant's body is detected is, for example, information on whether or not the occupant's body is detected to be swaying, and information on whether or not the occupant's body is swaying is detected. If so, it is information on the body wobble angle.
  • the information on whether or not the movement of the occupant's hand is detected includes, for example, in addition to the information on whether or not the movement of the occupant's hand is detected, when the movement of the occupant's hand is detected, it is the right hand or the left hand. It may include information that can identify whether or not.
  • the information on whether or not the movement of the occupant's face is detected is, for example, the information on whether or not the occupant's head is swaying, and the information on whether or not the occupant's head is swaying is detected. It is the information of the angle.
  • the temperature detection unit 15 detects the temperature of the occupant based on the position-imparted captured image output from the occupant detection unit 13 and the temperature image acquired by the temperature image acquisition unit 12.
  • the temperature of the occupant means the temperature of the occupant's hand and the temperature of the face. That is, the temperature detection unit 15 detects the temperature of the occupant's hand and the temperature of the face based on the position-imparted captured image output from the occupant detection unit 13 and the temperature image acquired by the temperature image acquisition unit 12. .. Specifically, first, the temperature detection unit 15 aligns the captured image and the temperature image after the position is assigned.
  • the alignment of the captured image after positioning and the temperature image performed by the temperature detection unit 15 associates the captured image after positioning and the temperature image with each other indicating the same spatial position.
  • the temperature detection unit 15 is based on a predetermined installation position of the image pickup device 2 and a temperature acquisition device 3. After the position is given, the position of the captured image and the temperature image can be aligned. Then, when the temperature detection unit 15 aligns the captured image and the temperature image after the position is assigned, which pixel in the temperature image is the pixel indicating the temperature of the occupant's hand or the temperature of the occupant's face. , Can be identified.
  • FIG. 3 is a diagram showing an example of an image of a temperature image after the temperature detection unit 15 has aligned with the captured image after the position has been assigned in the first embodiment.
  • FIG. 3 for convenience, the position of the occupant's eyes (see 201 in FIG. 3), the position of the occupant's mouth (see 202 in FIG.
  • the temperature image is shown reflecting the position (see 203 in FIG. 3), the position of the occupant's body (see 204 in FIG. 3), and the position of the occupant's hand (see 205 in FIG. 3).
  • the occupant detection unit 13 determines the position of the occupant's face, the position of the occupant's mouth, the position of the occupant's eyes, the position of the occupant's hand, and the position of the occupant's hand, as shown in FIG. It is assumed that the image was captured after the position was assigned, which was output from the occupant detection unit 13 when the position of the occupant's body was detected.
  • the temperature detection unit 15 passes through, for example, the position of the occupant's face, in other words, the position of the occupant's jaw, on the temperature image, and is the smallest circle including the position of the occupant's eyes and the position of the occupant's mouth.
  • the range indicated by is defined as a range for detecting the temperature of the occupant's face (hereinafter referred to as "face temperature detection range").
  • face temperature detection range a range for detecting the temperature of the occupant's face
  • the "minimum” is not limited to the “minimum”, and may be substantially the minimum.
  • the above-mentioned method for setting the face temperature detection range is only an example. The range in which the temperature detection unit 15 sets the face temperature detection range can be appropriately set.
  • the temperature detection unit 15 detects, for example, the average value of the pixel values of the pixels whose at least a part is included in the face temperature detection range as the temperature of the occupant's face.
  • the method of detecting the temperature of the occupant's face as described above is only an example, and the temperature detection unit 15 may detect the temperature of the occupant's face by another method.
  • the temperature detection unit 15 refers to a pixel having the largest area within the face temperature detection range (hereinafter referred to as “face selection pixel”) among a plurality of pixels whose at least a part is included in the face temperature detection range. It may be selected and the pixel value of the face selection pixel may be detected as the temperature of the occupant's face.
  • face selection pixels the temperature detection unit 15 detects, for example, the pixel value of any face selection pixel among the plurality of face selection pixels as the temperature of the occupant's face.
  • the temperature detection unit 15 passes through, for example, the position of the occupant's hand, in other words, one point on the base of the occupant's thumb and one point on the base of the little finger on the temperature image, and these two points are defined as the diameter.
  • the range indicated by the circle is defined as a range for detecting the temperature of the occupant's hand (hereinafter referred to as "hand temperature detection range").
  • the temperature detection unit 15 sets the hand temperature detection range for each of the right hand and the left hand.
  • the above-mentioned method for setting the hand temperature detection range is only an example.
  • the range in which the temperature detection unit 15 sets the manual temperature detection range can be appropriately set.
  • the temperature detection unit 15 detects, for example, the average value of the pixel values of the pixels whose at least a part is included in the hand temperature detection range as the temperature of the occupant's hand.
  • the method of detecting the temperature of the occupant's hand as described above is only an example, and the temperature detection unit 15 may detect the temperature of the occupant's hand by another method.
  • the temperature detection unit 15 refers to a pixel having the largest area within the hand temperature detection range (hereinafter referred to as “hand selection pixel”) among a plurality of pixels whose at least a part is included in the hand temperature detection range. It may be selected and the pixel value of the hand-selected pixel may be detected as the temperature of the occupant's hand.
  • the temperature detection unit 15 detects, for example, the pixel value of any hand-selected pixel among the plurality of hand-selected pixels as the temperature of the occupant's hand.
  • the temperature detection unit 15 When the temperature detection unit 15 detects the temperature of the occupant's hand and the temperature of the face, the temperature detection unit 15 obtains the detected information on the temperature of the occupant's hand and the temperature of the face (hereinafter referred to as "temperature detection information") when the temperature image is acquired. It is output to the arousal degree estimation unit 16 in association with the information related to.
  • the temperature detection unit 15 may set the acquisition date and time of the temperature image to, for example, the creation date and time of the temperature image attached to the temperature image.
  • the arousal degree estimation unit 16 estimates the arousal degree of the occupant based on the movement of the occupant detected by the motion detection unit 14 and the temperature of the occupant's hand and face detected by the temperature detection unit 15. In the first embodiment, as an example, the arousal degree estimation unit 16 sets the arousal degree to a degree indicated by five stages of "level 1" to "level 5". The greater the degree of arousal, the higher the degree to which the occupant is awake. The method by which the arousal degree estimation unit 16 estimates the arousal degree of the occupant will be described in detail below.
  • the arousal degree estimation unit 16 first determines whether or not the motion detection unit 14 has detected the movement of the occupant's hand. Specifically, the arousal degree estimation unit 16 determines whether or not the motion detection notification information including the information indicating that the hand movement has been detected is output from the motion detection unit 14. When the motion detection unit 14 detects the movement of the occupant's hand, the arousal degree estimation unit 16 estimates the occupant's arousal degree based on the movement of the occupant's hand. Specifically, the arousal degree estimation unit 16 estimates that the occupant is in an awake state because of the movement of the occupant's hand, and sets the arousal degree of the occupant to "level 5". It can be said that there is a high possibility that the occupant is awake if there is a movement of the occupant's hand.
  • the arousal degree estimation unit 16 detects the occupant's movement detected by the motion detection unit 14 and the temperature of the occupant's hand detected by the temperature detection unit 15. And the occupant's arousal level is estimated based on the temperature of the face. Specifically, the arousal degree estimation unit 16 according to a rule constructed based on a preset condition (hereinafter referred to as "determination condition") (hereinafter referred to as "awakening degree estimation rule”), the arousal degree of the occupant. To estimate. The arousal degree estimation rule is pre-constructed by the combination of the logical sum or the logical product of the judgment conditions. If there is no movement of the occupant's hand, it is highly possible that the occupant's arousal level is reduced.
  • the judgment conditions and the arousal degree estimation rule will be described with specific examples.
  • the following conditions (A) to (E) are defined as the determination conditions.
  • the number of times "the occupant's head is swaying" at an angle of 20 degrees or more in the past 5 minutes is 2 or more
  • the temperature of the occupant's hand is within -5 ° C with respect to the temperature of the occupant's face.
  • (A) to (D) are conditions for determining the movement of the occupant that appears when the occupant feels drowsy. When (A) to (D) are satisfied, it can be said that the occupant feels drowsy. When (D) has the above-mentioned contents, it can be determined from the above-mentioned head-swaying angle that the occupant's head is swaying at an angle of 20 degrees or more.
  • (E) when a person feels drowsy, the blood flow in the peripheral part such as the fingertip increases, and the temperature of the peripheral part rises to become close to the face or deep body temperature. It is a condition for judging that. If (E) is satisfied, it can be said that the occupant feels drowsy.
  • the following rules (1) to (6) are constructed as the arousal degree estimation rules.
  • the arousal degree estimation unit 16 acquires the motion detection notification information from the motion detection unit 14, it is assumed that the motion detection notification information is stored in the storage unit. Further, when the arousal degree estimation unit 16 acquires the temperature detection information from the temperature detection unit 15, it is assumed that the temperature detection information is stored in the storage unit.
  • the arousal degree estimation unit 16 estimates the arousal degree of the occupant according to the arousal degree estimation rule based on the motion detection notification information and the temperature detection information stored in the storage unit.
  • the arousal degree estimation unit 16 stores the motion detection notification information and the temperature detection information, but this is only an example.
  • the motion detection unit 14 stores the motion detection notification information in the storage unit
  • the temperature detection unit 15 stores the temperature detection information in the storage unit
  • the arousal degree estimation unit 16 refers to the storage unit. Then, the arousal degree of the occupant may be estimated.
  • the above-mentioned determination condition is only an example.
  • a determination condition related to body movement may be added, such as "the number of times the occupant's body is swaying at an angle of 20 degrees or more is 2 or more in the past 5 minutes". ..
  • the content of the determination condition is determined experimentally in advance.
  • the arousal degree estimation rule is constructed in advance using the determination conditions determined experimentally in advance.
  • the arousal degree estimation unit 16 first determines whether or not the movement of the occupant's hand is detected, and if the movement of the occupant's hand is detected, the occupant is in an awake state, in other words, the said person.
  • the occupant's arousal level is estimated to be "level 5".
  • the arousal degree estimation unit 16 does not estimate the arousal degree of the occupant according to the arousal degree estimation rule, in other words, the arousal degree of the occupant using the temperature of the hand and the temperature of the face.
  • the movement of the occupant's hand is detected by the motion detection unit 14, so that the occupant is in the awake state in the arousal degree estimation unit 16.
  • the occupant's arousal level is not estimated using the occupant's hand temperature and face temperature.
  • the arousal degree estimation unit 16 estimates the occupant's arousal degree using the movement of the occupant, the temperature of the occupant's hand, and the temperature of the face.
  • the temperature image output from the temperature acquisition device 3 has a low frame rate.
  • the temperature detection unit 15 can accurately detect the temperature of the occupant's hand and the temperature of the face from the temperature image when the occupant's hand does not move, as compared with the case where the occupant's hand moves.
  • the arousal degree estimation unit 16 estimates the arousal degree of the occupant in the order as described above, and estimates the arousal degree of the occupant using the temperature of the occupant's hand and the temperature of the face detected based on the temperature image. Can be done rationally.
  • the arousal degree estimation unit 16 outputs information regarding the estimated arousal degree of the occupant (hereinafter referred to as “awakening degree information”) to the output unit 17.
  • the arousal degree information includes, for example, information on the level of the arousal degree determined by the arousal degree estimation unit 16.
  • the output unit 17 outputs the arousal degree information output from the arousal degree estimation unit 16. Specifically, the output unit 17 outputs the arousal degree information to, for example, an alarm output control device (not shown), an air conditioning control device (not shown), or an automatic operation control device (not shown).
  • the alarm output control device, the air conditioning control device, and the automatic driving control device are mounted on the vehicle.
  • the alarm output control device outputs an alarm for alerting the occupants in the vehicle to drowsiness.
  • the air-conditioning control device controls the air-conditioning so as to suppress drowsiness when the arousal degree information is output from the arousal degree estimation unit 16.
  • the automatic driving control device switches the driving control method of the vehicle from manual driving to automatic driving.
  • the vehicle has an automatic driving function. Even if the vehicle has an automatic driving function, the driver can manually drive the vehicle by himself / herself.
  • FIG. 4 is a flowchart for explaining the operation of the occupant state detection device 1 according to the first embodiment.
  • the captured image acquisition unit 11 acquires the captured image output from the image pickup device 2 (step ST401).
  • the captured image acquisition unit 11 outputs the acquired captured image to the occupant detection unit 13.
  • the temperature image acquisition unit 12 acquires the temperature image output from the temperature acquisition device 3 (step ST402).
  • the temperature image acquisition unit 12 outputs the acquired temperature image to the temperature detection unit 15.
  • the occupant detection unit 13 detects occupant information based on the captured image acquired by the captured image acquisition unit 11 in step ST401 (step ST403). Specifically, the occupant detection unit 13 detects the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face. The occupant detection unit 13 provides information indicating the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face in the motion detection unit 14 and the temperature detection unit. Output to 15. Specifically, the occupant detection unit 13 outputs the image captured after the position is assigned to the motion detection unit 14 and the temperature detection unit 15.
  • the motion detection unit 14 detects the motion of the occupant based on the captured image acquired by the captured image acquisition unit 11 in step ST401 (step ST404). Specifically, the motion detection unit 14 detects the motion of the occupant based on the position-assigned captured image output from the occupant detection unit 13 in step ST403. The motion detection unit 14 outputs the motion detection notification information to the arousal degree estimation unit 16 in association with the information regarding the acquisition date and time of the captured image.
  • the temperature detection unit 15 detects the temperature of the occupant based on the position-assigned image captured by the occupant detection unit 13 output in step ST403 and the temperature image acquired by the temperature image acquisition unit 12 in step ST402. (Step ST405). Specifically, the temperature detection unit 15 determines the temperature of the occupant's hand and the temperature of the face based on the position-imparted captured image output from the occupant detection unit 13 and the temperature image acquired by the temperature image acquisition unit 12. Is detected. When the temperature detection unit 15 detects the temperature of the occupant's hand and the temperature of the face, the temperature detection unit 15 outputs the temperature detection information to the arousal degree estimation unit 16 in association with the information regarding the acquisition date and time of the temperature image.
  • the arousal degree estimation unit 16 first determines whether or not the motion detection unit 14 has detected the movement of the occupant's hand in step ST404 (step ST406). Specifically, the arousal degree estimation unit 16 determines whether or not the motion detection notification information including the information indicating that the hand movement has been detected is output from the motion detection unit 14. When the motion detection unit 14 detects the movement of the occupant's hand (when “YES” in step ST406), the arousal degree estimation unit 16 estimates the occupant's arousal degree based on the movement of the occupant's hand.
  • the arousal degree estimation unit 16 estimates that the occupant is in an awake state because of the movement of the occupant's hand, and sets the arousal degree of the occupant to "level 5" (step ST407).
  • the arousal degree estimation unit 16 outputs the arousal degree information to the output unit 17. Then, the operation of the occupant state detection device 1 proceeds to step ST409.
  • the arousal degree estimation unit 16 detects the occupant's movement in step ST404.
  • the degree of arousal of the occupant is estimated based on the temperature of the occupant's hand and the temperature of the face detected by the temperature detection unit 15 in step ST405 (step ST408).
  • the arousal degree estimation unit 16 estimates the arousal degree of the occupant according to the arousal degree estimation rule constructed based on the determination condition.
  • the arousal degree estimation unit 16 outputs the arousal degree information to the output unit 17. Then, the operation of the occupant state detection device 1 proceeds to step ST409.
  • step ST409 the output unit 17 outputs the arousal degree information output from the arousal degree estimation unit 16 (step ST409).
  • step ST401 and step ST402 In the flowchart of FIG. 4, it is assumed that the occupant state detection device 1 is operated in the order of step ST401 and step ST402, but this is only an example.
  • the order of the operation of step ST401 and the operation of step ST402 may be reversed or may be performed in parallel. Further, the operation of step ST402 may be performed by the time the operation of step ST405 is performed. Further, in the flowchart of FIG. 4, it is assumed that the occupant state detection device 1 is operated in the order of step ST404 and step ST405, but this is only an example.
  • the order of the operation of step ST404 and the operation of step ST405 may be reversed or may be performed in parallel.
  • the temperature of the occupant's hand used for estimating the arousal degree of the occupant of the vehicle comes into contact with the driver's finger at the peripheral edge of the steering wheel while the driver holds the steering wheel. It was obtained from the hand temperature sensor located at the position.
  • the temperature of the hand cannot be acquired if the position where the occupant grips the steering wheel shifts.
  • the method of acquiring the temperature of the occupant's hand as disclosed in the prior art has been a method that puts a load on the occupant.
  • the occupant state detection device 1 acquires an captured image of the occupant and a temperature image showing the temperature of the surface of the occupant's body measured in a non-contact manner, and detects the occupant based on the captured image.
  • the occupant's arousal level is estimated based on the movement and the temperature of the occupant's hand and face detected based on the temperature image.
  • the occupant state detection device 1 can estimate the arousal degree of the occupant based on the temperature of the occupant's hand and the temperature of the face regardless of the position where the occupant holds the handle.
  • the occupant state detection device 1 first determines whether or not the movement of the occupant's hand is detected, and if the occupant's hand movement is detected, the occupant's hand is detected. The occupant's arousal level is estimated based on the movement, and if the occupant's hand movement is not detected, the occupant's arousal level is calculated based on the occupant's movement, the temperature of the occupant's hand, and the temperature of the occupant's face. I tried to estimate. Thereby, the occupant state detection device 1 can reasonably estimate the occupant's arousal degree using the temperature of the occupant's hand and the temperature of the face detected based on the temperature image.
  • the arousal degree estimation unit 16 indicates the movement of the occupant detected by the movement detection unit 14, the temperature of the occupant's hand detected by the temperature detection unit 15, and the temperature of the occupant's hand.
  • the occupant's arousal level was estimated based on the temperature of the face. Specifically, for example, as in the determination condition (E) of the above example, the arousal degree estimation unit 16 detects the relative temperature change of the hand according to the temperature of the occupant's hand and the temperature of the face, and the temperature changes thereof. The relative temperature change of the hand was used to estimate the occupant's arousal level. However, this is just one example.
  • the arousal degree estimation unit 16 may not use the temperature of the occupant's face when estimating the arousal degree of the occupant. That is, when the movement of the hand is not detected, the arousal degree estimation unit 16 is based on the movement of the occupant detected by the motion detection unit 14 and the temperature of the occupant's hand detected by the temperature detection unit 15. The degree of arousal may be estimated.
  • the determination condition (E) may be a condition for a change in the temperature of the occupant's hand.
  • the temperature of the occupant when the arousal degree estimation unit 16 is used for estimating the rousal degree of the occupant may be at least the temperature of the occupant's hand.
  • the temperature detection unit 15 may detect the temperature of the occupant's hand as the temperature of the occupant.
  • the temperature acquisition device 3 is a non-contact temperature array sensor such as a thermopile
  • the temperature array sensor has a large temperature detection error in pixel units, and the difference between one pixel and another pixel is used.
  • the temperature detection error is small. Therefore, the temperature change of the occupant's hand when estimating the arousal degree of the occupant is not detected only from the temperature of the occupant's hand, but is detected by using the temperature of the occupant's face as the temperature of the occupant's hand. It is possible to more accurately detect the temperature change of the occupant's hand by detecting from the difference in the temperature of the occupant's face. As a result, the arousal degree estimation unit 16 can estimate the arousal degree of the occupant with higher accuracy.
  • the arousal degree estimation unit 16 follows the occupant's arousal degree estimation rule constructed based on the determination condition. I was trying to estimate the degree of arousal. Not limited to this, when the motion detection unit 14 does not detect the movement of the occupant's hand, the arousal degree estimation unit 16 is based on a trained model in machine learning (hereinafter referred to as “machine learning model”). Since it is also possible to estimate the arousal level of the occupant, it will be described below.
  • machine learning model machine learning
  • FIG. 5 is a diagram showing a configuration example of the occupant state detecting device 1a in the case where the occupant state detecting device 1a estimates the arousal degree of the occupant based on the machine learning model 18 in the first embodiment. ..
  • the same components as those of the occupant state detection device 1 described with reference to FIG. 1 are designated by the same reference numerals, and duplicate description will be omitted.
  • the occupant state detection device 1a is different from the occupant state detection device 1 described with reference to FIG. 1 in that it includes a machine learning model 18. Further, in the occupant state detection device 1a, the specific operation of the arousal degree estimation unit 16a is different from the specific operation of the arousal degree estimation unit 16 in the occupant state detection device 1.
  • the machine learning model 18 is a machine learning model that inputs information on the movement of the occupant and information on the temperature of the occupant's hand and the temperature of the face, and outputs information indicating the degree of arousal of the occupant.
  • Information about occupant movements includes information about occupant's eye movements, occupant's mouth movements, occupant's body movements, occupant's hand movements, or occupant's face movements.
  • the machine learning model 18 is generated in advance by learning using teacher data and a correct label for the degree of arousal.
  • the correct arousal label is, for example, a level indicating the arousal level.
  • the correct label may be, for example, a drowsiness evaluation index from facial expressions by NEDO (New Energy and Industrial Technology Development Organization), or the Karolinska Drowsiness Scale (KSS). It may be the stage of the degree of drowsiness indicated by. Further, the correct answer label may be, for example, a level indicating the degree of arousal, which is independently set by the administrator of the occupant state detection device 1a.
  • the machine learning model 18 is provided in the occupant state detecting device 1a, but this is only an example.
  • the machine learning model 18 may be provided in a place outside the occupant state detecting device 1a where the occupant state detecting device 1a can be referred.
  • the arousal degree estimation unit 16a is based on the movement of the occupant detected by the motion detection unit 14, the temperature of the occupant's hand and face detected by the temperature detection unit 15, and the machine learning model 18. To estimate.
  • the arousal degree estimation unit 16a first determines whether or not the motion detection unit 14 has detected the movement of the occupant's hand. Specifically, the arousal degree estimation unit 16a determines whether or not the motion detection notification information including the information indicating that the movement of the hand has been detected is output from the motion detection unit 14. When the motion detection unit 14 detects the movement of the occupant's hand, the arousal degree estimation unit 16a estimates the occupant's arousal degree based on the movement of the occupant's hand.
  • the arousal degree estimation unit 16a estimates that the occupant is in an awake state because of the movement of the occupant's hand, and sets the arousal degree of the occupant to "level 5". It can be said that there is a high possibility that the occupant is awake if there is a movement of the occupant's hand.
  • the arousal degree estimation unit 16a detects the occupant's movement detected by the motion detection unit 14 and the temperature of the occupant's hand detected by the temperature detection unit 15. And based on the face temperature and the machine learning model 18, the occupant's arousal level is estimated. Specifically, the arousal degree estimation unit 16a inputs the motion detection notification information output from the motion detection unit 14 and the temperature detection information output from the temperature detection unit 15 into the machine learning model 18, and the arousal degree of the occupant. Get the information that indicates.
  • step ST407 is an operation in which the operation of the arousal degree estimation unit 16 described above is replaced with the operation of the arousal degree estimation unit 16a.
  • the arousal degree estimation unit 16a When the movement detection unit 14 does not detect the movement of the occupant's hand in the occupant state detection device 1a (when "NO" in step ST406), the arousal degree estimation unit 16a has the movement detection unit 14 in step ST404.
  • the arousal level of the occupant is estimated based on the detected movement of the occupant, the temperature of the occupant's hand and face detected by the temperature detection unit 15 in step ST405, and the machine learning model 18 (step ST408).
  • the arousal degree estimation unit 16a inputs the motion detection notification information output from the motion detection unit 14 and the temperature detection information output from the temperature detection unit 15 into the machine learning model 18, and the arousal degree of the occupant. Get the information that indicates.
  • the arousal degree estimation unit 16a outputs the arousal degree information to the output unit 17. Then, the operation of the occupant state detection device 1 proceeds to step ST409.
  • the occupant state detection device 1a acquires an image of the occupant imaged and a temperature image showing the temperature of the surface of the occupant's body measured in a non-contact manner, and detects the occupant based on the captured image.
  • the occupant's arousal level is estimated based on the movement, the temperature of the occupant's hand and face detected based on the temperature image, and the machine learning model 18.
  • the occupant state detection device 1a can estimate the arousal degree of the occupant based on the temperature of the occupant's hand regardless of the position where the occupant holds the steering wheel.
  • the occupant state detection device 1a first determines whether or not the movement of the occupant's hand is detected, and if the occupant's hand movement is detected, the occupant's hand is detected. The occupant's arousal level is estimated based on the movement, and if the occupant's hand movement is not detected, it is based on the occupant's movement, the occupant's hand temperature and the occupant's face temperature, and the machine learning model 18. The occupant's arousal level was estimated. Thereby, the occupant state detection device 1a can reasonably estimate the occupant's arousal degree using the temperature of the occupant's hand and face detected based on the temperature image.
  • the occupant state detection device 1a estimates the occupant's arousal degree using the machine learning model 18, if a large amount of teacher data can be prepared, the occupant's arousal degree is estimated according to the arousal degree estimation rule. Compared with, the estimation accuracy of the arousal degree of the occupant can be improved.
  • the arousal degree estimation unit 16a may not use the temperature of the occupant's face when estimating the arousal degree of the occupant. That is, when the movement of the hand is not detected, the arousal degree estimation unit 16a includes the movement of the occupant detected by the movement detection unit 14, the temperature of the occupant's hand detected by the temperature detection unit 15, and the machine learning model 18. The occupant's arousal level may be estimated based on the above. As described above, the temperature of the occupant when the arousal degree estimation unit 16a is used for estimating the arousal degree of the occupant may be at least the temperature of the occupant's hand.
  • the temperature detection unit 15 may detect the temperature of the occupant's hand as the temperature of the occupant.
  • the machine learning model 18 is a machine learning model that inputs information on the movement of the occupant and information on the temperature of the occupant's hand and outputs information indicating the degree of arousal of the occupant.
  • the occupant state detection device 1 may be configured to estimate the arousal degree of the occupant in consideration of the attributes of the occupant.
  • FIG. 6 is a diagram showing a configuration example of the occupant state detection device 1b in the case where the arousal degree of the occupant is estimated in consideration of the attributes of the occupant in the first embodiment.
  • the same components as those of the occupant state detection device 1 described with reference to FIG. 1 are designated by the same reference numerals, and duplicate description will be omitted.
  • the occupant state detection device 1b is different from the occupant state detection device 1 described with reference to FIG. 1 in that it includes an attribute extraction unit 19.
  • the specific operation of the arousal degree estimation unit 16b is different from the specific operation of the arousal degree estimation unit 16 in the occupant state detection device 1.
  • the attribute extraction unit 19 extracts the attributes of the occupant based on the captured image acquired by the captured image acquisition unit 11.
  • the attribute of the occupant is, for example, the age of the occupant, the gender of the occupant, or the physique of the occupant.
  • the captured image acquisition unit 11 outputs the acquired captured image to the occupant detection unit 13 and the attribute extraction unit 19.
  • the attribute extraction unit 19 may extract the attributes of the occupant from the captured image by using a known image recognition processing technique.
  • the attribute extraction unit 19 outputs the extracted information regarding the attributes of the occupant (hereinafter referred to as “occupant attribute information”) to the arousal degree estimation unit 16b.
  • the arousal estimation unit 16b is based on the movement of the occupant detected by the motion detection unit 14, the temperature of the occupant's hand and face detected by the temperature detection unit 15, and the attributes of the occupant extracted by the attribute extraction unit 19. To estimate the occupant's arousal level.
  • the arousal degree estimation unit 16b estimates the occupant's arousal degree based on the movement of the occupant's hand. Specifically, the arousal degree estimation unit 16b estimates that the occupant is in an awake state because of the movement of the occupant's hand, and sets the occupant's arousal degree to "level 5". It can be said that there is a high possibility that the occupant is awake if there is a movement of the occupant's hand.
  • the arousal degree estimation unit 16b detects the occupant's movement detected by the motion detection unit 14 and the temperature of the occupant's hand detected by the temperature detection unit 15. And the occupant's arousal level is estimated based on the temperature of the face. Specifically, the arousal degree estimation unit 16b estimates the arousal degree of the occupant according to the arousal degree estimation rule constructed based on the determination condition. At this time, the arousal degree estimation unit 16b corrects the determination condition based on the attributes of the occupants extracted by the attribute extraction unit 19. Then, the arousal degree estimation unit 16b applies the corrected determination condition to the awakening degree estimation rule, and estimates the arousal degree of the occupant.
  • the arousal degree estimation unit 16b corrects the determination condition according to the gender of the occupant. Specifically, for example, when the occupant is a female, the arousal degree estimation unit 16b corrects the determination condition (E) to "the temperature of the hand is within -3 ° C with respect to the temperature of the face". Generally, it is said that females have a higher body temperature than males.
  • the arousal degree estimation unit 16b determines the determination condition (E) so as to narrow the range of the difference between the temperature of the hand and the temperature of the face, which is determined to have a high degree of indicating the degree of arousal when the occupant is a female. To correct.
  • the arousal degree estimation unit 16b corrects the determination condition according to the age of the occupant. Specifically, for example, when the occupant is elderly, the arousal degree estimation unit 16b sets the determination condition (A) to "the number of times the occupant is blinking four times or more in the past 10 seconds". Judgment condition (B) is "the time when the occupant is closing his eyes for 4 seconds or more in the past 10 seconds", and judgment condition (C) is "the occupant is squeezing in the past 3 minutes”.
  • the degree of arousal such as "the number of times” is 1 or more "and the judgment condition (D) is” the number of times "the occupant's head is swaying" at an angle of 20 degrees or more in the past 5 minutes is 1 or more ".
  • Judgment conditions (A) to (D) are corrected so that the condition for which the degree of indicating the above is high is determined becomes strict.
  • the arousal degree estimation unit 16b corrects the determination condition according to the physique of the occupant. Specifically, for example, when the occupant is fat, the arousal degree estimation unit 16b corrects the determination condition (E) to "the temperature of the hand is within -3 ° C with respect to the temperature of the face". Generally, it is said that a fat person has a higher body temperature than a thin person. Therefore, the arousal degree estimation unit 16b determines the determination condition (E) so as to narrow the width of the difference between the temperature of the hand and the temperature of the face, which is determined to have a high degree of indicating the degree of arousal when the occupant is fat. To correct.
  • the arousal degree estimation unit 16b estimates the arousal degree of the occupant according to the awakening degree estimation rule to which the corrected determination condition is applied, and outputs the awakening degree information to the output unit 17.
  • step ST407 is an operation in which the operation of the arousal degree estimation unit 16 described above is replaced with the operation of the arousal degree estimation unit 16b.
  • the attribute extraction unit 19 extracts the occupant's attributes based on the captured image acquired by the captured image acquisition unit 11 and awakens the occupant attribute information by the time the operation of step ST408 is performed. It is output to the degree estimation unit 16b.
  • the arousal degree estimation unit 16b When the movement detection unit 14 does not detect the movement of the occupant's hand in the occupant state detection device 1b (when "NO" in step ST406), the arousal degree estimation unit 16b has the movement detection unit 14 in step ST404.
  • the occupant's arousal level is estimated based on the detected movement of the occupant, the temperature of the occupant's hand and face detected by the temperature detection unit 15 in step ST405, and the attributes of the occupant extracted by the attribute extraction unit 19. (Step ST408).
  • the arousal degree estimation unit 16b estimates the arousal degree of the occupant according to the arousal degree estimation rule constructed based on the determination condition.
  • the arousal degree estimation unit 16b corrects the determination condition based on the attributes of the occupants extracted by the attribute extraction unit 19. Then, the arousal degree estimation unit 16b applies the corrected determination condition to the awakening degree estimation rule, and estimates the arousal degree of the occupant. The arousal degree estimation unit 16b outputs the arousal degree information to the output unit 17. Then, the operation of the occupant state detection device 1 proceeds to step ST409.
  • the occupant state detection device 1b acquires an image of the occupant imaged and a temperature image showing the temperature of the surface of the occupant's body measured in a non-contact manner, and detects the occupant based on the captured image.
  • the occupant's arousal level is estimated based on the movement, the temperature of the occupant's hand and face detected based on the temperature image, and the attributes of the occupant extracted based on the captured image.
  • the occupant state detection device 1b can estimate the arousal degree of the occupant based on the temperature of the occupant's hand and the temperature of the face regardless of the position where the occupant holds the handle, and can also estimate the occupant's arousal degree.
  • the occupant state detection device 1b first determines whether or not the movement of the occupant's hand is detected, and if the occupant's hand movement is detected, the occupant's hand is detected. The occupant's arousal level is estimated based on the movement, and if the occupant's hand movement is not detected, the occupant's movement, the temperature of the occupant's hand and the temperature of the occupant's face, and the attributes of the occupant are used. The occupant's arousal level is estimated. Thereby, the occupant state detection device 1b can reasonably estimate the occupant's arousal degree using the temperature of the occupant's hand and face detected based on the temperature image.
  • the arousal degree estimation unit 16b may not use the temperature of the occupant's face when estimating the arousal degree of the occupant. That is, when the movement of the hand is not detected in the arousal degree estimation unit 16b, the movement of the occupant detected by the movement detection unit 14, the temperature of the occupant's hand detected by the temperature detection unit 15, and the attribute extraction unit 19 The degree of arousal of the occupant may be estimated based on the extracted attributes of the occupant. As described above, the temperature of the occupant when the arousal degree estimation unit 16b is used for estimating the arousal degree of the occupant may be at least the temperature of the occupant's hand. In this case, the temperature detection unit 15 may detect the temperature of the occupant's hand as the temperature of the occupant.
  • the configuration of the occupant state detecting device 1b as described above may be applied to the occupant state detecting device 1a described with reference to FIG. That is, the occupant state detection device 1a shown in FIG. 5 may be configured to include the attribute extraction unit 19.
  • the machine learning model 18 inputs information on the movement of the occupant, information on the temperature of the occupant's hand and face, and the attribute of the occupant, and outputs information indicating the degree of awakening of the occupant.
  • the arousal degree estimation unit 16a includes the movement of the occupant detected by the motion detection unit 14, the temperature of the occupant's hand and face detected by the temperature detection unit 15, the attributes of the occupant extracted by the attribute extraction unit 19, and the machine.
  • the occupant's arousal level is estimated based on the learning model 18.
  • the arousal degree estimation unit 16a estimates the occupant's arousal degree based on the movement of the occupant's hand.
  • the arousal degree estimation unit 16a detects the movement of the occupant by the motion detection unit 14 and the temperature of the occupant's hand detected by the temperature detection unit 15.
  • the degree of arousal of the occupant is estimated based on the temperature of the face, the attribute of the occupant extracted by the attribute extraction unit 19, and the machine learning model 18.
  • FIG. 7 is a diagram showing a configuration example of the occupant state detection device 1c in the case where the temperature detection unit 15 detects the temperature of the occupant without using the captured image after the position is assigned in the first embodiment.
  • the configuration example of the occupant state detection device 1c shown in FIG. 7 is different from the configuration example of the occupant state detection device 1 shown in FIG. 1 in that there is no arrow indicating the flow of information from the occupant detection unit 13 to the temperature detection unit 15. different.
  • the specific operation of the temperature detection unit 15a is different from the specific operation of the temperature detection unit in the occupant state detection device 1.
  • the temperature detection unit 15a detects the temperature of the occupant's hand and the temperature of the face from the temperature distribution in the temperature image based on the temperature image acquired by the temperature image acquisition unit 12. In this way, when the temperature detection unit 15a detects the temperature of the occupant's hand and the temperature of the face without using the captured image after the position is assigned, the position of the occupant's hand and the position of the occupant's face cannot be aligned. As a result, the temperature detection accuracy of the temperature detection unit 15a is lowered. However, the temperature detection unit 15a can omit the process of aligning the captured image and the temperature image after the position is given. In the occupant state detection device 1a described with reference to FIG. 5, the temperature detection unit 15 does not use the position-assigned captured image output from the occupant detection unit 13, but the temperature image acquired by the temperature image acquisition unit 12. Therefore, the temperature of the occupant's hand and the temperature of the face may be detected.
  • the temperature detection units 15 and 15a are based on the temperature image and the temperature of the occupant's face.
  • the temperature of the facial part may be detected in more detail.
  • the temperature detection units 15 and 15a may detect, for example, the temperature of the forehead or the temperature of the cheeks as the temperature of the face of the occupant.
  • the temperature of the forehead or the temperature of the cheeks is considered to be close to the core body temperature of a person.
  • the nose since the nose is a peripheral part, the blood flow of the nose increases and the temperature of the nose rises when the person feels drowsy, similar to the hand.
  • the temperature detection units 15 and 15a subdivide the face temperature according to the parts, and the temperature of the forehead or the cheek temperature, which is considered to be close to the core body temperature of a person, is the temperature of the occupant's face, except for the nose, which is the peripheral part.
  • the occupant detection unit 13 detects the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face.
  • the information indicating the above is output to the motion detection unit 14 and the temperature detection units 15, 15a, but this is only an example.
  • the occupant detection unit 13 is the motion detection unit 14 or the temperature detection unit among the information indicating the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's body, the position of the occupant's hand, or the position of the occupant's face.
  • the motion detection unit 14 After narrowing down to the necessary information in the unit 15, it may be output to the motion detection unit 14 or the temperature detection units 15 and 15a.
  • the occupant detection unit 13 refers to information regarding the position of the occupant's eyes, the position of the occupant's mouth, the position of the occupant's face, and the position of the occupant's body (hereinafter referred to as "eye, mouth, facial body position information”. ) Is output to the motion detection unit 14, and information regarding the position of the occupant's face and the position of the occupant's hand (hereinafter referred to as “face and hand position information”) is output to the temperature detection units 15 and 15a. good.
  • the motion detection unit 14 may move the occupant's eyes, the occupant's mouth, the occupant's face, or the occupant's body based on the eye, mouth, and facial body position information output from the occupant detection unit 13. Detect movement. Further, the temperature detection units 15 and 15a detect the temperature of the occupant's hand and the temperature of the face based on the face / hand position information output from the occupant detection unit 13.
  • the motion detection unit 14 may have the function of the occupant detection unit 13. That is, the motion detection unit 14 may have a function of detecting occupant information. In this case, the motion detection unit 14 outputs the captured image after the position is assigned to the temperature detection unit 15. Further, in this case, the occupant state detection devices 1, 1a, 1b, 1c can be configured not to include the occupant detection unit 13. Further, in this case, with respect to the operation of the occupant state detection devices 1, 1a, 1b, 1c described using the flowchart of FIG. 4, the operation of step ST403 is performed by the motion detection unit 14.
  • the occupant is the driver of the vehicle, but this is only an example.
  • the occupant is a occupant of a vehicle other than the driver, and the occupant state detection devices 1, 1a, 1b can also estimate the arousal degree of the occupant other than the driver.
  • the occupant state detection devices 1, 1a and 1b are mounted on the vehicle and estimate the rousal degree of the occupant of the vehicle, but this is only an example.
  • the occupant state detecting devices 1, 1a, 1b can also estimate the rousing degree of the occupant of the moving body in various moving bodies.
  • FIGS. 8A and 8B are diagrams showing an example of the hardware configuration of the occupant state detecting devices 1, 1a, 1b, 1c according to the first embodiment.
  • the occupant state detecting devices 1, 1a, 1b, and 1c all have a hardware configuration as shown in FIGS. 8A and 8B.
  • the functions of the output unit 17 and the attribute extraction unit 19 are realized by the processing circuit 801. That is, the occupant state detecting devices 1, 1a, 1b, 1c include a processing circuit 801 for controlling to estimate the awakening degree of the occupant of the moving body.
  • the processing circuit 801 may be dedicated hardware as shown in FIG. 8A, or may be a CPU (Central Processing Unit) 805 that executes a program stored in the memory 806 as shown in FIG. 8B.
  • CPU Central Processing Unit
  • the processing circuit 801 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an SoC (System-on-a-chip), or an ASIC (Application). Application Special Integrated Circuit), FPGA (Field-Programmable Gate Array), or a combination thereof is applicable.
  • the processing circuit 801 is the CPU 805, the captured image acquisition unit 11, the temperature image acquisition unit 12, the occupant detection unit 13, the motion detection unit 14, the temperature detection units 15, 15a, and the arousal degree estimation units 16, 16a,
  • the functions of 16b, the output unit 17, and the attribute extraction unit 19 are realized by software, firmware, or a combination of software and firmware. That is, the captured image acquisition unit 11, the temperature image acquisition unit 12, the occupant detection unit 13, the motion detection unit 14, the temperature detection units 15, 15a, the arousal degree estimation units 16, 16a, 16b, and the output unit 17.
  • the attribute extraction unit 19 is realized by a CPU 805 that executes a program stored in an HDD (Hard Disk Drive) 802, a memory 806, or the like, or a processing circuit 801 such as a system LSI (Large-Scale Integration). Further, the programs stored in the HDD 802, the memory 806, or the like include the captured image acquisition unit 11, the temperature image acquisition unit 12, the occupant detection unit 13, the motion detection unit 14, the temperature detection units 15, 15a, and the arousal degree. It can also be said that the procedure or method of the estimation unit 16, 16a, 16b, the output unit 17, and the attribute extraction unit 19 is executed by the computer.
  • HDD Hard Disk Drive
  • a memory 806, or the like or a processing circuit 801 such as a system LSI (Large-Scale Integration).
  • the programs stored in the HDD 802, the memory 806, or the like include the captured image acquisition unit 11, the temperature image acquisition unit 12, the occupant detection unit 13, the motion detection unit 14, the temperature detection units 15, 15a
  • the memory 806 is, for example, a RAM, a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), an EEPROM (Electrically Erasable Projector), or an EEPROM (Electrically Erasable Molecular) volatile Read.
  • a semiconductor memory, a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, a DVD (Digital Versaille Disc), or the like is applicable.
  • the function of the attribute extraction unit 19 may be partially realized by dedicated hardware and partly realized by software or firmware.
  • the captured image acquisition unit 11, the temperature image acquisition unit 12, and the output unit 17 are realized by the processing circuit 801 as dedicated hardware, and the occupant detection unit 13, the motion detection unit 14, and the temperature detection unit 14 are realized.
  • the functions of 15, 15a, the arousal estimation units 16, 16a, 16b, and the attribute extraction unit 19 can be realized by the processing circuit 801 reading and executing the program stored in the memory 806. ..
  • a memory 806 is used as a storage unit (not shown). Note that this is only an example, and the storage unit (not shown) may be configured by HDD 802, SSD (Solid State Drive), DVD, or the like. Further, the occupant state detecting devices 1, 1a, 1b, 1c include a device such as an image pickup device 2 or a temperature acquisition device 3, and an input interface device 803 and an output interface device 804 for performing wired communication or wireless communication.
  • the occupant state detection devices 1, 1b, 1c are the captured image acquisition unit 11 that acquires the captured image of the occupant, and the occupant's body measured in a non-contact manner.
  • the temperature image acquisition unit 12 that acquires a temperature image representing the surface temperature
  • the motion detection unit 14 that detects the movement of the occupant based on the image captured by the captured image acquisition unit 11, and the temperature image acquisition unit 12 acquire the temperature image.
  • the temperature detection units 15 and 15a that detect the temperature of the occupant's hand, the movement of the occupant detected by the motion detection unit 14, and the temperature of the occupant's hand detected by the temperature detection units 15 and 15a.
  • the occupant state detecting devices 1, 1b, 1c can estimate the arousal degree of the person based on the temperature of the person's hand regardless of the position where the person (occupant) holds the steering wheel.
  • the movement of the occupant detected by the motion detection unit 14 includes the movement of the occupant's hand
  • the arousal degree estimation units 16 and 16b include the movement of the occupant's hand.
  • the movement detection unit 14 detects the movement of the occupant's hand
  • the arousal degree of the occupant is estimated based on the movement of the occupant's hand
  • the movement detection unit 14 does not detect the movement of the occupant's hand.
  • the occupant's arousal level is estimated based on the occupant's movement detected by the motion detection unit 14 and the occupant's hand temperature detected by the temperature detection units 15 and 15a. Therefore, the occupant state detecting devices 1, 1b, 1c can reasonably estimate the occupant's arousal degree using the temperature of the occupant's hand and face detected based on the temperature image.
  • the occupant state detection device 1a is a temperature image showing the temperature of the surface of the occupant's body measured in a non-contact manner with the captured image acquisition unit 11 that acquires the captured image of the occupant.
  • the motion detection unit 14 Based on the temperature image acquisition unit 12 that acquires the temperature image acquisition unit 12, the motion detection unit 14 that detects the movement of the occupant based on the image captured image acquired by the image capture image acquisition unit 11, and the temperature image acquired by the temperature image acquisition unit 12.
  • the temperature detection unit 15 that detects the temperature of the occupant's hand, the information on the occupant's movement detected by the motion detection unit 14, the information on the occupant's hand temperature detected by the temperature detection unit 15, and the machine learning model 18.
  • the occupant state detecting device 1a can estimate the arousal degree of the person based on the temperature of the person's hand regardless of the position where the person (occupant) holds the steering wheel.
  • the movement of the occupant detected by the motion detection unit 14 includes the movement of the occupant's hand, and in the arousal degree estimation unit 16a, the motion detection unit 14 is used.
  • the arousal degree of the occupant is estimated based on the movement of the occupant's hand, and when the motion detection unit 14 does not detect the movement of the occupant's hand, it is based on the machine learning model 18. It was configured to estimate the occupant's arousal level. Therefore, the occupant state detection device 1a can reasonably estimate the occupant's arousal degree using the temperature of the occupant's hand and face detected based on the temperature image.
  • the occupant state detection devices 1, 1a, 1b, and 1c are in-vehicle devices mounted on the vehicle, and the captured image acquisition unit 11, the temperature image acquisition unit 12, and the occupant detection unit 13 are used.
  • the motion detection unit 14, the temperature detection unit 15, 15a, the arousal degree estimation unit 16, 16a, 16b, the output unit 17, and the attribute extraction unit 19 are attached to the occupant state detection devices 1, 1a, 1b, 1c. It was supposed to be prepared. Not limited to this, the captured image acquisition unit 11, the temperature image acquisition unit 12, the occupant detection unit 13, the motion detection unit 14, the temperature detection units 15, 15a, the arousal degree estimation units 16, 16a, 16b, and the like.
  • the output unit 17 and the attribute extraction unit 19 a part of the in-vehicle device and the server are assumed to be mounted on the in-vehicle device of the vehicle and the other is provided in the server connected to the in-vehicle device via the network.
  • the occupant state detection system may be configured with and.
  • the occupant state detection device is configured so that the arousal degree of the person can be estimated based on the temperature of the person's hand regardless of the position where the person holds the handle. It can be applied to an occupant state estimation device that estimates the degree of arousal.
  • 1,1a, 1b, 1c Crew state detection device 2 Imaging device, 3 Temperature acquisition device, 11 Captured image acquisition unit, 12 Temperature image acquisition unit, 13 Crew detection unit, 14 Motion detection unit, 15, 15a Temperature detection unit, 16, 16a, 16b Awakening degree estimation unit, 17 output unit, 18 machine learning model, 19 attribute extraction unit, 801 processing circuit, 802 HDD, 803 input interface device, 804 output interface device, 805 CPU, 806 memory.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Social Psychology (AREA)
  • Psychiatry (AREA)
  • Biophysics (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Veterinary Medicine (AREA)
  • Animal Behavior & Ethology (AREA)
  • Public Health (AREA)
  • Surgery (AREA)
  • Molecular Biology (AREA)
  • Medical Informatics (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Human Computer Interaction (AREA)
  • Automation & Control Theory (AREA)
  • Mathematical Physics (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Child & Adolescent Psychology (AREA)
  • Dentistry (AREA)
  • Hospice & Palliative Care (AREA)
  • Educational Technology (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Physiology (AREA)
  • Psychology (AREA)
  • Oral & Maxillofacial Surgery (AREA)
  • Developmental Disabilities (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)
  • Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
PCT/JP2020/026913 2020-07-09 2020-07-09 乗員状態検出装置および乗員状態検出方法 Ceased WO2022009401A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2022534604A JP7286022B2 (ja) 2020-07-09 2020-07-09 乗員状態検出装置および乗員状態検出方法
PCT/JP2020/026913 WO2022009401A1 (ja) 2020-07-09 2020-07-09 乗員状態検出装置および乗員状態検出方法
DE112020007404.8T DE112020007404T5 (de) 2020-07-09 2020-07-09 Insassenzustandserkennungsvorrichtung und insassenzustandserkennungsverfahren
US17/928,015 US12240469B2 (en) 2020-07-09 2020-07-09 Occupant state detection device and occupant state detection method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/026913 WO2022009401A1 (ja) 2020-07-09 2020-07-09 乗員状態検出装置および乗員状態検出方法

Publications (1)

Publication Number Publication Date
WO2022009401A1 true WO2022009401A1 (ja) 2022-01-13

Family

ID=79552356

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/026913 Ceased WO2022009401A1 (ja) 2020-07-09 2020-07-09 乗員状態検出装置および乗員状態検出方法

Country Status (4)

Country Link
US (1) US12240469B2 (https=)
JP (1) JP7286022B2 (https=)
DE (1) DE112020007404T5 (https=)
WO (1) WO2022009401A1 (https=)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024037353A (ja) * 2022-09-07 2024-03-19 いすゞ自動車株式会社 車両制御装置
WO2026013777A1 (ja) * 2024-07-10 2026-01-15 三菱電機モビリティ株式会社 運転支援装置、運転支援方法、および、運転支援システム

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE112020007404T5 (de) * 2020-07-09 2023-05-17 Mitsubishi Electric Corporation Insassenzustandserkennungsvorrichtung und insassenzustandserkennungsverfahren

Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001137194A (ja) * 1999-11-12 2001-05-22 Matsushita Electric Ind Co Ltd 居眠り検出装置
JP2005075198A (ja) * 2003-09-02 2005-03-24 Calsonic Kansei Corp 車両用空調装置
US20060220915A1 (en) * 2005-03-21 2006-10-05 Bauer James A Inter-vehicle drowsy driver advisory system
JP2012164040A (ja) * 2011-02-04 2012-08-30 Nissan Motor Co Ltd 覚醒低下検出装置
US20160311440A1 (en) * 2015-04-22 2016-10-27 Motorola Mobility Llc Drowsy driver detection
JP2017127616A (ja) * 2015-07-22 2017-07-27 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 覚醒度予測方法および覚醒度予測装置
WO2018118958A1 (en) * 2016-12-22 2018-06-28 Sri International A driver monitoring and response system
JP2018127112A (ja) * 2017-02-08 2018-08-16 パナソニックＩｐマネジメント株式会社 覚醒度推定装置及び覚醒度推定方法
US20190049267A1 (en) * 2018-03-28 2019-02-14 Intel Corporation Safety enhanced computer assisted driving method and apparatus
US10322728B1 (en) * 2018-02-22 2019-06-18 Futurewei Technologies, Inc. Method for distress and road rage detection

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
RU2011119211A (ru) * 2008-11-13 2012-12-20 Асер Рич Лимитед. Система и способ повышения безопасности транспортного средства путем улучшения знания ситуации водителем транспортного средства
JP2011123653A (ja) 2009-12-10 2011-06-23 Sumitomo Rubber Ind Ltd 運転者覚醒度検査装置
US9789869B2 (en) * 2012-11-02 2017-10-17 Toyota Jidosha Kabushiki Kaisha Driving support apparatus and driving support method
KR20160139756A (ko) * 2015-05-28 2016-12-07 주식회사 엠투브 차량간 비콘 메시지 송수신 시스템 및 방법
US10716502B2 (en) 2015-07-22 2020-07-21 Panasonic Intellectual Property Corporation Of America Method for predicting arousal level and arousal level prediction apparatus
US10137848B2 (en) * 2016-01-13 2018-11-27 Ford Global Technologies, Llc System identifying a driver before they approach the vehicle using wireless communication protocols
US10562412B1 (en) * 2016-03-24 2020-02-18 Xsensor Technology Corporation Intelligent seat systems
JP2019074791A (ja) * 2017-10-12 2019-05-16 株式会社デンソー 車両用運転支援装置
WO2019221240A1 (ja) * 2018-05-16 2019-11-21 社会医療法人蘇西厚生会 まつなみリサーチパーク 安全運転支援システム
JP2021530069A (ja) * 2018-06-26 2021-11-04 カッツ，イテイ 状況的ドライバ監視システム
DE112020007404T5 (de) * 2020-07-09 2023-05-17 Mitsubishi Electric Corporation Insassenzustandserkennungsvorrichtung und insassenzustandserkennungsverfahren

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2001137194A (ja) * 1999-11-12 2001-05-22 Matsushita Electric Ind Co Ltd 居眠り検出装置
JP2005075198A (ja) * 2003-09-02 2005-03-24 Calsonic Kansei Corp 車両用空調装置
US20060220915A1 (en) * 2005-03-21 2006-10-05 Bauer James A Inter-vehicle drowsy driver advisory system
JP2012164040A (ja) * 2011-02-04 2012-08-30 Nissan Motor Co Ltd 覚醒低下検出装置
US20160311440A1 (en) * 2015-04-22 2016-10-27 Motorola Mobility Llc Drowsy driver detection
JP2017127616A (ja) * 2015-07-22 2017-07-27 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ 覚醒度予測方法および覚醒度予測装置
WO2018118958A1 (en) * 2016-12-22 2018-06-28 Sri International A driver monitoring and response system
JP2018127112A (ja) * 2017-02-08 2018-08-16 パナソニックＩｐマネジメント株式会社 覚醒度推定装置及び覚醒度推定方法
US10322728B1 (en) * 2018-02-22 2019-06-18 Futurewei Technologies, Inc. Method for distress and road rage detection
US20190049267A1 (en) * 2018-03-28 2019-02-14 Intel Corporation Safety enhanced computer assisted driving method and apparatus

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
CHOI, H ET AL.: "Driver Drowsiness Detection based on Multimodal using Fusion of Visual-feature and Bio-signal", 2018 INTERNATIONAL CONFERENCE ON INFORMATION AND COMMUNICATION TECHNOLOGY CONVERGENCE (ICTC, 2018, pages 1249 - 1251, XP033447928, DOI: 10.1109/ICTC.2018.8539478 *
MATSUO, H ET AL.: "Prediction of Drowsy Driving by Monitoring Driver's Behavior", PROCEEDINGS OF THE 21ST INTERNATIONAL CONFERENCE ON PATTERN RECOGNITION (ICPR2012, 2012, pages 3390 - 3393, XP032330073 *
PRATAMA, B.G ET AL.: "A Review on Driver Drowsiness Based on Image, Bio-Signal, and Driver Behavior", 2017 3RD INTERNATIONAL CONFERENCE ON SCIENCE AND TECHNOLOGY - COMPUTER (ICST, 2017, pages 70 - 75, XP033143580, DOI: 10.1109/ICSTC.2017.8011855 *

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2024037353A (ja) * 2022-09-07 2024-03-19 いすゞ自動車株式会社 車両制御装置
WO2026013777A1 (ja) * 2024-07-10 2026-01-15 三菱電機モビリティ株式会社 運転支援装置、運転支援方法、および、運転支援システム

Also Published As

Publication number Publication date
DE112020007404T5 (de) 2023-05-17
US12240469B2 (en) 2025-03-04
US20230211791A1 (en) 2023-07-06
JP7286022B2 (ja) 2023-06-02
JPWO2022009401A1 (https=) 2022-01-13

Similar Documents

Publication Publication Date Title
Ramzan et al. A survey on state-of-the-art drowsiness detection techniques
US11535280B2 (en) Method and device for determining an estimate of the capability of a vehicle driver to take over control of a vehicle
EP3844553B1 (en) Method and system for gaze estimation
CN115439832B (zh) 监测车辆乘员的方法和系统
WO2019136449A2 (en) Error correction in convolutional neural networks
JP2002352228A (ja) 画像処理システム及びこれを用いた運転支援システム
WO2022009401A1 (ja) 乗員状態検出装置および乗員状態検出方法
CN114423343A (zh) 认知功能推测装置、学习装置及认知功能推测方法
JP7240910B2 (ja) 乗員観察装置
Hajari et al. A review of issues and challenges in designing Iris recognition Systems for noisy imaging environment
Rosero-Montalvo et al. Multivariate approach to alcohol detection in drivers by sensors and artificial vision
US11077814B2 (en) Occupant eye(s) observation device
CN101642376A (zh) 疲劳检测装置及方法
KR101745774B1 (ko) 안구전도 시선 트래킹 기술 기반 눈글 인식 방법 및 시스템
JP7611839B2 (ja) 異常状態推定装置および異常状態推定方法
JP5139470B2 (ja) 眠気度推定装置および眠気度推定方法
Alam et al. Vision-based driver’s attention monitoring system for smart vehicles
WO2018051836A1 (ja) 虹彩検出装置、虹彩検出方法、虹彩検出プログラム、及び虹彩検出プログラムを記録した記録媒体
Panicker et al. Open-eye detection using iris–sclera pattern analysis for driver drowsiness detection
Al-Rahayfeh et al. Application of head flexion detection for enhancing eye gaze direction classification
JP7296069B2 (ja) 視線入力装置、および視線入力方法
Badri et al. Analysis of Driver Drowsiness Detection System Based on Landmarks and MediaPipe
Kamble Driver drowsiness detection system based on yawn detection
JP5493676B2 (ja) 眼位置認識装置
CN118115922A (zh) 一种基于机器视觉的行为状态综合评价方法和装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20944377

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022534604

Country of ref document: JP

Kind code of ref document: A

122 Ep: pct application non-entry in european phase

Ref document number: 20944377

Country of ref document: EP

Kind code of ref document: A1

WWG Wipo information: grant in national office

Ref document number: 17928015

Country of ref document: US