WO2024047856A1 - 眠気推定装置および眠気推定方法 - Google Patents

眠気推定装置および眠気推定方法 Download PDF

Info

Publication number
WO2024047856A1
WO2024047856A1 PCT/JP2022/033046 JP2022033046W WO2024047856A1 WO 2024047856 A1 WO2024047856 A1 WO 2024047856A1 JP 2022033046 W JP2022033046 W JP 2022033046W WO 2024047856 A1 WO2024047856 A1 WO 2024047856A1
Authority
WO
WIPO (PCT)
Prior art keywords
drowsiness
reliability
state
overdetection
factor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/033046
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
有実子 岡本
奈津季 田原
雄大 中村
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Mitsubishi Electric Corp
Original Assignee
Mitsubishi Electric Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Mitsubishi Electric Corp filed Critical Mitsubishi Electric Corp
Priority to PCT/JP2022/033046 priority Critical patent/WO2024047856A1/ja
Priority to JP2024543738A priority patent/JP7612118B2/ja
Publication of WO2024047856A1 publication Critical patent/WO2024047856A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems

Definitions

  • the present disclosure relates to drowsiness estimation technology.
  • a drowsiness determination function is one of the functions of a passenger monitoring system (hereinafter sometimes simply referred to as "PMS") that monitors vehicle drivers and other occupants.
  • PMS passenger monitoring system
  • the drowsiness determination function is a function that estimates the drowsiness level of the occupant by capturing information about the occupant using a sensor such as a camera. Traffic accidents can be prevented by issuing a warning to alert the occupants based on the estimation results.
  • drowsiness features features such as the occupant's eyes closed or yawning are quantified as feature quantities (hereinafter referred to as "drowsiness features"), and changes in the quantified drowsiness features are comprehensively judged. Based on this, the presence or absence of drowsiness and the degree of drowsiness are determined.
  • drowsiness features feature quantities
  • changes in the quantified drowsiness features are comprehensively judged. Based on this, the presence or absence of drowsiness and the degree of drowsiness are determined.
  • Patent Document 1 In order to resolve the difficulties associated with such rule-based models, there is a technique that uses a machine learning model to determine drowsiness, such as the technique disclosed in Patent Document 1, for example. Further, paragraph 0020 of Patent Document 1 states that a machine learning model and a rule-based model may be combined.
  • the numerical value of the drowsiness feature may change in the same way when there is drowsiness and when there is no drowsiness. Therefore, it is difficult to distinguish between these cases using a technique that simply uses a machine learning model to determine drowsiness.
  • Patent Document 1 states that a machine learning model and a rule-based model may be combined, there is no mention of a specific method of combination. Therefore, the problem with the prior art is that it is not clear how to estimate drowsiness by combining a machine learning model and a rule-based model.
  • the present disclosure has been made to solve such problems, and aims to provide a drowsiness estimation technology that combines a machine learning model and a rule-based model to estimate the drowsiness of a vehicle occupant.
  • a drowsiness estimation device calculates a drowsiness score indicating the degree of drowsiness of a vehicle occupant using a trained machine learning model based on an image of the face of the vehicle occupant.
  • a drowsiness score calculation unit determines whether an overdetection factor that affects the calculation of the drowsiness score exists, and when it is determined that the overdetection factor exists, the overdetection factor exists.
  • An over-detection factor index indicating that the over-detection factor index and the reliability of the over-detection factor index are calculated on a rule basis, the calculated drowsiness score, the calculated over-detection factor index, and the calculation thereof.
  • the drowsiness of the occupant is determined to be one of a plurality of drowsiness states including a first drowsiness state with a low degree of drowsiness and a second drowsiness state with a degree of drowsiness higher than the first drowsiness state. and a state transition determination unit that determines which of the sleepiness states the user is in.
  • the drowsiness estimation device it is possible to estimate the drowsiness of a vehicle occupant by combining a machine learning model and a rule-based model.
  • FIG. 2 is a block diagram showing a configuration example of a drowsiness estimation device.
  • FIG. 2 is a diagram illustrating an example of a hardware configuration of a drowsiness estimation device.
  • FIG. 2 is a diagram illustrating an example of a hardware configuration of a drowsiness estimation device.
  • 3 is a flowchart showing a schematic operation of the drowsiness estimation device. It is a flowchart which shows the operation of the overdetection factor reliability calculation part of a drowsiness estimation device.
  • FIG. 3 is a diagram for explaining a method of calculating a rate of opening both eyelids. It is a flowchart which shows the operation of the state transition judgment part of a drowsiness estimating device.
  • FIG. 3 is a state transition diagram between a plurality of drowsy states.
  • FIG. 1 is a block diagram illustrating a configuration example of a drowsiness estimation device 100 according to Embodiment 1 of the present disclosure.
  • a vehicle 1 includes an imaging device 2 and a drowsiness estimation device 100.
  • the imaging device 2 is a device for imaging the interior of the vehicle 1.
  • the imaging device 2 is installed, for example, in the front part of the vehicle interior of the vehicle 1, and images an area including the face of a passenger such as a driver of the vehicle 1 from the front.
  • the imaging device 2 includes one visible light camera, multiple visible light cameras, one infrared camera, or multiple infrared cameras.
  • a light source (not shown) is provided that irradiates an area including the driver's face with infrared rays for imaging.
  • This light source is composed of, for example, an LED (Light Emitting Diode).
  • the drowsiness estimation device 100 includes a face information detection section 10 and a drowsiness determination section 20. Further, the face information detection section 10 includes a video acquisition section 11 , a face detection section 12 , and a facial parts detection section 13 .
  • the drowsiness determination unit 20 also includes a feature value calculation unit 14 , a drowsiness score calculation unit 15 , an overdetection factor reliability calculation unit 16 , and a state transition determination unit 17 .
  • the drowsiness estimating device 100 also includes a control section (hereinafter simply referred to as "control section") not shown. In FIG. 1, illustration of the control unit is omitted in order to show the general flow of data.
  • the control unit controls the operation of each functional unit of the imaging device 2 and the drowsiness estimation device 100. Specifically, the control unit instructs the imaging device 2 to capture an image of the driver using the in-vehicle camera. The control unit also instructs the video acquisition unit 11, the face detection unit 12, the facial parts detection unit 13, the feature amount calculation unit 14, the drowsiness score calculation unit 15, the overdetection factor reliability calculation unit 16, and the state transition determination unit 17. control the operation timing and the exchange of information.
  • the video acquisition unit 11 acquires a facial image from a video (imaging information) including a plurality of frame images captured by the imaging device 2, and outputs the acquired facial image to the control unit.
  • a face image means an image in which a partial region including a face is extracted from the entire region of each frame image.
  • the face detection unit 12 receives the face image acquired by the video acquisition unit 11 from the control unit, detects a face from the face image, and outputs data of the detected face to the control unit as a face detection result.
  • the face detection unit 12 is a classifier using a general algorithm, for example, a Haar-Like detector combined with Adaboost or Casecade.
  • the facial parts detection unit 13 receives the face detection result outputted by the face detection unit 12 from the control unit, detects facial parts such as eyes or mouth from the face data included in the face detection result, and detects the detected facial parts. It is output to the control unit as a facial parts detection result. Facial parts can be detected using general image processing techniques, for example, by detecting edges using a differential filter.
  • the facial parts detection unit 13 may calculate the degree of opening of the facial parts from the shape of the detected facial parts, and may include information indicating the degree of opening of the facial parts in the facial parts detection result and output it.
  • the degree of eye opening may be calculated as information indicating the degree of opening of facial parts.
  • the degree of eye opening is determined by detecting the position of the inner corner of the eye, the position of the outer corner of the eye, and the position of the highest point of the upper eyelid.
  • the flatness of the eye can be determined by dividing by the distance, and the value obtained by dividing the flatness by a predetermined reference value can be calculated.
  • the facial parts detection unit 13 may calculate the degree of opening of the mouth as information indicating the degree of opening of the mouth.
  • the facial parts detection unit 13 may calculate the reliability of the degree of eye opening and include information indicating the reliability of the calculated degree of eye opening in the facial parts detection result and output it.
  • the reliability of the degree of eye opening indicates whether the degree of eye opening can be stably measured.
  • the facial parts detection unit 13 is configured to acquire both eyes of the passenger from the face detection result, and if only one eye of the passenger can be acquired, it is determined that there is a landscape reflection.
  • the reliability of the degree of eye opening is also set low when the driver turns his or her head to the side (when the Yaw angle of the face is large). The reliability of the degree of eye opening may be calculated by taking into consideration a plurality of indicators such as the presence or absence of scenery reflection and the Yaw angle of the face.
  • the feature quantity calculation unit 14 receives the facial parts detection results from the control unit, calculates the drowsiness feature quantity representing a sign of drowsiness appearing in the facial part detection results, and outputs the calculated drowsiness feature quantity to the control unit.
  • Examples of the drowsiness feature amount include the eye-closed time ratio, which captures the increase in the driver's blinks due to drowsiness, and the number of yawns, which captures the yawning due to drowsiness.
  • the eye-closed time ratio is calculated by calculating the eye-closed time ratio in a certain period of time. Eye closure means a state in which the distance between the upper and lower eyelids is less than or equal to a predetermined threshold.
  • the number of yawns is calculated by counting the number of yawns in a certain period of time.
  • Yawning refers to a state in which the mouth remains open for a predetermined threshold or more for a predetermined period of time. Distinguish between cases where the mouth is open due to conversation and cases where the mouth is open due to yawning by determining whether the mouth remains open for more than a predetermined threshold for a predetermined period of time. be able to.
  • the feature amount calculation unit 14 may calculate only one drowsiness feature amount, such as only the eye-closed time ratio, or may calculate a plurality of drowsiness feature amounts.
  • the drowsiness score calculation unit 15 receives the value of the drowsiness feature amount from the control unit, calculates the drowsiness score, and outputs the calculated drowsiness score to the control unit. Calculation of the drowsiness score is performed using a trained machine learning model that has learned the relationship between the drowsiness feature amount and the drowsiness score using a machine learning model such as a support vector machine (SVM) or a random forest. That is, the drowsiness score calculation unit 15 calculates the drowsiness score by inputting one or more feature quantities calculated by the feature quantity calculation unit 14 into the learned model and obtaining the drowsiness score from the learned model.
  • the method of outputting the sleepiness score may be a discrete value divided into multiple stages, or a continuous value such as 0 to 1, for example.
  • the overdetection factor reliability calculation unit 16 receives the facial parts detection results from the control unit, determines whether or not there is an overdetection factor based on the facial parts detection results, and if it is determined that there is an overdetection factor, the overdetection factor is calculated.
  • the reliability of the overdetection factor index indicating the presence of the detection factor is further calculated.
  • the reliability includes, for example, a first reliability with a low reliability and a second reliability with a higher reliability than the first reliability, and the overdetection factor reliability calculation unit 16 calculates the overdetection factor index.
  • the first reliability or the second reliability is calculated as the reliability of.
  • the reliability may be further divided.
  • the overdetection factor reliability calculation unit 16 outputs to the control unit an overdetection factor index indicating that an overdetection factor exists, and the reliability of the overdetection factor index indicating the probability that the overdetection factor exists.
  • the over-detection factor index means an index that indicates whether a factor that can lead to over-detection is occurring. Overdetection means that the driver is erroneously determined to be drowsy even though the driver does not actually feel drowsy. Overdetection is a concept that is distinguished from non-detection, and non-detection means that the driver is mistakenly determined to be not drowsy, even though the driver actually feels drowsy. Examples of such overdetection factor indicators include, for example, a downward gaze indicator that indicates that the driver's line of sight is directed downward, and a conversation indicator that indicates that the driver is having a conversation with a fellow passenger.
  • the downward gaze index by distinguishing between closed eyes and downward gaze based on the distance between the upper and lower eyelids, it is possible to determine whether it is downward gaze. That is, since the distance between the upper and lower eyelids is generally longer when looking downward than when the eyes are closed, it is possible to distinguish between closed eyes and downward looking based on the distance between the upper and lower eyelids.
  • the conversation index openings where the upper and lower lips are equal to or greater than a predetermined threshold and mouth closings where the upper and lower lips are less than the threshold are detected, and the speed of mouth opening and closing movements is calculated.
  • the speed of opening and closing movements is faster in speaking than in yawning, so yawning and speaking can be distinguished based on the speed of opening and closing movements.
  • the reliability of the overdetection factor index is a value that represents the certainty of the overdetection factor index, that is, the probability that the overdetection factor exists.
  • the reliability of the downward-looking index decreases when there is a possibility that the downward-looking index cannot be correctly determined. For example, the time series variance of the coordinates of the upper and lower eyelids is calculated, and if the value is greater than or equal to a threshold value, it is determined that the coordinates of the upper and lower eyelids cannot be stably detected, and the reliability is determined to be low. Conversely, if it is less than the threshold, it is determined that the reliability is high.
  • the reliability of the downward vision index may be determined using the reliability of the degree of eye opening received as the face detection result and a preset threshold.
  • a predetermined threshold if the time-series variance of the speed of opening and closing movements is greater than or equal to a predetermined threshold, the reliability is determined to be low, and if it is less than the threshold, the reliability is determined to be high. It's fine.
  • the state transition determining unit 17 receives the drowsiness score, the over-detection factor index, and the reliability of the over-detection factor index from the control unit, determines the drowsiness state of the driver, and outputs the determined drowsiness state.
  • the over-detection factor index and the reliability of the over-detection factor index will not be output if they do not exist for the current frame, so in such a case, the state transition determination unit 17 will not output the over-detection factor index Can not accept.
  • the state transition determination unit 17 may indicate current drowsiness. maintain the condition. Further, if the drowsiness score is smaller than the drowsiness lowering threshold, a transition is made to a weaker drowsiness state, and the drowsiness state after the transition is output. Note that the drowsiness lowering threshold is a threshold for determining a transition from the current drowsiness state to a weaker drowsiness state.
  • the state transition determination unit 17 checks whether there is an overdetection factor index, and if the result of the check is that there is no overdetection factor, the drowsiness state transitions according to the drowsiness score. It's okay.
  • the drowsiness increase threshold is a threshold for determining a transition from the current drowsiness state to a stronger drowsiness state.
  • the state transition determination unit 17 determines that the increase in the drowsiness score is due to the overdetection factor rather than an increase in drowsiness. It is not necessary to determine that there is a drowsiness state and not change the drowsiness state. That is, the current sleepy state may be maintained. As a result of checking the over-detection factor index, if there is an over-detection factor and the reliability of the over-detection factor index is low, the state transition determination unit 17 may make a correction to make the transition of the drowsiness state more difficult.
  • the correction may be made by increasing the threshold for transitioning from a first drowsiness state where the degree of drowsiness is low to a second drowsiness state where the degree of drowsiness is higher than the first drowsiness state.
  • the correction may be performed by transitioning to an intermediate state between the first drowsiness state and the second drowsiness state.
  • Each functional unit of the drowsiness estimation device 100 is realized by a processing circuit.
  • a processing circuit even if it is a dedicated processing circuit 100a as shown in FIG. 2A, executes a program stored in a memory 100c as shown in FIG. 2B. It may be the processor 100b.
  • the dedicated processing circuit 100a is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, or an application specific integrated circuit (ASIC). , FPGA (field-programmable gate array), or a combination of these.
  • Each functional unit may be realized by a plurality of separate processing circuits, or each functional unit may be realized by a single processing circuit.
  • each functional unit is realized by software, firmware, or a combination of software and firmware.
  • Software and firmware are written as programs and stored in memory 100c.
  • the processor 100b implements each functional unit by reading and executing programs stored in memory. Examples of the memory 100c include non-volatile or Includes volatile semiconductor memory, magnetic disks, flexible disks, optical disks, compact disks, minidisks, and DVDs.
  • the processing circuit can implement each functional unit using hardware, software, firmware, or a combination thereof.
  • the drowsiness score calculation unit 15 calculates a drowsiness score indicating the degree of drowsiness of the driver using a trained machine learning model based on an image of the driver's face.
  • the drowsiness score calculation unit 15 calculates the drowsiness extracted from the driver's face using a trained machine learning model that has learned the relationship between the drowsiness feature extracted from the image of the face and the drowsiness score. Input the features into a machine learning model to obtain the driver's drowsiness score.
  • the overdetection factor reliability calculation unit 16 determines whether there is an overdetection factor that affects the calculation of the drowsiness score based on the image of the driver's face, and determines whether the overdetection factor exists. When it is determined that this is the case, the reliability of the overdetection factor index is calculated. These determination and calculation processes are performed on a rule basis. As an example, the over-detection factor reliability calculation unit 16 determines whether an over-detection factor exists based on the driver's facial part detection results, and if the over-detection factor exists, the reliability of the over-detection factor index is Calculate. For example, it is determined whether downward gaze exists based on the distance between the upper and lower eyelids, and in the case of downward gaze, the reliability of the downward gaze index is calculated from the time-series variance of the coordinates of the upper and lower eyelids.
  • step ST11 and step ST12 may be performed either first or at the same time.
  • step ST13 the state transition determining unit 17 determines the driver's drowsiness state based on the calculated drowsiness score, the calculated over-detection factor index, and the reliability of the calculated over-detection factor index.
  • the overdetection factor reliability calculation unit 16 will be explained in more detail using the flowchart in FIG. 4. Although the flowchart of FIG. 4 is described based on the case where the overdetection factor is downward gaze, the process may be performed in the same flow as FIG. 4 in the case of other overdetection factors such as conversation. As a premise at the start of the flowchart of FIG. 4, it is assumed that the overdetection factor reliability calculation unit 16 has received information regarding the open/closed state of the eyes, such as the degree of eye opening, as a facial part detection result.
  • the overdetection factor reliability calculation unit 16 analyzes the state of the eyes and determines whether downward gaze is present or not according to a plurality of states of the eyes. In addition, when the overdetection factor reliability calculation unit 16 determines that downward gaze is present, the overdetection factor reliability calculation unit 16 further determines the reliability of the determination that downward gaze is present. A more detailed explanation will be given below.
  • step ST101 it is confirmed whether the eyes are closed. Whether or not the eyes are closed can be confirmed, for example, by determining whether the degree of eye opening received as the facial parts detection result is less than or equal to a predetermined threshold. If "NO", it is determined in step ST111 that there is no downward view. That is, if the eye state is not closed, in other words, if the eye state is open, the process proceeds to step ST111, and it is determined in step ST111 that there is no downward gaze. On the other hand, if "YES” in step ST101, that is, if it is determined that the eyes are closed, the process proceeds to steps ST102 to ST104, in which both eyelids are opened, which is used to determine whether the eyes are truly closed or whether the eyes are looking downward. Calculate the rate. Note that if the cause of overdetection is conversation, open eyes may be determined instead of closed eyes.
  • step ST102 curve fitting is performed for the upper and lower eyelids.
  • FIG. 5 shows how curve fitting is performed for one human eye.
  • both an upper eyelid curve and a lower eyelid curve connecting the inner corner coordinates and the outer corner coordinates are calculated.
  • the apex coordinates of the upper eyelid are calculated from the calculated upper eyelid curve
  • the apex coordinates of the lower eyelid are calculated from the calculated lower eyelid curve.
  • step ST103 the distance between the eyelids shown in FIG. 5 is measured based on the vertex coordinates of the upper and lower eyelids calculated in step ST102.
  • the inter-eyelid distance is the difference between the vertex coordinates of the upper eyelid and the vertex coordinates of the lower eyelid.
  • “when the eyes are closed” in the denominator means a case where it is determined that the eyes are closed using the above-mentioned degree of eye opening.
  • “Distance between eyelids when eyes are closed” is the distance between both eyelids in such a case, and is a reference value measured for each driver. This reference value is calculated by averaging the distance between the eyelids when the eyes are closed multiple times over a certain period of time.
  • step ST106 it is determined whether the eyelid opening rate is greater than or equal to a preset threshold.
  • a preset threshold the state of the eyes determined to be closed is distinguished from a state in which the eyes are truly closed and a state in which the eyes are determined to be closed despite downward gaze. Note that when the cause of overdetection is conversation, instead of the eyelid opening rate, it may be determined whether the speed of opening and closing movements is equal to or higher than a threshold value.
  • step ST106 determines whether the eyelid opening rate is less than the threshold. If “NO” in step ST106, that is, if the eyelid opening rate is less than the threshold, it is considered that the driver really closes his eyes. Therefore, in such a case, it is determined in step ST111 that there is no downward view.
  • step ST106 determines whether the eyelid opening rate is equal to or higher than the threshold value. If “YES” in step ST106, that is, if the eyelid opening rate is equal to or higher than the threshold value, it is considered that the driver is not closing his eyes but looking down. Therefore, in such a case, it is determined that downward viewing is present (steps ST109 and ST110). That is, a flag for the downward view indicator is set.
  • the overdetection factor reliability calculation unit 16 may further determine the reliability of the determination that downward gaze is present, that is, the reliability of the downward gaze index (steps ST107 to ST108).
  • step ST107 time-series variance of the coordinates of the upper and lower eyelids for determining the reliability of the downward vision index is calculated. If the time-series variance is equal to or greater than a predetermined threshold (“YES” in step ST108), it is considered that the result of upper and lower eyelid curve fitting in step ST102 is unstable for some reason, so step ST109 It is determined that downward gaze is present (low reliability). In other words, although downward gaze is a factor in overdetection, it is determined that the reliability is low. On the other hand, if the time-series variance is less than a predetermined threshold (“NO” in step ST108), it is determined in step ST110 that there is downward looking (high reliability).
  • the variance of the coordinates of the upper and lower eyelids is used as the reliability, but the time-series average value of the reliability of the degree of eye opening received as the face detection result may be taken and used as the reliability of the downward vision index.
  • step ST113 it is determined whether the drowsiness score output from the drowsiness score calculation unit 15 is equal to or higher than the drowsiness increase threshold for transitioning from the current drowsiness state to a drowsiness state with stronger drowsiness.
  • the drowsiness state is divided into multiple stages, and a threshold value for the drowsiness score for transition between each state is determined in advance.
  • the sleepiness score p is a sleepiness state with stronger sleepiness than "sleepiness state 2".
  • step ST113 determines whether or not the drowsiness increase threshold T23 for transitioning to "drowsiness state 3" is greater than or equal to. If the current sleepiness state is "Sleepiness State 1,” whether the sleepiness score p is equal to or higher than the sleepiness increase threshold T12 for transitioning from "Drowsiness State 1" to "Sleepiness State 2,” which is a sleepiness state with stronger sleepiness. Determine. If the determination result in step ST113 is "NO”, the process proceeds to step ST118, and a state transition is determined based on the drowsiness score. On the other hand, if "YES”, the process proceeds to step ST114.
  • step ST114 the overdetection factor index that is the output of the overdetection factor reliability calculation unit 16 is checked to determine whether there is an overdetection factor. If “NO”, the process proceeds to step ST118, and a state transition is determined based on the drowsiness score. If "YES”, the process proceeds to step ST115.
  • step ST115 the reliability that is the output of the overdetection factor reliability calculation unit 16 is checked, and it is determined whether the reliability is high. If "YES”, it is determined in step ST116 that there is no state transition. In other words, when the reliability of the overdetection factor is high, for example, when the reliability of downward gaze is high, the evaluation of the driver's drowsiness is not performed between multiple drowsiness states, but is performed on the previous frame. maintain a judgment regarding the sleepiness state level.
  • correction is performed in step ST117 so that the sleepiness state is less likely to increase.
  • the correction method may be a method of adjusting a transition threshold for the drowsiness score, or a method of making the transition to a state that is the average of the destination drowsiness state based on the drowsiness score and the current drowsiness state.
  • the drowsiness score calculated using the machine learning model based on the change in the feature amount is corrected using the overdetection factor index and its reliability determined using the rule-based method. Therefore, even in cases where it is difficult to distinguish between drowsiness and non-drowsiness using only a machine learning model, it is possible to distinguish between the cases using a rule-based method.
  • the drowsiness level can be estimated more accurately than when simply using the overdetection factor index.
  • the drowsiness estimation device 100 determines the transition of a drowsiness state using a drowsiness score based on a machine learning model, an overdetection factor index based on a rule, and its reliability.
  • the drowsiness state may increase depending on the determination result for a certain frame, and the drowsiness state may immediately decrease depending on the determination result for the next frame.
  • actual sleepiness does not decrease immediately after increasing.
  • the transition thresholds between different levels of drowsiness may be set to different values.
  • the transition threshold for determining the transition from the first drowsiness state where the level of drowsiness is stronger to the second drowsiness state where the level of drowsiness is weaker is set as the transition threshold for determining the transition from the second drowsiness state to the first drowsiness state. It may be set lower than the transition threshold for determining transition to the state.
  • sleepiness states with different sleepiness levels are represented in three stages.
  • the relationship between these three sleepiness states is ⁇ Sleepiness State 1'', which is the sleepiness state with the least sleepiness level, ⁇ Sleepiness State 3'', which is the sleepiness state with the strongest sleepiness level, and ⁇ Sleepiness State 3'', which is the sleepiness state with the strongest sleepiness level, and ⁇ Sleepiness State 3'', which is the sleepiness state with the strongest sleepiness level.
  • the user is in a "sleepy state 2" which is an intermediate sleepy state.
  • the drowsiness descending threshold T21 for determining the transition from drowsiness state 2 to drowsiness state 1 is lower by ⁇ than the drowsiness increase threshold T12 for determining the transition from drowsiness state 1 to drowsiness state 2.
  • the drowsiness decrease threshold T32 from drowsiness state 3 to drowsiness state 2 is set lower than the drowsiness increase threshold T23 from drowsiness state 2 to drowsiness state 3 by ⁇ .
  • the state transition determination unit 17 determines the transition of the sleepy state based on the state transition threshold set as described above. That is, in step ST118 in FIG. 6, the state transition determination section 17 determines the drowsiness state transition using the drowsiness score received from the drowsiness score calculation section 15, the drowsiness increase threshold, and the drowsiness decrease threshold.
  • the drowsiness estimation device (100) of Appendix 1 calculates a drowsiness score indicating the degree of drowsiness of the vehicle occupant using a trained machine learning model based on an image of the face of the vehicle occupant. (15), based on the image, determining whether there is an overdetection factor that affects the calculation of the drowsiness score, and when it is determined that the overdetection factor exists, the overdetection factor exists.
  • An over-detection factor reliability calculation unit (16) that calculates an over-detection factor index indicating that the over-detection factor index and the reliability of the over-detection factor index on a rule basis, the calculated drowsiness score, the calculated over-detection factor index, Based on the reliability of the calculated overdetection factor index, the drowsiness of the occupant is determined to be a first drowsiness state in which the degree of drowsiness is low and a second drowsiness state in which the degree of drowsiness is higher than the first drowsiness state. and a state transition determination unit (17) that determines which of a plurality of drowsiness states including drowsiness states.
  • the drowsiness estimating device is the drowsiness estimating device described in appendix 1, and the reliability of the overdetection factor index is higher than the first reliability, which is lower than the first reliability. and a second reliability with a high reliability, and the state transition determination unit determines whether the calculated reliability is higher than the second reliability when the calculated overdetection factor index exists. In this case, the determination regarding the previous drowsiness state is maintained without transitioning the determination regarding the drowsiness state among the plurality of drowsiness states.
  • the drowsiness estimation device is the drowsiness estimation device described in Supplementary note 1 or 2, in which the state transition determination unit is configured to detect the calculated overdetection factor index when the calculated overdetection factor index exists.
  • the reliability is the first reliability, correction is made so that the transition from the first drowsiness state to the second drowsiness state becomes more difficult.
  • the drowsiness estimating device is the drowsiness estimating device described in appendix 3, in which the state transition determination unit detects the calculated reliability when the calculated overdetection factor index exists. is the first reliability, the correction is performed by increasing the threshold for transitioning from the first drowsiness state to the second drowsiness state.
  • the drowsiness estimating device is the drowsiness estimating device described in appendix 3, in which the state transition determination unit determines the calculated reliability when the calculated overdetection factor index exists.
  • the correction is performed by transitioning to an intermediate state between the first drowsiness state and the second drowsiness state.
  • the drowsiness estimating device is the drowsiness estimating device described in appendix 2, in which the overdetection factor is the downward gaze of the occupant, and the overdetection factor reliability calculation unit is configured to The reliability of the overdetection factor index is calculated when the rate of eyelid opening of both eyelids is equal to or higher than a predetermined threshold value.
  • the drowsiness estimating device is the drowsiness estimating device described in appendix 6, in which the overdetection factor reliability calculation unit calculates the method of detecting drowsiness when the time-series variance of the vertex coordinates of the upper and lower eyelids is equal to or greater than a predetermined threshold.
  • the reliability of the overdetection factor index is determined to be the first reliability, and when the time series variance of the vertex coordinates of the upper and lower eyelids is less than the predetermined threshold, the overdetection factor index is determined to be the first reliability.
  • the reliability of the index is determined to be the second reliability.
  • the drowsiness estimation device is the drowsiness estimation device described in appendix 6, in which the overdetection factor reliability calculation unit calculates the reliability of the overdetection factor index based on the reliability of the degree of eye opening. .
  • the drowsiness estimating device according to appendix 9 is the drowsiness estimating device described in appendix 2, wherein the overdetection factor is the conversation of the occupant, and the overdetection factor reliability calculation unit is the drowsiness estimation device according to appendix 2.
  • the reliability of the overdetection factor index is calculated when the speed of movement is equal to or higher than a predetermined threshold.
  • the drowsiness estimation device is the drowsiness estimation device described in appendix 9, in which the overdetection factor reliability calculation unit calculates that the time-series variance of the vertex coordinates of the upper and lower lips is equal to or greater than a predetermined threshold. In this case, it is determined that the reliability of the overdetection factor index is the first reliability, and when the time series variance of the vertex coordinates of the upper and lower lips is less than the predetermined threshold, the overdetection is performed. The reliability of the factor index is determined to be the second reliability.
  • the drowsiness estimation device is the drowsiness estimation device described in any one of appendices 1 to 10, and includes a feature amount calculation unit (14) that calculates a drowsiness feature amount representing a sign of drowsiness based on the image.
  • the trained machine learning model is a trained model that has learned the relationship between the drowsiness feature and the drowsiness score
  • the drowsiness score calculation unit uses the trained machine learning model to: A drowsiness score of the occupant is calculated from the calculated drowsiness feature amount.
  • the drowsiness estimation device is the drowsiness estimation device described in any one of appendices 1 to 11, and includes a threshold value for determining a transition from the first drowsiness state to the second drowsiness state.
  • the value of is set higher than the threshold value for determining the transition from the second drowsiness state to the first drowsiness state.
  • the drowsiness estimation method of appendix 13 is a drowsiness estimation method performed by a drowsiness estimation device including a drowsiness score calculation unit (15), an overdetection factor reliability calculation unit (16), and a state transition determination unit (17), a step (ST11) in which the drowsiness score calculation unit calculates a drowsiness score indicating the degree of drowsiness of the vehicle occupant using a trained machine learning model based on an image of the face of the vehicle occupant;
  • the detection factor reliability calculation unit determines whether there is an overdetection factor that affects the calculation of the drowsiness score based on the image, and when it is determined that the overdetection factor exists, the overdetection factor is determined.
  • the drowsiness estimation technology of the present disclosure can be used as a technology for realizing the drowsiness estimation function of PMS.

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Traffic Control Systems (AREA)
PCT/JP2022/033046 2022-09-02 2022-09-02 眠気推定装置および眠気推定方法 Ceased WO2024047856A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/JP2022/033046 WO2024047856A1 (ja) 2022-09-02 2022-09-02 眠気推定装置および眠気推定方法
JP2024543738A JP7612118B2 (ja) 2022-09-02 2022-09-02 眠気推定装置および眠気推定方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/033046 WO2024047856A1 (ja) 2022-09-02 2022-09-02 眠気推定装置および眠気推定方法

Publications (1)

Publication Number Publication Date
WO2024047856A1 true WO2024047856A1 (ja) 2024-03-07

Family

ID=90099028

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/033046 Ceased WO2024047856A1 (ja) 2022-09-02 2022-09-02 眠気推定装置および眠気推定方法

Country Status (2)

Country Link
JP (1) JP7612118B2 (https=)
WO (1) WO2024047856A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118692062A (zh) * 2024-08-28 2024-09-24 杭州锐见智行科技有限公司 一种疲劳驾驶检测方法、装置、电子设备及可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010029537A (ja) * 2008-07-30 2010-02-12 Toyota Motor Corp 覚醒度判定装置
JP2011152218A (ja) * 2010-01-26 2011-08-11 Toyota Motor Corp 開眼度判定装置及び開眼度判定方法
JP2013156707A (ja) * 2012-01-26 2013-08-15 Nissan Motor Co Ltd 運転支援装置
JP2019068933A (ja) * 2017-10-06 2019-05-09 カルソニックカンセイ株式会社 推定装置及び推定方法
JP2019087170A (ja) * 2017-11-10 2019-06-06 アルパイン株式会社 瞼開閉判定装置および眠気検知装置

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010029537A (ja) * 2008-07-30 2010-02-12 Toyota Motor Corp 覚醒度判定装置
JP2011152218A (ja) * 2010-01-26 2011-08-11 Toyota Motor Corp 開眼度判定装置及び開眼度判定方法
JP2013156707A (ja) * 2012-01-26 2013-08-15 Nissan Motor Co Ltd 運転支援装置
JP2019068933A (ja) * 2017-10-06 2019-05-09 カルソニックカンセイ株式会社 推定装置及び推定方法
JP2019087170A (ja) * 2017-11-10 2019-06-06 アルパイン株式会社 瞼開閉判定装置および眠気検知装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN118692062A (zh) * 2024-08-28 2024-09-24 杭州锐见智行科技有限公司 一种疲劳驾驶检测方法、装置、电子设备及可读存储介质

Also Published As

Publication number Publication date
JPWO2024047856A1 (https=) 2024-03-07
JP7612118B2 (ja) 2025-01-10

Similar Documents

Publication Publication Date Title
KR101386823B1 (ko) 동작, 안면, 눈, 입모양 인지를 통한 2단계 졸음운전 방지 장치
US9117358B2 (en) Method for classification of eye closures
US11453401B2 (en) Closed eye determination device
US10311698B2 (en) Method and device for detecting a tiredness and/or sleeping state of a driver of a vehicle
JP6973928B2 (ja) 瞼開閉判定装置および眠気検知装置
CN110211335A (zh) 用于对车辆的驾驶员的微睡进行识别的方法和装置
JP7550879B2 (ja) 睡眠検出装置及び睡眠検出システム
CN111696312B (zh) 乘员观察装置
JP7240910B2 (ja) 乗員観察装置
US20070291231A1 (en) Eyeglass detection method
WO2024047856A1 (ja) 眠気推定装置および眠気推定方法
WO2023017595A1 (ja) 乗員状態判定装置、乗員状態判定方法、および、乗員状態判定システム
CN118876988B (zh) 一种驾驶状态监测方法、装置、电子设备及存储介质
KR20190044818A (ko) 운전자 모니터링 장치 및 방법
KR102906380B1 (ko) 운전자의 뇌졸증 전조증상을 검출하기 위한 장치 및 방법
CN107809952A (zh) 用于在使用眼睛张开度的情况下对眨眼事件和仪表察看进行区分的方法和装置
JP7812001B2 (ja) 開瞼度検出装置、開瞼度検出方法、および眠気判定システム
JP7843933B2 (ja) 眠気推定装置および眠気推定方法
JP7091632B2 (ja) 安全運転促進装置及び安全運転促進方法
JP2022161318A (ja) 顔認識装置
JP2022166702A (ja) 顔認識装置
Ibnouf et al. Drowsy driver detection system for poor light conditions
JP7594843B2 (ja) 乗員状態検知装置および乗員状態検知方法
JP7546815B2 (ja) 脇見判定装置および脇見判定方法
JP7595820B2 (ja) 乗員状態判定装置、乗員状態判定システム、乗員状態判定方法及びプログラム

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22957450

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024543738

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22957450

Country of ref document: EP

Kind code of ref document: A1