WO2023122439A1 - Awareness checker for enhancing collaborative driving supervision - Google Patents

Awareness checker for enhancing collaborative driving supervision Download PDF

Info

Publication number
WO2023122439A1
WO2023122439A1 PCT/US2022/081290 US2022081290W WO2023122439A1 WO 2023122439 A1 WO2023122439 A1 WO 2023122439A1 US 2022081290 W US2022081290 W US 2022081290W WO 2023122439 A1 WO2023122439 A1 WO 2023122439A1
Authority
WO
WIPO (PCT)
Prior art keywords
driver
vehicle
awareness
state
attention
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2022/081290
Other languages
English (en)
French (fr)
Inventor
Shabin MAHADEVAN
Rachel Gray ALEXANDER
Steffen Kuhn
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Arriver Software LLC
Original Assignee
Arriver Software LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Arriver Software LLC filed Critical Arriver Software LLC
Priority to JP2024537534A priority Critical patent/JP2025500379A/ja
Priority to EP22844415.4A priority patent/EP4452720A1/en
Priority to CN202280082875.3A priority patent/CN118434612A/zh
Priority to KR1020247019521A priority patent/KR20240122452A/ko
Publication of WO2023122439A1 publication Critical patent/WO2023122439A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W40/09—Driving style or behaviour
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/005—Handover processes
    • B60W60/0053—Handover processes from vehicle to occupant
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/005—Handover processes
    • B60W60/0059—Estimation of the risk associated with autonomous or manual driving, e.g. situation too complex, sensor failure or driver incapacity
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/0818—Inactivity or incapacity of driver
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/08—Interaction between the driver and the control system
    • B60W50/14—Means for informing the driver, warning the driver or prompting a driver intervention
    • B60W2050/146—Display means
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
    • B60W2420/403—Image sensing, e.g. optical camera
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2420/00—Indexing codes relating to the type of sensors based on the principle of their operation
    • B60W2420/40—Photo, light or radio wave sensitive means, e.g. infrared sensors
    • B60W2420/408—Radar; Laser, e.g. lidar
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2520/00—Input parameters relating to overall vehicle dynamics
    • B60W2520/10—Longitudinal speed
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/225—Direction of gaze
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00—Input parameters relating to occupants
    • B60W2540/229—Attention level, e.g. attentive to driving, reading or sleeping
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/20—Static objects
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/80—Spatial relation or speed relative to objects
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2555/00—Input parameters relating to exterior conditions, not covered by groups B60W2552/00, B60W2554/00
    • B60W2555/20—Ambient conditions, e.g. wind or rain
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2556/00—Input parameters relating to data
    • B60W2556/40—High definition maps
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2556/00—Input parameters relating to data
    • B60W2556/45—External transmission of data to or from the vehicle
    • B60W2556/50—External transmission of data to or from the vehicle of positioning data, e.g. GPS [Global Positioning System] data

Definitions

  • the invention relates to advanced driver assistance systems generally and, more particularly, to a method and/or apparatus for implementing an awareness checker for enhancing collaborative driving supervision .
  • ADAS advanced driver assistance systems
  • SAE L0 to L2+ Supervised advanced driver assistance systems
  • SAE L0 to L2+ Supervised advanced driver assistance systems
  • SAE L0 to L2+ Supervised advanced driver assistance systems
  • SAE L0 to L2+ Supervised advanced driver assistance systems
  • SAE L0 to L2+ Supervised advanced driver assistance systems
  • SAE L0 to L2+ Supervised advanced driver assistance systems
  • SAE L0 to L2+ Supervised advanced driver assistance systems
  • the invention concerns an apparatus comprising an interface and a control circuit .
  • the interface may be conf igured to ( i) receive sensor-based information f rom a plurality of sensor signals f rom a vehicle platform of a vehicle and environmental information about an environment of the vehicle , and ( ii) present one or more control signals to the vehicle platform .
  • the control circuit may be conf igured to ( i) detect whether an attention state of a driver is in an attentive state or an inattentive state in response to one or more of the plurality of sensor signals f rom the vehicle platform and a driver monitoring system, ( ii) assess whether the driver is suf f iciently attentive by monitoring the one or more of the plurality of sensor signals f rom the vehicle platform and the driver monitoring system, and ( iii) maintain an awareness of the driver to the environment of the vehicle by proactively interacting with the driver to determine whether a perception of the driver of the environment of the vehicle corresponds with the environmental information and the sensor-based information received by the interface .
  • FIG . 1 is a diagram illustrating a system in accordance with an embodiment of the invention .
  • FIG . 2 is a diagram illustrating an example implementation of an environmental monitoring systems block of FIG .
  • FIG . 3 is a diagram illustrating an implementation of an advanced driver-assistance systems (ADAS) feature control including an awareness checker in accordance with an example embodiment of the present invention .
  • ADAS advanced driver-assistance systems
  • FIGS . 4 - 12 are diagrams illustrating example interactions between a driver and a system in accordance with an example embodiment of the invention .
  • FIG . 13 is a diagram illustrating an awareness checking process in accordance with an example embodiment of the invention .
  • FIG . 14 is a diagram illustrating an electronic control unit implementing an advanced driver-assistance systems (ADAS) feature control system in accordance with an example embodiment of the invention .
  • ADAS advanced driver-assistance systems
  • Embodiments of the present invention include providing an awareness checker for enhancing collaborative driving supervision that may ( i) make the task of ADAS supervision more collaborative , ( ii) proactively engage the driver instead of react ively escalating af ter distraction, ( iii) focus on rewarding good engagement via collaboration (positive scale) over punishing bad supervising behavior (negative scale) , ( iv) nudge the driver to monitor the driving environment when the feature is active thereby reducing the possibly for non-driving related task involvement (NDRT) , (v) employ an in-vehicle digital assistant as a user interface to explore the idea of interactive safety, (vi) enable human- autonomy collaboration via interaction and positive reinforcement , (vii) provide a three level monitoring for driver engagement detection during assisted driving, and/or (viii) be implemented as one or more integrated circuits .
  • an awareness checker for enhancing collaborative driving supervision may ( i) make the task of ADAS supervision more collaborative , ( ii) proactively engage the driver instead of react
  • ADAS supervision is made more collaborative by proactively engaging the driver instead of reactively escalating responses after driver distraction is detected.
  • a system may be provided that focuses on rewarding good engagement via collaboration (positive scale) over punishing bad supervising behavior (negative scale) .
  • the feature control system may nudge the driver to monitor the driving environment when the feature is active thereby reducing the possibility for secondary, non-driving related task (NDRT) involvement .
  • NDRT non-driving related task
  • a new component e.g. , hardware, software, or combination thereof
  • the new component is generally referred to as an awareness checker.
  • the awareness checker may employ an in-vehicle digital assistant as a user interface to explore the idea of interactive safety (similar to level 3 in E-Gas three level monitoring below) .
  • the awareness checker generally enables human- autonomy collaboration via interaction and positive reinforcement.
  • the driver monitoring concept may be carried out in 3 levels.
  • a first level (or Level 1) may be implemented similarly to existing systems (e.g. , hyper traffic jam assistance (HTJA) attention monitor) .
  • a second level (e.g. , Level 2) may be implemented similarly to a human machine interaction monitor (HMIM) disclosed in co-pending U.S. patent application 17/493,144, filed October 4, 2021, which is herein incorporated by reference in its entirety.
  • a third level (e.g. , Level 3) may be implemented as an awareness checker (AC) to detect whether the driver has their mind on supervision of the autonomous feature.
  • Level 1 may be designated as a function level
  • Level 2 may be designated as a function monitoring level
  • Level 3 may be designated as a controller monitoring level.
  • Level 1 may contain the engine control functions (e.g. , implementation of requested engine torque, component monitoring, and input/output variable diagnostic to control the system reactions if a fault is detected.
  • Level 2 may detect the defective process of Level 1 functional software (e.g. , by monitoring the calculated torque values or vehicle acceleration, etc.) . In a case where a fault is detected, system reactions may be triggered.
  • Level 3 may implement a monitoring module. The monitoring module may be an independent part of the function controller (e.g. , ASIC or controller) , which tests the correctly executed program during a question-answer process.
  • ASIC application-specific integrated circuit
  • a system (or apparatus) 90 may implement an advanced driver-assistance system (ADAS) .
  • the system 90 may comprise a vehicle platform 92, a driver monitoring system (DMS) 94, a human machine (or vehicle) interface (HMI) 96, environmental monitoring systems (EMS) 98, and a function control module 100.
  • DMS driver monitoring system
  • HMI human machine interface
  • EMS environmental monitoring systems
  • the vehicle platform 92, the driver monitoring system (DMS) 94, the environment monitoring systems (EMS) 98, and the function control module 100 may be implemented as Automotive Safety Integrity Level (ASIL)
  • ASIL Automotive Safety Integrity Level
  • HMI human machine interface
  • QM Quality Management
  • Automotive Safety Integrity Level is a risk classification scheme defined by the ISO 26262 - Functional Safety for Road Vehicles standard. This is an adaptation of the Safety Integrity Level (SIL) used in IEC 61508 for the automotive industry.
  • the ASIL classification helps defining the safety requirements needed to be in line with the ISO 26262 standard to keep the risk to an acceptable level.
  • the ASIL is established by performing a risk analysis of a potentially hazardous scenario by looking at the Severity, Exposure and Controllability of the vehicle operating scenario. The safety goal for that hazardous scenario in turn carries the ASIL requirements.
  • the ASILs range from ASIL D, representing the highest degree of risk of a hazardous scenario turning into a mishap and highest degree of rigor needed to be applied in the assurance of the resultant safety requirements, to QM, representing applications with no automotive hazardous scenarios with unacceptable risk and, therefore, no safety requirements to manage under the ISO 26262 safety processes.
  • the level QM referring to "Quality Management", means that risk associated with a hazardous event is not unreasonable and does not therefore require safety measures in accordance with ISO 26262.
  • the intervening levels are simply a range of varying degrees of hazard risk levels and degrees of assurance and engineering rigor required.
  • ASILs establish safety requirements, based on the severity, probability and controllability of hazardous scenarios, for automotive components to be compliant with ISO 26262.
  • Systems like airbags, anti-lock brakes, and power steering require an ASIL D grade- the highest rigor applied to safety assurance-because the risks associated with their failure are the highest .
  • components like wiper systems require only an ASIL A grade. Headlights and brake lights generally would be ASIL B, as would be rear lights due to risk of rear collisions, while automatic emergency brake systems would generally be ASIL C due to risks associated with the unintended deceleration.
  • the vehicle platform 92, the DMS 94, the HMI 96, and the EMS 98 may provide input signals to the function control module 100.
  • the vehicle platform 92 may provide a number of input signals (e.g. , VEHICLE SPEED, WINDOWS & DOORS LOCK STATUS, TIRE PRESSURE, HVAC STATUS, ETC.) .
  • the signal VEHICLE SPEED may communicate vehicle (longitudinal) speed to the function control module 100.
  • the signal WINDOWS & DOORS LOCK STATUS may communicate whether windows are open or closed, and whether doors are open, closed, locked, or unlocked.
  • the signal TIRE PRESSURE may communicate an inflation status (e.g.
  • the signal HVAC STATUS may communicate whether an air conditioner (AC) or heater (HTR) are engaged, blower speed, passenger compartment temperature, etc.
  • the DMS 94 may provide input signals communicating information related to driver awareness (e.g. , driver eye movement, driver hand positions, steering angle, brake and accelerator pedal positions, etc.) .
  • the HMI 96 may provide a first input signal (e.g. , ACTIVATION REQUEST) , a second input signal (e.g. , DEACTIVATION REQUEST) , and a third signal (e.g. , DRIVER RESPONSE) to the function control module 100.
  • the signal ACTIVATION REQUEST may communicate a request from the driver to activate an ADAS feature controlled by the function control module 100.
  • the signal DEACTIVATION REQUEST may communicate a request from the driver to de-activate the ADAS feature controlled by the function control module 100.
  • the signal DRIVER RESPONSE may communicate responses from the driver to queries presented (e.g.
  • the HMI 96 may optionally present an input signal (e.g. , DRIVER INFO) communicating information about the particular driver operating the vehicle.
  • DRIVER INFO an input signal communicating information about the particular driver operating the vehicle.
  • the signal VEHICLE SPEED may be implemented as ASIL, and the signals ACTIVATION REQUEST, DEACTIVATION REQUEST, and DRIVER RESPONSE may be implemented as QM.
  • the function control module 100 may provide output signals to the vehicle platform 92 and the HMI 96.
  • the function control module 100 may present an output signal (e.g. , DECELERATION REQUEST) to the vehicle platform 92.
  • the signal DECELERATION REQUEST may be configured to allow the function control module 100 to bring the vehicle to a safe stop.
  • the signal DECELERATION REQUEST may be implemented as ASIL.
  • the function control module 100 may also present a number of signals (e.g. , DRIVER WARNING, QUESTIONS TO DRIVER, DRIVER MIND-ON-TASK STATUS, etc.) to the HMI 96.
  • the signal DRIVER WARNING may communicate information to cause the HMI 96 to present a particular warning to the driver.
  • the signal QUESTIONS TO DRIVER may communicate information to cause the HMI 96 to present a particular question to the driver .
  • the signal DRIVER MIND -ON- TASK STATUS may communicate information to cause the HMI 96 to present a particular avatar state to the driver .
  • the signals DRIVER WARNING and QUESTIONS TO DRIVER may be implemented as QM .
  • the signal DRIVER MIND-ON-TASK STATUS may be implemented as QM .
  • the signal DRIVER MIND-ON-TASK STATUS may be implemented as AS IL .
  • the function control module 100 may comprise a block (or circuit ) 102 and a block (or circuit ) 104 .
  • the block 102 may be implemented as a driver awareness estimator (DAE) .
  • the block 104 may be implemented as an ADAS feature mode manager .
  • the block 104 may be implemented as an autopilot mode manager .
  • other autonomous features may be implemented accordingly .
  • the blocks 102 and 104 are generally implemented as AS IL .
  • the signal VEHICLE SPEED may be presented to a f irst input of the block 102 and a f irst input of the block 104 .
  • the signals f rom the DMS 94 may be presented to a second input of the block 102 .
  • the signals from the HMI 96 may be presented to a third input of the block 102 and a second input of the block 104.
  • the signals from the EMS 98 may be presented to a fourth input of the block 102.
  • the block 102 may present a signal (e.g. , AWARENESS LEVEL) to a third input of the block 104 and a signal (e.g. , SUFFICIENTLY ATTENTIVE) to a fourth input of the block 104.
  • the signals AWARENESS LEVEL and SUFFICIENTLY ATTENTIVE may be implemented as ASIL.
  • the signal DRIVER INFO may be presented to a fifth input of the block 102.
  • the signal DRIVER INFO may be implemented as QM.
  • the block 102 may be configured as a driver awareness estimator (DAE) to methodically detect whether a driver is attentive and reach an appropriate safe state when the driver is no longer attentive.
  • DAE driver awareness estimator
  • the driver awareness estimator 102 may (i) provide a driver awareness level escalation regime that generates warnings at a number (e.g. , 3) of different levels via the HMI 96, (ii) detect insufficient attention related to short term shared visual attention (e.g.
  • the driver awareness estimator 102 may comprise a block (or circuit) 110, a block (or circuit) 112, and a block (or circuit) 114.
  • the block 110 may be implemented as an attention (or awareness) monitor.
  • the block 112 may be implemented as a human machine interaction monitor (HMIM) .
  • the block 114 may be implemented as an awareness checker (AC) .
  • the blocks 110 and 112 are generally implemented as ASIL.
  • the block 114 may be implemented as QM.
  • the signal DRIVER MIND-ON-TASK STATUS is subscribed to by safety related elements of the function control module 100 (e.g. , the block 110, the block 112, etc.)
  • the block 114 may be implemented as ASIL.
  • the signal VEHICLE SPEED may be presented to a first input of the block 110, a first input of the block 112, and a first input of the block 114.
  • the signals from the DMS 94 may be presented to a second input of the block 110, a second input of the block 112, and a second input of the block 114.
  • the signals from the HMI 96 may be presented to a third input of the block 114.
  • the signals from the EMS 98 may be presented to a fourth input of the block 114.
  • the block 110 may present the signal AWARENESS LEVEL to a third input of the block 112 and a fifth input of the block 114 .
  • the signal AWARENESS LEVEL may be implemented as AS IL .
  • the block 112 may present the signal SUFFICIENTLY ATTENTIVE to the third input of the block 104 .
  • the signal SUFFICIENTLY ATTENTIVE may be implemented as AS IL .
  • the block 114 may present a signal (e . g . , DRIVER MIND -ON- TASK STATUS) as an output signal .
  • the signal DRIVER MIND-ON-TASK STATUS may communicate a determination by the block 114 of an awareness level of the driver to the external and internal environments of the vehicle .
  • the signal MIND-ON-TASK STATUS may be stored in a local memory for use by other modules of the function control module 100 .
  • the signal DRIVER MIND-ON-TASK STATUS may be implemented as QM .
  • the signal DRIVER MIND- ON-TASK STATUS is subscribed to by the block 110 and/or the block 112
  • the signal DRIVER MIND-ON-TASK STATUS may be implemented as AS IL .
  • the signal DRIVER INFO may be presented to a f if th input of the block 112 and a sixth input of the block 114 .
  • the signal DRIVER INFO may be implemented as QM .
  • the block 110 , the block 112 , and the block 114 may be conf igured as a driver awareness estimator (DAE) to methodically detect whether a driver is attentive and, when the driver is no longer attentive , reach an appropriate safe state .
  • the attent ion monitor 110 may provide a driver awareness level escalation regime that generates warnings at a number (e . g . , 3 ) of dif ferent levels via the HMI 96 and may detect insuf f icient attention related to short term shared visual attention (e . g . , f rom involvement in secondary tasks) .
  • the attention monitor 110 may be implemented similarly to existing production attention tracking features on the market today that employ algorithms to track and categorize visual attention of drivers .
  • the driver awareness estimator (DAE) 102 does not rely solely on the attention monitor 110 to keep the driver functionally vigilant (and thereby meet a safety goal) due to numerous human factors related challenges in keeping the driver engaged in the driving task .
  • the driver awareness estimator (DAE) 102 may account for foreseeable misuse and the types of abuse of similar features that have been documented in the market today . For example , misuse can result in edge cases where the attention monitor 110 labels the driver as fully aware thereby keeping an automation feature controlled by the function control module 100 active while the driver cannot intervene in case of any hazardous event .
  • the driver awareness estimator (DAE) 102 in accordance with an embodiment of the invention generally utilizes the HMIM 112 and the AC 114 to provide additional functionality to methodically detect a driver who is no longer suf f iciently attentive to act as a safety net for the attention monitor 110 .
  • the HMIM 112 is generally conf igured to detect insuf f icient attention when the HMI 96 is not working or there is driver misuse .
  • the HMIM 112 may look at an of f - road glance distribution pattern of the driver by analyzing the toggle behavior between awareness levels reported by attention monitor 110 over a longer duration of time than the duration (or window) of time used by the attention monitor 110 .
  • the HMIM 112 may also look at the awareness of the driver to a surrounding environment of the vehicle by subscribing to and analyzing the signal DRIVER MIND-ON-TASK STATUS reported by awareness checker 114 .
  • long- term glance distribution patterns and driver mind-on- task status may be used to af fect the kinesthetics and longitudinal control of the vehicle platform 92 .
  • the HMIM 112 may focus on a longer- term assessment based on toggle behavior between attentiveness states reported by the attention monitor 110 .
  • a tunable (or programmable) number of transitions and an acceptable total time within each state of awareness may be def ined .
  • assisted driving e.g. , adaptive cruise control (ACC) , etc.
  • glance behavior as the gold standard, driver engagement may be calculated based on the glance distribution patterns.
  • the HMIM 112 may be configured to take into account transitions between different states of the signal DRIVER MIND-ON-TASK STATUS when calculating (or assessing) driver engagement.
  • the HMIM 112 generally prevents the driver from repeatedly entering into lower awareness states over a longer duration of time that may affect driver controllability by assessing the longer-term glance patterns and/or the mind-on- task status of the driver, and then triggering a transfer of control to the driver and a transition of the vehicle to a safe state (e.g. , via the signal DECELERATION REQUEST, etc.) .
  • the HMIM 112 may be implemented similarly to a human machine interaction monitor (HMIM) disclosed in co-pending U.S. patent application 17/493,144, filed October 4, 2021, which is herein incorporated by reference in its entirety.
  • the AC 114 may be implemented as part of the driver awareness estimator (DAE) 102 for evaluating driver attentiveness.
  • DAE driver awareness estimator
  • the DAE 102 may monitor the driver attention level (e.g. via the attention monitor 110) using eyes-on-road information provided by the driver monitoring camera inside the cabin. Based on the eyes-on-road information, the driver awareness level may be categorized into four different states: aware, temporarily unaware, unaware, and out of the loop.
  • the AC 114 may be configured to proactively maintain the driver in a continuously aware state (e.g. , using positive reinforcement or reward techniques) .
  • maintaining the driver in the aware state may be accomplished by asking questions about the external world in proximity of the vehicle as perceived by environmental sensors and/or the status of the vehicle inside the cabin as perceived by in-cabin sensors.
  • the specific contents of the questions may change dynamically based on the available information at hand for the AC 114 (e.g. , from the vehicle platform 92 and the environmental monitoring systems 98) .
  • the AC 114 cannot ask about a weather condition unless the AC 114 is aware of the current weather status (e.g. , from the internet, or cameras with a view of the exterior of the vehicle, etc.) .
  • the AC 114 may ask questions related to roadside objects or traffic signs, which may be validated against the information provided by perception or localization modules of the feature control module 100.
  • the driver mind-on- task status may be categorized into three different states: happy (or unconcerned) , concerned, and cautious.
  • FIG. 2 a diagram is shown illustrating an implementation of environmental monitoring systems of an advanced driver-assistance systems (ADAS) feature control including an awareness checker in accordance with an example embodiment of the present invention.
  • the function control module 100 may further comprise a localization circuit 103, a map interface 105, and a perception circuit 109.
  • the environmental monitoring systems 98 may comprise a block (or circuit) 210, a block (or circuit) 212, a block (or circuit) 214, a block (or circuit) 216, and a block (or circuit) 218.
  • the circuit 210 may implement a high-definition (HD) digital map.
  • the circuit 212 may implement satellite-based positioning.
  • the circuit 212 may comprise a global positioning system (GPS) or global navigation satellite system (GNSS) receiver.
  • GPS global positioning system
  • GNSS global navigation satellite system
  • the circuit 212 may be configured to determined a position of a vehicle based on satellite signals received.
  • the circuit 214 may implement a number of cameras (e.g.
  • the circuit 216 may implement a number of radar sensors (e.g. , forward looking radar, corner and/or side looking radar, rear looking radar, etc.) .
  • the circuit 218 may implement a wireless communication interface.
  • the circuit 218 may be configured to obtain weather information (e.g. , from the internet) .
  • the circuit 210 may have an input that may receive raw position information (e.g. , latitude, longitude, etc.) from the satellite-based positioning circuit 212.
  • the circuit 210 may be configured to present map horizon data to an input of the localization circuit 103 and an input of the map interface circuit 105.
  • the circuit 212 may be configured to also present the raw position data to the localization circuit 103.
  • the localization circuit 103 may be configured to present vehicle location information to the map interface circuit 105.
  • the map interface circuit 105 may be configured to generate map-based environmental information (e.g. , landmark and traffic signs information) in response to the map horizon data received from the HD map 210 and the vehicle location information received from the localization circuit 103.
  • the map interface circuit 105 may be configured to present the map-based environmental information to an input of the awareness checker 114.
  • the circuit 214 may comprise a number of on-board cameras including, but not limited to, a forward looking camera (FLC) 220.
  • the forward looking camera (FLC) 220 may present one or more signals (e.g. , VISION DETECTIONS) communicating vision detections to an input of the localization module (or circuit) 103 and an input of the perception module (or circuit) 109.
  • the circuit 216 may include, but is not limited to, front corner/side radars (FCR & FSR or FCSR) 222 and forward looking radar (FLR) 224.
  • the circuit 216 may present one or more signals (e.g. , RADAR DETECTIONS) communicating radar detections to an input of the circuit 103 and an input of the circuit 109.
  • the front corner/side radars (FCSR) 222 may present a first portion of the signal RADAR DETECTIONS communicating radar detections to an input of the circuit 103, and the forward looking radar (FLR) 224 may present a second portion of the signal RADAR DETECTIONS communicating radar detections to a second input of the perception module 109.
  • the localization circuit 103 may be configured to generate the vehicle location information presented to the map interface 105 in response to the raw position data received from the satellite-based positioning circuit 212, the map horizon data received from the HD map 210, the vision detections received from the FLC 220, and the radar detections received from the FCSR 222.
  • the perception module 109 may be configured to generate signals communicating static and dynamic object reporting in response to the vision detections from the forward looking camera (FLC) 220 and the radar detections from the forward looking radar (FLR) 224.
  • the static and dynamic object reporting signals generated by the perception module 109 may be presented to an input of the awareness checker 114.
  • the perception module 109 may be implemented as a software component.
  • the perception module 109 may be utilized in a SAE L2+ automation feature such as Hyper Traffic Jam Assistance (HTJA) .
  • the perception module 109 may utilize image data from the FLC 220 and point cloud data from a number of FCSRs 222a-222b and the FLR 224 to (i) detect the presence of landmarks, traffic signs and lights, pedestrians, and other objects, and (ii) analyze oncoming traffic, which may be further utilized by the awareness checker 114 to determine the awareness status of the driver (e.g. , as subjects of questions presented to the driver) .
  • the perception module 109 generally performs sensor fusion of the on-board sensors.
  • the perception module 109 generally fuses the image data from the FLC 220 and the point cloud data from the FCSRs 222a-222b and FLR 224 to (i) detect the presence of landmarks, traffic signs and lights, pedestrians, and other objects, (ii) track objects (or targets) in the environment around the vehicle, and (iii) analyze oncoming traffic .
  • the perception module 109 may detect objects in the surrounding environment of the vehicle based on the on-board sensor data.
  • the objects detected by the perception module 109 may be used as a cross-check on objects identified in the map data.
  • the map data may describe roadways and segments thereof and may also describe buildings and other items or objects (e.g. , lampposts, crosswalks, curbs, etc.) , location and directions of traffic lanes or lane segments (e.g. , the location and direction of a parking lane, a turning lane, a bicycle lane, or other lanes within a particular roadway) , traffic control data (e.g. , the location and instructions of signage, traffic lights, or other traffic control devices) , and/or any other map data that provides information to assist the ADAS system 90 in comprehending and perceiving the surrounding environment of the vehicle .
  • traffic control data e.g. , the location and instructions of signage, traffic lights, or other traffic control devices
  • the perception module 109 may be configured to determines a state for one or more of the objects in the surrounding environment of the vehicle.
  • the state generally describes a current state (or features) of the one or more objects.
  • the state for each object may describe an estimate of a current location (or position) of each object, a current speed (or velocity) of each object, a current acceleration of each object, a current heading of each object, a current orientation of each object, a size/shape/f ootprint (e.g. , as represented by a bounding shape such as a bounding polygon or polyhedron) of each object, a type/class (e.g.
  • the perception module 109 may also be configured to detect object free areas (e.g. , as represented by a bounding shape such as a bounding polygon or polyhedron) .
  • the perception module 109 may be configured to update state information for each object over time. Thus, the perception module 109 may detect and track objects, such as other vehicles, that are near the ego vehicle over time.
  • the perception module 109 may comprise a number of modules including, but not limited to, an object free area (OFA) module (or circuit) , a target tracking (TT) module (or circuit) , and a static perception (SP) module (or circuit) .
  • the perception module 109 may also comprise road estimation and electronic horizon reconstruction modules (not shown) , which may be used to produce self -generated map information from the on-board sensor-based information.
  • the object free area module may be configured to detect object free areas.
  • the object free area module may have a polygon output that may present a bounding shape such as a bounding polygon or polyhedron representing each object free area.
  • the target tracking module may be configured to detect and track objects, such as other vehicles, that are near the ego vehicle over time.
  • the target tracking module may have an output that may present a target tracking output.
  • the polygon output of the OFA module and the target tracking output of the target tracking module may be presented to inputs of the static perception module.
  • the static perception module my be configured to generate the static and dynamic object reporting signals that are presented to the awareness checker 114 in response to the polygon output received from the OFA module and the target tracking output received from the target tracking module.
  • the static perception module may use object information from the target tracking output of the target tracking module combined with analysis of the object free area (OFA) polygon output of the OFA module in order to provide the direction of the traffic and a confidence of the detection.
  • OFA object free area
  • the static perception module For detecting landmarks, traffic signs and lights, pedestrians, and other objects, the static perception module generally utilizes the object information from the target tracking output combined with analysis of the OFA polygon output to provide intersection type, object type, object location, landmark location, and intersection/ob j ect/landmark confidence .
  • the awareness checker 114 may be configured to generate the signal DRIVER MIND -ON- TASK STATUS based upon the static and dynamic object reporting signals received from a static perception module of the perception module 109, the landmark and traffic signs information received from the map interface 105, the weather information received from the circuit 218, the driver awareness level received from the awareness monitor 110, and the signals received from the vehicle platform 92.
  • the awareness checker 114 may be configured to store the signal DRIVER MIND-ON-TASK STATUS for use by other modules of the function control module 100.
  • the attention monitor 110 may subscribe to the signal DRIVER MIND -ON- TASK STATUS for use in determining whether to escalate the awareness level of the driver .
  • the HMIM 112 may subscribe to the signal DRIVER MIND-ON-TASK STATUS for use in determining whether the driver is suf f iciently attentive .
  • the apparatus (or system) 100 may be mounted totally within, or at least partially within a vehicle 50 .
  • the system (or apparatus) 100 may be implemented as part of an advanced driver-assistance systems (ADAS) electronic control unit (ECU) 90 .
  • ADAS advanced driver-assistance systems
  • the system 100 implementing the driver attentiveness estimator (DAE) 102 may be implemented within the ADAS ECU 90 of the vehicle 50 .
  • the ADAS ECU 90 may be connected to the vehicle platform 92 of the vehicle 50 .
  • the vehicle 50 may include the driver monitoring system (DMS) 94 , the human machine interface (HMI ) 96 , a forward looking camera ( FLC) 220 , a number of corner radar sensors 222a- 222d, a number of f ront side radar sensors (not shown) , a forward looking radar ( FLR) sensor 224 , a high-def inition (HD) map receiver 230 , a global navigation satellite system (GNSS) receiver 232 , and an inertial measurement unit ( IMU) 234 .
  • the vehicle 50 may also include LIDAR sensors and/or sonar sensors (not shown) .
  • the forward looking camera (FLC) 220 is generally used to detect and identify objects and road features in front of the vehicle 50.
  • the forward looking camera (FLC) 220 may be configured to provide stereoscopic vision with a 100-degree field of view (FOV) .
  • the forward looking camera (FLC) 220 may be used to detect road markings (e.g. , lane markings, etc.) , road signs, traffic lights, structures, etc.
  • the corner radar sensors 222a-222d and the forward looking radar (FLR) sensor 224 (and LIDAR and/or sonar sensors when present) are generally used to detect and track objects.
  • each of the corner radar sensors 222a-222d may have a 140-degree FOV.
  • the forward looking radar sensor (FLR) 224 may have two FOVs, an 18-degree FOV for long-range sensing and a 90-degree FOV for short range sensing.
  • the IMU 234 generally reports the orientation, angular velocity and acceleration, and forces acting on the vehicle 50.
  • the DMS 94, the HD map receiver 230, the GNSS receiver 232, the FLC 220, the FCRs 222a-222b, and the FLR 224 may be connected to the system 90.
  • the DMS 94, the HD map receiver 230, the GNSS receiver 232, the FLC 220, the FCRs 222a-222b, and the FLR 224 may be connected to the system 90 via one or more vehicle buses of the vehicle 50.
  • the DMS 94, the HD map receiver 230, the GNSS receiver 232, the FLC 220, the FCRs 222a-222b, and the FLR 224 may be connected to the system 90 via a wireless protocol.
  • the DMS 94 may convey driver attentiveness information to the system 90.
  • the FLC 220 may convey surrounding road information (e.g. , lane widths, marker types, lane marker crossing indications, and video) to the system 90.
  • the GNSS receiver 232 may convey position data (e.g. , latitude value, longitude value, adjustment information and confidence information) to the system 90.
  • the HD map receiver 230 may transfer map data to the system 90.
  • the FLC 220 may implement an optical sensor.
  • the FLC 220 may be an optical camera.
  • the FLC 220 is generally operational to provide the surrounding road information (or image data) to the system 90.
  • the road information may include, but is not limited to, lane width data, marker type data, lane change indicators, and video of a roadway ahead of the vehicle 50 within the field of view of the FLC 220.
  • the FLC 220 may be a color camera. The color may be useful for distinguishing between solid-yellow lane markers (e.g. , leftmost lane markers) from solid-white lane markers (e.g. , rightmost lane markers) .
  • the FLC 220 may provide an estimated lane width for at least a current lane in the center of the field of view of the FLC 220. In some embodiments, the FLC 220 may provide estimated lane widths for the lane(s) neighboring the center lane. In other embodiments, the FLC 220 may provide estimated lane widths for all of the lanes within the field of view of the FLC 220. The lane widths may be determined using standard image recognition methods and standard analysis methods implemented in the FLC 220. The FLC 220 may also identify all lane markers within the field of view of the FLC 220. When the FLC 220 crosses over a lane marker, the FLC 220 may notify the system 90 that a lane change is occurring. Identification of the lane markers and the lane changes may be determined using standard image recognition methods and standard analysis methods implemented in the FLC 220. The FLC 220 may transfer the road information to the system 90 via a vehicle bus or a wireless protocol.
  • Example sensors may include, but are not limited to, radar sensors, light detection and ranging (LiDAR) sensors, inertial sensors, thermal imaging sensors, and/or acoustic sensors. Some of the sensors may detect objects on the side of the road to provide estimations of a left boundary and a right boundary of the road. From the left boundary and the right boundary, a width of the road may be calculated. From the calculated width, an estimation of how many lanes probably fit within the width may be made based on a standard lane width.
  • LiDAR light detection and ranging
  • the sensors may estimate the current lane that the vehicle 50 occupies based on the relative distances of the sensors on the vehicle 50 to the left boundary and the right boundary of the road and the estimated number of lanes .
  • Lane crossovers may be determined by the sensors based on the estimated numbers of lanes and changes in the relative distances to the lef t boundary and/or the right boundary .
  • the system 90 may implement a control circuit (e . g . , an electronic control unit ) .
  • the system 90 is generally operational to keep track of the current lane that the vehicle 50 occupies and correct the current position of the vehicle 50 to a center of the current lane .
  • the tracking may be based on the satellite position data received in the GNSS receiver 232 , the map data received f rom the HD map receiver 230 , and the road information received in the vision detections f rom the FLC 220 and the radar detections received f rom the FCRs 222a- 222b and the FLR 224 .
  • the satellite position data may include an adj ustment value and a corresponding conf idence value .
  • the GNSS receiver 232 may implement a satellitenavigation device .
  • the GNSS receiver 232 may include a Global Positioning System (GPS) receiver .
  • GPS Global Positioning System
  • Other types of satellite-navigation devices may be implemented to meet the design criteria of a particular application .
  • the GNSS receiver 232 is generally operational to provide the latitude data and the longitude data of the vehicle 50 based on the GNSS signals received f rom a number of satellites .
  • the GNSS receiver 232 may also be operational to adj ust the latitude data and the longitude data based on the adjustment value and a corresponding confidence value received from the system 90.
  • the confidence value may have a range from zero (e.g. , unreliable) to one (e.g.
  • the GNSS receiver 232 may correct the latitude data and the longitude data per the adjustment value. If the confidence value is below a low threshold (e.g. , ⁇ 0.3) , the GNSS receiver 232 may ignore the adjustment value. If the confidence value is between the high threshold and the low threshold, the GNSS receiver 232 may apply a correction to both the latitude data and the longitude data that is a linear weighting based on the degree of confidence.
  • a high threshold e.g. , >0.7
  • a low threshold e.g. ⁇ 0.3
  • the HD map receiver 230 may implement a radio- frequency receiver.
  • the HD map receiver 230 may be operational to receive the map data from an antenna (not shown) .
  • the map data may be converted to a digital form and presented to the system 90.
  • the AC 114 may be configured as described above in connection with FIGS. 1 and 2.
  • the AC 114 may be configured to monitor the output of the attention monitor 110.
  • the AC 114 may be configured to provide mind-on-task assessments of the driver that may be utilized by the attention monitor 110 and/or the HMIM 112 to determine when to transition control back to the driver and send the vehicle into a safe stop mode in response to the driver to be sufficiently unaware to safely supervise the active automation feature.
  • FIG. 4 a diagram is shown illustrating the awareness checker (AC) 114 when the automation feature is being activated.
  • a picture 500a is shown illustrating a view within a cockpit (or cabin or passenger compartment) of a vehicle from the perspective of the driver.
  • the driver activates the automation feature (e.g. , pushing a button on the steering wheel hub, the dash, etc.) .
  • the AC 114 assesses the state of the driver (e.g. , using the signal AWARENESS LEVEL) .
  • FIG. 5 a diagram is shown illustrating the AC 114 when the automation feature is activated.
  • a picture 500b is shown illustrating a view within a cockpit of a vehicle from the perspective of the driver.
  • the driver is driving hands-free with the ADAS feature remaining active on the highway.
  • the awareness checker 114 determines the driver is attentive, the awareness checker 114 signals the virtual assistant to reflect a state of happiness.
  • the reflection of happiness may be implemented as a smiling face avatar on a display of the HMI 96.
  • the happiness state may be indicated by a color of the avatar (e.g. , green for happiness) .
  • the happiness state may be displayed across the cabin of the vehicle using ambient light, audio, etc.
  • FIG. 6 a diagram is shown illustrating the AC 114 proactively checking the awareness of the driver through positive reinforcement.
  • a picture 500c is shown illustrating the view within the cockpit of the vehicle from the perspective of the driver.
  • the awareness checker 114 may signal the virtual assistant to provide positive reinforcement (e.g. , with a compliment) .
  • the virtual assistant may use speakers of an audio system of the vehicle to give the driver a message such as "I love how you keep us safe by watching the road,” "you are good at keeping a safe distance from other vehicles,” etc. Other messages may be implemented accordingly.
  • FIG. 7 a diagram is shown illustrating the AC 114 proactively checking the awareness of the driver through inquiry.
  • a picture 500d is shown illustrating the view within the cockpit of the vehicle from the perspective of the driver.
  • the awareness checker 114 may periodically signal the virtual assistant to ask a question about the environment of the vehicle.
  • the virtual assistant may use the speakers of the audio system of the vehicle to ask a question such as "What color is the car in front of you?"
  • FIG. 8 a diagram is shown illustrating the AC 114 assessing the awareness of the driver based on a response of the driver to a question.
  • a picture 500e is shown illustrating the view within the cockpit of the vehicle from the perspective of the driver.
  • the driver answers the question presented by the virtual assistant correctly.
  • the virtual assistant may reflect happiness for the correctly answered question (e.g. , by continuing to present a smiling avatar face on the display, etc . ) .
  • FIG. 9 a diagram is shown illustrating the AC 114 assessing the awareness of the driver based on a response of the driver to a question.
  • a picture 500f is shown illustrating the view within the cockpit of the vehicle from the perspective of the driver.
  • the driver may answer the question presented by the virtual assistant incorrectly.
  • the virtual assistant may reflect a state of concern for the incorrectly answered question by showing a caution state.
  • the caution state may be exhibited by presenting the avatar having a concerned expression.
  • the caution state may be indicated by changing a color of the avatar (e.g. , to yellow) .
  • the virtual assistant may use speakers of an audio system of the vehicle to ask the driver a follow up question such as "Are you sure?" or "Are we looking at the car in front of us?" .
  • FIG. 10 a diagram is shown illustrating the AC 114 detecting the driver being inattentive.
  • a picture 500g is shown illustrating the view within the cockpit of the vehicle from the perspective of the driver.
  • the driver may be driving hands-free with the ADAS feature remaining active on the highway.
  • the driver may check a text message and becomes inattentive.
  • a display on the dash that is part of the HMI 96 may alert the driver to pay attention to the road.
  • the virtual assistant may reflect the caution state.
  • FIG. 11 a diagram is shown illustrating the AC 114 detecting the driver being inattentive.
  • a picture 500h is shown illustrating the view within the cockpit of the vehicle from the perspective of the driver.
  • a display on the dash that is part of the HMI 96 alerts the driver to pay attention to the road.
  • the AC 114 may work in cooperation with the HMIM 112 to restore the attention of the driver to the task at hand.
  • the virtual assistant may reflect the caution request state presented via the HMI 96 using the display.
  • the virtual assistant may also use the speakers of the audio system of the vehicle to give the driver the message "Eyes on the road please! .
  • FIG. 12 a diagram is shown illustrating the AC 114 after determining the driver has continued to be distracted.
  • a picture 500i is shown illustrating the view within the cockpit of the vehicle from the perspective of the driver.
  • the display on the dash that is part of the HMI 96 may be used by the attention monitor 110 and/or the HMIM 112 to alert the driver to take control of the vehicle.
  • the virtual assistant may reflect the deactivation request presented by the HMI 96.
  • the virtual assistant may reflect concern and use the speakers of the audio system of the vehicle to give the driver the message "Please Take Control!
  • the deactivation request state may also be exhibited by changing the color of the personal assistance avatar (e.g. , to red) .
  • the vehicle may begins to decelerate to 5 kph.
  • the AC 114 may provide notification of the mind-on-task status of the driver by using the signal DRIVER-MIND- ON_TASK STATUS.
  • the AC 114 may set the signal DRIVER-MIND-ON_TASK STATUS to reflect happiness with the awareness level of the driver.
  • the AC 114 may provide notification by set the signal DRIVER-MIND-ON_TASK STATUS to a state other than happiness.
  • the AC 114 may implement an escalation scheme in the levels presented by the signal DRIVER-MIND-ON_TASK STATUS (e.g.
  • the attention monitor 110 and/or the HMIM 112 may utilize the signal DRIVER-MIND-ON_TASK STATUS in assessing the suf f iciency of awareness exhibited by the driver .
  • the attention monitor 110 may be conf igured to move between an aware state to a temporarily unaware state based on the signal DRIVER MIND -ON- TASK STATUS .
  • the HMIM 112 may track transitions in the levels of the signal DRIVER MIND-ON-TASK STATUS and make an assessment based on a respective threshold .
  • the attention monitor 110 and/or the HMIM 112 may then initiate a request that the driver take over operation of the vehicle and safely discontinue the automation feature when the awareness (or attention) level is considered insuf f icient for the particular automation feature . Because discontinuing activity of the automation feature at high speed may be unsafe as well , particularly when the driver is known to not be aware , the HMI 96 may be used to present a request for the driver to take over operation of the vehicle . In an example , the attention monitor 110 and/or the HMIM 112 may facilitate a safe transition by slowly degrading operation of the automation feature to a safe state (e . g . , decelerating to a safe speed and/or a safe stop) before completely discontinuing collaborative operation .
  • a safe state e . g .
  • a flow diagram is shown illustrating an example awareness checking process in accordance with an embodiment of the invention.
  • a method (or process) 600 may be implemented to proactively maintain the driver in a continuously aware state.
  • the method 600 may be implemented by the awareness checker 114.
  • the method 600 may use positive reinforcement or reward techniques.
  • the method 600 may comprise a step (or state) 602, a step (or state) 604, a decision step (or state) 606, a step (or state) 608, a step (or state) 610, a step (or state) 612, a step (or state) 614, a decision step (or state) 616, a step (or state) 618.
  • the method 600 may begin in the step 602 and move to the decision step 604.
  • the method 600 may determine whether an automation feature (e.g. , auto pilot, adaptive cruise control, hyper traffic jam assistance, etc.) has been activated.
  • the method 600 may subscribe to the signal AWARENESS LEVEL to determine whether the driver is considered to be in an aware state by the attention monitor 110.
  • the method 600 may remain in the decision step 604.
  • the method 600 may move to the step 606.
  • the method 600 may acquire environmental information about the surroundings of the vehicle and conditions within the vehicle.
  • the method 600 may utilize information about the external world in proximity of the vehicle as perceived by environmental sensors (e.g. , cameras, radars, a wireless communication transceiver, an inertial measurement unit, a high-definition map, satellite signal receiver, etc.) and/or the status of the vehicle inside the cabin as perceived by in-cabin sensors (e.g. , cameras, radars, HVAC sensors, etc.) .
  • environmental sensors e.g. , cameras, radars, a wireless communication transceiver, an inertial measurement unit, a high-definition map, satellite signal receiver, etc.
  • the environmental information may be stored in a memory of an electronic control unit (ECU) for reference as needed.
  • the acquisition of the environmental information may be a continuous process while the automation feature is active.
  • the method 600 may be configured to maintain the driver in the aware state by asking the driver questions about the external world in proximity of the vehicle as perceived by environmental sensors and/or the status of the vehicle inside the cabin as perceived by in-cabin sensors.
  • the method 600 may determine the specific contents of the questions based on the available information on hand from the various sensors (e.g. , from the vehicle platform 92 and the environmental monitoring systems 98) .
  • the method 600 may be configured to dynamically change the question asked based on the available information. For example, the awareness checker 114 cannot ask about a weather condition unless the awareness checker
  • the awareness checker 114 may ask questions related to roadside objects or traffic signs, which may be validated against the information provided by perception or localization modules of the feature control module 100.
  • the method 600 may move to the step 610.
  • the method 600 may present the selected question and wait for a response from the driver.
  • the awareness checker 114 may be configured to utilize a the human machine interface (HMI) 96 of the vehicle to communicate the question to the driver and receive an answer to the question from the driver.
  • the awareness checker 114 may be configured to use speakers of a radio or an infotainment system of the vehicle to ask the question.
  • the awareness checker 114 may be configured to use a microphone of the vehicle platform or infotainment system of the vehicle to receive the answer to the question.
  • the method 600 may move to the decision step 612.
  • the method 600 may determine whether the answer received from the driver matches the information about the external world in proximity of the vehicle as perceived by the environmental sensors and/or the status of the vehicle inside the cabin as perceived by in-cabin sensors. When the answer from the driver substantially matches the information about the external world in proximity of the vehicle and/or the status of the vehicle inside the cabin, the method 600 may return to the step 604. When the answer from the driver does not match the information about the external world in proximity of the vehicle and/or the status of the vehicle inside the cabin, the method 600 may move to the step 614.
  • the awareness checker 114 may be configured generally to use positive reinforcement or reward techniques rather than punishment.
  • the method 600 may calmly ask the driver a followup question (e.g. , to confirm the answer, etc.) rather than reacting negatively (e.g. , deactivating the feature, correcting the driver, etc.) to an answer that is perceived to be incorrect.
  • the method 600 may move to the decision step 616.
  • the method 600 may determine whether the followup answer received from the driver matches the information about the external world in proximity of the vehicle as perceived by the environmental sensors and/or the status of the vehicle inside the cabin as perceived by in-cabin sensors. When the answer from the driver substantially matches the information about the external world in proximity of the vehicle and/or the status of the vehicle inside the cabin, the method 600 may return to the step 604 . When the answer f rom the driver still does not match the information about the external world in proximity of the vehicle and/or the status of the vehicle inside the cabin, the method 600 may move to the step 618 . In the step 618 , the method 600 may change a value of a mind-on- task status maintained by the awareness checker 114 according to a def ined escalation scheme . The process 600 may then return to the step 604 .
  • the change in value of a mind-on- task status may be used by the attention monitor 110 and/or the HMIM 112 to may be used to af fect the kinesthetics and longitudinal control of the vehicle platform 92 (e . g . , according to respective escalation schemes of the attention monitor 110 and the HMIM 112 ) .
  • an apparatus 800 may implement an electronic control unit or module (ECU) .
  • the electronic control module (ECU) 800 may be implemented as a domain controller (DC) .
  • the ECU 800 may be implemented as an active safety domain master (ASDM) .
  • the ECU 800 may be configured to control activation of one or more features (or functions) of an ADAS component of a vehicle.
  • the function control 100 may be implemented within the ECU 800.
  • the ECU 800 may be connected to the vehicle platform 92, the driver monitoring system (DMS) 94, the human machine interface (HMI) 96, and the environmental monitoring systems (EMS) 98.
  • the environmental monitoring systems (EMS) 98 may comprise the HD map receiver 210, the GNSS receiver 212, the cameras 214, the radar sensors 216, a wireless communication transceiver 218, the inertial monitoring unit (IMU) 230, and an electronic bus 802 of the vehicle.
  • the ECU 800 may be configured to (i) receive the signals VEHICLE SPEED, DEACTIVATION REQUEST, ACTIVATION REQUEST, DRIVER RESPONSE, and DRIVER INFO from the vehicle systems and (ii) communicate the signals DECELERATION REQUEST, QUESTIONS TO DRIVER, DRIVER MIND-ON- TASK STATUS, and DRIVER WARNING to the systems of the vehicle.
  • the ECU 800 may be connected to a block (or circuit) 802.
  • the circuit 802 may implement an electronic bus of the vehicle.
  • the electronic bus 802 may be configured to transfer data between the ECU 800 and the vehicle platform 92, the DMS 94, the HMI 96, the HD map receiver 210, the GNSS receiver 212, the cameras 214 (e.g. , the forward looking camera (FLC) 220) and corner viewing, side viewing and/or rear viewing cameras) , the radars 216 (e.g. , corner/side radar sensors 222a-222n, the forward looking radar (FLR) sensor 224, etc.) , the wireless communication transceiver 218, and the inertial measurement unit 230.
  • FLC forward looking camera
  • FLR forward looking radar
  • the electronic bus 802 may be implemented as a vehicle Controller Area Network (CAN) bus.
  • the electronic bus 802 may be implemented as an electronic wired network and/or a wireless network (e.g. , Wi-Fi, BLUETOOTH, ZIGBEE, etc.) .
  • the electronic bus 802 may connect one or more components of the vehicle 50 to enable a sharing of information in the form of digital signals (e.g. , a serial bus, an electronic bus connected by wiring and/or interfaces, a wireless interface, etc.) .
  • the ECU 800 generally comprises a block (or circuit) 820, a block (or circuit) 822, and a block (or circuit) 824.
  • the circuit 820 may implement a processor.
  • the circuit 822 may implement a communication port.
  • the circuit 824 may implement a memory.
  • Other blocks (not shown) may be implemented (e.g. , filters, clocks/timers , I/O ports, power connectors, interfaces, etc.) .
  • the number and/or types of circuits implemented by the module 800 may be varied according to the design criteria of a particular implementation.
  • the processor 820 may be implemented as a microcontroller, a multi-thread microprocessor, or any combination thereof .
  • the processor 820 may comprise a block (or circuit) implementing the localization module 103, a block (or circuit) implementing the mode manage 104, a block (or circuit) implementing the perception module 109, a block (or circuit) implementing the attention monitor 110, a block (or circuit) implementing the human machine interaction monitor 112, a block (or circuit) implementing the awareness checker 114, a block (or circuit) 850 implementing a GNSS module, and/or a block (or circuit) 852 implementing a map module.
  • the map module 852 may implement the map interface 105.
  • the processor 820 may comprise other components such as a filter and a clock (not shown) .
  • the processor 820 may be a combined (e.g. , integrated) chipset implementing processing functionality.
  • the processor 820 may be comprised of a number of separate circuits (e.g. , a microcontroller, a multithread microprocessor, a digital signal processor (DSP) , a graphics processing unit (GPU) , etc.) .
  • the design of the processor 820 and/or the functionality of various components of the processor 820 may be varied according to the design criteria of a particular implementation.
  • the processor 820 is shown sending data to and/or receiving data from the vehicle platform 92, the communication port 822, and/or the memory 824. However, the processor 820 may be configured to implement additional data and/or control paths to meet design criteria of a particular application.
  • the memory 824 may comprise a block (or circuit) 860 and a block (or circuit) 862.
  • the block 860 may store data utilized by the driver awareness (or attentiveness) estimator (DAE) 102 and the mode manager 104.
  • the block 862 may store computer readable instructions (e.g. , instructions readable by the processor 820) .
  • the DAE data 860 may store various data sets 870a-870n.
  • the data set 870a may be utilized by the attention monitor 110 to store awareness states (or levels) .
  • the data set 870a may keep track of transitions to the temporarily unaware state, the unaware state, and the out-of -loop state.
  • the data set 870b may be utilized by the human machine interaction monitor (HMIM) 112 to store an attentiveness state (or level) , a long-term glance distribution, a count of transitions to the temporarily unaware state, a count of transitions to the unaware state, a count of transitions to the out-of -loop state, etc.
  • the data set 870c may be utilized by the awareness checker 114 to store driver mind-on-task status.
  • the data set 870d may be utilized by the awareness checker 114 to store a lookup table or database of questions that may be used to query the driver.
  • the data set 870e may be utilized to store driver information that may be used by the HMIM 112 and the awareness checker 114.
  • the memory 824 may also be configured to store other data sets 870n (e.g. , environmental data, etc.) .
  • the other data sets 870n may comprise parameters (e.g. , coefficients) and/or calibration data used to transform data received from the various sensors (e.g. , FLC, FLR, FCR, FCS, IMU, etc.) of the vehicle.
  • the calibration data 870n may provide many sets of coefficients (e.g. , one set of coefficients for each of the sensors) .
  • the calibration data 870n may be updatable.
  • the calibration data 870n may store current values as coefficients for the sensors and as the data from the sensors drifts the module 800 may update the calibration data 870n in order to maintain accuracy.
  • the format of the calibration data 870n may vary based on the design criteria of a particular implementation.
  • the other data 870n may store glance distributions for a plurality of drivers.
  • the other data 870n may store past data values of the calibration data and/or current data values of the calibration data. The past and current data values of the calibration data may be compared to determine trends used to extrapolate and/or predict potential future values for the calibration data.
  • the processor 820 may be configured to execute stored computer readable instructions (e.g. , the instructions 862 stored in the memory 824) .
  • the processor 820 may perform one or more steps based on the stored instructions 862.
  • steps of the instructions 862 may be executed/perf ormed by the processor 820 and may implement one or more of the attention monitor 110, the human machine interaction monitor 112, the awareness checker 114, the localization module 103, the map interface 105, the perception module 109, and the mode manager 104.
  • the instructions executed and/or the order of the instructions 862 performed by the processor 820 may be varied according to the design criteria of a particular implementation .
  • the communication port 822 may allow the module 800 to communicate with external devices such as the vehicle platform 92, the driver monitoring system 94, the human machine interface 96, and the environmental monitoring systems 98.
  • the module 800 is shown connected to the external electronic bus 802.
  • information from the module 800 may be communicated to an infotainment device for display and/or presentation to a driver.
  • a wireless connection e.g. , Wi-Fi, BLUETOOTH, cellular, etc.
  • a portable computing device e.g. , a smartphone, a tablet computer, a notebook computer, a smart watch, etc.
  • a portable computing device e.g. , a smartphone, a tablet computer, a notebook computer, a smart watch, etc.
  • information from the module 800 may be displayed and/or otherwise communicated (e.g. , aurally, etc.) to a user.
  • the ECU 800 may further comprise a filter and a clock.
  • the filter may be configured to perform a linear quadratic estimation.
  • the filter may implement a Kalman filter.
  • the filter may operate recursively on input data to produce a statistically optimal estimate.
  • the filter may be used to calculate the position coordinates and/or estimate the accuracy of the position coordinates utilized by the GNSS module 850 and the map module 852.
  • the filter may be implemented as a separate module.
  • the filter may be implemented as part of the memory 824 (e.g. , the stored instructions 862) .
  • the implementation of the filter may be varied according to the design criteria of a particular implementation.
  • the clock may be configured to determine and/or track a time. The time determined by the clock may be stored as a time stamp in the other data 870n. In some embodiments, the clock may be configured to compare time stamps received from the GNSS receiver 212.
  • the module 800 may be configured as a chipset, a system on chip (SoC) and/or a discrete device.
  • the module 800 may be implemented as an electronic control unit (ECU) .
  • the module 800 may be configured to control activation of one or more ADAS f eatures/functions .
  • an underlying objective of the HMIM 112 and the AC 114 in accordance with an embodiment of the invention is to provide a monitoring functionality that ensures the sufficient controllability of a supervising driver to possible hazardous events.
  • the HMIM 112 and the AC 114 may achieve suf f icient controllability by ensuring driver engagement .
  • the HMIM 112 and the AC 114 may ensure driver engagement by monitoring glance distribution patterns and driver mind-on task status as a measure of attentiveness .
  • a few example functionality iterations of the HMIM 112 and the AC 114 may be implemented to mitigate false positives during decision making .
  • the HMIM 112 may check the delta change in eye glance shif t pre and post HMI warnings to determine whether the warning is being conveyed to the driver and to safeguard against omission of HMI messages .
  • the HMIM 112 may subscribe to a signal indicating "eyes on road” in real time f rom a camera of the driver monitoring system 94 .
  • the signal indicating "eyes on road” may be used as feedback to quickly evaluate whether there is improvement in glance distribution af ter each escalation warning . I f not , the HMIM 112 may fail safe appropriately .
  • the HMIM 112 may intentionally try to take the eyes of the driver of f the road momentarily when the attention monitor 110 reports the driver being in the "aware" state for a longer duration than expected to safeguard against false positive f rom the camera of the DMS 94 and attention monitor attention levels .
  • the HMIM 112 may send directed prompts to divert attention of the driver away f rom the road (e . g . , when judged to be safe to do so by subscribing to environmental information from onboard sensors, GNSS, HD map, etc.) and verify whether the front-end signal chain (e.g. , DMS 94, attention monitor 110, awareness checker 114, etc.) detect the diverted attention. If not, the HMIM 112 may fail safe appropriately.
  • the front-end signal chain e.g. , DMS 94, attention monitor 110, awareness checker 114, etc.
  • HMIM 112 and the AC 114 may be integrated into the HMIM 112 and the AC 114 as inputs to form a holistic driver state estimation.
  • the HMIM 112 and the AC 114 may subscribe to hands on steering wheel, pedal information, seat sensors, seat belt status, etc. to form a holistic driver state estimation model.
  • the HMIM 112 and the AC 114 may leverage feedback from each of these inputs to detect and mitigate inattentiveness.
  • the HMIM 112 and the AC 114 may be developed with artificial intelligence / machine learning (AI/ML) based non-deterministic algorithms to baseline a driver attentiveness profile for each individual driver and track inattentiveness against the baseline of the particular driver.
  • AI/ML artificial intelligence / machine learning
  • each vehicle may implement a generic HMIM 112 and the AC 114 that over time may customize themselves by tracking and learning about an inattentiveness profile of the driver by baselining the inattentiveness profile of the driver against the glance distribution of the same driver during manual driving. Inattentiveness during supervised driving may then be flagged when the inattentiveness exceeds the threshold noted during prior manual drivin .

Landscapes

  • Engineering & Computer Science (AREA)
  • Automation & Control Theory (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Traffic Control Systems (AREA)
  • Emergency Alarm Devices (AREA)
PCT/US2022/081290 2021-12-22 2022-12-09 Awareness checker for enhancing collaborative driving supervision Ceased WO2023122439A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP2024537534A JP2025500379A (ja) 2021-12-22 2022-12-09 協調型運転監督を強化するための意識度チェッカ
EP22844415.4A EP4452720A1 (en) 2021-12-22 2022-12-09 Awareness checker for enhancing collaborative driving supervision
CN202280082875.3A CN118434612A (zh) 2021-12-22 2022-12-09 用于增强协作驾驶监督的警觉性检查器
KR1020247019521A KR20240122452A (ko) 2021-12-22 2022-12-09 협업형 주행 감독을 강화하기 위한 인식 검사기

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US17/559,733 US12600363B2 (en) 2021-12-22 2021-12-22 Awareness checker for enhancing collaborative driving supervision
US17/559,733 2021-12-22

Publications (1)

Publication Number Publication Date
WO2023122439A1 true WO2023122439A1 (en) 2023-06-29

Family

ID=84981769

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2022/081290 Ceased WO2023122439A1 (en) 2021-12-22 2022-12-09 Awareness checker for enhancing collaborative driving supervision

Country Status (7)

Country Link
US (1) US12600363B2 (https=)
EP (1) EP4452720A1 (https=)
JP (1) JP2025500379A (https=)
KR (1) KR20240122452A (https=)
CN (1) CN118434612A (https=)
TW (1) TW202332608A (https=)
WO (1) WO2023122439A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026057343A1 (de) * 2024-09-10 2026-03-19 Volkswagen Aktiengesellschaft Verfahren zur prüfung einer aufmerksamkeit eines insassen eines fahrzeuges und vorrichtung sowie fahrzeug umfassend eine entsprechende vorrichtung

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP4162336B1 (en) * 2020-06-09 2024-05-15 Volvo Autonomous Solutions AB Method and control unit for handling safe stop mode of a vehicle
US20240067110A1 (en) * 2022-08-31 2024-02-29 Qualcomm Incorporated Power management integrated circuit (pmic) power supply monitoring without external monitoring circuitry
EP4484242B1 (en) * 2023-06-27 2025-09-10 Volvo Car Corporation Method for preparing a human driver to take over control of a vehicle from an autonomous driving system, data processing apparatus, computer program, computer-readable storage medium, and vehicle
CN116691700A (zh) * 2023-07-26 2023-09-05 广汽本田汽车有限公司 自动驾驶车辆应急辅助接管方法、系统、装置及存储介质
US12365354B2 (en) * 2023-08-24 2025-07-22 GM Global Technology Operations LLC Driver attention detection
TWI906705B (zh) * 2023-12-05 2025-12-01 優車智能股份有限公司 駕駛員介入引導系統及駕駛員介入引導方法

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130325478A1 (en) * 2012-05-22 2013-12-05 Clarion Co., Ltd. Dialogue apparatus, dialogue system, and dialogue control method
US20140091917A1 (en) * 2012-08-30 2014-04-03 Robert Bosch Gmbh Interactive attentiveness enhancement
DE102015013319A1 (de) * 2015-10-14 2017-04-20 Audi Ag Verfahren zum Betrieb eines Kraftfahrzeugs, Kraftfahrzeug und Computerprogramm
WO2021067380A1 (en) * 2019-10-01 2021-04-08 TeleLingo D/B/A dreyev Methods and systems for using artificial intelligence to evaluate, correct, and monitor user attentiveness
EP3910612A1 (en) * 2019-01-08 2021-11-17 Sony Semiconductor Solutions Corporation Information processing device, mobile device, method, and program
DE102020003018A1 (de) * 2020-05-19 2021-11-25 Daimler Ag Verfahren zur Ermittlung einer Reaktionsfähigkeit

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102008056343B4 (de) * 2008-11-07 2024-07-25 Bayerische Motoren Werke Aktiengesellschaft Warnsystem für ein Kraftfahrzeug
DE102009041187A1 (de) * 2009-08-13 2011-02-17 Volkswagen Ag Verfahren und Vorrichtung zur Adaption von Parametern eines Fahrerassistenzsystems
JP6213282B2 (ja) * 2014-02-12 2017-10-18 株式会社デンソー 運転支援装置
JP2016057814A (ja) * 2014-09-09 2016-04-21 セイコーエプソン株式会社 頭部装着型表示装置、頭部装着型表示装置の制御方法、情報システム、および、コンピュータープログラム
US10241509B1 (en) * 2014-11-13 2019-03-26 State Farm Mutual Automobile Insurance Company Autonomous vehicle control assessment and selection
JP7080598B2 (ja) * 2017-07-21 2022-06-06 ソニーセミコンダクタソリューションズ株式会社 車両制御装置および車両制御方法
US11188074B1 (en) * 2017-11-29 2021-11-30 United Services Automobile Association (Usaa) Systems and methods for remotely controlling operation of a vehicle
DE112019002145T5 (de) * 2018-04-26 2021-03-04 Sony Semiconductor Solutions Corporation Informationsverarbeitungsvorrichtung, mobile vorrichtung, informationsverarbeitungssystem, verfahren und programm
US10576994B1 (en) * 2018-08-22 2020-03-03 Nio Usa, Inc. Autonomous system operator cognitive state detection and alerting
US10647333B1 (en) * 2018-08-31 2020-05-12 Uatc, Llc Vehicle operator awareness system
WO2020054458A1 (ja) * 2018-09-13 2020-03-19 ソニーセミコンダクタソリューションズ株式会社 情報処理装置、移動装置、および方法、並びにプログラム
US11345415B2 (en) * 2019-03-14 2022-05-31 lEE INTERNATIONAL ELECTRONICS & ENGINEERING S.A. Vehicle occupant detection
US11772671B2 (en) * 2019-06-03 2023-10-03 Toyota Research Institute, Inc. Systems and methods for predicting control handback
US11587461B2 (en) * 2019-10-23 2023-02-21 GM Global Technology Operations LLC Context-sensitive adjustment of off-road glance time
KR20220063856A (ko) * 2020-11-10 2022-05-18 현대자동차주식회사 자율 주행 제어 방법 및 장치
US12077165B2 (en) * 2021-08-02 2024-09-03 Allstate Insurance Company Real-time driver analysis and notification system

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130325478A1 (en) * 2012-05-22 2013-12-05 Clarion Co., Ltd. Dialogue apparatus, dialogue system, and dialogue control method
US20140091917A1 (en) * 2012-08-30 2014-04-03 Robert Bosch Gmbh Interactive attentiveness enhancement
DE102015013319A1 (de) * 2015-10-14 2017-04-20 Audi Ag Verfahren zum Betrieb eines Kraftfahrzeugs, Kraftfahrzeug und Computerprogramm
EP3910612A1 (en) * 2019-01-08 2021-11-17 Sony Semiconductor Solutions Corporation Information processing device, mobile device, method, and program
WO2021067380A1 (en) * 2019-10-01 2021-04-08 TeleLingo D/B/A dreyev Methods and systems for using artificial intelligence to evaluate, correct, and monitor user attentiveness
DE102020003018A1 (de) * 2020-05-19 2021-11-25 Daimler Ag Verfahren zur Ermittlung einer Reaktionsfähigkeit

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2026057343A1 (de) * 2024-09-10 2026-03-19 Volkswagen Aktiengesellschaft Verfahren zur prüfung einer aufmerksamkeit eines insassen eines fahrzeuges und vorrichtung sowie fahrzeug umfassend eine entsprechende vorrichtung

Also Published As

Publication number Publication date
KR20240122452A (ko) 2024-08-12
TW202332608A (zh) 2023-08-16
US20230192100A1 (en) 2023-06-22
US12600363B2 (en) 2026-04-14
JP2025500379A (ja) 2025-01-09
CN118434612A (zh) 2024-08-02
EP4452720A1 (en) 2024-10-30

Similar Documents

Publication Publication Date Title
US12600363B2 (en) Awareness checker for enhancing collaborative driving supervision
US10558217B2 (en) Method and apparatus for monitoring of an autonomous vehicle
US11926259B1 (en) Alert modality selection for alerting a driver
US20180326994A1 (en) Autonomous control handover to a vehicle operator
US10591909B2 (en) Handheld mobile device for adaptive vehicular operations
EP4279350B1 (en) Lane keeping based on lane position unawareness
US20220306113A1 (en) Customization of autonomous-driving lane changes of motor vehicles based on drivers' driving behaviours
CN114347996A (zh) 车辆行为监控
US20070126565A1 (en) Process for monitoring blind angle in motor vehicles
US20230115240A1 (en) Advanced driver-assistance systems feature activation control using digital map and on-board sensing to confirm safe vehicle operation
US12125167B2 (en) Image processing device, image processing method, and image processing system
US11663939B1 (en) Augmented reality head-up display for generating a contextual graphic signifying a visually occluded object
US11912313B2 (en) Human machine interaction monitor
JP7732594B2 (ja) 処理システム及び情報提示装置
Molloy et al. Safety Assessment for Autonomous Systems' Perception Capabilities
Cho et al. Operational design domain (ODD) framework for driver-automation systems
US11155260B1 (en) Autonomous vehicle entity vector-based situational awareness scoring matrix
US20170232892A1 (en) Vehicle notification apparatus
CN119339565A (zh) 车辆交通标志预测
Thorat et al. Analysis of recent trends and developments in assisted and automated driving systems: a systematic review
Leilabadi et al. Systematic test case design for autonomous vehicles
Maharajpet et al. The Intersection of Technology and Driving: A Review of Advanced Driver Assistance Systems
Yang Traffic Operations with Connected and Automated Vehicles
Hamilton et al. The role of vision sensors in future intelligent vehicles
CN120431760A (zh) 一种基于5g-v2x智能终端交叉路口车辆碰撞预警方法及系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22844415

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 202447032505

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 202280082875.3

Country of ref document: CN

REG Reference to national code

Ref country code: BR

Ref legal event code: B01A

Ref document number: 112024011354

Country of ref document: BR

WWE Wipo information: entry into national phase

Ref document number: 2024537534

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2022844415

Country of ref document: EP

Effective date: 20240722

ENP Entry into the national phase

Ref document number: 112024011354

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20240605