US20160171888A1 - Reducing driver distraction in spoken dialogue - Google Patents

Reducing driver distraction in spoken dialogue Download PDF

Info

Publication number
US20160171888A1
US20160171888A1 US15/005,483 US201615005483A US2016171888A1 US 20160171888 A1 US20160171888 A1 US 20160171888A1 US 201615005483 A US201615005483 A US 201615005483A US 2016171888 A1 US2016171888 A1 US 2016171888A1
Authority
US
United States
Prior art keywords
dialogue
workload
reducing
user
anticipated
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US15/005,483
Inventor
Eli Tzirkel-Hancock
Omer Tsimhoni
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
GM Global Technology Operations LLC
Original Assignee
GM Global Technology Operations LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by GM Global Technology Operations LLC filed Critical GM Global Technology Operations LLC
Priority to US15/005,483 priority Critical patent/US20160171888A1/en
Publication of US20160171888A1 publication Critical patent/US20160171888A1/en
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/09Arrangements for giving variable traffic instructions
    • G08G1/0962Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/09626Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages where the origin of the information is within the own vehicle, e.g. a local storage device, digital map
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/16Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/18Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state for vehicle drivers or machine operators
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/01Detecting movement of traffic to be counted or controlled
    • G08G1/0104Measuring and analyzing of parameters relative to traffic conditions
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/09Arrangements for giving variable traffic instructions
    • G08G1/0962Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967Systems involving transmission of highway information, e.g. weather, speed limits
    • GPHYSICS
    • G08SIGNALLING
    • G08GTRAFFIC CONTROL SYSTEMS
    • G08G1/00Traffic control systems for road vehicles
    • G08G1/16Anti-collision systems
    • G08G1/166Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60WCONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2540/00Input parameters relating to occupants
    • B60W2540/22Psychological state; Stress level or workload
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B60VEHICLES IN GENERAL
    • B60YINDEXING SCHEME RELATING TO ASPECTS CROSS-CUTTING VEHICLE TECHNOLOGY
    • B60Y2302/00Responses or measures related to driver conditions
    • B60Y2302/09Reducing the workload of driver

Definitions

  • a driver of a motor vehicle may become distracted under certain conditions, which are typically characterized by increased mental workload.
  • a related situation occurs in a vehicle capable of autonomous operation, where the nominal driver may lose situation awareness, which may be unsafe under certain anticipated workload conditions requiring driver response.
  • Automated spoken and multimodal dialogue systems are becoming more common in motor vehicles and can be a factor in reducing potential distraction and/or reduced situation awareness of the driver.
  • Embodiments of the invention provide systems and methods for reducing user distraction associated with automated dialogue by monitoring user workload, by providing extended system dialogue acts to compensate for distraction, and by modifying the reward that assesses dialogue performance in order to optimize automated dialogue responsiveness that decreases user distraction associated with automated dialogue.
  • situations non-aware and “situation non-awareness” connote at least a partial unawareness of the fill situation in which the autonomous motor vehicle is operating.
  • the driver may have some awareness of the situation, but is potentially lacking the full awareness that a situation-aware driver would be expected to have.
  • a situation-aware driver has sufficient cognitive capacity directed to the driving task of taking control of the vehicle when necessary.
  • a driver with impaired situation-awareness lacks information to take responsible control of the vehicle should the need arise.
  • Embodiments of the invention may be applied advantageously in a broad range of situations where a human user participates in automated dialogue while operating, or supervising the operation of, apparatus or systems.
  • the human user may be faced by a high workload burden and may become distracted from the operation of the apparatus or systems.
  • the user may have a heavy anticipated workload and become non-aware of the present situation and factors thereof that could affect the operation which the user is supervising.
  • the present disclosure details certain embodiments of the invention which are applicable to the non-limiting example of a user who is a driver of a motor vehicle.
  • the case of users who are supervising or overseeing the operation of apparatus or systems is likewise illustrated herein by the non-limiting example of a user who is nominally a driver of an autonomous motor vehicle. It is understood, however, that embodiments of the present invention are broadly applicable to other and more general cases as well.
  • a user who operates an industrial process system and/or supervises the operation of an industrial process system can also benefit from embodiments of the present invention.
  • autonomous vehicle denotes a vehicle which has one or more automated systems for performing one or more common driving tasks without direct driver involvement.
  • systems for autonomous vehicles include, but are not limited to: autopilot systems for aircraft and ships; and cruise control systems and automated lane-centering systems for automobiles and trucks. Although such systems can alleviate considerable workload from the pilot or driver, they may not necessarily be able to handle all situations which may arise.
  • a trained human operator designated as the nominal “pilot” or “driver” supervises or oversees the operation of the autonomous vehicle, and is intended to be able to take over partial or full control in the event that a situation arises which the automated system cannot fully handle.
  • the driver may have to intervene by applying the brake, should traffic conditions suddenly change.
  • the driver of an autonomous vehicle may be relieved of much of the driving workload, but must nevertheless remain alert to the present situation at all times, and must remain ready to intervene as necessary.
  • the driver of an autonomous vehicle has an anticipated workload.
  • the anticipated workload may be similar in some respects to the actual workload handled by an actual driver, and in some cases may even exceed the workload of actual driving, such as in a case where sudden intervention is required.
  • the driver is situation non-aware and the anticipated workload exceeds a certain level, then the potential for serious consequences exists.
  • an increase in anticipated workload for the driver of an autonomous vehicle can lead to a potentially-dangerous condition if the driver is situation non-aware, paralleling the case where an increase in workload for the driver of a regular vehicle can lead to a condition of driver distraction.
  • FIG. 1 conceptually illustrates the regimes of interest according to certain embodiments of the present invention.
  • a regime set 100 applies to the driver of a regular vehicle, and a regime set 130 applies to the driver of an autonomous vehicle.
  • Basic regimes 151 pertain to measurement.
  • a regime 101 the workload associated with driving is measured for the driver of a regular vehicle, and in a regime 131 the anticipated workload associated with driving an autonomous vehicle is measured.
  • Secondary regimes 153 pertain to control of automated dialogues.
  • a regime 103 the workload associated with automated dialogue in a regular vehicle can be controlled, and in a regime 133 the workload associated with automated dialogue in an autonomous vehicle can be controlled.
  • Optimization regimes 155 provide latitude for adjusting the control of regimes 153 .
  • a regime 105 provides latitude to keep the combined driving workload 101 and automated dialogue workload 103 below a distraction threshold 110 .
  • distraction threshold 110 is a conceptual threshold rather than an operational threshold.
  • the system seeks to reduce distraction based on past experience actualized in a learning phase (as discussed below), in place of measuring the distraction directly.
  • a regime 135 provides latitude to keep the combined driving anticipated workload 131 and automated dialogue workload 133 below a situation non-awareness threshold 140 .
  • a region 137 below threshold 140 the driver is situation-aware, whereas in a region 139 above threshold 140 the driver is situation non-aware.
  • an offline learning process is used to develop a new dialogue policy for an automated dialogue system using a training database of example dialogues.
  • the new dialogue policy is developed through a learning process which confers penalties for creating dialogues which empirically create distraction/situation non-awareness. (In these embodiments, the term “penalty” denotes a negative reward.)
  • the new dialogue policy reduces workload/anticipated workload if the dialogue is similar to dialogue examples seen in the training process exceeding a threshold.
  • embodiments of the invention can optimize the automated dialogue to reduce the levels of distraction/situation non-awareness.
  • a method for reducing user distraction associated with interaction with an automated dialogue system comprising;
  • a method for reducing user situation non-awareness associated with an automated dialogue system comprising:
  • a dialogue system for reducing user distraction associated with interaction with automated dialogue comprising:
  • a dialogue system for reducing user situation non-awareness associated with interaction with automated dialogue comprising:
  • FIG. 1 illustrates operational regimes for reducing driver distraction and situation non-awareness, according to certain embodiments of the invention.
  • FIG. 2A conceptually illustrates a system according to certain embodiments of the invention, for reducing user distraction in real time.
  • FIG. 2B conceptually illustrates a system according to certain embodiments of the invention, for reducing user situation non-awareness in real time.
  • FIG. 3A conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce user distraction.
  • FIG. 3B conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce user situation non-awareness.
  • FIG. 4A is a flowchart of a method according to certain embodiments of the invention, for reducing user distraction in real time.
  • FIG. 4B is a flowchart of a method according to certain embodiments of the invention, for reducing user situation non-awareness in real time.
  • FIG. 5 is a flowchart of a method according to specific embodiments of the invention, for reducing driver distraction in real time.
  • FIG. 2A conceptually illustrates a system according to certain embodiments of the invention, for reducing distraction in real time for a driver of a regular motor vehicle.
  • An input stage 201 receives speech audio and/or multimodal input to a speech and multimodal understanding unit 203 , which outputs processed user dialogue acts 205 into a dialogue control unit 207 .
  • dialogue control unit 207 outputs system dialogue acts 209 into a speech and multimodal generation unit 211 for generating speech audio and/or multimodal output 213 .
  • one or more workload parameters 221 are input into a workload estimation unit 223 which outputs a workload measure 225 into dialogue control unit 207 .
  • a workload-responsive dialogue policy 235 is available to dialogue control unit 207 for responding to high workload situations with appropriate workload-reducing system dialogue acts 209 .
  • Workload estimates may be prepared according to factors including, but not limited to: vehicle parameters, such as steering, brakes, and safety systems; road conditions, such as road bends and traffic; weather conditions, such as rain and fog; time of day; and driver attributes, such as eye, head, and hand movements.
  • a workload estimation may be obtained from factors including, but not limited to:
  • workload can be estimated according to a user model.
  • workload may be predicted according to factors including, but not limited to:
  • a driver distraction input 231 is used as penalties 233 in a learning process, as described below.
  • the output of the learning process is used to create dialogue policy 235 .
  • Dialogue policy 235 thus bridges between the learning process—shown in FIG. 2A conceptually as an off-line creation phase 236 of dialogue policy 235 —and the interactive dialogue system—shown in FIG. 2A conceptually as a dialogue-time application 237 of dialogue policy 235 .
  • FIG. 2B conceptually illustrates a system according to certain embodiments of the invention, for reducing situation non-awareness in dialogue time for a driver of an autonomous motor vehicle.
  • an input stage 201 receives speech audio and/or input to a speech and multimodal understanding unit 203 , which outputs processed user dialogue acts 205 into a dialogue control unit 207 .
  • dialogue control unit 207 outputs system dialogue acts 209 into a speech and multimodal generation unit 211 for generating speech audio and/or multimodal output 213 .
  • one or more anticipated workload parameters 251 are input into an anticipated workload estimation unit 253 Which outputs an anticipated workload measure 255 into dialogue control nit 207 .
  • An anticipated workload-responsive dialogue policy 285 is available to dialogue control unit 207 for responding to high anticipated workload ions with appropriate workload-reducing system dialogue acts 209 .
  • Anticipated workload is the cognitive workload the driver would incur should it become necessary to take over control of the vehicle. If the driver is participating in an automated dialogue, this may impact situation awareness, particularly in an environment of high anticipated workload. According to certain embodiments of the invention, an anticipated workload estimate may be obtained from factors including, but not limited to:
  • a prediction of anticipated workload may be useful for adjusting the dialogue policy to increase future situation awareness, and anticipated workload may be predicted according to factors including, but not limited to:
  • a driver situation non-awareness input 281 is used as penalties 283 in a learning process, as described below.
  • the output of the learning process is used to create dialogue policy 285 .
  • Dialogue policy 285 thus bridges between the learning process—shown in FIG. 2B conceptually as an off-line creation phase 286 of dialogue policy 285 —and the interactive dialogue system—shown in FIG. 2B conceptually as a dialogue-time application 287 of dialogue policy 285 .
  • FIG. 3A conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce driver distraction.
  • User dialogue acts 301 are input into a user model 303 , the output of which are a set of beliefs 305 that are used as input to workload-responsive dialogue policy 235 , which has been formulated through an off-line policy learning process 307 .
  • the new policy developed through this process is therefore sensitive to the beliefs.
  • policy 235 is used to govern dialogue control unit 207 to output system dialogue acts 209 in response to the beliefs and workload estimate 225 .
  • Policy learning process 307 receives input from an interaction log 309 with reward/penalty 331 input to guide the learning process in creating a policy that meets the desired goal of reducing driver distraction.
  • Input to reward/penalty 331 includes penalties according to a driver distraction assessment 321 .
  • Driver distraction assessment 321 can be obtained from a driver's subjective impression of being distracted. Because the learning process takes place off-line, driver assessments can be obtained after completion of the driving session, in a vehicle or a vehicle simulator, in which the dialogues recorded in interaction log 309 were obtained.
  • an off-line process for policy learning takes place in a laboratory, for which a reward is also assigned off-line.
  • policy learning takes place in the vehicle itself, or an off-vehicle server, such as at a scheduled time, or after sufficient dialogue is recorded in the dialogue log, in cases where the reward is measured automatically.
  • driver interaction and/or performance-metrics can be used, such as braking response time given the measured headway from a lead vehicle and observation of driver head and eye movement.
  • Driver feedback is typically limited to off-line availability, but automated assessments may be done in real-time during driving sessions.
  • driver distraction assessment 321 which involves penalties for distraction
  • other dialogue metrics 323 can be used, some of which may involve rewards.
  • FIG. 3B conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce driver situation non-awareness in an autonomous vehicle.
  • User dialogue acts 301 are input into a user model 303 , the output of which are a set of beliefs 305 that are used as input to anticipated workload-responsive dialogue policy 285 , which has been formulated through off-line policy learning process 307 .
  • policy 285 is used to govern dialogue control unit 207 to output system dialogue acts 209 in response to anticipated workload estimate 255 .
  • reward/penalty 343 receives input from a driver situation non-awareness assessment 331 .
  • dialogue-time measurement of driver situation awareness enables policy learning, and driver situation awareness may be obtained in ways including, but not limited to:
  • an autonomous mode where one or more autonomous systems are in control of respective vehicle operating functions
  • a driver control mode where the driver assumes control of the vehicle.
  • Autonomous control can be partial control of vehicle operating functions, a non-limiting example of which is automatic cruise control in a vehicle.
  • an automated dialogue system in an autonomous vehicle should be able to handle switching between these two modes. According to an embodiment of the invention, this is done by switching between two appropriate dialogue policies; in this embodiment, the learning phase policy parameters are developed separately, and at dialogue time the appropriate policy is selected, consistent with the driving mode.
  • FIG. 4A is a flowchart of a method according to certain embodiments of the invention, for reducing driver distraction during a dialogue.
  • system dialogue turn 407 includes a workload-reducing dialogue act 409 and/or a regular system dialogue turn with a workload-reducing modification 411 .
  • FIG. 4B is a flowchart of a method according to other embodiments of the invention, for reducing driver situation non-awareness in an autonomous vehicle during a dialogue.
  • system dialogue turn 427 includes an anticipated workload-reducing dialogue act 429 and/or a regular system dialogue turn with an anticipated workload-reducing modification 431 .
  • workload-reducing and anticipated workload-reducing dialogue turns may have features including, but not limited to: pauses and suggestions for pauses (see below); termination of dialogue and suggestions for terminating dialogue.
  • Workload-reducing and anticipated workload-reducing modifications may have features including, but not limited to: breaking up dialogue turns into simpler sentences; presenting alternatives sequentially, rather than together; and phrasing questions for answering by “yes-no” responses; preferring speech modality to tactile and visual modality (see below).
  • Certain embodiments of the invention provide a spectrum of pause handling, and termination ranging as follows by degree of user involvement, which is selectable by the system:
  • the difference between pausing a dialogue and terminating the dialogue is that a paused dialogue may be resumed at a later time from the point at which the dialogue was paused, whereas a terminated dialogue is stopped and may not be resumed (but may be restarted).
  • the terms “suspend”, “suspending”, “suspension”, etc., herein denote that a dialogue has been interrupted by either being paused or being terminated. Whether or not a suspended dialogue may be later resumed depends on whether the dialogue was paused or terminated, but in either case the dialogue is interrupted at the time the suspension goes into effect.
  • FIG. 5 is a flowchart of a method according to specific embodiments of the invention.
  • a hazard warning 503 is received.
  • Hazard warning 503 can be signaled by a variety of presently-known methods, including, but not limited to: detection of a hazard ahead by an on-board radar system; and hazard notification by a navigational system, based on road condition information supplied to the navigational system.
  • the hazard warning includes a warning of a braking condition of another vehicle ahead of the vehicle.
  • the dialogue is immediately suspended via an immediate dialogue suspension action 507 .
  • the method illustrated in FIG. 5 may also be considered as a special case of the method illustrated in FIG. 4A , wherein user workload parameter 403 includes hazard warning 503 that a hazard has been detected, and system dialogue turn 407 includes immediate dialogue suspension action 507 .
  • immediate dialogue suspension action 507 is included in workload-reducing dialogue act 409 or in workload-reducing dialogue modification 411 .
  • workload estimation FIG. 2A
  • workload estimation 223 includes immediate dialogue suspension action 507 .
  • the method illustrated in FIG. 5 may additionally be considered as special case of the method illustrated in FIG. 4B , wherein user anticipated workload parameter 423 includes hazard warning 503 that a hazard has been detected, and system dialogue rum 427 includes immediate dialogue suspension action 507 .
  • immediate dialogue suspension action 507 is included in anticipated workload-reducing dialogue act 429 or in anticipated workload-reducing dialogue modification 431 .
  • anticipated workload estimation 253 FIG. 2B
  • anticipated workload estimation 253 includes immediate dialogue suspension action 507 .
  • a dialogue may be simplified by one or more of the following:
  • a prediction of upcoming increased workload can trigger the speeding up of a dialogue. For example, if the driver is approaching an area of congested traffic or other abnormal driving conditions, the automated dialogue system can receive a prediction that workload will soon increase, and may decide to accelerate an ongoing dialogue so that the dialogue will complete before the workload increases.
  • a dialogue may be speeded up by one or more of the following:
  • a computer product includes a set of executable commands for performing the one or both of the above methods on a computer, wherein the executable commands are contained within a tangible computer-readable non-transitory data storage medium including, but not limited to: computer media such as magnetic media and optical media; computer memory; semiconductor memory storage; flash memory storage; data storage devices and hardware components; and the tangible non-transitory storage devices of a remote computer or communications network; such that when the executable commands of the computer product are executed, the computer product causes the computer to perform one or both of the above methods.
  • a tangible computer-readable non-transitory data storage medium including, but not limited to: computer media such as magnetic media and optical media; computer memory; semiconductor memory storage; flash memory storage; data storage devices and hardware components; and the tangible non-transitory storage devices of a remote computer or communications network; such that when the executable commands of the computer product are executed, the computer product causes the computer to perform one or both of the above methods.
  • a “computer” is any data processing apparatus for executing a set of executable commands to perform a method of the present invention, including, but not limited to: personal computer; workstation; server; gateway; router; multiplexer, demultiplexer; modulator, demodulator; switch; network; processor; controller; digital appliance, tablet computer; mobile device, mobile telephone; any other device capable of executing the commands.

Abstract

Methods and systems for reducing driver distraction and situation non-awareness related to a dialogue of an automated dialogue system in a vehicle. For a dialogue policy learning session, driver distraction is introduced as an input into a penalty assigner that assesses dialogue quality, and dialogue acts are extended to include dialogues and dialogue act presentation styles which reduce driver workload related to dialogues. The automated dialogue system policy is developed during the learning process by optimizing the penalties, an that automated dialogue workload is reduced in response to increased workload or anticipated workload on the driver. Methods and systems are presented for responding to both actual workload in regular vehicles as well as anticipated workload in autonomous vehicles.

Description

    CROSS REFERENCE TO RELATED APPLICATIONS
  • This is a Continuation Application of U.S. patent application Ser. No. 13/893,433 entitled “Reducing driver distraction in spoken dialogue”, filed May 14, 2013, which claims benefit of U.S. Provisional Patent Application Ser. No. 61/652,600, filed May 29, 2012, entitled “Reducing driver distraction in spoken dialogue”, the disclosure of which is hereby incorporated by reference and the priority of which is hereby claimed pursuant to 37 CFR 1.78(a) (4) and (5)(i).
  • BACKGROUND
  • A driver of a motor vehicle may become distracted under certain conditions, which are typically characterized by increased mental workload. A related situation occurs in a vehicle capable of autonomous operation, where the nominal driver may lose situation awareness, which may be unsafe under certain anticipated workload conditions requiring driver response. Automated spoken and multimodal dialogue systems are becoming more common in motor vehicles and can be a factor in reducing potential distraction and/or reduced situation awareness of the driver.
  • It is thus desirable to have methods for reducing driver distraction and/or reducing driver situation unawareness while engaging in spoken dialogue with an automated dialogue system. This goal is met by the present invention.
  • SUMMARY
  • Embodiments of the invention provide systems and methods for reducing user distraction associated with automated dialogue by monitoring user workload, by providing extended system dialogue acts to compensate for distraction, and by modifying the reward that assesses dialogue performance in order to optimize automated dialogue responsiveness that decreases user distraction associated with automated dialogue.
  • In a similar fashion, additional embodiments of the invention provide systems and methods for reducing user situation non-awareness associated with automated dialogue in an autonomous vehicle or similar system. According to embodiments of the invention, the terms “situation non-aware” and “situation non-awareness” connote at least a partial unawareness of the fill situation in which the autonomous motor vehicle is operating. The driver may have some awareness of the situation, but is potentially lacking the full awareness that a situation-aware driver would be expected to have. A situation-aware driver has sufficient cognitive capacity directed to the driving task of taking control of the vehicle when necessary. A driver with impaired situation-awareness lacks information to take responsible control of the vehicle should the need arise.
  • Embodiments of the invention may be applied advantageously in a broad range of situations where a human user participates in automated dialogue while operating, or supervising the operation of, apparatus or systems. In certain circumstances the human user may be faced by a high workload burden and may become distracted from the operation of the apparatus or systems. In cases where the user is supervising or overseeing the operation of apparatus or systems, the user may have a heavy anticipated workload and become non-aware of the present situation and factors thereof that could affect the operation which the user is supervising.
  • For clarity of illustration, the present disclosure details certain embodiments of the invention which are applicable to the non-limiting example of a user who is a driver of a motor vehicle. The case of users who are supervising or overseeing the operation of apparatus or systems is likewise illustrated herein by the non-limiting example of a user who is nominally a driver of an autonomous motor vehicle. It is understood, however, that embodiments of the present invention are broadly applicable to other and more general cases as well. In another non-limiting example, a user who operates an industrial process system and/or supervises the operation of an industrial process system can also benefit from embodiments of the present invention.
  • Distraction and Situation Non-Awareness
  • In order to safely and effectively handle a motor vehicle, the driver must continually receive sensory input from many different sources, and respond appropriately and in a timely fashion to those inputs. There is a component of the accumulated workload which is associated with the driver's participation in automated dialogues, and this factor is addressed in the present disclosure.
  • If the cognitive workload in managing the inputs and responses exceeds a certain level, the driver may become distracted from the task of driving, with potentially serious consequences. A person who is nominally the “driver” of an autonomous vehicle, however, does not have the same ongoing responsibilities. The term “autonomous vehicle” herein denotes a vehicle which has one or more automated systems for performing one or more common driving tasks without direct driver involvement. Examples of systems for autonomous vehicles include, but are not limited to: autopilot systems for aircraft and ships; and cruise control systems and automated lane-centering systems for automobiles and trucks. Although such systems can alleviate considerable workload from the pilot or driver, they may not necessarily be able to handle all situations which may arise. Thus, a trained human operator, designated as the nominal “pilot” or “driver” supervises or oversees the operation of the autonomous vehicle, and is intended to be able to take over partial or full control in the event that a situation arises which the automated system cannot fully handle. In the non-limiting example of a cruise-control system, the driver may have to intervene by applying the brake, should traffic conditions suddenly change.
  • The driver of an autonomous vehicle may be relieved of much of the driving workload, but must nevertheless remain alert to the present situation at all times, and must remain ready to intervene as necessary. Thus, in place of the workload itself, the driver of an autonomous vehicle has an anticipated workload. The anticipated workload may be similar in some respects to the actual workload handled by an actual driver, and in some cases may even exceed the workload of actual driving, such as in a case where sudden intervention is required. According to certain embodiments of the invention, if the driver is situation non-aware and the anticipated workload exceeds a certain level, then the potential for serious consequences exists.
  • Therefore, according to certain embodiments of the invention, an increase in anticipated workload for the driver of an autonomous vehicle can lead to a potentially-dangerous condition if the driver is situation non-aware, paralleling the case where an increase in workload for the driver of a regular vehicle can lead to a condition of driver distraction.
  • FIG. 1 conceptually illustrates the regimes of interest according to certain embodiments of the present invention. A regime set 100 applies to the driver of a regular vehicle, and a regime set 130 applies to the driver of an autonomous vehicle. Basic regimes 151 pertain to measurement. In a regime 101 the workload associated with driving is measured for the driver of a regular vehicle, and in a regime 131 the anticipated workload associated with driving an autonomous vehicle is measured. Secondary regimes 153 pertain to control of automated dialogues. In a regime 103 the workload associated with automated dialogue in a regular vehicle can be controlled, and in a regime 133 the workload associated with automated dialogue in an autonomous vehicle can be controlled. Optimization regimes 155 provide latitude for adjusting the control of regimes 153. A regime 105 provides latitude to keep the combined driving workload 101 and automated dialogue workload 103 below a distraction threshold 110. According to an embodiment of the invention, distraction threshold 110 is a conceptual threshold rather than an operational threshold. In this embodiment, the system seeks to reduce distraction based on past experience actualized in a learning phase (as discussed below), in place of measuring the distraction directly.
  • In a region 107 below threshold 110 the driver is not distracted, whereas in a region 109 above threshold 110 the driver is distracted. Likewise, a regime 135 provides latitude to keep the combined driving anticipated workload 131 and automated dialogue workload 133 below a situation non-awareness threshold 140. In a region 137 below threshold 140 the driver is situation-aware, whereas in a region 139 above threshold 140 the driver is situation non-aware.
  • Reducing Distraction and Situation Non-Awareness
  • According to certain embodiments of the invention, an offline learning process is used to develop a new dialogue policy for an automated dialogue system using a training database of example dialogues. The new dialogue policy is developed through a learning process which confers penalties for creating dialogues which empirically create distraction/situation non-awareness. (In these embodiments, the term “penalty” denotes a negative reward.) Then, in dialogue-time situations, the new dialogue policy reduces workload/anticipated workload if the dialogue is similar to dialogue examples seen in the training process exceeding a threshold.
  • In this fashion, embodiments of the invention can optimize the automated dialogue to reduce the levels of distraction/situation non-awareness.
  • Therefore, according to an embodiment of the invention there is provided a method for reducing user distraction associated with interaction with an automated dialogue system, the method comprising;
      • receiving, by a processor, a user workload parameter;
      • responsively to the user workload parameter, controlling the automated dialogue system to perform a system dialogue turn that reduces the user workload associated with interacting with the automated dialogue system, wherein the system dialogue turn includes a dialogue act selected from a group consisting of:
      • a workload-reducing dialogue act; and
      • a regular dialogue act modified by a workload-reducing dialogue modification.
  • Also, according to another embodiment of the invention, there is provided a method for reducing user situation non-awareness associated with an automated dialogue system, the method comprising:
      • receiving, by a processor, a user anticipated workload parameter;
      • responsively to the user anticipated workload parameter, controlling the automated dialogue system to perform a system dialogue turn that reduces the user workload associated with interacting with the automated dialogue system, wherein the system dialogue turn includes a dialogue act selected from a group consisting of:
      • a workload-reducing dialogue act; and
      • a regular dialogue act modified by a workload-reducing modification.
  • In addition, according to a further embodiment of the invention, there is provided a dialogue system for reducing user distraction associated with interaction with automated dialogue, the system comprising:
      • a dialogue control unit;
      • a storage device containing a dialogue policy;
      • a workload estimation unit operative to:
      • receive a workload parameter indicative of a user workload; and
      • compute a workload estimate; and
      • input the workload estimate into the dialogue control unit.
  • Moreover, according to still another embodiment of the invention, there is provided a dialogue system for reducing user situation non-awareness associated with interaction with automated dialogue, the system comprising:
      • a dialogue control unit;
      • a storage device containing a dialogue policy;
      • an anticipated workload estimation unit operative to;
      • receive an anticipated workload parameter indicative of an anticipated user workload; and
      • compute an anticipated workload estimate; and
      • input the anticipated workload estimate into the dialogue control unit.
    BRIEF DESCRIPTION OF THE DRAWINGS
  • The subject matter disclosed may best be understood by reference to the following detailed description when read with the accompanying drawings in which:
  • FIG. 1 illustrates operational regimes for reducing driver distraction and situation non-awareness, according to certain embodiments of the invention.
  • FIG. 2A conceptually illustrates a system according to certain embodiments of the invention, for reducing user distraction in real time.
  • FIG. 2B conceptually illustrates a system according to certain embodiments of the invention, for reducing user situation non-awareness in real time.
  • FIG. 3A conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce user distraction.
  • FIG. 3B conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce user situation non-awareness.
  • FIG. 4A is a flowchart of a method according to certain embodiments of the invention, for reducing user distraction in real time.
  • FIG. 4B is a flowchart of a method according to certain embodiments of the invention, for reducing user situation non-awareness in real time.
  • FIG. 5 is a flowchart of a method according to specific embodiments of the invention, for reducing driver distraction in real time.
  • For simplicity and clarity of illustration, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.
  • DETAILED DESCRIPTION
  • FIG. 2A conceptually illustrates a system according to certain embodiments of the invention, for reducing distraction in real time for a driver of a regular motor vehicle. An input stage 201 receives speech audio and/or multimodal input to a speech and multimodal understanding unit 203, which outputs processed user dialogue acts 205 into a dialogue control unit 207. In response, dialogue control unit 207 outputs system dialogue acts 209 into a speech and multimodal generation unit 211 for generating speech audio and/or multimodal output 213. In these embodiments, one or more workload parameters 221 are input into a workload estimation unit 223 which outputs a workload measure 225 into dialogue control unit 207. A workload-responsive dialogue policy 235 is available to dialogue control unit 207 for responding to high workload situations with appropriate workload-reducing system dialogue acts 209. Workload estimates may be prepared according to factors including, but not limited to: vehicle parameters, such as steering, brakes, and safety systems; road conditions, such as road bends and traffic; weather conditions, such as rain and fog; time of day; and driver attributes, such as eye, head, and hand movements.
  • According to certain embodiments of the invention, a workload estimation may be obtained from factors including, but not limited to:
      • vehicle parameters, such as:
        • steering,
        • brakes,
        • safety systems
      • road conditions, such as
        • road-bends or
        • traffic
      • weather conditions, such as
        • heavy-rain or
        • fog
      • time of day
      • driver attributes, such as movement of
        • eyes
        • head, and
        • hands
  • In another embodiment of the invention, workload can be estimated according to a user model. These different embodiments regarding workload estimation can be combined together or used separately.
  • Further embodiments of the invention provide an estimate of future workload, which may be useful for adjusting dialogue policy to reduce future driver distraction. In these embodiments workload may be predicted according to factors including, but not limited to:
      • road conditions
      • weather conditions, and
      • time of day
  • According to these embodiments, to prepare workload-responsive dialogue policy 235, a driver distraction input 231 is used as penalties 233 in a learning process, as described below. The output of the learning process is used to create dialogue policy 235. Dialogue policy 235 thus bridges between the learning process—shown in FIG. 2A conceptually as an off-line creation phase 236 of dialogue policy 235—and the interactive dialogue system—shown in FIG. 2A conceptually as a dialogue-time application 237 of dialogue policy 235.
  • FIG. 2B conceptually illustrates a system according to certain embodiments of the invention, for reducing situation non-awareness in dialogue time for a driver of an autonomous motor vehicle. As before, an input stage 201 receives speech audio and/or input to a speech and multimodal understanding unit 203, which outputs processed user dialogue acts 205 into a dialogue control unit 207. In response, dialogue control unit 207 outputs system dialogue acts 209 into a speech and multimodal generation unit 211 for generating speech audio and/or multimodal output 213. In these embodiments, one or more anticipated workload parameters 251 are input into an anticipated workload estimation unit 253 Which outputs an anticipated workload measure 255 into dialogue control nit 207. An anticipated workload-responsive dialogue policy 285 is available to dialogue control unit 207 for responding to high anticipated workload ions with appropriate workload-reducing system dialogue acts 209.
  • Anticipated workload is the cognitive workload the driver would incur should it become necessary to take over control of the vehicle. If the driver is participating in an automated dialogue, this may impact situation awareness, particularly in an environment of high anticipated workload. According to certain embodiments of the invention, an anticipated workload estimate may be obtained from factors including, but not limited to:
      • vehicle parameters, such as:
        • steering,
        • brakes,
        • safety systems
      • road conditions, such as
        • road-bends or
        • traffic
      • weather conditions, such as
        • heavy-rain or
        • fog
      • time of day
  • According to these embodiments, a prediction of anticipated workload may be useful for adjusting the dialogue policy to increase future situation awareness, and anticipated workload may be predicted according to factors including, but not limited to:
      • road conditions
      • weather conditions, and
      • time of day
  • According to these embodiments, to prepare anticipated workload-responsive dialogue policy 285, a driver situation non-awareness input 281 is used as penalties 283 in a learning process, as described below. The output of the learning process is used to create dialogue policy 285. Dialogue policy 285 thus bridges between the learning process—shown in FIG. 2B conceptually as an off-line creation phase 286 of dialogue policy 285—and the interactive dialogue system—shown in FIG. 2B conceptually as a dialogue-time application 287 of dialogue policy 285.
  • FIG. 3A conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce driver distraction. User dialogue acts 301 are input into a user model 303, the output of which are a set of beliefs 305 that are used as input to workload-responsive dialogue policy 235, which has been formulated through an off-line policy learning process 307. The new policy developed through this process is therefore sensitive to the beliefs. As also illustrated in FIG. 2A, policy 235 is used to govern dialogue control unit 207 to output system dialogue acts 209 in response to the beliefs and workload estimate 225.
  • Policy learning process 307 receives input from an interaction log 309 with reward/penalty 331 input to guide the learning process in creating a policy that meets the desired goal of reducing driver distraction. Input to reward/penalty 331 includes penalties according to a driver distraction assessment 321. Driver distraction assessment 321 can be obtained from a driver's subjective impression of being distracted. Because the learning process takes place off-line, driver assessments can be obtained after completion of the driving session, in a vehicle or a vehicle simulator, in which the dialogues recorded in interaction log 309 were obtained. According to various embodiments of the invention, an off-line process for policy learning takes place in a laboratory, for which a reward is also assigned off-line. In other embodiments, policy learning takes place in the vehicle itself, or an off-vehicle server, such as at a scheduled time, or after sufficient dialogue is recorded in the dialogue log, in cases where the reward is measured automatically.
  • Besides direct feedback from the driver, visual inspection of driver interaction and/or performance-metrics can be used, such as braking response time given the measured headway from a lead vehicle and observation of driver head and eye movement. Driver feedback is typically limited to off-line availability, but automated assessments may be done in real-time during driving sessions. In addition to driver distraction assessment 321, which involves penalties for distraction, other dialogue metrics 323 can be used, some of which may involve rewards.
  • In a similar manner, FIG. 3B conceptually illustrates a system according to certain embodiments of the invention, for offline policy learning to reduce driver situation non-awareness in an autonomous vehicle. User dialogue acts 301 are input into a user model 303, the output of which are a set of beliefs 305 that are used as input to anticipated workload-responsive dialogue policy 285, which has been formulated through off-line policy learning process 307. As also illustrated in FIG. 2B, policy 285 is used to govern dialogue control unit 207 to output system dialogue acts 209 in response to anticipated workload estimate 255. Also in a similar manner, reward/penalty 343 receives input from a driver situation non-awareness assessment 331.
  • According to certain embodiments of the invention, dialogue-time measurement of driver situation awareness enables policy learning, and driver situation awareness may be obtained in ways including, but not limited to:
      • as feedback from the driver;
      • by visual inspection of driver interactions; and
      • by measuring driver eye and head movement, e.g.
        • driver's eyes focused on the road implies a high driver situation awareness; but
        • driver's eyes focused elsewhere than on the road implies a low driver situation awareness.
    Driving Modes in an Autonomous Vehicle
  • There are two modes for operating an autonomous vehicle: an autonomous mode, where one or more autonomous systems are in control of respective vehicle operating functions; and a driver control mode, where the driver assumes control of the vehicle. Autonomous control can be partial control of vehicle operating functions, a non-limiting example of which is automatic cruise control in a vehicle. According to certain embodiments of the invention, an automated dialogue system in an autonomous vehicle should be able to handle switching between these two modes. According to an embodiment of the invention, this is done by switching between two appropriate dialogue policies; in this embodiment, the learning phase policy parameters are developed separately, and at dialogue time the appropriate policy is selected, consistent with the driving mode. In another embodiment of the invention, there is a combined dialogue policy that supports both modes, and at dialogue time the mode is input to the policy along with both workload and anticipated workload estimates, and with penalty for driver distraction and penalty for driver situation non-awareness.
  • Method
  • FIG. 4A is a flowchart of a method according to certain embodiments of the invention, for reducing driver distraction during a dialogue.
  • In a step 401 a driver workload parameter 403 is received. Then, in a step 405 a system dialogue turn 407 is performed according to workload-responsive dialogue policy 235. According to these embodiments, system dialogue turn 407 includes a workload-reducing dialogue act 409 and/or a regular system dialogue turn with a workload-reducing modification 411.
  • In a similar manner, FIG. 4B is a flowchart of a method according to other embodiments of the invention, for reducing driver situation non-awareness in an autonomous vehicle during a dialogue.
  • In a step 421 a driver anticipated workload parameter 423 is received. Then, in a step 425 a system dialogue turn 427 is performed according to anticipated workload-responsive dialogue policy 285. According to these embodiments, system dialogue turn 427 includes an anticipated workload-reducing dialogue act 429 and/or a regular system dialogue turn with an anticipated workload-reducing modification 431.
  • According to the above embodiments, workload-reducing and anticipated workload-reducing dialogue turns may have features including, but not limited to: pauses and suggestions for pauses (see below); termination of dialogue and suggestions for terminating dialogue. Workload-reducing and anticipated workload-reducing modifications may have features including, but not limited to: breaking up dialogue turns into simpler sentences; presenting alternatives sequentially, rather than together; and phrasing questions for answering by “yes-no” responses; preferring speech modality to tactile and visual modality (see below).
  • Certain embodiments of the invention provide a spectrum of pause handling, and termination ranging as follows by degree of user involvement, which is selectable by the system:
      • Pause with or without a prompt, until workload is reduced;
      • Prompt the user before pausing and allow the user a limited time to cancel the pause;
      • Suggest pausing as a choice to the user;
      • Pause according to user request;
      • Pause until instructed otherwise; and
      • Terminate the dialogue
      • Prompt the user before terminating and allow the user a limited time to cancel;
      • Suggest terminating the dialogue to the user
      • Terminate upon user request.
  • According to related embodiments of the present invention, the difference between pausing a dialogue and terminating the dialogue is that a paused dialogue may be resumed at a later time from the point at which the dialogue was paused, whereas a terminated dialogue is stopped and may not be resumed (but may be restarted). The terms “suspend”, “suspending”, “suspension”, etc., herein denote that a dialogue has been interrupted by either being paused or being terminated. Whether or not a suspended dialogue may be later resumed depends on whether the dialogue was paused or terminated, but in either case the dialogue is interrupted at the time the suspension goes into effect.
  • FIG. 5 is a flowchart of a method according to specific embodiments of the invention. In a step 501 a hazard warning 503 is received. Hazard warning 503 can be signaled by a variety of presently-known methods, including, but not limited to: detection of a hazard ahead by an on-board radar system; and hazard notification by a navigational system, based on road condition information supplied to the navigational system. In a related embodiment, the hazard warning includes a warning of a braking condition of another vehicle ahead of the vehicle. In a step 505, in response to hazard warning 503, the dialogue is immediately suspended via an immediate dialogue suspension action 507.
  • The method illustrated in FIG. 5 may also be considered as a special case of the method illustrated in FIG. 4A, wherein user workload parameter 403 includes hazard warning 503 that a hazard has been detected, and system dialogue turn 407 includes immediate dialogue suspension action 507. In this case, immediate dialogue suspension action 507 is included in workload-reducing dialogue act 409 or in workload-reducing dialogue modification 411. It a related embodiment, workload estimation (FIG. 2A) includes hazard warning 503. In another related embodiment, workload estimation 223 includes immediate dialogue suspension action 507.
  • The method illustrated in FIG. 5 may additionally be considered as special case of the method illustrated in FIG. 4B, wherein user anticipated workload parameter 423 includes hazard warning 503 that a hazard has been detected, and system dialogue rum 427 includes immediate dialogue suspension action 507. In this case, immediate dialogue suspension action 507 is included in anticipated workload-reducing dialogue act 429 or in anticipated workload-reducing dialogue modification 431. In a related embodiment, anticipated workload estimation 253 (FIG. 2B) includes hazard warning 503. In another related embodiment, anticipated workload estimation 253 includes immediate dialogue suspension action 507.
  • According to embodiments of the invention, a dialogue may be simplified by one or more of the following:
      • Breaking up compound requests for information to requests for separate single items of information;
      • Presenting alternatives separately in sequential sentences rather than together in a single sentence; and
      • Presenting questions in low-level or yes/no answer form.
  • According to other embodiments of the invention, a prediction of upcoming increased workload can trigger the speeding up of a dialogue. For example, if the driver is approaching an area of congested traffic or other abnormal driving conditions, the automated dialogue system can receive a prediction that workload will soon increase, and may decide to accelerate an ongoing dialogue so that the dialogue will complete before the workload increases. A dialogue may be speeded up by one or more of the following:
      • Reducing the number of prompts by aggregating information in fewer prompts;
      • Presenting information visually, rather than aurally; and
      • Using implicit confirmation rather than explicit confirmation. For example, if the driver requested information on nearby Chinese restaurants, the dialogue system could respond with an implicit confirmation such as “What price range Chinese restaurant do you seek?” rather than first asking for explicit confirmation that the request was for Chinese restaurants.
    Computer Product
  • A computer product according to the above method embodiments includes a set of executable commands for performing the one or both of the above methods on a computer, wherein the executable commands are contained within a tangible computer-readable non-transitory data storage medium including, but not limited to: computer media such as magnetic media and optical media; computer memory; semiconductor memory storage; flash memory storage; data storage devices and hardware components; and the tangible non-transitory storage devices of a remote computer or communications network; such that when the executable commands of the computer product are executed, the computer product causes the computer to perform one or both of the above methods.
  • In these embodiment, a “computer” is any data processing apparatus for executing a set of executable commands to perform a method of the present invention, including, but not limited to: personal computer; workstation; server; gateway; router; multiplexer, demultiplexer; modulator, demodulator; switch; network; processor; controller; digital appliance, tablet computer; mobile device, mobile telephone; any other device capable of executing the commands.
  • While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will now occur to those of ordinary skill in the art. It is, therefore, to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

Claims (18)

What is claimed is:
1. A method for reducing user distraction associated with interaction with an automated dialogue system, the method comprising:
receiving, by a processor, a user workload parameter;
responsively to the user workload parameter, controlling the automated dialogue system to perform a system dialogue turn that reduces the user workload associated with interacting with the automated dialogue system, wherein the system dialogue turn includes a dialogue act selected from a group consisting of:
a workload-reducing dialogue act; and
a regular dialogue act modified by a workload-reducing dialogue modification.
2. The method of claim 1, wherein the workload-reducing dialogue act comprises an act selected from a group consisting of:
a dialogue pause;
a suggestion of a dialogue pause;
a dialogue termination;
a suggestion of a dialogue termination.
3. The method of claim 2, further comprising a spectrum of pause-handling and termination handling choices ranging by degree of user involvement.
4. The method of claim 1, wherein the workload-reducing dialogue modification comprises a dialogue modification selected from a group consisting of:
a switch to voice modality;
breaking up compound prompts into items;
a change to a yes/no prompt; and
speeding up of a dialogue.
5. The method of claim 4, wherein the speeding up of a dialogue act is in response to a predicted workload increase.
6. The method of claim 1, wherein the user workload parameter includes a hazard warning of a hazard ahead of the vehicle, and wherein at least one of the workload reducing act or the workload reducing dialogue modification includes an immediate suspension of the dialogue.
7. The method of claim 6, wherein the hazard warning includes a warning of a braking condition of another vehicle ahead of the vehicle.
8. A method for reducing user situation non-awareness associated with an automated dialogue system, the method comprising:
receiving, by a processor, a user anticipated workload parameter;
responsively to the user anticipated workload parameter, controlling the automated dialogue system to perform a system dialogue turn that reduces the user workload associated with interacting with the automated dialogue system, wherein the system dialogue turn includes a dialogue act selected from a group consisting of:
a workload-reducing dialogue act; and
a regular dialogue act modified by a workload-reducing modification.
9. The method of claim 8, wherein the workload-reducing dialogue act comprises an act selected from a group consisting of:
a dialogue pause;
a suggestion of a dialogue pause;
a dialogue termination;
a suggestion of a dialogue termination.
10. The method of claim 9, further comprising a spectrum of pause-handling and termination handling choices ranging by degree of user involvement.
11. The method of claim 6, wherein the workload-reducing dialogue modification comprises a dialogue modification selected from a group consisting of:
a switch to voice modality;
breaking up compound prompts into items;
a change to a yes/no prompt; and
speeding up of a dialogue.
12. The method of claim 11, wherein the speeding up of a dialogue act is in response to a predicted anticipated workload increase.
13. A dialogue system for reducing user distraction associated with interaction with automated dialogue, the system comprising:
a dialogue control unit;
a storage device containing a dialogue policy;
a workload estimation unit operative to:
receive a workload parameter indicative of a user workload; and
compute a workload estimate; and
input the workload estimate into the dialogue control unit.
14. The dialogue system of claim 13, wherein the workload parameter includes a hazard warning of a hazard ahead of the vehicle, and wherein the workload estimate includes the hazard warning.
15. The dialogue system of claim 14, wherein the hazard warning includes a warning of a braking condition of another vehicle ahead of the vehicle.
16. A dialogue system for reducing user situation non-awareness associated with interaction with automated dialogue, the system comprising:
a dialogue control unit;
a storage device containing a dialogue policy;
an anticipated workload estimation unit operative to:
receive an anticipated workload parameter indicative of an anticipated user workload; and
compute an anticipated workload estimate; and
input the anticipated workload estimate into the dialogue control unit.
17. The dialogue system of claim 16, wherein the anticipated workload parameter includes a hazard warning of a hazard ahead of the vehicle, and wherein the anticipated workload estimate includes the hazard warning.
18. The dialogue system of claim 17, wherein the hazard warning includes a warning of a braking condition of another vehicle ahead of the vehicle.
US15/005,483 2012-05-29 2016-01-25 Reducing driver distraction in spoken dialogue Abandoned US20160171888A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/005,483 US20160171888A1 (en) 2012-05-29 2016-01-25 Reducing driver distraction in spoken dialogue

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US201261652600P 2012-05-29 2012-05-29
US13/893,433 US9251704B2 (en) 2012-05-29 2013-05-14 Reducing driver distraction in spoken dialogue
US15/005,483 US20160171888A1 (en) 2012-05-29 2016-01-25 Reducing driver distraction in spoken dialogue

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US13/893,433 Continuation US9251704B2 (en) 2012-05-29 2013-05-14 Reducing driver distraction in spoken dialogue

Publications (1)

Publication Number Publication Date
US20160171888A1 true US20160171888A1 (en) 2016-06-16

Family

ID=49669534

Family Applications (2)

Application Number Title Priority Date Filing Date
US13/893,433 Expired - Fee Related US9251704B2 (en) 2012-05-29 2013-05-14 Reducing driver distraction in spoken dialogue
US15/005,483 Abandoned US20160171888A1 (en) 2012-05-29 2016-01-25 Reducing driver distraction in spoken dialogue

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US13/893,433 Expired - Fee Related US9251704B2 (en) 2012-05-29 2013-05-14 Reducing driver distraction in spoken dialogue

Country Status (2)

Country Link
US (2) US9251704B2 (en)
CN (1) CN103454930B (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180157980A1 (en) * 2016-12-07 2018-06-07 International Business Machines Corporation Cognitive dialog system for driving safety

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9418674B2 (en) * 2012-01-17 2016-08-16 GM Global Technology Operations LLC Method and system for using vehicle sound information to enhance audio prompting
JP6515764B2 (en) 2015-09-28 2019-05-22 株式会社デンソー Dialogue device and dialogue method
US20170221480A1 (en) * 2016-01-29 2017-08-03 GM Global Technology Operations LLC Speech recognition systems and methods for automated driving
US20170287476A1 (en) * 2016-03-31 2017-10-05 GM Global Technology Operations LLC Vehicle aware speech recognition systems and methods
JP2019535089A (en) 2016-08-19 2019-12-05 マサチューセッツ インスティテュート オブ テクノロジー System and method for providing visual distribution management
US10525984B2 (en) 2016-08-19 2020-01-07 Massachusetts Institute Of Technology Systems and methods for using an attention buffer to improve resource allocation management
DE102016014383A1 (en) * 2016-12-02 2018-06-07 Lucas Automotive Gmbh Extended driver attention module for driver assistance systems
GB2558671B (en) * 2017-01-17 2020-02-19 Jaguar Land Rover Ltd Communication control apparatus and method
JP7204739B2 (en) * 2018-03-30 2023-01-16 ソニーセミコンダクタソリューションズ株式会社 Information processing device, mobile device, method, and program
US11427216B2 (en) 2019-06-06 2022-08-30 GM Global Technology Operations LLC User activity-based customization of vehicle prompts

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050192730A1 (en) * 2004-02-29 2005-09-01 Ibm Corporation Driver safety manager
US7365769B1 (en) * 2000-07-06 2008-04-29 Donald Mager Activating a vehicle's own brake lights and/or brakes when brake lights are sensed in front of the vehicle, including responsively to the proximity of, and/or rate of closure with, a forward vehicle
US7831433B1 (en) * 2005-02-03 2010-11-09 Hrl Laboratories, Llc System and method for using context in navigation dialog
US8583439B1 (en) * 2004-01-12 2013-11-12 Verizon Services Corp. Enhanced interface for use with speech recognition

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6598018B1 (en) * 1999-12-15 2003-07-22 Matsushita Electric Industrial Co., Ltd. Method for natural dialog interface to car devices
US6964023B2 (en) * 2001-02-05 2005-11-08 International Business Machines Corporation System and method for multi-modal focus detection, referential ambiguity resolution and mood classification using multi-modal input
US7639148B2 (en) * 2003-06-06 2009-12-29 Volvo Technology Corporation Method and arrangement for controlling vehicular subsystems based on interpreted driver activity
DE10342541A1 (en) 2003-09-15 2005-05-12 Daimler Chrysler Ag Workload-dependent dialogue
DE102004048013A1 (en) * 2004-10-01 2006-04-06 Robert Bosch Gmbh Method and device for driver assistance
DE102004051963A1 (en) * 2004-10-25 2006-05-04 Robert Bosch Gmbh Method for driver assistance
US7652583B2 (en) * 2007-03-20 2010-01-26 Deere & Company Method and system for maintaining operator alertness
US8022831B1 (en) * 2008-01-03 2011-09-20 Pamela Wood-Eyre Interactive fatigue management system and method
CN102047304B (en) * 2008-08-05 2013-04-03 松下电器产业株式会社 Driver awareness degree judgment device, method, and program
JP4500369B2 (en) * 2008-09-19 2010-07-14 パナソニック株式会社 Distraction detection device, distraction detection method, and computer program
US8260482B1 (en) * 2010-04-28 2012-09-04 Google Inc. User interface for displaying internal state of autonomous driving system
DE102010049086A1 (en) * 2010-10-21 2012-04-26 Gm Global Technology Operations Llc (N.D.Ges.D. Staates Delaware) Method for assessing driver attention
US8902054B2 (en) * 2011-02-10 2014-12-02 Sitting Man, Llc Methods, systems, and computer program products for managing operation of a portable electronic device
US20130131907A1 (en) * 2011-11-17 2013-05-23 GM Global Technology Operations LLC System and method for managing misuse of autonomous driving
US20130325482A1 (en) 2012-05-29 2013-12-05 GM Global Technology Operations LLC Estimating congnitive-load in human-machine interaction
US9007198B2 (en) * 2012-11-02 2015-04-14 Toyota Motor Engineering & Manufacturing North America, Inc. Adaptive Actuator interface for active driver warning
US9085303B2 (en) * 2012-11-15 2015-07-21 Sri International Vehicle personal assistant
US9530409B2 (en) * 2013-01-23 2016-12-27 Blackberry Limited Event-triggered hands-free multitasking for media playback
US9505412B2 (en) * 2013-08-02 2016-11-29 Honda Motor Co., Ltd. System and method for detection and utilization of driver distraction level

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7365769B1 (en) * 2000-07-06 2008-04-29 Donald Mager Activating a vehicle's own brake lights and/or brakes when brake lights are sensed in front of the vehicle, including responsively to the proximity of, and/or rate of closure with, a forward vehicle
US8583439B1 (en) * 2004-01-12 2013-11-12 Verizon Services Corp. Enhanced interface for use with speech recognition
US20050192730A1 (en) * 2004-02-29 2005-09-01 Ibm Corporation Driver safety manager
US7831433B1 (en) * 2005-02-03 2010-11-09 Hrl Laboratories, Llc System and method for using context in navigation dialog

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180157980A1 (en) * 2016-12-07 2018-06-07 International Business Machines Corporation Cognitive dialog system for driving safety
US10832148B2 (en) * 2016-12-07 2020-11-10 International Business Machines Corporation Cognitive dialog system for driving safety

Also Published As

Publication number Publication date
US9251704B2 (en) 2016-02-02
CN103454930A (en) 2013-12-18
CN103454930B (en) 2016-08-10
US20130321171A1 (en) 2013-12-05

Similar Documents

Publication Publication Date Title
US9251704B2 (en) Reducing driver distraction in spoken dialogue
US20230094154A1 (en) Manual control re-engagement in an autonomous vehicle
Merat et al. Transition to manual: Driver behaviour when resuming control from a highly automated vehicle
CN106557160B (en) Vehicle-user interaction system
Walch et al. From car-driver-handovers to cooperative interfaces: Visions for driver–vehicle interaction in automated driving
EP3384475B1 (en) System and method of conversational adjustment based on user's cognitive state
JP7455456B2 (en) Method and system for enhanced warning based on driver status in hybrid driving
Wilson et al. Driver trust & mode confusion in an on-road study of level-2 automated vehicle technology
US11491994B2 (en) Systems and methods for detecting and dynamically mitigating driver fatigue
Naujoks et al. Controllability of partially automated driving functions–does it matter whether drivers are allowed to take their hands off the steering wheel?
US20190111933A1 (en) Method for producing control data for rule-based driver assistance
KR20200010503A (en) Dynamic adjustment of notification output provision to reduce user distraction and / or to reduce the use of computing resources
Naujoks et al. Driving performance at lateral system limits during partially automated driving
KR20200108827A (en) Method and system for switching driving mode based on self-awareness parameters in hybrid driving
CN111699521A (en) Method and system for driver state based driving mode switching in hybrid driving
Strand et al. Exploring end-user experiences: self-perceived notions on use of adaptive cruise control systems
EP2276646A1 (en) Method and system for modifying a drive plan of a vehicle towards a destination
Gold Modeling of take-over performance in highly automated vehicle guidance
Schieben et al. Evaluation of three different interaction designs for an automatic steering intervention
Wandtner Non-driving related tasks in highly automated driving-Effects of task characteristics and drivers' self-regulation on take-over performance
US10915770B2 (en) Method for assisting a driver of an ego-vehicle in making use of a gap between vehicles, corresponding driver assistance system and vehicle equipped with such driver assistance system
Sharma et al. Human factors of vehicle automation
JP2018063489A (en) Operator state estimation device
CN111372830A (en) Method and system for risk-based driving mode switching in hybrid driving
KR20160038158A (en) Detecting system and Method of dangerous driving for driver tendency

Legal Events

Date Code Title Description
STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION