DE102013209780B4 - Method and dialog system for improving vehicle safety by estimating a cognitive load of driving-related activities through a human-machine interface - Google Patents

Abstract

A method for improving vehicle safety by estimating a cognitive load of driving-related activities through a human-machine interface, the method comprising using a computer to:identify an expression of cognitive load within a user expression generated by a user, interacting with a dialog system (100), the dialog system (100) comprising one or more sensors, and wherein the user expression comprises applying pressure to a steering wheel, the pressure exceeding a threshold pressure; andusing a dynamic Bayesian network to estimate a level of cognitive load experienced by the user interacting with the dialog system (100) based on the expression of the cognitive load; wherein the interaction system (100) performs a system interaction action, wherein a machine action generator (230) transforms the system action into a machine interaction action, and wherein the machine interaction action comprises redirecting a vehicle to a selected location or providing some form of haptic feedback.

Description

BACKGROUND OF THE PRESENT INVENTION

The present invention relates generally to dialog systems, and more particularly to estimating the cognitive load of users who interact with them. Cognitive load can be viewed as a measure of the mental effort experienced by the user and can be expressed explicitly or non-explicitly during interaction with the system. Estimating the cognitive load during a user interaction facilitates more accurate determination of the user's real goals. When implemented in touring vehicles, such assessments can aid in determining the cognitive load related to driving activities.

Such systems are used in many different applications, including but not limited to automotive safety, telemetry systems used to remotely service vehicles, or infotainment activities that facilitate the capture or tracking of recreational matters of interest according to the expressed intent during dialog sessions. It should be recognized that such systems and methods also have application in any vehicle installation, including train and plane travel and pleasure cruises.

Typical driver-related activities that can cause cognitive load in a driver include road conditions, traffic conditions, passenger activities, ride comfort and ease of use, driving or travel time, and driving experience.

US 2011/0 207 099 A1 discloses a method for measuring the cognitive load of a person participating in an online lesson, the cognitive load of the person being transmitted to the teacher. Here, word-based input is first received from the person who creates it while performing a task. Furthermore, the input can also take place as text input via a keyboard or a digital pen. Predetermined grammatical features are identified from the word-based input. The identified grammatical features are finally weighted and combined based on the task to obtain a measure indicative of the person's cognitive load, with the task being classified based on predetermined tasks and user profiles. Further state of the art is out U.S. 6,731,307 B1 and US 2006 / 0 074 670 A1.

SUMMARY

The object of the invention is to provide an improved method and dialog system for improving vehicle safety by estimating a cognitive load of driving-related activities through a human-machine interface.

A method with the features of claim 1 and a dialog system with the features of claim 5 are provided to solve the problem. Advantageous developments of the invention can be found in the dependent claims, the description and the drawings.

character list

• 1 ein schematisches Blockdiagramm einer Hardware, die in Dialogsystemen verwendet wird, gemäß einer Ausführungsform der vorliegenden Erfindung ist;
• 2 ein schematisches Blockdiagramm von primären Softwaremodulen, die in einem Dialogsystem verwendet werden, gemäß einer Ausführungsform der vorliegenden Erfindung ist;
• 3 ein Ablaufplan, der ein Verfahren darstellt, das vom System von 1 und 2 verwendet wird, gemäß einer Ausführungsform der vorliegenden Erfindung ist;
• 4 ein teilweises Bayessches Netz, das im Systemen von 1 und 2 verwendet wird, zum statistischen Modellieren der Auswirkung der kognitiven Belastung auf Anwenderzielabschätzungen ist; und
• 5 ein nichtflüchtiges, computerlesbares Medium, auf dem Befehle zum statistischen Modellieren der kognitiven Belastung eines Anwenders, der mit einem Dialogsystem wechselwirkt, gespeichert sind, gemäß einer Ausführungsform der vorliegenden Erfindung darstellt.

The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. However, the invention as regards components, features, method of operation and advantages may best be understood by reference to the following detailed description and accompanying drawings, in which:

• 1 Figure 12 is a schematic block diagram of hardware used in dialog systems according to an embodiment of the present invention;
• 2 Figure 12 is a schematic block diagram of primary software modules used in a dialog system according to an embodiment of the present invention;
• 3 a flowchart depicting a procedure used by the system of 1 and 2 is used according to an embodiment of the present invention;
• 4 a partial Bayesian network used in the systems of 1 and 2 is used to statistically model the impact of cognitive load on user goal estimates; and
• 5 Figure 11 illustrates a non-transitory, computer-readable medium storing instructions for statistically modeling the cognitive load of a user interacting with a dialog system, according to an embodiment of the present invention.

It will be appreciated that for the sake of clarity, the elements shown in the figures have not necessarily been drawn to scale, and reference numbers may be repeated in different figures to indicate corresponding or analogous elements.

DETAILED DESCRIPTION OF THE PRESENT INVENTION

In the following detailed description, numerous details are set forth in order to provide a thorough understanding of the invention. However, it should be understood by those skilled in the art that the present invention may be practiced without these specific details. For the sake of clarity, well-known methods, procedures, and components are not described in detail.

The present invention is a dialog system effective to model the cognitive load of users interacting with the system.

• „Anwendertätigkeit“ bezieht sich auf einen Anwenderausdruck, der in irgendeiner Modalität oder Kombination von Modalitäten ausgedrückt wird, während er mit einem Dialogsystem wechselwirkt. Die Anwendertätigkeit kann eine explizite Zielaussage, eine Bestätigung oder eine Antwort auf eine Maschinen-Dialog-Handlung und einen Ausdruck von kognitiver Belastung umfassen.

The following terms are used throughout this document:

• "User action" refers to a user expression expressed in any modality or combination of modalities while interacting with a dialog system. User action may include an explicit goal statement, an acknowledgment or response to a machine dialog action, and an expression of cognitive load.

For example, the goal statement may be directed to performing an activity such as booking reservations at a restaurant or requesting information or providing information.

An expression of cognitive load can be expressed as either disfluency embedded in user activity, or as an explicit statement indicating cognitive load, or a combination of both. Disfluencies are regionally and time-sensitive in that they reflect deviations from cultural standards of expression that vary from one region to another and from one time period to another, and thus a disfluency in one region cannot be considered a disfluency in another region, equally changing standards of expression also change over time and therefore disfluencies are evaluated in the relevant social context. As indicated, the present invention is operative in any of a variety of modalities of expression; verbal expression, physical contact or through symbolism.

• - Aussprachefehler
• - Abgeschnittene Worte oder Sätze mitten in der Äußerung
• - Füllworte von nicht lexikalischen Vokabeln wie z. B. „uh“, „ähm“, „gut“, „err“ und „yea“
• - Füllworte von lexikalischen Vokabeln wie z. B. „mal sehen“
• - Wiederholungen von Worten, Ausdrücken oder Silben
• - Verbesserte Äußerungen, in denen der Sprecher seine eigenen Versprecher korrigiert
• - Verlängerte Pausen zwischen Worten
• - Wortaustausch wie z. B. „Wie viel ... teuer ist es?“
• - Artikulationsfehler wie z. B. „Hier auf links wenden“
• - Falsche Starts wie „Ja es ist ... tatsächlich ist es ...“

Typical examples of verbal influenza include:

• - pronunciation error
• - Cut off words or sentences in the middle of the utterance
• - Filler words of non-lexical vocabulary such as e.g. B. "uh", "um", "gut", "err" and "yea"
• - Filler words from lexical vocabulary such as B. "Let's see"
• - Repetition of words, phrases or syllables
• - Improved utterances where the speaker corrects his own slips of the tongue
• - Extended pauses between words
• - exchange of words such as B. "How much... expensive is it?"
• - Articulation errors such as B. "Turn left here"
• - False starts like "yes it is... actually it is..."

Explicit statements indicating cognitive load include, but are not limited to, “wait”, “hang in there”, “go on”, “say that again”, “please repeat”, “go back”.

Examples of visual non-smoothness include, but are not limited to, facial gestures and unusual hand movements that can be detected by an imaging system, such as tapping on the steering wheel or dashboard.

Examples of influencing mediated by physical contact include, but are not limited to, applying more than normal pressure to the steering wheel, tapping the steering wheel or dashboard with the above predetermined standards of force or frequency, or applying force to a portion of a dashboard, lacking a device actuator such as a switch or button, or touching a touch screen on a portion lacking a virtual device actuator.

"User dialog actions" refer, according to embodiments, to a dialog system's understanding of user actions, including any associated non-fluency or statement indicative of cognitive load, in any modality or combination of modalities. User interaction actions are also referred to as "user interaction actions" or "observation variables". Understanding user actions can be achieved via a language or multimodal understanding system within the dialog system.

"Machine dialog actions" refer to actions taken by a dialog control module in any modality or combination of modalities based on an assumption of the user's goal, the application of a policy, and other relevant parameters. According to embodiments, machine dialog actions are converted into machine actions by a machine action generator.

"Dialog Control Module" refers to a component of the dialog system that applies a policy that controls the interaction between a user and the dialog system, as will be discussed further.

The present invention relates to human-machine interaction systems, and more particularly relates to interaction systems configured to model effects of cognitive load, the cognitive load may emanate from driving-related activities or from other sources.

Some human-machine interaction systems are configured to statistically model user goals based on explicit input that conveys the user's actions to the system. Embodiments of the present invention can also statistically model effects of cognitive load generated by driving-related or other activities that result in inaccurate estimation of user goals.

In addition to manually operated vehicles, embodiments of the present system may also have application in autonomous vehicles. The dialog system in these applications can assess a level of expected cognitive load to be suffered by a driver when converting autonomous driving to manual driving.

Turning now to the characters, 1 Figure 12 is a schematic diagram of a statistically based multi-modal dialog system according to an embodiment of the present invention.

Dialogue system 100 includes one or more processors or controllers 20, memory 30, long-term data storage 40, input devices 50, and output devices 60.

The processor or controller 20 includes a central processing unit or multiple processors. Memory 30 may be random access memory (RAM), read only memory (ROM). It should be appreciated that image data, code, and other relevant data structures are stored in the memory and/or storage devices noted above.

Memory 30 includes, but is not limited to, random access memory, flash memory, or any other short-term storage device.

Long-term data storage devices 40 include, but are not limited to, a hard disk drive, a floppy disk drive, a compact disk drive, or any combination of such devices.

The dialog system 100 includes, among other things, one or more sensors, a digital camera, and a video camera of a computer view 10. Image data may also be input into the dialog system 100 from non-dedicated devices or databases.

Non-limiting examples of input devices 50 include, but are not limited to, audio sensing and touch actuated input devices, including touch sensors, located in proximity to other device actuating means such as buttons, knobs, switches, and touch screens.

Non-limiting examples of output devices 60 include, but are not limited to, visual, audio, and haptic feedback devices. It should be appreciated that according to one embodiment, input devices 50 and output devices 60 may be combined into a single device.

2 12 illustrates primary modules of a statistical dialog system including an understanding module 220, a dialog control module 225, and a machine action generator module 230 according to embodiments of the present invention. The understanding module 220 is configured according to embodiments of the invention to identify user actions from user expressions in dialog with a dialog system. Either nonfluency, explicit user expressions indicative of cognitive load, or a combination of both may be included in the list of identified user actions according to embodiments. The output of the understanding module 220 is a scored trust list of user interactions, according to embodiments.

The dialog control module 225 is configured, according to an embodiment of the invention, to apply a user model with probability distributions of the user's cognitive load and the user's goals and apply a policy to decide an optimal system dialog action to achieve the user's real goal.

The machine action generator 230 is configured to transform the system dialog action into a machine action in accordance with embodiments of the present invention.

3 12 depicts a flowchart of the primary steps involved in modeling the cognitive load of a user interacting with a dialog system, in accordance with embodiments of the present invention.

In step 300, a user expression is captured in any of the relevant modalities with the appropriate input device identified above.

At step 310, a comprehension module identifies user interaction actions with nonfluency and statements indicative of cognitive load, as noted above in an embodiment of the invention. Examples of verbal imfluency include the pronunciation errors noted above, clipping, lexical and non-lexical filler words, repetitions, enhanced utterances, and lengthened pauses. These obscurities can be recognized by a speech recognition system module and analyzed by a semantic analyzer and passed to a dialog control module as part of a list of alternatives, as discussed further.

Similarly, visual and tactile disfluencies can also be used as indicators of cognitive load, as noted above.

The following is an example of verbal disfluency expressed as a false start when requesting Chinese food: $$\text{'}\text{'}Wasist...wOGibt\text{e}scHinesiscHesEssen?\text{'}\text{'}$$

wobei „Inform“ der Typ der Anwenderdialoghandlung ist, „Essen“ ein Attribut ist, „Essen=chinesisch“ ein Attributwertpaar ist, „Unflüssigkeit“ ein zweites Attribut ist und „Unflüssigkeit“ = ‚falscher Start‘ ein zweites Attributwertpaar ist. Die Anwesenheit des Attributwertpaars „Unflüssigkeit“ = ‚falscher Start‘ bedeutet, dass Informationen hinsichtlich chinesischen Essens mit einer speziellen Unflüssigkeit angefordert wurden, die gemäß einer bestimmten Ausführungsform als ‚falscher Start‘ definiert ist.Such a statement can be analyzed as a user dialogue action embedded with attributes for nonfluency or explicit expressions of cognitive load. For example, the above statement can be parsed as: $$InfO\mathrm{right}m\left(Essen=cHinesiscH,unfl\ddot{\mathrm{and}}ssiGkeit=\text{'}falscHe\mathrm{right}Sta\mathrm{right}t\text{'}\right)$$

where "Inform" is the type of user interaction action, "Eat" is an attribute, "Eat=Chinese" is an attribute-value pair, "Unfluent" is a second attribute, and "Unfluent" = 'wrong start' is a second attribute-value pair. The presence of the attribute-value pair 'unliquid' = 'false start' means that information regarding Chinese food has been requested with a particular unliquid, which according to a particular embodiment is defined as 'false start'.

wobei die Pause in die Anwenderdialogaussage als Attributwertpaar eingebettet ist.In a second example, a request for information about Chinese food, where the user explicitly asks for a time delay, for example by saying "wait", can be parsed as: $$InfO\mathrm{right}m\left(Essen=cHinesiscH,expli\mathrm{e.g}it=\text{'}Pa\mathrm{and}se\text{'}\right)$$

where the pause is embedded in the user dialog statement as an attribute-value pair.

Additional attributes include 'resume', 'repeat' and 'return', among others.

After analyzing, user interaction actions are assigned confidence point values that are determined to most likely represent the user action, according to certain embodiments.

In step 320, a user model operative to model cognitive load using the user interaction actions identified in step 310 and other factors determines a target list and associated probabilities and optionally a cognitive load estimate. Application models that may be used include, but are not limited to, Bayesian networks, neural networks, or any other model that provides such functionality.

In step 330, a dialog system according to an embodiment of the invention applies a policy to the resulting target list to decide on a machine dialog action. In certain embodiments, the policy may be determined in advance from a policy learning process using dialog success metrics, rewards, and interaction logs.

In step 340, a dialog system according to embodiments performs a system dialog action 340 based on the policy decision made in step 330. Examples of machine interaction actions include, but are not limited to, asking the user for more information, requesting verbal confirmation, diverting a vehicle to a selected location, playing selected music, providing some form of haptic feedback, or any combination of the above.

4 FIG. 12 illustrates a partial dynamic Bayesian network, generally designated 400, that models cognitive load in a human-machine interaction, performed at step 320 of FIG 3 can be used.

In particular, in each dialog round, in certain embodiments, the cognitive load variable 410 depends on previous dialog round variables; a variable of the previous user goal 415; a variable 420 of the previous machine dialogue action and a variable 425 of the previous cognitive load.

Further, parameters of probability distributions representing the dependence of the cognitive load variable 410 on each of those variables are shown in nodes 415A, 420A, and 425A according to embodiments. In particular, workload variable 410 depends on parameter 415A associated with previous user goal variable 415, parameter 420A associated with observed machine interaction action 420, and parameter 425A associated with cognitive load 425. These parameters can be calculated using a database of dialogue patterns in a dedicated learning session. Current user dialog patterns can be used for learning; or conversation logs from multiple users may be used in a learning stage according to embodiments. Additionally, according to embodiments, the parameters may be learned by expectation propagation. The workload variable 410 may represent any one of three levels of cognitive workload, according to one embodiment of the invention; accept "low", "medium" and "high".

Continuing with the dynamic Bayesian network, cognitive workload 410 may in turn be modeled as an unconstrained dependency for user activity 435, which in turn is modeled as dependent on user goal 430, according to embodiments.

The dependency of user activity 435 on workload is also parameterized as represented by parameter 435A, as noted above.

User interaction action 440 is an observation variable or an observed user interaction action variable and is modeled as directly dependent on user action 435 in certain embodiments.

In operation, the cognitive workload 410 may be estimated by expectation propagation in the Bayesian network given the observed variables 440 and 420 according to embodiments.

As an illustrative example of how informal dependencies can affect current cognitive load, in certain embodiments, assuming that the previous user goal 415 is labor intensive, then there would be a correspondingly high conditional probability that the current cognitive workload 410 differed from the previous user goal 415 depends. For example, in a previous round of dialogue, a user goal of finding an unspecified piece of "rock" music from a very large selection may contribute to the current cognitive load.

Likewise, a previous machine dialog action 420 to display a long list of song titles for user selection may also impact the current cognitive load 410 . The previous cognitive load of 425 may affect the current cognitive load of node 410 in certain embodiments.

The user model of dependencies may be used to calculate a probability of user goals using Bayesian expectational propagation network methods according to embodiments of the invention. It should be recognized that neural network models and other models that provide such functionality may also be used according to certain embodiments.

Embodiments of the present invention also include provisions for estimating cognitive load based on data obtained from data acquisition devices or systems unrelated to the dialogue system. This can be done by modeling such acquired data as an additional observed node with appropriate dependencies in the Bayesian network model.

5 Figure 1 represents non-limiting computer-readable media containing executable code for configuring a computer system to execute the improved cognitive load dialogue system described above, according to embodiments of the present invention.

Although certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. Therefore, it is to be understood that the appended claims should cover all such modifications and changes as fall within the true spirit of the invention.

Claims (8)

A method for improving vehicle safety by estimating a cognitive load of driving-related activities through a human-machine interface, the method comprising using a computer to: identify an expression of cognitive load within a user expression generated by a user, interacting with a dialog system (100), the dialog system (100) comprising one or more sensors, and wherein the user expression comprises applying pressure to a steering wheel, the pressure exceeding a threshold pressure; and using a dynamic Bayesian network to estimate a level of cognitive load experienced by the user interacting with the dialog system (100) based on the expression of the cognitive load; wherein the dialog system (100) performs a system dialog action, wherein a machine action generator (230) converts the system action into a machine dialog action transformed, and wherein the machine interaction action includes redirecting a vehicle to a selected location or providing some form of haptic feedback. procedure after claim 1 , wherein the dialogue system (100) comprises a multimodal dialogue system. procedure after claim 1 wherein the dialog system (100) receives input from at least one data acquisition device unrelated to the dialog system (100). procedure after claim 1 , further comprising selecting a system interaction action based at least in part on target probabilities determined by the dynamic Bayesian network. Dialogue system (100) for improving vehicle safety by estimating a cognitive load of driving-related activities of a user interacting with the system, the system comprising: a processor (20) configured to: recognizing an expression of cognitive load in a user expression sensed by the dialog system (100), the dialog system (100) including one or more sensors, and wherein the user expression is an application of pressure to a steering wheel, the pressure a Exceeds threshold pressure includes; and use a user model to estimate a level of cognitive load experienced by the user based at least in part on the expression of the cognitive load; wherein the interaction system (100) performs a system interaction action, wherein a machine action generator (230) transforms the system action into a machine interaction action, and wherein the machine interaction action comprises redirecting a vehicle to a selected location or providing some form of haptic feedback. system after claim 5 , where the user model is implemented as a dynamic Bayesian network comprising an observed user dialog-action variable that depends directly or indirectly on a cognitive load variable. system after claim 6 , where the cognitive load variable depends on at least one previous dialogue round variable. system after claim 7 wherein the previous conversational round variable is selected from the group consisting of a previous cognitive load variable, a previous user goal variable, and a previous machine conversational action variable.

Images