FR3100367A1 - IT agent with advanced response to a user message - Google Patents

Abstract

The invention relates to the processing of a message (ENR) sent by a user (UT) by an IT agent, comprising: - recognizing key words in the message, - determining a response to the message based at least on the words - recognized keys, - additionally detecting at least one emotion of the user in the transmitted message, and - adapting the aforementioned response to the user's message on the basis also of the detected emotion Figure of the abstract: Figure 1

Description

Computer agent with advanced response to a user message

This disclosure concerns the field of human/machine interaction and more particularly a voice assistant.

“Voice assistant” means a computer agent dialoguing with a human user (in audio or written communication via a touch screen or other). Thanks to voice recognition techniques, the agent recognizes the words spoken by the user. According to rules, the recognized words trigger actions, such as answering a user's question (" What will the weather be like tomorrow ? "), turning on a light, etc.

The applications of conversational agents are multiple (providing advice to a customer during a purchase, after-sales service, guiding a traveler, etc.).

These voice assistants (in audio mode) consist of a connected speaker located within reach of the user. This speaker contains a loudspeaker and a microphone, and is connected to a wide area network such as the Internet. When the speaker detects, locally, a start keyword to start recording a user request, the speaker activates a mode for recording the sentences spoken by the user. This record is sent (over the internet) to a remote server to be processed and a response produced in return.

Usually, such a server only recognizes spoken words and therefore does not detect information (which can be very relevant) about the current mood of the user.

Indeed, the voice assistants currently available are not able to adapt to the emotions felt by their user, in particular those that the assistant himself would have provoked.

For example, current voice assistants are not able to speak softer to a user when they feel fear. For example again, current voice assistants are not able to take on a more playful tone when addressing a happy user, or to make humor with it (by telling him a joke for example) .

Summary

This disclosure improves this situation.

A method implemented by a computer agent, for processing a message transmitted by a user, is proposed, the method comprising:
- recognize keywords in the message, and
- determining a response to the message based at least on the recognized keywords,
the method further comprising:
- detecting at least one emotion of the user in the transmitted message, and
- adapting said response to the message on the basis of the detected emotion.

Thus, the present disclosure proposes the implementation of a computer agent (such as a voice assistant for example), which has the ability to be "empathetic", and thus to be able to adapt, in a way continuous and automatic, to the emotions felt by the user (these emotions can change at any time).

Emotion can be detected in characteristic words of the user's message. For example, the computer agent can recognize certain words of annoyance characterizing an emotion such as anger and/or fear, and adapt the content of the message response to this emotion detection.

Nevertheless, in addition or as a variant, in an application in particular where the message is a voice message, the emotion can also be detected on the basis of at least one biometric parameter measured on the user during a pronunciation of the message by the 'user.

This biometric parameter can be, for example, a voice intonation in the message spoken by the user, a facial expression, a posture, a particular blood flow of the user, or other, noted during the pronunciation of the message.

For example, the voice message is acquired by at least one microphone and the biometric parameter is at least one voice intonation in the message spoken by the user. Thus, a louder voice volume in the user's message with, for example, a faster rate can characterize anger and/or fear. Similarly, a variation in tone (for example more serious with a lower rate) can characterize a common emotion of sadness, etc.

Alternatively or in addition, the voice message is acquired by at least one camera and the biometric parameter is at least one of a facial expression and a posture of the user when speaking the message. Thus, for example, the analysis of facial expression and/or posture can confirm the emotion detected in the content of the message and possibly in the intonation of the user's message.

For example, it is possible to detect at least one emotion among joy, anger, sadness and fear.

It is also possible to assign a score to each emotion detected and the adaptation of the aforementioned response can then be a function of this score. Such a score can be based on selected criteria (as presented in the tables of the detailed description below) and by evaluation of selected parameters such as variations in intonation in the voice, or in the vocal volume, or the speed of pronunciation of words by the user.

Thus, the response of the IT agent to the user's message can be finely adapted to the emotion detected in the message. This adaptation can be in the form of the response. For example, if the computer agent is a voice assistant, therefore capable of generating an audio signal to be broadcast in response to the message transmitted, the adaptation of the response on the basis of the emotion detected can comprise for example at least one element among a volume modulation of the audio signal, a choice of a synthetic voice among several possible voices, a particular voice intonation, or else an appropriate rate of phrasing in the audio signal.

Adaptation to the detected emotion can also be at the very bottom of the computer agent's response. Thus, for example, the response based on the detected emotion may comprise at least one group of pre-recorded words, added or not to the response depending on the detected emotion. It can be for example a "joke" pre-recorded in a database and which can be played if the user is detected to be in a good mood (joy being the predominant emotion).

In addition or as a variant, in an embodiment where the IT agent is configured to respond to the message sent according to a choice among a plurality of items of a conversational menu, this choice can be a function of both:
- keywords recognized in the message sent, and
- the user's emotion detected in the transmitted message.

For example, a voice assistant of a navigation system does not insist on obtaining a destination street number, if annoyance of the user is observed (the detected emotion being anger) during the repetition of the house number question. In this case, the voice assistant determines that the destination is "anywhere" on that street.

The present disclosure also relates to a device comprising a processing circuit configured for the implementation of a method according to the present disclosure. It may be for example a device of the connected speaker type, comprising a voice assistance computer module (this module corresponding to the aforementioned computer agent) and which is configured so as not to require recourse to a remote server. In this case, this device can locally process both the content of the user's message and the detection of the emotion present.

The present disclosure also relates to a system for the implementation of such a method and comprising a computer agent connected via a network to at least one server, this server comprising a processing circuit configured to detect at least one emotion of the user in the transmitted message, and adapting the response to the user's message further based on the detected emotion. For example, a first server can be provided for the recognition of keywords in the content of the message itself and a second server in particular for the detection of emotions in the message and the adaptation of the response to the emotions detected. Furthermore, these first and second servers can be grouped together in the same global server.

This computer agent can be a voice assistant (for example a speaker connected to the aforementioned network). Nevertheless, as a variant, it may be in other applications a conversational agent of the "chatbot" type for example, or even a contextual automatic responder (telephone or SMS/MMS) which adapts their response according to at least one of the following parameters: the user (and his current emotions), the context of reception by the user (car, meeting) (and the emotions which may be aroused by such a context), or others.

The present disclosure also relates to a computer program comprising instructions for implementing the method above, when these instructions are executed by a processor of a processing circuit.

It also relates to a non-transitory recording medium readable by a computer on which is recorded a program for the implementation of the method when this program is executed by a processor.

Other characteristics, details and advantages will appear on reading the detailed description below, and on analyzing the attached drawings, on which:

Fig. 1

schematically illustrates an embodiment of a computer agent device in a system for implementing the method defined above.

Fig. 2

illustrates steps of the method according to one embodiment.

The drawings and the description below contain, for the most part, certain elements. They may therefore not only serve to better understand this disclosure, but also contribute to its definition, where applicable.

Reference is made to FIG. 1 in which the user UT communicates with a voice assistant, for example in the form of a connected speaker EC (here therefore in audio mode). The user can pronounce at least one ENR sentence then input by a microphone MIC that includes the connected speaker EC. This last EC also includes:
- an IN input interface for receiving an audio signal corresponding to the spoken sentence ENR, this IN interface being connected to
- a processor PROC cooperating with a working memory MEM (capable of storing data of a computer program within the meaning of the present disclosure) for the implementation of at least part of the steps of the method defined above,
- a communication interface COM with at least one server SER (via for example an extended network RES) in order to transmit a signal corresponding to the sentence ENR to the server SER and to obtain in return a response to the sentence ENR which is appropriate at the same time to the content of this sentence but also, within the meaning of the present disclosure, to the current emotion of the user having pronounced this ENR sentence.

To this end, the server SER can implement a detection of this emotion, as well as an interrogation of a database BDD of appropriate rules to respond to the detected emotion.

This appropriate response is then transmitted back from the server SER to the connected speaker and can be played in the form of an audio signal by a loudspeaker HP of the speaker, connected to the processor PROC by an output interface OUT of audio signal.

Similarly, the server SER may comprise a processing circuit including a processor cooperating with a working memory (capable of storing data from a computer program within the meaning of the present disclosure) and a communication interface, via a network such as the network extended RES, with in particular the connected speaker EC and the aforementioned database BDD.

More generally, the processing circuit (comprising the processor PROC, the working memory MEM, etc.) of the connected speaker EC can be shaped:
- to process the signal of the ENR message initially to detect the emotions present in the ENR message, locally, determine the appropriate intonations of a response to the message according to the emotions detected, and send the ENR message to the server in parallel SER so that the latter returns an appropriate response content to the ENR message;
or simply to send the signal of the message ENR to the server SER so that the latter processes both the content of the message ENR and the emotions detected in this message; in this case, the content of the response, itself, can take account of the emotions detected in the ENR message.

Of course, other realizations are still possible. For example, if the capacities of the processing circuit of the connected speaker allow it, it is possible to process locally at the level of the connected speaker EC both the content of the ENR message and the emotions present.

Emotions can be detected in various ways:
- by voice intonations present in the ENR message,
- by the very type of expressions used in the content of the ENR message,
- by data from one or more signal sensors other than the audio signal of the ENR message: it may be for example facial expressions of the user while he pronounces his ENR message and which are picked up by a camera CAM connected to the processor PROC and thus recognized by the processing circuit PROC, MEM.

Indeed, the solutions capable of detecting and interpreting emotions can be of different types, as detailed below.

An emotion, such as joy, fear, anger, sadness, is defined as a complex psychophysiological feeling having a relatively sudden onset and a relatively short duration. Today there are solutions capable of detecting and interpreting an individual's emotions using different data, for example by analysis:
- facial expressions; and or
- vocal expressions; and or
- body posture; and or
- textual expressions without voice intonation (written texts); and or
- blood flow, neural or brain activity via non-invasive methods (using biometric sensors).

These solutions are based on models, more or less complex, mapping human emotions. The simplest models simply map the four emotions generally considered to be primary: fear, anger, joy, and sadness. Other, more comprehensive models include more emotions by listing four so-called "primary" emotions (fear, anger, joy, sadness) and combining them with memory and thinking mechanisms to add four more emotions say "secondary" (confidence, related to joy; disgust, related to sadness; anticipation, related to anger; and surprise, related to fear).

Reference is now made to FIG. 2 in which, during a first step S1, a message such as a sentence pronounced ENR by the user UT is acquired by the microphone MIC, as an audio signal recording intended to be treated for its content and the emotions present in the message. At the same time, it is also possible to acquire biometric data from the user while he pronounces the sentence, for example using a CAM camera filming the user's facial expressions or even his posture, or others, in step S2. Advantageously, these signals are acquired at an acquisition timestamp HR which is stored in memory, to ensure a good correspondence between the content of the message at a given instant and the expression of the emotion detected at this same instant. .

At step S3, one or more emotions are detected, with a possible determination of respective scores linked to basic emotions as presented in the tables below, and this at successive timestamps HR. It is then sought, in step S4, a way in which the voice assistant can respond to this emotion by modulating for example the vocal tone of the response to be given to the message, and/or the rate of the phrasing of the response, or other. For this purpose, the connected speaker EC can refer (possibly via an associated server) to the aforementioned BDD database of emotions and response rules to these emotions. By way of non-limiting examples, the content of the BDD database can link the emotions detected in the user's message (when the latter interacts with the voice assistant), as well as the respective intensities of these , to tuning parameters of the audio signal that the voice assistant produces to implement corresponding personalizations of the voice assistant in response to the detected emotions, as follows:

Emotion detected
(fear, anger, joy or sadness) Intensity of the emotion detected
[1, 5] Voice assistant personalization rules Fear < or = 3 Tone: normal
Speed: normal
Volume: standard Fear > 3 Tone: cheerful
Speed: slow voice
Volume: standard Anger < or = 4 Tone: normal
Speed: normal
Volume: low Anger > 4 Tone: normal
Speed: slow voice
Volume: low Joy … …
Insertion of a predetermined grouping of words (joke type, etc.) Sadness … … … … …

Voice assistant personalization rules (available on the BDD database), for an appropriate response to a message, can indeed be based on an adapted tone of the audio response signal (for example female voice, rather than male), rapid rate of phrasing in the response, appropriate volume of the response audio signal, and/or others.

More complex rules defining the personalization of the voice assistant according to the four basic emotions detected (joy, anger, sadness, fear) can be provided as presented in the example below of a simultaneous detection of two basic emotions in the voice of the user:

Emotion detected #1
(fear, anger, joy or sadness) Intensity of emotion #1
[1, 5] Emotion detected #2
(fear, anger, joy or sadness) Intensity of emotion #2
[1, 5] Voice Assistant Customization Fear > 4 Anger > 4 Tone: cheerful
Speed: slow voice
Volume: low Fear > 4 Sadness > 3 Tone: cheerful
Speed: fast voice
Volume: standard … … … … …

Referring again to Figure 2, once the voice assistant personalization rules are thus defined in step S5 for an appropriate response to the emotions present in the ENR message, the message is played in step S6 by the loudspeaker HP of the connected speaker EC, with selected audio signal modulations, thus personalizing the voice assistant's response to the emotions detected in the REC message.

The present disclosure thus makes it possible to implement an "empathetic" voice assistant, that is to say capable of adapting, in a continuous and automatic way, to the emotions felt by the user. Thus, as the emotions felt by the user evolve, the voice assistant can, for example, change the tone of the synthetic voice used or even the volume thereof. The user experience is thus more sensitive and truly personalized. Other more complex rules can be suggested, such as very short-term learning and adaptation, if the assistant notices that his personalization on the contrary reinforces the first emotion detected (for example the synthetic voice used by the voice assistant, in response to the first emotion detected which is anger, leads to an increase in anger: the personalization then offers a different predefined personalization).

Of course, this disclosure is not limited to the embodiments described above by way of example; it extends to other variants.

Thus, for example, a single server has been described above with reference to FIG. 1, but of course several servers can be provided:
- a server for recognizing the content of the message on the one hand (and the factual response to be given);
- and a separate server for the recognition of the emotions present in the message on the other hand (and the appropriate intonation of the response to be given).

Furthermore, in the example illustrated above, the intensity of the emotions is evaluated by assigning an intensity score corresponding to an integer between 1 and 5 in the tables above (the value "1" corresponding to the intensity the weakest and the value "5" corresponding to the strongest intensity). However, a greater fineness of detection can make it possible to have a scale for example between 1 and 7 (even up to 10).

The voice assistant personalization rules can be predefined by a voice assistant operator and/or are also modifiable by the user by learning their emotions and reactions.

The personalization of the voice assistant can consist, as indicated previously, in the modification of the synthetic voice used by the voice assistant in its interaction with the user and in particular:
- use of a male or female synthesized voice;
- a modification of the tone and/or the harmonies of the synthesized voice (for example by using a normal tone, a playful tone, or a sad tone, or even using a sung, melodious or conversely dissonant voice);
- a modification of the speed of the synthetic voice (for example: using a normal voice, a slow voice, or a fast voice);
- a modulation of the sound volume of the synthesized voice (for example: use a normal volume, a high volume, or a low volume).

However, other ways to customize the voice assistant are possible, such as:
- order the voice assistant to stop "talking" to the user or to stop systematically presenting the same items in its menu;
- order the voice assistant to play humor with the user: for example, play a pre-recorded "joke" randomly during the exchange with the user (a sound recording available on the SER server or on the connected database BDD or in the memory MEM);
- modify a lamp color of the connected speaker (for example the color of the luminous halo emitted by the connected speaker);
- display emoji (pictograms) on a screen on the connected speaker;
- order the voice assistant to recommend specific content (music, films, etc.) to the user (for example, recommend content identified as being more "joyful" to a user whose content is sad identified as a “joyful” state;
- order the voice assistant to suggest particular activities to the user (invite a friend who is available this evening, etc.) and a predefined rule thus makes it possible to suggest to a user feeling angry to invite a friend or other.

Claims (15)

Method implemented by a computer agent, for processing a message transmitted by a user, the method comprising:
- recognize keywords in the message, and
- determining a response to the message based at least on the recognized keywords,
the method further comprising:
- detecting at least one emotion of the user in the transmitted message, and
- adapting said response to the message on the basis of the detected emotion. A method according to claim 1, wherein the emotion is detected in characteristic words of the user's message. Method according to one of Claims 1 and 2, in which the message is a voice message and the emotion is detected on the basis of at least one biometric parameter measured on the user during a pronunciation of the message by the user. A method according to claim 3, wherein the biometric parameter is at least one of voice intonation in the message spoken by the user, facial expression, posture, and blood flow of the user, while speaking the message . A method according to claim 4, wherein the voice message is acquired by at least one microphone and the biometric parameter is at least one voice intonation in the message spoken by the user. Method according to one of Claims 4 and 5, in which the voice message is acquired by at least one camera and the biometric parameter is at least one of a facial expression and a posture of the user at the time of pronouncing the message. Method according to claim 6, in which a score is assigned to each emotion detected and the adaptation of said response is a function of said score. Method according to one of the preceding claims, in which the computer agent is a voice assistant capable of generating an audio signal to be broadcast in response to the message transmitted and the adaptation of the response on the basis of the detected emotion comprises at least one element among a volume modulation of the audio signal, a choice of a synthetic voice among several possible voices, a voice intonation, a rate of phrasing in the audio signal. Method according to one of the preceding claims, in which the response based on the detected emotion comprises at least one group of pre-recorded words, added or not to the response depending on the detected emotion. Method according to one of the preceding claims, in which, the IT agent being configured to respond to the message sent according to a choice from among a plurality of items of a conversational menu, said choice is a function of both:
- keywords recognized in the message sent, and
- the user's emotion detected in the transmitted message. Device comprising a processing circuit configured for implementing the method according to one of the preceding claims. System for implementing the method according to one of Claims 1 to 11 comprising a computer agent connected via a network to at least one server, the server comprising a processing circuit configured to detect at least one emotion of the user in the transmitted message, and tailoring said response to the message further based on the detected emotion. System according to the preceding claim, in which the IT agent is a voice assistant. Computer program comprising instructions for implementing the method according to one of Claims 1 to 11, when said instructions are executed by a processor of a processing circuit. Non-transitory recording medium readable by a computer on which is recorded a program for implementing the method according to one of Claims 1 to 11 when this program is executed by a processor.