FR3111044A1 - Prediction of negative emotional situations in a connected habitat to promote services invoking positive emotions - Google Patents

Abstract

A method for predicting an emotional state of a user and suggesting a predefined digital service is proposed. The method comprises obtaining signals of interest measuring respectively a behavior of a user, a state of the environment and a context external to the environment. The method comprises a determination (P5) of time series of interest HUM_DB, IN_DB, OUT_DB, an obtaining (P6) of a MOD model structuring temporal and multimodal correlations between the values of the time series of interest and the emotional state as a function of time, an inference (P7), at a current instant, of an emotion score (E) corresponding to a prediction of the emotional state at a later instant, a triggering (P10) of a routine of animation of a man-machine interface to suggest a predefined digital service, and upon obtaining (P11) feedback from the user, triggering (P12) of the predefined digital service. A corresponding computer program, recording medium and processing circuit are also provided. Abstract Figure: Figure 2

Description

Prediction of negative emotional situations in a connected habitat to promote services invoking positive emotions

This disclosure falls within the field of data science.

More particularly, the present disclosure relates to methods of emotional state prediction and service triggering, as well as computer programs, computer-readable storage devices, and processing circuitry for carrying out such methods.

In the context of home support for fragile people (e.g. seniors, people with disabilities, isolated people, etc.), it is essential to be able to assess the autonomy of the occupants. Currently, tele-vigilance services implement this type of evaluation based on actimetry systems.

In this context, it appears necessary to also have a better understanding of the occupants' emotions, in order to provide a more robust measure of autonomy and adapted services. This measurement is one of the strong indicators for evaluating the good physical, social and moral health of individuals by the medical profession during televigilance services.

Indeed, emotions can reveal variations in autonomy. It is trivial to say that boredom, depression, anger or, on the contrary, joy, dynamism or spirit are indicative of a person's well-being and these characteristics influence their health and staying at home.

The analysis of the emotions of people at home is a current area of research.

Various proposed solutions are described below. These solutions differ in the nature of the data analyzed, including facial expressions, voice, gestures, data from home automation sensors or data from health sensors. Depending on the nature of the data analyzed, the underlying mathematical models also differ.

It was proposed in B. Nagarajan and V. R. M. Oruganti, "Group Emotion Recognition in Adverse Face Detection," 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), to implement group emotion recognition at from images of faces. The technical difficulties encountered for this recognition of emotions are linked to various heterogeneities in the images to be analyzed. Indeed, on some images, people's heads may be partially obscured. The pose of the body may differ from image to image or from person to person in the same image. Various interior and exterior settings, such as the brightness and orientation of the light sources as well as the definition of the images to be analyzed can also be heterogeneous. The architecture of the proposed algorithm relies on convolutional neural networks to extract the features of a face and on a Bayesian network to model the temporal correlations between these features. However, recognizing a person's emotions by focusing only on images of the person's face is not realistic for a consumer application. Indeed, even without considering the aforementioned technical difficulties, an additional limitation is the very low rate of acceptance of home cameras.

It has been proposed in J. Jia et al., "Inferring Emotions From Large-Scale Internet Voice Data," in IEEE Transactions on Multimedia, vol. 21, no. 7, p. 1853-1866, July 2019, a neural network model that uses acoustic information, textual information and three indicators: a temporal indicator, a descriptive indicator and a geosocial indicator. The objective is to provide contextual information giving meaning to acoustic information received by a voice assistant based on the duration of a request, the subject of the request and the location of the user. . A limitation of this approach is that determining emotion requires both that the person has a voice assistant and that they use it regularly. It would be desirable, in order to more robustly estimate the emotional state of a person, to exploit more sources of information. Another limitation of this approach is the nature of the exploited information extracted from the acoustic information, which boils down to the duration and the object of the requests. It would be desirable, always to estimate in a more robust way the emotional state of a person, to be able to extract and exploit, from acoustic information, other semantic data.

It was proposed in Z. Yang and S. S. Narayanan, "Modeling Dynamics of Expressive Body Gestures In Dyadic Interactions," in IEEE Transactions on Affective Computing, vol. 8, no. 3, p. 369-381, 1 July-Sept. 2017, a statistical framework for modeling the dynamics of bodily gestures in dyadic interactions. Body gestures are a non-verbal channel of expression, important in affective communication. They convey the attitudes and emotions that unfold dynamically during an interpersonal interaction. First, the construction of a generic model based on Gaussian mixing models represents subject-independent gestural variability. Then, each gesture sequence is described as a concatenation of semantic gestures that are derived from a structure of a hidden Markov model built in parallel. A disadvantage of this approach is that some emotions are not perceptible through body movement, for example disgust. Moreover, certain gestures may be common to several types, or classes, of emotions, hence the possibility of confusion and incorrect determination. For example, a person's gestures may be similar when the person is excited or happy.

It was proposed in D. Fedotov, Y. Matsuda and W. Minker, "From Smart to Personal Environment: Integrating Emotion Recognition into Smart Houses," 2019 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), the combination of a dialogue and emotion recognition system in a connected home. The authors believe that these two systems are complementary and that they make it possible to constantly monitor the state of the user, interact with him in real time and in reaction to mood changes. The objective is to adapt the dialogue system to the perceived mood, confused in these works with the emotion of the person. However, in a home support service for the well-being and health assessment of occupants, a person's mood and emotion must be distinguished.

It has been proposed in K. Lin, F. Xia, W. Wang, D. Tian and J. Song, “System Design for Big Data Application in Emotion-Aware Healthcare,” in IEEE Access, vol. 4, p. 6901-6909, 201, a system improving health services taking into account the emotional factor. Indeed, existing health systems focus only on logical reasoning but ignore the user's emotion factor, yet considered an important factor of impact on human health. The sensors used are bracelets, watches, connected glasses, etc. In this approach, the sensors are worn by the users, which is an obstacle for medical monitoring. Indeed, the acceptance of worn, intrusive sensors is generally low. In addition, their use is unsuitable, for example for periods of care, toilets, etc.

There is therefore a need to be able to remedy the drawbacks mentioned above and to propose a robust prediction of so-called negative emotional situations of people in a connected habitat in order to encourage said people to have so-called more positive emotions. It is also desirable that the sources used be multimodal.

Summary

This disclosure improves the situation.

A method is proposed for predicting an emotional state of a user and for suggesting a predefined digital service to the user, the method being implemented by a processing circuit connected to a plurality of communicating equipment installed in a user-occupied environment, the method comprising:
- obtaining a plurality of signals of interest acquired by the communicating equipment, the plurality of signals of interest comprising at least one signal of interest of a first type measuring a behavior of the user, a signal of interest of a second type measuring a state of the environment and a signal of interest of a third type indicative of a context external to the environment,
- for each signal of interest obtained, a determination, on the basis of said signal obtained, of a time series of interest formed of a sequence of values as a function of time,
- obtaining a model structuring the temporal and multimodal correlations between the values of the time series of interest and the emotional state of the user as a function of time,
- on the basis of the model obtained and the time series of interest obtained, an inference, at a current instant, of an emotion score corresponding to a prediction of an emotional state of the user at a later instant,
- on the basis of the inferred emotion score, triggering an animation routine of a man-machine interface to suggest to the user at least one predefined digital service associated with the inferred emotion score, and
- upon obtaining feedback from the user, received by the man-machine interface, triggering of the predefined digital service via a control signal transmitted to at least one device of the plurality of communicating devices.

The proposed method aims to respond to the aforementioned drawbacks and to propose a robust prediction of “negative” emotional situations in a connected habitat to promote services that encourage people to have “more positive” emotions. An example of emotion classification is described in Parrott, W. (2001), Emotions in Social Psychology, Key Readings in Social Psychology, Philadelphia: Psychology Press. In this example, fifteen classes of emotions are defined: neutral (1), disgusted (2), panicked (3), anxious (4), angry (5), cold anger (6), desperate (7), sad (8), enthusiastic (9), happy (10), interested (11), annoyed (12), ashamed (13), proud (14) and contemptuous (15). In an example implementation using the emotion classes defined above, the emotion classes disgusted, panicked, anxious, angry, desperate, sad, bored and ashamed can be considered negative and the emotion classes enthusiastic , happy and interested can be considered positive.

• des informations propres aux comportements des individus, par exemple liées aux expressions faciales, à la voix, aux gestes, etc.,
• des informations de la maison connectée par exemple liées à la température, à l’humidité, à la consommation d’énergie, etc., et
• des informations extérieures à la maison, par exemple liées aux prévisions météorologiques, aux actualités, etc.

The proposed process is innovative on the sources used which are multimodal from three areas, namely:

• information specific to the behavior of individuals, for example related to facial expressions, voice, gestures, etc.,
• information from the connected home, for example relating to temperature, humidity, energy consumption, etc., and
• information outside the home, for example related to weather forecasts, news, etc.

The proposed method is, moreover, original in its end-to-end design. Indeed, the implementation of the method makes it possible to process raw data received from communicating equipment, to infer an emotion score indicating a predicted emotional situation, and to promote an adapted service in order to influence this predicted emotional situation. A recognition of “negative” emotions is a situational reference to promote services invoking “positive” emotions.

The characteristics exposed in the following paragraphs can, optionally, be implemented. They can be implemented independently of each other or in combination with each other:

In one example, the method further comprises a collection of signals over time from the plurality of communicating equipment and a selection of the signals of interest from among the signals collected.

Indeed, the method is applicable in many possible configurations of domestic installations, in particular no restriction is imposed on the nature of the communicating equipment present in the domestic installation and the sensors equipping them.

The relative relevance of the signals collected for the prediction of the emotional situation of the inhabitants can for example be evaluated during a calibration phase of the model, so that it is possible, on the basis of this evaluation, to select only the signals obtained having a proven correlation with a subsequent emotional situation of the user. Thus, the use of the resources of the processing circuit is optimized for the prediction of the emotional situation of the user.

In one example, the method further includes temporal segmentation of the signals of interest, and the time series of interest are determined based on the segmented signals.

Temporal segmentation makes it possible to have chronologically synchronized time series of interest. For example, a temporal segmentation in time intervals of 30 seconds has the effect that all the time series of interest comprise, for each time interval of 30 seconds, an associated value of quantity of interest. This temporal synchronization makes it possible, for example, to establish multimodal correlations between the values of different quantities of interest during the same time interval, or during consecutive time intervals. Thus, it is possible to structure in the model, for example, a correlation between the speech rate of the user during a given time interval and his gestures during a following time interval.

In one example, the method further includes processing the signals of interest, such as low-pass filtering, normalization, or resampling, and the time series of interest are determined based on the processed signals.

This processing makes it possible, in general, to shape the signals of interest with a view to inferring the emotional situation of the user. For example, low-pass filtering makes it possible to denoise the information, normalization makes it possible to standardize the data and data resampling makes it possible to synchronize the sources.

In one example, the values of the time series of interest are obtained by classifying the signals of interest by modality according to a discrete scale of values associated with said signal.

This classification by modality makes it possible, by synthesizing the information contained in the signals of interest, to highlight recurring sub-sequences within each time series of interest. Each recurrent sub-sequence thus highlighted can serve as a basis for a temporal correlation. Thus, the classification by modality of a signal of interest favors the detection of temporal correlations relative to this signal of interest.

Similarly, the classification by modality of several signals of interest has the effect of favoring the detection of multimodal correlations between these signals of interest.

In one example, the temporal and multimodal correlations of the model are obtained by training based on:
- for each time series of interest, a reference subsection formed of a portion of the sequence of values, the portion covering a reference time period, and
- a knowledge base listing, for a plurality of instants of the reference time period, a reference emotion score indicative of the emotional state of the user at said instant.

Thus, after this learning, the time series of interest continue to be acquired in the same way as during the reference time period, using the same communicating equipment.

A subsequent user emotion score can then be inferred based on the correlations established, under the same conditions, between the emotion scores of the knowledge base and the reference subsections of the time series d 'interest.

In one example, the model includes a given temporal correlation between several values of a given time series of interest, and the inference of the emotion score is based on the given temporal correlation.

A particular subsequence can be identified within a time series of interest. For example, a facial expression indicating boredom may be detected repeatedly over several consecutive time points. It is thus possible to infer that at a later moment, this facial expression will be maintained. On this basis, it is thus possible to infer a later emotional state “bored”.

In one example, the time series of interest are grouped into a plurality of informational domains, the model comprises, for at least one given informational domain, a given multimodal correlation between a plurality of time series of interest of said given informational domain, and the inference of the emotion score is based on the given multimodal correlation.

For example, an emotion score can be inferred based on a multimodal correlation with news and weather information. For example, sunny weather, associated with a positively perceived current event for a given person, can generally be correlated with a positive emotional state of the given person.

In one example, the method comprises obtaining a latent time series formed of a sequence of values as a function of time, the model obtained comprises a given temporal correlation between several values of the latent time series, the inference of the score d emotion includes determining, at the current time, the value of the latent time series at the current time based on the given time correlation, and inferring the emotion score is based on the determined value of the latent time series.

A latent time series makes it possible to represent an unmeasurable quantity, for example in the absence of an adequate sensor, but nevertheless correlated both with one or more measurable quantities and with the emotional state of the person. This latent time series can be manipulated as such by the model. Thus it is possible to structure, using the model, a temporal correlation between different values of any time series, whether it relates to a measurable quantity or a latent quantity.

In one example, the method comprises, on the basis of the model and the time series obtained, a determination, at a current instant, of a level of confidence associated with the inferred emotion score, the method further comprises a comparison of the level confidence determined with a threshold, and the triggering of the routine is based on the result of the comparison.

Thus, for a given inferred emotion score, it is possible to recommend a predefined service to a user, for example, only if the multimodal and temporal correlations have made it possible to infer the emotion score with a sufficient level of confidence. . This avoids suggesting services to a user based on erroneous predictions of his subsequent emotional state.

In one example, a plurality of suggested digital services are defined by respective links stored in a correspondence table in association with respective emotion score thresholds and, if a threshold is exceeded by the emotion score of the user, at least one link of a corresponding digital service is read to suggest said corresponding digital service to the user via the man-machine interface.

Thus, a service is only recommended to the user if the intensity of a predicted emotion is sufficiently high. This has the effect of reserving the suggestion of services to the user at times when the execution of these services would be most useful to him.

In one example, the respective links are stored in the correspondence table according to an order of preference, in the event of a threshold being exceeded by the user's emotion score, a plurality of corresponding digital service links are read according to the order of preference to suggest said corresponding digital services to the user via the man-machine interface, the user's return to the issued invitation comprises a selection of a digital service among said corresponding digital services, and the the order of preference is updated based on said selection.

Thus, a selection of services is offered to the user and the services usually preferred from this selection will be put forward more, or more frequently.

A computer program is also proposed comprising instructions for implementing the above method when this program is executed by a processor.

There is also proposed a non-transitory recording medium readable by a computer on which is recorded a program for the implementation of the method above when this program is executed by a processor.

A processing circuit is also proposed comprising a processor connected to the above non-transitory recording medium and to a communication interface.

Other characteristics, details and advantages will appear on reading the detailed description below, and on analyzing the appended drawings, in which:

Fig. 1

schematically illustrates the structure of a processing circuit, in a particular embodiment, for the implementation of the proposed determination method.

Fig. 2

represents a flowchart of a general algorithm of a computer program, in a particular embodiment, for the implementation of the proposed determination method.

Fig. 3

represents, in the form of a dynamic Bayesian network, an emotion prediction model that can be derived, in a particular embodiment, from the proposed determination method.

The drawings and the description below contain, for the most part, certain elements. They may therefore not only serve to better understand this disclosure, but also contribute to its definition, where applicable.

Reference is now made to Figure 1, which shows an example of a processing circuit integrated into communicating equipment, installed in a space occupied by a person, such as a dwelling.

The processing circuit comprises at least one processor PROC 100 connected to at least one memory MEM 200 and to at least one communication interface COM 300.

A communicating device is characterized by its ability to receive and transmit information via the communication interface. For example, in a domestic installation, various communicating devices can be configured to communicate with each other through a local communication network and/or an extended communication network. Communication can be wired and/or by radio waves.

Some communicating equipment can also be equipped with sensors to collect data.

Examples of communicating equipment include smartphones, tablets, computers, connected televisions, network gateways, connected speakers, watches, connected bracelets or headbands, household appliances, weather stations, electricity, gas or water meters, connected lighting, connected cameras, connected thermostats, etc.

Reference is now made to Figure 2, which shows a flowchart of a general algorithm of a computer program stored on the memory MEM (200) of the processing circuit shown in Figure 1.

This processing circuit is configured to obtain, or receive SIGN OBT (P1), continuously, a stream of signals. Each signal carries at least one type of digital information.

If the processing circuit is integrated into communicating equipment provided with one or more sensors, then the reception of signals can be carried out by acquisition using the said sensor(s).

For example, if the communicating equipment includes a camera, then it is possible to collect a video recording of the person, as a signal carrying several types of digital information. This video recording can be analyzed by the processing circuit in order to identify said digital information. It is thus possible to collect different facial characteristics or facial expressions, different gestures or postures, movements of the person, ambient light, etc.

The reception of signals can be carried out in parallel via the communication interface of the processing circuit, receiver. Thus, it is possible to receive signals transmitted by other communicating equipment, transmitters, of the domestic installation.

For example, a connected speaker, as transmitting communicating equipment, can collect an audio recording as a signal carrying digital information. The processing of this audio recording directly by the connected speaker can make it possible to extract, for example, an ambient sound level, a person's speech rate, an estimate of the person's position, etc. It is possible to transmit each type of digital information extracted separately. In other words, the connected speaker can transmit a plurality of signals. Each signal carries a type of digital information to the receiving communicating equipment.

Alternatively, the connected speaker can directly transmit the audio recording to the receiving communicating equipment, without prior processing. In this case, the connected speaker transmits a signal, audio, carrying several types of digital information.

For example, a weather station, as transmitting communicating equipment, can be fitted with sensors to continuously collect various home automation digital information. Examples include ambient temperature in different rooms, humidity, atmospheric pressure, carbon dioxide levels, etc. This digital home automation information can be transmitted in the form of time series, therefore in the form of sequences of timestamped values.

In general, depending on the nature and configuration of the various pre-existing communicating equipment within a particular domestic installation, it is possible to obtain a wide variety of digital information. In principle, the digital information thus obtained is heterogeneous both in nature and in format.

Three categories of information obtained are defined below.

A first category of information concerns information describing the behavior of the person. For example, it can be features of the face, voice, gestures, etc. This type of information can come, for example, from communicating equipment comprising audio and/or video sensors. Subcategories can be defined. For example, a first subcategory may encompass facial features. A second sub-category can group the characteristics of the voice.

A second category of information concerns information related to the space occupied by the person. For example, it may be door opening indicators, water, gas or electricity consumption readings, temperature, brightness or sound level measurements, etc. This type of information can be acquired, for example, by appropriate sensors present on various connected home equipment.

A third category of information concerns one or more information related to a context outside the home. For example, it can be weather information, news feeds, air pollution indicators, etc. This type of information can be downloaded by a communicating device, for example from a remote server.

In the context of disclosure, it is considered that at least one digital information is obtained for each category of information defined above.

Other digital information not belonging to any of the aforementioned categories, or belonging to several of the aforementioned categories, can be obtained.

In parallel with the collection of data by the reception of the flow of signals, it is possible for the processing circuit to operate, for each category or sub-category of information, a selection SIGN SELEC (S2) of signals of interest treat.

For example, it is possible to process only a selection of 50 home automation information from among 200 home automation information collected.

Similarly, it is possible to process only a selection of 10 facial characteristics out of 50 collected.

The advantage of making such a selection is to minimize the exchange of data between communicating equipment by ignoring the characteristics that have no influence on the emotional state of the person.

For example, it is possible to track a person's emotional state over an extended period of weeks or months. This monitoring can be implemented by repeated iterations of the proposed method. This follow-up can highlight, by cross-checking, that certain information obtained is decorrelated from the emotional state of the said person. Based on this finding, this information may be disregarded in the future.

Moreover, a sociological study can highlight that certain information obtained can be ignored a priori.

In order to homogenize the signals with a view to their subsequent analysis, it is possible for the processing circuit to define SIGN WIND (S3) an analysis window, or a measurement step.

For example, home automation information such as an ambient sound level can be recorded continuously, be segmented into consecutive intervals of a predefined duration, and be represented by a time series corresponding to a succession of average ambient sound levels during each elapsed interval.

For example, meteorological information such as an outside ambient temperature can be recorded discreetly so as to form a time series corresponding to a succession of instantaneous values.

Thus, it is possible to represent each type of information obtained as a time series formed of a sequence of timestamped values. In other words, it is possible to associate with each measurement a corresponding instant i and with each time series comprising n measurements a sequence of corresponding consecutive instants (0, 1, …, i-1, i, i+1 , … n-1, n), each instant corresponding to a predefined time interval.

In an example implementation, it is possible to segment the data every minute without overlapping.

The processing circuit can implement various SIGN PROC (P4) specific processing of the collected and segmented signals of interest.

The processing is, in a non-exhaustive way, low-pass filtering to denoise the information, normalization to standardize the data, resampling of the data to synchronize the sources, etc.

The signals thus obtained can be classified SIGN CLASS (P5) by modality. The term modality refers to the different values that can be taken with respect to a type of information.

In an example implementation, features of people's faces can be ranked on an emotional intensity scale of 1 to 5, to designate the intensity of emotions such as neutral, happy, disgusted, angry, or annoyed, respectively. . For voices, the rate of speech can for example be classified on a scale of 1 to 4, indicating respectively a slow, neutral, fast or very fast rate. The intensity of the gestures can be classified on a scale of 1 to 3, with 1 denoting the person at rest, 2 denoting gestures of normal intensity and 3 denoting energetic gestures.

Similarly, the information characterizing the house can be categorized on a scale of values. For example, it is possible using a contactor to determine at any time whether a door is open or closed. The time series resulting from the classification by modality can for example be an alternation of values equal to 1 when the door is open and to 2 when the door is closed. The consumption of water, gas or electricity can be compared to predefined thresholds and be classified by modality according to a scale for example of 1 to 4, corresponding respectively to very low, low, high or very high consumption. The same applies to ambient temperature, brightness and noise.

Contextual information from outside the home can also be classified. An example of a scale from 1 to 4 can refer to sunny, rainy, snowy or cloudy weather respectively. It can also be reported on the major news of the day using such scales. Thus, it is possible to assign a modality of 1 if the main news is political, 2 if it is sports news, etc. It is also possible to consider each type of news separately and to assign a modality to each type of news (politics, sports, health, leisure, people, etc.) according to the presence or absence of this type of news. topicality, an assessment of their interest in the person and/or of their reassuring or disturbing nature.

Thus, databases HUM_DB, IN_DB, OUT_DB are made up to respectively store the history of pre-classified information for the three categories of information, or informational domains, describing the state of the inhabitants (e.g. faces, voices, gestures, etc.), from inside the home (e.g. temperature, humidity, energy consumption, etc.) and from outside the home (e.g. news/news, weather, pollution index, etc. .).

A pre-learned MOD model is loaded MOD LD (P6) by the processing circuit. The MOD model structures temporal and multimodal correlations between the different sources of information.

Reference is now made to an example implementation, where the MOD model is a dynamic Bayesian network. The inventors have observed, by experimental validation, the robustness of the prediction of an emotional state of a person on the basis of such a model.

Of course, alternatively, any type of probabilistic model, establishing temporal and multimodal correlations, is applicable. However, the results obtained in this way do not necessarily have equivalent robustness.

The MOD model can be determined beforehand, or learned, during a calibration phase.

This determination is based on digital information previously collected over a predefined period, known as the reference period.

For example, it is possible to rely on:
- a subsection of the HUM_DB, IN_DB, OUT_DB databases, corresponding to data collected over a period of one week, and
- a knowledge base EM_DB labeling the emotions of the person during this same period.

In an example implementation, fifteen emotion classes can be used, either neutral (1), disgusted (2), panicked (3), anxious (4), angry (5), cold anger (6), desperate (7), sad (8), enthusiastic (9), happy (10), interested (11), bored (12), ashamed (13), proud (14) and contemptuous (15).

The labels, designating the classes of emotions, can be provided by the person himself, for example in the form of a logbook. Alternatively, the labels can be provided by a trusted third party, i.e. the family, a loved one, the medical profession, etc.

The labels of the emotions thus make it possible to learn temporal correlations and multimodal correlations between the databases HUM_DB, IN_DB, OUT_DB and the labels of emotions EM_DB.

These correlations are an example of a custom MOD model provided in an initial production release. The MOD model is of course brought to evolve over time according to the evolution of the accumulated digital data.

Different MOD models, corresponding to different people, in different dwellings, with different domestic installations, can be brought together in order to constitute typical, generic models, which can be applicable by default to a new person in a new dwelling having a domestic installation comprising a set of given communicating devices. In this way, it is possible to dispense with, in certain examples of implementations, a calibration phase.

Reference is now made to Figure 3, representing an example implementation of the MOD model as a dynamic Bayesian network.

The emotional state E _i of a person at a given instant i potentially depends on the recent history of the emotional states of the person. In other words, there is a temporal correlation between the emotional state E of the person, on the one hand, at a future instant i+1 and, on the other hand, at the current instant i, at previous instant i-1, etc. This temporal correlation is represented in Figure 3 by the arrows connecting the emotional states E _i-1 , E _i and E _i+1 .

Moreover, there are multimodal correlations between the emotional state E _i+1 of the person at a future instant i+1, on the one hand, and different variables at the current instant i, on the other hand. These multimodal correlations are represented in FIG. 3 by the arrows connecting L _i , F _i , V _i , G _i , T _i , H _i , C _i , N _i and M _i with E _i+1 .

Mention may in particular be made of variables of the first informational domain, designating states of the person, such as characteristics of the face, denoted F, of the voice, denoted V or of gestures, denoted G.

We can also cite variables from the second informational domain, designating home automation information, such as indoor temperature, denoted T, indoor humidity, denoted H, or energy consumption, denoted C.

We can also cite variables from the third informational domain, designating contextual information outside the home, such as news information, denoted N, or weather, denoted M.

We can also cite latent (non-observable) variables, denoted L. Such variables represent non-measurable quantities, for example due to a lack of sensors to measure them. In this case, a quantity L is modeled by multimodal and/or temporal correlation with other data obtained. For example, indoor brightness can be correlated with natural lighting, i.e. sunshine, and artificial lighting. Thus, the luminosity can be taken into account by the MOD model even in the absence of a luminosity sensor. Indeed, assuming the above correlations, the luminosity at a given instant can be modeled based on meteorological information, on communicating lighting operating data and/or on modeled luminosity values at instants previous ones.

It will also be noted that the information produced by the latent variables L, the inhabitants, the residence and the external context are dependent in time, that is to say that the instants i-1 and i, in particular, influence the instant i+1. In other words, there are internal temporal correlations to each of these variables.

It will also be noted that the voice and the gesture influence the characteristics of the face. These multimodal correlations are represented in Figure 3 by the arrows connecting V _i and G _i with F _i+1 . Similarly, the face and the voice influence the gestures and that the face and the gestures influence the voice.

Similarly, information from the house is correlated (for example: temperature and humidity influence energy consumption, etc.). Also, information from the external context can be linked (eg: weather and news, etc.).

In an example implementation, it is possible to exclude a priori certain correlations.

For example, it is possible to provide that, in the MOD model, no correlation is structured between different informational domains. No arrow in Figure 3 connects the variables F, V and G with the variables T, H and C for example. We then consider that these informational domains are independent. For example, it is possible to consider that the information produced by the inhabitants is not directly influenced either by home automation information or by external context information, and vice versa.

Similarly, it is possible to provide for a latent variable to be determined only by temporal correlation and not by multimodal correlation. Thus, this latent variable is independent of the information measured.

The correlations that can be excluded a priori only concern the interactions between different obtained and/or latent variables, and not the correlations between an obtained or latent variable, on the one hand, and an emotional state, on the other hand. Indeed, the emotional states E are potentially correlated with the variables obtained for each informational domain and with each latent variable L.

The processing circuit infers, or predicts EMOT PRED (P7), an emotional state E of a person at a later instant i+1 based on the temporal and multimodal correlations structured by the MOD model.

These correlations are applied to the recent history of the information obtained, i.e. to the information associated with the current instant i and to the information associated with one or more previous instants i-1, i-2, etc.

• celles issues des bases de données HUM_DB, IN_DB, OUT_DB, et
• les valeurs des variables latentes L exploitées par le modèle MOD.

Such information obtained includes:

• those from the HUM_DB, IN_DB, OUT_DB databases, and
• the values of the latent variables L exploited by the MOD model.

Said information obtained also includes the current emotional state of the person, as inferred by a previous iteration.

Thus, the proposed process makes it possible, at any time, both to determine the current emotional state of a person and to predict its evolution over time.

In certain exemplary embodiments, a confidence measure Z can be associated CONF COMP (S8), by the processing circuit, with the emotional situation of the person at time i+1 inferred by the MOD model.

For example, a maximum likelihood value that the emotional situation is in state 3 (i.e. class representing “panicked”) is calculated. The higher this value, the greater the confidence in the prediction.

In certain exemplary embodiments, it may be decided to take into account TEST DIFF (P9) both the inferred emotional situation and the confidence measure Z thus determined in order to offer a service.

For example, a double condition may be provided, such as:
- if the confidence measure Z is greater than a predefined threshold A (on a scale of 0 to 1, the threshold can be set at 0.8 in an example implementation), and
- if the person's predicted emotional situation is considered "negative",
then an adapted service can be offered to the person.

Otherwise, provision may be made to offer no service, to stop the current iteration of the method at the current instant i, and to trigger a subsequent iteration of the method at the subsequent instant i+1.

In an example implementation, the "negative" classes can be chosen as follows: disgusted (2), panicked (3), anxious (4), angry (5), cold anger (6), desperate (7), sad (8), bored (12) and ashamed (13).

The emotional situation at the lower instant being inferred, a service recommendation, adapted to the prediction, can be triggered by the processing circuit.

Thus, an animation routine of a man-machine interface of any communicating equipment of the domestic installation is triggered to present SERV PRES (P10) the recommended service to the person.

In an example implementation, the service recommendation is executed on the best broadcast channel for the person, such as the living room TV, a communicating speaker, a smartphone, etc.

The most suitable communicating device for this purpose can be selected, for example, on the basis of the functionalities offered by the various communicating devices in the installation.

Another criterion can be an estimate of the distance, at the current time, between the various communicating devices and the person, making it possible to choose, among the various communicating devices of the installation capable of triggering the services offered, the one being the closest. of the person at the current time.

The animation routine makes it possible to suggest to the person at least one predefined numerical service associated with the inferred emotion score.

For example, if the predicted situation is "freaked out", the associated services in a PREF preference model can be in order of preference "a comedy video recommendation", "an incentive to call a friend", or "a increase in home comfort through automatic adjustment of brightness and heating”.

Other examples of services can be described for each "negative" emotional situation according to people's preferences and habits.

These preferences can be defined by the people themselves or by a trusted third party, such as the family, a loved one, or the medical profession.

These preferences can be set, rearranged or updated automatically by learning the services used in the past in correlation with the predicted emotions.

The goal is always to evoke or maintain a positive emotional situation through appropriate services.

The processing circuit can then evaluate TEST AGR (P11) the return of the person to the request for the adapted service. Indeed, if a list of possible services is suggested, each service in the list can either be activated, refused, or ignored by the person.

If a service is activated by the person, then it is possible to execute the service SERV EXEC (P12) proposed.

It is also possible, if necessary, to update the PREF UPD (P13) model of the person's PREF preferences according to the predicted emotional situation and the accepted service.

It is also possible to update the MOD model according to the return of the person to the request of the adapted service. Indeed, it can be provided that the person can signal an incorrect prediction of his subsequent emotional state, or can signal when the prediction is correct, in order to improve the predictions during subsequent iterations of the method.

Finally, when a proposed service is activated and executed, or when a proposed service is refused, or even when no proposed service is activated at the end of a predefined period of time, the iteration of the method to the current instant ends. The processing circuit can then initiate a new iteration of the method for a later time.

Claims (15)

Method for predicting an emotional state of a user and suggesting a predefined digital service to the user, the method being implemented by a processing circuit connected to a plurality of communicating equipment installed in an occupied environment by the user, the method comprising:
- obtaining a plurality of signals of interest acquired by the communicating equipment, the plurality of signals of interest comprising at least one signal of interest of a first type measuring a behavior of the user, a signal of interest of a second type measuring a state of the environment and a signal of interest of a third type indicative of a context external to the environment,
- for each signal of interest obtained, a determination (P5), on the basis of said signal obtained, of a time series of interest (HUM_DB, IN_DB, OUT_DB) formed of a sequence of values as a function of time,
- obtaining (P6) a MOD model structuring temporal and multimodal correlations between the values of the time series of interest and the emotional state of the user as a function of time,
- on the basis of the model obtained and the time series of interest obtained, an inference (P7), at a current instant, of an emotion score (E) corresponding to a prediction of an emotional state of the user at a later time,
- on the basis of the inferred emotion score (E), triggering (P10) of an animation routine of a man-machine interface to suggest to the user at least one predefined digital service associated with the score of emotion (E) inferred, and
- upon obtaining (P11) a return from the user, received by the man-machine interface, a triggering (P12) of the predefined digital service via a control signal transmitted to at least one device of the plurality of communicating equipment. Method according to the preceding claim, further comprising a collection (P1) of signals over time from the plurality of communicating equipment and a selection (P2) of the signals of interest from among the signals collected. Method according to one of the preceding claims, further comprising temporal segmentation (P3) of the signals of interest, and in which the time series of interest are determined on the basis of the segmented signals. Method according to one of the preceding claims, further comprising processing (P4) of the signals of interest, such as low-pass filtering, normalization, or resampling, and in which the time series of interest are determined based on the processed signals. Method according to one of the preceding claims, in which the values of the time series of interest are obtained by classifying the signals of interest by modality according to a discrete scale of values associated with said signal. Method according to one of the preceding claims, in which the temporal and multimodal correlations of the MOD model are obtained by learning on the basis of:
- for each time series of interest, a reference subsection formed of a portion of the sequence of values, the portion covering a reference time period, and
- a knowledge base listing, for a plurality of instants of the reference time period, a reference emotion score indicative of the emotional state of the user at said instant. Method according to one of the preceding claims, in which:
- the model includes a given temporal correlation between several values of a given time series of interest, and
- the inference of the emotion score is based on the given temporal correlation. Method according to one of the preceding claims, in which:
- the time series of interest are grouped into a plurality of informational domains,
- the model comprises, for at least one given informational domain, a given multimodal correlation between a plurality of time series of interest of said given informational domain, and
- the inference of the emotion score is based on the given multimodal correlation. Method according to one of the preceding claims, in which:
- the method comprises obtaining a latent time series formed of a sequence of values as a function of time,
- the model obtained includes a given temporal correlation between several values of the latent time series, and
- the inference (P7) of the emotion score comprises a determination, at the current instant, of the value of the latent time series at the current instant on the basis of the given temporal correlation, and the inference (P7 ) of the emotion score is based on the determined value of the latent time series. Method according to one of the preceding claims,
- the method comprising, on the basis of the model and the time series obtained, a determination (P8), at a current instant, of a confidence level (Z) associated with the inferred emotion score,
- the method comprising a comparison (P9) of the determined confidence level (Z) with a threshold (A), and
- the triggering (P10) of the routine being based on the result of the comparison. Method according to one of the preceding claims, in which a plurality of suggested digital services are defined by respective links stored in a correspondence table (PREF) in association with respective emotion score thresholds and, in the event of exceeding threshold by the user's emotion score, at least one link of a corresponding digital service is read to suggest said corresponding digital service to the user via the man-machine interface. Process according to the preceding claim, in which:
- the respective links are stored in the correspondence table according to an order of preference,
- in case of exceeding a threshold by the emotion score of the user, a plurality of links of corresponding digital services are read according to the order of preference to suggest said corresponding digital services to the user via the interface man-machine,
- the user's return to the invitation sent comprises a selection of a digital service from among said corresponding digital services, and
- the order of preference is updated on the basis of said selection. Computer program comprising instructions for implementing the method according to one of Claims 1 to 11 when this program is executed by a processor. Non-transitory recording medium readable by a computer on which is recorded a program for implementing the method according to one of Claims 1 to 11 when this program is executed by a processor. Processing circuit comprising a processor PROC (100) connected to a communication interface COM (300) and to a non-transitory recording medium MEM (200) according to claim 14.