KR20230110999A - Apparatus and method for emotion recognition based on physiological signal and lifelog - Google Patents

Abstract

The present invention relates to an emotion recognition device based on a biological signal and a life log and a method thereof. According to the present invention, the emotion recognition device comprises: a biological signal and life log data collector that collects a biological signal and life log data and extracts feature values from the biological signal and the life log data to generate learning data and inference data; an emotion recognition model generator that generates an emotion recognition model through machine learning based on the learning data; and an emotion recognition and cause analyzer that recognizes an emotion through inference by inputting the inference data into an emotion recognition model.

Description

Emotion recognition device and method based on biosignal and lifelog

The present invention relates to an apparatus and method for recognizing an individual's emotion or mood.

BACKGROUND OF THE INVENTION Recently, as society develops and becomes more sophisticated, more and more people feel emotional and psychological anxiety in social relationships. As a result, it is expected that the demand for technology that helps to read individual emotions or moods and improve them when negative emotions appear will increase.

However, many people tend to ignore these emotional and psychological insecurities or suppress them internally without expressing their emotional experiences under stress. In such a situation where people have difficulties in expressing emotions socially and culturally, it makes it difficult to recognize emotions through facial expressions or voices that are revealed to the outside. Therefore, a method that has been actively researched in the field of emotion and emotion recognition in recent years is to recognize emotions through various bio-signals. In physiological psychological research, it is known that there is a strong correlation between human emotions and biological responses, and biosignals can be obtained relatively easily with sensors, etc., and emotional recognition using them is less sensitive to social and cultural differences, so it has recently received a lot of attention. In particular, as the performance of wearable devices such as smart watches and fitness bands has improved along with the development of IT technology, it has become possible to acquire biosignals easily and accurately, and as recognition performance has improved with the development of AI technology, the possibility of using technology for recognizing user's emotions based on biosignals is increasing.

However, emotion recognition technology using bio-signals has dealt mainly with emotions lasting for a short time (10 seconds to 10 minutes). In general, when people receive emotional stimuli, changes in biosignals appear by the autonomic nervous system. For example, when people are embarrassed or nervous, their heart beats faster, sweat all over their body, or their body temperature rises. These bio-signals are composed of sympathetic and parasympathetic nerves and are generated by the autonomic nervous system, which regulates the heartbeat or the body's exocrine glands. The autonomic nervous system regulates the balance of the sympathetic and parasympathetic nerves so that the body can maintain a stable situation according to various emotional stimuli. The sympathetic nervous system mainly works in a state of excitement/tension, and the parasympathetic nervous system mainly responds in a state of mental stability. In addition, these bio-signal responses generally do not appear for a long time and end after a relatively short period of time. For example, when people are momentarily emotionally stimulated, changes in biosignals such as heartbeat, sweat, and body temperature appear for a short period of time and then return to a stable state.

On the other hand, in order to identify emotional and psychological anxiety and respond appropriately to it, it is necessary to recognize emotions for a relatively long time (1 hour to 1 day).

An object of the present invention is to provide a device and method for recognizing a mood, which is an emotion that appears for a long time, based on lifelog data such as behavior, situation, sleep, and the like, as well as biological signals highly correlated with the mood. To this end, the present invention first collects bio-signals and lifelog data from the user, and extracts various features from them. At this time, features are extracted by combining biosignal characteristics for each behavior and situation so that emotions of a relatively long time can be effectively recognized. With the features extracted in this way, an emotion (mood) recognition model is created through machine learning. Thereafter, the apparatus and method according to the present invention can recognize a mood based on the emotion recognition model, and in particular, can recognize negative emotions that cause emotional and psychological anxiety.

In addition, the apparatus and method according to the present invention, when a negative mood is recognized, by analyzing the behavior or situation that causes it based on a personal emotion model, to find a lifestyle or behavior pattern that causes negative emotion, and to provide the analysis result to the user.

The object of the present invention is not limited to the object mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the description below.

An emotion recognition device according to an embodiment of the present invention for achieving the above object is a biosignal and lifelog data collector that collects biosignals and lifelog data, extracts feature values from the biosignals and lifelog data, and generates learning data and inference data; an emotion recognition model generator generating an emotion recognition model through machine learning based on the learning data; and an emotion recognition and cause analyzer for recognizing emotion through reasoning by inputting the reasoning data to the emotion recognition model.

In one embodiment of the present invention, the bio-signal and lifelog data collector may include: a bio-signal measuring unit that collects the bio-signal through measurement; a lifelog data collection unit that collects the lifelog data; a data logger that collects the bio-signals and the lifelog data in synchronization with time; a signal pre-processing unit dividing the synchronized biosignal according to at least one criterion of situation and behavior; and a feature extraction unit extracting the feature value from the divided bio-signals and the lifelog data.

In one embodiment of the present invention, the bio-signal and lifelog data collector may include a normalization unit normalizing the characteristic values.

In one embodiment of the present invention, the signal pre-processing unit may divide the synchronized bio-signal after removing noise included in the synchronized bio-signal.

In one embodiment of the present invention, the emotion recognition and cause analyzer may include an emotion recognition unit inputting the reasoning data to the emotion recognition model and recognizing emotion through reasoning; A negative emotion detecting unit that determines whether the recognized emotion is a negative emotion; and an emotion cause analyzer that analyzes a cause of the recognized emotion based on the reasoning data when the recognized emotion is a negative emotion.

In one embodiment of the present invention, the emotion cause analysis unit may analyze the cause of the recognized emotion through a correlation analysis between variables constituting the reasoning data and the recognized emotion.

In one embodiment of the present invention, the emotion recognition device may further include a lifestyle coach. In this case, the emotion recognition and cause analyzer analyzes the cause of the recognized emotion based on the reasoning data, and the lifestyle coaching device is associated with the recognized emotion based on the analysis result of the emotion recognition and cause analyzer. It may provide information about at least one of lifestyle habits and behavioral patterns.

Further, the emotion recognition method according to an embodiment of the present invention includes a learning data generation step of generating learning data based on the collected first bio-signals and first lifelog data; An emotion recognition model generating step of generating an emotion recognition model through machine learning based on the learning data; a data generation step of generating inference data based on the collected second bio-signals and second lifelog data; and an emotion recognition and cause analysis step of inputting the reasoning data into the emotion recognition model and recognizing the emotion through reasoning.

In one embodiment of the present invention, the emotion recognition method may further include a lifestyle coaching step. In this case, the emotion recognition and cause analysis step may be to analyze the cause of the recognized emotion based on the reasoning data, and the lifestyle coaching step may be to provide information on at least one of lifestyle habits and behavioral patterns related to the recognized emotion based on the analysis result of the emotion recognition and cause analysis step.

In an embodiment of the present invention, the generating data for learning may include generating data for learning by dividing the first bio-signal according to at least one criterion of situation and behavior, and then extracting feature values from the divided first bio-signal and the first lifelog data, and generating data for inference may include dividing the second bio-signal according to at least one criterion of situation and action, and then extracting feature values from the divided second bio-signal and the second lifelog data to extract feature values from the divided second bio-signal and the second lifelog data. may be to generate

According to an embodiment of the present invention, mood recognition is possible by combining lifelog data such as behavior, situation, sleep, etc., which appear over a relatively long period of time, that is, behavior, situation, and sleep that are highly related to mood, as well as characteristics of biological signals for each behavior and situation, and using them as new feature variables.

In addition, according to one embodiment of the present invention, by predicting the user's mood and analyzing the behavior or situation that causes it when negative emotion is predicted, it is possible to find and improve lifestyle habits or behavior patterns that cause negative emotion. There is an effect that it can be supported.

Effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

1 is a block diagram showing the configuration of an emotion recognition device according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a bio-signal and lifelog data collector according to an embodiment of the present invention;
3 is a block diagram showing the configuration of an emotion recognition and cause analyzer according to an embodiment of the present invention.
4 is a flowchart illustrating a method for recognizing emotion according to an embodiment of the present invention.

In this specification, the emotion of a long time is expressed with the term 'mood'. The present invention relates to a device and method for recognizing a long-time emotion, that is, a 'mood'. In order to recognize 'mood', it is necessary to approach differently from when recognizing emotions that last for a short time, such as joy, sadness, and anger. Mood has a close relationship with lifelogs such as behavior, situation, and sleep, as well as biosignal responses caused by complex emotions during the previous time. Therefore, in order to recognize the mood, it is necessary to deal with not only biosignals collected over a long period of time, but also lifelog data such as behavior, situation, and sleep that are deeply related to the mood. In particular, biological signals tracked by behavior or situation are expected to be effective in finding moods. For example, it is assumed that an application that predicts the user's mood immediately after waking up and informs him or her of what lifestyle habits or behavioral patterns should be corrected if the user is not in a good mood. First, it is not realistic to predict the user's mood immediately after waking up with biosignal data collected for a short time after waking up. This is because the mood immediately after waking up of the user corresponds to the emotion that is maintained for a relatively long time, that is, the mood, rather than the emotion at that moment. In addition, it can be seen that the mood immediately after waking up is directly or indirectly influenced by not only the lifelog itself, such as the previous day's actions, situations, and sleep, but also individual actions and emotions in the situation. Therefore, in order to recognize these moods, it can be said that it is advantageous to extract and utilize features from individual behaviors or biosignals for each situation.

In the present invention, an emotion recognition model (mood recognition model) is constructed by extracting meaningful features by using lifelog data as well as biological signals and learning these features. In addition, when negative emotions are recognized through the emotion recognition model, it provides a lifestyle that can improve negative emotions through the process of analyzing which actions and situations have a great impact on emotions.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in a variety of different forms, and only these embodiments make the disclosure of the present invention complete, and those skilled in the art It is provided to completely inform the scope of the invention, and the present invention is only defined by the scope of the claims. Meanwhile, terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" does not preclude the presence or addition of one or more other components, steps, operations, and/or elements in which a stated element, step, operation, and/or element is present.

In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate overall understanding in describing the present invention, the same reference numerals will be used for the same means regardless of the drawing numbers.

1 is a block diagram showing the configuration of an emotion recognition device (10, also referred to as a 'mood recognition device') according to an embodiment of the present invention.

The emotion recognition device 10 according to an embodiment of the present invention includes a biosignal and lifelog data collector 100 that collects biosignals and lifelog data, an emotion recognition model generator 200 that generates an emotion recognition model through machine learning using the biosignals and lifelog data, and an emotion recognition and cause analyzer 300 that recognizes emotions by inputting biosignals and lifelog data into an emotion recognition model and analyzes the cause of the recognized emotions.

In addition, the emotion recognition device 10 according to an embodiment of the present invention may further include a lifestyle coach 400 that recommends a customized lifestyle based on the analysis result obtained from the emotion recognition and cause analyzer 300 and domain knowledge.

The bio-signal and lifelog data collector 100 collects the bio-signal and lifelog data, extracts feature values from the bio-signal and lifelog data, and generates learning data or inference data.

The emotion recognition model generator 200 generates an emotion recognition model through machine learning based on training data.

The emotion recognition and cause analyzer 300 recognizes emotion through inference by inputting reasoning data to an emotion recognition model. In addition, the emotion recognition and cause analyzer 300 may analyze the cause of the recognized emotion based on the reasoning data. The emotion recognition and cause analyzer 300 may analyze the cause of positive emotion as well as negative emotion. In some cases, it may be determined whether the recognized emotion is negative emotion and the cause of the emotion may be analyzed only when the emotion is negative.

The lifestyle coach 400 is related to the emotion recognized by the emotion recognition and cause analyzer 300 through a domain knowledge DB search based on the emotion recognition and analysis results of the emotion recognition and cause analyzer 300. Information on at least one of lifestyle and behavioral patterns may be provided. For example, the lifestyle coach 400 may recommend a customized lifestyle capable of improving negative emotions.

2 is a block diagram showing the configuration of a biosignal and lifelog data collector 100 according to an embodiment of the present invention.

The biosignal and lifelog data collector 100 may include a biosignal measurer 110, a lifelog data collector 120, a data logger 130, a signal preprocessor 140, a feature extractor 150, and a normalizer 160. The bio-signal and lifelog data collector 100 shown in FIG. 2 is according to an embodiment, and components of the bio-signal and lifelog data collector 100 are not limited to the embodiment shown in FIG. 2, and can be added, changed, or deleted as needed.

The biosignal measurer 110 collects biosignals through measurement. That is, the bio-signal measurer 110 may measure various bio-signals from the user. The bio-signal measuring unit 110 measures various bio-signals necessary for mood recognition through sensors attached to the user's body, and typically measures electrocardiogram (ECG), galvanic skin response (GSR), skin temperature (SKT), photo-plethysmography (PPG), respiration rates, and electroencephalogram (EEG) signals. In addition, biosignals useful for recognizing emotion or mood may be added. The biosignal measurer 110 transmits the biosignal measured from the user's body to the data logger 130 connected to the Internet via a wireless interface such as Wi-Fi or mobile communication.

The lifelog data collection unit 120 collects lifelog data. The lifelog data collection unit 120 may collect information experienced by individuals in daily life as lifelog data. For example, the lifelog data collection unit 120 may collect information (lifelog data) on behavior, situations, and sleep. The lifelog data collection unit 120 may collect lifelog data through a method in which a user records information directly through a smartphone app or automatically records information collected through a wearable device in association with a smartphone. As wearable devices for lifelog collection, glasses for image input, necklaces for sound input, and bracelets or belts for position and behavior input can be used. [Table 1] and [Table 2] show examples of behavior-related lifelog data collected by the lifelog data collection unit 120. In addition to the examples of [Table 1] and [Table 2], behavioral information useful for recognizing emotion or mood may be additionally included in lifelog data related to behavior.

Meanwhile, [Table 3] shows an example of situation-related lifelog data collected by the lifelog data collection unit 120.

In addition to the examples in [Table 3], geographical data, weather, noise, illumination, etc. may be included in context-related lifelog data, and other context information useful for recognizing emotion or mood may be additionally included in context-related lifelog data.

The lifelog data collection unit 120 may collect sleep-related lifelog data using various wearable devices such as a smart watch, a smart band, a smart ring, and a sensor under the bed. The sleep-related lifelog data includes actual sleep time, sleep efficiency, and sleep pattern, and may additionally include sleep information useful for recognizing emotion or mood.

The lifelog data collection unit 120 collects emotion or mood labels for instructional learning. Emotion or mood labels can be collected in various forms depending on the service to be used by recognizing emotions or moods. Representatively, through a smartphone app, the mood before going to bed and immediately after waking up can be collected by selecting the following 5-step Likert scale.

1. very unpleasant

2. unpleasant

3. Moderate

4. Good

5. Very good

In addition, the lifelog data collection unit 120 may collect emotion or mood labels in various ways according to service needs.

The lifelog data collection unit 120 transmits the collected lifelog data to the data logger 130 connected to the Internet via a wireless interface such as Wi-Fi or mobile communication.

The data logger 130 synchronizes and collects the biosignals and lifelog data collected by the biosignal measurement unit 110 and the lifelog data collection unit 120 with respect to time. The data logger 130 may generally be a large-capacity data collection server, and may have various functions including personal authentication management so that it can be managed for each user. Wearable devices, including smart phones, can transmit data to the data logger 130 periodically or according to set transmission conditions in conjunction with the data logger 130 .

The signal pre-processor 140 removes noise included in the bio-signal synchronized by the data logger 130 and then divides the synchronized bio-signal according to at least one criterion of situation, behavior, and sleep. The signal pre-processing unit 140 divides the biosignal for each action, situation, or progress unit of sleep. For example, the signal pre-processing unit 140 divides the biosignals according to the start and end times of travel among the behaviors listed in [Table 1] and [Table 2] and converts them into travel-related biosignals. In addition, the signal pre-processing unit 140 may divide the biosignal into data in the same way for detailed items of 'movement' (for example, 'commuting to and from work and work-related movement') according to the needs of the emotion recognition device 10. That is, the signal pre-processing unit 140 divides the bio signal from the start of attendance to the end of attendance for one movement (commuting and work-related movement) specified by the start and end of attendance, and allocates it to the corresponding movement. The signal preprocessing unit 140 may divide the biosignal in the same way for each situation listed in Table 3. In addition, the signal pre-processing unit 140 may divide bio signals for each sleep based on the start time and end time of sleep.

The feature extraction unit 150 extracts features (feature values) from each divided bio-signal. The feature extractor 150 extracts various feature values from biosignals divided by action, situation, and sleep units. For example, the feature extractor 150 may extract heart rate variability (HRV) and its derived index with respect to an electrocardiogram (ECG) signal. Heart rate variability (HRV) is an index that reflects the activity of the autonomic nervous system. It quantifies the fluctuation characteristics of the R-R interval, which means the interval between the R in the adjacent QRS complex and the next R in the electrocardiogram (ECG) signal. Heart rate variability (HRV) is a useful feature for determining the level of activity or balance of sympathetic and parasympathetic nerves, and is widely used to identify emotions or emotions. Features related to heart rate variability (HRV) can be largely divided into time domain features and frequency domain features.

First, the time domain feature of heart rate variability (HRV) is obtained through statistical calculation with a Normal to Normal (NN) interval, which means a normal R-R interval in the R-R interval. Typically, MeanNN, which means the average of NN intervals, SDNN (standard deviation of the NN interval), which means the standard deviation of NN intervals, root-mean-square of successive differences (RMSSD), which means the root-mean-square (RMS) of the sum of the squared differences between adjacent NN intervals, NN50 count to the number of all NN intervals (the number of NN intervals in which the difference between consecutive NN intervals exceeds 50 ms). pNN50 and the like.

In addition, the frequency domain characteristics of heart rate variability (HRV) are closely related to the autonomic nervous system including the parasympathetic nervous system and the sympathetic nervous system, so it is useful for evaluating the balance of the sympathetic-parasympathetic nervous system in the cardiovascular system. In order to extract the frequency domain features, a process of converting the time series data of the R peak interval into the frequency domain through FFT (Fast Fourier Transform) is required. This process is called spectrum analysis, and representative frequency domain features obtained through spectrum analysis of heart rate variability (HRV) are shown in [Table 4].

Frequency domain characteristics of heart rate variability (HRV) meaning Total power Variance of NN intervals ULF Power in the ultra-low frequency band below 0.003 Hz VLF Power in the ultra-low frequency band between 0.003-0.4 Hz LF Power in the low frequency band between 0.04-0.15 Hz HF Power in the high frequency band between 0.15-0.4 Hz LF/HF Ratio of LF power to HF power

The feature extractor 150 may extract features for biosignals such as GSR, SKT, PPG, and EEG in addition to the features described above. Most of the biosignal-related features to be extracted by the feature extractor 150 basically include average values and standard deviations, and other features widely used in emotion recognition may be included. Of course, features extracted from biosignals have been widely used in conventional emotion recognition. However, the present invention does not simply extract features from biosignals for a short period of time, but extracts features of biosignals for each behavior and situation listed in [Table 1], [Table 2], and [Table 3]. It can be said that the biggest difference is. The reason why characteristics are extracted by dividing them into actions and situations is that mood, which means a relatively long period of time, is not irrelevant to daily life patterns. For example, if the patterns of vital signals measured during conversations with family members and the mood after waking up the next morning appear to have a strong correlation with each other, emotions during conversations with family members can be seen as an important feature influencing moods. If this concept is extended to other behaviors and situations, it is necessary to learn the emotion recognition model built in the emotion recognition device 10 by extracting characteristics of biosignals for each behavior and situation.

In addition, the feature extractor 150 may extract feature values from lifelog data in addition to feature values based on bio-signals. For example, the feature extractor 150 may extract various lifelog data feature values, such as calories, exercise time, smartphone activity history (or usage time for each smartphone app), and smartphone usage time, from lifelog data.

The normalization unit 160 normalizes the feature values extracted by the feature extraction unit 150 . In order for various features (feature values) to be input into the emotion recognition model with equal weight, it is necessary to normalize the feature values. For example, the normalizer 160 may normalize feature values of all feature variables to values between -1 and 1. This normalization method can be usefully utilized when generally feature values have a normal distribution. [Equation 1] is an equation that can be used to normalize feature values of feature variables using the mean and standard deviation of feature values.

In [Equation 1], X _norm means a normalized feature and X means an original feature value. μ means the average of all feature values, and σ means the standard deviation of all feature values.

Meanwhile, the normalization unit 160 may use another normalization method if the feature values are not close to a normal distribution. [Equation 2] is an equation that can be used to normalize a feature value of a feature variable to a range between 0 and 1 using the minimum and maximum values of the feature values.

In [Equation 2], X _norm means a normalized feature and X means an original feature value. X _min and X _max mean the minimum and maximum values of each feature value.

The normalization unit 160 normalizes feature values extracted from lifelog data, such as calories, exercise time, smartphone use time, smartphone activity history, sleep time, and sleep efficiency, as well as feature values extracted from biosignals. In this specification, the (normalized) feature values provided for learning the emotion recognition model are referred to as 'learning data', and the (normalized) feature values provided for reasoning using the emotion recognition model are referred to as 'inference data'.

The emotion recognition model generator 200 shown in FIG. 1 generates an emotion recognition model through machine learning based on learning data. That is, the emotion recognition model generator 200 receives learning data from the biosignal and lifelog data collector 100 . Since the emotion recognition model generator 200 uses a conventional machine learning method, detailed description of the machine learning method used by the emotion recognition model generator 200 is omitted in this specification. The emotion recognition model generator 200 may be representative machine learning algorithms such as KNN (K-Nearest Neighbor), SVC (Support Vector Classification), random forest, and AdaBoost, which are representative machine learning algorithms widely used in the field of emotion recognition. In addition, machine learning algorithms useful for recognizing emotions or moods may be additionally included. In addition, when learning without extracting features from bio-signal and lifelog data, deep learning algorithms such as Deep Neural Network (DNN), Convolutional Neural Network (CNN), and Recurrent Neural Network (RNN) can be used to create emotion recognition models.

3 is a block diagram showing the configuration of an emotion recognition and cause analyzer 300 according to an embodiment of the present invention.

The emotion recognition and cause analyzer 300 may include an emotion recognition unit 310 , a negative emotion detection unit 320 and an emotion cause analysis unit 330 . The emotion recognition and cause analyzer 300 shown in FIG. 3 is according to an embodiment, and the components of the emotion recognition and cause analyzer 300 are not limited to the embodiment shown in FIG. 3, and can be added, changed or deleted as needed.

The emotion recognition model obtained through learning in the emotion recognition model generation unit 200 is used by the emotion recognition and cause analyzer 300 to recognize the user's emotion based on the data for reasoning. The biosignal and lifelog data collected from the user are input to the emotion recognition unit 310 via the biosignal and lifelog collection unit 100 .

The emotion recognition unit 310 recognizes emotion through reasoning by inputting inference data generated by the bio-signal and lifelog data collector 100 into an emotion recognition model. The user's emotion recognized by the emotion recognition unit 310 is input to the negative emotion detection unit 320 .

The negative emotion detecting unit 320 determines whether the emotion recognized by the emotion recognizing unit 310 is a negative emotion.

The emotion cause analysis unit 330 analyzes the cause of the emotion recognized by the emotion recognition unit 310 based on the data for reasoning. When the negative emotion detecting unit 320 determines that the emotion recognized by the emotion recognizing unit 310 is negative emotion, the emotion cause analysis unit 330 analyzes the cause of the corresponding negative emotion based on the reasoning data.

The emotion cause analysis unit 330 may analyze characteristics related to behaviors, situations, and sleep that greatly affect negative emotions based on features (feature values) included in inference data. The emotion cause analysis unit 330 may analyze characteristics related to behaviors, situations, or sleep that greatly affect not only negative emotions but also positive emotions. The analysis tool used by the emotion cause analysis unit 330 is a tool capable of analyzing a correlation between feature variables (variables constituting inference data) and emotions recognized by the emotion recognition unit 310 . That is, the emotion cause analysis unit 330 may analyze the cause of the emotion recognized by the emotion recognition unit 310 through correlation analysis between feature variables constituting inference data and emotions recognized by the emotion recognition unit 310. Representative analysis tools that can be used by the emotion cause analysis unit 330 include Pearson's correlation coefficient, Spearman's correlation coefficient, Kendall's tau, and the like. In addition, various tools capable of measuring the correlation between two variables can be utilized.

The lifestyle coach 400 shown in FIG. 1 provides information on at least one of a user's lifestyle and behavioral pattern related to the emotion recognized by the emotion recognition and cause analyzer 300 based on the analysis result of the emotion recognition and cause analyzer 300. In addition, the lifestyle coach 400 searches the domain knowledge DB using the analysis result of the emotional cause analysis unit 330, extracts related domain knowledge (information), and provides customized services to users through the extracted domain knowledge (information). For example, the domain knowledge DB may store lifestyle habits or behavior patterns that cause negative emotions, along with information on improvement of those lifestyle habits or behavior patterns. The lifestyle coach 400 may provide customized improvement services for lifestyle habits or behavior patterns that cause negative emotions to users through domain knowledge DB searches. As an example of the customized improvement service, if it is analyzed that the user has negative emotions due to the stress caused by yesterday's company work, it may be recommended to meditate or take a walk so that the user can get away from work for a while and take psychological stability.

4 is a flowchart illustrating an emotion recognition method (mood recognition method) according to an embodiment of the present invention.

The emotion recognition method according to an embodiment of the present invention includes steps S510 to S540, and may further include step S550.

Step S510 is a step of generating data for learning. The biosignal and lifelog data collector 100 collects biosignals (referred to as 'first biosignal' to distinguish them from the biosignals of step S530) and lifelog data (referred to as 'first lifelog data' to distinguish them from the lifelog data of step S530), and generates learning data based on the first biosignals and the first lifelog data. The bio-signal and lifelog data collector 100 synchronizes the first bio-signal and the first lifelog data with respect to time, removes noise included in the synchronized first bio-signal, divides the synchronized first bio-signal according to at least one criterion of situation, behavior, and sleep, and extracts features (feature values) from the divided first bio-signal and the first lifelog data to generate learning data. The biosignal and lifelog data collector 100 may normalize features (feature values) to generate training data. For details of step S510, reference may be made to the above description of the bio-signal and lifelog data collector 100 and its components.

Step S520 is a step of generating an emotion recognition model. The emotion recognition model generator 200 generates an emotion recognition model through machine learning based on training data. For details of step S520 , reference may be made to the above description of the emotion recognition model generator 200 .

Step S530 is a step of generating data for inference. The biosignal and lifelog data collector 100 collects biosignals (referred to as 'second biosignal' to distinguish them from the biosignals of step S510) and lifelog data (referred to as 'second lifelog data' to distinguish them from the lifelog data of step S510), and generates inference data based on the second biosignals and the second lifelog data. The bio-signal and lifelog data collector 100 synchronizes the second bio-signal and the second lifelog data with respect to time, removes noise included in the synchronized second bio-signal, divides the synchronized second bio-signal according to at least one criterion of situation, behavior, and sleep, and extracts features (feature values) from the divided second bio-signal and the second lifelog data to generate data for inference. For details of step S530, reference may be made to the above description of the bio-signal and lifelog data collector 100 and its components.

Step S540 is an emotion recognition and cause analysis step. The emotion recognition and cause analyzer 300 recognizes emotion through inference by inputting reasoning data to an emotion recognition model. In addition, the emotion recognition and cause analyzer 300 may analyze the cause of the recognized emotion based on the reasoning data. The emotion recognition and cause analyzer 300 may determine whether the recognized emotion is negative emotion, and analyze the cause of the emotion only when it is determined to be negative emotion. For details on step S540 , reference may be made to the above description of the emotion recognition and cause analyzer 300 and its components.

Step S550 is a lifestyle coaching step. The lifestyle coach 400 may provide information on at least one of lifestyle habits and behavior patterns related to emotions recognized in step S540 based on the analysis result in step S540. In addition, the lifestyle coach 400 searches the domain knowledge DB using the analysis result of step S540, extracts related domain knowledge (information), and utilizes the extracted domain knowledge (information) to provide customized services to users. For example, if the emotion recognized in step S540 is 'lethargic', and as a result of the analysis in step S540, 'working hours', which is a characteristic value of lifelog data related to 'work', appears to be the feature value with the highest correlation with 'lethargic' among user behavior patterns, the lifestyle coach 400 uses the domain knowledge DB search result along with information that one of the causes of 'lethargy' is 'working hours' (specific values can be displayed). We can provide a customized improvement service that recommends meditation or a walk so that you can take a break from stress and get psychological stability. For details of step S550 , reference may be made to the above description of the lifestyle coach 400 .

In the description with reference to FIG. 4 , each step may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the invention. Also, some steps may be omitted if necessary, and the order of steps may be changed. For example, after generating the emotion recognition model, steps S510 and S520 may be omitted. That is, only steps S530 to S550 may be performed for emotion recognition and lifestyle coaching. Also, only steps S530 and S540 may be performed for emotion recognition only.

Meanwhile, even if other omitted contents, the contents of FIGS. 1 to 3 may be applied to the contents of FIG. 4 . Also, the contents of FIG. 4 may be applied to the contents of FIGS. 1 to 3 .

For reference, components according to an embodiment of the present invention may be implemented in software or hardware form such as a digital signal processor (DSP), a field programmable gate array (FPGA), or an application specific integrated circuit (ASIC), and may perform predetermined roles.

However, 'components' are not meant to be limited to software or hardware, and each component may be configured to be in an addressable storage medium or configured to reproduce one or more processors.

Thus, as an example, a component includes components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables.

Components and the functionality provided within them may be combined into fewer components or further separated into additional components.

Meanwhile, it will be understood that each block of the flowchart drawings and combinations of the flowchart drawings can be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions executed by the processor of the computer or other programmable data processing equipment create means for performing the functions described in the flowchart block(s). Since computer program instructions can also be loaded on a computer or other programmable data processing equipment, it is also possible that a series of operating steps are performed on a computer or other programmable data processing equipment to create a computer-executed process so that the computer or other programmable data processing equipment performing instructions provides steps for executing the functions described in the flowchart block(s).

Additionally, each block may represent a module, segment, or portion of code that includes one or more executable instructions for executing specified logical function(s). It should also be noted that in some alternative implementations it is possible for the functions mentioned in the blocks to occur out of order. For example, two blocks shown in succession may in fact be executed substantially concurrently, or the blocks may sometimes be executed in reverse order depending on their function.

The term '~group' or '~unit' used in the embodiments of the present invention means software or a hardware component such as FPGA or ASIC, and '~group' or '~unit' performs certain roles. However, '~ group' or '~ part' is not limited to software or hardware. '~ group' or '~ unit' may be configured to be in an addressable storage medium and may be configured to reproduce one or more processors. Thus, as an example, '~group' or '~unit' includes components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. Functions provided within the components and '~ group' or '~ unit' may be combined into a smaller number of components and '~ group' or '~ unit' or further separated into additional components and '~ group' or '~ unit'. In addition, the components and '~ group' or '~ unit' may be implemented to play one or more CPUs in a device or a secure multimedia card.

The above-described emotion recognition method (mood recognition method) has been described with reference to flowcharts presented in the drawings. Although the method is shown and described as a series of blocks for simplicity, the invention is not limited to the order of the blocks, and some blocks may occur in a different order or concurrently with other blocks than shown and described herein, and various other branches, flow paths, and sequences of blocks may be implemented that achieve the same or similar results. Also, not all blocks shown may be required for implementation of the methods described herein.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art can variously modify and change the present invention without departing from the spirit and scope of the present invention described in the claims below. It will be understood that it can be changed.

10: emotion recognition device (mood recognition device)
100: biosignal and lifelog data collector
110: biosignal measuring unit
120: lifelog data collection unit
130: data logger
140: signal pre-processor
150: feature extraction unit
160: normalization unit
200: emotion recognition model generator
300: emotion recognition and cause analyzer
310: emotion recognition unit
320: negative emotion detection unit
330: emotional cause analysis unit
400: Lifestyle Coach

Claims (1)

a bio-signal and life-log data collector that collects bio-signals and lifelog data, extracts feature values from the bio-signals and lifelog data, and generates data for learning and reasoning;
an emotion recognition model generator generating an emotion recognition model through machine learning based on the learning data; and
an emotion recognition and cause analyzer inputting the data for inference into the emotion recognition model and recognizing emotion through inference;
Emotion recognition device comprising a.

Images