WO2023103512A1 - Emotion monitoring method and emotion monitoring apparatus - Google Patents

Abstract

Provided in the present application are an emotion monitoring method and an emotion monitoring apparatus, which are conducive to improving the accuracy of emotion recognition. The method comprises: acquiring a photoplethysmographic (PPG) signal of a user in real time, acquiring a surface electromyography (SEMG) signal of the user in real time, and acquiring an ACC signal of the user in real time; in view of the PPG signal, the SEMG signal, age information of the user and gender information of the user, compiling statistics on emotion classification results of the user at a plurality of moments within a first time period; on the basis of the emotion classification results of the user at the plurality of moments within the first time period, determining an emotion state evaluation result of the user within the first time period; in view of the PPG signal, the ACC signal, the age information of the user and the gender information of the user, determining a sleep quality evaluation result of the user within the first time period; and according to the emotion state evaluation result and the sleep quality evaluation result, determining whether to perform emotion early warning.

Description

Mood monitoring method and mood monitoring device

This application claims the priority of the Chinese patent application with application number 202111510699.2 and application title "Emotional Monitoring Method and Emotional Monitoring Device" filed with the China Patent Office on December 10, 2021, the entire contents of which are hereby incorporated by reference in this application .

technical field

The present application relates to the field of terminal equipment, and more specifically, to an emotion monitoring method and an emotion monitoring device.

Background technique

In contemporary society, people pay more and more attention to physical and mental health. The accumulation of negative emotions such as anxiety, tension, anger, depression, sadness, and pain for a long time may lead to negative mental illnesses such as mania, depression, autism, and anxiety.

Research provided by psychology and physiology shows that there is a great relationship between physiological reactions and people's emotional state. In order to pay attention to the user's physical and mental health, wearable devices represented by smart bracelets have gradually become popular, which can monitor the user's emotions, Exercise, sleep, health indicators, etc. are monitored.

At present, emotion monitoring is mainly based on the measurement of physiological signals. In a possible implementation, since different emotions have different time-domain and frequency-domain characteristics, it is possible to collect the user's photoplethysmography (PPG) The pulse rate variability (PRV) sequence is extracted from the PPG signal, and the extracted PRV sequence is analyzed in time domain and frequency domain to obtain time domain features and frequency domain features, and the time domain features and The frequency domain features are used as input to the neural network for emotion recognition.

However, the above methods cannot fully reflect the physiological characteristics of different emotions, and the accuracy of emotion recognition is not high.

Contents of the invention

The present application provides an emotion monitoring method and an emotion monitoring device, which are beneficial to improving the accuracy of emotion recognition.

In the first aspect, an emotion monitoring method is provided, which is applied to a terminal device equipped with a pulse wave sensor, an electromyographic electrode sensor, and an acceleration sensor. The method comprises: acquiring the user's photoplethysmography PPG signal in real time through the pulse wave sensor, acquiring the user's surface electromyography (surface electromyography, SEMG) signal in real time through the myoelectric electrode sensor, and acquiring the user's ACC signal in real time through the acceleration sensor . Combining the PPG signal, SEMG signal, user's age information and user's gender information, the classification results of the user's emotions at multiple moments in the first time period are counted, and the classification results include positive emotions, calm emotions or negative emotions. Based on the classification results of the user's emotions at multiple moments in the first time period, an evaluation result of the user's emotional state in the first time period is determined. Combining the PPG signal, the ACC signal, the age information of the user, and the gender information of the user, the sleep quality evaluation result of the user in the first time period is determined. According to the emotional state evaluation results and the sleep quality evaluation results, it is determined whether to carry out emotional early warning.

In this application, the terminal device can combine the user's PPG signal, SEMG signal, ACC signal, age information and gender information to obtain the user's emotional state evaluation result in the first time period, and combine the PPG signal, ACC signal, user's age information and The user's gender information, determine the user's sleep quality evaluation results in the first time period, and comprehensively consider the user's emotions according to the user's emotional state evaluation results and sleep quality evaluation results in the first time period, which is conducive to improving the accuracy of emotion recognition rate to provide users with better mental health services.

In combination with the first aspect, in some implementations of the first aspect, combined with PPG signals, SEMG signals, user age information, and user gender information, the classification results of user emotions at multiple moments within the first time period are counted , including: performing feature extraction on the PPG signal, SEMG signal, age information of the user, and gender information of the user at the first moment among the plurality of moments, to obtain multiple sets of biometric features of the user at the first moment. Feature fusion is performed on multiple sets of biometric features at the first moment to obtain the fused features of the user at the first moment. Based on the fusion features at the first moment, the classification result of the user's emotion at the first moment is obtained.

In this application, the terminal device can obtain multiple sets of biometric features of the user at the first moment through feature extraction, and the interpretability and representation ability of the fused features are strong. Emotion recognition based on the fused features is conducive to improving the accuracy of emotion recognition. Rate.

In combination with the first aspect, in some implementations of the first aspect, feature fusion is performed on multiple sets of biometric features at the first moment to obtain the fused features of the user at the first moment, including: calculating multiple sets of biometric features through the flexible maximum transfer function The weight of each set of biometrics in biometrics. According to each set of biometric features and the weight of each set of biometric features, the fusion features of the user at the first moment are obtained.

In this application, different physiological features have different weight values, so that the importance of different physiological signals can be obtained, making the emotion recognition result more accurate and reliable.

With reference to the first aspect, in some implementations of the first aspect, the multiple sets of biological characteristics include at least one of the following: heart rate characteristics, breathing characteristics, blood oxygen characteristics, blood sugar characteristics, blood pressure characteristics, or epidermal electromyography characteristics.

In conjunction with the first aspect, in some implementations of the first aspect, based on the classification results of the user's emotions at multiple moments within the first time period, determining the user's emotional state evaluation results in the first time period includes: The number of negative emotions of the user in the first time period and the duration of each negative emotion are obtained from the classification results of emotions at multiple moments. Based on the number of times of negative emotions and the duration of each negative emotion, an evaluation result of the user's emotional state in the first time period is determined.

In this application, the terminal device can obtain the number of negative emotions of the user in the first period of time and the duration of each negative emotion. The longer the accumulated negative emotion time, the lower the emotional state evaluation score, which is conducive to accurate evaluation of the user. Emotions are assessed during the first time period.

With reference to the first aspect, in some implementations of the first aspect, combining the PPG signal, the ACC signal, the user's age information, and the user's gender information to determine the user's sleep quality evaluation result in the first time period, including: based on ACC The component of the signal on the coordinate axis is used to obtain the number of sleep interruptions of the user in the first time period. Based on the PPG signal, the heart rate change information of the user in the first time period is acquired. According to the heart rate change information, the user's total sleep duration, deep sleep duration, and light sleep duration in the first period of time are obtained. Combining the number of sleep interruptions, total sleep duration, deep sleep duration, and light sleep duration, the user's sleep evaluation result in the first time period is determined.

In this application, since the user's sleep conditions may be different under different emotions, the terminal device obtains the user's sleep evaluation result by collecting the user's heart rate change information during the sleep period, and the sleep evaluation result is conducive to improving the accuracy of emotion recognition. .

In combination with the first aspect, in some implementations of the first aspect, determining whether to perform an emotional early warning according to the emotional state evaluation result and the sleep quality evaluation result includes: obtaining the Describe the emotion monitoring results of the first time period. When the emotion monitoring result satisfies the preset condition, it is determined to carry out an emotion early warning.

In this application, if the terminal device detects that the user's emotional monitoring results in the first time period meet the preset conditions, then the terminal device can give the user an emotional warning, which is conducive to paying attention to the user's mental health in a timely manner and improving the user's usage. experience.

In combination with the first aspect, in some implementations of the first aspect, according to the emotional state evaluation results and the sleep quality evaluation results, the user's emotional monitoring results in the first time period are obtained, including: quantified values of the emotional state evaluation results The weighted summation is performed with the quantified value of the sleep quality evaluation result to obtain the user's emotional score in the first time period. When the emotion monitoring result satisfies the preset condition, determining to perform an emotional early warning includes: determining to perform an emotional early warning when the emotional score is less than or equal to a preset threshold.

With reference to the first aspect, in some implementations of the first aspect, the user's emotional age is determined according to the user's emotion monitoring results in the first time period, the number of negative emotions, and the user's age information. At least one of emotional age, emotional monitoring results, or emotional statements is displayed.

In the present application, the terminal device may display at least one of emotional age, emotion monitoring results, or emotional statements to the user, which is beneficial for the user to adjust emotion.

With reference to the first aspect, in some implementation manners of the first aspect, peaks and valleys of the ACC signal are obtained. According to the peak and trough of the ACC signal, it is determined whether the user is in a non-exercise state. Combining PPG signal, SEMG signal, user's age information and user's gender information, counting the classification results of user's emotions at multiple moments in the first time period, including: combining PPG signal when the user is in a non-exercising state , SEMG signal, age information of the user, and gender information of the user, and the classification results of the user's emotions at multiple moments in the first time period are counted.

In this application, since the PPG signal is greatly disturbed by motion, emotion recognition needs to be realized in a non-motion state, so the terminal device can judge whether the user is in a non-motion state through the ACC signal, and if the user is in a motion state, the collected PPG signal and The SEMG signal is an invalid signal and needs to be collected again, which is conducive to obtaining effective PPG signals and SEMG signals and improving the accuracy of emotion recognition.

In a second aspect, an emotion monitoring device is provided, including: for performing the method in any possible implementation manner of the above first aspect. Specifically, the apparatus includes a module configured to execute the method in any possible implementation manner of the foregoing first aspect.

In a third aspect, another emotion monitoring device is provided, including a processor, the processor is coupled to a memory, and can be used to execute instructions in the memory, so as to implement the method in any possible implementation manner of the above first aspect. Optionally, the device further includes a memory. Optionally, the device further includes a communication interface, and the processor is coupled to the communication interface.

In an implementation manner, the emotion monitoring apparatus is a terminal device. When the emotion monitoring device is a terminal device, the communication interface may be a transceiver, or an input/output interface.

In another implementation manner, the emotion monitoring device is a chip configured in a terminal device. When the emotion monitoring device is a chip configured in a terminal device, the communication interface may be an input/output interface.

In a fourth aspect, a processor is provided, including: an input circuit, an output circuit, and a processing circuit. The processing circuit is configured to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in any possible implementation manner of the first aspect above.

In a specific implementation process, the above-mentioned processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, a gate circuit, a flip-flop, and various logic circuits. The input signal received by the input circuit may be received and input by, for example but not limited to, the receiver, the output signal of the output circuit may be, for example but not limited to, output to the transmitter and transmitted by the transmitter, and the input circuit and the output The circuit may be the same circuit, which is used as an input circuit and an output circuit respectively at different times. The present application does not limit the specific implementation manners of the processor and various circuits.

In a fifth aspect, a processing device is provided, including a processor and a memory. The processor is used to read instructions stored in the memory, and may receive signals through the receiver and transmit signals through the transmitter, so as to execute the method in any possible implementation manner of the first aspect above.

Optionally, there are one or more processors, and one or more memories.

Optionally, the memory may be integrated with the processor, or the memory may be separated from the processor.

In the specific implementation process, the memory can be a non-transitory (non-transitory) memory, such as a read-only memory (read only memory, ROM), which can be integrated with the processor on the same chip, or can be set in different On the chip, the application does not limit the type of the memory and the arrangement of the memory and the processor.

It should be understood that a related data interaction process such as sending indication information may be a process of outputting indication information from a processor, and receiving capability information may be a process of receiving input capability information from a processor. In particular, processed output data may be output to the transmitter, and input data received by the processor may be from the receiver. Wherein, the transmitter and the receiver may be collectively referred to as a transceiver.

The processing device in the fifth aspect above can be a chip, and the processor can be implemented by hardware or by software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc.; when implemented by software When implemented, the processor may be a general-purpose processor, which is realized by reading the software code stored in the memory, and the memory may be integrated in the processor, or it may be located outside the processor and exist independently.

In a sixth aspect, a computer program product is provided, and the computer program product includes: a computer program (also referred to as code, or instruction), which, when the computer program is run, causes the computer to perform any one of the possible implementations in the first aspect above. methods in methods.

In a seventh aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program (also referred to as code, or an instruction) which, when run on a computer, enables the computer to execute the above-mentioned first aspect. A method in any of the possible implementations.

Description of drawings

FIG. 1 is a schematic diagram of a terminal device applicable to an embodiment of the present application;

FIG. 2 is a schematic diagram of a physical structure of a terminal device provided in an embodiment of the present application;

Fig. 3 is a schematic flow chart of an emotion monitoring method provided by an embodiment of the present application;

Fig. 4 is a schematic diagram of an emotion classification process provided by an embodiment of the present application;

Fig. 5 is a curve diagram of heart rate changes during sleep according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a system architecture of a terminal device provided in an embodiment of the present application;

FIG. 7 is a schematic interface diagram of a terminal device provided in an embodiment of the present application;

FIG. 8 is a schematic waveform diagram of an ACC signal provided by an embodiment of the present application;

Fig. 9 is a schematic flow chart of another emotion monitoring method provided by the embodiment of the present application;

Fig. 10 is a schematic block diagram of an emotion monitoring device provided by an embodiment of the present application;

Fig. 11 is a schematic block diagram of another emotion monitoring device provided by an embodiment of the present application.

Detailed ways

The technical solution in this application will be described below with reference to the accompanying drawings.

Before introducing the emotion monitoring method and the emotion monitoring device provided in the embodiments of the present application, the following explanations are made first.

First, in the embodiments shown below, each term and English abbreviation, such as PPG signal, ACC signal, pulse wave sensor, etc., are all illustrative examples given for convenience of description, and should not constitute any limitation to this application . This application does not exclude the possibility of defining other terms that can achieve the same or similar functions in existing or future agreements.

Second, the first, second and various numbers in the embodiments shown below are only for convenience of description, and are not used to limit the scope of the embodiments of the present application. For example, distinguishing between different terminal devices, etc.

Third, "at least one" means one or more, and "multiple" means two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three types of relationships, for example, A and/or B, which can mean: A exists alone, A and B exist at the same time, and B exists alone, where A, B can be singular or plural. The character "/" generally indicates that the contextual objects are an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (one) of a, b and c may represent: a, or b, or c, or a and b, or a and c, or b and c, or a, b and c, wherein a, b, c can be single or multiple.

Generally, emotions can be divided into three categories: calm, positive, and negative. Positive emotions can include excitement, relaxation, surprise, surprise, etc., and negative emotions can include anxiety, tension, anger, frustration, sadness, pain, etc.

Emotional age is a measure that marks the level of emotional development. Different age groups may correspond to different levels of emotional development. A person's emotions are consistent with the emotional expression at what age, that is, he has a corresponding emotional age. Generally speaking, people's emotional management ability tends to gradually increase with age, and gradually grows from childhood naughtiness, childhood innocence, youth vigor, and adult perseverance, to middle age and old age. However, although some people are gradually growing and maturing physically, their hearts have not matured. Therefore, according to the time law of the development of emotional quotient (EQ), we can try to use a standard for classifying emotional control ability according to age to measure a person's emotional age. Table 1 is an exemplary EQ comparison table.

Table I

In this application, the emotional age can be summarized into four periods: childish period, active period, stable period and wisdom period.

Among them, childhood corresponds to the first grade in Table 1. During this period, emotions change quickly, are difficult to control, and each emotion lasts for a short time.

The active period corresponds to the second grade in Table 1. During this period, emotions are easily affected by studies, love affairs, and family relationships. Negative emotions are easily aroused and last for a long time, and they have a certain ability to self-regulate.

The stable period corresponds to the third and fourth grades in Table 1. During this period, students know how to restrain their emotions, are not easy to arouse negative emotions, and have strong control ability.

The wisdom period corresponds to the fifth, sixth and seventh grades in Table 1, and the mood changes little during this period.

Gender differences can also bring about emotional differences. Studies have found that women are more emotionally expressive than men, that is, women tend to exaggerate the emotions they feel, and men's heart rate changes are more obvious when they are emotional, which means that men are more sensitive to emotions. The intensity of experience is greater and more affected by emotions, but they are reluctant to express it for various reasons.

Emotions may lead to some physiological reactions, such as changes in autonomic nerve activity, including frequency of breathing, speed of exhalation and inhalation, quality of respiration, heart rate, blood vessel volume, blood pressure, skin electricity, endocrine glands, etc. For another example, brain wave changes, for example, α wave, β wave, δ wave, θ wave change with emotion, and α wave changes the most. In addition, emotions may also cause adverse reactions such as dizziness and pain.

Breathing changes under different emotions: Exemplarily, breathing is 20 breaths/minute when calm. Compared with calm emotions, when happy, the breathing depth is not large, the frequency is slightly faster, and the rhythm is relatively regular. Exemplarily, the breathing is 23 breaths/minute when calm. When sad, the respiratory rate is very slow, and the interval is long, for example, the breathing rate is 9 breaths/minute when sad. When in fear, the breathing rate is very fast, there are intermittent and pauses, the amplitude is irregular, and the trembling state is present. Exemplarily, the breathing rate is 64 times/min when in fear. When angry, the breathing rate increases, and the breathing depth increases sharply. Exemplarily, the breathing rate is 40 breaths/minute when angry.

Heart rate changes under different emotions: the heart rate is 60-80 beats per minute when the mood is calm, and the heart rate is accelerated by the influence of adrenal hormones when the mood is happy, but the rhythm is regular. When sad, the heart beats faster, the arteries constrict, and in severe cases, there is arrhythmia. Rapid heartbeat during fear, accompanied by arrhythmia. When angry, the heart beats faster, and there is a feeling of palpitation and palpitation.

Changes in blood, muscles, sleep, electrical signals, etc. may occur during negative emotions. Among them, negative emotions may lead to increased secretion of stress hormones and vasoactive substances such as cortisol, norepinephrine, epinephrine, and catecholamines, resulting in systemic vasoconstriction, increased heart rate, increased blood pressure, and changes in blood oxygen levels. Negative emotions can cause muscles to tense and contract. Negative emotions can lead to poor sleep quality, difficulty falling asleep, and dreaminess.

Through the investigation on the physiological changes of emotional changes, it is found that the respiration, heart rate, blood pressure, blood sugar, blood oxygen, and electromyographic signals are different in different emotions, and the changes in these physiological indicators caused by positive emotions and negative emotions are different. At present, photoplethysmography (photo plethysmo graphic, PPG) signals can be applied to the monitoring of heart rate, blood pressure, blood oxygen, blood sugar, respiration and other physiological indicators.

Exemplarily, blood glucose monitoring can be based on infrared light with a wavelength range of 1250-1333 nm, 1600-1666 nm or 2025 nm. The heart rate can be calculated according to the timing peak and frequency domain characteristics obtained from the PPG signal.

Exemplarily, blood oxygen saturation can be judged based on the information of infrared (600nm-800nm) and near-infrared (800nm-1000nm) light respectively detected to detect oxygenated hemoglobin HbO2 and hemoglobin Hb so as to monitor blood oxygen.

Exemplarily, since the transmission speed of the pulse wave is directly related to the blood pressure, the pulse wave transmission is fast when the blood pressure is high, and the pulse wave transmission is slow when the blood pressure is low. Therefore, the systolic and diastolic blood pressure of the human pulse wave can be estimated by establishing the characteristic equation, so as to realize non-invasive continuous blood pressure monitoring.

Exemplarily, the respiratory signal may be extracted through wavelet decomposition or empirical mode decomposition (empirical mode decomposition, EMD) of the PPG signal.

FIG. 1 is a schematic diagram of a hardware structure of a terminal device 100 applicable to an embodiment of the present application. As shown in FIG. 1 , the terminal device 100 includes a pulse wave sensor 101 , a myoelectric electrode sensor 102 , an acceleration sensor 103 and a signal processing unit 104 .

Wherein, the pulse wave sensor 101 obtains the user's PPG signal, and sends the pulse wave signal to the signal processing unit 104 . The myoelectric electrode sensor 102 is used to obtain the user's SEMG signal, and send the SEMG signal to the signal processing unit 104 . The acceleration sensor 103 is used to obtain the user's ACC signal, and send the ACC signal to the signal processing unit 104 . The signal processing unit 104 is used to receive PPG signals, SEMG signals, and ACC signals to monitor and identify user emotions.

Optionally, the terminal device 100 may further include a display 105 , a storage unit 106 , interactive hardware 107 , a wireless unit 108 and a battery 109 .

Wherein, the user may perform operations such as touch on the terminal device 100 through the interactive hardware 107 to realize the interactive operation between the user and the terminal device 100 . After the signal processing unit 104 obtains the user's emotion monitoring result, it can display the emotion monitoring result to the user through the display 105 , and store the emotion monitoring result in the storage unit 106 . The signal processing unit 104 may also send the emotion monitoring result to other terminal devices associated with the terminal device 100 or to the cloud through the wireless unit 108 .

The terminal equipment in the embodiment of the present application may refer to user equipment, access terminal, subscriber unit, subscriber station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, wireless communication device, user agent or user device . The terminal in the embodiment of the present application may be a mobile phone, a tablet computer (pad), a computer with a wireless transceiver function, a virtual reality (virtual reality, VR) terminal, an augmented reality (augmented reality, AR) terminal, a hybrid Reality (mixed reality, MR) terminals, extended reality (extended reality, XR) terminals, holographic display terminals, wireless terminals in industrial control, wireless terminals in self driving, or connected to wireless Other processing equipment of the modem, vehicle equipment, terminals in the 5G network or terminals in the future evolution network, etc.

As an example but not a limitation, in this embodiment of the present application, the terminal device may also be a wearable device. Wearable devices can also be called wearable smart devices, which is a general term for the application of wearable technology to intelligently design daily wear and develop wearable devices, such as glasses, gloves, watches, clothing and shoes. A wearable device is a portable device that is worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not only a hardware device, but also achieve powerful functions through software support, data interaction, and cloud interaction. Generalized wearable smart devices include full-featured, large-sized, complete or partial functions without relying on smart phones, such as smart watches or smart glasses, etc., and only focus on a certain type of application functions, and need to cooperate with other devices such as smart phones Use, such as various smart bracelets and smart jewelry for physical sign monitoring.

In addition, the terminal device may also be a terminal device in an Internet of Things (internet of things, IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, so as to realize the intelligent network of human-machine interconnection and object interconnection. The present application does not limit the specific form of the terminal device.

It should be understood that, in this embodiment of the present application, the terminal device may be a device for realizing the function of the terminal device, or may be a device capable of supporting the terminal device to realize the function, such as a chip system, and the device may be installed in the terminal. In the embodiment of the present application, the system-on-a-chip may be composed of chips, or may include chips and other discrete devices.

In the following, an introduction will be made by taking the terminal device 100 as an example of a smart watch. FIG. 2 is a schematic diagram of a physical structure of a smart watch 200 provided by an embodiment of the present application. As shown in FIG. 2 , the smart watch 200 includes a dial 201 and a strap 202 . The pulse wave sensor 101 and the myoelectric electrode sensor 102 shown in FIG. 1 can be located at the bottom of the dial 201 , and the acceleration sensor 103 and the signal processing unit 104 can be located inside the dial 201 . The user can wear the terminal device 100 on the wrist through the wristband 202 to collect the user's PPG signal, SEMG signal and ACC signal. Optionally, a display screen 105 is configured on the dial 201, and the user can input age information and/or gender information through a touch operation on the display screen 105, the PPG signal, SEMG signal, ACC signal collected by the terminal device, and the input user age Information and/or gender information may be stored in storage unit 106 .

It should be noted that the orientation words such as "upper" and "bottom" used by the terminal device 100 shown in the embodiment of the present application are mainly used to describe the device, and they do not form a limitation on the orientation of the terminal device 100 in actual application scenarios. .

Fig. 3 is a schematic flowchart of an emotion monitoring method 300 provided by an embodiment of the present application. The method 300 can be applied to a terminal device equipped with a pulse wave sensor, an electromyographic electrode sensor, and an acceleration sensor. The hardware structure of the terminal device in the embodiment of the present application can be the same as the hardware structure of the terminal device 100 shown in FIG. 1 above, and the physical structure It may be the smart watch 200 shown in FIG. 2 , which is not limited in this embodiment of the present application. Method 300 includes the following steps:

S301. Obtain the user's PPG signal in real time through the photoplethysmography sensor, obtain the user's epidermal myoelectricity SEMG signal in real time through the myoelectric electrode sensor, and obtain the user's ACC signal in real time through the acceleration sensor.

S302. Combining the PPG signal, SEMG signal, age information of the user, and gender information of the user, count the classification results of the user's emotions at multiple moments in the first time period, and the classification results include positive emotions, calm emotions, or negative emotions.

S303. Based on the classification results of the user's emotions at multiple moments in the first time period, determine the user's emotional state evaluation results in the first time period.

S304. Combining the PPG signal, the ACC signal, the age information of the user, and the gender information of the user, determine the sleep quality evaluation result of the user in the first time period.

S305. Determine whether to perform an emotional warning according to the emotional state evaluation result and the sleep quality evaluation result.

Exemplarily, the above-mentioned pulse wave sensor may be a PPG sensor, a pressure sensor or an ultrasonic sensor.

Exemplarily, in the embodiment of the present application, the PPG signal, the SEMG signal, the ACC signal, the age information of the user, and the gender information of the user may be collectively referred to as physiological signals.

In the embodiment of the present application, the terminal device 100 may obtain the user's emotion classification results at multiple measurement moments within the first time period according to the collected user's PPG signal, SEMG signal, user's age information, and user's gender information. The terminal device 100 can obtain the user's sleep quality evaluation result in the first time period according to the collected user's PPG signal, ACC signal, user's age information, and user's gender information, and combine the user's emotional state evaluation result in the first time period The result and the sleep quality evaluation result together obtain the user's emotion monitoring result in the first time period. The above emotion monitoring result combines PPG signal, SEMG signal, ACC signal, user's age information and user's gender information, and also considers the user's Therefore, it is beneficial to obtain more accurate emotional monitoring results, timely warn users of emotions, and then provide protection for users' mental health.

As an optional embodiment, S302 includes: the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user at the first moment among the multiple moments. Feature extraction is performed on the PPG signal, the SEMG signal, the user's age information, and the user's gender information to obtain multiple sets of biometric features of the user at the first moment. Feature fusion is performed on multiple sets of biometric features at the first moment to obtain the fused features of the user at the first moment. Based on the fusion feature at the first moment, the classification result of the user's emotion at the first moment is obtained.

In the embodiment of the present application, the first moment among the multiple moments in the first time period is taken as an example to introduce the emotion classification process at the first moment. It should be understood that the first moment may be any of the multiple moments. At different moments, the user may be affected by the surrounding things to produce changes in behavior and emotion, so each moment has its corresponding PPG signal, SEMG signal and ACC signal.

Exemplarily, the first time period is 24 hours, and the terminal device 100 can classify the user's emotions in this hour according to the collected physiological signals of the user every hour. In this example, the multiple moments are 24 At this moment, the terminal device 100 can obtain 24 emotion classification results within the first time period.

Exemplarily, positive emotions may be represented by a value of 1, calm emotions may be represented by a value of 0, and negative emotions may be represented by a value of -1.

Fig. 4 is a schematic diagram of an emotion classification process provided by an embodiment of the present application. Exemplarily, FIG. 4 shows the emotion classification process at the first moment above. As shown in FIG. 4, the terminal device 100 can perform feature extraction on the collected SEMG signals, PPG signals, age information, and gender information to obtain multiple sets of biological features, and then perform feature fusion on multiple sets of biological features to obtain fusion features, and The fusion features are input into the classification network for classification, and the result of emotion classification is obtained.

Optionally, the terminal device 100 performs feature extraction on the SEMG signal, including preprocessing the SEMG signal, to obtain a preprocessed SEMG signal.

Exemplarily, the preprocessing of the SEMG signal may be filtering the SEMG signal by wavelet transform. In the embodiment of the present application, the biological feature obtained from the filtered SEMG signal may be referred to as an epidermal myoelectric feature.

Optionally, the terminal device 100 performs feature extraction on the user's age information and gender information, including that the terminal device 100 performs preprocessing on the user's age information and gender information. Exemplarily, preprocessing the age information and gender information of the user may be performing numerical processing on the age information and gender information of the user.

Exemplarily, the actual age of the user can be classified and then numerically processed. For example, 0-10 years old can be represented by the value 1, 11-20 years old can be represented by the value 2, and 21-30 years old can be represented by the value 3. 31-40 years old can be represented by the value 4, 41-50 years old can be represented by the value 5, 51-60 years old can be represented by the value 6, 61-70 years old can be represented by the value 7, and those over 70 years old can be represented by the value 8.

Exemplarily, a male can be represented by a value of 0, and a female can be represented by a value of 1.

Optionally, the terminal device 100 performs feature extraction on the PPG signal, including preprocessing the PPG signal. The terminal device 100 can input the preprocessed PPG signal to the feature extraction network for feature extraction to obtain multiple sets of biological features of the user, and can obtain the feature value corresponding to each set of biological features according to the user's biological feature regression.

Exemplarily, the preprocessing of the PPG signal may be to filter the PPG signal, for example, wavelet transform is used to implement the filtering of the PPG signal.

In a possible implementation manner, the feature extraction network may perform signal decomposition and frequency domain signal transformation on the preprocessed PPG signal to extract biological features.

Wherein, when performing signal decomposition on the preprocessed PPG signal, for example, the terminal device 100 may detect the time domain feature of the preprocessed PPG signal, extract the time domain feature value and the optimal band of the preprocessed PPG signal, Perform waveform segmentation on the extracted optimal band to obtain multiple single-period waveforms, and perform sparse decomposition of the single-period waveform to obtain the atomic characteristic parameters of the signal.

Wherein, when performing frequency-domain signal transformation on the preprocessed PPG signal, for example, Fourier transform may be used to obtain the frequency-domain eigenvalue of the PPG signal.

After performing signal decomposition and frequency domain signal transformation on the preprocessed PPG signal, the terminal device 100 may perform feature fusion on the atomic feature parameter feature, time domain feature value, frequency domain feature value and the preprocessed PPG signal.

Exemplarily, feature fusion may be implemented by means of feature concatenation or feature summation.

In a possible implementation manner, the feature extraction network may be a sequential network, for example, a gated recurrent unit (GRU) network. Among them, GRU is a deformed network structure of recurrent neural network (RNN), which can preserve the important characteristics of the input data through the gate function. In the embodiment of the present application, at least one of the user's heart rate feature, breathing feature, blood oxygen feature, blood sugar feature, and blood pressure feature may be extracted through the feature extraction network.

It should be understood that the above manner of extracting features from the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user is only an example, and is not limited in this embodiment of the present application.

From the above description, it can be known that the breathing, heart rate, blood pressure, blood sugar, blood oxygen, electromyographic signals, etc. of different emotions are different, and the changes of these physiological indicators caused by positive emotions and negative emotions are different. The acquired biological characteristics can clearly reflect the changes of biological signals under different emotions, and the interpretability and representation ability are strong.

As an optional embodiment, performing feature fusion on multiple sets of biometric features at the first moment to obtain the fused features of the user at the first moment includes: calculating each of the multiple sets of biological features through a softmax function The weight of the group biometric. According to each set of biometric features and the weight of each set of biometric features, the fused features of the user at the first moment are obtained.

As shown in FIG. 4 , after obtaining multiple sets of biometric features, the terminal device 100 may perform feature fusion on the multiple sets of biometric features to obtain fused features. Exemplarily, the feature fusion in the embodiment of the present application is an adaptive feature fusion, and each set of biological features has a corresponding feature weight. Among them, the feature weight is related to the sample, and the feature weight can be adjusted adaptively according to different samples, which is beneficial to obtain the importance of features of different dimensions, and makes the fusion feature more expressive.

Exemplarily, there are N groups of biological characteristics, and these N groups of biological characteristics are expressed as P=[P ₁ ; P ₂ ; ... P _N ] in the form of a feature matrix, and each group of biological characteristics has M elements, then the feature matrix The dimension of P is N*M, where * represents matrix multiplication.

Exemplarily, the weight of each group of biological characteristics can be calculated by the weight function F(W, P)=softmax(W _M*N *P*P ^T +B _M*1 , axis=0), wherein, W _M*N Represents an M*N-dimensional trainable weight matrix, B _M*1 represents an M*1-dimensional trainable bias matrix, axis=0 represents each row performs softmax separately, and finally obtains an M*N-dimensional weight matrix.

After the weight matrix F(W, P) is obtained, the M*N-dimensional weight matrix F(W, P) is matrix-multiplied with the N*M-dimensional biological characteristic matrix P, so that each group of biological characteristics can be assigned a corresponding After that, the fusion feature matrix can be obtained through the diagonal function Diag(x), that is, the fusion feature matrix feature=Diag(F(W,P)*P), where Diag(x) means taking diagonal elements to form a new matrix.

As an optional embodiment, S303 includes: obtaining the number of negative emotions of the user in the first time period and the duration of each negative emotion from the classification results of emotions at multiple moments. Based on the number of negative emotions and the duration of each negative emotion, the evaluation result of the user's emotional state in the first time period is determined.

In the embodiment of the present application, assuming that the first time period is 24 hours, and the terminal device 100 performs emotion recognition on the user every hour, then the first time period can be divided into 24 sub-time periods, and the duration of each sub-time period is 1 hour. The terminal device 100 can obtain the emotion classification result of each sub-time period based on the PPG signal, SEMG signal, user's age information and gender information collected in each sub-time period, and express the emotion recognition results of these 24 sub-time periods in the form of a set as (0,0,0,0,1,0,0,0,0,-1,0,0,0,1,1,0,0,0,0,-1,-1,0,0, 1), where 0 indicates calm emotion, 1 indicates positive emotion, and -1 indicates negative emotion.

The terminal device 100 may acquire the number of times of negative emotions in the first time period from the set of emotion classification results. In this example, it can be seen that the user has three negative emotions in the first time period. Further, the terminal device 100 can obtain the information of each negative emotion according to changes in respiration, heart rate, blood pressure, blood sugar, blood oxygen, and electromyographic signals reflected in the PPG signal and SEMG signal collected in the sub-time period when the user appears negative emotion. Duration.

Exemplarily, the terminal device 100 continuously acquires the PPG signal and the SEMG signal, and may perform feature extraction on the PPG signal and the SEMG signal every 1s, every 5s, or every 10s, starting at the initial moment when the terminal device 100 detects negative emotions. The duration of the emotion begins to accumulate until the terminal device 100 detects no negative emotion, then the accumulated duration from the beginning moment when the terminal device 100 initially detects the negative emotion to the end moment when the terminal device does not detect the negative emotion is the negative emotion. Duration.

Exemplarily, the evaluation result of the emotional state of the user in the first time period may be expressed in the form of scores. The emotional state evaluation score score_e can be obtained by the following formula:

Among them, T represents the duration of the first time period, n represents the number of times negative emotions appear in the first time period, and t _i represents the duration of the i-th negative emotion. It can be seen from the above formula that the longer the accumulated negative emotion time, the lower the emotional state evaluation score.

Optionally, the terminal device 100 may display real-time emotions obtained at each moment in the first time period to the user. It should be understood that the real-time emotion may be expressed in the form of sentences or images, which is not limited in this embodiment of the present application.

Exemplarily, when the emotion classification result is a positive emotion, the terminal device 100 may display "Keep a good mood!", "Good mood! Keep going!", "Energetic day" or "I'm in a great mood~" and so on.

Exemplarily, when the emotion classification result is a positive emotion, the terminal device 100 may also display patterns such as "sun", "like", "applause" or "cheer".

Exemplarily, when the emotion classification result is negative emotion, for example, the terminal device 100 may display sentences such as "Come on, be happy!", "Sunshine always comes after a storm" or "Dark clouds go away".

Exemplarily, when the emotion classification result is a negative emotion, the terminal device 100 may also display patterns such as "encouragement", "encouragement" or "hug", for example.

It should be understood that the above-mentioned form of expressing emotion is only an example, and other forms may also be used to reflect the user's real-time emotion, which is not limited in this embodiment of the present application.

As an optional embodiment, S304 includes: obtaining the number of sleep interruptions of the user in the first time period based on the components of the ACC signal on the coordinate axis. The heart rate change information of the user in the first time period is acquired based on the PPG signal, and the total sleep duration, deep sleep duration, and light sleep time of the user in the first time period are obtained according to the heart rate change information. Combined with the number of sleep interruptions, total sleep duration, deep sleep duration, and light sleep duration, the sleep evaluation result of the user in the first time period is determined.

In the embodiment of the present application, the gravitational acceleration components of the ACC signal on the x-axis, y-axis and z-axis are different between the moving state and the non-moving state. Wherein, the value of the x-axis direction represents the horizontal movement of the terminal device 100 , the value of the y-axis direction represents the vertical movement of the terminal device 100 , and the value of the z-axis direction represents the spatial vertical direction of the terminal device 100 . In the non-moving state, the modulus value of the gravitational acceleration component of the ACC signal on the x-axis, y-axis and z-axis should be stable at about 9.8. Therefore, during the sleep process of the user, if the terminal device 100 detects that the modulus variation of the gravitational acceleration component of the ACC signal on the x-axis, y-axis and z-axis exceeds the first preset threshold, and the duration is greater than or equal to the second With a preset threshold, the terminal device 100 may consider that the user's sleep is interrupted.

Fig. 5 is a graph of changes in heart rate during sleep according to an embodiment of the present application. As shown in FIG. 5 , when the user goes from falling asleep to light sleep (referred to as light sleep for short) and then to deep sleep (referred to as deep sleep for short), the heart rate gradually decreases slowly. Exemplarily, the user starts to fall asleep at time t1, and is in a light sleep state during the period of time t1-t2, and the duration is denoted by _T1 . After time t2, the user's heart rate tends to be stable, and the user is in a deep sleep state during the period of time t2-t3, and the duration is represented by _T2 . Exemplarily, the heart rate is stable at 55-60 beats/min during deep sleep. After time t3, the user's heart rate begins to rise slowly, and the user is in a light sleep state during the period from t3 to t4. The duration is represented by _T3 until the user gradually wakes up at time t4. Thus, the total sleep duration T_s=T ₁ +T ₂ +T ₃ , the deep sleep duration T_d=T ₂ , and the light sleep duration T_q=T ₁ +T ₃ of the user in the first time period can be obtained.

Combined with the user's total sleep duration, deep sleep duration, light sleep duration, and sleep interruption times, the user's sleep quality evaluation results in the first time period can be obtained. Wherein, the sleep quality evaluation result may be expressed in the form of scores, and the user's sleep quality evaluation score is recorded as score_s. Table 2 shows a possible sleep quality assessment score.

Table II

Score	90～100	80～90	60～80	40～60	<40
total sleep time	7～8 hours	≥9 hours	5～6 hours	3 to 4 hours	1～2 hours
Proportion of deep sleep time	>25%	15%～25%	10%～15%	3%～10%	<3%
Proportion of light sleep duration	<50%	50%～65%	65%～75%	75%～90%	>90%
sleep interruptions	≤1	2～4	4～6	6～8	>8

A logical judgment code for obtaining sleep quality evaluation scores is schematically given below:

As an optional embodiment, S305 includes: obtaining an emotion monitoring result of the user in the first time period according to the emotional state evaluation result and the sleep quality evaluation result. In the case that the emotion monitoring result satisfies the preset condition, it is determined to carry out an emotion early warning.

In the embodiment of the present application, the user's emotional state evaluation results and sleep quality evaluation results can be combined to determine the user's emotion monitoring results in the first time period. Such monitoring is more accurate and comprehensive, and is conducive to improving the accuracy of emotion recognition.

Exemplarily, the terminal device 100 may quantify the emotional state evaluation result and the sleep quality evaluation result to obtain the emotional state evaluation score score_e and the sleep quality evaluation score score_s. The terminal device 100 may perform weighted summation of the emotional state evaluation score score_e and the sleep quality evaluation score score_s to obtain the user's emotional score score_out in the first time period, and when the emotional score score_out is less than or equal to the third preset threshold, Be sure to give an emotional warning.

The sentiment score score_out can be obtained by the following formula:

score_out=α×score_s+(1-α)×score_e

Among them, α represents the correction factor. Exemplarily, the value of α is 0.5.

In a possible implementation, the terminal device 100 may display early warning information to the user, and the early warning information carries the user's emotion monitoring results in the first time period. For example, the terminal device 100 displays the emotional score to the user, and displays the corresponding Emotional statements to remind users to pay attention to emotional health.

In another possible implementation manner, in the terminal device system architecture 600 shown in FIG. Wherein, the terminal device 110 is a terminal device connected to the terminal device 100 through a network, for example, the terminal device 100 and the terminal device 110 are connected through Bluetooth.

Exemplarily, the above warning information may include the user's PPG signal, ACC signal and SEMG signal, and the terminal device 110 may combine the age information and gender information input by the user on the terminal device 110 according to the obtained PPG signal, ACC signal and SEMG signal , to obtain the user's emotion monitoring result, and display the emotion monitoring result and/or the corresponding emotional statement to the user so as to serve as an early warning to the user.

Exemplarily, the early warning information may include the emotion monitoring results obtained by the terminal device 100 according to the user's PPG signal, ACC signal and SEMG signal, and the terminal device 110 may display the emotion monitoring results and/or corresponding emotional statements to the user to play a role in the user's actions. The role of early warning.

Table 3 shows possible display contents of the terminal device 100 or the terminal device 110 under different scores.

Table three

sentiment score	Show statement example
90～100	You are in a good mood today, please keep it up
70～90	the sun is always behind the storm
<70	How can you see a rainbow without experiencing wind and rain

Exemplarily, the emotional state evaluation results and the sleep quality evaluation results are presented in the form of evaluation grades. For example, the emotional state evaluation results and sleep quality evaluation results are divided into five levels: A, B, C, D, and E. When the user's emotional state evaluation results are A or B, and the sleep quality evaluation results are A or B, the terminal device 100 It can be obtained that the emotion monitoring result of the user in the first time period is a high level, indicating that the user is in a good mood. When the evaluation result of the user's emotional state is C and the evaluation result of sleep quality is C, the terminal device 100 may obtain that the user's emotion monitoring result in the first time period is at a medium level, indicating that the mood is normal. When the user's emotional state evaluation result is D or E, and the sleep quality evaluation result is D or E, the terminal device 100 may obtain that the user's emotional monitoring result in the first time period is low, indicating abnormal or negative mood.

Exemplarily, when the terminal device 100 obtains that the user's mood monitoring result in the first time period is low, the terminal device 100 may send an early warning message to the terminal device 110 .

It should be understood that the terminal device 100 may display at least one of emotional state evaluation results, sleep quality evaluation results, or emotional monitoring results to the user, which is not limited in this embodiment of the present application.

It should also be understood that the above-mentioned emotional state evaluation results, sleep quality evaluation results, and emotional monitoring results are only examples, and can also be displayed to users in other forms such as data histograms, data pie charts, line graphs, and emoticons. The embodiment does not limit this.

Optionally, the terminal device 100 may store the early warning information in the cloud, and the terminal device 110 may periodically obtain the early warning information from the cloud. Exemplarily, the terminal device 100 stores the user's emotional monitoring results every 24 hours in the cloud, and the terminal device 110 can obtain early warning information from the cloud every 24 hours, every 48 hours, or every 72 hours, and The emotion monitoring results show the corresponding warning content to the user.

Exemplarily, if the emotional monitoring results show that the user's emotional score in the past 24 hours, 48 hours or 72 hours is low or low, the terminal device 110 may consider using the health application program (application, APP) displays words encouraging the user to refuel, and can also display the user's emotional state evaluation results and/or sleep quality evaluation results through the health APP.

For example, if the emotional monitoring results show that the user's emotional score in the past 24 hours, 48 hours, or 72 hours is low or at a low level, the terminal device 110 can also warn the user through the monitoring APP, in a tactful manner Prompt users for communication or psychological counseling. For example, the monitoring APP can be used to display sentences such as "The weather is really nice, go out and walk more", "Chat with friends and family often", or "Let's find someone to talk about bad emotions" to the user.

In a possible scenario, the terminal device 100 can be a smart watch worn by an underage user, the terminal device 110 can be a parent's mobile phone, and the parent's mobile phone is associated with the underage user's smart watch, and the parent can obtain the information of the underage user through the mobile phone. According to at least one of the user's emotional state evaluation results, sleep quality evaluation results, or emotional monitoring results, timely communication and psychological counseling are provided to underage users.

For example, if the emotional monitoring results show that the user's emotional score in the past 24 hours, 48 hours or 72 hours is low or at a low level, the terminal device 110 can also use the video APP and music APP on the terminal device 110 to Or a news app can intelligently recommend positive multimedia resources to users, thereby alleviating the negative emotions of users.

It should be understood that the period (24 hours, 48 hours or 72 hours) for the terminal device 110 to obtain the early warning information from the cloud is only an example, and the embodiment of the present application does not limit the period for obtaining the early warning information.

As an optional embodiment, the method 300 further includes: determining the user's emotional age according to the user's emotion monitoring result in the first time period, the number of negative emotions, and the user's age information. At least one of emotional age, emotional monitoring results, or emotional statements is displayed to the user.

In the embodiment of the present application, the terminal device 100 can obtain the user's emotional age, so that the user can understand his own ability to control emotions based on the emotional age, which is beneficial to improving user experience.

Table 4 shows a corresponding relationship between emotion score, number of negative emotions and emotional age, which is for reference only.

Table four

sentiment score	Number of Negative Emotions x	emotional age
80～100	0	wisdom period
80～100	1≤x≤3	stable period
80～100	≥3	Childhood
<70	-	active period

In the case where the physical structure of the terminal device 100 is a smart watch 200, FIG. 7 is a schematic interface diagram of a smart watch 200 provided in an embodiment of the present application. As shown in FIG. 7 , the smart watch 200 can display the user's emotional age, emotional score and corresponding emotional statements. For example, the user's emotional score is 98 points, the emotional age is a stable period, and the emotional statement is "keep a good mood!".

As an optional embodiment, the method 300 further includes: acquiring peaks and valleys of the ACC signal. Determine whether the user is in a non-exercise state based on the peaks and troughs of the ACC signal. S302 includes: when it is determined that the user is in a non-exercise state, combining the PPG signal, SEMG signal, user's age information and user's gender information, and counting the classification results of the user's emotions at multiple moments within the first time period .

In the embodiment of the present application, since the PPG signal is significantly disturbed by motion, emotion monitoring needs to be performed in a non-exercise state. The terminal device 100 can determine whether the PPG signal and the SEMG signal are stable signals collected when the user is in a non-exercising state according to the peak and trough of the ACC signal.

FIG. 8 is a schematic diagram of a waveform of an ACC signal provided by an embodiment of the present application. Exemplarily, the duration of the ACC signal recognized by the terminal device 100 is 10s. As shown in FIG. 8 , the waveform of the ACC signal within the recognition duration has 1 peak and 1 valley, wherein the peak value is W ₁ , and the value of the valley is W _2. If the difference between W ₁ and W ₂ is less than or equal to the fourth preset threshold, and the sum of the number of peaks and valleys is less than or equal to the fifth preset threshold, then the terminal device 100 can determine that the user is in a non-exercise state, and collect The PPG signal and SEMG signal are available signals and can be used for subsequent emotional monitoring.

FIG. 9 is a schematic flow chart of another emotion monitoring method 900 provided by an embodiment of the present application. The method 900 can be executed by the terminal device 100, and the method 900 includes the following steps:

S901. Obtain the user's PPG signal, SEMG signal, ACC signal, age information of the user, and gender information of the user in real time.

In this step, the PPG signal can be obtained by the pulse wave sensor, the SEMG signal can be obtained by the myoelectric electrode sensor, and the ACC signal can be obtained by the acceleration sensor.

S902. Determine whether the user is in a non-exercise state according to the ACC signal. When the user is in a non-exercise state, execute S903. If the user is in a state of exercise, the collected PPG signal and SEMG signal are invalid signals and need to be collected again.

S903. Classify the user's emotion according to the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user, and obtain an emotion classification result.

In this step, the terminal device 100 may periodically perform emotion recognition on the collected PPG signal, SEMG signal, user's age information, and user's gender information to obtain multiple emotion classification results at different times. Exemplarily, the terminal device 100 can perform feature extraction on PPG signals, SEMG signals, user age information, and user gender information, and perform feature fusion on the extracted biometric features. The specific feature extraction and feature fusion processes have been described above. description and will not be repeated here.

It should be understood that the emotion classification results in this step include multiple emotion classification results at different moments.

S904. According to the number of occurrences of negative emotions counted in the emotion classification results and the duration of each negative emotion, the user's emotional state evaluation score is obtained.

In this step, it is assumed that the emotion classification results include the user’s emotion classification results for each hour in 24 hours, and the user’s emotional state evaluation in 24 hours can be obtained according to the number of negative emotions that have been counted within 24 hours and the duration of a single session. point.

S905: Monitor the sleep quality of the user according to the PPG signal, the ACC signal, the user's age information, and the user's gender information, and obtain the user's sleep quality evaluation score.

In this step, the terminal device 100 can obtain the number of sleep interruptions of the user according to the ACC signal, and obtain the time when the user enters sleep, the time when the user exits sleep, the duration of deep sleep, and the duration of light sleep according to the change of the user's heart rate obtained from the PPG signal, and then The user's sleep quality is evaluated by combining the number of sleep interruptions, the time of entering sleep, the time of exiting sleep, the duration of deep sleep, the duration of light sleep, and the user's age and gender to obtain the user's sleep quality evaluation score. Among them, the movement state lasting more than 5s can be regarded as sleep interruption.

S906. Perform weighted summation of the user's emotional state evaluation score and sleep quality evaluation score to obtain the user's emotional score.

S907. When the user's emotional score is less than or equal to the emotional threshold, send early warning information to the cloud, where the early warning information carries at least one of the user's emotional state evaluation score, sleep quality evaluation score, or emotional score.

In this step, the terminal device 100 can send early warning information to the cloud when it detects that the user's emotional score is low, and the terminal device 110 associated with the terminal device 100 can obtain the early warning information from the cloud to further determine whether the user's emotional score is low. 110 provides emotional alerts to users, and uses APP to push positive energy information and display encouraging sentences to remind users to pay attention to emotional health.

S908. Obtain the user's emotional age according to the user's emotional score and the number of occurrences of negative emotions.

S909, displaying at least one of emotional age, emotional score, or emotional sentence to the user.

Optionally, the terminal device 100 may also send the emotional age to the terminal device 110, and the terminal device 110 displays at least one of the emotional age, the emotional score, or the emotional statement.

In the embodiment of the present application, when the terminal device 100 determines that the collected PPG signal and SEMG signal are relatively stable according to the ACC signal, it can obtain the biological characteristics of the user under different emotions according to the user's PPG signal and SEMG signal, such as heart rate, Blood pressure, blood oxygen, blood sugar, muscle contraction, etc., combined with the user's age and gender to evaluate the emotional state. Since the user's sleep conditions may be different under different emotions, the terminal device 100 can also monitor the user's sleep quality according to the PPG signal, ACC signal, user's age and gender, and obtain the user's sleep quality evaluation result, and further combine the user's The emotional state evaluation results and the sleep quality evaluation results determine the user's emotional score. The emotional score obtained in this way can more accurately reflect the user's emotions in a period of time, which is conducive to improving the accuracy of emotion recognition.

It should be understood that the sequence numbers of the above processes do not mean the order of execution, and the execution order of each process should be determined by its functions and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

The emotion monitoring method according to the embodiment of the present application is described in detail above with reference to FIG. 1 to FIG. 9 , and the emotion monitoring device according to the embodiment of the present application will be described in detail below in conjunction with FIG. 10 and FIG. 11 .

FIG. 10 shows a schematic block diagram of an emotion monitoring device 1000 provided by an embodiment of the present application, and the device 1000 includes an acquisition module 1010 and a processing module 1020 .

Wherein, the acquiring module 1010 is configured to: acquire the user's photoplethysmography PPG signal in real time through the pulse wave sensor, acquire the user's epidermal electromyography SEMG signal in real time through the myoelectric electrode sensor, and acquire the user's ACC signal in real time through the acceleration sensor. The processing module 1020 is used to: combine the PPG signal, the SEMG signal, the user's age information and the user's gender information, and count the classification results of the user's emotions at multiple moments within the first time period. The classification results include positive emotions, calm emotions, or positive emotions. Negative emotions; based on the classification results of the user's emotions at multiple moments in the first time period, determine the user's emotional state evaluation results in the first time period; combine PPG signal, ACC signal, user's age information and user's gender information , determine the sleep quality evaluation result of the user in the first time period; and, determine whether to perform an emotional warning according to the emotional state evaluation result and the sleep quality evaluation result.

Optionally, the processing module 1020 is configured to: perform feature extraction on the PPG signal, the SEMG signal, the user's age information, and the user's gender information at the first moment among the multiple moments, to obtain multiple sets of biological characteristics of the user at the first moment . Performing feature fusion on multiple sets of biometric features at the first moment to obtain the fused features of the user at the first moment; and, based on the fused features at the first moment, obtaining a classification result of the user's emotions at the first moment.

Optionally, the processing module 1020 is configured to: use the flexible maximum transfer function to calculate the weight of each set of biological features in multiple sets of biological features; and, according to each set of biological features and the weight of each set of biological features, obtain the fusion features.

Optionally, the multiple sets of biological characteristics include at least one of the following: heart rate characteristics, breathing characteristics, blood oxygen characteristics, blood glucose characteristics, blood pressure characteristics, or epidermal myoelectric characteristics.

Optionally, the obtaining module 1010 is configured to: obtain the number of negative emotions of the user in the first time period and the duration of each negative emotion from the classification results of emotions at multiple moments. The processing module 1020 is configured to: determine the evaluation result of the user's emotional state in the first time period based on the number of negative emotions and the duration of each negative emotion.

Optionally, the processing module 1020 is configured to: obtain the number of sleep interruptions of the user in the first time period based on the components of the ACC signal on the coordinate axis. The obtaining module 1010 is configured to: obtain the heart rate change information of the user in the first time period based on the PPG signal. The processing module 1020 is used to: obtain the total sleep duration, deep sleep duration, and light sleep duration of the user in the first time period according to the heart rate change information; and, combining the number of sleep interruptions, total sleep duration, deep sleep duration, and light sleep duration, Determine the sleep evaluation result of the user in the first time period.

Optionally, the processing module 1020 is configured to: obtain the emotional monitoring result of the user in the first time period according to the emotional state evaluation result and the sleep quality evaluation result; and, when the emotional monitoring result satisfies the preset condition, determine Be emotionally alert.

Optionally, the processing module 1020 is configured to: perform a weighted summation of the quantized value of the emotional state evaluation result and the quantized value of the sleep quality evaluation result to obtain the user's emotional score in the first time period; and, when the emotional score is less than Or when it is equal to the preset threshold, it is determined to perform an emotional warning.

Optionally, the processing module 1020 is configured to: determine the user's emotional age according to the user's emotion monitoring results in the first time period, the number of negative emotions, and the user's age information; and display the emotional age, emotion monitoring results or emotional sentences at least one of the

Optionally, the acquiring module 1010 is configured to: acquire peaks and valleys of the ACC signal. The processing module 1020 is used to: determine whether the user is in a non-exercise state according to the peak and trough of the ACC signal; and, when the user is in a non-exercise state, combine the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user , to collect statistics on the classification results of the user's emotions at multiple moments in the first time period.

In an optional example, those skilled in the art may understand that the apparatus 1000 may specifically be the terminal device in the foregoing embodiment, or the functions of the terminal device in the foregoing embodiment may be integrated in the apparatus 1000 . The above functions can be implemented by hardware, or can be implemented by executing corresponding software by hardware. The hardware or software includes one or more modules corresponding to the above functions. Apparatus 1000 may be configured to execute various processes and/or steps corresponding to the terminal device in the foregoing method embodiments.

It should be understood that the apparatus 1000 here is embodied in the form of functional modules. The term "module" here may refer to an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group processor, etc.) and memory, incorporated logic, and/or other suitable components to support the described functionality. In the embodiment of the present application, the device 1000 in FIG. 10 may also be a chip or a chip system, for example: a system on chip (system on chip, SoC).

Fig. 11 shows a schematic block diagram of another emotion monitoring device 1100 provided by an embodiment of the present application. The apparatus 1100 includes a processor 1110 , a transceiver 1120 and a memory 1130 . Wherein, the processor 1110, the transceiver 1120 and the memory 1130 communicate with each other through an internal connection path, the memory 1130 is used to store instructions, and the processor 1110 is used to execute the instructions stored in the memory 1130 to control the transceiver 1120 to send signals and /or to receive a signal.

It should be understood that the apparatus 1100 may specifically be the terminal device in the foregoing embodiments, or, the functions of the terminal device in the foregoing embodiments may be integrated into the apparatus 1100, and the apparatus 1100 may be used to perform various functions corresponding to the terminal apparatus in the foregoing method embodiments. steps and/or processes. Optionally, the memory 1130 may include read-only memory and random-access memory, and provides instructions and data to the processor. A portion of the memory may also include non-volatile random access memory. For example, the memory may also store device type information. The processor 1110 may be configured to execute instructions stored in the memory, and when the processor executes the instructions, the processor may execute various steps and/or processes corresponding to the electronic device in the foregoing method embodiments.

It should be understood that, in the embodiment of the present application, the processor 1110 may be a central processing unit (central processing unit, CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits ( ASIC), Field Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.

In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the methods disclosed in connection with the embodiments of the present application may be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register. The storage medium is located in the memory, and the processor executes the instructions in the memory, and completes the steps of the above method in combination with its hardware. To avoid repetition, no detailed description is given here.

Those skilled in the art can appreciate that the modules and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed by hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may use different methods to implement the described functions for each specific application, but such implementation should not be regarded as exceeding the scope of the present application.

Those skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the above-described system, device, and module can refer to the corresponding process in the foregoing method embodiment, and details are not repeated here.

In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the modules is only a logical function division. In actual implementation, there may be other division methods. For example, multiple modules or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or modules may be in electrical, mechanical or other forms.

The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or may be distributed to multiple network modules. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module in each embodiment of the present application may be integrated into one processing module, each module may exist separately physically, or two or more modules may be integrated into one module.

If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (read-only memory, ROM), random access memory (random access memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only the specific implementation of the application, but the protection scope of the embodiment of the application is not limited thereto, and any skilled person familiar with the technical field can easily think of changes within the technical scope disclosed in the embodiment of the application Or replacement, should be covered within the scope of protection of the embodiments of the present application. Therefore, the scope of protection of the embodiments of the present application should be based on the scope of protection of the claims.

Claims (23)

1. An emotion monitoring method, characterized in that it is applied to a terminal device deployed with a pulse wave sensor, an electromyography electrode sensor and an acceleration sensor, the method comprising:

   Obtain the user's photoplethysmography PPG signal in real time through the pulse wave sensor, obtain the user's epidermal electromyography SEMG signal in real time through the myoelectric electrode sensor, and obtain the user's ACC signal in real time through the acceleration sensor;

   Combining the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user, the classification results of the user's emotions at multiple moments within the first time period are counted, and the classification results include Positive, calming or negative emotions;

   Based on the classification results of the user's emotions at multiple moments in the first time period, determine the user's emotional state evaluation results in the first time period;

   Combining the PPG signal, the ACC signal, the age information of the user, and the gender information of the user, determine the sleep quality evaluation result of the user in the first time period;

   According to the evaluation result of the emotional state and the evaluation result of the sleep quality, it is determined whether to carry out an emotional warning.
2. The method according to claim 1, characterized in that, combining the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user to count the user's time in the first time period The classification results of emotions at multiple moments, including:

   performing feature extraction on the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user at the first moment of the plurality of moments to obtain multiple sets of information about the user at the first moment biometrics;

   performing feature fusion on multiple sets of biometric features at the first moment to obtain the fused features of the user at the first moment;

   Based on the fusion feature at the first moment, a classification result of the user's emotion at the first moment is obtained.
3. The method according to claim 2, wherein the feature fusion of multiple sets of biological features at the first moment to obtain the fusion features of the user at the first moment includes:

   calculating a weight for each set of biometrics in the plurality of sets of biometrics via a softmax transfer function;

   According to each set of biometric features and the weight of each set of biometric features, the fused features of the user at the first moment are obtained.
4. The method according to claim 2 or 3, wherein the multiple sets of biological characteristics include at least one of the following:

   Heart rate characteristics, respiratory characteristics, blood oxygen characteristics, blood glucose characteristics, blood pressure characteristics or epidermal myoelectric characteristics.
5. The method according to any one of claims 1-4, characterized in that, based on the classification results of the user's emotions at multiple moments within the first time period, it is determined that the user is in the Emotional state evaluation results in the first time period, including:

   Obtain the number of negative emotions of the user in the first time period and the duration of each negative emotion from the classification results of emotions at the multiple moments;

   Based on the number of negative emotions and the duration of each negative emotion, determine the user's emotional state evaluation result in the first time period.
6. The method according to any one of claims 1-5, wherein the user is determined by combining the PPG signal, the ACC signal, the age information of the user, and the gender information of the user. The sleep quality evaluation results in the first time period include:

   Obtaining the number of sleep interruptions of the user in the first time period based on the components of the ACC signal on the coordinate axis;

   Obtaining heart rate change information of the user in the first time period based on the PPG signal;

   Obtain the total sleep duration, deep sleep duration, and light sleep duration of the user in the first time period according to the heart rate change information;

   Combining the number of sleep interruptions, the total sleep duration, the deep sleep duration, and the light sleep duration, determine the sleep evaluation result of the user in the first time period.
7. The method according to any one of claims 1-6, wherein the determining whether to carry out an emotional early warning according to the evaluation result of the emotional state and the evaluation result of the sleep quality comprises:

   Obtaining an emotion monitoring result of the user in the first time period according to the emotional state evaluation result and the sleep quality evaluation result;

   In the case that the emotion monitoring result satisfies the preset condition, it is determined to carry out an emotion early warning.
8. The method according to claim 7, wherein the obtaining the emotional monitoring result of the user in the first time period according to the emotional state evaluation result and the sleep quality evaluation result includes:

   performing weighted summation of the quantified value of the emotional state evaluation result and the quantized value of the sleep quality evaluation result to obtain the emotional score of the user in the first time period;

   In the case where the emotional monitoring result satisfies the preset condition, determining to carry out an emotional early warning includes:

   If the emotional score is less than or equal to a preset threshold, it is determined to perform an emotional early warning.
9. The method according to claim 7 or 8, characterized in that the method further comprises:

   determining the user's emotional age according to the user's emotional monitoring results in the first time period, the number of negative emotions, and the user's age information;

   Displaying at least one of the emotion age, the emotion monitoring result or the emotion statement.
10. The method according to any one of claims 1-9, further comprising:

    Obtain the peak and trough of the ACC signal;

    determining whether the user is in a non-exercise state according to the peak and trough of the ACC signal;

    The combining the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user, and counting the classification results of the user's emotions at multiple moments within the first time period include:

    When the user is in a non-exercise state, combine the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user to count the multiple The classification results of emotions under the moment.
11. A mood monitoring device, characterized in that it comprises:

    The acquisition module is used to obtain the user's photoplethysmography PPG signal in real time through the pulse wave sensor, obtain the user's epidermal electromyography SEMG signal in real time through the myoelectric electrode sensor, and obtain the user's ACC signal in real time through the acceleration sensor;

    A processing module, configured to combine the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user to count the classification results of the user's emotions at multiple moments within the first time period, The classification results include positive emotions, calm emotions or negative emotions;

    The processing module is further configured to: determine the evaluation result of the user's emotional state in the first time period based on the classification results of the user's emotions at multiple moments in the first time period;

    The processing module is further configured to: combine the PPG signal, the ACC signal, the age information of the user, and the gender information of the user to determine the sleep quality evaluation result of the user in the first time period;

    The processing module is further configured to: determine whether to carry out an emotional warning according to the evaluation result of the emotional state and the evaluation result of the sleep quality.
12. The device according to claim 11, wherein the processing module is used for:

    performing feature extraction on the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user at the first moment of the plurality of moments to obtain multiple sets of information about the user at the first moment biometrics;

    performing feature fusion on multiple sets of biometric features at the first moment to obtain the fused features of the user at the first moment;

    Based on the fusion feature at the first moment, a classification result of the user's emotion at the first moment is obtained.
13. The device according to claim 12, wherein the processing module is used for:

    calculating a weight for each set of biometrics in the plurality of sets of biometrics via a softmax transfer function;

    According to each set of biometric features and the weight of each set of biometric features, the fused features of the user at the first moment are obtained.
14. The device according to claim 12 or 13, wherein the multiple sets of biological characteristics include at least one of the following:

    Heart rate characteristics, respiratory characteristics, blood oxygen characteristics, blood glucose characteristics, blood pressure characteristics or epidermal myoelectric characteristics.
15. The device according to any one of claims 11-14, wherein the acquisition module is used for:

    Obtain the number of negative emotions of the user in the first time period and the duration of each negative emotion from the classification results of emotions at the multiple moments;

    The processing module is configured to: determine the user's emotional state evaluation result in the first time period based on the number of negative emotions and the duration of each negative emotion.
16. The device according to any one of claims 11-15, wherein the processing module is used for:

    Obtaining the number of sleep interruptions of the user in the first time period based on the components of the ACC signal on the coordinate axis;

    The obtaining module is configured to: obtain the heart rate change information of the user in the first time period based on the PPG signal;

    The processing module is further configured to: obtain the total sleep duration, deep sleep duration, and light sleep duration of the user in the first time period according to the heart rate change information;

    The processing module is further configured to: combine the sleep interruption times, the total sleep duration, the deep sleep duration, and the light sleep duration to determine the sleep evaluation result of the user in the first time period.
17. The device according to any one of claims 11-16, wherein the processing module is configured to:

    Obtaining an emotion monitoring result of the user in the first time period according to the emotional state evaluation result and the sleep quality evaluation result;

    In the case that the emotion monitoring result satisfies the preset condition, it is determined to carry out an emotion early warning.
18. The device according to claim 17, wherein the processing module is used for:

    performing weighted summation of the quantified value of the emotional state evaluation result and the quantized value of the sleep quality evaluation result to obtain the emotional score of the user in the first time period;

    If the emotional score is less than or equal to a preset threshold, it is determined to perform an emotional early warning.
19. The device according to claim 17 or 18, wherein the processing module is used for:

    determining the user's emotional age according to the user's emotional monitoring results in the first time period, the number of negative emotions, and the user's age information;

    Displaying at least one of the emotion age, the emotion monitoring result or the emotion statement.
20. The device according to any one of claims 11-19, wherein the acquiring module is configured to:

    Obtain the peak and trough of the ACC signal;

    The processing module is configured to: determine whether the user is in a non-exercise state according to the peak and trough of the ACC signal;

    The processing module is further configured to: when the user is in a non-exercise state, combine the PPG signal, the SEMG signal, the age information of the user, and the gender information of the user to make statistics on the user's Classification results of emotions at multiple moments in the first time period.
21. An emotion monitoring device, characterized in that it includes: a processor, the processor is coupled with a memory, the memory is used to store a computer program, when the processor invokes the computer program, the device executes the following steps: The method of any one of claims 1-10.
22. A computer-readable storage medium, characterized by being used for storing a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1-10.
23. A computer program product, characterized in that the computer program product includes computer program code, and when the computer program code is run on a computer, the computer implements the method according to any one of claims 1-10 .

Images