WO2022100187A1

WO2022100187A1 - Mobile terminal-based method for identifying and monitoring emotions of user

Info

Publication number: WO2022100187A1
Application number: PCT/CN2021/113012
Authority: WO
Inventors: 李志刚; 李斯羽; 问静波; 齐振翮; 张娜娜
Original assignee: 西北工业大学
Priority date: 2020-11-11
Filing date: 2021-08-17
Publication date: 2022-05-19
Also published as: CN112370058A

Abstract

A mobile terminal-based method for identifying and monitoring the emotions of a user, which is different from carrying out single prediction in the traditional sense purely on the basis of expression, behavior and language, rather, in said method, various pieces of data generated by a user, such as communication, sleep, APP usage and the like, are collected by means of a mobile phone and a bracelet that the user carries daily, a data set that covers the user's life to a high degree, and then an accurate perception of the emotions of the user is completed. During data collection, the collection process is simplified and user-friendliness is improved, emotional features of a user are described from a multi-dimensional perspective to improve the accuracy of recognition results; meanwhile, in a data processing stage, key features are extracted for modeling, reducing the complexity of a subsequent machine learning algorithm and thus shortening the algorithm running time.

Description

A method for identifying and monitoring users' emotions based on mobile terminals

technical field

The present invention relates to the field of data processing, in particular to a method for identifying and monitoring a user's emotion based on a mobile terminal.

Background technique

At present, user emotion recognition is mainly divided into two modules: user emotion recognition based on traditional methods (face-based user emotion recognition, text-based user emotion recognition, voice-based user emotion recognition and physiological feature-based user emotion recognition) and User emotion recognition based on a novel method (user emotion recognition based on gesture behavior).

The traditional emotion recognition method has good applicability for capturing the user's instantaneous emotion in scientific research, but in real life, the user's emotion may change every moment. Compared with the analysis of instantaneous emotion, Users may be more inclined to get the average sentiment over a certain period of time. User emotion recognition based on traditional methods only describes user emotion from a single perspective, which reduces the accuracy of emotion recognition. At the same time, user emotion recognition based on traditional methods sometimes needs to wear complex equipment for data collection. For example, user emotion analysis based on user brain waves requires users to wear a responsible EEG collection device to collect physiological characteristics. In addition, in the subsequent training of the network, the current emotion recognition based on traditional methods uses open source data sets to train the network, and the data may have problems such as low naturalness and poor soundness, which will directly affect emotions recognition accuracy.

SUMMARY OF THE INVENTION

In view of the above defects, the present invention provides a method for identifying and monitoring users' emotions based on a mobile terminal. The present invention collects the diversified daily data generated by mobile terminal devices such as smart phones and smart bracelets, and extracts characteristic values. Create a model to describe the user's emotions as accurately as possible from multiple perspectives without affecting the user's daily work, and at the same time provide the analysis results to other programs in need for use.

The technical scheme of the present invention is: a method for identifying and monitoring the emotion of a user based on a mobile terminal, the method comprising the following steps:

Step 1: collect user data to generate a data set; the data heart rate data and emotion data; the data set includes a heart rate data set and an emotion data set;

Step 2: Preliminary data processing, the preliminary processing includes supplementing the missing characteristic values of some data in the data set, integrating the data, and extracting corresponding characteristic values;

Step 3: Perform emotion recognition and select a model for recognition.

Further, a method for identifying and monitoring a user's emotion based on a mobile terminal, the heart rate data is divided into daytime heart rate data and nighttime heart rate data, the daytime heart rate data is obtained by a heart rate sensor; the nighttime heart rate data is obtained by Triaxial accelerometer and heart rate sensor are obtained.

Further, a method for identifying and monitoring user's emotion based on a mobile terminal, wherein the emotion data includes a positive class, a general class, and a negative class.

Further, in a mobile terminal-based method for identifying and monitoring the user's emotion, the feature value is supplemented by an automatic strategy based on other known values and an average value.

Further, a method for identifying and monitoring the user's emotion based on the mobile terminal, the extraction of the corresponding characteristic value is that after the data classification collected by the APP carries out graphic visualization observation, the corresponding characteristic value is integrated and extracted for part of the data.

Further, a method for identifying and monitoring a user's emotion based on a mobile terminal, wherein the emotion feature involves an APP index, a physiological index, a sleep index, and a communication index;

The emotional features are described as:

in

for APP indicators,

for physiological indicators,

for sleep indicators,

for communication indicators.

Further, a method for identifying and monitoring a user's emotion based on a mobile terminal, the model includes an SVM model and a neural network model.

The invention collects the daily data of the user through the smart phone and the smart bracelet, finds out the characteristic data for network training, thus obtains the mapping relationship between the user's external performance data and the internal emotion, and identifies the user's internal emotion. Collecting users' daily data based on mobile devices can solve the problems of low data naturalness and poor soundness in traditional emotion recognition. First, the high popularity of smartphones in my country can ensure the amount of data that can be collected; second, different people use different mobile phones, which can ensure the specificity of data and the soundness of experimental results; finally, because mobile phones are portable and easy to carry, different Compared with traditional emotion recognition, the new type of emotion recognition based on smart phones can easily obtain the data generated by the user daily to recognize the user's emotion without the need for the user to wear an additional data collection device.

The present invention is a method for identifying and monitoring the user's emotion based on the mobile terminal, which is different from the traditional single prediction based on expression, behavior and language. Various data generated by the user, such as communication, sleep, APP usage, etc., are collected to build a data set that covers the user's life, and then complete the accurate perception of the user's emotions. In the process of data collection, the invention simplifies the collection process, improves user friendliness, describes the user's emotional characteristics from a multi-dimensional perspective, improves the accuracy of the recognition result, and at the same time in the data processing stage, extracts key features for modeling, reducing subsequent The complexity of the machine learning algorithm reduces the algorithm running time.

Description of drawings

Figure 1 is a flowchart of emotion recognition and monitoring based on mobile terminal users.

Detailed ways

The technical solution of the present invention is further described below with reference to the accompanying drawings: as shown in FIG. 1 , a method for identifying and monitoring a user's emotion based on a mobile terminal includes the following steps.

Step 1: Collect user data to generate a data set; the data heart rate data and emotion data; the data set includes a heart rate data set and an emotion data set.

In order to obtain a more prepared result, the heart rate data is divided into daytime heart rate data and nighttime heart rate data, the daytime heart rate data is obtained by a heart rate sensor; the nighttime heart rate data is obtained by a triaxial acceleration sensor and a heart rate sensor. The emotion data includes positive class, general class, and negative class.

Specifically, first design a mobile APP to collect the data generated by users every day. Record the usage time of α kinds of applications through UsageStats through the APP, and get the collection

The pedometer function is realized through the newly added STEP DETECTOR and STEP COUNTER sensors of Android, and the number of steps s and movement distance d of the user in the day is recorded, and the collection is obtained.

Collect the user's total communication duration ct and times cf through PhoneStateListener to get the data

Secondly, design a bracelet APP to record the daytime heart rate data through the heart rate sensor carried on the bracelet

Set the time node to record the nighttime heart rate data through the three-axis acceleration sensor and heart rate sensor carried by the bracelet

and triaxial acceleration data

Collect the user's daily emotional data, divide the emotions into three categories: 1→positive, 0→normal, -1→negative, and get the user's daily emotional data set

According to the daily emotional changes of users corresponding to the data generated by the mobile terminal every day, the data set can be obtained:

Step 2: Preliminary data processing, which includes supplementing some data in the data set with missing eigenvalues, integrating the data, and extracting corresponding eigenvalues; the eigenvalues are supplemented using automatic strategies based on other known data. The average value is used for supplementation; the extraction of the corresponding feature value is that after the data collected by the APP is classified for graphic visualization, the corresponding feature value is extracted from the integration of some data; the emotional feature involves the APP index and the physiological index. , sleep indicators, communication indicators.

The specific method is as follows: First, the collected data is managed, which is mainly divided into three parts: First, for the situation that some data in the data set have missing eigenvalues, an automatic strategy is used here based on other known values. Supplement; second, data integration is performed. After the data collected by the APP is classified and visualized, the corresponding feature values are extracted from the integration of some data.

APP indicator

The APP indicators include the type of APP, the usage time of the APP, and the number of times the APP is used. Considering the types of APPs on the market and the impact on users' emotions, APPs are currently only divided into three categories: work, entertainment, and social interaction. According to different APP types, the total working hours of the user in the day can be obtained

Total entertainment time of the day

Total social time of the day

Considering the different degrees of influence of the usage time of different APP types on user emotions, the APP indicators can be obtained by weighting the total duration of different types of APPs:

Physiological indicators

First, obtain the daily movement distance s and steps d of the user through the mobile APP, and get the set

Secondly, design a bracelet APP, set the time node through the heart rate sensor carried on the bracelet, collect heart rate data and three-axis acceleration every 1h, and obtain the maximum heart rate set:

Take the variance of the daytime heart rate data to get the daytime heart rate abnormal value:

Physiological indicators can then be obtained:

sleep indicators

Set the time node, collect heart rate data and triaxial acceleration every 30 minutes at night, and get the maximum heart rate set:

At the same time, the data of the corresponding three-axis acceleration of the bracelet in the same time interval at night are collected, and the data set is obtained:

The obtained three-axis accelerated data set is processed as follows to obtain parameters

value:

Get the dataset corresponding to each time interval at night

First, determine whether the user is in a sleep state or an awake state. The three-axis acceleration sensor is the mainstay, and the heart rate is assisted. When the acceleration calculation weight is less than the heart rate, that is

When the user is in sleep state, summing the time can get the total sleep time of the user

The sleep state can be further classified, with the heart rate as the leading factor and the acceleration sensor as the auxiliary, and the sleep time can be divided into deep sleep time and light sleep time:

And then get the sleep indicator:

communication indicators

communication data

represents the communication time, and

Indicates the number of communications, and the average duration of communications can be obtained

Based on the above indicators, the emotional characteristics are described as follows:

The meanings of the above symbols are as follows:

——The total duration of the application ∝ used by the i-th user on the j-th day;

——The total number of steps of the i-th user on the j-th day;

——The movement distance of the i-th user on the j-th day;

——The total communication time of the i-th user on the j-th day;

——The total number of communications of the i-th user on the j-th day;

——The heart rate data of the ith user in the εth time interval on the jth day;

——The daytime heart rate data set of the ith user on the jth day;

——The heart rate data set of the ith user on the jth day;

—The nighttime triaxial acceleration of the ith user on the jth day in the εth time interval;

——The nighttime three-axis acceleration data set of the ith user on the jth day;

——The emotional data of the ith user on the jth day;

——The total length of time that the i-th user uses the work APP on the j-th day;

——The total length of time that the i-th user uses the entertainment APP on the j-th day;

——The total length of time that the i-th user uses the social APP on the j-th day;

h_m _ε ——the maximum heart rate during the day in the εth time interval of the ith user on the jth day;

——The maximum set of daytime heart rate of the ith user on the jth day;

——The data set of the movement distance s and the number of steps d of the ith user on the jth day;

——The abnormal value of the daytime heart rate of the ith user on the jth day;

——The physiological index value of the ith user on the jth day;

——The maximum heart rate of the ith user at night on the jth day.

Step 3: Perform emotion recognition and select a model for recognition. The models include SVM models and neural network models.

model selection

In order to choose a better model, a variety of classification methods that can identify and monitor user emotions are compared.

Based on SVM model

SVM is a very common and effective classifier that builds a maximum margin hyperplane in the space by mapping the vector to a higher latitude space.

The present invention uses the default Graph of TensorFlow to create ops, uses placeholder() as the input container of data, and instructs TensorFlow to run the corresponding nodes of the graph through sess.run(). Through continuous training with the training data collected, a usable classification model is obtained.

The main steps of the experiment are as follows: parse the dataset and cross-validate, preprocess the data, draw the data flow diagram, define the Gaussian kernel function, create the dual loss function, create the classification kernel function, create the classification function, set the optimizer and initialize, train and evaluate the model .

Based on neural network model

Neural network model, also known as artificial neural network, is a derivative of using mathematical methods to simulate the working principle of human neurons in the brain, and is divided into input layer, hidden layer and output layer. The input layer inputs feature data, defines the number of hidden layers and the number of neurons in each layer, and the output layer is used to output the results.

The neural network model constructed in this project is trained using the data set generated by the user's daily life collected on the mobile terminal. Through continuous optimization, an available emotion recognition model is trained to recognize and monitor the user's daily emotion. Two high-level TensorFlow are mainly used here: APIEstimator and Dataset to build neural network models. The collected data is divided into two categories, one is used as a sample for training, and the other is used for evaluation. At the same time, TensorBoard is used to visualize the training process.

The experiment is divided into the following steps: import and parse the dataset, describe the data, define the model type, define the training input function, train the model, define the test model, and evaluate the model.

Claims

A method for identifying and monitoring users' emotions based on a mobile terminal, characterized in that:

The method includes the following steps:

Step 1: collect user data to generate a data set; the data heart rate data and emotion data; the data set includes a heart rate data set and an emotion data set;

Step 2: Preliminary data processing, the preliminary processing includes supplementing the missing characteristic values of some data in the data set, integrating the data, and extracting corresponding characteristic values;

Step 3: Perform emotion recognition and select a model for recognition.
The method for identifying and monitoring user's emotions based on a mobile terminal according to claim 1, wherein the heart rate data is divided into daytime heart rate data and nighttime heart rate data, and the daytime heart rate data is passed through a heart rate sensor. Obtained; the nighttime heart rate data is obtained through a three-axis acceleration sensor and a heart rate sensor.
The method for identifying and monitoring the user's emotion based on a mobile terminal according to claim 1, wherein the emotion data includes a positive class, a general class, and a negative class.
The method for identifying and monitoring the user's emotion based on a mobile terminal according to claim 1, wherein the feature value is supplemented by an automatic strategy based on other known values and an average value.
The method for identifying and monitoring users' emotions based on a mobile terminal according to claim 1, wherein the extraction of corresponding feature values is performed by classifying and visualizing the data collected by the APP. The corresponding eigenvalues are extracted by integration.
The method for identifying and monitoring the user's emotion based on a mobile terminal according to claim 5, wherein the emotion feature involves an APP index, a physiological index, a sleep index, and a communication index;

The emotional features are described as:

in
for APP indicators,
for physiological indicators,
for sleep indicators,
for communication indicators.
The method for identifying and monitoring the user's emotion based on a mobile terminal according to claim 1, wherein the model comprises an SVM model and a neural network model.