WO2020119245A1 - 一种基于可穿戴手环的情绪识别系统及方法 - Google Patents

一种基于可穿戴手环的情绪识别系统及方法 Download PDF

Info

Publication number
WO2020119245A1
WO2020119245A1 PCT/CN2019/111531 CN2019111531W WO2020119245A1 WO 2020119245 A1 WO2020119245 A1 WO 2020119245A1 CN 2019111531 W CN2019111531 W CN 2019111531W WO 2020119245 A1 WO2020119245 A1 WO 2020119245A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
physiological
module
heart rate
wearer
Prior art date
Application number
PCT/CN2019/111531
Other languages
English (en)
French (fr)
Inventor
舒琳
余洋
徐向民
Original Assignee
华南理工大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华南理工大学 filed Critical 华南理工大学
Publication of WO2020119245A1 publication Critical patent/WO2020119245A1/zh

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/024—Detecting, measuring or recording pulse rate or heart rate
    • A61B5/02438—Detecting, measuring or recording pulse rate or heart rate with portable devices, e.g. worn by the patient
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • A61B5/316—Modalities, i.e. specific diagnostic methods
    • A61B5/318—Heart-related electrical modalities, e.g. electrocardiography [ECG]
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/68—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801—Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/6802—Sensor mounted on worn items
    • A61B5/681—Wristwatch-type devices
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7225—Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device

Definitions

  • the invention relates to the field of wearable devices, in particular to an emotion recognition system and method based on a wearable bracelet.
  • the present invention provides an emotion recognition system based on a wearable bracelet, which integrates a heart rate, ECG, and galvanoelectric monitoring module on the wearable bracelet, and aims at the collected physiology Emotional recognition and feedback of the data in time.
  • the measurement method is simple and easy to wear. You can directly see the real-time changes of physiological signals on the worn bracelet, and you can quickly get the emotion recognition results.
  • the invention also provides an emotion recognition method based on the wearable bracelet.
  • An emotion recognition system based on a wearable bracelet including a physiological signal acquisition module, a physiological signal preprocessing module, a physiological signal feature extraction module, an emotion classification module, and a comprehensive evaluation module;
  • the physiological signal collection module collects three kinds of physiological data of the wearer's ECG, heart rate, and skin electrical energy respectively through the ECG sensor, the heart rate sensor, and the skin electrical sensor deployed on the end of the wearable bracelet; Data segmentation, after denoising, and transmitted to the physiological signal feature extraction module; the physiological signal feature extraction module performs feature extraction on the three types of physiological data, and the extracted features include linear features, nonlinear features, time-domain features, and frequency-domain features ;
  • the emotion classification module performs emotion recognition on three kinds of physiological data and outputs an emotional state based on each physiological data; the comprehensive evaluation module adopts weight-based voting decision rules to make voting decisions on the emotional state output from the three physiological data, and comprehensively determine The current emotional state label of the wearer of the bracelet is obtained, and the recognition result is obtained.
  • the emotion recognition system further includes a user feedback module.
  • the user feedback module feeds back a description of the current emotional state to the emotion classification module according to the difference between the recognition result and the real emotional feeling of the wearable bracelet wearer at the current moment; the emotion classification module based on the current feedback
  • the difference between emotional state description and recognition results dynamically adjusts the parameters of the emotion classification module and comprehensive evaluation module to form a personalized emotion recognition algorithm that more closely matches each wearable bracelet wearer.
  • the emotion recognition method of the present invention is implemented using the following technical solution: An emotion recognition method based on a wearable bracelet, including the following steps:
  • Step 1 Collect the three physiological data of the wearer's heart rate, ECG, and galvanic skin;
  • Step 2 Pre-process the three types of physiological data separately, including signal amplification and denoising, to obtain relatively pure physiological signals;
  • Step 3 Extract the corresponding physiological parameters from the pre-processed physiological signals, and calculate the linear, non-linear, time-domain, and frequency-domain characteristic parameters of the electrocardiographic data, as well as the statistical characteristic parameters of the heart rate data and the skin electrical data, to obtain three physiological parameters Characteristic parameters of data;
  • Step 4 Use different classifiers to perform emotion recognition on the characteristic parameters of the three kinds of physiological data of the current wearer to obtain the three kinds of emotion tags of the current wearer;
  • Step 5 Comprehensive evaluation, by setting the three emotion tags and their weights, the initial weight of the central electrical data is the highest, the initial weight of the heart rate data is lower than that of the electrocardiographic data, and the initial weight of the picoelectric data is the lowest. Through weight-based voting Rules to form the user's final emotional state label.
  • the present invention is based on a wearable bracelet, integrating multiple modules such as data collection, data preprocessing, feature extraction, emotion classification, comprehensive evaluation, user feedback, etc., which can comprehensively and accurately evaluate the current emotional state of the wearer of the bracelet , Improve the wearer's awareness and management of their emotions, can have a healthier mental state.
  • the present invention has the following technical effects:
  • the present invention can directly see the real-time changes of physiological signals on the worn bracelet.
  • the wearer can understand the changes of their physiological signals in real time.
  • the emotion classification module and comprehensive evaluation module can combine heart rate, electrocardiogram and skin electrical A physiological signal to accurately identify the current emotional state of the wearer.
  • the user feedback module can make a more accurate emotional feedback description based on the emotion recognition result and the real emotion feeling at the current moment.
  • the emotion classification module first performs individual emotion recognition for each physiological signal, then determines the voting weight based on the degree of influence of emotional changes on the physiological signal, and dynamically adjusts the classifier parameters and comprehensive evaluation algorithm based on the wearer's feedback on the recognition result
  • the weight of the voter gets the final emotional state label of the wearer.
  • the accuracy of algorithm recognition is improved.
  • the wearer's personal data and database standard data are also integrated, and a "personalized" emotion recognition classifier suitable for each wearer is generated by training.
  • the present invention is based on a wearable bracelet, which comprehensively utilizes three different types of physiological signals, on the one hand, it greatly improves the description of physiological signals to the wearer's true emotions, and effectively compensates for the current wearable bracelet can not identify the wearer Emotional state or emotion recognition based on a single physiological signal; on the other hand, the physiological signal of the current wearer can be collected simply and conveniently. Compared with other complicated collection devices, the volume is smaller, convenient to wear, and the cost is lower.
  • the present invention fully considers the characteristics of extracting physiological signals from the body surface, adopts a double denoising scheme from the hardware level and the software level, and in the denoising algorithm, fully considers what kind of noise is affected by each physiological signal. Large, select the algorithm with the best denoising effect to preprocess the data.
  • the amplifying circuit set on the skin electrical signal is located inside the bracelet dial, and the skin electrical and heart rate work in the same way. Only when this mode is selected, the sensor and the amplifier circuit at the end of the bracelet will work to amplify the galvanic signal.
  • the median filter selects an observation window composed of odd numbers, arranges the observation window data, and retains the median, which has the advantage of real-time processing. Wavelet processing thresholds the coefficients on the scale to effectively remove noise such as baseline drift.
  • the data set used for training the model comes from the experimental data collected in the standard laboratory environment (sound insulation), using the Chinese Standard Video Material Library (CEVS) as the experimental material
  • CEVS Chinese Standard Video Material Library
  • the classifier model is based on the standard emotional state data set, and in the parameter setting of the model and weights, the use of Grid Search, cross-validation and other methods can improve the accuracy rate and reflect the emotional state attributes more objectively and comprehensively.
  • FIG. 1 is an overall block diagram of an emotion recognition system based on a wearable bracelet.
  • Figure 2 is a schematic diagram of the external structure of the bracelet.
  • Figure 3 is a schematic diagram of the bracelet body.
  • Figure 4 is a schematic diagram of a positive single cycle ECG.
  • Figure 5 is a heart rate graph under neutral emotions.
  • Figure 6 is a schematic diagram of picoelectric changes.
  • Heart rate refers to the number of heartbeats per minute in a normal person in a quiet state, also called quiet heart rate, which is generally 60-100 beats/min.
  • Electrocardiogram is the synthesis of action potentials produced by myocardial cells during the pulse of the human heart. If you place the two electrodes on the body surface, you can record the changes in ECG through the potential difference between the two points on the body surface, forming a continuous curve, called an electrocardiogram.
  • Skin electrical is an emotional physiological indicator, which represents the change of skin electrical conduction when the body is stimulated.
  • Physiological signals such as electrocardiogram, heart rate, and picoelectricity contain the content of emotional changes.
  • the change of mood can be distinguished from the rate of change, linear characteristics, nonlinear characteristics, time-domain characteristics, and frequency-domain characteristics of these physiological signals.
  • Studies have shown that in fear, heart rate changes are significantly faster than sadness and neutral emotions.
  • the three basic emotion states of happiness, sadness and disgust and the neutral emotional state are significantly different from each other in the high-frequency components of heart rate variability.
  • Skin electricity is usually closely related to the degree of emotion activation, and it is also an effective indicator for identifying basic emotions.
  • Negative emotions such as depression and anxiety can significantly affect skin electricity.
  • Dermal electricity is also considered to be an effective indicator for diagnosing patients with mood disorders such as depression and anxiety, and has a certain predictive effect on suicidal behaviors based on patients with depression.
  • the overall structure of the wearable bracelet-based emotion recognition system is shown in FIG. 1, which mainly includes a physiological signal acquisition module, a physiological signal preprocessing module, a physiological signal feature extraction module, an emotion classification module, a comprehensive evaluation module, and User feedback module.
  • the physiological signal collection module is to collect the three physiological signals of the wearer's ECG, heart rate, and skin electrical signals through the ECG sensor, the heart rate sensor, and the skin electrical sensor deployed on the end of the wearable bracelet; specifically, the physiological signal collection module includes the heart rate Signal monitoring module, ECG signal monitoring module and picoelectric signal monitoring module. among them:
  • the heart rate signal monitoring module is a sensor using the PPG principle, and a PPG light sensor is provided at the position of the wristband; the bracelet body is located at the back of the bracelet (close to the back of the hand), and the PPG light sensor is based on the received photoelectric signal of the subcutaneous tissue. , Perform filtering and amplification processing, and calculate the current heart rate value according to the signal peak monitored per unit time; the sampling frequency of the heart rate is 25 Hz.
  • Using a heart rate signal monitoring module based on PPG light sensing to measure heart rate has the following advantages:
  • the melanin on the skin absorbs a lot of light with shorter wavelengths, and most of the green light entering the skin is absorbed by red blood cells, so the blood absorbs more than other tissues
  • the green light as the light source signal, the signal to noise ratio is better than other light sources.
  • the principle of PPG measuring heart rate is that when light transmits through skin tissue and then reflects to the photosensitive sensor, there will be a certain loss of light.
  • the body and PPG light sensor remain relatively still, the absorption of light by muscles, bones, veins and other tissues is basic
  • the absorption of light is basic
  • the absorption of light changes.
  • the light signal is converted into an electrical signal, the AC part of the AC changes.
  • the peak is extracted and calculated. The number of peaks per unit time, the wearer's heart rate value is calculated, as shown in formula (1):
  • N represents the number of pulse wave peaks monitored
  • T represents the time interval for recording changes in the pulse wave.
  • the ECG signal monitoring module adopts a two-electrode structure, and one end of the electrode is integrated in the rivet used for fixing the wristband of the wristband.
  • the contacts are connected, and the other end electrode is integrated under the touch point of the bracelet body.
  • the ECG signal monitoring module mainly includes three parts.
  • the first part of the electrode is a contact with the human body (ie ECG electrode 1).
  • the contact for fixing the wristband of the bracelet is fixed by a metal buckle In the rubber of the wristband, it is connected to the contact point that is connected to the wristband and the bracelet dial through the wire buried in the wristband (ie, the ECG electrode connection wire 2); After the other end of the wristband is connected, the wristband can be worn on the wearer's wrist.
  • the second part of the electrode is the ECG electrode contact 4 under the touch screen of the bracelet dial.
  • the third part of the electrode is the contact point of the bracelet dial and the ECG electrode connection wire buried in the wristband.
  • the contact contact groove of the bracelet wristband needs to protrude outward, and all the contacts on the bracelet are made of metal Spring-type contacts, which can ensure that all contacts are in full contact, and will not cause a disconnection between the electrode and the bracelet.
  • the sampling frequency of the ECG signal is 256 Hz.
  • Picoelectric signal monitoring module mainly uses a flexible sensor integrated in the wristband (ie, Picoelectric electrode 5), such as electrodes of fabric or conductive rubber substrate.
  • the Picoelectric electrode mainly includes two parts, the first part is a flexible sensor embedded in The central area of the wristband of the bracelet is kept in contact with the wrist joint of the wrist, and is connected with the electric skin electrode connection 6 buried in the wristband.
  • the second part is that the wire extends along the direction of the base of the wrist strap for fixing the bracelet dial, and extends to the galvanic electrode contact 7 which is in contact with the bracelet dial, so that the flexible sensor and the bracelet body are connected.
  • the sampling frequency is 25 Hz.
  • the charging dial 8 is also provided on the bracelet dial.
  • the physiological signal preprocessing module mainly includes a signal amplification module based on a hardware circuit, a denoising module, and a signal denoising algorithm based on software.
  • noise sources include power frequency interference, baseline drift, myoelectric interference, and motion interference.
  • the hardware the common mode rejection ratio of the circuit is improved, and an analog filter is set to denoise the signal.
  • wavelet transform, median filtering and other methods are used. Among them, for the three physiological signals, all affected by motion interference, wavelet transform is used to perform threshold value processing on the frequency signal containing noise to separate the noise and the signal.
  • Physiological signal preprocessing module In the processing of heart rate data, the collected heart rate signal needs to extract the spikes according to the photoelectric volume pulse wave tracing. The number of spikes in the statistical unit time is the corresponding heart rate value; first, the signal is amplified, then The wavelet transform method is used to denoise the signal, and then the position of the peak is monitored, and the number of occurrences of the peak is counted. In ECG data processing, the signal-to-noise ratio of the ECG signal collected at the body surface is very low.
  • the hardware aspect improves the signal-to-noise ratio, sets an analog filter for denoising, uses median filtering to remove the baseline drift, and the selected window time length is 200ms , Adopt 50Hz notch filter to remove power frequency interference.
  • the threshold method of wavelet transform is used to remove EMG interference. Movement interference is a relatively difficult point in signal removal. Due to the large-scale overlap of noise and ECG spectrum, independent component analysis (ICA) is used Remove the motion interference of ECG signal. The picoelectric signal changes slowly, and the wavelet transform threshold method is used to remove its motion interference.
  • the data segmentation length of skin electrical and heart rate is 2min
  • the data segmentation length of ECG is 10s. Then, according to a 2min heart rate data, a 2min skin electrical data, and 12 10s length ECG data, the wearer's 2min physiological signal data is formed, and the data set is processed by the physiological signal feature extraction module.
  • the physiological signal feature extraction module is divided into three parts, and all algorithms are deployed on the cloud platform. Separate extraction based on three physiological signals, covering the linear, non-linear, time-domain and frequency-domain characteristics of the three physiological signals.
  • the feature extraction of heart rate data and picoelectric data is mainly based on statistical characteristics, and statistics are made from the change range, maximum value, minimum value, rate of change, first-order difference, and second-order difference of data.
  • the feature extraction of ECG data is mainly analyzed from four levels: time domain, frequency domain, linearity, and nonlinearity of ECG signals, focusing on analyzing the changes of ECG data to analyze the impact of emotional changes on three physiological signal changes .
  • Figure 4 shows the change of heart rate for a period of time.
  • the first-order difference average of heart rate change is calculated according to heart rate change, as shown in formula (2), where X n represents time
  • the corresponding heart rate value at t n , N represents the length of this segment of heart rate value.
  • X n represents the corresponding heart rate value at time t n
  • N represents the length of this segment of heart rate value
  • the average value of the absolute value of the first-order difference of the original signal refers to the average value of the heart rate of the wearer monitoring the wearer in the neutral emotional state. This data is generated by the user's historical data, and the cloud Continuously update based on the collected data.
  • the calculation process of the normalized first-order difference average value of the heart rate change is shown in formula (4), and the calculation process of the normalized second-order difference average value of the heart rate change is shown in formula (5).
  • Figure 5 shows the ECG data after a single cycle of denoising.
  • the extracted features mainly include linear features, nonlinear features, time-domain features, and frequency-domain features.
  • the specific indicators and calculation process are as follows:
  • SDNN the standard deviation of all sinus cardiac RR intervals
  • NN50 the number of adjacent NN>50ms
  • PNN50 the number of adjacent NNs >50ms as a percentage of the total number of sinus beats
  • SDSD standard deviation of the difference between adjacent RR intervals
  • RR_MEAN average value of RR gap
  • ECG Min, Max, Mean, Var after analyzing the baseline drift
  • Wavelets Use db6 wavelet, 3-layer decomposition processing, respectively count the maximum, minimum, median, standard deviation of 3-layer high-frequency details and 1-layer low-frequency approximation;
  • VLF ultra low frequency
  • LF low frequency
  • HF high frequency
  • the characteristic parameters of picoelectric data are specifically covered as follows.
  • Figure 6 shows the ECG data after a single cycle of denoising.
  • the extracted features mainly include the statistical characteristic parameters of a piece of 2min picoelectric data, including the maximum, minimum, mean, variance, rate of change (if the rate of change is positive when rising, the rate of change is negative when falling), and the first-order difference average Value, second-order difference average, normalized first-order difference average, normalized second-order difference average, variation range, sum of squares of sequence difference, etc.
  • the emotion classification module performs emotion recognition on three physiological signals separately, and then obtains the emotional state corresponding to each physiological signal.
  • the classifiers used include SVM, KNN, RF, DT, GBDT, AdaBoost, etc.
  • the parameters of the classifier can be set according to the specific classifier.
  • Several classifiers based on sklearn can set the reference parameters as follows: SVM first normalizes the data, the parameters mainly choose the RBF kernel function, the initial value of C is set to 5, and the initial value of ganma is set to 0.4; the default selection of LDA is'lsqr' Solve the least square QR and calculate the covariance matrix of each category.
  • the maximum number of iterations of the RF weak learner is set at 900, and the samples outside the bag are used to evaluate the quality of the samples. Due to the small size of the data set used to train the classifier, the main method of setting the feature cut point for DT is best; GBDT Mainly set subsample, the initial value can be set to 0.5; all parameters of Adaboost can choose the default value.
  • the training set of the classifier used comes from the data set collected using China Standard Emotional Video Material Library (CEVS).
  • CEVS China Standard Emotional Video Material Library
  • GridSearch is used to try out the influence of each parameter on the recognition result through looping through all the listed candidate parameters, and the parameter with the best recognition rate is used as the parameter used by the final classifier.
  • each specific parameter set a reasonable minimum step value, control one parameter at a time, and gradually perform loop training according to the step value until the best recognition accuracy rate is obtained.
  • the above classifier will be used to predict the classifier with the best accuracy of 50% cross-validation in training to obtain the emotional label of each physiological signal.
  • the parameters corresponding to each model have been adjusted, so that under cross-validation, the accuracy of the recognition results for the three emotions is the highest.
  • the comprehensive evaluation module is based on the voting rules of weights to obtain the final emotional label of the wearer at the current moment. Based on the weighted voting rules, the amount of ECG data is large, sensitive to changes in emotional state, and the rate of change is fast, while the relative change of heart rate data is relatively slow, and the change of skin electrical data is the slowest.
  • the initial weight held by ECG data is higher than that of skin electrical data and heart rate data.
  • the initial weight of heart rate data is lower than that of ECG data, and the initial weight of skin electrical data is the lowest.
  • the initial weight of ECG data can be set to 50%
  • the initial weight of heart rate data can be set to 30%
  • the initial weight of picoelectric data can be set to 20%.
  • the current emotional state of the wearer is generated by summarizing the three emotional tags and corresponding weights. Each time a label based on three physiological signals and a corresponding weight ratio is voted, the final label generated is the wearer's emotional state at the current moment. Use “positive”, “negative”, “neutral” to characterize the wearer's emotional state.
  • the weight of the ECG data may be higher, and the weight of the skin electrical data and the heart rate data are the same and lower than the weight of the ECG.
  • the tags generated by the 12-segment ECG data take the "mode" of each segment as the total emotional tag represented by the ECG data, and use the three physiological signal tags combined with the voting weights held by the three to generate the total emotional tag As the final recognition result.
  • the setting standard of the initial weight is determined by the accuracy rate of the emotion classification of the standard data set, and the weight with the highest accuracy rate in all recognition is taken as the initial weight.
  • the weights are also gradually adjusted based on the current optimal accuracy. After each update, the optimal weight combination of the three physiological signals can still be maintained.
  • the algorithm implementation process of the user feedback module is shown in Figure 8, which is based on the "personalized" emotion recognition scheme for each wearer of the bracelet, for each user's actual emotional subjective feelings and differences in usage scenarios. Sex, adjust to the wearer. The wearer can feedback his own emotions through the mobile terminal in time according to the emotion recognition results fed back in real time. The emotion classification module and comprehensive evaluation module dynamically adjust the classifier parameters and voting weights according to the wearer feedback results, so that the emotion recognition results are consistent with user feedback, and finally form a "personalized” emotion recognition algorithm with each bracelet wearer. Get more accurate emotion recognition results.
  • the historical physiological signals collected from the wearer are first added to the standard database to form part of the data set, and then feature extraction is performed, and the model is continuously retrained with individual tags. Compare the new data set during training against the current data, whether the accuracy of recognition is improved, if there is no improvement, start using Grid_Search to adjust the parameters of the model, and then compare the accuracy rate, if there is no increase, then the voting weight Adjust to find the highest accuracy, the best model, and the weight combination of voting to get the best emotion recognition algorithm for each wearer.
  • the emotion recognition method of the present invention includes the following steps:
  • Step 1 Wear the bracelet correctly according to the prescribed posture, and collect the three physiological data of the wearer's heart rate, electrocardiogram, and galvanic electrocardiogram according to the use requirements;
  • Step 2 Pre-process the three types of physiological data separately, including signal amplification and denoising to obtain relatively pure physiological signals. Calculate the wearer's real-time heart rate value from the PPG measurement signal, and divide the obtained physiological signal according to a fixed length;
  • Step 3 The physiological signal extraction module extracts the corresponding physiological parameters from the pre-processed physiological signals, and mainly calculates the linear, non-linear, time-domain and frequency-domain characteristic parameters of the electrocardiographic data, and the statistical characteristic parameters of the heart rate data and the galvanic data , Get the characteristic parameters of three kinds of physiological data;
  • Step 4 Use different classifiers to perform emotion recognition on the characteristic parameters of the three kinds of physiological data of the current wearer to obtain the three kinds of emotion tags of the current wearer;
  • Step 5 Comprehensive evaluation, by setting the three emotion tags and their weights, the initial weight of the central electrical data is the highest, the initial weight of the heart rate data is lower than that of the electrocardiographic data, and the initial weight of the picoelectric data is the lowest. Through weight-based voting Rules to form the user's final emotional state label;
  • Step 6 According to user feedback, adjust the parameters of the emotion classification algorithm and voting weights in real time to form a specific algorithm model that is more suitable for each user, and form a more accurate wearable bracelet emotion recognition through interaction with the wearer Algorithm model.

Landscapes

  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Surgery (AREA)
  • Public Health (AREA)
  • Veterinary Medicine (AREA)
  • General Health & Medical Sciences (AREA)
  • Animal Behavior & Ethology (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Pathology (AREA)
  • Biomedical Technology (AREA)
  • Heart & Thoracic Surgery (AREA)
  • Medical Informatics (AREA)
  • Signal Processing (AREA)
  • Artificial Intelligence (AREA)
  • Psychiatry (AREA)
  • Physiology (AREA)
  • Cardiology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Social Psychology (AREA)
  • Developmental Disabilities (AREA)
  • Educational Technology (AREA)
  • Hospice & Palliative Care (AREA)
  • Psychology (AREA)
  • Child & Adolescent Psychology (AREA)
  • Power Engineering (AREA)
  • Evolutionary Computation (AREA)
  • Fuzzy Systems (AREA)
  • Mathematical Physics (AREA)
  • Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
  • Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)

Abstract

基于可穿戴手环的情绪识别系统及方法，系统包括生理信号采集模块、生理信号预处理模块、生理信号特征提取模块、情绪分类模块、综合评价模块；生理信号采集模块采集佩戴者的心电、心率、皮电三种生理数据；生理信号预处理模块对三种生理数据进行数据切分，去噪之后传输至生理信号特征提取模块；生理信号特征提取模块对三种生理数据分别进行特征提取；情绪分类模块针对三种生理数据进行情绪识别，输出三种情绪状态；综合评价模块采用基于权重的投票决策规则，对三种情绪状态进行投票决策，综合确定可穿戴手环佩戴者当前的情绪状态标签，得到识别结果。通过基于可穿戴手环的情绪识别系统及方法能提高佩戴者对自身情绪的认知和管理能力，使其拥有更加健康的心理状态。

Description

一种基于可穿戴手环的情绪识别系统及方法 技术领域
本发明涉及可穿戴设备领域，具体是一种基于可穿戴手环的情绪识别系统及方法。
背景技术
近年来，移动互联网技术取得了飞速的发展，可穿戴设备已经在人们的生活中扮演着十分重要的角色，逐渐成为消费类电子中比较流行的产品。它帮助人们养成健康的生活方式，更合理规划工作与生活。可穿戴电子设备方便携带，测量准确，拓展能力强等优势深受消费者和开发者的喜爱。当今社会，人们面临的工作和生活压力与日俱增，长期的压力给人们的情绪带来严重的影响，负面的情绪极易引发抑郁症、心脏病、高血压、内分泌失调等健康问题。因此，尽早发现潜在的情绪问题，能够帮助我们更好地清除情绪障碍，使工作和生活变得更加美好。
目前大多数品类的手环，都只是采集了心率数据，并在手环端实时显示心率的情况，或者只是在移动端的APP进行了单位时间(天或者小时)心率变化的分析；而没有采集更多的生理信号，也反映不出单位时间内佩戴者情绪的变化情况，更没有针对情绪状态有可能带来的影响做出及时的反馈。达不到针对佩戴者情绪及时监测，提醒佩戴者注意调节情绪的效果。即使现在也有测量心电信号的手表，但是也只是局限在手腕处获取原始信号，没有针对佩戴者的情绪变化情况进行及时识别和反馈。
发明内容
针对现有可穿戴设备所存在的不足，本发明提供一种基于可穿戴手环的情绪识别系统，该系统在可穿戴手环上集成心率、心电、皮电监测模块，并针对采集的生理数据及时进行情绪识别并反馈，测量方法简单，方便穿戴，可直接在佩戴的手环上看到生理信号实时的变化情况，并能够很快得到情绪的识别结果。
本发明还提供一种基于可穿戴手环的情绪识别方法。
本发明情绪识别系统采用如下技术方案来实现：一种基于可穿戴手环的情绪识别系统，包括生理信号采集模块、生理信号预处理模块、生理信号特征提取模块、情绪分类模块、综合评价模块；
生理信号采集模块通过部署在可穿戴手环端的心电传感器、心率传感器、皮电传感器来分别采集佩戴者的心电、心率、皮电三种生理数据；生理信号预处理模块对三种生理数据进行数据切分，去噪之后传输至生理信号特征提取模块；生理信号特征提取模块对三种生理数 据分别进行特征提取，所提取的特征包括线性特征、非线性特征、时域特性、频域特征；
情绪分类模块针对三种生理数据进行情绪识别，基于每一种生理数据输出一个情绪状态；综合评价模块采用基于权重的投票决策规则，对三种生理数据输出的情绪状态进行投票决策，综合确定可穿戴手环佩戴者当前的情绪状态标签，得到识别结果。
所述情绪识别系统还包括用户反馈模块，用户反馈模块根据识别结果和可穿戴手环佩戴者当前时刻真实情绪感受的差异，向情绪分类模块反馈当前情绪状态描述；情绪分类模块根据所反馈的当前情绪状态描述和识别结果之间的差异，动态调整情绪分类模块和综合评价模块的参数，形成与每一位可穿戴手环佩戴者更加匹配的个性化情绪识别算法。
本发明情绪识别方法采用如下技术方案来实现：一种基于可穿戴手环的情绪识别方法，包括以下步骤：
步骤1：采集手环佩戴者的心率、心电、皮电三种生理数据；
步骤2：对三种生理数据分别进行预处理，包括信号放大、去噪处理，得到相对纯净的生理信号；
步骤3：对预处理之后的生理信号提取相应的生理参数，计算心电数据的线性、非线性、时域、频域特征参数，以及心率数据和皮电数据的统计特征参数，得到三种生理数据的特征参数；
步骤4：使用不同的分类器，分别对当前佩戴者的三种生理数据的特征参数进行情绪识别，得到当前佩戴者的三种情绪标签；
步骤5：综合评价，通过对三种情绪标签及其权重设置，其中心电数据的初始权重最高，心率数据的初始权重较心电数据低，皮电数据的初始权重最低，通过基于权重的投票规则，形成用户最终的情绪状态标签。
从以上技术方案可知，本发明基于可穿戴手环，集成了数据采集、数据预处理、特征提取、情绪分类、综合评价、用户反馈等多模块，能全面准确评价手环佩戴者当前的情绪状态，提高了佩戴者对于自身情绪的认知和管理能力，能拥有一个更加健康的心理状态。与现有技术相比，本发明具有如下技术效果：
1、本发明可直接在佩戴的手环上看到生理信号实时的变化情况，佩戴者能够实时了解自身的生理信号变化情况，情绪分类模块和综合评价模块能够结合心率、心电和皮电三种生理信号，准确识别佩戴者当前的情绪状态。用户反馈模块能基于情绪识别结果和当前时刻真实情绪感受，做出更精准的情绪反馈描述。情绪分类模块先针对每一种生理信号进行单独的情绪识别，然后基于情绪变化对生理信号的影响程度，确定投票权重，并根据佩戴者对于识别结果的反馈，动态调整分类器参数和综合评价算法投票器的权重，得到佩戴者最终的情绪状 态标签。提高了算法识别的准确率，同时也综合佩戴者个人数据和数据库标准数据，训练生成适合用于每一位佩戴者的“个性化”情绪识别分类器。
2、本发明基于可穿戴手环，综合利用了三种不同类型的生理信号，一方面极大地提升了生理信号对于佩戴者真实情感的描述，有效弥补了当前可穿戴手环不能识别佩戴者的情绪状态或者基于单一生理信号的情感识别；另一方面，能够简单、方便地采集到当前佩戴者的生理信号，比起其他繁杂的采集设备，体积更小，方便穿戴，成本更低。
3、本发明充分考虑从体表提取生理信号的特点，从硬件层面和软件层面，采用双重的去噪方案，且在去噪算法上，充分考虑每一种生理信号受到何种噪声的影响更大，选择去噪效果最好的算法对数据进行预处理。
4、皮电信号上所设置的放大电路位于手环表盘内部，皮电和心率的工作方式相同。只有在选择该模式的情况下，传感器和手环端的放大电路才会工作，对皮电信号进行放大处理。中值滤波选取一个奇数组成的观察窗，将观察窗数据排列，保留中值，具有实时化处理的优势。小波处理对尺度上的系数进行阈值处理，能够有效的去除基线漂移等噪声。
5、本发明在情绪分类算法的模型训练时，用于训练模型的数据集均来自于在标准实验室环境下(隔音)，使用中国标准视频素材库(CEVS)作为实验材料收集到的实验数据，分类器模型是基于标准情绪状态数据集建立，且在模型和权重的参数设置上，使用Grid Search、交叉验证等方法能够在提升准确率同时，更加客观全面的反映情绪状态属性。
附图说明
图1为基于可穿戴手环的情绪识别系统整体框图。
图2是手环的外形结构示意图。
图3是手环本体示意图。
图4是正向单周期心电示意图。
图5是中性情绪下心率图。
图6是皮电变化示意图。
图7是三种生理信号的情绪识别流程图。
图8是基于用户反馈的算法动态更新流程图。
具体实施方式
下面结合附图及实施例对本发明做进一步详细说明，但本发明的实施方式并不限于此。
心率是指正常人安静状态下每分钟心跳的次数，也叫安静心率，一般为60-100次/分。心 电(electrocardiogram，ECG)是人体心脏脉搏时，由心肌细胞产生的动作电位综合而成。如果将两个电极放置在体表，就可以通过体表两点间的电位差记录心电的变化，形成一条连续的曲线，称之为心电图。皮电是一项情绪生理指标，代表机体受到刺激时皮肤电传导的变化。
心电、心率、皮电等生理信号包含着情感变化的内容。可以从这些生理信号的变化率、线性特性、非线性特性、时域特性、频域特性中分辨情绪的变化情况。研究表明：在恐惧情绪时，心率的变化明显快于悲伤情绪和中性情绪。高兴、悲伤和厌恶这三种基本情绪和中性情绪状态在心率变异性的高频成分上彼此都存在着显著性差异。皮肤电通常与情绪的激活程度密切相关，同时也是识别基本情绪的有效指标，抑郁和焦虑等负向情绪会显著的影响皮肤电。皮肤电也被认为是诊断抑郁症、焦虑症等情绪障碍患者的一个有效指标，对基于抑郁症患者的自杀行为都有一定的预测作用。
本实施例中，基于可穿戴手环的情绪识别系统的整体结构如图1所示，主要包括生理信号采集模块、生理信号预处理模块、生理信号特征提取模块、情绪分类模块、综合评价模块以及用户反馈模块。生理信号采集模块是通过部署在可穿戴手环端的心电传感器、心率传感器、皮电传感器来采集佩戴者的心电、心率、皮电三种生理信号；具体来说，生理信号采集模块包括心率信号监测模块、心电信号监测模块和皮电信号监测模块。其中：
心率信号监测模块是采用PPG原理的传感器，在腕带位置设有PPG光传感器；手环本体在位于手环背部(紧贴手背部位方向)，PPG光传感器根据接收到的皮下组织的光电信号后，进行滤波和放大处理，根据单位时间监测到的信号波峰计算出当前的心率值；心率的采样频率是25Hz。采用基于PPG光传感的心率信号监测模块测量心率，有以下优势：皮肤上的黑色素会吸收大量波长较短的光，进入皮肤的绿光大部分被红细胞吸收，因而血液要比其他组织吸收更多的光；而绿光作为光源信号，信噪比要优于其他的光源。PPG测量心率的原理是当光透射皮肤组织然后再反射到光敏传感器时，光照会有一定的损失，当身体和PPG光传感器保持相对静止时，肌肉、骨骼、静脉等组织对光的吸收是基本没有变化的，人体动脉里有血液的流动，这样对光的吸收就有所变化，光信号转换成电信号的时候，交流AC部分就有变化，根据光电容积脉搏波描记，提取出尖峰，计算单位时间内峰值的个数，就计算出佩戴者的心率值，如公式(1)所示：
Figure PCTCN2019111531-appb-000001
其中，N表示监测到脉搏波峰值的个数，T表示记录脉搏波变化的时间间隔。
如图2、3所示，心电信号监测模块采用双电极结构，一端电极集成在手环腕带用于固定的铆钉处，通过埋藏在腕带中的心电电极连接线和手环本体金属触点相连接，另一端电极集 成在手环本体触摸点下方。在本实施例中，心电信号监测模块主要包括三个部分，第一部分电极是与人体接触的触点(即心电电极1)，用于固定手环腕带的触点通过一金属扣固定在腕带的橡胶中，与通过埋藏在腕带之中的导线(即心电电极连接线2)连接到腕带和手环表盘接触的触点；将金属扣通过手环卡扣孔3与手环腕带的另一端连接后，便可将手环佩戴在佩戴者的手腕处。第二部分电极为手环表盘触摸屏下方的心电电极触点4，当进入采集心电模式时，一只手的拇指按住触点，然后沿着腕带方向紧握，这样既能保证信号采集输入点和人体的充分接触，另外一方面也能保证另一只手与手环表盘电极的接触。第三部分电极是手环表盘和埋在腕带中的心电电极连接线的接触触点，手环腕带的触点接触槽需向外凸出一部分，手环上所有的触点采用金属弹簧式触点，这样能够保证所有的触点充分接触，不会造成电极和手环之间断路。心电信号的采样频率256Hz。
皮电信号监测模块主要使用集成在腕带中的柔性传感器(即皮电电极5)，如织物或者导电橡胶基材的电极，皮电电极主要包括两个部分，第一部分是柔性传感器内嵌在手环腕带的中部区域，和手腕的腕关节保持接触，通过与埋藏在腕带中的皮电电极连线6进行连接。第二部分是导线沿着腕带用于固定手环表盘的底座方向延伸，延伸至与手环表盘接触的皮电电极触点7，使柔性传感器和手环本体进行连接。采样频率为25Hz。手环表盘上还设有充电触点8。
生理信号预处理模块主要包括基于硬件电路的信号放大模块、去噪模块，以及基于软件的信号去噪算法。在手环测量的三种生理信号中，噪声来源包括工频干扰、基线漂移、肌电干扰、运动干扰等。硬件上，采用提高电路的共模抑制比，设置模拟滤波器对信号去噪。在软件上，采用小波变换、中值滤波等方法。其中，针对三种生理信号，均受到运动干扰的影响，采用小波变换，对包含噪声的频率信号进行阈值法处理，分离噪声和信号。
生理信号预处理模块，在心率数据处理上，由于采集的心率信号需要根据光电容积脉搏波描记，提取出尖峰，统计单位时间内尖峰的个数就是对应的心率值；首先进行信号的放大，然后利用小波变换方法对信号进行去噪，之后监测波峰所在位置，统计波峰出现的次数。在心电数据处理上，体表处采集的心电信号信噪比非常低，硬件方面提高信噪比，设置模拟滤波器进行去噪，使用中值滤波去除基线漂移，选用的窗口时间长度为200ms，采用50Hz陷波滤波器去除工频干扰。在皮电数据处理上，采用小波变换的阈值法去除肌电干扰，运动干扰是信号去除中比较难的一点，由于噪声和心电的频谱有大规模的重叠，因此采用独立成分分析(ICA)去除心电信号的运动干扰。皮电信号变化缓慢，用小波变换阈值法来去除其运动干扰。皮电和心率的数据切分长度为2min，心电的数据切分长度为10s。之后按照一段2min的心率数据、一段2min的皮电数据、12段10s长度的心电数据共同构成佩戴者2min的生理 信号数据，组成数据集交由生理信号特征提取模块进行处理。
生理信号特征提取模块分为三大部分，所有算法均部署在云平台。基于三种生理信号进行单独提取，涵盖三种生理信号的线性特征、非线性特征、时域特性、频域特征等。本实施例中，对于心率数据和皮电数据的特征提取主要是基于统计特征，从数据的变化范围、最大值、最小值、变化率、一阶差分、二阶差分变化情况进行统计。心电数据的特征提取，主要从心电信号的时域、频域、线性、非线性四个层面进行分析，着重通过分析心电数据的变化情况来分析情绪变化对三种生理信号变化的影响。
具体来说，心率数据的特征参量具体涵盖如下，图4所示为一段心率的变化情况，根据心率变化计算心率变化一阶差分平均值，如公式(2)所示，其中，X n表示时间t n时对应的心率值，N表示这一段心率值的长度。
Figure PCTCN2019111531-appb-000002
心率变化二阶差分平均值如公式(3)所示，X n表示时间t n时对应的心率值，N表示这一段心率值的长度。
Figure PCTCN2019111531-appb-000003
归一化之后原始信号一阶差分绝对值的平均值，这里的归一化是指在手环监测佩戴者在中性情绪状态下，心率的平均值，这个数据由用户的历史数据产生，云端根据收集到的数据进行不断的更新。心率变化归一化一阶差分平均值的计算过程如式(4)所示，心率变化归一化二阶差分平均值的计算过程如式(5)所示。
Figure PCTCN2019111531-appb-000004
Figure PCTCN2019111531-appb-000005
2min内心率变化的范围如式(6)所示：
HR range＝Heart max-Heart min   (6)
心率序列之间差值的平方和的平均值如式(7)所示：
Figure PCTCN2019111531-appb-000006
心率变化的斜率计算过程如式(8)所示：
Figure PCTCN2019111531-appb-000007
具体来说，心电数据的特征参量具体涵盖如下，图5所示为单一周期去噪之后的心电数据。提取的特征主要包括线性特征、非线性特征、时域特征、频域特征，具体指标和计算过程如下：
SDNN：全部窦性心博RR间期的标准差；
NN50：相邻NN之差>50ms的个数；
PNN50：相邻NN之差>50ms的个数占总窦性心搏个数的百分比；
SDSD：相邻RR间期差值的标准差；
RR_MEAN：RR间隙的平均值；
ECG：分析基线漂移之后Min、Max、Mean、Var；
Wavelets：使用db6小波，3层分解处理，分别统计3层高频细节和1层低频近似的最大值、最小值、中位数、标准差；
VLF(超低频)、LF(低频)、HF(高频)的能量。
具体来说，皮电数据的特征参量具体涵盖如下，图6所示为单一周期去噪之后的心电数据。提取的特征主要包括一段2min皮电数据的统计特征参量，包括最大值、最小值、均值、方差、变化率(若上升时变化率为正，下降时变化率为负数)，以及一阶差分平均值、二阶差分平均值、归一化一阶差分平均值、归一化二阶差分平均值、变化范围、序列差值的平方和等。
情绪分类模块是单独针对三种生理信号进行情绪识别，然后得出每一种生理信号对应的情绪状态，如图7所示，是整个情绪分类的流程图。所使用的分类器包括SVM、KNN、RF、DT、GBDT、AdaBoost等。分类器的参数可根据具体的分类器进行设置。基于sklearn的几个分类器可设置参考参数如下：SVM首先针对数据进行归一化，参数主要选择RBF核函数，C初始值设置为5，ganma初始值设置为0.4；LDA中默认选择’lsqr’最小平方QR求解，并计算每个类别的协方差矩阵。RF的弱学习器最大迭代次数设置在900，采用袋外样本来评估样本的好坏；由于用于训练分类器的数据集规模不大，DT的主要设置特征切分点的方式为best；GBDT主要设置subsample，初始值可设置为0.5；Adaboost所有参数选择默认值即可。针对三种不同生理信号，所使用的分类器训练集均来自于使用中国标准情绪视频素材库(CEVS)采集的数据集。在模型参数的设定上，使用GridSearch在所有列举的候选参数中，通过循环遍历，尝试每一个参数对识别结果的影响，去识别率最好的参数作为最终分类器使用的参数。 在每一组参数中，根据每一个具体的参数，设定合理最小化的步进值，每次控制一个参数，逐步按照步进值进行循环训练，直至得到最好的识别准确率才终止循环。针对每一种生理信号，都会使用以上分类器中，在训练中五折交叉验证准确率最好的分类器进行预测，得到每一种生理信号的情绪标签。以上使用标准数据库进行模型训练的时候，都已经将每一个模型对应的参数进行过调参处理，使得在交叉验证下，针对三种情绪的识别结果准确率最高。
综合评价模块是基于权重的投票法则，得出佩戴者当前时刻最终的情绪标签。基于权重的投票规则，心电数据量大，随情绪状态的变化敏感，且变化率快，而心率数据相对变化较慢，皮电数据变化最慢。心电数据所持有初始权重较皮电数据和心率数据高，心率数据初始权重较心电数据初始权重低，皮电数据的初始权重最低。例如：心电数据的初始权重可设置为50％，心率数据的初始权重可设置为30％，皮电数据的初始权重可设置为20％。佩戴者当前的情绪状态是总结三个情绪标签和对应的权重产生。每一次基于三种生理信号的标签，以及相应的权重比例进行投票，产生的最终标签就是佩戴者当前时刻的情绪状态。使用“正向”、“负向”、“中性”表征佩戴者的情绪状态。
综合评价模块在设置的初始权重时候，也可以心电数据所占的权重更高，皮电数据和心率数据所占比的权重相同且低于心电所占权重。12段心电数据产生的标签，取每一段标签“众数”作为心电数据所代表的总情绪标签，用三种生理信号的标签结合三者所持有的投票权重，产生总的情绪标签作为最终的识别结果。初始权重的设定标准，由标准数据集合情绪分类的准确率决定，取所有识别中准确率最高的权重作为初始的权重。在模型的更新过程中，权重也是基于当前最优准确率进行逐步调整，在每一次更新之后，仍能保持三个生理信号最佳的权重组合。
用户反馈模块的算法实现流程如图8所示，这是基于每一位手环的佩戴者进行“个性化”情绪识别的方案，针对每一位用户实际的情绪主观感受差异和使用场景的多样性，进行适配佩戴者的调整。佩戴者可根据实时反馈的情绪识别结果，及时通过移动端反馈自身的情绪感受。情绪分类模块和综合评价模块根据佩戴者反馈结果，动态调整分类器参数和投票权重，让情绪识别结果与用户反馈一致，最终形成与每一位手环佩戴者“个性化”的情绪识别算法，得到更加准确的情绪识别结果。
具体的说，在分类器动态调整中，首先将从佩戴者采集到的历史生理信号加入标准数据库，共同构成数据集的一部分，然后进行特征的提取，不断用个体标签进行模型再训练。在训练中对比新的数据集针对当前数据，识别的准确率有无提升，若无提升，则开始使用Grid_Search调整模型的参数，之后在进行准确率的对比，如没有上升，则在进行投票权重调整，寻找最高准确率和最佳模型、投票的权重组合，得出适合每一位佩戴者自身最佳的情绪 识别算法。
本发明情绪识别方法包括以下步骤：
步骤1：按照规定姿势正确佩戴手环，按照使用要求采集佩戴者的心率、心电、皮电三种生理数据；
步骤2：对三种生理数据分别进行预处理，包括信号放大去噪等处理，得到相对纯净的生理信号。从PPG测量信号中计算出佩戴者实时心率值，并将所得到的生理信号按照固定长度进行切分；
步骤3：生理信号提取模块对预处理之后的生理信号提取相应的生理参数，主要计算心电数据的线性、非线性、时域、频域特征参数，以及心率数据和皮电数据的统计特征参数，得到三种生理数据的特征参数；
步骤4：使用不同的分类器，分别对当前佩戴者的三种生理数据的特征参数进行情绪识别，得到当前佩戴者的三种情绪标签；
步骤5：综合评价，通过对三种情绪标签及其权重设置，其中心电数据的初始权重最高，心率数据的初始权重较心电数据低，皮电数据的初始权重最低，通过基于权重的投票规则，形成用户最终的情绪状态标签；
步骤6：根据用户反馈，实时调整情绪分类算法的参数和投票的权重，形成更加适配每一位用户的特定算法模型，在通过与佩戴者的交互中形成更加精准的可穿戴手环情绪识别算法模型。
上述实施例为本发明较佳的实施方式，但本发明的实施方式并不受上述实施例的限制，其他的任何未背离本发明的精神实质与原理下所作的改变、修饰、替代、组合、简化，均应为等效的置换方式，都包含在本发明的保护范围之内。

Claims (10)

  1. 一种基于可穿戴手环的情绪识别系统，其特征在于，包括生理信号采集模块、生理信号预处理模块、生理信号特征提取模块、情绪分类模块、综合评价模块；
    生理信号采集模块通过部署在可穿戴手环端的心电传感器、心率传感器、皮电传感器来分别采集佩戴者的心电、心率、皮电三种生理数据；生理信号预处理模块对三种生理数据进行数据切分，去噪之后传输至生理信号特征提取模块；生理信号特征提取模块对三种生理数据分别进行特征提取，所提取的特征包括线性特征、非线性特征、时域特性、频域特征；
    情绪分类模块针对三种生理数据进行情绪识别，基于每一种生理数据输出一个情绪状态；综合评价模块采用基于权重的投票决策规则，对三种生理数据输出的情绪状态进行投票决策，综合确定可穿戴手环佩戴者当前的情绪状态标签，得到识别结果。
  2. 根据权利要求1所述的情绪识别系统，其特征在于，所述情绪识别系统还包括用户反馈模块，用户反馈模块根据识别结果和可穿戴手环佩戴者当前时刻真实情绪感受的差异，向情绪分类模块反馈当前情绪状态描述；情绪分类模块根据所反馈的当前情绪状态描述和识别结果之间的差异，动态调整情绪分类模块和综合评价模块的参数，形成与每一位可穿戴手环佩戴者更加匹配的个性化情绪识别算法。
  3. 根据权利要求1所述的情绪识别系统，其特征在于，所述生理信号采集模块包括心率信号监测模块、心电信号监测模块和皮电信号监测模块；其中：
    心电信号监测模块采用双电极结构：一端电极集成在手环腕带用于固定的铆钉处，通过埋藏在腕带中的心电电极连接线和手环本体金属触点相连接，另一端电极集成在手环本体触摸点下方；
    心率信号监测模块是基于PPG原理的光传感部件，放置在手环本体的背部；
    皮电信号监测模块是集成在手环腕带之中的柔性传感器，位于手环腕带两侧中间部位。
  4. 根据权利要求1所述的情绪识别系统，其特征在于，所述生理信号预处理模块采用提高电路的共模抑制比设置模拟滤波器对信号去噪，采用小波变换、中值滤波方法处理信号。
  5. 根据权利要求1所述的情绪识别系统，其特征在于，所述生理信号特征提取模块，对心率数据和皮电数据的特征提取基于统计特征，从数据的变化范围、最大值、最小值、变化率、一阶差分、二阶差分变化情况进行统计；对心电数据的特征提取，从心电数据的时域、频域、线性、非线性四个层面进行分析，通过分析心电数据的变化情况来分析情绪变化对三种生理信号变化的影响；
    心率数据的特征参数包括：心率变化一阶差分平均值、二阶差分平均值，心率变化归一化一阶差分绝对值平均值，持续不变时间的最大值、最小值、平均值，持续上升变化的斜率，持续下降变化的斜率，相邻两个差值平方和的平均值；
    心电数据的特征参量包括全部窦性心博RR间期的标准差SDNN、相邻NN之差>50ms的个数NN50、相邻NN之差>50ms的个数占总窦性心搏个数的百分比PNN50、相邻RR间期差值的标准差SDSD、RR间隙的平均值RR_MEAN、小波变换统计特征，以及超低频VLF、低频LF、高频HF的频谱能量；
    皮电数据的特征参数包括最大值、最小值、均值、方差、变化率，一阶差分平均值、二阶差分平均值、归一化一阶差分平均值、归一化二阶差分平均值、变化范围、序列差值的平方和。
  6. 根据权利要求4所述的情绪识别系统，其特征在于，心率变化一阶差分平均值的计算公式如下：
    Figure PCTCN2019111531-appb-100001
    其中，X n表示时间t n时对应的心率值，N表示这一段心率值的长度；
    心率变化二阶差分平均值的计算如公式如下：
    Figure PCTCN2019111531-appb-100002
    心率变化归一化一阶差分平均值的计算过程如下：
    Figure PCTCN2019111531-appb-100003
    心率变化归一化二阶差分平均值的计算过程如下：
    Figure PCTCN2019111531-appb-100004
  7. 根据权利要求1所述的情绪识别系统，其特征在于，情绪分类模块单独针对三种生理数据进行情绪识别，得出每一种生理数据对应的情绪状态；所使用的分类器包括SVM、KNN、RF、DT、GBDT及AdaBoost；针对每一种生理数据，使用以上分类器中，在训练中五折交叉验证准确率最好的分类器进行预测，得到每一种生理数据的情绪标签。
  8. 根据权利要求7所述的情绪识别系统，其特征在于，综合评价模块基于权重的投票规则，得出佩戴者当前时刻最终的情绪标签；基于权重的投票规则中，心电数据所持有初始权重较皮电数据和心率数据高；佩戴者当前的情绪状态总结三个情绪标签和对应的权重产生；每一次基于三种生理数据的情绪标签，以及相应的权重比例进行投票，产生的最终标签为佩戴者当前时刻的情绪状态。
  9. 一种基于可穿戴手环的情绪识别方法，其特征在于，包括以下步骤：
    步骤1：采集手环佩戴者的心率、心电、皮电三种生理数据；
    步骤2：对三种生理数据分别进行预处理，包括信号放大、去噪处理，得到相对纯净的生理信号；
    步骤3：对预处理之后的生理信号提取相应的生理参数，计算心电数据的线性、非线性、时域、频域特征参数，以及心率数据和皮电数据的统计特征参数，得到三种生理数据的特征参数；
    步骤4：使用不同的分类器，分别对当前佩戴者的三种生理数据的特征参数进行情绪识别，得到当前佩戴者的三种情绪标签；
    步骤5：综合评价，通过对三种情绪标签及其权重设置，其中心电数据的初始权重最高，心率数据的初始权重较心电数据低，皮电数据的初始权重最低，通过基于权重的投票规则，形成用户最终的情绪状态标签。
  10. 根据权利要求9所述的情绪识别方法，其特征在于，还包括：
    步骤6：根据用户反馈，实时调整情绪分类算法的参数和投票的权重，形成更加适配每一位用户的特定算法模型，在通过与手环佩戴者的交互中形成更加精准的可穿戴手环情绪识别算法模型。
PCT/CN2019/111531 2018-12-12 2019-10-16 一种基于可穿戴手环的情绪识别系统及方法 WO2020119245A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201811518440.0A CN109620262B (zh) 2018-12-12 2018-12-12 一种基于可穿戴手环的情绪识别系统及方法
CN201811518440.0 2018-12-12

Publications (1)

Publication Number Publication Date
WO2020119245A1 true WO2020119245A1 (zh) 2020-06-18

Family

ID=66073106

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/111531 WO2020119245A1 (zh) 2018-12-12 2019-10-16 一种基于可穿戴手环的情绪识别系统及方法

Country Status (2)

Country Link
CN (1) CN109620262B (zh)
WO (1) WO2020119245A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112297023A (zh) * 2020-10-22 2021-02-02 新华网股份有限公司 智能陪护机器人系统

Families Citing this family (39)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109620262B (zh) * 2018-12-12 2020-12-22 华南理工大学 一种基于可穿戴手环的情绪识别系统及方法
CN110025323B (zh) * 2019-04-19 2021-07-27 西安科技大学 一种婴幼儿情绪识别方法
CN109907746A (zh) * 2019-04-22 2019-06-21 深圳仙苗科技有限公司 一种可采集紫外线与心电信号的健康手环及其使用方法
CN110379511B (zh) * 2019-05-27 2022-03-11 康岁医院投资管理有限公司 一种关于体质监测数据的分析及预警方法
CN110141259B (zh) * 2019-06-04 2021-09-28 清华大学 一种基于无线通信测量认知负荷与心理压力的方法及装置
CN110881987B (zh) * 2019-08-26 2022-09-09 首都医科大学 一种基于可穿戴设备的老年人情绪监测系统
CN110599999A (zh) * 2019-09-17 2019-12-20 寇晓宇 数据交互方法、装置和机器人
CN110717542A (zh) * 2019-10-12 2020-01-21 广东电网有限责任公司 一种情绪识别方法、装置和设备
CN111144436A (zh) * 2019-11-15 2020-05-12 北京点滴灵犀科技有限公司 基于可穿戴设备的情绪压力筛查及危机预警方法和装置
CN110916631B (zh) * 2019-12-13 2022-04-22 东南大学 基于可穿戴生理信号监测的学生课堂学习状态评测系统
CN111241583B (zh) * 2020-01-13 2023-03-31 桂林电子科技大学 可穿戴设备分类属性个性化本地差分隐私保护方法及系统
CN111248928A (zh) * 2020-01-20 2020-06-09 北京津发科技股份有限公司 压力识别方法及装置
CN111184521B (zh) * 2020-01-20 2023-01-06 北京津发科技股份有限公司 一种压力识别手环
CN113780546A (zh) * 2020-05-21 2021-12-10 华为技术有限公司 一种评估女性情绪的方法及相关装置、设备
CN112656387A (zh) * 2020-06-15 2021-04-16 杭州星迈科技有限公司 一种可穿戴设备数据的处理方法、装置和计算机设备
CN112206394A (zh) * 2020-10-09 2021-01-12 安徽美心信息科技有限公司 一种基于vr技术的心理训练互动系统及训练方法
CN112006700B (zh) * 2020-10-27 2021-07-16 成都大学 一种用于眼动仪的情感识别系统及方法
CN114403825B (zh) * 2020-10-28 2024-02-09 深圳市科瑞康实业有限公司 一种脉搏波信号识别方法和装置
CN112101823B (zh) * 2020-11-03 2021-03-02 四川大汇大数据服务有限公司 多维度情绪识别管理方法、系统、处理器、终端及介质
CN112618911B (zh) * 2020-12-31 2023-02-03 四川音乐学院 基于信号处理的音乐反馈调节系统
CN112618913A (zh) * 2021-01-06 2021-04-09 中国科学院心理研究所 一种基于多模态的自适应情绪调节系统与方法
CN112924660B (zh) * 2021-01-26 2023-09-26 上海浩创亘永科技有限公司 一种扫描系统及其扫描方法
CN112998711A (zh) * 2021-03-18 2021-06-22 华南理工大学 一种基于可穿戴装置的情绪识别系统及方法
CN113080970A (zh) * 2021-04-06 2021-07-09 北京体育大学 一种可穿戴情绪识别手环
CN113440122B (zh) * 2021-08-02 2023-08-22 北京理工新源信息科技有限公司 一种基于生命体征的情绪波动监测识别大数据预警系统
CN114053550A (zh) * 2021-11-19 2022-02-18 东南大学 一种基于高频心电的耳机式情绪压力调节装置
CN114259214A (zh) * 2021-12-21 2022-04-01 北京心华科技有限公司 身心健康数据检测、调节、筛查方法及系统
CN114391846B (zh) * 2022-01-21 2023-12-01 中山大学 一种基于过滤式特征选择的情绪识别方法及系统
CN114334090B (zh) * 2022-03-02 2022-07-12 博奥生物集团有限公司 一种数据分析方法、装置及电子设备
CN114515149B (zh) * 2022-03-17 2023-07-18 天津大学 一种基于多模态情感模型的情感识别装置
CN115192040B (zh) * 2022-07-18 2023-08-11 天津大学 基于庞加莱图和二阶差分图的脑电情绪识别方法及装置
CN115568853A (zh) * 2022-09-26 2023-01-06 山东大学 一种基于皮电信号的心理压力状态评估方法及系统
CN115633961A (zh) * 2022-10-28 2023-01-24 福州大学 基于动态加权决策融合恐高情绪识别模型构建方法与系统
CN115715680A (zh) * 2022-12-01 2023-02-28 杭州市第七人民医院 一种基于结缔组织电势的焦虑判别方法及装置
CN116077071B (zh) * 2023-02-10 2023-11-17 湖北工业大学 一种智能康复按摩方法、机器人及存储介质
CN116327123B (zh) * 2023-03-13 2023-08-18 深圳市雅为智能技术有限公司 一种基于智能手表的睡眠监测系统及方法
CN116636847A (zh) * 2023-06-02 2023-08-25 南京航空航天大学 一种基于腕部可穿戴设备的情感评估方法及系统
CN117204855B (zh) * 2023-10-26 2024-03-01 厚德明心(北京)科技有限公司 一种基于交互设备的用户心理状态评估方法及系统
CN117426774B (zh) * 2023-12-21 2024-04-09 深圳腾信百纳科技有限公司 一种基于智能手环的用户情绪评估方法及系统

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030139654A1 (en) * 2002-01-23 2003-07-24 Samsung Electronics Co., Ltd. System and method for recognizing user's emotional state using short-time monitoring of physiological signals
CN103584872A (zh) * 2013-10-29 2014-02-19 燕山大学 一种基于多生理参数融合的心理压力评估方法
CN108634969A (zh) * 2018-05-16 2018-10-12 京东方科技集团股份有限公司 情绪检测设备、情绪检测系统、情绪检测方法、存储介质
CN108742660A (zh) * 2018-07-02 2018-11-06 西北工业大学 一种基于可穿戴设备的情绪识别方法
CN108922626A (zh) * 2018-08-22 2018-11-30 华南师范大学 一种体征参数模型构建方法和体征参数评价方法
CN109620262A (zh) * 2018-12-12 2019-04-16 华南理工大学 一种基于可穿戴手环的情绪识别系统及方法

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060122474A1 (en) * 2000-06-16 2006-06-08 Bodymedia, Inc. Apparatus for monitoring health, wellness and fitness
TWI470564B (zh) * 2012-02-21 2015-01-21 Wistron Corp 使用者情緒偵測方法與應用其之手寫輸入電子裝置
CN103679214B (zh) * 2013-12-20 2017-10-20 华南理工大学 基于在线面积估计和多特征决策融合的车辆检测方法
WO2018035160A1 (en) * 2016-08-15 2018-02-22 The Regents Of The University Of California Bio-sensing and eye-tracking system
CN107220591A (zh) * 2017-04-28 2017-09-29 哈尔滨工业大学深圳研究生院 多模态智能情绪感知系统
CN108309328A (zh) * 2018-01-31 2018-07-24 南京邮电大学 一种基于自适应模糊支持向量机的情绪识别方法
CN108805088B (zh) * 2018-06-14 2021-05-28 南京云思创智信息科技有限公司 基于多模态情绪识别系统的生理信号分析子系统

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030139654A1 (en) * 2002-01-23 2003-07-24 Samsung Electronics Co., Ltd. System and method for recognizing user's emotional state using short-time monitoring of physiological signals
CN103584872A (zh) * 2013-10-29 2014-02-19 燕山大学 一种基于多生理参数融合的心理压力评估方法
CN108634969A (zh) * 2018-05-16 2018-10-12 京东方科技集团股份有限公司 情绪检测设备、情绪检测系统、情绪检测方法、存储介质
CN108742660A (zh) * 2018-07-02 2018-11-06 西北工业大学 一种基于可穿戴设备的情绪识别方法
CN108922626A (zh) * 2018-08-22 2018-11-30 华南师范大学 一种体征参数模型构建方法和体征参数评价方法
CN109620262A (zh) * 2018-12-12 2019-04-16 华南理工大学 一种基于可穿戴手环的情绪识别系统及方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112297023A (zh) * 2020-10-22 2021-02-02 新华网股份有限公司 智能陪护机器人系统

Also Published As

Publication number Publication date
CN109620262A (zh) 2019-04-16
CN109620262B (zh) 2020-12-22

Similar Documents

Publication Publication Date Title
WO2020119245A1 (zh) 一种基于可穿戴手环的情绪识别系统及方法
CN107569226B (zh) 基于压电传感获取hrv的方法及应用
Nguyen et al. A lightweight and inexpensive in-ear sensing system for automatic whole-night sleep stage monitoring
Liang et al. Automatic stage scoring of single-channel sleep EEG by using multiscale entropy and autoregressive models
CN109843163A (zh) 用于标记睡眠状态的方法和系统
CN105147248A (zh) 基于生理信息的抑郁症评估系统及其评估方法
CN204931634U (zh) 基于生理信息的抑郁症评估系统
CN107742534A (zh) 患者存活性预测系统
US10966662B2 (en) Motion-dependent averaging for physiological metric estimating systems and methods
CN110135285B (zh) 一种使用单导设备的脑电静息态身份认证方法及装置
CN106175754B (zh) 睡眠状态分析中清醒状态检测装置
CN114781465B (zh) 一种基于rPPG的非接触式疲劳检测系统及方法
Yan et al. Emotion classification with multichannel physiological signals using hybrid feature and adaptive decision fusion
Cecchi et al. Physical stimuli and emotions: EDA features analysis from a wrist-worn measurement sensor
CN111317446B (zh) 基于人体肌肉表面电信号的睡眠结构自动分析方法
Tiwari et al. A comparative study of stress and anxiety estimation in ecological settings using a smart-shirt and a smart-bracelet
CN113855053A (zh) 基于肌电的佩戴式肌肉阈值监测系统
CN109394203A (zh) 精神障碍康复期情绪监测与干预方法
CN115211858A (zh) 一种基于深度学习的情绪识别方法、系统及可存储介质
Wang et al. Emotionsense: An adaptive emotion recognition system based on wearable smart devices
CN114587288A (zh) 一种睡眠监测方法、装置以及设备
CN106333676B (zh) 清醒状态下的脑电信号数据类型的标注装置
CN106175698B (zh) 睡眠状态分析中睡眠周期检测装置
GB2600126A (en) Improvements in or relating to wearable sensor apparatus
CN115054248B (zh) 情绪监测方法和情绪监测装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19896805

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19896805

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 05.11.2021)

122 Ep: pct application non-entry in european phase

Ref document number: 19896805

Country of ref document: EP

Kind code of ref document: A1