CN115840890A - Emotion recognition method and device based on non-contact physiological signals - Google Patents
Emotion recognition method and device based on non-contact physiological signals Download PDFInfo
- Publication number
- CN115840890A CN115840890A CN202310158469.7A CN202310158469A CN115840890A CN 115840890 A CN115840890 A CN 115840890A CN 202310158469 A CN202310158469 A CN 202310158469A CN 115840890 A CN115840890 A CN 115840890A
- Authority
- CN
- China
- Prior art keywords
- contact
- emotion
- stress
- pulse wave
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
Abstract
The invention discloses an emotion recognition method and device based on non-contact physiological signals, and relates to the technical field of intelligent recognition. The method comprises the following steps: acquiring non-contact emotion perception data to be identified; inputting emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model; and realizing emotion recognition based on the non-contact physiological signal according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model. The invention designs a cognitive pressure and stress emotion induction experiment oriented to non-contact emotion perception verification, establishes a non-contact physiological signal detection and stress emotion recognition model by analyzing an association mechanism between non-contact emotional characteristics and stress emotion, realizes emotion perception based on non-contact physiological signals, and has the advantage of non-contact compared with the traditional physiological signal emotion perception method; compared with the emotion perception method based on expression voice, the method has the advantages that physiological signals are difficult to control autonomously, and the real emotion is expected to be revealed.
Description
Technical Field
The invention relates to the technical field of intelligent recognition, in particular to a method and a device for emotion recognition based on non-contact physiological signals.
Background
Emotion is the basis of human experience and influences multiple daily tasks such as cognition and perception in human life. In the research of artificial intelligence, one of the essential intelligence is the ability to identify, analyze, understand and express emotion. The Turing awarder Yann LeCun also emphasizes: "it is impossible to have intelligence without emotion". The concept of "emotion calculation" was first proposed by professor rosaled Picard at the university of massachusetts multimedia laboratory and defines emotion calculation as "calculation related to, derived from, or capable of exerting an influence on emotion". The emotion calculation has wide application prospects in the aspects of medical health, public safety and the like, and the application of the emotion calculation in medical mental health such as schizophrenia and suicide screening is also reviewed in the Nature article of 2020 by the Li Fei research group of Stanford university. At present, compared with researches in other artificial intelligence fields such as computer vision and natural language understanding, the research on emotion calculation is still in a relatively immature stage, namely, the acquisition of emotion information and the recognition of emotion are the premise of the research on emotion calculation, and conditions are provided for further analysis and understanding.
The emotion perception channel is multidimensional, if the emotion perception can be realized in a non-contact manner based on expressions and voices, the expressions can be recognized through a camera after imaging and combining with a computer vision algorithm, and the voices can also be collected through a microphone and then combining with a voice signal processing method. Furthermore, the fluctuation of emotion causes a change in physiological signals through the regulation of the autonomic nervous system of the human body, and thus the perception of emotional changes can be achieved based on the analysis of the physiological signals. Most of changes in the physiological signals cannot be subjectively controlled and cannot be hidden, and compared with dimensionalities such as expressions and voices, the real emotional states can be well reflected.
However, the conventional acquisition of physiological emotional characteristics mainly faces a bottleneck of limiting application scenarios due to contact signal acquisition. Physiological signal collection is realized by a contact-type sensor, a testee senses a signal collection device or introduces extra emotion, research results are further influenced, preparation work such as electrode patches is required to be arranged in advance, and application scenes are limited. The non-contact non-inductive measurement has great application value in the field of emotion calculation. Autonomic nervous activity during a stress response reflects an individual's ability to cope with a sudden situation. In high-pressure working environments (such as aviation and command), the perception of the stress and the stress level of key post workers is realized through a non-contact method, and the perception is important for improving the safety. However, the non-contact signal still faces the problem of low signal-to-noise ratio, which is the core difficulty for emotion perception based on the non-contact signal, and the noise challenge caused by motion, illumination and the like needs to be overcome. At present, a reliable and effective non-contact emotion perception method is lacked, the weak telemetering photoplethysmography pulse signal needs to be represented and enhanced urgently, a robust extraction method of physiological emotion characteristics is provided, and therefore a key technical support is provided for non-contact emotion perception.
Disclosure of Invention
The invention provides a method for acquiring physiological emotion characteristics, which aims at solving the problems that the conventional physiological emotion characteristics are mainly subjected to contact signal acquisition and the bottleneck of an application scene is limited and non-contact signals are subjected to low signal to noise ratio.
In order to solve the technical problems, the invention provides the following technical scheme:
in one aspect, the present invention provides a method for emotion recognition based on a non-contact physiological signal, the method being implemented by an electronic device, the method including:
s1, non-contact emotion perception data to be recognized are obtained.
And S2, inputting emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model.
And S3, emotion recognition based on the non-contact physiological signals is realized according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model.
Optionally, the construction process of the non-contact physiological signal detection and stress emotion perception model in S2 includes:
and S21, acquiring stress emotion data of the testee by finishing the stress emotion inducing task.
S22, carrying out non-contact physiological signal detection corresponding to the emotion data to obtain a non-contact pulse wave signal.
And S23, performing feature extraction on the non-contact pulse wave signals to obtain non-contact emotional features.
And S24, completing non-contact physiological signal detection and construction of a stress emotion perception model according to the stress emotion data and the non-contact emotional characteristics.
Optionally, the stress-emotion inducing tasks in S21 include a first-stage cognitive stress inducing task and a second-stage stress-induced emotion task.
The stress emotion inducing task further comprises a stress source; the stress source includes a social assessment threat task, a time stress task, and a loud audible feedback task.
Emotional stress data includes visual imaging data, physiological data, and episodic subjective self-reports.
Optionally, the non-contact physiological signal detection is performed on the corresponding emotional data in S22 to obtain a non-contact pulse wave signal, including:
the method comprises the steps of obtaining an original signal of visual imaging data in stress emotion data through a constructed space-time characteristic learning model, projecting the original signal to a plane orthogonal to the original signal, learning multiple color space transformation weights through multiple two-dimensional convolution weights of a first layer in the space-time characteristic learning model, and obtaining a non-contact pulse wave signal.
Optionally, the performing feature extraction on the non-contact pulse wave signal in S23 includes:
extracting heart rate variability features of the non-contact pulse wave signals, and extracting peripheral hemodynamic information features of the non-contact pulse wave signals.
Optionally, extracting heart rate variability features of the non-contact pulse wave signal comprises:
s231, modeling the non-contact pulse wave signal by a quasi-periodic signal, wherein the frequency of the non-contact pulse wave signal comprises an average heart rate and a change part caused by heart rate variability.
S232, constructing the instantaneous frequency of the non-contact pulse wave signal, wherein the instantaneous frequency of the non-contact pulse wave signal comprises the average heart rate and the instantaneous frequency change part caused by heart rate variability.
And S233, solving to obtain the instantaneous frequency caused by heart rate variability by adopting an instantaneous frequency extraction method based on pulse frequency demodulation and a discrete energy analysis algorithm, and completing the extraction of the heart rate variability characteristics of the non-contact pulse wave signal.
Optionally, the instantaneous frequency extraction method based on pulse frequency demodulation includes:
using a center frequency of
And the high cutoff frequency is->
And the low cutoff frequency is->
The band-pass filter extracts a first harmonic of the non-contact pulse wave signal; wherein BW is the bandwidth of the first harmonic of the non-contact pulse wave signal, BW is driven by the heart rate variability information of the fundamental sideband.
Optionally, the extracting peripheral hemodynamic information features of the non-contact pulse wave signal includes:
extracting the surrounding blood volume pulse waveform envelope characteristic of the non-contact pulse wave signal, and extracting the vasodilation and contraction movement characteristic of the non-contact pulse wave signal.
The method comprises the steps of extracting peripheral blood volume pulse waveform envelope characteristics of non-contact pulse wave signals, including performing Butterworth filtering with high cutoff frequency of 0.7Hz and low cutoff frequency of 3Hz, and completing extraction of the peripheral blood volume pulse waveform envelope characteristics of the non-contact pulse wave signals.
And extracting the vasodilation and contraction movement characteristics of the non-contact pulse wave signals, wherein the extraction of the vasodilation and contraction movement characteristics of the non-contact pulse wave signals is completed by performing Butterworth filtering with high cutoff frequency of 0.009Hz and low cutoff frequency of 0.2 Hz.
Optionally, in S24, completing the non-contact physiological signal detection and the construction of the stress emotion perception model according to the stress emotion data and the non-contact emotional characteristics, including:
and S241, recording the periodic subjective self-report in the corresponding emotional feeling data by adopting a Likter scale, and corresponding the subjective self-report estimation score to the perception stress emotion scores of three levels by using a clustering algorithm to obtain a truth value label of the periodic subjective self-report.
And S242, establishing a non-contact physiological signal detection and stress emotion perception model by adopting a data-driven method according to the periodic subjective self-report, the truth value label of the periodic subjective self-report, the heart rate variability characteristic of the non-contact pulse wave signal and the peripheral hemodynamic information characteristic of the non-contact pulse wave signal.
On the other hand, the invention provides an emotion recognition device based on a non-contact physiological signal, which is applied to realize an emotion recognition method based on the non-contact physiological signal, and comprises the following steps:
and the acquisition module is used for acquiring the non-contact emotion perception data to be identified.
And the input module is used for inputting the emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model.
And the output module is used for realizing emotion recognition based on the non-contact physiological signal according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model.
Optionally, the input module is further configured to:
and S21, acquiring stress emotion data of the testee by finishing the stress emotion inducing task.
S22, carrying out non-contact physiological signal detection corresponding to the emotion data to obtain a non-contact pulse wave signal.
And S23, performing feature extraction on the non-contact pulse wave signals to obtain non-contact emotional features.
And S24, completing non-contact physiological signal detection and construction of a stress emotion perception model according to the stress emotion data and the non-contact emotional characteristics.
Optionally, the stress-mood inducing tasks include a first-stage cognitive stress inducing task and a second-stage stress-inducing mood task.
Stress emotion inducing tasks also include stress source; the stressors include social assessment threat tasks, temporal stress tasks, and loud audible feedback tasks.
The stress mood data includes visual imaging data, physiological data and episodic subjective self-reports.
Optionally, the input module is further configured to:
the method comprises the steps of obtaining an original signal of visual imaging data in stress emotion data through a constructed space-time characteristic learning model, projecting the original signal to a plane orthogonal to the original signal, learning multiple color space transformation weights through multiple two-dimensional convolution weights of a first layer in the space-time characteristic learning model, and obtaining a non-contact pulse wave signal.
Optionally, the input module is further configured to:
the heart rate variability characteristics of the non-contact pulse wave signals are extracted, and the peripheral hemodynamic information characteristics of the non-contact pulse wave signals are extracted.
Optionally, the input module is further configured to:
s231, modeling the non-contact pulse wave signal by a quasi-periodic signal, wherein the frequency of the non-contact pulse wave signal comprises an average heart rate and a change part caused by heart rate variability.
S232, constructing the instantaneous frequency of the non-contact pulse wave signal, wherein the instantaneous frequency of the non-contact pulse wave signal comprises the average heart rate and the instantaneous frequency change part caused by heart rate variability.
And S233, solving to obtain the instantaneous frequency caused by heart rate variability by adopting an instantaneous frequency extraction method based on pulse frequency demodulation and a discrete energy analysis algorithm, and completing the extraction of the heart rate variability features of the non-contact pulse wave signals.
Optionally, the input module is further configured to:
using a center frequency of
And the high cutoff frequency is->
And the low cutoff frequency is->
The band-pass filter extracts a first harmonic of the non-contact pulse wave signal; wherein, BW is the bandwidth of the first harmonic of the non-contact pulse wave signal, and BW is driven by the heart rate variability information of the fundamental sideband.
Optionally, the input module is further configured to:
extracting the surrounding blood volume pulse waveform envelope characteristic of the non-contact pulse wave signal, and extracting the vasodilation and contraction movement characteristic of the non-contact pulse wave signal.
The method comprises the steps of extracting peripheral blood volume pulse waveform envelope characteristics of non-contact pulse wave signals, including performing Butterworth filtering with high cutoff frequency of 0.7Hz and low cutoff frequency of 3Hz, and completing extraction of the peripheral blood volume pulse waveform envelope characteristics of the non-contact pulse wave signals.
And extracting the vasodilation and contraction movement characteristics of the non-contact pulse wave signals, wherein the extraction of the vasodilation and contraction movement characteristics of the non-contact pulse wave signals is completed by performing Butterworth filtering with high cutoff frequency of 0.009Hz and low cutoff frequency of 0.2 Hz.
Optionally, the input module is further configured to:
and S241, recording the periodic subjective self-report in the corresponding emotional feeling data by adopting a Likter scale, and corresponding the subjective self-report estimation score to the perception stress emotion scores of three levels by using a clustering algorithm to obtain a truth value label of the periodic subjective self-report.
And S242, establishing a non-contact physiological signal detection and stress emotion perception model by adopting a data-driven method according to the periodic subjective self-report, the truth value label of the periodic subjective self-report, the heart rate variability characteristic of the non-contact pulse wave signal and the peripheral hemodynamic information characteristic of the non-contact pulse wave signal.
In one aspect, an electronic device is provided, which includes a processor and a memory, where at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the method for emotion recognition based on non-contact physiological signals.
In one aspect, a computer-readable storage medium is provided, in which at least one instruction is stored, and the at least one instruction is loaded and executed by a processor to implement the above method for emotion recognition based on non-contact physiological signals.
Compared with the prior art, the technical scheme at least has the following beneficial effects:
according to the scheme, a cognitive pressure and stress emotion induction experiment oriented to non-contact emotion perception verification is designed, a non-contact physiological signal detection and stress emotion perception model is established by analyzing an association mechanism between non-contact emotional characteristics and stress emotion, emotion perception based on non-contact physiological signals is achieved, and compared with a traditional physiological signal emotion perception method, the method has the advantage of non-contact; compared with the emotion perception method based on expression voice, the method has the advantages that physiological signals are difficult to control autonomously, and the real emotion is expected to be revealed.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flow chart of a method for emotion recognition based on a non-contact physiological signal according to an embodiment of the present invention;
fig. 2 is a main flow chart of a non-contact physiological signal-based emotion recognition method provided by an embodiment of the invention;
FIG. 3 is a block diagram of a non-contact physiological signal based emotion recognition device provided by an embodiment of the invention;
fig. 4 is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings of the embodiments of the present invention. It is to be understood that the embodiments described are only a few embodiments of the present invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the invention without any inventive step, are within the scope of protection of the invention.
As shown in fig. 1, an embodiment of the present invention provides an emotion recognition method based on a non-contact physiological signal, which may be implemented by an electronic device. As shown in fig. 1, a flowchart of a method for emotion recognition based on a non-contact physiological signal, a processing flow of the method may include the following steps:
s1, non-contact emotion perception data to be recognized are obtained.
In a possible implementation, the non-contact emotion perception data may be expression data acquired by camera imaging, voice data acquired by a microphone, or the like.
And S2, inputting emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model.
Optionally, the construction process of the non-contact physiological signal detection and stress emotion perception model in S2 includes S21-S24:
and S21, acquiring stress emotion data of the tested object by finishing the stress emotion induction task.
Optionally, the stress-emotion inducing tasks in S21 include a first-stage cognitive stress inducing task and a second-stage stress-induced emotion task.
Stress emotion inducing tasks also include stress source; the stressors include social assessment threat tasks, temporal stress tasks, and loud audible feedback tasks.
The stress mood data includes visual imaging data, physiological data and episodic subjective self-reports.
In one possible embodiment, as shown in fig. 2, two types of stress emotion inducing tasks can be used, the first phase being cognitive stress induction, such as the Stroop color word test and mental arithmetic test, and the second phase being stress emotion inducing, such as public interview, the two phases being employed on different subjects to avoid sequential effects using the latin square design. The two-stage stress response induction experiments are proved by related researches to be capable of effectively inducing stress emotion and observable physiological signals and cortisol change. Although Stroop and mental tasks have been shown to cause stress responses, the experimentally induced stress is not strong enough to detect distinguishable non-contact physiological signals. Therefore, on the basis of previous studies, it was also planned to introduce further pressure sources, including: (1) Social assessment threat, namely, close observation and assessment are carried out on the performance of a person, three main test experimenters are arranged in the opposite direction of a tested person in the experiment, and the tested person is informed to be assessed; (2) Time pressure, which will set time limits in both Stroop and mental tasks, and (3) loud audible feedback, especially a sharp high-pitched audible feedback that is emitted when an answer is wrong.
S22, carrying out non-contact physiological signal detection corresponding to the emotion data to obtain a non-contact pulse wave signal.
Optionally, the step S22 may include:
the method comprises the steps of obtaining an original signal of visual imaging data in stress emotion data through a constructed space-time characteristic learning model, projecting the original signal to a plane orthogonal to the original signal, learning multiple color space transformation weights through multiple two-dimensional convolution weights of a first layer in the space-time characteristic learning model, and obtaining a non-contact pulse wave signal.
In one possible implementation, the non-contact pulse wave recovery is to extract subtle continuous color changes of the face due to the absorption and reflection characteristics of the face in a continuous sequence of video frames. It is therefore an intrinsic feature analysis problem in video frame sequences that can be modeled by spatio-temporal feature characterization learning. In the design of a network structure, in order to eliminate light intensity change in the skin tone direction, signals can be projected to a plane orthogonal to the light intensity change, therefore, a plurality of two-dimensional convolution weights of a first layer in the network structure are learned through space-time characteristic representation, a plurality of color space transformation weights are learned, original signals are projected to the plane orthogonal to the original signals to weaken the influence of ambient light change, and then non-contact pulse waves with stronger signal-to-noise ratio are extracted.
And S23, performing feature extraction on the non-contact pulse wave signals to obtain non-contact emotional features.
Optionally, the step S23 may include: extracting heart rate variability features of the non-contact pulse wave signals and extracting peripheral hemodynamic information features of the non-contact pulse wave signals.
Optionally, the extracting of the heart rate variability feature of the non-contact pulse wave signal comprises:
s231, modeling the non-contact pulse wave signal by a quasi-periodic signal, wherein the frequency of the non-contact pulse wave signal comprises an average heart rate and a change part caused by heart rate variability.
S232, constructing the instantaneous frequency of the non-contact pulse wave signal, wherein the instantaneous frequency of the non-contact pulse wave signal comprises the average heart rate and the instantaneous frequency change part caused by heart rate variability.
And S233, solving to obtain the instantaneous frequency caused by heart rate variability by adopting an instantaneous frequency extraction method based on pulse frequency demodulation and a discrete energy analysis algorithm, and completing the extraction of the heart rate variability characteristics of the non-contact pulse wave signal.
In a possible embodiment, although the accuracy of the recovered pulse wave signal can be improved in step S22, the pulse wave signal obtained by the telemetry method still has a relatively low signal-to-noise ratio compared to the contact type, so that the extraction of the heart beat interval based on peak point positioning is susceptible to significant interference, and thus the invention provides a robust extraction method for heart rate variability features in a non-contact pulse wave signal under the condition of a low signal-to-noise ratio. The heart rate variability describes the change of heart beating intervals, the time period which can be provided by the non-contact emotion perception requirement is usually short, and the corresponding short-time heart rate variability analysis statistical characteristics mainly comprise the standard deviation of the heart beating intervals and the root-mean-square difference between continuous heart beating intervals, which respectively correspond to the low-frequency components and the high-frequency components of the heart rate variability. The pulse wave signal is modeled as a quasi-periodic signal whose frequency is made up of the average heart rate and the varying part caused by the variability of the heart rate. Specifically, it can be expressed as the following formula (1):
wherein the content of the first and second substances,
representing a non-contact pulse wave signal; />
Is indicated to be at>
Moment->
The amplitude of the subharmonic;
is indicated to be at>
Moment->
Instantaneous phase of the subharmonic.
Further, the air conditioner is provided with a fan,
is momentarily frequency->
Including the average heart rate and the instantaneous frequency variation caused by the heart rate variability, as shown in the following formula (2): />
Wherein the content of the first and second substances,
represents the average heart rate; />
Representing the instantaneous frequency change caused by heart rate variability.
Further, the frequency energy of the pulse wave signal is mainly concentrated on
Of the first harmonics, the first harmonic can be expressed by the following formula (3):
wherein, the first and the second end of the pipe are connected with each other,
and &>
Corresponding to the amplitude adjustment and frequency adjustment components, respectively.
Further, inspired by an algorithm of contact type pulse wave signal processing, a pulse frequency domain demodulation method is adopted to estimate the instantaneous frequency of the fundamental wave signal. Assuming the first harmonic
The bandwidth of (b) is described as BW. BW is driven by the heart rate variability information of the fundamental sideband. Using a centreHas a frequency of->
The high and low cutoff frequencies are respectively->
And
the harmonic component can be separated out by the bandpass filter, wherein the harmonic component is present in the filter>
Indicates the ^ th or greater than currently analyzed>
A signal segment. The instantaneous frequency extraction is a typical frequency demodulation problem. To utilize a time-period independent discrete energy analysis algorithm, the raw signal can be represented as a discrete sequence, where m corresponds to discrete samples of the signal, and the following formula describes the specific steps:
and by filtering
Remove by>
The generated high-frequency component is represented by the following formulae (5) (6):
further, to avoid the false estimation caused by sudden discontinuity of signal due to sudden sparse noise artifact, the threshold is set as formula (5)
. So that the instantaneous frequency of the heart rate variability can be determined by &>
And (4) calculating.
Optionally, the peripheral hemodynamic information feature extraction on the non-contact pulse wave signal includes:
extracting the surrounding blood volume pulse waveform envelope characteristic of the non-contact pulse wave signal and extracting the vasodilation and contraction movement characteristic of the non-contact pulse wave signal.
The peripheral blood volume pulse waveform envelope feature extraction of the non-contact pulse wave signal comprises the step of carrying out Butterworth filtering with high cutoff frequency of 0.7Hz and low cutoff frequency of 3Hz, and the peripheral blood volume pulse waveform envelope feature extraction of the non-contact pulse wave signal is completed.
The extraction of the vasodilatation and contraction movement characteristics of the non-contact pulse wave signals comprises the step of carrying out Butterworth filtering with high cutoff frequency of 0.009Hz and low cutoff frequency of 0.2Hz, and the extraction of the vasodilatation and contraction movement characteristics of the non-contact pulse wave signals is completed.
In one possible embodiment, the recovered pulse wave further contains peripheral hemodynamic information, including the envelope of the peripheral blood volume pulse waveform, vasodilation, etc. The specific extraction method comprises the following steps: for heart rate variability analysis, high frequency components (0.15-0.4 Hz) and low frequency components (0.04-0.15 Hz) were extracted, and the LF/HF (low/high frequency) ratio was calculated as the ratio of power in the range of 0.01-0.15Hz to power in the range of 0.15-0.4 Hz. The normalized power is calculated by dividing the low and high frequency power by the total power of 0.04-0.4 Hz. The LF/HF ratio is dimensionless and therefore not affected by normalization. For the envelope characteristics of the peripheral blood volume pulse waveform, butterworth filtering is carried out on the high cut-off frequency and the low cut-off frequency of 0.7Hz and 3Hz respectively, then the integral of the power spectrum of the signal in the same frequency band is calculated, and the measurement of the signal amplitude is obtained. For the vasodilatory-systolic motion description, butterworth filtering is also applied to the signals (with cut-off frequencies of 0.009 and 0.2 Hz), the resulting signals are vasodilatory estimates, and then the integrals of the power spectra of the signals in the same frequency band are calculated, thus enabling the extraction of the pulse wave time-frequency domain variability features.
And S24, completing non-contact physiological signal detection and construction of a stress emotion perception model according to the stress emotion data and the non-contact emotional characteristics.
Optionally, the step S24 may include S241-S242:
and S241, recording the periodic subjective self-report in the corresponding emotional feeling data by adopting a Likter scale, and corresponding the subjective self-report estimation score to the perception stress emotion scores of three levels by using a clustering algorithm to obtain a truth value label of the periodic subjective self-report.
And S242, establishing a non-contact physiological signal detection and stress emotion perception model by adopting a data-driven method according to the periodic subjective self-report, the truth value label of the periodic subjective self-report, the heart rate variability characteristic of the non-contact pulse wave signal and the peripheral hemodynamic information characteristic of the non-contact pulse wave signal.
In one possible embodiment, the subjective report of the subject is collected by questionnaire after step S21 and recorded using the likert scale. The self-reported differences among individuals with perceived psychological stress are often large, and therefore a standardized clustering method is adopted to calibrate subjective evaluation. And the subjective evaluation scores of the participants are corresponding to the perception stress emotion scores of three levels through a clustering algorithm so as to obtain related truth value labels, and a data-driven method is adopted to establish a stress emotion estimation model by combining the extracted multiple pulse waves and the hemodynamic physiological indexes. After the model is built, the step one does not appear in the actual use, but the non-contact physiological emotion characteristics output in the step two and the step three are directly input into the pre-built model to directly output the non-contact emotion perception result.
And S3, emotion recognition based on the non-contact physiological signals is realized according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model.
Based on emotion perception of physiological signal sensing, application scenes are limited due to perception of an acquisition device or additional emotion to be introduced; the emotion perception method based on video expression analysis is difficult to reveal real emotion due to the fact that macroscopic expression can be disguised; compared with the non-contact pulse wave extraction method, the non-contact pulse wave extraction method has the application potential of stress emotion assessment in natural open scenes due to the non-contact, rapid and universal autonomic nervous system reaction capability. The pulse wave signal not only contains heart rate information, but also changes the blood volume caused by the propagation of the pulse pressure in the artery, thereby changing the pulse shape and the time characteristics, and can provide a great deal of information concerning the vascular system, including autonomic nerve functions and blood vessel characteristics. Based on the method, visual imaging data, physiological data and periodic subjective self-reports related to psychological stress level are collected through stress emotion induction, emotion characteristic extraction is carried out by combining non-contact physiological signal detection, a non-contact physiological signal detection and stress emotion perception model is established through correlation mechanism analysis between emotion characteristics and stress emotion, and non-contact stress emotion perception based on physiological signals is achieved. In addition, aiming at the problem that the HRV analysis peak value positioning error with low signal-to-noise ratio is high, an instantaneous frequency extraction method based on pulse frequency demodulation is provided.
In the embodiment of the invention, a cognitive pressure and stress emotion induction experiment oriented to non-contact emotion perception verification is designed, a non-contact physiological signal detection and stress emotion perception model is established by analyzing an association mechanism between non-contact emotional characteristics and stress emotion, emotion perception based on non-contact physiological signals is realized, and compared with the traditional physiological signal emotion perception method, the method has the advantage of non-contact; compared with the emotion perception method based on expression voice, the method has the advantages that physiological signals are difficult to control autonomously, and the real emotion is expected to be revealed.
As shown in fig. 3, an embodiment of the present invention provides an emotion recognition apparatus 300 based on a non-contact physiological signal, where the apparatus 300 is applied to implement an emotion recognition method based on a non-contact physiological signal, and the apparatus 300 includes:
an obtaining module 310, configured to obtain non-contact emotion perception data to be recognized.
And the input module 320 is used for inputting the emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model.
And the output module 330 is configured to implement emotion recognition based on the non-contact physiological signal according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model.
Optionally, the input module 320 is further configured to:
and S21, acquiring stress emotion data of the testee by finishing the stress emotion inducing task.
S22, carrying out non-contact physiological signal detection corresponding to the emotion data to obtain a non-contact pulse wave signal.
And S23, performing feature extraction on the non-contact pulse wave signals to obtain non-contact emotional features.
And S24, completing non-contact physiological signal detection and construction of a stress emotion perception model according to the stress emotion data and the non-contact emotional characteristics.
Optionally, the stress-mood inducing tasks include a first-stage cognitive stress inducing task and a second-stage stress-inducing mood task.
The stress emotion inducing task further comprises a stress source; the stressors include social assessment threat tasks, temporal stress tasks, and loud audible feedback tasks.
The stress mood data includes visual imaging data, physiological data and episodic subjective self-reports.
Optionally, the input module 320 is further configured to:
the method comprises the steps of obtaining an original signal of visual imaging data in stress emotion data through a constructed space-time characteristic learning model, projecting the original signal to a plane orthogonal to the original signal, learning multiple color space transformation weights through multiple two-dimensional convolution weights of a first layer in the space-time characteristic learning model, and obtaining a non-contact pulse wave signal.
Optionally, the input module 320 is further configured to:
extracting heart rate variability features of the non-contact pulse wave signals, and extracting peripheral hemodynamic information features of the non-contact pulse wave signals.
Optionally, the input module 320 is further configured to:
s231, modeling the non-contact pulse wave signal by a quasi-periodic signal, wherein the frequency of the non-contact pulse wave signal comprises an average heart rate and a change part caused by heart rate variability.
S232, constructing the instantaneous frequency of the non-contact pulse wave signal, wherein the instantaneous frequency of the non-contact pulse wave signal comprises the average heart rate and the instantaneous frequency change part caused by heart rate variability.
And S233, solving to obtain the instantaneous frequency caused by heart rate variability by adopting an instantaneous frequency extraction method based on pulse frequency demodulation and a discrete energy analysis algorithm, and completing the extraction of the heart rate variability characteristics of the non-contact pulse wave signal.
Optionally, the input module 320 is further configured to:
using a center frequency of
And the high cutoff frequency is->
And the low cutoff frequency is->
The band-pass filter extracts a first harmonic wave of the non-contact pulse wave signal; wherein, BW is the bandwidth of the first harmonic of the non-contact pulse wave signal, and BW is driven by the heart rate variability information of the fundamental sideband.
Optionally, the input module 320 is further configured to:
extracting the surrounding blood volume pulse waveform envelope characteristic of the non-contact pulse wave signal, and extracting the vasodilation and contraction movement characteristic of the non-contact pulse wave signal.
The method comprises the steps of extracting peripheral blood volume pulse waveform envelope characteristics of non-contact pulse wave signals, including performing Butterworth filtering with high cutoff frequency of 0.7Hz and low cutoff frequency of 3Hz, and completing extraction of the peripheral blood volume pulse waveform envelope characteristics of the non-contact pulse wave signals.
And extracting the vasodilation and contraction movement characteristics of the non-contact pulse wave signals, wherein the extraction of the vasodilation and contraction movement characteristics of the non-contact pulse wave signals is completed by performing Butterworth filtering with high cutoff frequency of 0.009Hz and low cutoff frequency of 0.2 Hz.
Optionally, the input module 320 is further configured to:
and S241, recording the periodic subjective self-report in the corresponding emotional feeling data by adopting a Likter scale, and corresponding the subjective self-report estimation score to the perception stress emotion scores of three levels by using a clustering algorithm to obtain a truth value label of the periodic subjective self-report.
And S242, establishing a non-contact physiological signal detection and stress emotion perception model by adopting a data-driven method according to the periodic subjective self-report, the truth value label of the periodic subjective self-report, the heart rate variability characteristic of the non-contact pulse wave signal and the peripheral hemodynamic information characteristic of the non-contact pulse wave signal.
In the embodiment of the invention, a cognitive pressure and stress emotion induction experiment oriented to non-contact emotion perception verification is designed, a non-contact physiological signal detection and stress emotion perception model is established by analyzing an association mechanism between non-contact emotional characteristics and stress emotion, emotion perception based on non-contact physiological signals is realized, and compared with the traditional physiological signal emotion perception method, the method has the advantage of non-contact; compared with the emotion perception method based on expression voice, the method has the advantages that physiological signals are difficult to control autonomously, and the real emotion is expected to be revealed.
Fig. 4 is a schematic structural diagram of an electronic device 400 according to an embodiment of the present invention, where the electronic device 400 may generate a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 401 and one or more memories 402, where at least one instruction is stored in the memory 402, and the at least one instruction is loaded and executed by the processor 401 to implement the following emotion recognition method based on a non-contact physiological signal:
s1, non-contact emotion perception data to be recognized are obtained.
And S2, inputting emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model.
And S3, emotion recognition based on the non-contact physiological signals is realized according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model.
In an exemplary embodiment, there is also provided a computer readable storage medium, such as a memory including instructions executable by a processor in a terminal to perform the above-described non-contact physiological signal based emotion recognition method. For example, the computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.
It will be understood by those skilled in the art that all or part of the steps for implementing the above embodiments may be implemented by hardware, or may be implemented by a program instructing relevant hardware, where the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, etc.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (10)
1. A method for emotion recognition based on a non-contact physiological signal, the method comprising:
s1, acquiring non-contact emotion perception data to be identified;
s2, inputting the emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model;
and S3, realizing emotion recognition based on the non-contact physiological signal according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model.
2. The method according to claim 1, wherein the construction process of the non-contact physiological signal detection and stress emotion perception model in S2 comprises:
s21, acquiring stress emotion data of the testee by completing a stress emotion inducing task;
s22, carrying out non-contact physiological signal detection on the stress emotion data to obtain a non-contact pulse wave signal;
s23, extracting the characteristics of the non-contact pulse wave signals to obtain non-contact emotional characteristics;
and S24, completing non-contact physiological signal detection and construction of a stress emotion perception model according to the stress emotion data and the non-contact emotion characteristics.
3. The method according to claim 2, wherein the stress-emotion-inducing tasks in S21 include a first-stage cognitive stress-inducing task and a second-stage stress-emotion-inducing task;
the stress emotion inducing task further comprises a pressure source; the pressure source comprises a social assessment threat task, a time pressure task and a loud sound feedback task;
the stress mood data includes visual imaging data, physiological data, and episodic subjective self-reports.
4. The method according to claim 2, wherein the non-contact physiological signal detection of the emotional stress data in S22 to obtain a non-contact pulse wave signal comprises:
and acquiring an original signal of visual imaging data in the stress emotion data through a constructed space-time characteristic representation learning model, projecting the original signal to a plane orthogonal to the original signal, and learning multiple color space transformation weights through multiple two-dimensional convolution weights of a first layer in the space-time characteristic representation learning model to obtain a non-contact pulse wave signal.
5. The method according to claim 2, wherein the feature extraction of the non-contact pulse wave signal in S23 includes:
extracting heart rate variability features of the non-contact pulse wave signal, and extracting peripheral hemodynamic information features of the non-contact pulse wave signal.
6. The method according to claim 5, wherein said extracting heart rate variability features of said non-contact pulse wave signal comprises:
s231, modeling the non-contact pulse wave signal by a quasi-periodic signal, wherein the frequency of the non-contact pulse wave signal comprises an average heart rate and a change part caused by heart rate variability;
s232, constructing the instantaneous frequency of the non-contact pulse wave signal, wherein the instantaneous frequency of the non-contact pulse wave signal comprises an average heart rate and an instantaneous frequency change part caused by heart rate variability;
and S233, solving to obtain the instantaneous frequency caused by heart rate variability by adopting an instantaneous frequency extraction method based on pulse frequency demodulation and a discrete energy analysis algorithm, and completing the extraction of the heart rate variability characteristics of the non-contact pulse wave signal.
7. The method of claim 6, wherein the instantaneous frequency extraction method based on pulse frequency demodulation comprises:
using a center frequency of
A high cut-off frequency of
A low cut-off frequency of
The band-pass filter extracts a first harmonic of the non-contact pulse wave signal; wherein BW is the bandwidth of the first harmonic of the non-contact pulse wave signal, the BW being driven by the heart rate variability information of the fundamental sideband.
8. The method according to claim 5, wherein the extracting peripheral hemodynamic information characteristic of the non-contact pulse wave signal comprises:
extracting peripheral blood volume pulse waveform envelope characteristics of the non-contact pulse wave signals and extracting vasodilatation and contraction movement characteristics of the non-contact pulse wave signals;
extracting the surrounding blood volume pulse waveform envelope characteristics of the non-contact pulse wave signals, wherein the extracting comprises Butterworth filtering with high cut-off frequency of 0.7Hz and low cut-off frequency of 3Hz, and the extracting of the surrounding blood volume pulse waveform envelope characteristics of the non-contact pulse wave signals is completed;
and extracting the vasodilatation and contraction movement characteristics of the non-contact pulse wave signals, wherein the extraction of the vasodilatation and contraction movement characteristics of the non-contact pulse wave signals is completed by performing Butterworth filtering with high cutoff frequency of 0.009Hz and low cutoff frequency of 0.2 Hz.
9. The method according to claim 2, wherein the step of completing the construction of the non-contact physiological signal detection and stress emotion perception model according to the stress emotion data and the non-contact emotional characteristics in the step S24 comprises:
s241, recording the staged subjective self-report in the stress emotion data by adopting a Likter scale, and corresponding the subjective self-report estimation score to perception stress emotion scores of three levels through a clustering algorithm to obtain a truth label of the staged subjective self-report;
and S242, establishing a non-contact physiological signal detection and stress emotion perception model by adopting a data-driven method according to the periodic subjective self-report, the truth value label of the periodic subjective self-report, the heart rate variability characteristics of the non-contact pulse wave signal and the peripheral hemodynamic information characteristics of the non-contact pulse wave signal.
10. An emotion recognition apparatus based on a non-contact physiological signal, characterized in that the apparatus comprises:
the acquisition module is used for acquiring non-contact emotion perception data to be identified;
the input module is used for inputting the emotion perception data into the constructed non-contact physiological signal detection and stress emotion perception model;
and the output module is used for realizing emotion recognition based on the non-contact physiological signal according to the emotion perception data and the non-contact physiological signal detection and stress emotion perception model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310158469.7A CN115840890B (en) | 2023-02-24 | 2023-02-24 | Emotion recognition method and device based on non-contact physiological signals |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310158469.7A CN115840890B (en) | 2023-02-24 | 2023-02-24 | Emotion recognition method and device based on non-contact physiological signals |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115840890A true CN115840890A (en) | 2023-03-24 |
| CN115840890B CN115840890B (en) | 2023-05-19 |
Family
ID=85580100
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310158469.7A Active CN115840890B (en) | 2023-02-24 | 2023-02-24 | Emotion recognition method and device based on non-contact physiological signals |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115840890B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758619A (en) * | 2023-08-17 | 2023-09-15 | 山东大学 | Facial video-based emotion classification method, system, storage medium and equipment |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107212896A (en) * | 2017-04-05 | 2017-09-29 | 天津大学 | A kind of emotional stability overall evaluation system and information processing method |
| CN107220591A (en) * | 2017-04-28 | 2017-09-29 | 哈尔滨工业大学深圳研究生院 | Multi-modal intelligent mood sensing system |
| US20200075040A1 (en) * | 2018-08-31 | 2020-03-05 | The Regents Of The University Of Michigan | Automatic speech-based longitudinal emotion and mood recognition for mental health treatment |
| US20210007607A1 (en) * | 2015-06-14 | 2021-01-14 | Facense Ltd. | Monitoring blood sugar level with a comfortable head-mounted device |
| CN113017630A (en) * | 2021-03-02 | 2021-06-25 | 贵阳像树岭科技有限公司 | Visual perception emotion recognition method |
| RU2750745C1 (en) * | 2020-09-23 | 2021-07-02 | Общество с ограниченной ответственностью «Оптические медицинские диагностические системы» (ООО «ОДС-мед») | Method and device for measuring the pulse wave propagation velocity when measuring blood pressure by oscillometric method |
| CN113229790A (en) * | 2021-05-17 | 2021-08-10 | 浙江大学 | Non-contact mental stress assessment system |
| US20210398337A1 (en) * | 2020-06-19 | 2021-12-23 | Microsoft Technology Licensing, Llc | Generating physio-realistic avatars for training non-contact models to recover physiological characteristics |
| US20220015654A1 (en) * | 2018-08-29 | 2022-01-20 | Sleep Advice Technologies S.R.L. | Photoplethysmography based detection of transitions between awake, drowsiness, and sleep phases of a subject |
| CN114305325A (en) * | 2020-09-30 | 2022-04-12 | 华为云计算技术有限公司 | Emotion detection method and device |
| CN115624322A (en) * | 2022-11-17 | 2023-01-20 | 北京科技大学 | Non-contact physiological signal detection method and system based on efficient space-time modeling |
-
2023
- 2023-02-24 CN CN202310158469.7A patent/CN115840890B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20210007607A1 (en) * | 2015-06-14 | 2021-01-14 | Facense Ltd. | Monitoring blood sugar level with a comfortable head-mounted device |
| CN107212896A (en) * | 2017-04-05 | 2017-09-29 | 天津大学 | A kind of emotional stability overall evaluation system and information processing method |
| CN107220591A (en) * | 2017-04-28 | 2017-09-29 | 哈尔滨工业大学深圳研究生院 | Multi-modal intelligent mood sensing system |
| US20220015654A1 (en) * | 2018-08-29 | 2022-01-20 | Sleep Advice Technologies S.R.L. | Photoplethysmography based detection of transitions between awake, drowsiness, and sleep phases of a subject |
| US20200075040A1 (en) * | 2018-08-31 | 2020-03-05 | The Regents Of The University Of Michigan | Automatic speech-based longitudinal emotion and mood recognition for mental health treatment |
| US20210398337A1 (en) * | 2020-06-19 | 2021-12-23 | Microsoft Technology Licensing, Llc | Generating physio-realistic avatars for training non-contact models to recover physiological characteristics |
| RU2750745C1 (en) * | 2020-09-23 | 2021-07-02 | Общество с ограниченной ответственностью «Оптические медицинские диагностические системы» (ООО «ОДС-мед») | Method and device for measuring the pulse wave propagation velocity when measuring blood pressure by oscillometric method |
| CN114305325A (en) * | 2020-09-30 | 2022-04-12 | 华为云计算技术有限公司 | Emotion detection method and device |
| CN113017630A (en) * | 2021-03-02 | 2021-06-25 | 贵阳像树岭科技有限公司 | Visual perception emotion recognition method |
| CN113229790A (en) * | 2021-05-17 | 2021-08-10 | 浙江大学 | Non-contact mental stress assessment system |
| CN115624322A (en) * | 2022-11-17 | 2023-01-20 | 北京科技大学 | Non-contact physiological signal detection method and system based on efficient space-time modeling |
Non-Patent Citations (1)
| Title |
|---|
| 杨乔生 等: "基于心率检测的便携式儿童情绪感知系统研制" * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116758619A (en) * | 2023-08-17 | 2023-09-15 | 山东大学 | Facial video-based emotion classification method, system, storage medium and equipment |
| CN116758619B (en) * | 2023-08-17 | 2023-11-24 | 山东大学 | Facial video-based emotion classification method, system, storage medium and equipment |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115840890B (en) | 2023-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Zhao et al. | EmotionSense: Emotion recognition based on wearable wristband | |
| CN108366732B (en) | Initial sleepiness test | |
| US8862581B2 (en) | Method and system for concentration detection | |
| RU2708807C2 (en) | Algorithm of integrated remote contactless multichannel analysis of psychoemotional and physiological state of object based on audio and video content | |
| CN108742660A (en) | A kind of Emotion identification method based on wearable device | |
| Park et al. | Real-time estimation of respiratory rate from a photoplethysmogram using an adaptive lattice notch filter | |
| CN111523601A (en) | Latent emotion recognition method based on knowledge guidance and generation counterstudy | |
| US20240081705A1 (en) | Non-contact fatigue detection system and method based on rppg | |
| Okada et al. | Monitoring emotion by remote measurement of physiological signals using an RGB camera | |
| CN115840890B (en) | Emotion recognition method and device based on non-contact physiological signals | |
| Rundo et al. | Innovative saliency based deep driving scene understanding system for automatic safety assessment in next-generation cars | |
| Yao et al. | Hypertension assessment based on feature extraction using a photoplethysmography signal and its derivatives | |
| Jerritta et al. | Emotion recognition from electrocardiogram signals using Hilbert Huang Transform | |
| Ren et al. | Comparison of the use of pupil diameter and galvanic skin response signals for affective assessment of computer users | |
| Daqrouq et al. | Wavelet based method for congestive heart failure recognition by three confirmation functions | |
| Athaya et al. | An efficient fingertip photoplethysmographic signal artifact detection method: A machine learning approach | |
| KR102029760B1 (en) | System for detecting event using user emotion analysis and method thereof | |
| Cosentino et al. | Human-humanoid robot social interaction: Laughter | |
| CN116392148A (en) | Electroencephalogram signal classification method, device, equipment and storage medium | |
| Huerta et al. | Comparison of supervised learning algorithms for quality assessment of wearable electrocardiograms with paroxysmal atrial fibrillation | |
| KR102488535B1 (en) | System and Method of Stress Level Classification Using Electrogastrogram | |
| CN114743236A (en) | Intelligent trial study and judgment method | |
| Wu et al. | Integration of affective computing techniques and soft computing for developing a human affective recognition system for U-learning systems | |
| Cai et al. | A facial electromyography activity detection method in silent speech recognition | |
| Wrigley | A theory and computational model of auditory selective attention |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |