CN113229790A

CN113229790A - Non-contact mental stress assessment system

Info

Publication number: CN113229790A
Application number: CN202110535959.5A
Authority: CN
Inventors: 何赛灵; 朱宇东; 杨双
Original assignee: Zhejiang University ZJU
Current assignee: Zhejiang University ZJU
Priority date: 2021-05-17
Filing date: 2021-05-17
Publication date: 2021-08-10

Abstract

The invention discloses a non-contact mental stress assessment system, which comprises a video acquisition module, a data processing module and an analysis and evaluation module; the video acquisition module is used for acquiring the upper half body video of the person to be detected; the data processing module is used for detecting the characteristic points of the divided human face regions and identifying the micro expression of the person to be detected; carrying out color amplification on the divided human face area by using Euler color amplification, and extracting pulse wave signals to obtain heart rate and heart rate variability physiological parameters; amplifying by using Euler motion to obtain a displacement signal and a respiration rate of the chest wall; performing data dimension reduction on the extracted micro-expression, heart rate and heart rate variability physiological parameters, displacement signals of the chest wall and respiration rate by using a data dimension reduction technology; and the analysis and evaluation module analyzes and evaluates the data output by the data processing module by using the non-contact mental stress evaluation model or trains the non-contact mental stress evaluation model. The invention has the advantages of few sensors, simple operation and low cost.

Description

Non-contact mental stress assessment system

Technical Field

The invention relates to the technical fields of optical engineering, medicine, psychology, computer algorithm and the like, in particular to a non-contact mental stress assessment system.

Background

Under the background of rapid social development, people are increasingly stressed by mental stress from the aspects of work, study, life and the like, if the people are in high-pressure work, study and life states for a long time, the physiological health of human bodies can be further influenced, high-risk chronic diseases such as hypertension, heart disease and the like are caused, insomnia and immunologic function decline are caused, and serious consequences such as depression and even suicide can be caused if the people are in strong mental stress for a long time and are not relieved in time. According to the statistical data of the world health organization in 2017, the number of depression patients worldwide reaches 3.5 hundred million, and the depression patients become the second largest diseases in 2020 or human beings. There are surveys showing that counselors with depressed, anxious mood are present in psychological counseling practice in colleges and universities at rates up to 80%. This negative mental state, if not discovered and intervened in a timely manner, can have serious consequences such as retrospect, mental aberration and even suicide.

Although various means and devices such as electrocardiosignals, electroencephalograms, medical detection, psychological tests and the like are used for carrying out mental stress assessment on a person to be tested, some need to wear a plurality of sensing electrodes, some need to carry out blood drawing or sample extraction assay, some need to carry out a series of psychological tests and the like. The methods are complicated or have strong invasion to the testee, so that the willingness of the testee to actively cooperate with the detection is reduced to a certain extent, and the illness state is delayed.

The system device has simple process, intelligent data processing and non-contact type mental stress assessment, and needs to appear urgently to make up the defects of the prior art.

Disclosure of Invention

In order to overcome the defects of the prior art, the invention aims to provide a non-contact mental stress assessment system.

A system for contactless assessment of mental stress comprising: the system comprises a video acquisition module, a data processing module and an analysis and evaluation module; the video acquisition module is used for acquiring the upper half body video of the person to be tested with mental stress; the data processing module divides the acquired video into a face area and a chest wall area by a cascade classifier in OpenCV; carrying out feature point detection on the divided human face region by using the Dlib, and identifying the micro expression of the person to be detected; carrying out color amplification on the divided human face area by using Euler color amplification, then extracting a pulse wave signal by using an IPPG technology, and finally obtaining a heart rate and heart rate variability physiological parameters from the pulse wave signal; carrying out motion amplification on the divided chest wall area by utilizing Euler motion amplification, and then obtaining a displacement signal and a respiration rate of the chest wall through pixel change; performing data dimension reduction on the extracted micro-expression, heart rate and heart rate variability physiological parameters, displacement signals of the chest wall and respiration rate by using a data dimension reduction technology; the analysis and evaluation module analyzes and evaluates the data output by the data processing module by using the non-contact mental stress evaluation model, or trains the non-contact mental stress evaluation model.

The non-contact mental stress assessment model is trained by the system, wherein the mental stress assessment of the testee of mental stress adopts medical and psychological methods, and comprises one or more of the following methods: salivary cortisol detection, alpha-salivary amylase detection, norepinephrine detection, electroencephalogram analysis, brain function imaging, subject complaints, mental stress assessment questionnaires and subject behavior observation.

In the data processing module, the acquired video is subjected to region division through a cascade classifier in the OpenCV, and the video local region of interest (ROI) is subjected to targeted processing from a Lagrangian view.

In the data processing module, the divided human face regions are subjected to feature point detection by using the Dlib, and the feature points are subjected to expression judgment and classification by using a trained machine learning model.

In the data processing module, the Euler color amplification is to establish a Gaussian pyramid through spatial filtering, then filter signals of each layer, amplify signals of a pulse wave frequency band of 0.6-3 Hz, and then reconstruct to obtain an amplified video; and an ideal band-pass filter is selected for filtering.

In the data processing module, the IPPG technology utilizes spectroscopy principle, blind source signal separation and color space variation principle to extract weak pulse signals from facial videos, the heart rate and the physiological parameters of heart rate variability are extracted from pulse wave signals, and the heart rate variability includes but is not limited to RR intervals, standard deviation linear features of RR intervals, approximate entropy and sample entropy nonlinear features.

The Euler motion amplification is to establish a Laplacian pyramid through spatial filtering, then filter signals of all layers, amplify signals of a breathing frequency band of 0.1-0.6 Hz, and then reconstruct to obtain an amplified video; the filtering uses a butterworth filter or a wireless impulse response filter.

The data dimension reduction technique includes, but is not limited to, principal component analysis.

The invention has the beneficial effects that:

the invention designs a non-contact mental stress evaluation system by using principles and technologies such as optical engineering technology, medical research, psychological research, data analysis algorithm and the like, and has the main characteristics that:

1) and the number of sensors is small due to non-contact detection. The camera is used for shooting the upper half body video of a human body, and the non-contact physiological signal detection technology and algorithm based on the camera can be used for detecting the physiological parameters and the human face micro expression of the human body under the condition that no sensing equipment is worn.

2) Inducing and testing human body stress. The human stress inducing experiment can enable a person to be tested to generate mental stress with different degrees, and corresponding mental stress is scientifically calibrated by a medical detection method (such as salivary cortisol detection, alpha-salivary amylase detection, norepinephrine detection, electroencephalogram analysis, brain function imaging and the like) and a psychological method (such as chief complaints of the person to be tested, a mental stress assessment questionnaire, behavior observation of the person to be tested and the like) so as to be used as a reference standard and a training sample for mental stress assessment. The step is to collect labels of different samples, to train a stress detection model, and to be removed in practical use.

3) Euler amplification algorithm. The Euler amplification algorithm can be used for amplifying the weak color change of the human face, which is originally generated due to the blood volume change, in color, and also can be used for amplifying the weak displacement of the chest wall of the human body, which is caused by breathing, so that the detection difficulty of the weak physiological signals of the human body is reduced, and the detection accuracy is improved. The Euler magnifying visual angle is used for integrally magnifying the video, and requirements for ROI selection are not accurate, so that the method is suitable for complex application scenes with high background noise or incapable of accurately dividing the ROI, and has an obvious effect of improving the accuracy of pulse wave detection and respiration detection.

4) And (5) analyzing the data characteristics. Under different mental stresses, non-contact pulse wave signals collected and extracted by a camera are utilized, dominant characteristics of human physiological signals are obtained through signal characteristic identification, respiration measurement, micro-expression identification, heart rate calculation, heart rate variability and the like, and nonlinear characteristics such as approximate entropy, sample entropy and the like of the human physiological signals are further calculated through nonlinear characteristic analysis.

5) And (5) training a machine learning model. The method comprises the steps of utilizing a dimensionality reduction algorithm (such as principal component analysis and the like) to carry out data dimensionality reduction on non-contact pulse wave signals, human face micro-expressions, respiratory signals and linear and nonlinear characteristics thereof which are subjected to medical and psychological calibration to obtain data characteristics after dimensionality reduction, then utilizing a supervised learning algorithm (such as a support vector machine, a neural network and the like) to train a non-contact mental stress assessment model, utilizing the model to carry out mental stress assessment on a person to be tested and reporting results.

6) The method has the advantages of simple process and easy experience, brings better experience to a person to be tested, and has high use value and social benefit.

Drawings

FIG. 1 is a schematic diagram of a system according to the present invention.

FIG. 2 is a flow chart of the operation of the present invention.

Detailed Description

The invention is further illustrated below with reference to the figures and examples.

As shown in fig. 1, a non-contact mental stress assessment system comprises a video acquisition module, a data processing module and an analysis and evaluation module which are connected in sequence. The data processing module comprises an ROI (region of interest) dividing unit, a micro-expression identification unit, a respiration detection unit, a pulse wave detection unit and a data dimension reduction unit. Wherein the ROI dividing unit is respectively connected with the micro-expression recognition unit, the respiration detection unit and the pulse wave detection unit and then connected with the data dimension reduction unit. The analysis and evaluation module comprises a training non-contact mental stress assessment model and an analysis unit.

Examples

As shown in fig. 2, a camera-based non-contact mental stress assessment system which obtains a non-contact mental stress assessment model through a model training stage Sa and achieves assessment and reporting of mental stress without any sample extraction, psychological testing and wearing of any sensor having an intrusion or contact feeling on a subject in an actual use stage Sb.

As shown in fig. 1, the present invention is realized by the following scheme: a non-contact mental stress assessment system mainly comprises: the device comprises a video acquisition module, a data processing module and an analysis and evaluation module. In a model training stage Sa, a video acquisition module, a data processing module and a training non-contact type mental stress assessment model unit in an analysis and evaluation module are utilized to perform stress induction experiments on a large number of training testees and train a non-contact type mental stress assessment model: the method comprises a step Sa1, wherein the Montreal imaging stress task is used to enable the testee to generate different levels of mental stress, and the levels of the mental stress at corresponding stages of the testee are accurately calibrated and recorded by using a salivary amylase test and survey scale.

And step Sa2 is performed while step Sa1 is performed, and the video acquisition module is used for recording upper body videos of the testee under various intensities of mental stress in the experimental process and uploading the upper body videos to the data processing module.

After the data processing module acquires the upper body videos of each stage of the examinee, the step Sa3 is performed, and the captured videos are subjected to region division by using functions in the ROI division unit, for example, a Haar cascade classifier in OpenCV, so as to be divided into face videos and chest videos.

After the areas are divided, Sa4, Sa5 and Sa6 and Sa4 are simultaneously carried out, wherein a micro expression recognition unit is used for carrying out feature point recognition on a facial video by using Dlib, for example, so as to extract micro expressions of the tested person under different mental stress degrees; the Sa5 is to perform euler color amplification on a face video, and the specific operation flow is to convert a shot RGB video into a YIQ video, then perform spatial decomposition on the YIQ video into a gaussian pyramid, then perform time-domain filtering on each layer by adopting an ideal band-pass filter, the band-pass is 0.6-3 Hz, amplify the obtained signal, re-synthesize the video by using inverse gaussian decomposition, and finally convert the YIQ into the RGB space, so that a face video with amplified color signals can be obtained; the Sa6 is to perform euler motion amplification on the chest video, and the specific operation flow is to convert the shot RGB video into the YIQ video, then perform spatial decomposition on the YIQ video into a laplacian pyramid, then perform time domain filtering on each layer by using a butterworth filter with a band pass of 0.1-0.6 Hz, then amplify the obtained signal, re-synthesize the video, and finally convert the YIQ back to the RGB space, so as to obtain the chest video with amplified motion signals.

After obtaining the human face video with amplified color signals, Sa7 is carried out, facial pulse waves of the person to be detected under various mental stress intensities are extracted from the facial video of the person to be detected by utilizing a non-contact physiological signal detection technology based on a camera, linear characteristics such as heart rate, heart rate variability and the ratio of the frequency domain intensity of the first harmonic wave and the second harmonic wave of the pulse waves are calculated by utilizing the facial pulse waves of the person to be detected after the facial pulse waves of the person to be detected are obtained, further nonlinear characteristics such as approximate entropy of the heart rate and the heart rate variability and sample entropy are calculated, and data characteristics of multiple dimensions are obtained; the Sa5 and Sa7 steps are mainly performed in the pulse wave detection unit.

After obtaining the chest video with amplified motion signals, performing step Sa8, obtaining a respiration curve represented by the chest by using the pixel change of the video with amplified motion signals, and extracting the respiration rate; the Sa6 and Sa8 steps are mainly performed in the respiration detection unit.

After obtaining the facial micro-expression, the heart rate variability and the respiration, the data dimension reduction unit is used for performing Sa9, and the data dimension reduction algorithm is used for performing data dimension reduction on the multidimensional data extracted in the steps Sa4, Sa7 and Sa8 to obtain the data after dimension reduction. After the data after dimension reduction is obtained, the data enters a non-contact type mental stress assessment model training unit in the data processing module and the analysis and evaluation module, and the non-contact type mental stress assessment model is trained by using the obtained data after dimension reduction and the mental stress calibration data obtained in the step Sa1 according to a support vector machine algorithm.

Mental stress assessment of subjects in the actual use phase Sb: step Sb1 is firstly carried out, and a video acquisition module is utilized to acquire the upper half body video of the person to be detected; performing step Sb2 after obtaining the video of the upper body half of the subject, dividing the region using the same algorithm as in step Sa3, and obtaining the facial microexpression, heart rate variability, and respiration of the subject using the same algorithm as in (Sa 4-Sa 8); then, Sb3 is performed, data dimension reduction is performed on the obtained high-dimensional data to obtain dimension-reduced data, and using the non-contact type mental stress assessment model obtained in the training stage Sa, the analysis unit is used to perform mental stress recognition on the dimension-reduced data and give a result report.

The embodiments in the above description can be further combined or replaced, and the embodiments are only described as preferred examples of the present invention, and do not limit the concept and scope of the present invention, and various changes and modifications made to the technical solution of the present invention by those skilled in the art without departing from the design concept of the present invention belong to the protection scope of the present invention. The scope of the invention is given by the appended claims and any equivalents thereof.

Claims

1. A system for contactless assessment of mental stress, comprising: the system comprises a video acquisition module, a data processing module and an analysis and evaluation module; the video acquisition module is used for acquiring the upper half body video of the person to be tested with mental stress; the data processing module divides the acquired video into a face area and a chest wall area by a cascade classifier in OpenCV; carrying out feature point detection on the divided human face region by using the Dlib, and identifying the micro expression of the person to be detected; carrying out color amplification on the divided human face area by using Euler color amplification, then extracting a pulse wave signal by using an IPPG technology, and finally obtaining a heart rate and heart rate variability physiological parameters from the pulse wave signal; carrying out motion amplification on the divided chest wall area by utilizing Euler motion amplification, and then obtaining a displacement signal and a respiration rate of the chest wall through pixel change; performing data dimension reduction on the extracted micro-expression, heart rate and heart rate variability physiological parameters, displacement signals of the chest wall and respiration rate by using a data dimension reduction technology; the analysis and evaluation module analyzes and evaluates the data output by the data processing module by using the non-contact mental stress evaluation model, or trains the non-contact mental stress evaluation model.

2. The system of claim 1, wherein said non-contact mental stress assessment model is trained by said system, wherein said mental stress assessment of said subject under mental stress employs medical and psychological methods comprising one or more of: salivary cortisol detection, alpha-salivary amylase detection, norepinephrine detection, electroencephalogram analysis, brain function imaging, subject complaints, mental stress assessment questionnaires and subject behavior observation.

3. The system according to claim 1, wherein the data processing module performs region partition on the acquired video through a cascaded classifier in OpenCV, and performs targeted processing on a local region of interest (ROI) of the video from a lagrangian perspective.

4. The system of claim 1, wherein the data processing module performs feature point detection on the divided human face regions by using Dlib, and performs expression judgment classification on the feature points by using a trained machine learning model.

5. The system of claim 1, wherein in the data processing module, the euler color amplification is to build a gaussian pyramid by spatial filtering, then filter signals of each layer, amplify signals of a pulse wave frequency band of 0.6-3 Hz, and then reconstruct to obtain an amplified video; and an ideal band-pass filter is selected for filtering.

6. The system of claim 1, wherein the IPPG technique utilizes spectroscopy, blind source signal separation and color space variation principles to extract weak pulse signals from facial video, and the heart rate and heart rate variability physiological parameters are extracted from pulse wave signals, and the heart rate variability includes but is not limited to RR intervals, standard deviation linear features of RR intervals, and approximate entropy and sample entropy non-linear features.

7. The system of claim 1, wherein the euler motion amplification is to build a laplacian pyramid by spatial filtering, then filter signals of each layer, amplify signals in a respiratory frequency band of 0.1-0.6 Hz, and then reconstruct the amplified video; the filtering uses a butterworth filter or a wireless impulse response filter.

8. The system of claim 1, wherein the data dimension reduction technique includes, but is not limited to, principal component analysis.