CN111714144A - Mental stress analysis method based on video non-contact measurement - Google Patents

Mental stress analysis method based on video non-contact measurement Download PDF

Info

Publication number
CN111714144A
CN111714144A CN202010722059.7A CN202010722059A CN111714144A CN 111714144 A CN111714144 A CN 111714144A CN 202010722059 A CN202010722059 A CN 202010722059A CN 111714144 A CN111714144 A CN 111714144A
Authority
CN
China
Prior art keywords
video
signal
calculating
face
module
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202010722059.7A
Other languages
Chinese (zh)
Inventor
李炳霖
嵇晓强
牛裕茸
刘振瑶
邵望舒
王美娇
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Changchun University of Science and Technology
Original Assignee
Changchun University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Changchun University of Science and Technology filed Critical Changchun University of Science and Technology
Priority to CN202010722059.7A priority Critical patent/CN111714144A/en
Publication of CN111714144A publication Critical patent/CN111714144A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/16—Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/165—Evaluating the state of mind, e.g. depression, anxiety
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/024—Detecting, measuring or recording pulse rate or heart rate
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
    • A61B5/024—Detecting, measuring or recording pulse rate or heart rate
    • A61B5/02405—Determining heart rate variability
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7253—Details of waveform analysis characterised by using transforms
    • A61B5/7257—Details of waveform analysis characterised by using transforms using Fourier transforms
    • A—HUMAN NECESSITIES
    • A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00—Measuring for diagnostic purposes; Identification of persons
    • A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235—Details of waveform analysis
    • A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
    • G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
    • G06V40/16—Human faces, e.g. facial parts, sketches or expressions
    • G06V40/168—Feature extraction; Face representation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/02—Preprocessing
    • G06F2218/04—Denoising

Abstract

A mental stress analysis method based on video non-contact measurement belongs to the technical field of mental stress evaluation and aims to solve the problems of poor comfort and poor measurement accuracy in the prior art. The mental stress analysis method based on video non-contact measurement specifically comprises the following steps: firstly, a face video is collected through a video image collection module; secondly, acquiring a video frame picture according to the face video through an image preprocessing module, identifying the face, removing the environmental background interference, and extracting a face region of interest (ROI) by adopting a skin extraction and multiple ROI method; thirdly, channel conversion is carried out through an IPPG signal extraction and processing module to obtain an original signal of the pulse rate signal, and then the IPPG signal is obtained through denoising and smoothing preprocessing; then, an RR interval of the IPPG signal is extracted through a characteristic value extraction module, and HRV analysis is carried out to obtain a heart rate and time domain frequency domain characteristic index; finally, a rapid assessment of mental stress is accomplished based on machine learning by a mental stress assessment module.

Description

Mental stress analysis method based on video non-contact measurement
Technical Field
The invention belongs to the technical field of mental stress assessment, and particularly relates to a mental stress analysis method based on video non-contact measurement.
Background
Psychological stress can affect the working efficiency and the life quality of people, and is also a main reason for the formation of diseases such as hypertension, myocardial infarction, anxiety, depression disorder and the like. It is therefore particularly necessary to be able to give objective psychological interventions. In the prior art, psychological stress measurement is generally in two forms, one is measurement based on questionnaires or scales in a self-reporting form, and the reporting questionnaires are too subjective and difficult to give more scientific and reasonable evaluation suggestions to doctors. The other is the measurement of physiological angles, the most important of which is the measurement of cardiovascular response. Currently, the most widely used methods for measuring cardiovascular responses are Electrocardiography (ECG) and photoplethysmography (PPG), both of which require the use of associated instruments under the guidance of a professional, the instruments are not portable, and contact measurement may cause irritation and discomfort to the skin of a patient. The professional operation of the contact instrument with the medical staff is likely to cause mental panic in the subject, thereby affecting the results of the mental stress measurement.
The Chinese patent with the publication number of CN106419937A discloses a mental stress analysis system based on the HRV theory of heart sounds, which collects heart sound signals through a heart sound signal collection module, is connected with a PC (personal computer) end through a wireless communication module to realize data transmission, obtains time domain and frequency domain indexes of the HRV through a self-adaptive algorithm to perform stress analysis, but experimental equipment is contact measurement collection, is inconvenient for remote home measurement and needs to be completed under the operation of professionals, and the equipment collection can cause mental panic emotion of patients, thereby affecting the accuracy of measurement results.
Chinese patent publication No. CN104755020A discloses a mental stress measuring system, which provides three methods for measuring mental stress through alarm clock stimulation for heart rate detection, combines an alarm clock with a PPG sensor, an ECG sensor, and a remote PPG vital sign camera, and reflects the condition of mental stress through heart rate. However, the heart rate is only used as an index for measuring the psychological stress state, and the individual heart rate is different in height and the judgment standard is not easy to be unified.
The Chinese patent with the publication number of CN107913075A discloses a mental stress assessment device based on multiple parameters and an assessment method thereof, the method adopts a nine-scale method and a three-scale method to judge a matrix to obtain the weight of a mental stress assessment system, and a mental stress formula is constructed according to the weight. However, the judgment matrix needs to be manually specified, which requires a large number of previous data references, and a certain deviation exists, which may cause inaccuracy of the final measurement result.
Disclosure of Invention
The invention aims to provide a mental stress analysis method based on video non-contact measurement, and solves the problems of poor comfort and poor measurement accuracy in the prior art.
In order to achieve the above purpose, the mental stress analysis method based on video non-contact measurement of the invention specifically comprises the following steps: firstly, a face video is collected through a video image collection module; secondly, acquiring a video frame picture according to the face video through an image preprocessing module, identifying the face, removing the environmental background interference, and extracting a face region of interest (ROI) by adopting a skin extraction and multiple ROI method; thirdly, channel conversion is carried out through an IPPG signal extraction and processing module to obtain an original signal of the pulse rate signal, and then the IPPG signal is obtained through denoising and smoothing preprocessing; then, an RR interval of the IPPG signal is extracted through a characteristic value extraction module, and HRV analysis is carried out to obtain a heart rate and time domain frequency domain characteristic index; finally, a rapid assessment of mental stress is accomplished based on machine learning by a mental stress assessment module.
The specific steps of acquiring the face video through the video image acquisition module are as follows: set for video image acquisition module's frame frequency and gather long, shoot the face, during the shooting, the human body sits quietly in quiet indoor, guarantees that light is sufficient, eyes are directly looked video image acquisition module, and video image acquisition module is to the host computer with the face video of gathering.
The image preprocessing module comprises the following specific processing steps:
1) acquiring a plurality of frame pictures of a video acquired by a video image acquisition module;
2) carrying out human face detection based on Haar + adaboost on each frame picture obtained in the step 1);
3) extracting a detected face region;
4) extracting the ROI of the face from the face region obtained in the step 3) by adopting two means to obtain the ROIs under two conditions;
the ROIs for both cases are specifically:
one of the situations is: converting the RGB picture of the face region obtained in the step 3) into a YCrCb space, setting the threshold values of Cr and Cb to convert the picture into a binary picture, removing the influence of five sense organs and hair, and extracting the face skin to obtain an ROI (region of interest) of one condition;
in another case: selecting a plurality of smooth skin areas from the face area obtained in the step 3), and taking the plurality of skin areas as the ROI in the second condition.
The specific processing steps in the IPPG signal extracting and processing module are as follows:
1) converting the face picture of the face area obtained in the step 3) in the image preprocessing module from an RGB space to a CIEXYZ color space, and then calculating from a CIE XYZ space to obtain a LUV color space;
2) extracting a U channel image;
3) combining the two ROI in the step 4) in the image preprocessing module, and taking a change curve graph of a pixel point as an original signal of the pulse rate signal;
4) and preprocessing an original signal of the pulse rate signal to obtain an IPPG signal.
The step 3) of taking the variation curve of the pixel point as the original signal of the pulse rate signal specifically comprises the following steps:
1) combining the ROI of one condition obtained in the step 4) in the image preprocessing module, performing AND operation on the skin binary image and the U-channel image obtained in the step 2 to obtain a U-channel image only containing skin information, and calculating a pixel point mean value of each image; taking a change curve graph of pixel point average values as a pulse rate signal of one condition;
selecting a plurality of smooth skin areas from the U-channel picture obtained in the step 2) by combining the ROI of the other condition obtained in the step 4) in the image preprocessing module, and calculating the pixel point mean value of the plurality of areas of each image; taking a change curve graph of the pixel point average value as a pulse rate signal in another situation;
2) the original waveforms of the pulse rate signals of the two cases are compared, and a group with low noise is selected as the original signal of the pulse rate signal.
The preprocessing of the original signal of the pulse rate signal specifically includes:
1) preprocessing an original signal of the pulse rate signal by adopting wavelet transformation to obtain a reconstructed pulse wave signal;
2) smoothing the pulse wave signal by adopting 5-point moving average filtering;
3) and (5) carrying out interpolation by adopting a cubic spline function to obtain an IPPG signal.
The characteristic value extraction module comprises the following specific steps:
1) positioning a peak point of the IPPG signal obtained by the IPPG signal extraction and processing module, and extracting the peak point;
2) calculating the time interval between adjacent peak points of the waveform after the peak points are positioned, and calculating an RR interval;
3) and calculating and obtaining the human heart rate, the time domain characteristic value and the frequency domain characteristic value based on RR interval analysis.
The specific steps of analyzing and calculating in the step 3) to obtain the human heart rate are as follows: respectively calculating heart rates of patients with regular heart rates and patients with irregular heart rates, judging whether the heart rates are regular or not by calculating whether the difference value between RR intervals is larger than 0.12s or not, judging whether the heart rates are regular or not if the difference value is not larger than 0.12s, calculating by adopting a formula (1), judging whether the heart rates are irregular or not if the difference value is not larger than 0.12s, and calculating by adopting a formula (2);
HR ═ 60 ÷ RR interval (1)
HR-10 sRR interval × 6 (2).
The time domain characteristic value and the frequency domain characteristic value obtained by analyzing and calculating in the step 3) are specifically as follows:
the HRV time domain analysis adopts the obtained RR interphase, and time domain characteristic values are obtained through calculation, wherein the time domain characteristic values comprise the standard deviation of the average RR interphase in the measuring period, the root mean square of the difference value of the adjacent RR interphase and the proportion of the number of the adjacent RR interphase larger than 50ms in the total number of the RR interphase in the measuring period;
calculating the average value of all RR intervals in the measuring period by adopting a formula (3); calculating the SDNN (standard deviation of average RR intervals during measurement) by adopting a formula (4); calculating the root mean square of RMSSD, namely the difference value between adjacent RR intervals by adopting a formula (5); calculating the proportion of PNN50, namely the number of adjacent RR intervals larger than 50ms, in the total number of RR intervals in the measurement period by adopting a formula (6); where N is the total number of heart beats during the measurement, RRiIs the ith RR interval, RRi+1Is the i +1 RR interval, meanRR is the average of the RR intervals of the N heart beats;
Figure BDA0002600386370000041
Figure BDA0002600386370000042
Figure BDA0002600386370000043
Figure BDA0002600386370000044
and the HRV frequency domain analysis is based on the RR interval, energy values are respectively calculated for signals of the LF and the HF sections based on fast Fourier transform, and then the LF/HF ratio is obtained and used as a frequency domain characteristic value.
The mental stress assessment module comprises the following specific steps:
1) dividing the data into an integral training set and an integral testing set; the data comprises heart rate, time domain characteristic value and frequency domain characteristic value in the characteristic value extraction module and actual mental pressure value;
2) dividing the whole training set in the step 1) into three groups of data, taking two groups of the three groups of data as a training set and taking the other group of data as a confirmation set in turn, training a model, calculating the error of the model on the confirmation set to obtain three error values, and averaging the three error values to obtain the average error of the model;
3) repeating the step 2), calculating the average errors of different models, finally comparing the average errors, and selecting the model with smaller average error as the final regression model;
4) adopting an integral test set test regression model, outputting a pressure estimation value, and calculating the difference value between the actual mental pressure value and the output pressure estimation value to obtain a measurement error;
5) and judging whether the measurement error is in a calibrated value range, if so, outputting the pressure estimation value as a measurement value, and if not, repeating the step 2) and the step 4) until the measurement error is in the calibrated value range. The video image acquisition module is a camera.
The invention has the beneficial effects that: the mental pressure analysis method based on video non-contact measurement comprises the steps of shooting a face video based on a mobile phone camera, preprocessing a video picture, selecting ROI and converting channels, preprocessing the signal to obtain an IPPG signal, extracting a signal RR interval, calculating a heart rate, analyzing HRV to extract time domain and frequency domain characteristic values, and evaluating the mental pressure of a human body based on machine learning. The device is simple, the operation is simple and convenient, the cost is low, the heart rate and heart rate variability indexes of a patient can be remotely measured at home, the mental pressure value is obtained, the device is applied to the fields of psychology and mental health medicine, and the device can be used as a professional psychology and mental health research analysis tool to provide objective professional analysis and auxiliary evaluation for researchers.
The invention automatically collects the face video to evaluate the real-time pressure, avoids complex equipment operation, improves the convenience and comfort of measurement and reduces psychological panic caused by environment and instruments.
Drawings
FIG. 1 is a general flow chart of a mental stress analysis method based on video non-contact measurement according to the present invention;
FIG. 2 is a flow diagram of an image pre-processing module according to the present invention;
FIG. 3 is a flow chart of an IPPG signal extraction and processing module of the present invention;
FIG. 4 is a flow diagram of a eigenvalue extraction module of the present invention;
FIG. 5 is a flow chart of the stress assessment module according to the present invention.
Detailed Description
The following further describes embodiments of the present invention with reference to the drawings.
Referring to fig. 1-5, the method for analyzing mental stress based on video non-contact measurement of the present invention specifically comprises: firstly, a video image acquisition module is used for acquiring a face video under a natural light condition; secondly, acquiring a video frame picture according to the face video through an image preprocessing module, identifying the face, removing environmental background interference, and extracting a face region of interest (ROI) by adopting a skin extraction and a method of averaging a plurality of ROI; thirdly, channel conversion is carried out through an IPPG signal extraction and processing module to obtain an original signal of the pulse rate signal, and then the IPPG signal is obtained through denoising and smoothing preprocessing; then, an RR interval of the IPPG signal is extracted through a characteristic value extraction module, and HRV analysis is carried out to obtain a heart rate and time domain frequency domain characteristic index; finally, a rapid assessment of mental stress is accomplished based on machine learning by a mental stress assessment module.
The method comprises the following specific steps of collecting the face video under the natural light condition through a video image collection module: set for video image acquisition module's frame frequency and gather long, shoot the face, during the shooting, the human body sits quietly in quiet indoor, guarantees that light is sufficient, eyes are directly looked video image acquisition module, and video image acquisition module is to the host computer with the face video of gathering.
The image preprocessing module comprises the following specific processing steps:
1) acquiring a plurality of frame pictures of a video acquired by a video image acquisition module;
2) carrying out human face detection based on Haar + adaboost on each frame picture obtained in the step 1);
3) extracting a detected face region;
4) extracting the ROI of the face from the face region obtained in the step 3) by adopting two means to obtain the ROIs under two conditions;
the ROIs for both cases are specifically:
one of the situations is: converting the RGB picture of the face region obtained in the step 3) into a YCrCb space, setting the threshold values of Cr and Cb to convert the picture into a binary picture, removing the influence of five sense organs and hair, and extracting the face skin to obtain an ROI (region of interest) of one condition;
in another case: selecting a plurality of smooth skin areas from the face area obtained in the step 3), and taking the plurality of skin areas as the ROI in the second condition.
The specific processing steps in the IPPG signal extracting and processing module are as follows:
1) converting the face picture of the face area obtained in the step 3) in the image preprocessing module from an RGB space to a CIEXYZ color space, and then calculating from a CIE XYZ space to obtain a LUV color space;
2) extracting a U channel image; the absorption coefficient of oxygenated and deoxygenated haemoglobin is between 540 and 577nm, corresponding to green/yellow wavelengths, when extracting the U channel, U represents a red to green indication, the IPPG information extracted will be more effective;
3) combining the two ROI in the step 4) in the image preprocessing module, and taking a change curve graph of a pixel point as an original signal of the pulse rate signal;
4) and preprocessing an original signal of the pulse rate signal to obtain an IPPG signal.
The step 3) of taking the variation curve of the pixel point as the original signal of the pulse rate signal specifically comprises the following steps:
1) combining the ROI of one condition obtained in the step 4) in the image preprocessing module, performing AND operation on the skin binary image and the U-channel image obtained in the step 2 to obtain a U-channel image only containing skin information, and calculating a pixel point mean value of each image; taking a change curve graph of pixel point average values as a pulse rate signal of one condition;
selecting a plurality of smooth skin areas from the U-channel picture obtained in the step 2) by combining the ROI of the other condition obtained in the step 4) in the image preprocessing module, and calculating the pixel point mean value of the plurality of areas of each image; taking a change curve graph of the pixel point average value as a pulse rate signal in another situation;
2) the original waveforms of the pulse rate signals of the two cases are compared, and a group with low noise is selected as the original signal of the pulse rate signal.
The preprocessing of the original signal of the pulse rate signal specifically includes:
1) preprocessing an original signal of the pulse rate signal by adopting wavelet transformation, and selecting a proper decomposition layer number by combining the frame frequency of a camera and the frequency (0.5-2Hz) of the pulse wave to obtain a reconstructed pulse wave signal;
2) smoothing the pulse wave signal by adopting 5-point moving average filtering;
3) and (4) interpolating by adopting a cubic spline function to obtain a more accurate peak point detection result and obtain an IPPG signal.
The characteristic value extraction module comprises the following specific steps:
1) positioning a peak point of the IPPG signal obtained by the IPPG signal extraction and processing module, and extracting the peak point;
2) calculating the time interval between adjacent peak points of the waveform after the peak points are positioned, and calculating an RR interval;
3) and calculating and obtaining the human heart rate, the time domain characteristic value and the frequency domain characteristic value based on RR interval analysis.
The specific steps of analyzing and calculating in the step 3) to obtain the human heart rate are as follows: respectively calculating heart rates of patients with regular heart rates and patients with irregular heart rates, judging whether the heart rates are regular or not by calculating whether the difference value between RR intervals is larger than 0.12s or not, judging whether the heart rates are regular or not if the difference value is not larger than 0.12s, calculating by adopting a formula (1), judging whether the heart rates are irregular or not if the difference value is not larger than 0.12s, and calculating by adopting a formula (2);
HR ═ 60 ÷ RR interval (1)
HR-10 sRR interval × 6 (2).
The time domain characteristic value and the frequency domain characteristic value obtained by analyzing and calculating in the step 3) are specifically as follows:
the HRV time domain analysis adopts the obtained RR interphase, and time domain characteristic values are obtained through calculation, wherein the time domain characteristic values comprise the standard deviation of the average RR interphase in the measuring period, the root mean square of the difference value of the adjacent RR interphase and the proportion of the number of the adjacent RR interphase larger than 50ms in the total number of the RR interphase in the measuring period;
calculating the average value of all RR intervals in the measuring period by adopting a formula (3); calculating the SDNN (standard deviation of average RR intervals during measurement) by adopting a formula (4); calculating the root mean square of RMSSD, namely the difference value between adjacent RR intervals by adopting a formula (5); calculating the proportion of PNN50, namely the number of adjacent RR intervals larger than 50ms, in the total number of RR intervals in the measurement period by adopting a formula (6); where N is the total number of heart beats during the measurement, RRiIs the ith RR interval, RRi+1Is the i +1 RR interval, meanRR is the average of the RR intervals of the N heart beats;
Figure BDA0002600386370000081
Figure BDA0002600386370000082
Figure BDA0002600386370000083
Figure BDA0002600386370000084
the HRV frequency domain analysis is based on RR interphase, energy values are respectively calculated for signals of an LF section and an HF section based on fast Fourier transform, then an LF/HF ratio is obtained and used as a frequency domain characteristic value, and HRV frequency domain indexes are shown in table 1.
TABLE 1
Figure BDA0002600386370000091
The mental stress assessment module comprises the following specific steps:
1) dividing the data into an integral training set and an integral testing set; the data comprises heart rate, time domain characteristic value and frequency domain characteristic value in the characteristic value extraction module and actual mental pressure value, wherein the actual mental pressure value is a mental pressure evaluation state result given by the psychology department of the professional hospital;
2) dividing the whole training set in the step 1) into three groups of data, taking two of the three groups of data as a training set and taking the other two parts of the three groups of data as a confirmation set in turn, namely dividing the whole training set into three groups of data, namely a model training set, a model training set and a model confirmation set, and training a regression model by using the whole training set; a gradient descent method is adopted to minimize a loss function, an initial position is randomly given at first, then partial differentiation of each parameter is calculated, learning rate is set, and the learning rate is further set each time until the algorithm converges to the minimum; thereby obtaining model parameters; calculating the error of the model on the model confirmation set to obtain three error values, and averaging the three error values to obtain the average error of the model;
3) repeating the step 2), calculating the average errors of different models, finally comparing the average errors, and selecting the model with smaller average error as the final regression model;
4) adopting an integral test set test regression model, outputting a pressure estimation value, and calculating the difference value between the actual mental pressure value and the output pressure estimation value to obtain a measurement error;
5) and judging whether the measurement error is in a calibrated value range, if so, outputting the pressure estimation value as a measurement value, and if not, repeating the step 2) and the step 4) until the measurement error is in the calibrated value range. The calibration value of the embodiment is [0-20], each interval 20 is a pressure state judgment, and the range exceeding the calibration value 20 is considered that the pressure state judgment is unreasonable, and the mental pressure value and the reality are inaccurate.
The video image acquisition module is a camera. The camera is a mobile phone camera or a computer camera.

Claims (10)

1. The mental stress analysis method based on video non-contact measurement is characterized by comprising the following specific steps: firstly, a face video is collected through a video image collection module; secondly, acquiring a video frame picture according to the face video through an image preprocessing module, identifying the face, removing the environmental background interference, and extracting a face ROI by adopting a skin extraction and a plurality of face interesting regions; thirdly, channel conversion is carried out through an IPPG signal extraction and processing module to obtain an original signal of the pulse rate signal, and then the IPPG signal is obtained through denoising and smoothing preprocessing; then, an RR interval of the IPPG signal is extracted through a characteristic value extraction module, and HRV analysis is carried out to obtain a heart rate and time domain frequency domain characteristic index; finally, the mental stress assessment module completes the assessment of the mental stress based on machine learning.
2. The mental stress analysis method based on video non-contact measurement according to claim 1, wherein the specific steps of acquiring the face video through the video image acquisition module are as follows: set for video image acquisition module's frame frequency and gather long, shoot the face, during the shooting, the human body sits quietly in quiet indoor, guarantees that light is sufficient, eyes are directly looked video image acquisition module, and video image acquisition module is to the host computer with the face video of gathering.
3. The mental stress analysis method based on video non-contact measurement according to claim 1 or 2, characterized in that the specific processing steps in the image preprocessing module are as follows:
1) acquiring a plurality of frame pictures of a video acquired by a video image acquisition module;
2) carrying out human face detection based on Haar + adaboost on each frame picture obtained in the step 1);
3) extracting a detected face region;
4) extracting the ROI of the face from the face region obtained in the step 3) by adopting two means to obtain the ROIs under two conditions;
the ROIs for both cases are specifically:
one of the situations is: converting the RGB picture of the face region obtained in the step 3) into a YCrCb space, setting the threshold values of Cr and Cb to convert the picture into a binary picture, removing the influence of five sense organs and hair, and extracting the face skin to obtain an ROI (region of interest) of one condition;
in another case: selecting a plurality of smooth skin areas from the face area obtained in the step 3), and taking the plurality of skin areas as the ROI in the second condition.
4. The mental stress analysis method based on video non-contact measurement according to claim 3, wherein the specific processing steps in the IPPG signal extraction and processing module are as follows:
1) converting the face picture of the face region obtained in the step 3) in the image preprocessing module from an RGB space to a CIE XYZ color space, and then calculating from the CIE XYZ space to obtain a LUV color space;
2) extracting a U channel image;
3) combining the two ROI in the step 4) in the image preprocessing module, and taking a change curve graph of a pixel point as an original signal of the pulse rate signal;
4) and preprocessing an original signal of the pulse rate signal to obtain an IPPG signal.
5. The analysis method of mental stress based on video non-contact measurement according to claim 4, wherein the original signal using the variation graph of the pixel points as the pulse rate signal in step 3) is specifically:
1) combining the ROI of one condition obtained in the step 4) in the image preprocessing module, performing AND operation on the skin binary image and the U-channel image obtained in the step 2 to obtain a U-channel image only containing skin information, and calculating a pixel point mean value of each image; taking a change curve graph of pixel point average values as a pulse rate signal of one condition;
selecting a plurality of smooth skin areas from the U-channel picture obtained in the step 2) by combining the ROI of the other condition obtained in the step 4) in the image preprocessing module, and calculating the pixel point mean value of the plurality of areas of each image; taking a change curve graph of the pixel point average value as a pulse rate signal in another situation;
2) the original waveforms of the pulse rate signals of the two cases are compared, and a group with low noise is selected as the original signal of the pulse rate signal.
6. The analysis method of mental stress based on video non-contact measurement as claimed in claim 4, wherein said pre-processing of the raw signal of the pulse rate signal specifically comprises:
1) preprocessing an original signal of the pulse rate signal by adopting wavelet transformation to obtain a reconstructed pulse wave signal;
2) smoothing the pulse wave signal by adopting 5-point moving average filtering;
3) and (5) carrying out interpolation by adopting a cubic spline function to obtain an IPPG signal.
7. The mental stress analysis method based on video non-contact measurement according to claim 4, wherein the characteristic value extraction module comprises the following specific steps:
1) positioning a peak point of the IPPG signal obtained by the IPPG signal extraction and processing module, and positioning the peak point;
2) obtaining a waveform after the peak points are positioned, and calculating the time interval between adjacent peak points to obtain an RR interval;
3) and calculating and obtaining the human heart rate, the time domain characteristic value and the frequency domain characteristic value based on RR interval analysis.
8. The mental stress analysis method based on video non-contact measurement according to claim 7, wherein the human heart rate obtained by the analysis and calculation in step 3) is specifically: respectively calculating heart rates of patients with regular heart rates and patients with irregular heart rates, judging whether the heart rates are regular or not by calculating whether the difference value between RR intervals is larger than 0.12s or not, judging whether the heart rates are regular or not if the difference value is not larger than 0.12s, calculating by adopting a formula (1), judging whether the heart rates are irregular or not if the difference value is not larger than 0.12s, and calculating by adopting a formula (2);
HR ═ 60 ÷ RR interval (1)
HR-10 sRR interval × 6 (2).
9. The mental stress analysis method based on video non-contact measurement according to claim 7, wherein the time domain characteristic value and the frequency domain characteristic value obtained by the analysis and calculation in step 3) are specifically:
the HRV time domain analysis adopts the obtained RR interphase, and time domain characteristic values are obtained through calculation, wherein the time domain characteristic values comprise the standard deviation of the average RR interphase in the measuring period, the root mean square of the difference value of the adjacent RR interphase and the proportion of the number of the adjacent RR interphase larger than 50ms in the total number of the RR interphase in the measuring period;
calculating the average value of all RR intervals in the measuring period by adopting a formula (3); calculating the SDNN (standard deviation of average RR intervals during measurement) by adopting a formula (4); calculating the root mean square of RMSSD, namely the difference value between adjacent RR intervals by adopting a formula (5); calculating the proportion of PNN50, namely the number of adjacent RR intervals larger than 50ms, in the total number of RR intervals in the measurement period by adopting a formula (6); where N is the total number of heart beats during the measurement, RRiIs the ith RR interval, RRi+1Is the i +1 RR interval, meanRR is the average of the RR intervals of the N heart beats;
Figure FDA0002600386360000031
Figure FDA0002600386360000032
Figure FDA0002600386360000041
Figure FDA0002600386360000042
and the HRV frequency domain analysis is based on the RR interval, energy values are respectively calculated for signals of the LF and the HF sections based on fast Fourier transform, and then the LF/HF ratio is obtained and used as a frequency domain characteristic value.
10. The analysis method of mental stress based on video non-contact measurement as claimed in claim 7, wherein the mental stress assessment module comprises the following specific steps:
1) dividing the data into an integral training set and an integral testing set; the data comprises heart rate, time domain characteristic value and frequency domain characteristic value in the characteristic value extraction module and actual mental pressure value;
2) dividing the whole training set in the step 1) into three groups of data, taking two groups of the three groups of data as a training set and taking the other group of data as a confirmation set in turn, training a model, calculating the error of the model on the confirmation set to obtain three error values, and averaging the three error values to obtain the average error of the model;
3) repeating the step 2), calculating the average errors of different models, finally comparing the average errors, and selecting the model with smaller average error as the final regression model;
4) adopting an integral test set test regression model, outputting a pressure estimation value, and calculating the difference value between the actual mental pressure value and the output pressure estimation value to obtain a measurement error;
5) and judging whether the measurement error is in a calibrated value range, if so, outputting the pressure estimation value as a measurement value, and if not, repeating the step 2) and the step 4) until the measurement error is in the calibrated value range.
CN202010722059.7A 2020-07-24 2020-07-24 Mental stress analysis method based on video non-contact measurement Pending CN111714144A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202010722059.7A CN111714144A (en) 2020-07-24 2020-07-24 Mental stress analysis method based on video non-contact measurement

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010722059.7A CN111714144A (en) 2020-07-24 2020-07-24 Mental stress analysis method based on video non-contact measurement

Publications (1)

Publication Number Publication Date
CN111714144A true CN111714144A (en) 2020-09-29

Family

ID=72573466

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010722059.7A Pending CN111714144A (en) 2020-07-24 2020-07-24 Mental stress analysis method based on video non-contact measurement

Country Status (1)

Country Link
CN (1) CN111714144A (en)

Cited By (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112370057A (en) * 2020-11-09 2021-02-19 平安科技（深圳）有限公司 Pressure evaluation method and device, computer equipment and storage medium
CN112597949A (en) * 2020-12-29 2021-04-02 清华大学 Psychological stress measuring method and system based on video
CN113229790A (en) * 2021-05-17 2021-08-10 浙江大学 Non-contact mental stress assessment system
CN113397519A (en) * 2021-08-05 2021-09-17 季华实验室 Cardiovascular health state detection device
CN113657345A (en) * 2021-08-31 2021-11-16 天津理工大学 Non-contact heart rate variability feature extraction method based on reality application scene
CN114271800A (en) * 2021-12-01 2022-04-05 西北工业大学 Non-invasive continuous blood pressure monitoring method in office environment and application
CN116889388A (en) * 2023-09-11 2023-10-17 长春理工大学 Intelligent detection system and method based on rPPG technology
CN117281516A (en) * 2023-11-24 2023-12-26 吾征智能技术(北京)有限公司 Mental stress monitoring system, method, medium and device
WO2023246182A1 (en) * 2022-06-20 2023-12-28 华中师范大学 Non-contact fatigue detection system and method based on rppg

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105193431A (en) * 2015-09-02 2015-12-30 杨静 Device for analyzing mental stress state of human body
CN106137226A (en) * 2016-07-29 2016-11-23 华南理工大学 A kind of stress appraisal procedure based on heart source property breath signal

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105193431A (en) * 2015-09-02 2015-12-30 杨静 Device for analyzing mental stress state of human body
CN106137226A (en) * 2016-07-29 2016-11-23 华南理工大学 A kind of stress appraisal procedure based on heart source property breath signal

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
SHUANG YANG: "Effect of defocus blur on the signal distribution of camera-based remote photoplethysmography", 《PROCEEDINGS OF SPIE》 *
王映丽: "基于人脸视频的心率特征提取方法研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 *

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112370057A (en) * 2020-11-09 2021-02-19 平安科技（深圳）有限公司 Pressure evaluation method and device, computer equipment and storage medium
CN112597949A (en) * 2020-12-29 2021-04-02 清华大学 Psychological stress measuring method and system based on video
CN113229790A (en) * 2021-05-17 2021-08-10 浙江大学 Non-contact mental stress assessment system
CN113397519A (en) * 2021-08-05 2021-09-17 季华实验室 Cardiovascular health state detection device
CN113657345A (en) * 2021-08-31 2021-11-16 天津理工大学 Non-contact heart rate variability feature extraction method based on reality application scene
CN113657345B (en) * 2021-08-31 2023-09-15 天津理工大学 Non-contact heart rate variability feature extraction method based on realistic application scene
CN114271800A (en) * 2021-12-01 2022-04-05 西北工业大学 Non-invasive continuous blood pressure monitoring method in office environment and application
WO2023246182A1 (en) * 2022-06-20 2023-12-28 华中师范大学 Non-contact fatigue detection system and method based on rppg
CN116889388A (en) * 2023-09-11 2023-10-17 长春理工大学 Intelligent detection system and method based on rPPG technology
CN116889388B (en) * 2023-09-11 2023-11-17 长春理工大学 Intelligent detection system and method based on rPPG technology
CN117281516A (en) * 2023-11-24 2023-12-26 吾征智能技术(北京)有限公司 Mental stress monitoring system, method, medium and device

Similar Documents

Publication Publication Date Title
CN111714144A (en) Mental stress analysis method based on video non-contact measurement
US7025729B2 (en) Apparatus for detecting sleep apnea using electrocardiogram signals
WO2019161609A1 (en) Method for analyzing multi-parameter monitoring data and multi-parameter monitor
WO2017024457A1 (en) Blood-pressure continuous-measurement device, measurement model establishment method, and system
WO2019161608A1 (en) Multi-parameter monitoring data analysis method and multi-parameter monitoring system
EP1284645A2 (en) System and device for multi-scale analysis and representation of electrocardiographic data
WO2005036455A2 (en) Method for displaying spectral trends in complex signals
CN112294282A (en) Self-calibration method of emotion detection device based on RPPG
CN110236508A (en) A kind of non-invasive blood pressure continuous monitoring method
CN112890792A (en) Cloud computing cardiovascular health monitoring system and method based on network camera
CN112788200B (en) Method and device for determining frequency spectrum information, storage medium and electronic device
US20240081705A1 (en) Non-contact fatigue detection system and method based on rppg
CN111714088B (en) Human body characteristic index detection method and system based on traditional Chinese medicine principle
CN109044275B (en) Non-invasive sensing sleep quality analysis system and method based on fuzzy inference system
CN112294272A (en) Monitor and irregular pulse rate identification method thereof
CN115299899A (en) Activity recognition and beat-to-beat blood pressure monitoring, analyzing and early warning system based on multiple sensors
CN111839488A (en) Non-invasive continuous blood pressure measuring device and method based on pulse wave
WO2022063047A1 (en) Photoplethysmography-based non-invasive diabetes prediction system and method
Madhav et al. Use of multi scale PCA for extraction of respiratory activity from photoplethysmographic signals
WO2021016892A1 (en) Method for analyzing regularity evaluation information, monitoring device and monitoring system
CN116434959A (en) Human body fatigue monitoring system based on mobile phone camera
Zhang et al. Development of a Continuous Blood Pressure Monitoring System based on Pulse Transit Time and Hemodynamic Covariates.
CN114587310A (en) Method for realizing invasive blood pressure waveform estimation based on photoplethysmography
CN113425282A (en) Respiration rate monitoring method and device based on multispectral PPG blind source separation method
TWI783904B (en) Non-contact physiological signal measuring method and device thereof

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination