CN117678998A - Non-contact heart rate detection method based on self-adaptive projection plane and feature screening - Google Patents
Non-contact heart rate detection method based on self-adaptive projection plane and feature screening Download PDFInfo
- Publication number
- CN117678998A CN117678998A CN202311853827.2A CN202311853827A CN117678998A CN 117678998 A CN117678998 A CN 117678998A CN 202311853827 A CN202311853827 A CN 202311853827A CN 117678998 A CN117678998 A CN 117678998A
- Authority
- CN
- China
- Prior art keywords
- heart rate
- signal
- time
- frequency
- projection plane
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 18
- 238000012216 screening Methods 0.000 title claims abstract description 16
- 238000000034 method Methods 0.000 claims abstract description 25
- 238000001914 filtration Methods 0.000 claims abstract description 9
- 238000012935 Averaging Methods 0.000 claims abstract description 7
- 239000013598 vector Substances 0.000 claims description 13
- 230000003044 adaptive effect Effects 0.000 claims description 12
- 238000010586 diagram Methods 0.000 claims description 5
- 238000001228 spectrum Methods 0.000 claims description 3
- 238000013186 photoplethysmography Methods 0.000 claims description 2
- 238000004590 computer program Methods 0.000 claims 2
- 238000003672 processing method Methods 0.000 claims 2
- 230000001131 transforming effect Effects 0.000 claims 1
- 230000008859 change Effects 0.000 abstract description 10
- 230000008569 process Effects 0.000 abstract description 5
- 230000009467 reduction Effects 0.000 abstract description 4
- 238000005286 illumination Methods 0.000 abstract description 3
- 239000008280 blood Substances 0.000 description 5
- 210000004369 blood Anatomy 0.000 description 5
- VYZAMTAEIAYCRO-UHFFFAOYSA-N Chromium Chemical compound [Cr] VYZAMTAEIAYCRO-UHFFFAOYSA-N 0.000 description 4
- 238000009532 heart rate measurement Methods 0.000 description 4
- 238000011160 research Methods 0.000 description 4
- 210000003491 skin Anatomy 0.000 description 4
- 230000001815 facial effect Effects 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 2
- 230000008921 facial expression Effects 0.000 description 2
- 238000012544 monitoring process Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000012805 post-processing Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 230000010349 pulsation Effects 0.000 description 2
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 230000036996 cardiovascular health Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 210000004207 dermis Anatomy 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000002565 electrocardiography Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 229910052736 halogen Inorganic materials 0.000 description 1
- 150000002367 halogens Chemical class 0.000 description 1
- 230000001121 heart beat frequency Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Abstract
The invention discloses a non-contact heart rate detection method based on a self-adaptive projection plane and feature screening, which comprises the following basic processes: acquiring a face video by using a camera, performing face detection and tracking to acquire an ROI, performing spatial averaging on pixel values in the ROI to obtain a time-varying RGB signal, converting the three-dimensional RGB signal into a one-dimensional BVP signal containing pulse information by using an rPPG algorithm based on a self-adaptive projection plane and feature screening, performing band-pass filtering on the obtained BVP signal to remove noise outside a normal heart rate range, and acquiring a final heart rate value by using a frequency domain Fourier transform and a time-frequency domain frequency tracking algorithm. Aiming at the key step of signal dimension reduction in the whole flow, the self-adaptive projection plane provided by the invention eliminates the interference generated by factors such as illumination change, face movement and the like in the process of detecting heart rate based on face video, screens by utilizing two characteristics of a time domain and a frequency domain which can reflect pulse signals on the projection plane, and finally, the obtained pulse signals have higher signal-to-noise ratio.
Description
Technical Field
The invention belongs to the field of biomedical signal processing, and particularly relates to a method for extracting pulse signals from face videos captured by a camera and calculating heart rate values.
Background
Heart rate is one of important physiological parameters for assessing cardiovascular health of a human body, and research on a detection method thereof has important significance. Traditional methods of measuring heart rate include electrocardiography, chest straps, wearable devices, and the like. These methods suffer from several drawbacks, including: the need to connect electrodes and other devices has a great hindrance to the free movement of the user, and the contact type measurement mode may cause discomfort to the skin of the subject and can only be used for short-term monitoring and the like. With the popularization of network cameras and mobile phone cameras, a non-contact heart rate measurement method based on face videos is widely studied. The detection method can extract the pulse wave signals containing heart rate information only through the facial videos acquired by the camera, does not need to contact detection equipment with a human body, is simple and efficient, has low measurement cost, can be used for long-term heart rate monitoring, and has a huge application prospect.
The principle based on face video heart rate detection is photoplethysmography PPG, which is an abbreviation for photosystem graph. It is an optical technique that detects changes in blood volume of a tissue microvascular bed by measuring changes in transmitted or reflected light. Specifically, when light irradiates the face, the heart of the person to be measured periodically beats, so that the blood volume in the capillaries under the dermis layer changes periodically, the absorption of the light by the blood also changes, and the change finally appears on the change of the facial skin color. Although this very small change is difficult for the human eye to perceive, it can be captured by a camera sensor.
Most current studies are conducted under ideal conditions of high signal to noise ratio, i.e. subjects are relatively stationary and well-conditioned. However, the reality scene is complex, and in order to realize the daily wide application of the technology, the factors such as illumination change, face movement and facial expression interference, even body building and the like, need to be considered. At present, the research also has a scene of considering the existence of interference such as motion, but most of the research focuses on post-processing of removing noise influence such as self-adaptive filtering after the pulse signal is acquired, and the research prospect of directly acquiring the pulse signal with high signal to noise ratio from the step of rPPG algorithm is certain.
Disclosure of Invention
The invention aims to design a method for effectively extracting pulse signals with high signal-to-noise ratio from face videos and calculating heart rate.
The invention provides a non-contact heart rate detection method based on a self-adaptive projection plane and feature screening. Mainly comprises the following steps: the human face detection and tracking acquire the ROI area, the space average acquires the RGB time-varying signal, the rPPG algorithm based on the self-adaptive projection plane and the feature screening is utilized to realize the dimension reduction from the three-dimensional RGB signal to the one-dimensional BVP signal, the filtering is carried out, and the core rate value is acquired by utilizing the FFT or the frequency tracking algorithm. The specific processing steps are as follows:
step 1, shooting a face video by using a camera, positioning a face region by using a face detection and tracking technology, and averaging face pixel signals to obtain a three-dimensional RGB signal.
And 2, passing the RGB signals through a new rPPG dimensionality reduction algorithm to obtain a one-dimensional pulse wave signal containing heart rate information.
And step 3, performing band-pass filtering on the pulse wave signals to remove noise interference.
And 4, converting the signal in the step 3 to a frequency domain to analyze a spectrogram thereof, and calculating a heart rate value according to the frequency corresponding to the peak point of the signal spectrogram, or converting the signal in the step 3 to a time-frequency domain to analyze a time-frequency diagram thereof, and obtaining a heart rate value sequence changing along with time by using a frequency tracking algorithm.
Innovations and advantages of the present invention:
(1) The invention fully considers noise influence caused by interference factors such as light change, face movement and the like in the non-contact heart rate measurement process based on the face video, and provides the self-adaptive projection plane (Adaptive Projection Plane, APP) which can eliminate the interference of most of noise and obtain pulse signals with high signal to noise ratio.
(2) Based on the self-adaptive projection plane and feature screening, a complete heart rate measurement flow is formed by combining a new rPPG algorithm with other steps, and accurate heart rate measurement can be realized under the scenes of static state, light change or slight rotation of the head and the like.
Drawings
FIG. 1 is a main flow chart of non-contact heart rate detection based on adaptive projection plane and feature screening according to the present invention;
fig. 2 is a distribution of RGB tracks in a three-dimensional space obtained from face videos in different states;
FIG. 3 is a detailed step illustration of determining an adaptive projection plane;
FIG. 4 is a comparison of an adaptive projection plane with a fixed projection plane used in the CHROM, POS algorithm;
FIG. 5 is a comparative schematic diagram of obtaining a core rate value using a spectrogram;
FIG. 6 is a schematic diagram showing a comparison of a heart rate value sequence obtained using a time-frequency plot;
Detailed Description
The heart rate detection method based on the self-adaptive projection plane and the feature screening can realize accurate pulse signal extraction in various scenes;
referring to fig. 1, the main body of the present invention can be summarized into three steps, the first step can be summarized into a signal preparation stage, including two steps a and b in fig. 1, mainly performing face positioning and tracking on a detected video to obtain an ROI area, and obtaining RGB color change signals by using spatial averaging; the second step is a signal dimension reduction stage, which is the most critical step in the whole process, and comprises the step c in fig. 1, wherein a two-step projection method based on an adaptive projection plane and feature screening is mainly used for acquiring a one-dimensional pulse signal containing heart rate information; the last step can be summarized as the signal post-processing stage, which includes the filtering operation of step d plus step c in fig. 1, mainly using band-pass filtering to remove the noise interference outside the normal heart rate range, and then performing time-frequency analysis to calculate the heart rate value.
The details of the first step are as follows, face feature point coordinates of 468 points are extracted from each frame of image mainly by using a face tracking and feature point positioning frame, and the convex shell of the face skin region can be conveniently obtained by using the detailed feature point coordinates. Performing an averaging operation on all frames that vary with time to obtain a time-varying raw RGB signal c 0 (t)＝(r 0 (t),g 0 (t),b 0 (t)) in the following manner:
vector c i,j (t)＝(r i,j (t),g i,j (t),b i,j (t)) T Describes a convex hull image of the skin region of each frame of the face, i.e. transposed red r, consisting of pixels with coordinates i, j i,j (t), green g i,j (t) and blue b i,j (t) a channel.
As shown in fig. 2, each point of the RGB trace extracted in different states (stationary, head translation, head rotation, etc.) is plotted in three-dimensional RGB space, and it is observed that they all take on a spindle-like arrangement. The direction in which the color change of each channel signal is the largest is mainly caused by interference such as illumination change, facial expression, facial movement and the like, and is the noise direction which needs to be removed in the process of extracting the pulse signals. The RGB trajectory points are fitted to a straight line using the least square method, and then a projection plane perpendicular to the straight line is selected as a candidate plane for final pulse signal selection. Unlike the fixed projection planes used in the currently prevailing algorithms CHROM, POS, the projection planes used here are adaptive, varying with varying signal energy, robust to various noise disturbances.
The figure three shows in detail the determination of the adaptive projection plane. Firstly, the least square idea is utilized to perform straight line fitting on RGB scattered points in a three-dimensional space, as shown in figure 3a. After the direction vector z of the straight line is obtained, one direction vector a on the plane is obtained after two coordinate points of the target vector are known by using the condition that the two vectors are perpendicular to each other, and then the other direction vector b on the plane is obtained by using the condition that the three vectors are perpendicular to each other, as shown in fig. 3 b-c. In this way, the final projection plane used is determined. This rpg algorithm uses the original RGB signal, so the determined projection plane needs to be moved to the vicinity of the fitted line with the midpoint coordinates of the fitted line, and the final adaptive projection plane is shown in fig. 3 d.
After the acquisition of the projection plane, one projection vector needs to be selected as the final projection vector. To fully understand how the projection axis on the fitting plane affects the projected signal quality, the pulsatility of each projection direction was analyzed using an exhaustive method and compared to two fixed projection planes used by the CHROM, POS algorithm. As defined by the PBV algorithm, the blood volume pulse vector u is given by the halogen lamp and the specified optical RGB filter pbv Measured value of [0.33,0.77,0.53 ]] T Can be according to u pbv To determine the pulsatility of the projection direction on each projection plane, assuming that one projection axis on the projection plane is px, the pulsatility of this direction is defined as:
wherein px is a projection vector of 3*1, u pbv The blood volume pulse vector 3*1 defined for the PBV algorithm, p, defines the pulsatility in the px direction. The pulsation intensity is the absolute value of p, which reflects the amplitude of the pulsation variation in the z-direction. As can be seen from fig. 4 (where light/dark color indicates areas with stronger/weaker pulse intensity), the proposed adaptive projection plane has the same characteristics as both the CHROM and POS planes, and has a wide area with stronger pulse intensity, which illustrates that the adaptive projection plane can be used to extract purer pulse wave signals.
From the above analysis, it can be seen that different projection directions on the fitting plane can produce pulse signals with different intensities, and that some features can be used for screening. The Kurtosis feature is a time-domain based analysis that assumes that the distribution of pulses in the time domain is non-gaussian, the lower the value the closer the signal is to a non-gaussian distribution; the SNR characteristic is based on a frequency domain analysis, which assumes that the pulse is a periodic component and exhibits a sharp peak in the frequency domain, and that the higher the value is, the more pronounced the peak characteristic in the spectrogram, and the better the heart rate component can be identified from the noise. The following equation is the definition of the Kurtosis feature and the generation of candidate signals based on that feature on the fitting plane:
wherein Kurtosis is i Representing the ith pulse signal on the fitting planeIs used to represent the standard deviation operation, μ (·) represents the averaging operation, and H represents the optimal pulse signal with the smallest Kurtosis value. The following equation is the definition of SNR characteristics and the generation of candidate signals based on the characteristics in the fitting plane:
wherein SNR is i Representing the ith pulse signal on the projection planeSNR value of +.>Then it is the corresponding frequency spectrum, f is the frequency unit Hz, U t (f) Is a binary template window, H represents the optimum pulse signal with the maximum SNR value.
After determining the last two candidate signals, the estimated pulse signals obtained by the three-communication linear combination 2*G-R-B method are used as pseudo-reference PPG signals for screening, and the final pulse signals are determined.
The third step is to obtain the estimated pulse signal and then to post-process. The signal is band-pass filtered using a butterworth filter of order 6, in the range 0.7-4Hz, which corresponds to the normal heart beat frequency range. Then, an FFT is used to calculate a heart rate value of a signal, and fig. 5a and b show a spectrogram of an estimated pulse signal and a spectrogram of a corresponding reference PPG signal, respectively, where it can be seen that the measurement result of the method of the present invention is consistent with the reference value. A frequency domain tracking algorithm may also be used to obtain a heart rate value sequence of a signal, and fig. 6 shows a frequency tracking result of an estimated pulse signal and a reference heart rate value sequence, and it can be seen that the estimated result is substantially consistent with the group_route. The result shows that the non-contact heart rate detection method based on the self-adaptive projection plane and the feature screening can accurately and effectively extract the heart rate value of the detected main body.
The foregoing is merely a preferred embodiment of the invention, and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present invention, which are intended to be comprehended within the scope of the present invention.
Claims (9)
1. A non-contact heart rate detection method based on adaptive projection planes and feature screening, the method comprising:
s1, shooting a video by using an RGB color camera, positioning an ROI region by face detection and tracking, and obtaining RGB time-varying signals of the whole face region by spatial averaging;
s2, reducing the dimension of the three-dimensional RGB signal into a one-dimensional BVP signal by using an rPPG (Remote PhotoPlethysmoGraphy ) algorithm based on the self-adaptive projection plane and feature screening;
s3, removing noise components except 0.7-4Hz in the one-dimensional BVP signal by using band-pass filtering;
s4, converting the filtered time domain waveform into a frequency domain to obtain a spectrogram of a signal, and obtaining a heart rate value of a section of signal by identifying peak points;
s5, transforming the filtered time domain waveform to a time frequency domain to obtain a time frequency diagram of the signal, and obtaining a heart rate value sequence of the whole section of signal through a frequency tracking algorithm.
2. The method for detecting a heart rate based on a face video according to claim 1, for the step of acquiring RGB time-varying signals S1, comprising: and tracking and positioning the human face in real time by using a human face detection frame, selecting the whole human face region as the ROI, and carrying out spatial averaging to obtain an RGB value of a frame of image, and splicing in a time domain to obtain an RGB three-dimensional time-varying signal.
3. The heart rate detection method based on face video according to claim 1, for the step of reducing the dimension of a three-dimensional RGB signal into a one-dimensional BVP signal S2, providing an rPPG innovation algorithm based on an adaptive projection plane and feature screening, which is characterized by comprising the following specific steps:
for RGB pixel points of a section of the tested video sequence, spindle-shaped distribution is presented in a three-dimensional space, and the RGB pixel points are fitted into a straight line by using a least square method;
the direction of the straight line is the direction in which each color channel changes the most, which is caused by noise interference such as movement, and a plane perpendicular to the direction is selected as a final projection plane, so that an estimated pulse signal with high signal-to-noise ratio can be obtained;
determining two candidate projection vectors on a projection plane by utilizing a time domain Kurtosis characteristic and a frequency domain SNR characteristic, and projecting RGB signals to obtain two candidate projection signals;
finally, an estimated pulse signal obtained by using an rPPG algorithm (2*G-R-B) with linear combination of three color channels is used as a pseudo PPG reference signal, and a final projection signal is selected from two candidate projection signals to be used as the estimated pulse signal.
4. The Kurtosis feature, SNR feature of claim 3 wherein:
wherein the method comprises the steps ofRepresenting an estimated pulse signal to be evaluated, wherein sigma (·) represents standard deviation operation, and mu (·) represents mean value taking operation;the frequency spectrum corresponding to the pulse signal is frequency unit of Hz, U t (f) Is a two-value template window.
5. The method for detecting heart rate based on face video according to claim 1, for the step of filtering S3, characterized in that: the estimated signal obtained in claim 3 is bandpass filtered using a six-order butterworth filter with passband range 0.7hz,4hz, which is the range in which the normal heart rate value lies.
6. The method for detecting heart rate based on face video according to claim 1, for the step of calculating the heart rate value S4 in the frequency domain, characterized by the following: the FFT algorithm is used for converting the time domain into the frequency domain to obtain a time-frequency diagram of the estimated pulse signal, and the influence of noise outside the normal heart rate range is removed due to the simple filtering operation in the last step, so that the frequency of the peak point on the frequency spectrum can be used as the estimated heart rate value.
7. The method for detecting heart rate based on face video according to claim 1, wherein the step of acquiring the heart rate sequence S5 in the time-frequency domain tracking is characterized by the following: the method described in claim 6 can only obtain one estimated heart rate value for a video segment, if the heart rate value sequence of the tested video is desired to be obtained in real time, a frequency tracking algorithm can be used, firstly, a time-frequency chart of the estimated pulse signal is obtained by using an STFT algorithm, then, the time-frequency chart is operated by using the frequency tracking algorithm, and the heart rate value sequence which changes with time is obtained.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the processor implements the signal processing method as claimed in claim 1 when executing the program.
9. A computer-readable storage medium, on which a computer program is stored, characterized in that the program, when being executed by a processor, implements the processing method as claimed in claim 1.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311853827.2A CN117678998A (en) | 2023-12-28 | 2023-12-28 | Non-contact heart rate detection method based on self-adaptive projection plane and feature screening |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311853827.2A CN117678998A (en) | 2023-12-28 | 2023-12-28 | Non-contact heart rate detection method based on self-adaptive projection plane and feature screening |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117678998A true CN117678998A (en) | 2024-03-12 |
Family
ID=90137167
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311853827.2A Pending CN117678998A (en) | 2023-12-28 | 2023-12-28 | Non-contact heart rate detection method based on self-adaptive projection plane and feature screening |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117678998A (en) |
-
2023
- 2023-12-28 CN CN202311853827.2A patent/CN117678998A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Bobbia et al. | Unsupervised skin tissue segmentation for remote photoplethysmography | |
| CN107529646B (en) | Non-contact heart rate measurement method and device based on Euler image amplification | |
| Lam et al. | Robust heart rate measurement from video using select random patches | |
| CN110269600B (en) | Non-contact video heart rate detection method based on multivariate empirical mode decomposition and combined blind source separation | |
| CN109820499B (en) | High anti-interference heart rate detection method based on video, electronic equipment and storage medium | |
| Feng et al. | Dynamic ROI based on K-means for remote photoplethysmography | |
| JP6521845B2 (en) | Device and method for measuring periodic fluctuation linked to heart beat | |
| Bobbia et al. | Remote photoplethysmography based on implicit living skin tissue segmentation | |
| Feng et al. | Motion artifacts suppression for remote imaging photoplethysmography | |
| WO2019030074A1 (en) | Device, system and method for determining a physiological parameter of a subject | |
| Gudi et al. | Efficient real-time camera based estimation of heart rate and its variability | |
| CN109977858A (en) | A kind of heart rate detection method and device based on image analysis | |
| CN111387959A (en) | Non-contact physiological parameter detection method based on IPPG | |
| Song et al. | Remote photoplethysmography with an EEMD-MCCA method robust against spatially uneven illuminations | |
| CN111243739A (en) | Anti-interference physiological parameter telemetering method and system | |
| Mironenko et al. | Remote photoplethysmography: Rarely considered factors | |
| Huang et al. | A motion-robust contactless photoplethysmography using chrominance and adaptive filtering | |
| Jiang et al. | Real-time quantifying heart beat rate from facial video recording on a smart phone using Kalman filters | |
| CN111938622B (en) | Heart rate detection method, device and system and readable storage medium | |
| Przybyło | A deep learning approach for remote heart rate estimation | |
| Li et al. | Comparison of region of interest segmentation methods for video-based heart rate measurements | |
| CN113591769B (en) | Non-contact heart rate detection method based on photoplethysmography | |
| US11583198B2 (en) | Computer-implemented method and system for contact photoplethysmography (PPG) | |
| CN114557685B (en) | Non-contact type exercise robust heart rate measurement method and measurement device | |
| CN113456042A (en) | Non-contact facial blood pressure measuring method based on 3D CNN |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |