CN112233813A

CN112233813A - Non-contact non-invasive heart rate and respiration measurement method and system based on PPG

Info

Publication number: CN112233813A
Application number: CN202011044656.5A
Authority: CN
Inventors: 郭宏博; 阎石; 朱文浩; 雷馥铭; 曹智杰
Original assignee: Lanzhou University
Current assignee: Lanzhou University
Priority date: 2020-09-28
Filing date: 2020-09-28
Publication date: 2021-01-15

Abstract

The invention discloses a non-contact non-invasive heart rate and respiration measuring method based on PPG, which comprises the following steps: the method comprises the following steps: acquiring a picture, and step two: obtaining a region of interest ROI, and step three: ICA and consistency analysis, step four: acquiring a change signal, and step five: gaussian filtering processing and multi-scale analysis, and the sixth step: and (4) performing fast Fourier transform. According to the invention, a novel scheme of extracting time domain heart rate and respiration value signals based on a histogram statistical thought is used, noise information contained in the obtained time domain signals is reduced, and the accuracy of heart rate detection is enhanced; a Gaussian filtering denoising method is innovatively used, so that the robustness of the system is greatly improved; establishing an Overlays algorithm with a large calculation amount, and accelerating hardware at a PL (programmable logic) end; the PS end utilizes a time overlapping method to improve algorithm parallelism, realizes the portability of hardware of the low-performance FPGA, and meets the requirements of real-time heart rate and respiratory value calculation on high frame number of the video.

Description

Non-contact non-invasive heart rate and respiration measurement method and system based on PPG

Technical Field

The invention belongs to the technical field of physiological information detection, and particularly relates to a non-contact noninvasive heart rate and respiration measuring method and system based on PPG.

Background

The heart rate measurement method is classified into contact measurement and non-contact measurement according to whether the heart rate measurement method is in contact with a detected person. The contact measurement method mainly comprises a pressure method, a resistance method and an electrocardiogram method. The pressure method has high accuracy, but the measuring equipment usually comprises a large-volume air pump, so the carrying is very inconvenient.

The resistance method is simple in equipment, but heart rate measurement errors are large through resistivity changes, and inconsistent results are generated along with changes of physiological and psychological states of a measured person. Although the ECG method, i.e., a professional electrocardiographic measurement method, can accurately measure various parameters of the heartbeat, it is difficult to realize telemedicine and long-term, daily examination because the placement and measurement operations of the conductive gel require specialized levels. The heart rate measurement can also adopt an infrared detection method, and the general infrared detection methods are divided into a finger clip type and an ear clip type, must be in contact with the surface and are not suitable for patients with skin surface burns and daily detection.

The non-contact method can be roughly classified into an electromagnetic detection method, an ultrasonic doppler detection method, a PPG detection method, and the like. Common to these methods is the measurement of subtle shifts and color changes in the skin surface during systole due to myocardial contraction, movement of the outer wall of the heart and soft tissue caused by changes in cardiac blood volume. The ultrasonic Doppler radar can be used for monitoring heart rate and respiration, but the components are large in size and expensive, and electromagnetic waves with specific frequency capable of penetrating tissues are adopted for a long time, so that the ultrasonic Doppler radar has certain harm to human health and is not suitable for daily physiological parameter detection.

The detection principle based on PPG (photoplethysmography) is: in a period of heartbeat, the blood volume changes periodically, when the heart contracts to pump blood outwards, the blood volume increases, the absorption amount of the light irradiated on the skin is large, and when the blood volume is maximum, the light intensity collected by the camera is weakest; when the heart relaxes, blood flows back, the blood volume is reduced, the absorption amount of the blood to the light irradiated on the skin is reduced, and when the blood volume is minimum, the light intensity collected by the camera is strongest. The optical signal is converted into an electric signal through the camera, so that the condition of blood volume change is obtained.

The detection method based on the PPG has achieved a good effect in a laboratory environment at present, and most of the frame processes of the detection method based on the PPG are scheme processes proposed by the Poh team based on MIT: firstly, aligning a person to be detected by using a camera, monitoring a face region by using a Viola-Jone face detection algorithm to be used as an ROI (region of interest), and taking the average value of each frame of image as a BVP (BVP) signal; then, the team removes noise in the signals by means of an independent component analysis method in blind source separation, and meanwhile, a band-pass filter with the frequency of [0.7,4] Hz is used for separating the heart rate signals; and finally, on the basis of the steps, calculating the heart rate by using the separated heart rate signals. But in daily application, the noise influence is large, the stability is poor and the relative static performance is good.

A non-contact human body heart rate measuring system based on an FPGA (electronic technology and software engineering) (page 80 in 20 years 2019), which uses hardware to realize a heart rate detection method based on PPG signals, but the method frame flow is based on an Poh team of MIT, and the system is greatly influenced by environmental factors and noise, has poor stability and better performance in relative stillness in daily application.

In summary, the contact heart rate measurement method and the non-contact heart rate measurement method have corresponding limitations, and the invention aims to solve the defects of inconvenience in carrying, large error, high cost, limitation in use and the like caused by different contact heart rate measurement methods and the problems of noise and stability of the conventional detection method based on PPG.

Disclosure of Invention

In order to solve the technical problems, the invention provides a non-contact non-invasive heart rate and respiration measuring method and system based on PPG, and the method comprises the following steps:

the method comprises the following steps: picture acquisition

And the network camera is used for extracting the incoming video signals frame by frame, so that the subsequent steps can be conveniently processed.

Step two: acquisition of a region of interest ROI

The method adopts an Adaboost face detection algorithm to detect a face area of a first frame, extracts a forehead part in the face area as an ROI (region of interest), and then adopts a method of combining Mean-Shift and Kalman filtering to track a characteristic area.

And the motion noise introduced when the human face has large-amplitude movement is reduced.

Step three: ICA and identity analysis

In the non-contact physiological signal extraction based on the PPG principle, physiological signals, motion artifacts, high-frequency noise and the like are mutually independent, observation signals R (T), G (T) and B (T) meet non-Gaussian distribution, and a mixed matrix formed by the observation signals R (T), G (T) and B (T) is a square matrix and meets the condition for carrying out ICA. And sequentially carrying out correlation analysis on the three groups of source signals separated by the ICA algorithm and the signal of the green channel, wherein the signal with the strongest correlation is used as the PPG signal.

Step four: change signal acquisition

The main steps of the extraction of the change signal are as follows: 1) based on the face detection technology, the ROI area is positioned. 2) The ROI region was equally divided into 25(5 × 5) sub-regions, and G-channel pixel intensity average M was extracted for each sub-region. 3) And extracting the difference value Q between each sub-region M value and the region M value of the previous frame. 4) And counting the number cnt of the region with the Q value larger than zero in the ROI region by using a binarization thought. 5) The time series I (t) is formed from cnt.

Step five: gaussian filtering processing and multi-scale analysis

Heart rate and respiration signals in the BVP signals are submerged by noise signals of a photosensitive element of the camera; in a real life scene, the illumination intensity of the facial skin of the detected person is uneven, which causes the signal-to-noise ratio of the BVP signal to be low. The method uses Gaussian filtering processing, replaces the pixel value in the center of the area with the weighted average value of all the pixel values of the area where the convolution kernel is located, can better eliminate the noise through the Gaussian filtering, and greatly improves the stability of heart rate measurement. The invention analyzes the photoelectric volume pulse wave signal by using a continuous wavelet transform method and extracts a stable respiration signal from the photoelectric volume pulse wave signal.

Step six: fast Fourier transform

In order to obtain frequency information in the signals, the frequency of heartbeat and breath is obtained by adopting a frequency domain calculation mode, a PPG signal in a time domain space is converted into a frequency domain space through fast Fourier transform, and the strongest frequency signal is found in a sliding window by a frequency spectrum and is used as the frequency of the heartbeat or breath. And selecting the frequency corresponding to the highest peak as a heart rate frequency to output within the range of 50-160 times per minute of the frequency spectrum, and selecting the frequency corresponding to the highest peak as a breathing frequency to output within the range of 10-30 times per minute of the frequency spectrum.

In the implementation mode of the system, the method is implemented by using the FPGA, and the whole system is divided into three hardware modules, namely a picture acquisition module, an FPGA image processing module and a physiological data display module. The method comprises the steps of completing picture acquisition through a network camera, completing FPGA image processing through the method described herein, and finally displaying physiological data in real time through mobile phone app.

Has the advantages that:

1. by using the novel scheme of extracting the time domain heart rate and respiration value signals based on the histogram statistical thought, the noise information contained in the obtained time domain signals is reduced, and the accuracy of heart rate detection is enhanced.

2. Aiming at the noises such as facial illumination shadow and the like, a Gaussian filtering denoising method is innovatively used, and the robustness of the system is greatly improved.

3. An acceleration scheme is researched, hardware transplantation and real-time heart rate and respiration detection are realized: the invention builds Overlays by an algorithm with larger computation amount, and performs hardware acceleration at the PL end; the PS end utilizes a time overlapping method to improve algorithm parallelism, realizes the portability of hardware of the low-performance FPGA, and meets the requirements of real-time heart rate and respiratory value calculation on high frame number of the video.

4. Accomplish the APP development, send the physiological data testing result to cell-phone APP end through bluetooth module or wiFi module to the physiological data change curve of the different time quantums of record user is preserved, makes the user realize remote monitoring and control, has improved user's handing-over mutual inductance, and the better management of convenience of customers is healthy.

Drawings

The following further description is made with reference to the accompanying drawings and detailed description:

FIG. 1 is a block diagram of a measurement method of the present invention.

Fig. 2 is a flow chart of the ICA algorithm.

Fig. 3 is a graph of physiological signals extracted from observed signals by the ICA.

Fig. 4 is a graph of a histogram based heart rate signal extraction with the addition of histogram statistics to the Algorithm curve and a control line.

Fig. 5 is a heart rate detection stability comparison graph before and after gaussian filtering, an Algorithm curve is added with an innovative Algorithm, and another line is a control group.

Fig. 6 is a diagram showing an ac component and a dc component in a photoplethysmographic signal.

Fig. 7 is a heart rate detection stability contrast chart before and after gaussian filtering.

FIG. 8 is a schematic diagram of a sliding window period analysis.

Fig. 9 is a schematic diagram of a hardware implementation module.

Detailed Description

As shown in fig. 1-9, the invention discloses a non-contact non-invasive heart rate and respiration detection method based on PPG, comprising the following steps:

step 1: picture acquisition

And extracting the video signals transmitted by the network camera frame by frame for subsequent processing.

Step 2: ROI acquisition

Compared with other parts, the human face has abundant capillaries and relatively large light absorption degree, and the BVP signal of the testee is extracted by selecting the face related area. Eye blinking motion and cheek muscle movement bring obvious noise signals to the BVP signal, reduce the signal-to-noise ratio of the heart rate signal, and the skin area of the nose which can be used as an ROI area is relatively small. The forehead area is relatively flat, the skin area which can be used as an ROI area is relatively large, and the forehead area contains a large number of capillary vessels. The method adopts an Adaboost face detection algorithm to detect a face area of a first frame, extracts a forehead part in the face area as an ROI (region of interest), and then adopts a method of combining Mean-Shift and Kalman filtering to track a characteristic area. And the motion noise introduced when the human face has large-amplitude movement is reduced.

And step 3: ICA and identity analysis

In the non-contact physiological signal extraction based on the PPG principle, physiological signals, motion artifacts, high-frequency noise and the like are mutually independent, observation signals R (T), G (T) and B (T) meet non-Gaussian distribution, and a mixing matrix formed by the observation signals R (T), G (T) and B (T) is a square matrix and meets the condition for ICA. The invention realizes the extraction of the source heart rate signal from the observation signal through the ICA algorithm. In the visible light band, because the blood has the maximum absorption capacity for the electromagnetic wave with the wavelength band of 510-590nm, namely the green light wave, the green signal can most represent the photoplethysmography signal which the invention wants to acquire, and three groups of source signals separated by the ICA algorithm are subjected to correlation analysis with the signal of the green channel in sequence, wherein the signal with the strongest correlation is taken as the PPG signal.

And 4, step 4: change signal extraction

Step 4-1: discrete differentiation; the extracted PPG signal consists of two parts: a PPG dc component and an ac component. The alternating current component of the pulse wave signal generally only accounts for 10% -20%, and the invention provides a discrete differentiation method for retaining the frequency information of the BVP signal and removing the basis quantity.

Step 4-2: a change signal extraction process; the system can realize remote heart rate monitoring in a range of 6 meters, the increase of the measurement distance can reduce the number of pixels of the forehead skin of a tested person, and the signal-to-noise ratio of the heart rate signal is reduced along with the reduction of an ROI (region of interest).

The main steps of the extraction of the change signal are as follows:

When the intensity of the reflected light increases, it represents the descending branch of the heartbeat/respiration signal, and when the intensity of the reflected light decreases, it represents the ascending branch of the heartbeat/respiration signal. Counting the number of pixel points with increased light intensity in the region of interest, and considering that the pixel points are in the ascending branches of heartbeat and respiration signals when the light intensity of most pixel points is in an increased state; otherwise, it is considered that the heart is in the descending branch of the heartbeat and respiratory signals. Ideally, a standard rectangular wave, called dppg (sensitivity Of ppg), is obtained, with the same frequency characteristics as the heartbeat and respiration signals.

In practical situations, due to the influence of ambient light variation, motion artifacts, uneven facial illumination and other factors, the facial reflected light variation signal is a superposition of the PPG signal and the noise signal. Namely, it is

ΔI_{Light variation}＝ΔI_{Change in blood volume}+ΔI_Noise(s)。

Ideally,. DELTA.I_{Change in blood volume}The size changes periodically, so when the blood volume change is large, the intensity of the PPG signal is strong and can be approximatedTo ignore noise signals when

ΔI_{Light variation}≈ΔI_{Change in blood volume}。

When the blood volume changes little, the strength of the PPG signal is weak, and the noise signal is not negligible. The extracted original signal is mixed with obvious noise signals, and the periodicity of the signals is seriously influenced.

For DPPG signal, when noise signal exists, delta I of certain pixel point of face_{Change in blood volume}Only slightly larger than Δ I_Noise(s)I.e. Delta I_{Light variation}Slightly greater than 0, Delta I_{Light variation}Therefore, the heartbeat and respiration signals can be correctly reflected. When the blood volume variation is large, the delta I of most pixel points_{Light variation}The extreme value of the heart beat signal and the respiration signal, namely the DPPG signal, can be correctly reflected. At this time, the PPG signal appears as aliasing of the noise signal with the same intensity as the heartbeat and respiration signals, and the heartbeat and respiration information cannot be reflected correctly.

In the rising branch of the heartbeat and respiration signals, because of the influence of noise, the cnt contributed by partial pixel points is-1 when the contribution is different from the real situation, but the cnt corresponding to most pixel points is 1, and in the falling branch of the heartbeat and respiration signals, the cnt corresponding to most pixel points is-1. By utilizing the accumulation effect, the noise resistance of the DPPG signal extracted by the pixel point difference density algorithm is better.

The DPPG signal is more noise resistant than the PPG signal.

And 5: gaussian filtering processing and multi-scale analysis

Heart rate and respiration signals in the BVP signals are submerged by noise signals of a photosensitive element of the camera; in a real life scene, the illumination intensity of the facial skin of the detected person is uneven, which causes the signal-to-noise ratio of the BVP signal to be low. The present invention uses a gaussian filtering process to replace the pixel value in the center of the region with a weighted average of all the pixel values of the region in which the convolution kernel is located. Therefore, the problem of low signal-to-noise ratio caused by uneven illumination intensity of facial skin can be solved well, meanwhile, the noise of the photosensitive element is mostly in Gaussian distribution, and the noise can be eliminated well through Gaussian filtering. Experiments prove that the stability of heart rate measurement is greatly improved by adding Gaussian filtering. The invention analyzes the photoelectric volume pulse wave signal by using a continuous wavelet transform method and extracts a stable respiration signal from the photoelectric volume pulse wave signal.

Step 6: fast Fourier transform

In a short time, the heart rate signal can be approximated to a steady-state signal, so that the time-frequency analysis can be performed on the time-domain signal in a windowing manner. Assuming that the width of the window function is W, performing time-frequency analysis on the PPG signal in the window through Fourier transform and calculating the heart rate value of the PPG signal. And then sliding the window by the step length L, and calculating a real-time heart rate value corresponding to the PPG signal of the next window. And obtaining a final heart rate value by using an arithmetic mean method according to the heart rate value calculated in one second.

The invention discloses a non-contact non-invasive heart rate and respiration measuring method and system based on PPG, which comprises the following steps:

2. Aiming at the noises such as the facial illumination shadow and the like, a Gaussian filtering denoising method is innovatively used, and the robustness of the system is greatly improved.

3. An acceleration scheme is researched, hardware transplantation and real-time heart rate and respiration detection are realized: in the project, Gaussian filtering and FFT with large iteration times and long operation time are used for building Overlays, and hardware acceleration is carried out at a PL (programmable logic) end; the PS terminal utilizes a time overlapping method: and meanwhile, in the subprocess, the Gaussian filtering algorithm and the FFT algorithm adopt the multithread programming, so that the throughput of data is increased, and the algorithm parallelism is improved.

The acceleration scheme realizes the portability of hardware of the low-performance FPGA and meets the requirement of real-time heart rate calculation on high frame number of videos.

Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A non-contact non-invasive heart rate and respiration measuring method based on PPG is characterized by comprising the following steps:

the method comprises the following steps: picture acquisition

Step two: acquisition of a region of interest ROI

Step three: ICA and identity analysis

Step four: change signal acquisition

Step five: gaussian filtering processing and multi-scale analysis

Step six: and (4) performing fast Fourier transform.

2. The PPG-based non-contact non-invasive heart rate and respiration measurement method of claim 1, wherein the first step is:

3. The PPG-based non-contact non-invasive heart rate and respiration measurement method of claim 1, wherein the second step is:

the method comprises the steps of detecting a face area of a first frame by adopting an Adaboost face detection algorithm, extracting a forehead part in the face area to be used as an ROI (region of interest), and then tracking a characteristic area by adopting a method of combining Mean-Shift and Kalman filtering.

4. The PPG-based non-contact non-invasive heart rate and respiration measurement method of claim 1, wherein the third step is:

in the non-contact physiological signal extraction based on the PPG principle, physiological signals, motion artifacts and high-frequency noise are mutually independent, observation signals R (T), G (T) and B (T) meet non-Gaussian distribution, and a mixing matrix formed by the observation signals R (T), G (T) and B (T) is a square matrix and meets the condition for ICA; and sequentially carrying out correlation analysis on the three groups of source signals separated by the ICA algorithm and the signal of the green channel, wherein the signal with the strongest correlation is used as the PPG signal.

5. The PPG-based non-contact non-invasive heart rate and respiration measurement method of claim 1, wherein the fourth step comprises the following specific steps:

1) based on the face detection technology, the ROI area is positioned;

2) equally dividing the ROI area into 25(5 x 5) sub-areas, and extracting the G channel pixel intensity average value M of each sub-area;

3) extracting the difference Q between the M value of each sub-region and the M value of the region of the previous frame;

4) counting the number cnt of the region with the Q value larger than zero in the ROI region by using a binarization thought;

5) the time series I (t) is formed from cnt.

6. The PPG-based non-contact non-invasive heart rate and respiration measurement method of claim 1, wherein the step five is:

using Gaussian filtering processing to replace the pixel value in the center of the area by the weighted average value of all the pixel values in the area where the convolution kernel is located; analyzing the photoelectric volume pulse wave signal by using a continuous wavelet transform method, and extracting a stable respiration signal from the photoelectric volume pulse wave signal.

7. The PPG-based non-contact non-invasive heart rate and respiration measurement method of claim 1, wherein the sixth step is:

in order to obtain frequency information in the signals, the frequency of heartbeat and breath is obtained by adopting a frequency domain calculation mode, a PPG signal in a time domain space is converted into a frequency domain space through Fast Fourier Transform (FFT), and the strongest frequency signal is found in a sliding window by a frequency spectrum and is used as the frequency of heartbeat or breath; and selecting the frequency corresponding to the highest peak as a heart rate frequency to output within the range of 50-160 times per minute of the frequency spectrum, and selecting the frequency corresponding to the highest peak as a breathing frequency to output within the range of 10-30 times per minute of the frequency spectrum.

8. A non-contact non-invasive heart rate and respiration measuring system based on PPG is characterized by comprising a picture acquisition module, an FPGA image processing module and a physiological data display module;

the picture acquisition module is used for completing picture acquisition through the network camera;

the FPGA image processing module is connected with the picture acquisition module and is used for completing FPGA image processing by the method of claims 1 to 7;

the physiological data display module is connected with the FPGA image processing module and comprises an HDMI display module and an APP display module,

the APP display module displays the physiological data in real time by using the mobile phone APP.