CN112890792A - Cloud computing cardiovascular health monitoring system and method based on network camera - Google Patents
Cloud computing cardiovascular health monitoring system and method based on network camera Download PDFInfo
- Publication number
- CN112890792A CN112890792A CN202011338427.4A CN202011338427A CN112890792A CN 112890792 A CN112890792 A CN 112890792A CN 202011338427 A CN202011338427 A CN 202011338427A CN 112890792 A CN112890792 A CN 112890792A
- Authority
- CN
- China
- Prior art keywords
- pulse wave
- cloud computing
- monitoring
- camera
- face
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 61
- 238000000034 method Methods 0.000 title claims abstract description 32
- 230000036996 cardiovascular health Effects 0.000 title claims abstract description 22
- 238000001514 detection method Methods 0.000 claims abstract description 42
- 239000008280 blood Substances 0.000 claims abstract description 30
- 210000004369 blood Anatomy 0.000 claims abstract description 30
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 claims abstract description 14
- 229910052760 oxygen Inorganic materials 0.000 claims abstract description 14
- 239000001301 oxygen Substances 0.000 claims abstract description 14
- 238000012545 processing Methods 0.000 claims abstract description 9
- 230000001815 facial effect Effects 0.000 claims abstract description 7
- 230000009467 reduction Effects 0.000 claims abstract description 3
- 230000036541 health Effects 0.000 claims description 15
- 238000000605 extraction Methods 0.000 claims description 14
- 238000004364 calculation method Methods 0.000 claims description 9
- 230000008859 change Effects 0.000 claims description 9
- 238000011156 evaluation Methods 0.000 claims description 6
- 238000010521 absorption reaction Methods 0.000 claims description 5
- 102000001554 Hemoglobins Human genes 0.000 claims description 4
- 108010054147 Hemoglobins Proteins 0.000 claims description 4
- 230000006835 compression Effects 0.000 claims description 4
- 238000007906 compression Methods 0.000 claims description 4
- 238000000354 decomposition reaction Methods 0.000 claims description 4
- 238000007781 pre-processing Methods 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 2
- 238000001914 filtration Methods 0.000 claims description 2
- 230000007246 mechanism Effects 0.000 claims description 2
- 238000010183 spectrum analysis Methods 0.000 claims description 2
- 230000002526 effect on cardiovascular system Effects 0.000 abstract description 12
- 238000005516 engineering process Methods 0.000 abstract description 9
- 230000007774 longterm Effects 0.000 abstract description 3
- 238000010586 diagram Methods 0.000 description 7
- 230000008569 process Effects 0.000 description 5
- 208000024172 Cardiovascular disease Diseases 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000001228 spectrum Methods 0.000 description 4
- 206010020772 Hypertension Diseases 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 230000007547 defect Effects 0.000 description 3
- 238000013186 photoplethysmography Methods 0.000 description 3
- 206010003210 Arteriosclerosis Diseases 0.000 description 2
- 208000011775 arteriosclerosis disease Diseases 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 210000000887 face Anatomy 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000011897 real-time detection Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 208000031226 Hyperlipidaemia Diseases 0.000 description 1
- 108010076504 Protein Sorting Signals Proteins 0.000 description 1
- 238000000862 absorption spectrum Methods 0.000 description 1
- 238000000149 argon plasma sintering Methods 0.000 description 1
- 210000001367 artery Anatomy 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 210000000988 bone and bone Anatomy 0.000 description 1
- 230000009084 cardiovascular function Effects 0.000 description 1
- 208000026106 cerebrovascular disease Diseases 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 201000010099 disease Diseases 0.000 description 1
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 1
- 239000003814 drug Substances 0.000 description 1
- 210000004709 eyebrow Anatomy 0.000 description 1
- 201000005577 familial hyperlipidemia Diseases 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000006996 mental state Effects 0.000 description 1
- 210000000056 organ Anatomy 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 208000020016 psychiatric disease Diseases 0.000 description 1
- 230000000241 respiratory effect Effects 0.000 description 1
- 230000000087 stabilizing effect Effects 0.000 description 1
- 210000001519 tissue Anatomy 0.000 description 1
- 230000002792 vascular Effects 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
- A61B5/02416—Detecting, measuring or recording pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/02028—Determining haemodynamic parameters not otherwise provided for, e.g. cardiac contractility or left ventricular ejection fraction
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/02028—Determining haemodynamic parameters not otherwise provided for, e.g. cardiac contractility or left ventricular ejection fraction
- A61B5/02035—Determining blood viscosity
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/145—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue
- A61B5/1455—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue using optical sensors, e.g. spectral photometrical oximeters
- A61B5/14551—Measuring characteristics of blood in vivo, e.g. gas concentration, pH value; Measuring characteristics of body fluids or tissues, e.g. interstitial fluid, cerebral tissue using optical sensors, e.g. spectral photometrical oximeters for measuring blood gases
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7225—Details of analog processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7253—Details of waveform analysis characterised by using transforms
- A61B5/7257—Details of waveform analysis characterised by using transforms using Fourier transforms
Abstract
The invention discloses a cloud computing cardiovascular health monitoring system based on a network monitoring camera and a method thereof, which are implemented by the following steps: 1) a front-end monitoring camera collects a facial video; 2) the cloud server side remotely acquires the uploaded video data stream; 3) selecting a monitored user face ROI to extract pulse waves; 4) and performing pulse wave noise reduction processing and estimating three cardiovascular parameters of heart rate, blood oxygen saturation and blood viscosity by using a corresponding algorithm. The invention provides a method for transmitting a remote video streaming picture by utilizing the existing network camera, and a cloud server side provides algorithm support required by an IPPG technology, so that the problems that the existing non-contact heart rate detection only supports local operation and is inconvenient for continuous real-time monitoring are solved, and the non-contact heart rate detection can realize long-term continuous detection in a real scene.
Description
Technical Field
The invention belongs to the field of medical video image processing, and particularly relates to a framework of a remote cardiovascular health monitoring system adopting a network camera and cloud computing and an implementation method thereof.
Background
Cardiovascular state is an important physiological index reflecting human health, and the main parameters are as follows: heart rate, blood oxygen saturation, and blood viscosity. It is proved to be an effective diagnosis means in the fields of cardiovascular and cerebrovascular diseases, hypertension, psychological and mental diseases and the like. According to the report issued by the national cardiovascular center, the number of cardiovascular diseases in China is nearly 2.9 hundred million, and the cardiovascular diseases are the most important threats to the health of residents in China. The cardiovascular health monitoring system can monitor important parameters of cardiovascular health in real time, and is particularly important for patients, particularly the old with cardiovascular disease risks.
Heart rate is an important parameter of cardiovascular health and refers to the number of beats of the heart per minute. The research on the medicine can reflect the cardiovascular function and mental state of human body. The blood oxygen saturation is the percentage of the volume of oxygenated hemoglobin (HbO2) to the volume of oxygenated hemoglobin and reduced hemoglobin (Hb) in the blood. I.e., the concentration of blood oxygen in the blood, which is an important physiological parameter of the respiratory cycle. The difference between HbO2 and Hb absorption spectra can be used, HbO2 mainly absorbs red light with the wavelength of 600-700, and Hb mainly absorbs near infrared light with the wavelength of 800-1000 for detection. According to the pulse wave waveform research of Loshichang et al, the blood viscosity can be judged by K value index, which can reflect the magnitude of vascular resistance. The K value parameter is influenced by the different pulse wave shapes, the normal range is 0.3-0.4, and the possibility of hyperlipemia is considered to exist when the K value exceeds 0.45. Hypertension and hyperlipidemia are considered to be major causes of arteriosclerosis, and if hypertension is present and the K value is too large, arteriosclerosis is considered to be likely to occur.
The clinical traditional cardiovascular monitoring technology is to bind twelve leads on an Electrocardiograph (ECG) to different parts of a human body to acquire accurate electrocardiographic data for analysis and processing. The problems exist in that: the cost is high, and the common family is difficult to bear the high selling price of the electrocardiograph; secondly, the following steps: the automation degree is low, the operation is complicated, and the requirement of professional knowledge on a user is high; thirdly, the method comprises the following steps: the use is inconvenient, the test subject cannot finish the detection process independently, and the comfort level is poor, so that the long-time real-time detection is not facilitated. In addition, the technology of performing heart rate detection by photoplethysmography (PPG), which is widely used on sports bracelets, is now under great development. However, there are also problems that contact measurement is necessary, measurable parameters are few, and facial image information analysis is missing.
Pavlidis et al originally proposed a non-contact heart rate detection method using a general camera, which is called imaging photoplethysmography (IPPG). The blood volume changes in the arteries and capillaries under the skin result in small changes in skin color that are not visible to the naked eye, but can be recorded by a camera. The method comprises the steps of acquiring a human face video, amplifying tiny color change information in the human face video through a specific algorithm to obtain pulse wave data for calculation. This technique has received attention and developed rapidly because of non-contact, multi-channel detection. However, the technology also has some problems in practical application, namely: a face video is collected firstly, and then an analysis processing flow is carried out after the video is collected, so that the real-time performance cannot be well met; secondly, the following steps: only local detection can be carried out, and the requirement of remote monitoring for the elderly living alone and the like cannot be met; thirdly, the method comprises the following steps: does not meet the requirement of continuous long-term detection. No methods or techniques to solve or improve on this problem have been seen.
Disclosure of Invention
The invention aims to overcome the defects of the traditional ECG technology and the existing IPPG technology in remotely monitoring heart rate and other cardiovascular parameters in real time, and provides a system and a method for remotely monitoring cardiovascular health in real time based on the combination of a network monitoring camera and cloud computing. According to the invention, a video picture is acquired by a common network camera and transmitted to the cloud server, and the cloud computing server detects the change of the skin color of the surface such as the human face in the picture by using an IPPG (Internet protocol packet) technology and calculates and monitors the cardiovascular parameters such as the heart rate, the blood oxygen saturation, the blood viscosity and the like of the human in the picture in real time.
In order to achieve the purpose of the invention, the invention provides a network camera-based cloud computing cardiovascular health monitoring system which comprises a network camera with a front end for collecting faces, a cloud computing server with a back end and corresponding android and IOS end cardiovascular health management software, wherein the cloud computing server obtains data uploaded by the camera, decompresses the data into an original video stream to extract and analyze IPPG signals, and the cardiovascular health management software displays cardiovascular health parameters and generates corresponding evaluation suggestions
Furthermore, the network monitoring camera at the front end is accessed to the internet to upload video data to a corresponding cloud computing server, and the camera should be arranged at a proper position to ensure that a complete user face picture can be obtained; the user can control the monitoring through the set gestures and mouth shapes.
The cloud computing server obtains a network camera picture in real time and carries out face detection, extracts human body pulse wave data of a detection area, and obtains cardiovascular health parameters including heart rate, blood oxygen and blood viscosity through an algorithm. The mobile phone software of the android and IOS terminal can remotely acquire the health parameters of the corresponding user on the cloud computing server in real time through the account number, give corresponding health evaluation and suggestions for specific parameters, and push emergency information through bound mobile phone short messages and phone notifications if necessary.
The invention also provides a corresponding implementation method of cardiovascular health monitoring based on the system, which comprises the following steps:
step 1: configuring the overall architecture of the system, and acquiring the video stream of a remote camera in real time by a server and carrying out image preprocessing operation;
step 2: monitoring the face state in a picture in real time through face detection, and selecting a Region Of Interest (ROI) for pulse wave extraction;
and step 3: after the face area is stable, extracting facial pulse waves, denoising original pulse waves, dynamically judging the waveform quality by adopting a sliding window mechanism, and automatically deleting current waveform data when the difference between the front and rear waveform changes is too large;
and 4, step 4: and acquiring the processed pulse wave data, and estimating the heart rate by adopting a power spectrum analysis method.
And 5: and (3) synchronously extracting the infrared video of the camera while collecting the RGB video, respectively calculating the absorption ratio of the red channel and the near infrared channel, and calculating the blood oxygen saturation.
Step 6: and (3) performing single-cycle decomposition on the pulse wave shape, and calculating the blood viscosity K value by using the shape difference of the pulse wave shape.
Preferably, step 1 specifically comprises the following steps:
step 1.1: acquiring compressed video data remotely transmitted by a network camera, and decompressing the compressed video data into an original RGB video stream;
step 1.2: the original RGB color model data is converted into HSV color model by utilizing the difference of absorption and reflection of light of blood and other body tissues, and H (Hue) channel is selected. Because the traditional IPPG pulse wave data only selects a single G (Green ) channel for calculation, the extracted color change is not comprehensive, and the H channel can convert the whole color change spectrum information of the face into a calculable numerical signal, thereby increasing the signal-to-noise ratio of the facial pulse wave extraction.
Preferably, the pulse wave extraction ROI selection in step 2 specifically includes the following 2 steps:
step 2.1: the cloud computing server performs a face detection algorithm on the video streaming picture in real time, and starts to extract pulse waves after the position of the recognition frame is stable for 3 s;
step 2.2: aiming at the edge of the face detection recognition frame, the pixel area which does not belong to the face exists, the area of face eyes and the like which can not detect the pulse color change exists, a skin detection algorithm is carried out on the basis of the face detection recognition frame, only the skin area with the color change in the recognition frame is ensured to participate in the pulse wave extraction, and the signal quality of the pulse wave extraction is further improved.
Preferably, step 3 specifically comprises the following steps:
step 3.1: firstly, removing high-frequency noise superposed on an original pulse wave signal and direct current signals generated by reflection areas such as bones and the like by using a narrow-band filter;
step 3.2: removing a possible linear trend (baseline drift) of the original waveform signal by adopting a sine wave fitting mode;
step 3.3: finally, after pulse wave data are segmented, comparison between Std (Standard Deviation) of each part and the whole waveform is detected respectively, the comparison is larger than a set threshold value, and a peak compression algorithm is adopted to remove local sharp pulse wave peaks, so that the whole amplitude of the waveform is uniform.
Compared with the prior art, the invention has the following beneficial effects:
1) the invention can realize the remote real-time non-contact monitoring of the cardiovascular parameters of the user through the network camera;
2) the defects that the traditional cardiovascular monitoring equipment is high in cost and complex in operation and a user cannot independently complete the monitoring process are overcome;
3) according to the technology, the real-time monitoring can be realized only by acquiring the face video without contacting electrodes or sensors with a human body, and the use experience of a subject is improved due to the cleanness and sanitation of the testing process;
4) the system has low cost, can utilize the existing network camera, only needs to upload video stream to a cloud computing server, and provides a specific monitoring algorithm, and can realize high-efficiency and accurate real-time monitoring function through software and hardware support;
5) the cloud computing server not only serves as a computing platform, but also serves as an analysis and maintenance platform of health data, a user does not need to pay attention to the detection result every time, the server side can regularly form a daily health report of the user and push the daily health report to a mobile phone or a desktop software side of the user, and messages are pushed in time when disease risks occur.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
The accompanying drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and together with the example serve to explain the principles of the invention and not to limit the invention.
Fig. 1 is an overall architecture diagram of a monitoring system according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a detection state provided by an embodiment of the present invention;
fig. 3 is a flowchart of an algorithm corresponding to the monitoring method according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a preferred ROI provided in accordance with an embodiment of the present invention;
FIG. 5 is a schematic diagram of a health management software interface according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the embodiments and features of the embodiments in the present application may be arbitrarily combined with each other without conflict.
As shown in fig. 1, the architecture of a cloud computing cardiovascular health monitoring platform based on a network monitoring camera includes a network camera for collecting monitoring pictures at a front end, a cloud computing server at a back end, and corresponding android, IOS and desktop health management software; the network camera can utilize the internet to transmit a monitoring picture to the cloud computing server in real time, after the server side acquires data uploaded by the camera and decodes the data to form an original video stream, cardiovascular state parameters of people in the picture, including heart rate, blood oxygen saturation, blood viscosity and the like, are computed, and corresponding evaluation suggestions are generated to remind timely.
The cloud computing server comprises a heart rate monitoring back end, a monitoring server and a database server which are sequentially connected, wherein background heart rate monitoring software runs on the heart rate monitoring back end (such as a monitoring PC); the monitoring server is connected to the network camera through a TCP/IP protocol via a switching routing system to obtain a monitoring video stream, and the monitoring server also performs data interaction with the database server and matched health management software.
Furthermore, the monitoring server also detects the user posture in the monitoring picture, and the user can send different control commands through different gestures, wherein the control commands comprise triggering monitoring, closing the camera for preset time, changing the heart rate monitoring period and the like.
Fig. 2 shows a schematic diagram of the detection state of a monitoring system. The network camera adopts a monitoring camera with infrared rays and is arranged at a proper position to ensure that a complete user face picture can be acquired. When the user enters the effective acquisition range (for example, 3m) of the camera and can be detected, the acquired human face video stream is sent to the cloud server host through the Internet. The user can trigger the monitoring platform to perform monitoring through specific gestures and mouth shapes.
Referring to fig. 3, the implementation steps of cardiovascular state parameter detection of the cloud computing cardiovascular health monitoring platform based on the network monitoring camera in the invention are as follows:
s1: when the network camera is used for remotely detecting the heart rate, firstly, an overall framework and a local acquisition environment are built, and the fact that the network camera can acquire complete facial data and can stably transmit the data to a cloud server end is ensured, as shown in fig. 2. And the server performs image preprocessing color space conversion on the acquired original video stream, converts an original RGB color model into an HSV color space, and selects an H channel to perform pulse wave extraction. The green (G) channel for extracting the IPPG signal from the traditional RGB color space video is converted into the HSV color space, and the hue (H) channel is selected for pulse wave extraction, so that all color change information of the face can be acquired, and the signal to noise ratio of the facial pulse wave extraction is greatly improved compared with the G channel.
The conversion relationship from RGB to H channel is shown in formula (1):
R’＝R/255G’＝G/255B’＝B/255
Cmax＝max(R’,G’,B’)Cmin＝min(R’,G’,B’)Δ＝Cmax-Cmin
wherein RGB is the pixel value of each channel in the original video color space, and f (H) is the H channel pixel value for extracting the pulse wave signal after conversion.
S2: the method for extracting pulse wave ROI optimization specifically comprises the following steps:
1) and face detection: and detecting the coordinates of the face region in real time and extracting the characteristic points of the main organs by adopting a DRMF (DRMF) method.
2) And skin detection: on the basis of the face detection rectangular recognition area, a skin detection algorithm is carried out to further remove the non-skin areas such as the edge background area of the face detection frame and five human faces. The YCrCb color space is adopted for judging the skin detection, and the Asian yellow race parameter is preferably used as the threshold value. The specific judgment formula is as follows:
133<＝Cr<＝173
77<＝C＝<＝127
as an example, fig. 4 shows a ROI preference diagram. The pulse wave extraction optimization method enables the ROI extracted by the IPPG signal to be in an original human face detection frame area, further carries out a skin detection algorithm on images in the frame, removes the edges of the detection frame and areas, such as eyes and eyebrows, of the face, which do not contain pulse wave signals or are not rich in pulse wave signals, and improves the accuracy of signal extraction.
S3: in the detection process, a plurality of threads can be started to respectively perform the functions of face detection, pulse wave acquisition, real-time heart rate calculation and the like. When the cloud server monitors that the face monitoring area in the picture is stable for 3s, the pixel mean value of the frame sequence H channel of the ROI acquired by the camera is automatically stored in the one-dimensional array with the length of 512. With reference to fig. 3, the real-time heart rate calculation function may be enabled when a full 256 frames are cyclically acquired. When the cycle acquisition is completed by 512 frames, the full-function processing of blood oxygen calculation, blood viscosity calculation and heart rate calculation can be carried out. And in the cyclic acquisition process, whether the current frame is available or not is judged according to the pixel jump size between adjacent frames. If available (e.g., pixel jump values within a predetermined threshold), the data is processed and saved, and if not, the data is discarded. For the pulse wave extracted from the selected ROI, the process of further noise reduction processing comprises the following steps:
1) firstly, an ideal band-pass FIR filter is utilized to filter the extracted pulse wave, the direct current and high-frequency noise of the original pulse wave are removed, the pass band range is 0.83-3Hz, and the corresponding heart rate range is 50-180 times/minute;
2) secondly, removing the possible trend deviation condition of the original pulse wave signal by adopting a sine wave fitting mode, and stabilizing the baseline of the pulse wave signal to the same level;
3) finally, dividing the extracted pulse wave into a plurality of sections, respectively calculating the size of each section Std (Standard development, Standard Deviation), judging that spike jump exists if the detected section Std is more than twice of the whole waveform Std, and starting a spike compression algorithm to enable the pulse wave amplitude to be uniform;
s4: calculating through the power spectrum to obtain a heart rate value, wherein the calculating steps are as follows:
storing pulse wave data of n frames (n is 1,2,3 … S) after filtering processing to obtain a one-dimensional BVP signal sequence B [ n ];
② to B [ n]Performing fast Fourier transform to obtain power spectrum PBvp：
F(t)＝FFT(B(t)) (3)
PBvp(t)＝|F(t)|2 (4)
Where the FFT is a fast fourier transform function.
Thirdly, calculating the heart rate value HRThe following steps:
T＝max{PBvp(t)} (5)
where T is the maximum power spectrum, S is the total captured video frames, and Fs is the frame rate at which the video stream is captured.
So far, one heart rate detection based on face video processing is basically finished.
S5: because the adopted monitoring camera can shoot the infrared video and the RGB video at the same time, the pulse wave I is extracted from the red channel and the near-infrared channelRAnd IIRespectively calculating the DC component and the AC component to obtain
And
can be processed by the same method to obtain IIDirect current component and alternating current component of
And
from Lambert-beer's law and knowledge of light scattering, Blood Oxygen Saturation (SpO)2) The formula is as follows:
SpO2＝A*R+B (7)
absorption ratio in the formula
A. B is two empirical constants, which can be obtained by calibrating a professional contact oximeter.
S6: and (3) performing single-cycle decomposition on the filtered pulse waves, and respectively calculating the index of the blood viscosity K value of each cycle, wherein the formula is as follows:
wherein, Pm is the mean value of the pulse wave, Ps is the peak value, and Pd is the valley value. And then, according to the waveform signal-to-noise ratio of the single-period pulse wave, weighting and calculating the K value index of the whole pulse waveform, wherein the formula is as follows:
wherein
Is a weighted K value index, xtx2…xnIs based onWeighting factor, k, determined by the quality of the periodic waveformtk2…knK value index of single period, n is the number of pulse wave after single period decomposition.
At this point, the acquisition and calculation of cardiovascular status parameters are completed, and the intuitive monitoring parameter presentation to the user can be performed through the health management software interface diagram shown in fig. 5. The mobile phone software and the PC client of the android and IOS terminals can remotely acquire health parameters of corresponding users on the cloud computing server in real time through account numbers, meanwhile, big data analysis can be carried out according to continuous monitoring for a period of time, corresponding health evaluation and suggestions, early warning and the like can be given to specific parameters to remind the users in time, and emergency information can be pushed through bound mobile phone short messages and phone notifications when necessary.
The implementation method can effectively overcome the defect that the current photoelectric heart rate sensor can only monitor in a contact mode, greatly reduces the high price of a computing terminal required by the current IPPG technology, and can save the cost of the computing terminal by utilizing cloud computing on one hand and realize real-time detection by combining with a monitoring picture on the other hand, so that the non-contact heart rate detection can realize long-term continuous detection in a real scene. In addition, the support of the multi-thread algorithm also enables the monitoring result to be efficient and accurate.
Although the embodiments of the present invention have been described above, the above description is only for the convenience of understanding the present invention, and is not intended to limit the present invention. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the scope of the present invention should be determined by the following claims.
Claims (10)
1. The cloud computing cardiovascular health monitoring system based on the network camera is characterized by comprising the network camera with the front end used for collecting a human face, a cloud computing server with the rear end and corresponding supporting android and IOS end cardiovascular health management software, wherein the cloud computing server obtains data uploaded by the camera and decompresses the data into an original video stream to extract IPPG signals and analyze the IPPG signals, and the cardiovascular health management software displays cardiovascular health parameters and generates corresponding evaluation suggestions.
2. The monitoring system of claim 1, wherein the network monitoring camera at the front end accesses the internet to upload video data to a corresponding cloud computing server, and the camera should be set at a suitable position to ensure that a complete user face picture can be obtained; the user can control the monitoring through the set gestures and mouth shapes.
3. The monitoring system of claim 1, wherein the cloud computing server obtains the network camera picture in real time and performs face detection, extracts human pulse wave data in the detection area, and obtains cardiovascular health parameters including heart rate, blood oxygen, and blood viscosity through an algorithm.
4. The monitoring system of claim 1, wherein the mobile software of the android and IOS terminals can remotely obtain health parameters of corresponding users on the cloud computing server in real time through an account, and can give corresponding health evaluation and suggestions for specific parameters, and if necessary, can push emergency information through bound mobile phone short messages and phone notifications.
5. A cloud computing cardiovascular health monitoring implementation method based on a network camera is characterized by comprising the following steps:
step 1: configuring an overall system architecture, and acquiring a video stream of a remote network camera in real time by a cloud computing server and carrying out image preprocessing operation;
step 2: monitoring the face state in a picture in real time through face detection, and selecting an ROI (region of interest) for pulse wave extraction;
and step 3: after the face area is stable, extracting facial pulse waves, denoising original pulse waves, dynamically judging the waveform quality by adopting a sliding window mechanism, and automatically deleting current waveform data when the difference between the front and rear waveform changes is too large;
and 4, step 4: acquiring the processed pulse wave data, and estimating the heart rate by adopting a power spectrum analysis method;
and 5: synchronously extracting the infrared video of the camera while collecting the RGB video, respectively calculating the absorption ratio of a red channel and a near infrared channel, and calculating the blood oxygen saturation;
step 6: and (3) performing single-cycle decomposition on the pulse wave shape, and calculating the blood viscosity K value by using the shape difference of the pulse wave shape.
6. The method of claim 5, wherein step 1 comprises the steps of:
step 1.1: acquiring compressed video data remotely transmitted by a network camera, and decompressing the compressed video data into an original RGB video stream;
step 1.2: and converting the original RGB color model data into an HSV color model, and selecting an H channel in the HSV color model for subsequent pulse wave extraction.
7. The method according to claim 5, wherein the ROI selection of step 2 specifically comprises:
step 2.1: the cloud computing server performs a face detection algorithm on the video streaming picture in real time, and starts to extract pulse waves after the position of the identification frame is stable;
step 2.2: aiming at the edge of the face detection recognition frame, a pixel area which does not belong to the face exists and an area which can not detect the pulse color change of the face, a skin detection algorithm is carried out on the basis of the face detection recognition frame, so that only the skin area with the color change in the recognition frame is ensured to participate in the pulse wave extraction, and the signal quality of the pulse wave extraction is further improved.
8. The method according to claim 5, wherein the pulse wave noise reduction processing includes the steps of:
step 3.1: firstly, carrying out narrow-band-pass filtering on an originally extracted pulse wave signal by using a narrow-band filter, and removing high-frequency noise superposed on the original pulse wave signal and a direct current signal generated in a reflection area;
step 3.2: removing a possible linear trend of the signal by adopting a sine wave fitting mode on the filtered signal;
step 3.3: and finally, after pulse wave data are segmented, respectively calculating the comparison between the standard deviation Std of each pulse wave and the overall waveform Std, judging whether a peak compression algorithm is started, and when the comparison result is greater than a set threshold value, starting the peak compression algorithm to remove local sharp pulse wave peaks, so that the overall amplitude of the waveform is uniform.
9. The method of claim 5, wherein the calculation of the blood oxygen saturation parameter comprises: and (3) respectively selecting an R channel and an infrared channel of an RGB image of the network camera to extract pulse waves and calculate the hemoglobin absorption ratio, and further obtaining the blood oxygen saturation parameter through calibration with a professional instrument.
10. The method of claim 5, wherein the de-noised pulse wave is decomposed for a single period, and K value indicators for each period are calculated respectively, and then weighted average is performed according to the quality of the waveform for each period to obtain K value indicators for the whole waveform, thereby estimating the blood viscosity indicator of the user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011338427.4A CN112890792A (en) | 2020-11-25 | 2020-11-25 | Cloud computing cardiovascular health monitoring system and method based on network camera |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011338427.4A CN112890792A (en) | 2020-11-25 | 2020-11-25 | Cloud computing cardiovascular health monitoring system and method based on network camera |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112890792A true CN112890792A (en) | 2021-06-04 |
Family
ID=76111313
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011338427.4A Pending CN112890792A (en) | 2020-11-25 | 2020-11-25 | Cloud computing cardiovascular health monitoring system and method based on network camera |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112890792A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113397505A (en) * | 2021-06-25 | 2021-09-17 | 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) | Physiological signal detection method and system |
| CN113397519A (en) * | 2021-08-05 | 2021-09-17 | 季华实验室 | Cardiovascular health state detection device |
| CN113712526A (en) * | 2021-09-30 | 2021-11-30 | 四川大学 | Pulse wave extraction method and device, electronic equipment and storage medium |
| CN113938622A (en) * | 2021-12-15 | 2022-01-14 | 慕思健康睡眠股份有限公司 | Blood pressure detection device based on asynchronously recorded video and storage medium |
| CN115914583A (en) * | 2023-02-28 | 2023-04-04 | 中国科学院长春光学精密机械与物理研究所 | Old people monitoring equipment and monitoring method based on visual identification |
| CN114343625B (en) * | 2021-12-17 | 2024-04-26 | 杭州电子科技大学 | Non-contact capillary blood gas parameter determination method based on color image analysis and application |
Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005211197A (en) * | 2004-01-28 | 2005-08-11 | Matsushita Electric Ind Co Ltd | Health care network camera and database server |
| CN103271743A (en) * | 2012-12-10 | 2013-09-04 | 中国人民解放军第一五二中心医院 | Non-contact oxyhemoglobin saturation measuring device based on imaging device |
| CN103826532A (en) * | 2011-08-22 | 2014-05-28 | Isis创新有限公司 | Remote monitoring of vital signs |
| CN104523263A (en) * | 2014-12-23 | 2015-04-22 | 华南理工大学 | Mobile internet based pregnant and lying-in woman health surveillance system |
| CN105188521A (en) * | 2013-03-14 | 2015-12-23 | 皇家飞利浦有限公司 | Device and method for obtaining vital sign information of a subject |
| US20160174913A1 (en) * | 2014-12-23 | 2016-06-23 | Intel Corporation | Device for health monitoring and response |
| CN105989357A (en) * | 2016-01-18 | 2016-10-05 | 合肥工业大学 | Human face video processing-based heart rate detection method |
| US20160302735A1 (en) * | 2013-12-25 | 2016-10-20 | Asahi Kasei Kabushiki Kaisha | Pulse wave measuring device, mobile device, medical equipment system and biological information communication system |
| CN106778695A (en) * | 2017-01-19 | 2017-05-31 | 北京理工大学 | A kind of many people's examing heartbeat fastly methods based on video |
| JP2017093760A (en) * | 2015-11-22 | 2017-06-01 | 国立大学法人埼玉大学 | Device and method for measuring periodic variation interlocking with heart beat |
| WO2017154477A1 (en) * | 2016-03-08 | 2017-09-14 | パナソニックＩｐマネジメント株式会社 | Pulse estimating device, pulse estimating system, and pulse estimating method |
| US20180068171A1 (en) * | 2015-03-31 | 2018-03-08 | Equos Research Co., Ltd. | Pulse wave detection device and pulse wave detection program |
| KR20180074066A (en) * | 2016-12-23 | 2018-07-03 | 순천향대학교 산학협력단 | Patients monitoring system and method thereof |
| CN109247944A (en) * | 2018-08-30 | 2019-01-22 | 合肥工业大学 | A kind of contactless method for detecting blood oxygen saturation based on low side color camera |
| KR20190023167A (en) * | 2017-08-28 | 2019-03-08 | 성균관대학교산학협력단 | Method and device for measuring blood viscosity using image |
| US20190239762A1 (en) * | 2018-02-08 | 2019-08-08 | Rochester Institute Of Technology | Opportunistic Plethysmography using Video Cameras |
| CN110279406A (en) * | 2019-05-06 | 2019-09-27 | 苏宁金融服务(上海)有限公司 | A kind of touchless pulse frequency measurement method and device based on camera |
| JP2019180666A (en) * | 2018-04-05 | 2019-10-24 | Ｗｉｎフロンティア株式会社 | Pulse wave detection terminal program, pulse wave analysis server program and pulse wave detection analysis terminal program |
| CN110384491A (en) * | 2019-08-21 | 2019-10-29 | 河南科技大学 | A kind of heart rate detection method based on common camera |
| CN111027485A (en) * | 2019-12-11 | 2020-04-17 | 南京邮电大学 | Heart rate detection method based on face video detection and chrominance model |
| CN111434472A (en) * | 2019-01-14 | 2020-07-21 | 漫谷科技股份公司 | Health management platform based on household intelligent robot and application thereof |
| KR20200099248A (en) * | 2019-02-13 | 2020-08-24 | 와이케이씨테크(주) | Estimation method of blood vessel elasticity and arrhythmia using skin image |
-
2020
- 2020-11-25 CN CN202011338427.4A patent/CN112890792A/en active Pending
Patent Citations (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005211197A (en) * | 2004-01-28 | 2005-08-11 | Matsushita Electric Ind Co Ltd | Health care network camera and database server |
| CN103826532A (en) * | 2011-08-22 | 2014-05-28 | Isis创新有限公司 | Remote monitoring of vital signs |
| CN103271743A (en) * | 2012-12-10 | 2013-09-04 | 中国人民解放军第一五二中心医院 | Non-contact oxyhemoglobin saturation measuring device based on imaging device |
| CN105188521A (en) * | 2013-03-14 | 2015-12-23 | 皇家飞利浦有限公司 | Device and method for obtaining vital sign information of a subject |
| US20160302735A1 (en) * | 2013-12-25 | 2016-10-20 | Asahi Kasei Kabushiki Kaisha | Pulse wave measuring device, mobile device, medical equipment system and biological information communication system |
| CN104523263A (en) * | 2014-12-23 | 2015-04-22 | 华南理工大学 | Mobile internet based pregnant and lying-in woman health surveillance system |
| US20160174913A1 (en) * | 2014-12-23 | 2016-06-23 | Intel Corporation | Device for health monitoring and response |
| US20180068171A1 (en) * | 2015-03-31 | 2018-03-08 | Equos Research Co., Ltd. | Pulse wave detection device and pulse wave detection program |
| JP2017093760A (en) * | 2015-11-22 | 2017-06-01 | 国立大学法人埼玉大学 | Device and method for measuring periodic variation interlocking with heart beat |
| CN105989357A (en) * | 2016-01-18 | 2016-10-05 | 合肥工业大学 | Human face video processing-based heart rate detection method |
| WO2017154477A1 (en) * | 2016-03-08 | 2017-09-14 | パナソニックＩｐマネジメント株式会社 | Pulse estimating device, pulse estimating system, and pulse estimating method |
| KR20180074066A (en) * | 2016-12-23 | 2018-07-03 | 순천향대학교 산학협력단 | Patients monitoring system and method thereof |
| CN106778695A (en) * | 2017-01-19 | 2017-05-31 | 北京理工大学 | A kind of many people's examing heartbeat fastly methods based on video |
| KR20190023167A (en) * | 2017-08-28 | 2019-03-08 | 성균관대학교산학협력단 | Method and device for measuring blood viscosity using image |
| US20190239762A1 (en) * | 2018-02-08 | 2019-08-08 | Rochester Institute Of Technology | Opportunistic Plethysmography using Video Cameras |
| JP2019180666A (en) * | 2018-04-05 | 2019-10-24 | Ｗｉｎフロンティア株式会社 | Pulse wave detection terminal program, pulse wave analysis server program and pulse wave detection analysis terminal program |
| CN109247944A (en) * | 2018-08-30 | 2019-01-22 | 合肥工业大学 | A kind of contactless method for detecting blood oxygen saturation based on low side color camera |
| CN111434472A (en) * | 2019-01-14 | 2020-07-21 | 漫谷科技股份公司 | Health management platform based on household intelligent robot and application thereof |
| KR20200099248A (en) * | 2019-02-13 | 2020-08-24 | 와이케이씨테크(주) | Estimation method of blood vessel elasticity and arrhythmia using skin image |
| CN110279406A (en) * | 2019-05-06 | 2019-09-27 | 苏宁金融服务(上海)有限公司 | A kind of touchless pulse frequency measurement method and device based on camera |
| CN110384491A (en) * | 2019-08-21 | 2019-10-29 | 河南科技大学 | A kind of heart rate detection method based on common camera |
| CN111027485A (en) * | 2019-12-11 | 2020-04-17 | 南京邮电大学 | Heart rate detection method based on face video detection and chrominance model |
Non-Patent Citations (8)
| Title |
|---|
| 关天一;宋春林;: "一种基于脸部视频及脉搏特征平面的心率检测算法", 信息技术与信息化, no. 10 * |
| 冯小智;: "基于Sooc云平台对心血管疾病患者健康远程计算机监护系统的设计与应用效果", 电脑知识与技术, no. 35 * |
| 刘丽佳: "光电容积脉搏波信号采集及预处理方法研究", 中国优秀硕士学位论文全文数据库 (医药卫生科技辑), pages 4 * |
| 吴楚宜;陈悦;鞠泽亮;李帅;张文杰;: "基于云服务的人体血压特征监控系统设计", 自动化应用, no. 07 * |
| 张磊;田泽懿;唐春晖;高秀敏;: "生理参数监测技术及设备的研究进展", 光学仪器, no. 02 * |
| 李全彬;曹汉清;: "基于皮肤光学模型的非接触式心率测量", 江苏师范大学学报(自然科学版), no. 01 * |
| 李全彬等: "基于皮肤光学模型的非接触式心率测量", 江苏师范大学学报（自然科学版）, pages 58 * |
| 陈旭: "基于人脸视频的心率测量算法研究", 中国优秀硕士学位论文全文数据库 (医药卫生科技辑), pages 49 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113397505A (en) * | 2021-06-25 | 2021-09-17 | 合肥综合性国家科学中心人工智能研究院(安徽省人工智能实验室) | Physiological signal detection method and system |
| CN113397519A (en) * | 2021-08-05 | 2021-09-17 | 季华实验室 | Cardiovascular health state detection device |
| CN113712526A (en) * | 2021-09-30 | 2021-11-30 | 四川大学 | Pulse wave extraction method and device, electronic equipment and storage medium |
| CN113712526B (en) * | 2021-09-30 | 2022-12-30 | 四川大学 | Pulse wave extraction method and device, electronic equipment and storage medium |
| CN113938622A (en) * | 2021-12-15 | 2022-01-14 | 慕思健康睡眠股份有限公司 | Blood pressure detection device based on asynchronously recorded video and storage medium |
| CN113938622B (en) * | 2021-12-15 | 2022-02-15 | 慕思健康睡眠股份有限公司 | Blood pressure detection device based on asynchronously recorded video and storage medium |
| CN114343625B (en) * | 2021-12-17 | 2024-04-26 | 杭州电子科技大学 | Non-contact capillary blood gas parameter determination method based on color image analysis and application |
| CN115914583A (en) * | 2023-02-28 | 2023-04-04 | 中国科学院长春光学精密机械与物理研究所 | Old people monitoring equipment and monitoring method based on visual identification |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112890792A (en) | Cloud computing cardiovascular health monitoring system and method based on network camera | |
| US10004410B2 (en) | System and methods for measuring physiological parameters | |
| JP6194105B2 (en) | Improved signal selection for acquiring remote photoplethysmographic waveforms | |
| Siddiqui et al. | A pulse rate estimation algorithm using PPG and smartphone camera | |
| CN107358220A (en) | A kind of human heart rate and the contactless measurement of breathing | |
| CN109044314B (en) | Non-contact heart rate monitoring method based on Euler video amplification | |
| Qiao et al. | Measuring heart rate and heart rate variability with smartphone camera | |
| CN112233813A (en) | Non-contact non-invasive heart rate and respiration measurement method and system based on PPG | |
| CN106236049A (en) | Blood pressure measuring method based on video image | |
| CN111714144A (en) | Mental stress analysis method based on video non-contact measurement | |
| CN110236508A (en) | A kind of non-invasive blood pressure continuous monitoring method | |
| CN112294282A (en) | Self-calibration method of emotion detection device based on RPPG | |
| WO2021164350A1 (en) | Method and device for generating photoplethysmography signal | |
| CN114781465A (en) | rPPG-based non-contact fatigue detection system and method | |
| EP4033495A1 (en) | Activity task evaluating system, and activity task evaluating method | |
| CN112120711A (en) | Noninvasive diabetes prediction system and method based on photoplethysmography | |
| Sharma et al. | Non-contact measurement of human heart rate using low cost video camera | |
| CN113317766A (en) | A311D-based embedded non-contact physiological parameter monitoring system | |
| Wang et al. | KLT algorithm for non-contact heart rate detection based on image photoplethysmography | |
| Zhuang et al. | Remote blood pressure measurement via spatiotemporal mapping of a short-time facial video | |
| TWI783904B (en) | Non-contact physiological signal measuring method and device thereof | |
| CN110200641A (en) | A kind of method and device based on touch screen measurement cognitive load and psychological pressure | |
| Lan et al. | A non-contact oxygen saturation detection method based on dynamic spectrum | |
| CN117557506A (en) | Device and method for detecting infant heart rate with high video precision | |
| Cruz et al. | Non-contact heart rate and respiratory rate monitoring system using histogram of oriented gradients |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |