CN114469036A - Remote heart rate monitoring method and system based on video images - Google Patents
Remote heart rate monitoring method and system based on video images Download PDFInfo
- Publication number
- CN114469036A CN114469036A CN202210100178.8A CN202210100178A CN114469036A CN 114469036 A CN114469036 A CN 114469036A CN 202210100178 A CN202210100178 A CN 202210100178A CN 114469036 A CN114469036 A CN 114469036A
- Authority
- CN
- China
- Prior art keywords
- heart rate
- image
- calculation
- region
- rate data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000012544 monitoring process Methods 0.000 title claims abstract description 55
- 238000004364 calculation method Methods 0.000 claims abstract description 67
- 239000011159 matrix material Substances 0.000 claims abstract description 41
- 238000001514 detection method Methods 0.000 claims abstract description 35
- 238000004458 analytical method Methods 0.000 claims abstract description 25
- 230000036541 health Effects 0.000 claims abstract description 13
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 238000012545 processing Methods 0.000 claims description 10
- 238000000354 decomposition reaction Methods 0.000 claims description 6
- 230000001815 facial effect Effects 0.000 claims description 6
- 238000012549 training Methods 0.000 claims description 4
- 238000000926 separation method Methods 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 2
- 206010007559 Cardiac failure congestive Diseases 0.000 abstract description 2
- 230000006806 disease prevention Effects 0.000 abstract description 2
- 210000001061 forehead Anatomy 0.000 description 10
- 230000008859 change Effects 0.000 description 9
- 230000008569 process Effects 0.000 description 9
- 208000024172 Cardiovascular disease Diseases 0.000 description 7
- 230000007774 longterm Effects 0.000 description 5
- 238000001228 spectrum Methods 0.000 description 5
- 239000008280 blood Substances 0.000 description 4
- 210000004369 blood Anatomy 0.000 description 4
- 239000003814 drug Substances 0.000 description 4
- 230000000694 effects Effects 0.000 description 4
- 238000010606 normalization Methods 0.000 description 4
- 208000017667 Chronic Disease Diseases 0.000 description 3
- 230000036772 blood pressure Effects 0.000 description 3
- 230000008030 elimination Effects 0.000 description 3
- 238000003379 elimination reaction Methods 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 238000009532 heart rate measurement Methods 0.000 description 3
- 238000011321 prophylaxis Methods 0.000 description 3
- 230000002829 reductive effect Effects 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 201000010099 disease Diseases 0.000 description 2
- 208000037265 diseases, disorders, signs and symptoms Diseases 0.000 description 2
- 230000002526 effect on cardiovascular system Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000001914 filtration Methods 0.000 description 2
- 230000000670 limiting effect Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 238000013186 photoplethysmography Methods 0.000 description 2
- 230000002265 prevention Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000029058 respiratory gaseous exchange Effects 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 238000011282 treatment Methods 0.000 description 2
- 101100356682 Caenorhabditis elegans rho-1 gene Proteins 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000010521 absorption reaction Methods 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000004872 arterial blood pressure Effects 0.000 description 1
- 210000001367 artery Anatomy 0.000 description 1
- QVGXLLKOCUKJST-UHFFFAOYSA-N atomic oxygen Chemical compound [O] QVGXLLKOCUKJST-UHFFFAOYSA-N 0.000 description 1
- 238000009530 blood pressure measurement Methods 0.000 description 1
- 210000004204 blood vessel Anatomy 0.000 description 1
- 230000036760 body temperature Effects 0.000 description 1
- 230000036471 bradycardia Effects 0.000 description 1
- 230000000747 cardiac effect Effects 0.000 description 1
- 230000036996 cardiovascular health Effects 0.000 description 1
- 210000000748 cardiovascular system Anatomy 0.000 description 1
- 210000000038 chest Anatomy 0.000 description 1
- 238000010219 correlation analysis Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 229940079593 drug Drugs 0.000 description 1
- 238000013399 early diagnosis Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000004217 heart function Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 208000015181 infectious disease Diseases 0.000 description 1
- 208000014674 injury Diseases 0.000 description 1
- 230000002452 interceptive effect Effects 0.000 description 1
- 210000004072 lung Anatomy 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 229910052760 oxygen Inorganic materials 0.000 description 1
- 239000001301 oxygen Substances 0.000 description 1
- 230000035778 pathophysiological process Effects 0.000 description 1
- 230000001766 physiological effect Effects 0.000 description 1
- 230000003449 preventive effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 230000036387 respiratory rate Effects 0.000 description 1
- 230000002441 reversible effect Effects 0.000 description 1
- 230000033764 rhythmic process Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000008733 trauma Effects 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
- A61B5/02416—Detecting, measuring or recording pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Cardiology (AREA)
- Biomedical Technology (AREA)
- Medical Informatics (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Engineering & Computer Science (AREA)
- Physiology (AREA)
- Heart & Thoracic Surgery (AREA)
- Physics & Mathematics (AREA)
- Molecular Biology (AREA)
- Surgery (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Public Health (AREA)
- Veterinary Medicine (AREA)
- Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
Abstract
The invention discloses a remote heart rate monitoring method and system based on video images, wherein the monitoring method comprises the following steps: acquiring a face image of a monitored object, and preprocessing the face image to obtain an analysis image; extracting a set area from the analysis image, and performing mean value calculation on the image information of the set area to obtain a heart rate calculation queue matrix; calculating a queue matrix according to the heart rate to obtain heart rate data of the set area; selecting different set areas for calculation, and calculating according to the weight value of each set area to obtain the heart rate data monitored at this time; and predicting the future health condition by combining the heart rate data monitored this time and the historical heart rate data of the monitored object. According to the scheme, heart rate monitoring is carried out on the detection object based on the image, the database is established, weak heart rate signals are extracted from complex background signals, the influence of noise signals is effectively removed, the prediction result is reliable, and a reference basis is provided for disease prevention.
Description
Technical Field
The invention relates to the technical field of heart rate monitoring, in particular to a remote heart rate monitoring method and system based on video images.
Background
With the rapid development of social economy, the change of national life style, especially the aging of population and the acceleration of urbanization process, the unhealthy life style of residents is increasingly prominent, the influence of cardiovascular disease risk factors on the health of residents is more obvious, and the cardiovascular disease incidence rate is still continuously increased. In the Chinese cardiovascular health and disease report 2020, it is pointed out that cardiovascular diseases in China are the first causes of total death of urban and rural residents, and the economic burden of residents and society due to cardiovascular diseases is increasing day by day. Because the monitoring of the physiological parameters, especially the monitoring of the change of the key physiological parameters reflecting the functions of the cardiovascular system, can realize the early discovery and early treatment of the cardiovascular diseases. Heart rate is one of the important physical parameters reflecting the health condition of human body, and people who have too fast heart rate for a long time have larger potential risks of cardiovascular diseases than people who have slow heart rate. It is seen that long term dynamic monitoring of heart rate is a useful measure for monitoring the health condition of a person.
The most common examination method of present hospital is to adopt Electrocardiograph (ECG) to examine patient's cardiac function, and the principle of electrocardiogram is because breathing, physiological activities such as heartbeat can make human thorax constantly fluctuate, and biological radar is through catching this fluctuation to convert it into voltage signal, carry on the processing on the chip, thereby isolate all important physiological information relevant with breathing, rhythm of the heart. The arterial pressure method is a method in which an air pump is used to pressurize an artery, and a pressure sensor is used to monitor a pressure signal of a pulse beat to obtain a heart rate, and is generally used in conjunction with blood pressure measurement. However, this method requires wearing a heart rate monitor, which is complicated. The method needs complicated equipment, needs operation of professional medical personnel, has strict requirements on the position and the posture of the measured person in the measuring process, and is not suitable for daily monitoring.
In recent years, heart rate detection methods based on a contact photoplethysmography (PPG) technology include a pulse oximeter and an electronic blood pressure pulse meter to measure the heart rate, and the two methods measure the number of times of effective pulse generation of the heart per minute, namely the pulse rate; the pulse rate and heart rate of a healthy person are generally equal in value, but some special cases may lead to inconsistent pulse rate and heart rate. This method can affect the treatment of a newborn, long-term continuous monitoring, home medicine, long-term seizure monitoring, burn or trauma patient monitoring, especially during an epidemic, tedious donning procedures and excessive exposure to contact, as well as increase the workload of medical personnel and risk of infection.
At the time of a hospital visit, a doctor typically needs to examine 4 vital signs of a patient: heart rate, blood pressure, respiratory rate, and body temperature. The detection for only 1 or 2 times every year is not enough, the continuous heart rate monitoring can establish a first preventive line for prevention, abnormal heart activity can be found in time through continuous and accurate electrocardio monitoring and analysis, and precious diagnosis and intervention time is won for patients. The long-term collection and monitoring of health data can only be accomplished through a non-invasive, very easy to use and widely available data collection process.
Because of the blood vessels on the face, the blood has a certain absorption capacity to natural light. The weak change is extracted through a camera to form a facial blood spectrum. The heart beats, the lung breathes, and blood pressure is high or low, and blood oxygen content all can influence the change of facial blood spectrum, through the analysis to the change, calculates human important physiological signal. Therefore, the non-contact heart rate measurement is carried out at the same time, the non-contact measurement can get rid of the burden and discomfort caused by various contact sensors, and the remote heart rate monitoring can be realized under the condition of not interfering the normal life of people. The research on the non-contact heart rate monitoring method has very important significance for early diagnosis and effective prevention of cardiovascular diseases.
In patent CN110547783A, a non-contact heart rate detection method, system, device, and storage medium are provided, in which a pixel value matrix of R, G, B channels of a face area is preset by obtaining multiple frames of face images, then a channel with the largest energy is selected as a target channel through fast fourier transform, then a signal is amplified through euler images, a cardiovascular pulse wave sequence is obtained, a frequency waveform of the cardiovascular pulse wave sequence is analyzed, and a frequency corresponding to the largest peak in the frequency waveform is selected as a target frequency. However, the pulse waveform contains many noise signals, and the method of denoising and blind source extraction is not provided in the invention.
The invention patent CN110236511A provides a video-based noninvasive heart rate measurement method, which adopts an improved Euler image amplification method to amplify weak signals of a face, adopts MTCNN to detect the face, and combines G signals as input signals for heart rate calculation. The MTCNN is used to improve the accuracy of face detection and thus the accuracy of heart rate calculation, but there is no detailed processing of noise processing and no correlation of historical heart rate data for the person for long-term monitoring.
The invention patent CN113408508B provides a non-contact heart rate measurement method based on a Transformer, which is characterized in that a human face key point model is obtained to obtain a human face interesting region image sequence, and then a Transformer model is trained to obtain a heart rate sequence. However, the method does not provide a noise processing method and a blind source signal extraction method, and meanwhile, the accuracy completely depends on training samples, but due to the particularity of the heart rate signals, the samples collected in general are insufficient, people in different age stages and different heart rate stages cannot be necessarily educed, and the learned model is easy to be under-fitted.
Prophylactic treatment of chronic diseases can reduce the total dose of required drugs and associated side effects, and reduce mortality and morbidity. In general, once the earliest clinical symptoms are found, prophylactic treatment should be initiated or enhanced immediately to prevent the progression and worsening of the clinical episode and to halt and reverse the pathophysiological processes. Thus, the ability to accurately monitor pre-onset indicators increases the effectiveness of prophylactic treatment of chronic diseases.
When a camera is used for sampling a physiological signal of a human body, the collected signal contains a background signal, a heart rate signal and interference noise, and how to remove the interference noise, it becomes very difficult to extract a weak heart rate signal from the background signal.
Therefore, an effective heart rate detection system and method based on video images is needed.
Disclosure of Invention
In order to overcome the defects in the prior art, the invention provides a remote heart rate monitoring method and system based on a video image, which can accelerate the human face detection speed and improve the system performance.
The above effects are specifically realized by the following technical scheme:
first, the present application provides a remote heart rate monitoring method based on video images, the monitoring method includes:
s1, acquiring a face image of the monitored object, and preprocessing the face image to obtain an analysis image;
s2, extracting a set area from the analysis image, and carrying out mean value calculation on the image information of the set area to obtain a heart rate calculation queue matrix;
s3, obtaining heart rate data of the set area according to the heart rate calculation queue matrix;
s4, selecting different set areas for calculation, and calculating according to the weight value of each set area to obtain the heart rate data monitored at this time;
and S5, predicting the future health condition by combining the heart rate data monitored this time and the historical heart rate data of the monitored object.
Further, the heart rate calculation queue matrix is:
X＝[x(1)，x(2)，...，x(t)，...x(N2)]Twherein x (t), t 1,2,., N2, mean of data obtained after the image information of the set region is subjected to averaging processing, N2 is a time interval between two face detections, and:
wherein i ', j' is a variable, i 'represents the abscissa position of a certain point in the region-of-interest image, j' represents the ordinate position of a certain point in the region-of-interest image, W2 is the width of the color image of the local region to which the set region belongs, and H2 is the height of the color image of the local region to which the set region belongs; mask [ ] is a region mask image value used to demarcate whether the region is a set region, and ROI2[ i ', j' ] is the image information of the region.
Further, the calculating the queue matrix according to the heart rate to obtain the heart rate data of the current monitoring specifically includes:
s3.1, removing outliers in the heart rate calculation queue matrix;
s3.2, normalizing the data in the heart rate calculation queue matrix;
s3.3, eliminating a trend item in the heart rate calculation queue matrix based on a smooth prior analysis method;
s3.4, performing blind source separation processing on the heart rate calculation queue matrix based on a singular value decomposition method to obtain a denoised heart rate signal;
and S3.5, calculating the arithmetic mean value of the denoised heart rate signals in the step S3.4, then carrying out FFT (fast Fourier transform), and finally carrying out peak frequency calculation to obtain the heart rate data of the set area.
As a preferred embodiment of the present application, the heart rate data formula of the set area is:
where fre is the peak frequency and fps is the regularization parameter in the smooth prior analysis.
Further, the heart rate data monitored this time obtained by calculation according to the weight values of the set regions is as follows:
Heartrate＝τ1*Heartrate1+τ2*Heartrate2
where τ 1 is the weight of the first setting region, τ 2 is the weight of the second setting region, and τ 1+ τ 2 is equal to 1.
In a preferred embodiment of the present application, the setting region is a region with small light interference.
As a preferred embodiment of the present application, the detection method further comprises:
extracting human face features according to the analysis image obtained in the step S1, judging whether the current monitoring object exists in the database, and if not, storing the human face image of the current monitoring object in the database.
Further, step S5 is to use an LSTM training heart rate monitoring method to predict that the difference between the future data and the historical mean value is greater than a set threshold, and issue an early warning to remind the monitoring subject or medical staff to take measures.
The application also provides a remote heart rate monitoring system based on video images, the monitoring system includes:
the system comprises a face image acquisition unit, a face image acquisition unit and a monitoring unit, wherein the face image acquisition unit is used for acquiring a face image of a monitored object and preprocessing the face image to obtain an analysis image;
the heart rate calculation queue matrix unit is used for extracting a set area from the analysis image, and performing mean value calculation on image information of the set area to obtain a heart rate calculation queue matrix;
the heart rate data calculation unit is used for obtaining heart rate data of the set area according to the heart rate calculation queue matrix; selecting different set areas for calculation, and calculating according to the weight value of each set area to obtain the heart rate data monitored this time;
and the heart rate data prediction unit is used for predicting the future health condition by combining the heart rate data monitored at this time and the historical heart rate data of the monitored object.
Further, the detection system also comprises a feature recognition unit and a database, wherein the feature recognition unit analyzes the image to extract the face features, judges whether the monitored object exists in the database or not, and stores the face image of the monitored object into the database if the monitored object does not exist.
Compared with the prior art, the invention has the following advantages:
the invention provides a remote heart rate monitoring method and system based on video images, and compared with the prior art, the method and system have the following advantages:
(1) the gray level image is zoomed, so that the human face detection speed is increased, and the system performance is improved;
(2) background signals and interference signals in the heart rate signals are effectively removed, and the heart rate detection precision is improved;
(3) the heart rate data of the user is monitored in real time, and the historical data of the user is reserved so as to judge the heart rate change trend of the user, so that the heart rate monitoring system has guiding significance for preventing and treating chronic diseases, can reduce the dosage of medicine use and side effects of the medicine use, and reduces the mortality and morbidity.
Drawings
In order to more clearly illustrate the technical solution of the present invention, the drawings needed to be used in the present invention will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art that other drawings can be obtained based on these drawings without inventive labor.
FIG. 1 is a schematic diagram of a video image-based remote heart rate monitoring system architecture according to the present invention;
fig. 2 is a flow chart of a remote heart rate monitoring method based on video images.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
Example 1
The embodiment provides a remote heart rate monitoring system based on video images, a block diagram of the system structure is shown in fig. 1, and the monitoring system comprises:
the system comprises a face image acquisition unit, a face image acquisition unit and a monitoring unit, wherein the face image acquisition unit is used for acquiring a face image of a monitored object and preprocessing the face image to obtain an analysis image;
the heart rate calculation queue matrix unit is used for extracting a set area from the analysis image, and performing mean value calculation on image information of the set area to obtain a heart rate calculation queue matrix;
the heart rate data calculation unit is used for obtaining heart rate data of the set area according to the heart rate calculation queue matrix; selecting different set areas for calculation, and calculating according to the weight value of each set area to obtain the heart rate data monitored at this time;
and the heart rate data prediction unit is used for predicting the future health condition by combining the heart rate data monitored at this time and the historical heart rate data of the monitored object.
Further, the detection system also comprises a feature recognition unit and a database, wherein the feature recognition unit analyzes the image to extract the face features, judges whether the monitored object exists in the database or not, and stores the face image of the monitored object into the database if the monitored object does not exist.
Example 2
Based on the above system, the present application also relates to a remote heart rate monitoring method based on video images, as shown in fig. 2, including the following steps:
face detection
(1) Acquiring a current frame image;
(2) converting the color image into a gray image;
face detection and tracking is performed on a grayscale image, so the color video image is first converted to a grayscale image. i, j is a variable, i represents the abscissa position of a certain point in the color image, j represents the ordinate position of a certain point in the color image, R [ i, j ] represents the horizontal position as i, the vertical position as the red (R) component value of the RGB color space at the pixel point of j, G [ i, j ] represents the horizontal position as i, the vertical position as the green (G) component value of the RGB color space at the pixel point of j, B [ i, j ] represents the horizontal position as i, the vertical position as the blue (B) component value of the RGB color space at the pixel point of j, Gray [ i, j ] represents the Gray level position as i, the vertical position as the value at the pixel point of j, and the method for transforming the color space into the Gray scale space comprises:
Gray[i，j]＝0.29*R[i，j]+0.587*G[i，j]+0.114*B[i，j],i＝1，2，...，W1,j＝1，2，...，H1；
where W1 is the width of the color image, H1 is the height of the color image, and G [ i, j ] is ∈ [0, 255 ].
The conversion of the color image into the gray image is to increase the speed of face detection, and no steps are necessary, and it is within the scope of the present invention to perform face detection directly without converting into the gray image.
(3) Scaling the gray level image;
in order to improve the detection efficiency, the grayscale image is reduced, and in the embodiment of the present invention, the image is reduced to 1/4 of the original image in an equal ratio. The current image scaling method is mature and is not in the research scope of the invention. The methods mentioned below are all performed on the reduced grayscale image. When the face template is 0 or reaches a fixed detection time threshold value N0, entering the step (4), otherwise, entering the step (5);
n0 represents the time interval of face detection, and in order to increase the detection speed, face detection is not performed every frame, but every N0 frames.
The scaling of the gray image is to increase the speed of face detection, and no steps are necessary, and it is within the scope of the invention to directly perform face detection without scaling.
(4) Carrying out face detection;
1) loading a caffe model by using an dnn module of opencv to perform face detection to obtain the proportion of the face position;
2) obtaining face position coordinates in the video image according to the proportion of the face positions detected in the step 1), namely, adopting a form of [ xN, yN ] to represent the position of the face where the same coordinates on the face are located, in the embodiment, using [ x1, y1] to represent the coordinates of the upper left corner of the face obtained by detection, and using [ x2, y2] to represent the coordinates of the lower right corner of the face obtained by detection; the upper right corner, the lower left corner, the middle position and the like can be additionally set according to actual calculation.
3) Representing the number of the detected human face frames by using a frame1, setting the number of the detected human face frames 1 as 0 when no human face is detected, returning to the step (1), otherwise, acquiring a human face template, adding 1 to the number of the detected human face frames 1, and entering the step (5);
(5) performing face tracking
1) Extracting feature points of the face template by using goodffeatureToTrack in opencv, and finding out corner points required by optical flow estimation;
2) using calcOpticalFlowPyrLK in opencv to realize KLT-based characteristic point tracking and obtaining the corner position after optical flow detection;
3) and (4) when the tracking fails, entering the step (4), otherwise, adding 1 to the frame number frame1 of the detected face, and entering the step 2. When the frame1 is more than N1, the method enters the step (4) to carry out face detection again.
Wherein, 0 represents how often to perform face detection again. In the embodiment of the present invention, N0 is 1-fps, frame1 indicates the number of frames in which a face is detected, and fps indicates the frame rate of a video;
2. face recognition
(1) Human face feature point extraction
Loading a pytorech model by using an dnn model based on opencv to extract human face features;
(2) feature comparison
Comparing the feature array of the currently detected face image with the feature arrays of the face images in all face databases to obtain a plurality of verification feature distances, wherein the calculation method of the feature distances comprises the following steps:
in the formula, d (n) is a feature distance between the current face image and the face image of the nth face in the face database, p (k) is a k-dimensional feature value of the current face image, q (k) is a k-dimensional feature value of the face image in the face database, and k is a variable and represents a dimension number. D (n) is a floating point number with the value range of [0,3 ].
(3) Similarity calculation
Obtaining corresponding similarity according to the characteristic distance between the current frame face image and the face image in the face database, and using sim (n) to represent that when the similarity between the current frame face image and the nth individual face image in the face database is calculated, the calculating method of the similarity is as follows:
(4) determination
After calculating the similarity between the current frame face image and the face images in all the face databases, taking the maximum similarity
And when the maximum similarity Max _ Sim is larger than TH1, judging that the current frame face image is the person in the current face database, finishing face recognition, otherwise, indicating that the current frame face is a new user, and establishing a face database for the user. The process of creating the face library is not within the scope of the present invention, and therefore, will not be described in detail.
Where N1 represents the number of users in the database, TH1 represents the face similarity threshold, and TH1 is 85 in the embodiment of the present invention.
The process is to establish data archive for each user so as to facilitate heart rate tracking monitoring of the user, and after the data volume of the user is increased, relatively more accurate data prediction can be obtained, and the heart rate condition (namely the health state) of the user can be more accurately known; of course, if a user does not use the system before, historical data can be obtained in a form of directly importing data and used as a basis for prediction.
3. Extracting regions of interest
After the acquired user face image data is preprocessed, an 'interested region' needs to be determined, wherein the interested region is represented by an interested region mask image in a figure with relatively small change of brightness and darkness in the expression change process, and the change rate of the brightness and darkness is relatively different in the face, so that the 'interested region' can be determined according to specific conditions. According to the scheme, the heart rate of a user is calculated in an interested area, in the embodiment, two areas, namely a forehead area and a cheek area, are selected as the interested area, and then the heart rate values calculated by the two areas are averaged; other regions can be selected as the region of interest and assigned with differential weights to calculate the heart rate, which is not limited in the manner.
(1) And selecting two areas, namely a forehead area and a cheek area, as the interested areas, and acquiring the positions of the interested areas according to the results of face detection and tracking.
The coordinates of the upper left corner of the forehead area are represented by [ x3, y3], the coordinates of the upper right corner are represented by [ x4, y4], and the calculation method is as follows:
the upper left-hand coordinates of the forehead area are represented by [ x3, y3], the upper right-hand coordinates are represented by [ x4, y4], and the calculation method is as follows:
x3＝x1+δ1*(x2-x1)
y3＝y1+δ2*(y2-y1)
x4＝x1+δ3*(x2-x1)
y4＝y1+δ4*(y2-y1)
the width W2 of the extracted forehead region is x4-x3, and H2 is y4-y 3.
Wherein δ 1, δ 2, δ 3, δ 4 are determined according to the proportion of facial features, and in the embodiment of the invention, δ 5 is 0.3, δ 6 is 0.5, δ 7 is 0.7, and δ 8 is 0.65.
The coordinates of the upper left corner of the cheek area are represented by [ x5, y5] and the coordinates of the upper right corner are represented by [ x6, y6], and the calculation method is as follows:
x5＝x1+δ5*(x2-x1)
y5＝y1+δ6*(y2-y1)
x6＝x1+δ7*(x2-x1)
y6＝y1+δ8*(y2-y1)
the width W3 ═ x6-x5, H3 ═ y6-y5 of the extracted cheek regions.
Wherein δ 5, δ 6, δ 7, δ 8 are determined according to the proportion of facial features, δ 5 is 0.3, δ 6 is 0.5, δ 7 is 0.7, δ 8 is 0.65 in the embodiment of the invention.
The method for calculating the heart rate in the forehead area will be described below by taking the forehead area as an example.
(2) The highlight region is detected and removed.
Setting a mask image of the non-interested region as 0, namely setting mask [ i, j ] as 0;
wherein i, j are variables, x3≤i≤x4，y3J is not less than j and not more than y4, Th2 is highlight detection threshold, Imax[i，j]＝max(R[i，j]，G[i，j]，B[i，j]) The maximum value of RGB three channels of a pixel point with the abscissa of I and the ordinate of j in the color image is represented, Imin[i，j]＝min(R[i，j]，G[i，j]，B[i，j]) And the minimum value of RGB three channels of the pixel point with the abscissa of i and the ordinate of j in the color image is represented. In the region of interest when
When the current pixel point is the highlight pixel point, TH2 is the threshold value of the highlight pixel point, mask [ i, j]0, remove from the region of interest, otherwise mask i, j]＝1。
(3) Extracting information of the region of interest;
the extracted information of the region of interest may be represented as follows:
ROI1[i′，j′]＝S[x4+i′，y4+j′]，1≤i′≤W2，1≤j′≤H2
wherein i ', j' is a variable, i 'represents the abscissa position of a certain point in the region-of-interest image, and j' represents the ordinate position of a certain point in the region-of-interest image.
σ (-) denotes the calculated standard deviation.
(4) Euler amplifying the signal to obtain an amplified region of interest ROI2[ i ', j' ];
the calculation of Euler video amplification is realized by a mature technology at present and is out of the research range of the patent.
(5) And (5) solving the mean value mean of the region of interest.
(6) When the frame is less than N2, storing mean in a heart rate calculation queue called as a heart rate time domain pulse signal, and repeating the step 1-3; when parameter is N2, go to step 4; when frame1 > N2, entering step (7); where N2 is how long the data is counted for heart rate calculation. In the embodiment of the present invention, N2 ═ 10 × fps, and this threshold value is related to the frame rate.
(7) Deleting the head data of the heart rate calculation queue, then pressing the currently calculated average value of the region of interest into the tail of the heart rate calculation queue, updating the heart rate statistical time frame2 plus 1, repeating the steps 1-3, and entering the step 4 when the frame2 is N3. Wherein, frame2 is the updated heart rate statistics time, and N3 is the heart rate update time.
4. Heart rate calculation
When the heart rate calculation is entered, the updated heart rate statistics time frame2 is set to 0. The heart rate calculation queue matrix is denoted by X and is a matrix of 1 row and N2 columns, where X ═ X (1), X (2),.. times, X (t),. times.x (N2)]T. Where t is a variable and x (t) represents the value of the t-th element in the heart rate calculation queue.
(1) Outlier removal
Some outliers may appear for the features extracted in the previous step. Deleting such inconsistent data helps achieve better performance.
(2) Normalizing data in a heart rate calculation queue
A normalization method based on the mean value and the variance of original data is adopted, the normalized value is 0-1, and the normalization formula is as follows:
(3) Heart rate trend term elimination based on smooth prior analysis
The heart rate time domain pulse signal is inevitably mixed with noise and various trends, interference of the trends is eliminated, and analysis accuracy is facilitated. Trend term refers to the entire process of time-dependent deviation from baseline, and trend term elimination is an important step in analytical data analysis. When there is an obvious trend term in the cardiac electrical signal without being eliminated, distortion occurs when correlation analysis and power spectral density analysis are performed, which causes the low frequency component to upwarp and even submerge the main frequency component, thereby seriously affecting the processing accuracy. Therefore, in order to improve data quality or process data into a form convenient for data processing, it is necessary to extract or cull the trend item.
In the embodiment of the invention, the adopted trend term based on the smooth prior analysis is eliminated.
The heart rate calculation queue matrix after the elimination of the trend term based on the smoothed prior analysis is represented by X ″, which is [ X "(1), X" (2),. ·, X "(t),. ·, X" (N2)]TThe formula can be expressed as:
in the embodiment of the invention, the 2 nd order differential operator matrix D of X is used2Expressed as follows:
(4) heart rate signal blind source separation based on singular value decomposition
Let the noisy signal X be S + V, S be [ S (2), S (2),. ·, S (N2) ], representing the original signal, V be [ V (1), V (2),. ·, V (N2) ], representing the noisy signal, and N2 representing the length of the signal. Constructing the noise-containing signal X' in the step (3) into a Hankel matrix of m multiplied by n (m is less than or equal to n):
where 1 < N2, m is the embedding dimension and m + N-1 is N2.
1) Singular value decomposition of H-U sigma VTThe matrix of singular values ^ diag (σ 1, σ 2.,. σ l.,. σ r) is obtained, and an operation of retaining only one of the singular values and setting all the remaining singular values to 0 is performed, so that r singular values are obtained. The first singular value is ^ (l) ═ diag (0,. sigma., σ l, 0,. sigma.).
2) Calculating corresponding singular value sigma l according to each obtained singular value
Obtaining a component signal X '(l) of the signal X' corresponding to each singular value through inversion;
3) respectively carrying out FFT (fast Fourier transform) on the r component signals X' (l) obtained in the step 2);
4) normalization;
5) band-pass filtering;
6) and calculating a maximum value and a minimum value to obtain a frequency value f corresponding to the maximum amplitude of each signal component X' (l) after conversion, and obtaining r frequency values which are f (1), f (2), f (r) in sequence.
7) First-order lag differencing is performed on the r frequency values f (1), f (2),.., f (r), resulting in a difference spectrum:
Δf(l)＝f(l)-f(l+1)
wherein l is variable, and l is more than or equal to 1 and less than or equal to r-1.
When the difference spectrum Deltaf (l) > Th3, l is marked as g, which is used as the effective order of noise reduction of singular value decomposition, and g < r. Where Th3 is a threshold value of the difference spectrum.
8) According to the obtained effective order g, the first g effective singular values of the matrix Λ are taken, the residual singular values are set to be 0, the singular values diag (rho 1, p2,.. once, rho g, 0,.. once, 0) after noise reduction are obtained, then the reconstruction matrix H 'is obtained according to the inverse process of singular value decomposition, and the denoised heart rate signal X' is obtained through inversion.
(5) Moving average filter
And taking a certain number of the obtained heart rate signals X 'in sequence to make an arithmetic mean value to obtain X'.
(6) FFT transformation
1) Performing FFT on the X 'and transforming the X' into a frequency domain;
2) filtering by a 0.6-4HZ band-pass filter;
3) obtaining a peak frequency fre through maximum and minimum value operation;
(7) calculating heart rate
Calculating the heart rate according to the peak frequency, wherein the heart rate calculation formula of the forehead area is as follows:
according to the steps 3 and 4, heart rate calculation values Heartrate2 of the cheek areas are obtained in sequence, and then weighted calculation is carried out:
Heartrate＝τ1*Heartrate1+τ2*Heartrate2
where τ 1 is a weight of the forehead region, τ 2 is a weight of the face region, and τ 1+ τ 2 is 1. Because the interference on the forehead area is the smallest, the weight is larger, and in the embodiment of the present invention, τ 1 is 0.6, and τ 2 is 0.4.
(8) Preserving heart rate data
And comparing the calculated heart rate with the last historical heart rate value of the person, deleting the heart rate data possibly under the influence of noise when the difference exceeds a certain threshold value which is larger than the threshold value, otherwise, storing the heart rate data of the person, and further removing the artifact.
5. Predicting future health conditions based on stored historical heart rate data
And when the stored historical heart rate data reaches a certain amount, forming a time sequence by the heart rate data of each time period, training a heart rate prediction algorithm by using the LSTM, verifying and testing, and finally predicting future heart rate data by using a trained LSTM prediction heart rate model. And when the difference between the predicted future data and the historical mean value is larger than a certain threshold value, giving out an early warning to remind the person or the medical care personnel of paying attention.
The method has the advantages of higher processing speed and higher calculation precision, meanwhile, historical data of the user can be used for predicting the future health condition of the user, when the difference between the predicted heart rate data and the historical data at a certain time is larger than a certain threshold value, the user is early warned, heart-related diseases can be found in advance, and the method has a very high guiding significance for disease prevention.
The present invention is not limited to the above-described alternative embodiments, and various other forms of products can be obtained by anyone in light of the present invention. The above detailed description should not be taken as limiting the scope of the invention, which is defined in the claims, and which the description is intended to be interpreted accordingly.
The above description is only exemplary of the present application and should not be taken as limiting the present application, as any modification, equivalent replacement, or improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.
Claims (10)
1. A remote heart rate monitoring method based on video images is characterized by comprising the following steps:
s1, acquiring a face image of the monitored object, and preprocessing the face image to obtain an analysis image;
s2, extracting a set area from the analysis image, and carrying out mean value calculation on the image information of the set area to obtain a heart rate calculation queue matrix;
s3, obtaining heart rate data of the set area according to the heart rate calculation queue matrix;
s4, selecting different set areas for calculation, and calculating according to the weight value of each set area to obtain the heart rate data monitored at this time;
and S5, predicting the future health condition by combining the heart rate data monitored this time and the historical heart rate data of the monitored object.
2. The video-image-based remote heart rate monitoring method according to claim 1, wherein the heart rate calculation queue matrix is:
X＝[x(1)，x(2)，...，x(t)，...x(N2)]Twherein x (t), t 1,2,., N2, mean of data obtained after the image information of the set region is subjected to averaging processing, N2 is a time interval between two face detections, and:
wherein i ', j' is a variable, i 'represents the abscissa position of a certain point in the region-of-interest image, j' represents the ordinate position of a certain point in the region-of-interest image, W2 is the width of the color image of the local region to which the set region belongs, and H2 is the height of the color image of the local region to which the set region belongs; mask [ ] is a region mask image value used to demarcate whether the region is a set region, and ROI2[ i ', j' ] is the image information of the region.
3. The video image-based remote heart rate monitoring method according to claim 1, wherein the step of obtaining the heart rate data of the current monitoring by calculating the queue matrix according to the heart rate specifically comprises:
s3.1, removing outliers in the heart rate calculation queue matrix;
s3.2, normalizing the data in the heart rate calculation queue matrix;
s3.3, eliminating a trend item in the heart rate calculation queue matrix based on a smooth prior analysis method;
s3.4, performing blind source separation processing on the heart rate calculation queue matrix based on a singular value decomposition method to obtain a denoised heart rate signal;
and S3.5, calculating the arithmetic mean value of the denoised heart rate signals in the step S3.4, then carrying out FFT (fast Fourier transform), and finally carrying out peak frequency calculation to obtain the heart rate data of the set area.
4. A method according to any one of claims 1 to 3, wherein the heart rate data formula for the predetermined area is:
where fre is the peak frequency and fps is the regularization parameter in the smooth prior analysis.
5. The video image-based remote heart rate monitoring method according to claim 4, wherein the heart rate data of the current monitoring obtained by calculating the weight values of the set regions is:
Heartrate＝τ1*Heartrate1+τ2*Heartrate2
where τ 1 is a weight of the first setting region, τ 2 is a weight of the second setting region, and τ 1+ τ 2 is equal to 1.
6. The method according to claim 1, wherein the defined area is an area with little light interference.
7. The method of claim 1, wherein the detecting further comprises:
extracting human face features according to the analysis image obtained in the step S1, judging whether the current monitoring object exists in the database, and if not, storing the human face image of the current monitoring object in the database.
8. The method according to claim 1, wherein step S5 is to use LSTM training heart rate monitoring method to predict that the future data and the historical mean difference are greater than a predetermined threshold, and issue an early warning to remind the monitoring subject or medical staff to take measures.
9. A video image-based remote heart rate monitoring system, the monitoring system comprising:
the system comprises a face image acquisition unit, a face image acquisition unit and a monitoring unit, wherein the face image acquisition unit is used for acquiring a face image of a monitored object and preprocessing the face image to obtain an analysis image;
the heart rate calculation queue matrix unit is used for extracting a set area from the analysis image, and performing mean value calculation on image information of the set area to obtain a heart rate calculation queue matrix;
the heart rate data calculation unit is used for obtaining heart rate data of the set area according to the heart rate calculation queue matrix; selecting different set areas for calculation, and calculating according to the weight value of each set area to obtain the heart rate data monitored at this time;
and the heart rate data prediction unit is used for predicting the future health condition by combining the heart rate data monitored at this time and the historical heart rate data of the monitored object.
10. The video image-based remote heart rate monitoring system according to claim 9, wherein the detection system further comprises a feature recognition unit and a database, the feature recognition unit analyzes the image to extract facial features, determines whether the current monitored object exists in the database, and stores the facial image of the current monitored object into the database if the current monitored object does not exist.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210100178.8A CN114469036A (en) | 2022-01-27 | 2022-01-27 | Remote heart rate monitoring method and system based on video images |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210100178.8A CN114469036A (en) | 2022-01-27 | 2022-01-27 | Remote heart rate monitoring method and system based on video images |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114469036A true CN114469036A (en) | 2022-05-13 |
Family
ID=81476952
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210100178.8A Pending CN114469036A (en) | 2022-01-27 | 2022-01-27 | Remote heart rate monitoring method and system based on video images |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114469036A (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102499664A (en) * | 2011-10-24 | 2012-06-20 | 西双版纳大渡云海生物科技发展有限公司 | Video-image-based method and system for detecting non-contact vital sign |
| CN105147274A (en) * | 2015-08-04 | 2015-12-16 | 河北工业大学 | Method for extracting heart rate from visible spectrum section face video signal |
| CN105996993A (en) * | 2016-04-29 | 2016-10-12 | 南京理工大学 | System and method for intelligent video monitoring of vital signs |
| CN108197595A (en) * | 2018-01-23 | 2018-06-22 | 京东方科技集团股份有限公司 | A kind of method, apparatus, storage medium and computer for obtaining evaluation information |
| CN110647815A (en) * | 2019-08-25 | 2020-01-03 | 上海贝瑞电子科技有限公司 | Non-contact heart rate measurement method and system based on face video image |
| CN110866498A (en) * | 2019-11-15 | 2020-03-06 | 北京华宇信息技术有限公司 | Portable heart rate monitoring device and heart rate monitoring method thereof |
| CN111248890A (en) * | 2020-01-20 | 2020-06-09 | 深圳大学 | Non-contact newborn heart rate monitoring method and system based on facial video |
| CN111938622A (en) * | 2020-07-16 | 2020-11-17 | 启航汽车有限公司 | Heart rate detection method, device and system and readable storage medium |
| WO2021017307A1 (en) * | 2019-07-31 | 2021-02-04 | 平安科技（深圳）有限公司 | Non-contact heart rate measurement method, system, device, and storage medium |
-
2022
- 2022-01-27 CN CN202210100178.8A patent/CN114469036A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102499664A (en) * | 2011-10-24 | 2012-06-20 | 西双版纳大渡云海生物科技发展有限公司 | Video-image-based method and system for detecting non-contact vital sign |
| CN105147274A (en) * | 2015-08-04 | 2015-12-16 | 河北工业大学 | Method for extracting heart rate from visible spectrum section face video signal |
| CN105996993A (en) * | 2016-04-29 | 2016-10-12 | 南京理工大学 | System and method for intelligent video monitoring of vital signs |
| CN108197595A (en) * | 2018-01-23 | 2018-06-22 | 京东方科技集团股份有限公司 | A kind of method, apparatus, storage medium and computer for obtaining evaluation information |
| WO2021017307A1 (en) * | 2019-07-31 | 2021-02-04 | 平安科技（深圳）有限公司 | Non-contact heart rate measurement method, system, device, and storage medium |
| CN110647815A (en) * | 2019-08-25 | 2020-01-03 | 上海贝瑞电子科技有限公司 | Non-contact heart rate measurement method and system based on face video image |
| CN110866498A (en) * | 2019-11-15 | 2020-03-06 | 北京华宇信息技术有限公司 | Portable heart rate monitoring device and heart rate monitoring method thereof |
| CN111248890A (en) * | 2020-01-20 | 2020-06-09 | 深圳大学 | Non-contact newborn heart rate monitoring method and system based on facial video |
| CN111938622A (en) * | 2020-07-16 | 2020-11-17 | 启航汽车有限公司 | Heart rate detection method, device and system and readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | The obf database: A large face video database for remote physiological signal measurement and atrial fibrillation detection | |
| US11445983B2 (en) | Non-invasive determination of disease states | |
| Sanyal et al. | Algorithms for monitoring heart rate and respiratory rate from the video of a user’s face | |
| Oweis et al. | QRS detection and heart rate variability analysis: A survey | |
| WO2019161609A1 (en) | Method for analyzing multi-parameter monitoring data and multi-parameter monitor | |
| CN111728602A (en) | Non-contact blood pressure measuring device based on PPG | |
| Fan et al. | Non-contact remote estimation of cardiovascular parameters | |
| Zhou et al. | The noninvasive blood pressure measurement based on facial images processing | |
| CN112294282A (en) | Self-calibration method of emotion detection device based on RPPG | |
| Chen et al. | Modulation model of the photoplethysmography signal for vital sign extraction | |
| CN114387479A (en) | Non-contact heart rate measurement method and system based on face video | |
| Hu et al. | Study on Real-Time Heart Rate Detection Based on Multi-People. | |
| CN117153405A (en) | Alcoholism risk assessment and early warning method combined with alcohol concentration detection | |
| Suleman et al. | Respiratory Events Estimation From PPG Signals Using a Simple Peak Detection Algorithm | |
| Sahin et al. | Non-contact heart rate monitoring from face video utilizing color intensity | |
| Simjanoska et al. | ECG-derived Blood Pressure Classification using Complexity Analysis-based Machine Learning. | |
| Panigrahi et al. | Video-based HR measurement using adaptive facial regions with multiple color spaces | |
| Liu et al. | Lightweight and interpretable convolutional neural network for real-time heart rate monitoring using low-cost video camera under realistic conditions | |
| CN114271800B (en) | Non-invasive continuous blood pressure monitoring method and application in office environment | |
| CN114469036A (en) | Remote heart rate monitoring method and system based on video images | |
| AV et al. | Non-contact heart rate monitoring using machine learning | |
| Hwang et al. | Phase-shifted remote photoplethysmography for estimating heart rate and blood pressure from facial video | |
| Hassan et al. | A real-time non-contact heart rate measurement based on imaging photoplethysmography (ippg)-power spectral density (psd) | |
| Balaraman et al. | Recent Innovations and Improvements in Remote Heart Rate and Heart Disease Measuring Methods Using RGB Camera | |
| Liao et al. | Camera-Based Heart Rate Variability for Estimating the Maturity of Neonatal Autonomic Nervous System |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220824 Address after: 214101 No. 16, Chunlei Road, Xishan Economic and Technological Development Zone, Wuxi City, Jiangsu Province Applicant after: Wuxi Boao Maya Medical Technology Co.,Ltd. Applicant after: Chengdu Maya Lightyear Technology Co.,Ltd. Address before: 214101 No. 16, Chunlei Road, Xishan Economic and Technological Development Zone, Wuxi City, Jiangsu Province Applicant before: Wuxi Boao Maya Medical Technology Co.,Ltd. |