CN115376179A - Non-contact heart rate measuring method and system - Google Patents
Non-contact heart rate measuring method and system Download PDFInfo
- Publication number
- CN115376179A CN115376179A CN202210416920.6A CN202210416920A CN115376179A CN 115376179 A CN115376179 A CN 115376179A CN 202210416920 A CN202210416920 A CN 202210416920A CN 115376179 A CN115376179 A CN 115376179A
- Authority
- CN
- China
- Prior art keywords
- signal
- heart rate
- modal
- face
- algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
Landscapes
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Engineering & Computer Science (AREA)
- Medical Informatics (AREA)
- Surgery (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Veterinary Medicine (AREA)
- Biomedical Technology (AREA)
- Heart & Thoracic Surgery (AREA)
- Public Health (AREA)
- Molecular Biology (AREA)
- Physics & Mathematics (AREA)
- Animal Behavior & Ethology (AREA)
- General Health & Medical Sciences (AREA)
- Cardiology (AREA)
- Physiology (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Psychiatry (AREA)
- Signal Processing (AREA)
- Measuring And Recording Apparatus For Diagnosis (AREA)
Abstract
The invention provides a non-contact heart rate measuring method and a non-contact heart rate measuring system, which belong to the technical field of physiological parameter detection equipment, and are used for acquiring multi-frame face images of a person to be measured; determining a human face interesting region in each frame of human face image; accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal; acquiring a chrominance signal based on the IPPG signal; and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal. The invention can separate the intensity, light intensity and color of the pixel value by CHROM algorithm, and eliminate noise; and performing a VMD algorithm on the signals, and decomposing the main signals into different modes by using the VMD algorithm according to the heartbeat frequency characteristics, so that the signal frequency ranges of the modes are not overlapped, and the complete heartbeat signals without harmonic wave residue are separated.
Description
Technical Field
The invention relates to the technical field of physiological parameter detection equipment, in particular to a non-contact heart rate measurement method and system.
Background
Conventional heart rate measurements require examination by the electrocardiographic technique of the chest leads. According to the method, a plurality of electrodes are required to be accurately arranged at the appointed parts of the body, and conductive paste is often required to be injected into the electrodes to enhance the conductivity so as to reduce the contact resistance, so that the detection process is time-consuming and labor-consuming, and the detection cannot be carried out under the motion state of a subject.
The Photoplethysmography (PPG) detects the heart rate, and its principle is: the pulsating blood propagating in the cardiovascular system alters the blood volume in the skin tissue, and the oxygenated blood circulation causes fluctuations in the amount of hemoglobin molecules and proteins, resulting in fluctuations in the optical absorption of the entire spectrum, whose curve over time is the pulse wave, which can be processed to extract the heart rate signal. But the application range of the PPG monitoring area is limited due to the problems that the PPG monitoring area is single in position, needs to be in contact with the skin of a person to be detected and the like.
An Imaging Photoplethysmography (IPPG) is a pulse wave measuring method based on Imaging, and the principle of the technology is that an RGB camera is used for capturing tiny color changes reflected by skin so as to identify the stage of blood circulation and extract a pulse wave signal. However, the imaging type photoplethysmography is sensitive to light and motion, and the heart rate can be accurately measured only under the condition that the light source is sufficient and the resting state is available.
Disclosure of Invention
The invention aims to provide a non-contact heart rate measuring method and a non-contact heart rate measuring system Based on a chroma remote-Based RPPG (Chrom-read only memory) and a Variable Mode Decomposition (VMD) so as to solve at least one technical problem in the background technology. In order to achieve the purpose, the invention adopts the following technical scheme:
in one aspect, the present invention provides a method for non-contact heart rate measurement, comprising:
acquiring a plurality of frames of face images of a person to be detected;
determining a human face interesting region in each frame of human face image;
accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
acquiring a chrominance signal based on the IPPG signal;
and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
Preferably, the acquiring the region of interest of the face includes: and acquiring a face mark point by using the landmark face recognition model, setting a face interesting region, and removing the environment information of the face edge by using a skin color detector.
Preferably, the skin tone detector uses a multi-color space skin tone detection algorithm.
Preferably, the acquiring the chrominance signal includes:
extracting RGB three channels of the IPPG signal, and then carrying out normalization processing on the RGB channel signal;
projecting the normalized RGB values to two orthogonal chromaticity vectors;
filtering the two orthogonal vectors by a Butterworth band-pass filter to obtain filtered signals;
a chrominance signal is calculated based on the filtered signal.
Preferably, a saddle point in orthogonal chromaticity vector calculation is iteratively solved by updating with an alternating direction multiplier algorithm, and modal components and multipliers of the augmented Lagrange functions are iteratively updated in a frequency domain by using the augmented Lagrange functions and a constrained variation model until an iteration termination condition is met, so that the modal components are obtained.
Preferably, the constraint variational model involved in the variational modal decomposition algorithm is as follows:
wherein, { u k Denotes the k-th modal component, { w k Denotes the center frequency of the kth modal component, K denotes the number of modal components,
representing partial derivative operation, delta (t) representing a unit pulse function, j representing an imaginary unit, x representing convolution operation, and f representing a target signal;
introducing a penalty factor alpha and a Lagrange multiplier lambda to solve the variational constraint problem, wherein the Lagrange function expression is augmented as follows:
preferably, the center frequency of the modal component is updated in the frequency domain using the following equation:
where ω denotes frequency, i denotes the i-th modal component, and d denotes derivation.
Updating λ in conjunction with:
where τ represents the fidelity coefficient, Λ represents the fourier transform, and n represents the number of iterations.
Preferably, the iteration termination condition is:
wherein ε represents the discrimination accuracy, and ε > 0.
In a second aspect, the invention provides a non-contact heart rate measurement system comprising:
the acquisition module is used for acquiring a plurality of frames of face images of a person to be detected;
the recognition module is used for determining a face interesting region in each frame of face image;
the first calculation module is used for accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
the second calculation module is used for acquiring a chrominance signal based on the IPPG signal;
and the third calculation module is used for carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In a third aspect, the invention provides a non-transitory computer readable storage medium for storing computer instructions which, when executed by a processor, implement the contactless heart rate measurement method as described above.
In a fourth aspect, the present invention provides an electronic device comprising: a processor, a memory, and a computer program; wherein a processor is connected with the memory, a computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute instructions to implement the contactless heart rate measurement method as described above.
The invention has the beneficial effects that:
the intensity, light intensity and color of pixel values can be separated through a CHROM algorithm, and noise is eliminated; and performing a VMD algorithm on the signals, and decomposing the main signals into different modes by using the VMD algorithm according to the heartbeat frequency characteristics, so that the signal frequency ranges of the modes are not overlapped with each other, and the heartbeat signals which are relatively complete and have no harmonic residue are separated.
Aiming at complex face conditions, particularly when non-skin pixel interference information such as hair, glasses or beard exists, a multi-color space skin color detection algorithm is adopted to remove non-skin color pixels, and an ROI with high stability and high signal-to-noise ratio is obtained.
When pulse wave signals are extracted, noise introduced due to face movement and illumination direction change inevitably exists in the pulse waves, in order to further eliminate the noise, an automatic optimization variational modal decomposition algorithm is adopted, the noise elimination can be realized, pulse wave information with high signal-to-noise ratio is obtained, and high-accuracy non-contact heart rate detection is realized.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the description of the embodiments are briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a flow chart of a method of non-contact heart rate measurement according to an embodiment of the invention;
FIG. 2 is a schematic diagram of a region of interest of a human face according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of a processing procedure of a face image according to an embodiment of the present invention;
FIG. 4 is a block diagram of an acquisition module according to an embodiment of the present invention;
fig. 5 is a flowchart illustrating automatic optimization of the number of modal components and the penalty factor according to an embodiment of the present invention.
Detailed Description
Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below by way of the drawings are illustrative only and are not to be construed as limiting the invention.
It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.
It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, various embodiments or examples and features of different embodiments or examples described in this specification can be combined and combined by one skilled in the art without contradiction.
For the purpose of facilitating an understanding of the present invention, the present invention will be further explained by way of specific embodiments with reference to the accompanying drawings, which are not intended to limit the present invention.
It should be understood by those skilled in the art that the drawings are merely schematic representations of embodiments and that the elements shown in the drawings are not necessarily required to practice the invention.
Example 1
This embodiment 1 provides a non-contact heart rate measurement system, uses this system to realize non-contact's heart rate and measures, need not to paint conductive paste to the skin, need not to utilize the appointed position of electrode contact health, and it is through accurate location face detection, combines the rejection and the noise removal of light intensity and color information, extracts the heart rate signal.
In this embodiment 1, the non-contact heart rate measurement system mainly includes the following functional modules:
the acquisition module is used for acquiring a plurality of frames of face images of a person to be detected;
the recognition module is used for determining a face interesting region in each frame of face image;
the first calculation module is used for accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
the second calculation module is used for acquiring a chrominance signal based on the IPPG signal;
and the third calculation module is used for carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In this embodiment 1, the non-contact heart rate measurement system is utilized to implement a non-contact heart rate measurement method, where the method includes:
acquiring a plurality of frames of face images of a person to be detected by using an acquisition module;
determining a face interesting region in each frame of face image by using an identification module;
accumulating and averaging skin pixels in all the human face interesting regions by using a first computing module to obtain an IPPG signal;
processing the IPPG signal by using a second calculation module to obtain a chrominance signal;
and performing a variation modal decomposition algorithm on the chrominance signal by using a third module to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In this embodiment 1, acquiring a region of interest of a human face includes: and acquiring a face mark point by using the landmark face recognition model, setting a face interesting region, and removing the environment information of the face edge by using a skin color detector.
In this embodiment 1, acquiring the chrominance signal includes:
extracting RGB three channels of the IPPG signal, and then carrying out normalization processing on the RGB channel signal;
projecting the normalized RGB values to two orthogonal chrominance vectors;
filtering the two orthogonal vectors by a Butterworth band-pass filter to obtain filtered signals;
a chrominance signal is calculated based on the filtered signal.
In this embodiment 1, an alternating direction multiplier algorithm is used to update a saddle point in the calculation of an iterative solution orthogonal chromaticity vector, and an augmented Lagrange function and a constrained variation model are used to iteratively update a modal component and an augmented Lagrange function multiplier in a frequency domain until an iteration termination condition is met, so as to obtain a modal component.
In this embodiment 1, the constraint variational model involved in the variational modal decomposition algorithm is:
wherein, { u k Denotes the k-th modal component, { w k Denotes the center frequency of the kth modal component, K denotes the number of modal components,
represents partial derivative operation, delta (t) represents unit pulse function, j represents imaginary unit, represents convolution operation, and f represents partial derivative operationA target signal;
introducing a penalty factor alpha and Lagrange multiplier lambda to solve the variational constraint problem, wherein the augmented Lagrange function expression is as follows:
in this embodiment 1, the center frequency of the modal component is updated in the frequency domain using the following equation:
where ω denotes frequency, i denotes the i-th modal component, and d denotes derivation.
Updating λ in conjunction with:
where τ denotes the fidelity coefficient, Λ denotes the fourier transform, and n denotes the number of iterations.
In this embodiment 1, the iteration termination condition is:
wherein ε represents the discrimination accuracy, and ε > 0.
In this embodiment 1, the non-contact heart rate measurement system and the heart rate measurement method implemented by using the system are Based on a colorimetric-Based remote photoplethysmography (CHROM) and a Variational Mode Decomposition (VMD), and can be used for non-contact heart rate measurement under complex illumination and large-amplitude motion, and have an important application value for real-time monitoring of vital signs in a complex environment.
Example 2
In this embodiment 2, a non-contact heart rate measurement system is provided, through obtaining human face image, carries out signal processing to human face image, finally obtains the heart rate signal.
In this embodiment 2, the non-contact heart rate measuring system mainly includes the following functional modules:
the acquisition module is used for acquiring multi-frame face images of a person to be detected;
the recognition module is used for determining a face interesting region in each frame of face image;
the first calculation module is used for performing accumulation and average on skin pixels in all the human face interesting regions to obtain an IPPG signal;
the second calculation module is used for acquiring a chrominance signal based on the IPPG signal;
and the third calculation module is used for carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In embodiment 2, the non-contact heart rate measurement method is implemented by using the non-contact heart rate measurement system, and the method includes:
acquiring a plurality of frames of face images of a person to be detected by using an acquisition module;
determining a human face interesting region in each frame of human face image by using an identification module;
accumulating and averaging skin pixels in all the human face interesting regions by using a first computing module to obtain an IPPG signal;
processing the IPPG signal by using a second calculation module to obtain a chrominance signal;
and performing a variation modal decomposition algorithm on the chrominance signal by using a third module to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In this embodiment 2, the acquisition module includes an industrial camera, and the industrial camera is used to supplement light to a human face by combining with a halogen lamp (whose light wave range includes visible light and invisible light ranges), so as to obtain better image quality. The captured human face video sequence is input into a PC (computer system) end through a USB3.0 data transmission line, and an identification module, a first calculation module, a second calculation module and a third calculation module in the PC end are used for processing images to finally obtain a heart rate signal.
Firstly, frame processing is carried out on the captured video, and a face image of each frame is obtained.
The acquiring of the human face region of interest comprises: and acquiring a face mark point by using the landmark face recognition model, setting a face interesting region, and removing the environment information of the face edge by using a skin color detector.
The face recognition program in the recognition module uses a landmark model generated by an Ensemble of Regression Trees (ERT) based on gradient boosting learning. And acquiring 68 personal face mark points through a landmark model, and setting a face region of interest as shown in fig. 2, wherein the image at the upper left corner is a mask image of the face mark points, the image at the upper right corner is an actual face mask image, the image at the lower left corner is an actual ROI image after a skin color detector is used, and the image at the lower right corner is a binary image of the actual ROI image. In the detection process, the face edge contains a small part of environmental information, and subsequently, a skin color detector is used to remove the environmental information of the face edge from the region of interest through the skin color detector.
And performing cumulative averaging on all the processed skin pixels to obtain an original IPPG signal.
In this embodiment 2, the obtaining the chrominance signal by using the second calculating module includes: extracting RGB three channels of the IPPG signal, and then carrying out normalization processing on the RGB channel signal; projecting the normalized RGB values to two orthogonal chrominance vectors; filtering the two orthogonal vectors by a Butterworth band-pass filter to obtain filtered signals; a chrominance signal is calculated based on the filtered signal. Specifically, the method comprises the following steps:
the original IPPG signal is processed using the CHROM algorithm, which comprises the following steps:
RGB three-channel extraction is carried out on the original IPPG signal, and then normalization is carried out on the RGB channel signalProcessing; projecting the normalized RGB values to two orthogonal chromaticity vectors X chrom And Y chrom ：
X chrom (t)＝3x r (t)-2x g (t) (2)
Y chrom (t)＝1.5x r (t)+x g (t)-1.5x b (t) (3)
The final output S (t) is:
S(t)＝X f -αY f ； (4)
wherein x is r Representing the red channel component, x, of the original IPPG signal g Representing the green channel component, x, of the original IPPG signal b Representing the blue channel component of the original IPPG signal, t representing time, the signal waveform varying with time t, X f 、Y f Are each X chrom ，Y chrom The signals are filtered by a fifth-order 0.7Hz-4Hz Butterworth band-pass filter; α is X f 、Y f The ratio of the standard deviations, S (t), is the IPPG signal processed through the CHROM algorithm.
After passing through the CHROM algorithm, most of the noise in the original IPPG signal is eliminated. And performing VMD algorithm on the S (t) to obtain a more accurate heart rate signal.
In this embodiment 2, an alternating direction multiplier algorithm is used to update saddle points in the iterative solution of orthogonal chromaticity vector calculation, and an augmented Lagrange function and a constraint variation model are used to iteratively update modal components and augmented Lagrange function multipliers in a frequency domain until an iteration termination condition is satisfied, so as to obtain modal components.
In this embodiment 2, the constraint variational model involved in the variational modal decomposition algorithm is:
wherein, { u k }＝{u 1 ,u 2 ,...,u k Denotes the k-th modal component, { w k }＝{w 1 ,w 2 ,...,w k Denotes the center frequency of the kth modal component, K denotes the number of modal components,
represents the partial derivative operation, δ (t) represents the unit impulse function, j represents the imaginary unit, x represents the convolution operation, f represents the target signal, and e represents the transcendental number.
Introducing a penalty factor alpha and a Lagrange multiplier lambda to solve the variational constraint problem, wherein the Lagrange function expression is augmented as follows:
in this embodiment 2, the saddle point in equation (7) is iteratively updated by using an alternative multiplier algorithm, and u is iteratively updated in the frequency domain k 、w k 、λ。
In this embodiment 2, the VMD algorithm decomposes the signal into 3 modal components (the number of modal components is determined by experiment), the penalty factor α is 2000, and the decomposition steps are as follows:
step 1: initialization
n is zero;
step 2: u. of k And w k Iteratively updated by equations (8) and (9), respectively:
where ω denotes frequency, i denotes the i-th modal component, and d denotes derivation.
And 3, step 3: updating λ by equation (10):
where τ represents the fidelity coefficient, Λ represents the fourier transform, and n represents the number of iterations.
And 4, step 4: and (4) repeating the steps 2 and 3 until an iteration termination condition is met, wherein the termination condition is given by a formula (11).
Wherein epsilon represents the discrimination precision, epsilon is larger than 0, superscript n is the iteration step number, and subscript k represents the current mode number.
And 5: 3 modal components are output.
And (3) outputting 3 modal components by the VMD algorithm, wherein the modal component with the largest peak value in the frequency spectrum is the heart rate signal.
In this embodiment 2, the proposed non-contact heart rate measurement system and the heart rate measurement method implemented by using the system are Based on a colorimetric-Based remote photoplethysmography (CHROM) and a Variational Mode Decomposition (VMD), and can be used for non-contact heart rate measurement under complex illumination and large-amplitude motion, and have an important application value for real-time monitoring of vital signs in a complex environment.
Example 3
In this embodiment 3, a non-contact heart rate measurement system is provided, through obtaining human face image, carries out signal processing to human face image, finally obtains the heart rate signal.
In this embodiment 3, the non-contact heart rate measuring system mainly includes the following functional modules:
the acquisition module is used for acquiring a plurality of frames of face images of a person to be detected;
the recognition module is used for determining a face interesting region in each frame of face image;
the first calculation module is used for accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
the second calculation module is used for acquiring a chrominance signal based on the IPPG signal;
and the third calculation module is used for carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In this embodiment 3, the non-contact heart rate measurement method is implemented by using the non-contact heart rate measurement system, and the method includes:
acquiring a plurality of frames of face images of a person to be detected by using an acquisition module;
determining a human face interesting region in each frame of human face image by using an identification module;
using a first calculation module to perform accumulation and average on skin pixels in all the human face interesting regions to obtain an IPPG signal;
processing the IPPG signal by using a second calculation module to obtain a chrominance signal;
and performing a variation modal decomposition algorithm on the chrominance signal by using a third module to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In this embodiment 3, as shown in fig. 4, the acquiring module includes an industrial camera A1, and the industrial camera is used in combination with a halogen lamp A2 (whose light wave range includes visible light and invisible light ranges) to fill in light for the human face of the subject A4, so as to obtain better image quality. The captured human face video sequence is input into a PC end A3 (computer system) through a USB3.0 data transmission line A5, an identification module, a first calculation module, a second calculation module and a third calculation module in the PC end are used for processing images, and finally a heart rate signal is obtained.
Firstly, frame processing is carried out on captured video, and each frame of face image is obtained.
The acquiring of the human face region of interest comprises: and acquiring a face mark point by using the landmark face recognition model, setting a face interesting region, and removing the environment information of the face edge by using a skin color detector.
The face recognition program in the recognition module uses a landmark model generated by an Ensemble of Regression Trees (ERT) based on gradient boosting learning. Acquiring 68 individual face mark points through a landmark model, setting a face interesting region as shown in a diagram at the upper left corner in FIG. 2, and newly generating mark points HL and mark points HR by using mark points 0, 8 and 16, wherein coordinate points of the HL and the HR are calculated as follows:
wherein, y p8 Is the ordinate, y, of the marking point 8 p0 Is the ordinate, x, of the index point 0 p0 Is marked on the abscissa, y, of point 0 p0 Is the ordinate, x, of the marker point 0 p16 Is the abscissa, y, of the marking point 16 p16 Is the ordinate of the marker point 16; x is the number of HR 、y HR And x HL 、y HL Decomposing into the horizontal and vertical coordinates of a new mark point HR and the horizontal and vertical coordinates of a new mark point HL; the human face region is obtained by connecting the mark point 0 to the mark point 16, HR and HL in sequence, the mouth region is removed by connecting the mark point 48 to the mark point 59 in sequence, and finally the human face mask image, namely the region of interest, is obtained.
In the detection process, the face edge contains a small part of environmental information inevitably, a skin color detector is used subsequently, the environmental information of the face edge is removed from an interested area through the skin color detector, specifically, the skin color detector removes non-skin interference as far as possible while retaining skin area information by using a multi-color space skin color detection algorithm, and the core method is as follows: and the accurate skin color pixel judgment is carried out by using three different color space skin color threshold values of RGB, ycrCb and CMYK.
The multi-color space skin color detection algorithm specifically comprises the following steps:
(1) The skin color threshold condition in the RGB color space is as follows:
R＞95∩G＞40∩B＞20∩R>G∩R＞B∩|R-G|>15
wherein R is a red channel value, G is a green channel value, and B is a blue channel value;
(2) The skin tone threshold condition in the YCrCb color space is as follows:
Cr＞135∩Cb＞85∩Yl>80∩Cr＜＝(1.5874*Cb)+20∩
Cr＞＝(0.3447*Cb)+76.2068∩Cr＞＝(-4.5653*Cb)+234.5652∩
Cr＜＝(-1.15*Cb)+301.78∩Cr＜＝(-2.2868*Cb)+433.85
wherein, yl is a color brightness value, cr is a red concentration offset value, and Cb is a blue component offset value;
(3) The skin tone threshold condition in the CMYK color space is as follows:
K＜0.8∩0<＝C<0.05∩0.1＜＝Y/M＜4.8
∩0.088＜Y＜1∩0＜C/Y＜1
where C is a cyan channel value, M is a magenta channel value, Y is a yellow channel value, and K is a black channel value;
and (3) calculating the face interesting region by using the threshold values from (1) to (3), reserving pixels which accord with the skin color threshold value condition in the RGB color space, the skin color threshold value condition in the YCrCb color space and the skin color threshold value condition in the CMYK color space in the face interesting region, and setting the RGB values of the pixels which do not accord with the threshold value condition to be (0, 0) so as to eliminate the environment information of the face edge and obtain the accurate face interesting region.
And performing cumulative averaging on all the processed skin pixels to obtain an original IPPG signal.
In this embodiment 3, the obtaining the chrominance signal by using the second calculating module includes: extracting RGB three channels of the IPPG signal, and then carrying out normalization processing on the RGB channel signal; projecting the normalized RGB values to two orthogonal chrominance vectors; filtering the two orthogonal vectors by a Butterworth band-pass filter to obtain filtered signals; a chrominance signal is calculated based on the filtered signal. Specifically, the method comprises the following steps:
the original IPPG signal is processed using the CHROM algorithm.
After passing through the CHROM algorithm, most of the noise in the original IPPG signal is eliminated. And performing VMD algorithm on the S (t) to obtain a more accurate heart rate signal.
In this embodiment 3, an alternating direction multiplier algorithm is used to update saddle points in the calculation of the iterative solution orthogonal chromaticity vectors, and the modal components and the multipliers of the augmented Lagrange function are updated in the frequency domain by using the augmented Lagrange function and the constraint variation model in an iterative manner until an iteration termination condition is satisfied, so as to obtain the modal components.
In this embodiment 3, the VMD algorithm decomposes the signal into K modal components.
The difference from embodiment 2 is that the modal component is no longer fixed to 3 and the penalty factor α is no longer fixed to 2000. Before modal decomposition, parameters, the number K of modal components and a penalty factor alpha need to be set. Inappropriate parameters may lead to modal aliasing of the signal, loss of signal frequency components, and the like. In order to solve the above situation, the invention provides an algorithm for automatically optimizing the number K of the mode components and the penalty factor alpha, and the automatic parameter confirmation is carried out.
As shown in fig. 5, the automatic optimization process of the number K of modal components and the penalty factor α includes:
(1) because the range of the main frequency domain of the heart rate is small (about between 0.7Hz and 4 Hz), the number K of modal components belongs to [1,4], the step length is 1, and 4K values are obtained; setting the range of the penalty factor alpha as alpha to [0, 2000], the step length of the penalty factor alpha is 400, and obtaining 6 penalty factor alpha values. If the selection of α is proper, the correlation between the modal components is small, and if the selection of α is improper, the correlation between the modal components becomes large.
(2) Combining 4K values and 6 penalty factor alpha values to obtain 24 parameter pairs, judging whether modal components obtained by each parameter pair meet the optimization correlation limiting condition and the frequency loss limiting condition, and reserving the parameter pairs meeting the optimization correlation limiting condition and the frequency loss limiting condition:
the optimization correlation limiting conditions are as follows:
wherein, C (·) represents a correlation function, K is a kth modal component, and K is the number of modal components; MC is the average value of the correlation between two adjacent modes under the fixed mode number K; c represents the number of optimization times of the parameter alpha, formula
Representing the ratio of the c-th optimization and the c-1 st optimization correlation; when the alpha is set to be too large, or the correlation between the modal components is suddenly increased, therefore, the judgment threshold value is set to be 0.5, and when the correlation ratio is smaller than 0.5, the c-1 time optimization value alpha is reserved as the alpha under the K value of the modal number; finally, 4 pairs of parameters, namely 4 pairs of K and alpha, are obtained. For example, [ K =1, [ α =400 ]]，[K＝2,α＝400]，[K＝3,α＝400]，[K＝4,α＝400]. Where α is uncertain, when the condition cannot be met, α iterates to 2000 stop and uses 2000 as the α value.
In the process of modal decomposition, frequency loss may occur, and the limiting conditions of the frequency loss are as follows:
wherein S (t) is an IPPG signal processed by a chrominance algorithm, | · |. Luminance |, luminance 2 Is a two-norm; pairs of parameters less than the threshold condition are retained.
(3) Selecting the optimal parameter pair from the parameter pairs reserved in the step (2) by using the method of maximum envelope kurtosis:
where k is the k-th modal component,
of component of the kth mode at a number K of modesA modulus of a hilbert transform; it should be noted that, since there is only one pair of parameter pairs for each mode number K in the remaining parameter pairs of step (2), the number of parameter pairs and the number of mode numbers K are equivalent in number.
Wherein the molecule
Is composed of
The fourth order central moment of (a) · is a square error, ek k Obtaining a kurtosis value after taking a module for the k modal component by Hilbert transform;
wherein the content of the first and second substances,
is a vector composed of the peak values after all modal components are subjected to Hilbert transform modulus under the K modal number,
is composed of
The peak value of the medium to maximum value,
maximum kurtosis values for all parameter pairs;
(4) finally return to
The medium mode number K and alpha finish the automatic optimization of the parameters.
The method comprises the steps that in K modal components output by an automatically optimized VMD algorithm, the modal component with the largest peak value in a frequency spectrum is a heart rate signal; the maximum peak frequency is solved through Fourier transformation, and then the final heart rate value can be obtained.
The specific steps involved in this example 3 are the same as those in example 2, and will not be described here again.
Example 4
In this embodiment 4, a non-contact heart rate measuring method is provided, where the method includes the following steps:
acquiring a plurality of frames of face images of a person to be detected;
determining a face interesting region in each frame of face image;
accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
acquiring a chrominance signal based on the IPPG signal;
and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
As shown in fig. 1, in the non-contact heart rate measurement method described in this embodiment 4, first, face detection is performed on an acquired face image through a landmark face recognition model to acquire a region of interest (ROI), then, skin color detection is performed to remove edge environment pixels, then, an original pulse wave signal (IPPG) is acquired, an RGB three-channel signal is extracted and generated, a chrominance signal is obtained by using a Chrom algorithm, a denoised IPPG signal is obtained by using a variational modal decomposition (VDM algorithm), and finally, a fourier transform is performed to obtain a heart rate.
In this embodiment 4, a human face video image of a subject is acquired by an industrial camera, and is used to supplement light to a human face by combining a halogen lamp 2 (whose light wave range includes visible light and invisible light ranges), so as to obtain better image quality. And inputting the captured human face video sequence into the PC terminal 3 through a USB3.0 data transmission line to wait for the PC terminal to process the image. Firstly, frame processing is carried out on the captured video, and a face image of each frame is obtained.
The face recognition program uses a landmark face recognition model generated by an Ensemble of Regression Trees (ERT) based on gradient boosting learning. The individual face marker points are then obtained 68 by the landmark face recognition model, the region of interest of the face being shown in fig. 2.
In the detection process, the face edge contains a small part of environment information, and subsequently, a skin color detector is used to remove the environment information of the face edge from the region of interest through the skin color detector.
And performing cumulative average on all the skin pixels after treatment to obtain the IPPG signal.
The original IPPG signal is processed by using a CHROM algorithm, and the steps are as follows:
RGB three-channel extraction is carried out on the original IPPG, and then normalization processing is carried out on RGB channel signals.
Projecting the normalized RGB values to two orthogonal chromaticity vectors X chrom And Y chrom The formula is as follows:
X chrom (t)＝3x r (t)-2x g (t) (2)
Y chrom (t)＝1.5x r (t)+x g (t)-1.5x b (t) (3)
the final output S (t) is:
S(t)＝X f -αY f ； (4)
wherein x is r Representing the red channel component, x, of the original IPPG signal g Representing the green channel component, x, of the original IPPG signal b Representing the blue channel component of the original IPPG signal, t representing time, the signal waveform varying with time t, X f 、Y f Are each X chrom ，Y chrom Is processed by a fifth-order Butterworth of 0.7Hz-4HzA signal filtered by a bandpass filter; α is X f 、Y f The ratio of the standard deviations, S (t), is the IPPG signal processed through the CHROM algorithm.
After passing through the CHROM algorithm, most of the noise in the original IPPG signal is eliminated. And performing VMD algorithm on the S (t) to obtain a more accurate heart rate signal.
The constraint variational model involved in the VMD algorithm is as follows:
wherein, { u k }＝{u 1 ,u 2 ,...,u k Denotes the k-th modal component, { w k }＝{w 1 ,w 2 ,...,w k Denotes the center frequency of the kth modal component, K denotes the number of modal components,
the partial derivative operation is shown, delta (t) is a unit pulse function, j is an imaginary unit, x is a convolution operation, f is a target signal, and e is an overtaking number.
And introducing a penalty factor alpha and a Lagrange multiplier lambda to solve the variational constraint problem. The resulting augmented Lagrange expression is as follows:
adopting an alternative way multiplier algorithm to update saddle points in the iterative solution (7), and iteratively updating u in the frequency domain k 、w k 、λ。
The VMD algorithm decomposes the signal into 3 modal components (the number of modal components is determined by experiment), the penalty factor α is 2000, and the decomposition steps are as follows:
(1) Initialization
n is zero;
(2)u k and w k Iteratively updated by equations (8) and (9), respectively:
where ω denotes frequency, i denotes the i-th modal component, and d denotes derivation.
(3) Update λ by equation (10):
where τ represents the fidelity coefficient, Λ represents the fourier transform, and n represents the number of iterations.
(4) And (5) repeating the steps 2 and 3 until an iteration termination condition is met, wherein the termination condition is given by the formula (11).
Wherein epsilon represents the discrimination precision, epsilon is larger than 0, superscript n is the iteration step number, and subscript k represents the current mode number.
(5) 3 modal components are output.
A schematic diagram of a processing process of a face image described in this embodiment 4 is shown in fig. 3, where a landmark is returned by landmark face recognition, an area of interest is drawn according to a diagram at the upper left corner in fig. 2, then, environmental information of an edge is removed by skin color detection, RGB three-channel processing is performed on a pulse wave, a CHROM algorithm is performed on the RGB three-channel signal to obtain a pulse wave signal with light intensity and color removed, modal decomposition denoising is performed by VDM, and a heart rate signal waveform and a fourier spectrogram are obtained by reconstruction.
In this embodiment 4, the landmark model is used to accurately locate the face, and the CHROM algorithm and the VMD algorithm are used to perform denoising processing on the original IPPG signal, so as to obtain an accurate heart rate, thereby achieving the purpose of obtaining the heart rate in a non-contact detection manner.
It solves several main technical problems:
the face tracking detection has a good tracking effect on the large-amplitude movement of the face, but under the condition that the face is relatively static, because each frame of picture of a video is relatively independent, the face identification mark point shakes, and thus noise is additionally introduced; the movement of the human face can cause the change of the light incidence angle and the skin color, so that noise is introduced, the intensity, the light intensity and the color of a pixel value can be separated through a CHROM algorithm, and the noise is eliminated; VMD is carried out to pulse wave signal, according to the heartbeat frequency characteristic, uses VMD algorithm with main signal decomposition for different modals, has guaranteed that signal frequency range does not overlap each other between each modality, separates out comparatively complete and no harmonic residual heartbeat signal.
Example 5
In this embodiment 5, a non-contact heart rate measuring method is provided, which includes the following steps:
acquiring a plurality of frames of face images of a person to be detected;
determining a human face interesting region in each frame of human face image;
accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
acquiring a chrominance signal based on the IPPG signal;
and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
As shown in fig. 1, in the non-contact heart rate measurement method described in this embodiment 5, first, a face detection is performed on an acquired face image through a landmark face recognition model to acquire a region of interest (ROI), then, skin color detection is performed to remove edge environment pixels, then, an original pulse wave signal (IPPG) is acquired, an RGB three-channel signal is extracted and generated, a chrominance signal is obtained by using a Chrom algorithm, a denoised IPPG signal is obtained by using a variational modal decomposition (VDM algorithm), and finally, a fourier transform is performed to obtain a heart rate.
In this embodiment 5, a video image of a human face of a subject is obtained by an industrial camera, and is used to supplement light to the human face by combining a halogen lamp (whose light wave range includes visible light and invisible light ranges), so as to obtain better image quality. And inputting the captured human face video sequence into a PC (personal computer) end through a USB (universal serial bus) 3.0 data transmission line to wait for the PC end to process the image. Firstly, frame processing is carried out on the captured video, and a face image of each frame is obtained.
The face recognition program uses a landmark face recognition model generated by an Ensemble of Regression Trees (ERT) based on gradient boosting learning. Then, 68 individual face mark points are obtained through a landmark model, and mark points HL and mark points HR are newly generated by using the mark points 0, 8 and 16, wherein the coordinate points of the HL and the HR are calculated as follows:
wherein, y p8 Is the ordinate, y, of the marking point 8 p0 Is the ordinate, x, of the marker point 0 p0 Is the abscissa, y, of the marker point 0 p0 Is the ordinate, x, of the index point 0 p16 Is the abscissa, y, of the marking point 16 p16 Is the ordinate of the marker point 16; x is the number of HR 、y HR And x HL 、y HL Decomposing into the horizontal and vertical coordinates of the new mark point HR and the horizontal and vertical coordinates of the new mark point HL; the mark point 0 to the mark point 16, the mark point HR and the mark point HL are connected in sequence to obtain a face area, the mark point 48 to the mark point 59 are connected in sequence to remove a mouth area, and finally a face mask image, namely a face interesting area, is obtained, as shown in the upper left corner of the image in FIG. 2.
In the detection process, the face edge contains a small part of environment information, and subsequently, a skin color detector is used to remove the environment information of the face edge from the region of interest through the skin color detector. Specifically, the skin color detector uses a multi-color space skin color detection algorithm to eliminate non-skin interference as much as possible while retaining skin area information, and the core method is as follows: and the accurate skin color pixel judgment is carried out by using three different color space skin color threshold values of RGB, ycrCb and CMYK.
The multi-color space skin color detection algorithm specifically comprises the following steps:
(1) The skin color threshold condition in the RGB color space is as follows:
R＞95∩G＞40∩B＞20∩R>G∩R＞B∩|R-G|>15
wherein R is a red channel value, G is a green channel value, and B is a blue channel value;
(2) The skin color threshold condition in the YCrCb color space is as follows:
Cr＞135∩Cb＞85∩Yl>80∩Cr＜＝(1.5874*Cb)+20∩
Cr＞＝(0.3447*Cb)+76.2068∩Cr＞＝(-4.5653*Cb)+234.5652∩
Cr＜＝(-1.15*Cb)+301.78∩Cr＜＝(-2.2868*Cb)+433.85
wherein, yl is a color brightness value, cr is a red concentration offset value, and Cb is a blue component offset value;
(3) The skin tone threshold condition in the CMYK color space is as follows:
K＜0.8∩0<＝C<0.05∩0.1＜＝Y/M＜4.8
∩0.088＜Y＜1∩0＜C/Y＜1
where C is a cyan channel value, M is a magenta channel value, Y is a yellow channel value, and K is a black channel value;
and (3) calculating the face interesting region by using the threshold values from (1) to (3), reserving pixels which accord with the skin color threshold value condition in the RGB color space, the skin color threshold value condition in the YCrCb color space and the skin color threshold value condition in the CMYK color space in the face interesting region, and setting the RGB values of the pixels which do not accord with the threshold value condition to be (0, 0) so as to eliminate the environment information of the face edge and obtain the accurate face interesting region.
And performing cumulative average on all the skin pixels after treatment to obtain the IPPG signal.
The original IPPG signal is processed by using a CHROM algorithm, and the steps are as follows:
RGB three-channel extraction is carried out on the original IPPG, and then normalization processing is carried out on RGB channel signals.
Projecting the normalized RGB values to two orthogonal chromaticity vectors X chrom And Y chrom 。
After passing through the CHROM algorithm, most of the noise in the original IPPG signal is eliminated. And performing VMD algorithm on the S (t) to obtain a more accurate heart rate signal.
The VMD algorithm decomposes the signal into K modal components.
The difference from embodiment 4 is that the modal component is no longer fixed to 3 and the penalty factor α is no longer fixed to 2000. Before carrying out modal decomposition, parameters, the number K of modal components and a penalty factor alpha need to be set. Inappropriate parameters may lead to modal aliasing of the signal, loss of signal frequency components, and the like. In order to solve the above situation, the invention provides an algorithm for automatically optimizing the number K of the mode components and the penalty factor alpha, and the automatic parameter confirmation is carried out.
The automatic optimization process of the number K of modal components and the penalty factor alpha comprises the following steps:
(1) because the range of the main frequency domain of the heart rate is small (about between 0.7Hz and 4 Hz), the number K of modal components belongs to [1,4], the step length is 1, and 4K values are obtained; setting the range of the penalty factor alpha as alpha epsilon [0, 2000], setting the step length of the penalty factor alpha as 400, and obtaining 6 penalty factor alpha values. The appropriate selection of alpha results in small correlation among modal components, and the inappropriate selection of alpha results in large correlation among modal components.
(2) Combining 4K values and 6 penalty factor alpha values to obtain 24 parameter pairs, judging whether modal components obtained by each parameter pair meet the optimization correlation limiting condition and the frequency loss limiting condition, and reserving the parameter pairs meeting the optimization correlation limiting condition and the frequency loss limiting condition:
the optimization correlation limiting conditions are as follows:
wherein, C (·) represents a correlation function, K is a kth modal component, and K is the number of modal components; MC is the average value of the correlation between two adjacent modes under the fixed mode number K; c represents the number of optimization times of the parameter alpha, formula
Representing the ratio of the c-th optimization and the c-1 st optimization correlation; when the alpha is set to be too large, or the correlation between the modal components is suddenly increased, therefore, the judgment threshold value is set to be 0.5, and when the correlation ratio is smaller than 0.5, the c-1 time optimization value alpha is reserved as the alpha under the K value of the modal number; finally, 4 pairs of parameters, namely 4 pairs of K and alpha, are obtained. For example, [ K =1, α =400]，[K＝2,α＝400]，[K＝3,α＝400]，[K＝4,α＝400]. Where α is uncertain, when the condition cannot be met, α iterates to 2000 stop and uses 2000 as the α value.
In the process of modal decomposition, frequency loss may occur, and the limiting conditions of the frequency loss are as follows:
wherein S (t) is an IPPG signal processed by a chrominance algorithm, | · |. Luminance |, luminance 2 Is a two-norm; pairs of parameters less than the threshold condition are retained.
(3) Selecting the optimal parameter pair from the parameter pairs reserved in the step (2) by using the method of maximum envelope kurtosis:
where k is the k-th modal component,
a modulus of a hilbert transform of a kth modal component at a temporal modal number K; it should be noted that, since there is only one pair of parameter pairs for each mode number K in the remaining parameter pairs of step (2), the number of parameter pairs and the number of mode numbers K are equivalent in number.
Wherein the molecule
Is composed of
Is the square error, ek k Obtaining a kurtosis value after the k-th modal component is subjected to Hilbert transform;
wherein the content of the first and second substances,
is a vector composed of the peak values after all modal components are subjected to Hilbert transform modulus under the K modal number,
is composed of
The peak value of the medium to maximum value,
maximum kurtosis values for all parameter pairs;
(4) finally return to
The medium mode number K and alpha finish the automatic optimization of the parameters.
The method comprises the following steps that in K modal components output by an automatic optimization VMD algorithm, the modal component with the maximum peak value in a frequency spectrum is a heart rate signal; the maximum peak frequency is solved through Fourier transformation, and then the final heart rate value can be obtained.
The specific steps involved in this example 5 are the same as those in example 4, and will not be described here again.
Example 6
Embodiment 6 of the present invention provides a non-transitory computer readable storage medium for storing computer instructions which, when executed by a processor, implement a method for contactless heart rate measurement, the method comprising:
acquiring a multi-frame face image of a person to be detected;
determining a human face interesting region in each frame of human face image;
accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
acquiring a chrominance signal based on the IPPG signal;
and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
Example 7
Embodiment 7 of the present invention provides a computer program (product) comprising a computer program for implementing a method of contactless heart rate measurement as described above when run on one or more processors, the method comprising:
acquiring a multi-frame face image of a person to be detected;
determining a human face interesting region in each frame of human face image;
accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
acquiring a chrominance signal based on the IPPG signal;
and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
Example 8
An embodiment 8 of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein a processor is connected with the memory, a computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to make the electronic device execute the non-contact heart rate measurement method as described above, the method comprising:
acquiring a plurality of frames of face images of a person to be detected;
determining a human face interesting region in each frame of human face image;
accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
acquiring a chrominance signal based on the IPPG signal;
and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
In summary, the non-contact heart rate detection method and system according to the embodiments of the present invention can separate the intensity, light intensity, and color of the pixel value by using the CHROM algorithm, and eliminate noise; and performing a VMD algorithm on the signals, and decomposing the main signals into different modes by using the VMD algorithm according to the heartbeat frequency characteristics, so that the signal frequency ranges of the modes are not overlapped with each other, and the heartbeat signals which are relatively complete and have no harmonic residue are separated.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts based on the technical solutions disclosed in the present invention.
Claims (10)
1. A method of non-contact heart rate measurement, comprising:
acquiring a multi-frame face image of a person to be detected;
determining a human face interesting region in each frame of human face image;
accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
acquiring a chrominance signal based on the IPPG signal;
and carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
2. The non-contact heart rate measurement method of claim 1, wherein acquiring a face region of interest comprises: and acquiring a face mark point by using a landmark face recognition model, setting a face interesting region, and removing the environment information of the face edge by using a skin color detector.
3. A method of contactless heart rate measurement according to claim 2, characterized in that the skin tone detector uses a multi-color space skin tone detection algorithm.
4. The method of non-contact heart rate measurement according to claim 1, wherein acquiring a chrominance signal comprises:
extracting RGB three channels of the IPPG signal, and then carrying out normalization processing on the RGB channel signal;
projecting the normalized RGB values to two orthogonal chromaticity vectors;
filtering the two orthogonal vectors by a Butterworth band-pass filter to obtain filtered signals;
a chrominance signal is calculated based on the filtered signal.
5. The method as claimed in claim 4, wherein an alternating direction multiplier algorithm is used to update saddle points in the iterative solution of orthogonal chrominance vector calculation, and modal components and enhanced Lagrange function multipliers are iteratively updated in the frequency domain by using an augmented Lagrange function and a constrained variation model until an iteration termination condition is satisfied to obtain the modal components.
6. The non-contact heart rate measurement method of claim 5, wherein:
the constraint variational model involved in the variational modal decomposition algorithm is as follows:
wherein, { u k Denotes the k-th modal component, { w k Denotes the center frequency of the kth modal component, K denotes the number of modal components,
representing partial derivative operation, delta (t) representing a unit pulse function, j representing an imaginary unit, x representing convolution operation, and f representing a target signal;
introducing a penalty factor alpha and Lagrange multiplier lambda to solve the variational constraint problem, wherein the augmented Lagrange function expression is as follows:
7. the method of non-contact heart rate measurement according to claim 6, wherein:
the center frequency of the modal component is updated in the frequency domain using the following equation:
where ω denotes frequency, i denotes the i-th modal component, and d denotes derivation.
Updating λ in conjunction with:
where τ represents the fidelity coefficient, Λ represents the fourier transform, and n represents the number of iterations.
8. The method of non-contact heart rate measurement according to claim 7, wherein the iteration termination condition is:
wherein ε represents the discrimination accuracy, and ε > 0.
9. A non-contact heart rate measurement system, comprising:
the acquisition module is used for acquiring a plurality of frames of face images of a person to be detected;
the recognition module is used for determining a face interesting region in each frame of face image;
the first calculation module is used for accumulating and averaging skin pixels in all the human face interesting regions to obtain an IPPG signal;
the second calculation module is used for acquiring a chrominance signal based on the IPPG signal;
and the third calculation module is used for carrying out a variation modal decomposition algorithm on the chrominance signal to obtain a plurality of modal components, wherein the modal component with the maximum peak value in the frequency spectrum is the heart rate signal.
10. An electronic device, comprising: a processor, a memory, and a computer program; wherein a processor is connected with a memory, a computer program being stored in the memory, the processor executing the computer program stored by the memory when the electronic device is running, to cause the electronic device to execute the instructions of the contactless heart rate measurement method according to any of the claims 1-8.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2021115252062 | 2021-12-14 | ||
| CN202111525206 | 2021-12-14 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115376179A true CN115376179A (en) | 2022-11-22 |
Family
ID=84060087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210416920.6A Pending CN115376179A (en) | 2021-12-14 | 2022-04-20 | Non-contact heart rate measuring method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115376179A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116999044A (en) * | 2023-09-07 | 2023-11-07 | 南京云思创智信息科技有限公司 | Real-time motion full-connection bidirectional consistent optical flow field heart rate signal extraction method |
-
2022
- 2022-04-20 CN CN202210416920.6A patent/CN115376179A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116999044A (en) * | 2023-09-07 | 2023-11-07 | 南京云思创智信息科技有限公司 | Real-time motion full-connection bidirectional consistent optical flow field heart rate signal extraction method |
| CN116999044B (en) * | 2023-09-07 | 2024-04-16 | 南京云思创智信息科技有限公司 | Real-time motion full-connection bidirectional consistent optical flow field heart rate signal extraction method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | Deepphys: Video-based physiological measurement using convolutional attention networks | |
| CN111407245B (en) | Non-contact heart rate and body temperature measuring method based on camera | |
| Subramaniam et al. | Estimation of the Cardiac Pulse from Facial Video in Realistic Conditions. | |
| CN107564048B (en) | Feature registration method based on bifurcation point | |
| Casado et al. | Face2PPG: An unsupervised pipeline for blood volume pulse extraction from faces | |
| Xia et al. | A global optimization method for specular highlight removal from a single image | |
| CN111281367A (en) | Anti-interference non-contact heart rate detection method based on face video | |
| Hu et al. | A novel spatial-temporal convolutional neural network for remote photoplethysmography | |
| CN115376179A (en) | Non-contact heart rate measuring method and system | |
| CN112396565A (en) | Method and system for enhancing and segmenting blood vessels of images and videos of venipuncture robot | |
| Das et al. | Bvpnet: Video-to-bvp signal prediction for remote heart rate estimation | |
| Schenk et al. | Automatic high-speed video glottis segmentation using salient regions and 3D geodesic active contours | |
| Nguyen et al. | Non-contact heart rate measurement from deteriorated videos | |
| Tania et al. | Computational complexity of image processing algorithms for an intelligent mobile enabled tongue diagnosis scheme | |
| Mehta et al. | Heart rate estimation from RGB facial videos using robust face demarcation and VMD | |
| Hu et al. | An end-to-end efficient framework for remote physiological signal sensing | |
| Mohamed et al. | On analyzing various density functions of local binary patterns for optic disc segmentation | |
| CN111814682A (en) | Face living body detection method and device | |
| Gu et al. | Automatic Tongue Image Segmentation Based on Thresholding and an Improved Level Set Model | |
| CN113361480B (en) | Human body pulse wave acquisition method based on face video | |
| CN114677713A (en) | Near-infrared light-based arm vein blood sampling point identification method and system | |
| Ben Salah et al. | Contactless heart rate estimation from facial video using skin detection and multi-resolution analysis | |
| Das et al. | A multiresolution method for non-contact heart rate estimation using facial video frames | |
| CN111685730A (en) | Non-contact physiological parameter detection method, system, terminal equipment and storage medium | |
| Zhang et al. | A gesture recognition method based on YCbCr and SURF for service robot interaction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |