CN111134650A - Heart rate information acquisition method and device, computer equipment and storage medium - Google Patents
Heart rate information acquisition method and device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN111134650A CN111134650A CN201911371181.8A CN201911371181A CN111134650A CN 111134650 A CN111134650 A CN 111134650A CN 201911371181 A CN201911371181 A CN 201911371181A CN 111134650 A CN111134650 A CN 111134650A
- Authority
- CN
- China
- Prior art keywords
- sub
- image
- heart rate
- signal
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
Abstract
The application relates to a heart rate information acquisition method and device, computer equipment and a storage medium. The method comprises the following steps: obtaining a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image; dividing the human face interesting region in each frame image into a plurality of sub-images; extracting a heart rate sub-signal of each sub-image sequence in a time domain; obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value; determining a photoplethysmography (PPG) signal representing pulse information in a video according to the frequency domain weight, the space domain weight and the heart rate sub-signal of each sub-image sequence; and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period. By adopting the method, the acquired heart rate information can be more accurate.
Description
Technical Field
The present application relates to the field of computer technologies, and in particular, to a heart rate information obtaining method and apparatus, a computer device, and a storage medium.
Background
The heart rate is an important index reflecting the physiological health state of a human body, and the heart rate of the human body needs to be detected in daily life of the human body, such as physical exercise. However, the traditional electrocardiogram used for representing the heart rate of a person is limited by the environment and cannot be used anytime and anywhere.
Therefore, research on the manner in which heart rate is obtained has begun. At present, the existing heart rate detection method can be based on the recognition of a face image and certain analysis so as to obtain the relevant data of the heart rate. However, the traditional method cannot exclude external interference, so that the obtained heart rate data is inaccurate.
Disclosure of Invention
In view of the above, it is necessary to provide a heart rate information acquiring method, a heart rate information acquiring apparatus, a computer device, and a storage medium, which can improve the accuracy of heart rate data.
In a first aspect, an embodiment of the present application provides a heart rate information obtaining method, where the method includes:
obtaining a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image;
dividing the human face interesting region in each frame image into a plurality of sub-images; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence;
extracting a heart rate sub-signal of each sub-image sequence in a time domain;
obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value;
determining a photoplethysmography (PPG) signal representing pulse information in a video according to the frequency domain weight, the space domain weight and the heart rate sub-signal of each sub-image sequence;
and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
In one embodiment, the obtaining the pulse rate approximation of the target object in the video includes:
acquiring a human face region of interest of each frame of image in the video;
carrying out chrominance feature extraction and average value calculation on each face region of interest to obtain a pixel chrominance feature average value;
performing band-pass filtering on the pixel chrominance characteristic average value to obtain a rough estimation heart rate signal;
and taking the frequency component with the maximum power spectral density of the roughly estimated heart rate signal as the approximate pulse rate value of the target image.
In one embodiment, the acquiring a region of interest of a human face of each frame of image in the video includes:
acquiring a previous face interesting region and a plurality of previous key points in a previous frame of image in the video;
carrying out face detection on a current frame image in the video to obtain a plurality of current key points of the current image; the previous key point and the current key point are in one-to-one correspondence;
obtaining the offset matrixes of the previous frame image and the current frame image according to the coordinate offset of the previous key point and the corresponding current key point;
determining a current face interesting region in the current frame image according to the previous face interesting region and the offset matrix;
when the previous frame image is a first frame image, the acquiring a previous face region of interest and a plurality of previous key points in the previous frame image in the video includes:
and carrying out face detection on the previous frame image to obtain the previous face interesting area and a plurality of previous key points of the previous frame image.
In one embodiment, the obtaining a frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximation includes:
acquiring the power spectral density of each heart rate sub-signal;
determining a power integration range according to the pulse rate approximate value and a preset integration offset;
determining effective spectral energy within the power integration range according to the power spectral density of the heart rate sub-signal and the power integration range;
determining spectral energy in a pass band according to the power spectral density and the pass band range of the heart rate sub-signal;
and taking the ratio of the effective spectral energy to the difference between the spectral energy in the passband and the effective spectral energy as the frequency domain weight of each sub-image sequence.
In one embodiment, the determining a photoplethysmography (PPG) signal characterizing pulse information in a video according to the frequency domain weight, the spatial domain weight, and the heart rate sub-signal of each sub-image sequence includes:
and weighting and summing each heart rate sub-signal according to the frequency domain weight and the space domain weight corresponding to the sub-image sequence to obtain the PPG signal.
In one embodiment, the spatial weight is determined according to the image spatial gradient and a preset adjusting parameter.
In one embodiment, the determining a heart rate fine estimation value from a frequency corresponding to a maximum power spectral density of the PPG signal within a preset time period includes:
taking a frequency corresponding to the maximum power spectral density of the PPG signal within a preset time period as a target frequency;
and determining the preset multiple of the target frequency as the fine heart rate estimation value.
In a second aspect, an embodiment of the present application provides a heart rate information obtaining apparatus, where the apparatus includes:
the acquisition module is used for acquiring a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image;
the extraction module is used for dividing a human face interesting region in each frame of image in the video into a plurality of sub-images, extracting a heart rate sub-signal of each sub-image sequence in a time domain, and obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence;
and the processing module is used for determining a PPG signal representing pulse information in a video according to the frequency domain weight, the spatial domain weight and the heart rate sub-signal of each sub-image sequence, and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
In a third aspect, an embodiment of the present application provides a computer device, including a memory and a processor, where the memory stores a computer program, and the processor implements the following steps when executing the computer program:
obtaining a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image;
dividing the human face interesting region in each frame image into a plurality of sub-images; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence;
extracting a heart rate sub-signal of each sub-image sequence in a time domain;
obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value;
determining a photoplethysmography (PPG) signal representing pulse information in a video according to the frequency domain weight, the space domain weight and the heart rate sub-signal of each sub-image sequence;
and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the following steps:
obtaining a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image;
dividing the human face interesting region in each frame image into a plurality of sub-images; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence;
extracting a heart rate sub-signal of each sub-image sequence in a time domain;
obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value;
determining a photoplethysmography (PPG) signal representing pulse information in a video according to the frequency domain weight, the space domain weight and the heart rate sub-signal of each sub-image sequence;
and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
According to the heart rate information acquisition method, the heart rate information acquisition device, the computer equipment and the storage medium, the computer equipment is used for acquiring the pulse rate approximate value of the target object in the video, the human face interesting region in each frame image is divided into the sub-images, each sub-image and the sub-images corresponding to the positions in other frame images form a sub-image sequence according to the time sequence, and the computer equipment can extract the heart rate sub-signals of each sub-image sequence in the time domain and representing the characteristic changes of different regions. And then the computer equipment obtains the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value, and the pulse rate approximate value is obtained by carrying out feature extraction on the basis of the whole video image, so that the result of rough estimation on the heart rate features can be represented, and the obtained frequency domain weight can be combined with the result of the rough estimation. Meanwhile, each sub-image sequence corresponds to a frequency domain weight, the characteristics of different regions can be weighted based on the frequency domain weight of each sub-image sequence, the specific gravity of a region of interest is highlighted, the specific gravity of a no-center region is reduced, and fine estimation of an image is realized.
Drawings
FIG. 1 is a diagram illustrating an internal structure of a computer device according to an embodiment;
fig. 2 is a schematic flow chart of a method for acquiring heart rate information according to an embodiment;
FIG. 2a is a schematic diagram of a region of interest of a human face in one embodiment;
fig. 2b is a schematic diagram illustrating division of a region of interest of a human face according to an embodiment;
fig. 3 is a schematic flow chart of a heart rate information obtaining method according to another embodiment;
fig. 4 is a schematic flow chart of a heart rate information obtaining method according to yet another embodiment;
fig. 5 is a schematic flow chart of a heart rate information obtaining method according to yet another embodiment;
fig. 5a is a schematic flow chart of a heart rate information obtaining method according to yet another embodiment;
fig. 6 is a schematic structural diagram of a heart rate information acquisition device according to an embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
The heart rate information acquisition method provided by the embodiment of the application can be suitable for the computer equipment shown in fig. 1. The computer device comprises a processor, a memory, a network interface, a database, a display screen and an input device which are connected through a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for storing videos in the following embodiments, and specific descriptions of the videos are provided in the following embodiments. The network interface of the computer device may be used to communicate with other devices outside over a network connection. Optionally, the computer device may be a server, a desktop, a personal digital assistant, other terminal devices such as a tablet computer, a mobile phone, and the like, or a cloud or a remote server, and the specific form of the computer device is not limited in the embodiment of the present application. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like. Of course, the input device and the display screen may not belong to a part of the computer device, and may be external devices of the computer device.
Those skilled in the art will appreciate that the architecture shown in fig. 1 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
The following describes the technical solutions of the present application and how to solve the above technical problems with specific examples. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
It should be noted that the execution subject of the method embodiments described below may be a heart rate information obtaining apparatus, which may be implemented by software, hardware, or a combination of software and hardware as part or all of the above computer device. The following method embodiments are described by taking the execution subject as the computer device as an example.
Fig. 2 is a schematic flow chart of a heart rate information obtaining method according to an embodiment. The embodiment relates to a specific process for automatically estimating the heart rate information of a target object through a video image by a computer device. As shown in fig. 2, includes:
s10, obtaining the pulse rate approximate value of the target object in the video; the pulse rate approximate value is obtained by performing feature extraction based on the whole video image.
Specifically, the computer device may read the pulse rate approximate value of the target object in the video in the memory, or may receive the pulse rate approximate value sent by other devices, and the computer device may also obtain the video, and perform feature extraction and identification processing on the frame image in the video to obtain the pulse rate approximate value. The video comprises a plurality of time-continuous frame images, and is obtained by shooting a target object. Alternatively, the pulse rate approximation may be obtained by performing feature extraction based on the entire video image and using an algorithm to estimate the pulse rate approximation according to the change of the frame image with time. Optionally, the computer device may further identify a region of interest (ROI) in each frame of image by using a preset identification model, for example, identify a face region in the person image, and perform feature extraction on the ROI to obtain a pulse rate approximate value.
S20, dividing the human face interesting region in each frame image into a plurality of sub-images; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence.
Specifically, the computer device divides the face region of interest in each frame image, and each frame image is divided to obtain a plurality of sub-images. Optionally, the division mode may be division according to well-type grids, the number of the well-type grids is not limited in this embodiment, and the division mode may be set according to required accuracy. Fig. 2a is a schematic diagram of a region of interest of a face in an embodiment, optionally, the division manner may be as shown in fig. 2b, and the division manner shown in fig. 2b is only an example and does not limit the present application. It should be noted that any one of the sub-images obtained by dividing the face roi of each frame of image corresponds to a sub-image of any other frame of image, that is, the dividing manner of the face roi for dividing each frame of image may be the same, so that the sub-images obtained by dividing the face roi of all the frame of images may be arranged according to the corresponding positions and time sequence, respectively, to obtain a plurality of sub-image sequences. Each sub-image sequence comprises a plurality of temporally successive sub-images.
And S30, extracting the heart rate sub-signals of each sub-image sequence in the time domain.
Specifically, the computer device may calculate, based on the intensity signal of the face video, a heart rate sub-signal of the chrominance feature of each sub-image sequence in the time domain, which may be denoted by V (x, y, t), and may be generally denoted by ci (t), where i represents a reference number of the sub-image sequence, and i is a positive integer to represent a feature of which sub-image sequence the heart rate sub-signal characterizes.
And S40, obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value.
Specifically, the computer device obtains the frequency domain weight of the sub-image sequence corresponding to each heart rate sub-signal according to the functional relationship between each heart rate sub-signal and the pulse rate approximation value. Alternatively, the power spectral density of the heart rate sub-signal within a period of time [0, T ] can be calculated, and then the signal-to-noise ratio between the spectral energy of the power spectral density within the region near the pulse rate and the spectral energy within the pass band can be calculated as the frequency domain weight according to the spectral energy of the power spectral density within the region near the pulse rate and in combination with the preset pass band range.
And S50, determining a photoplethysmography (PPG) signal representing pulse information in the video according to the frequency domain weight, the spatial domain weight and the heart rate sub-signal of each sub-image sequence.
Specifically, the computer device may perform weighted summation according to the frequency domain weight of each sub-image sequence, the spatial domain weight of each sub-image sequence, and the heart rate sub-signal of each sub-image sequence, so as to obtain a photoplethysmography (PPG) signal representing pulse information in the video.
And S60, determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
Specifically, the computer device performs fast fourier transform on the PPG signal, performs power spectral density analysis to obtain a frequency corresponding to the maximum power, and further calculates a heart rate fine estimation value according to the frequency corresponding to the maximum power and a certain functional relationship, for example, the heart rate fine estimation value may be multiplied by a corresponding coefficient or a certain offset is superimposed.
In this embodiment, the computer device obtains the pulse rate approximate value of the target object in the video, and divides the region of interest in each frame of image into a plurality of sub-images, and because each sub-image and the sub-images corresponding to the positions in the other frame of images form a sub-image sequence according to the time sequence, the computer device can extract a plurality of heart rate sub-signals of each sub-image sequence in the time domain and representing the characteristic changes of different regions. And then the computer equipment obtains the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value, and the pulse rate approximate value is obtained by carrying out feature extraction on the basis of the whole video image, so that the result of rough estimation on the heart rate features can be represented, and the obtained frequency domain weight can be combined with the result of the rough estimation. Meanwhile, each sub-image sequence corresponds to a frequency domain weight, the characteristics of different regions can be weighted based on the frequency domain weight of each sub-image sequence, the specific gravity of a region of interest is highlighted, the specific gravity of a no-center region is reduced, and fine estimation of an image is realized.
Optionally, on the basis of the foregoing embodiment, a possible implementation manner of the foregoing S10 may be as shown in fig. 3, and includes:
and S11, acquiring the human face interesting region of each frame of image in the video.
Specifically, the computer device may perform face detection on each frame of image in the video to obtain a face roi of each frame of image, for example, the face roi may be obtained by detecting the image with an artificial neural network model, or the face roi of the next frame of image is estimated according to a detection result of the previous frame of image with a tracking algorithm, which is not limited in this embodiment.
And S12, extracting the chrominance feature of each human face region of interest and calculating the average value to obtain the pixel chrominance feature average value.
Specifically, the computer equipment adopts the face movement robustness to extract the judgment chrominance characteristic of the region of interest of the face. That is, the angle between the incident light source and the face changes due to the face motion, which causes the brightness and color of the face to change, and the image caused by the face motion can be weakened by the extraction of the chromaticity characteristics. Specifically, the chromaticity characteristics may be calculated by combining R, G, B three channels, and the formula C ═ X may be adoptedf-αYfOr a variation of this formula, where α ═ σ (Xf)/σ (Yf), and Xf and Yf can be band-pass filtered by X and Y, and the pixel chrominance feature average c is calculated by extracting chrominance featuresThe signal of (a), wherein X ═ 3Rn-2Gn, Y ═ 1.5Rn + Gn-1.5Bn, where Rn ═ R/μ (R), Gn ═ G/μ (G), and Bn ═ B/μ (B) are normalized color channels. Since the influence of expression changes such as chewing on non-rigid movements is the same in the three color channels, extraction of the colorimetric features in the above manner can cancel out the interference caused by non-rigid movements by the characteristics of the different color channels.
And S13, performing band-pass filtering on the pixel chrominance characteristic average value to obtain a rough estimation heart rate signal.
Specifically, the computer device may perform band-pass filtering on the average value of the chrominance features of the pixels, and optionally, the bandwidth of the band-pass filtering may be selected to be [0.5Hz,5Hz [ ]]To obtain a coarse estimated heart rate signal, which can be used
And (4) showing. Optionally, other bandwidths may be selected for filtering, which is not limited in this embodiment.
And S14, taking the frequency component with the maximum power spectral density of the roughly estimated heart rate signal as the approximate value of the pulse rate of the target image.
Specifically, since the PPG signal reflects the trend of the pulse, the PPG signal includes a fundamental oscillation frequency similar to the pulse rate, and therefore the computer device obtains the above-mentioned roughly estimated heart rate signal for a period of time [0, T]Inner power spectral density, and then the largest frequency component in the power spectrum is taken as the pulse rate PR approximation. Alternatively, a formula may be used
Or the approximate value of PR is obtained by the deformation calculation of the formula. Wherein
Is composed of
At [0, T]Power Spectral Density (PSD) over time.
In this embodiment, a face region of interest of each frame of image in the video is obtained, and chrominance feature extraction and average value calculation are performed on each face region of interest to obtain an average value of pixel chrominance features, so that interference generated by non-rigid motion is offset through mutual operation of different color channels. And finally, the frequency component with the maximum power spectral density of the roughly estimated heart rate signal is used as the pulse rate approximate value of the target image, and the method can remove the interference generated by non-rigid motion from the obtained pulse rate approximate value, so that the obtained pulse rate approximate value is more accurate, and further the heart rate information is more accurate.
Optionally, one possible implementation manner of step S11 in the foregoing embodiment may be as shown in fig. 4, and includes:
and S111, acquiring a previous face interesting region and a plurality of previous key points in a previous frame of image in the video.
S112, carrying out face detection on the current frame image in the video to obtain a plurality of current key points of the current image; and the previous key point and the current key point are in one-to-one correspondence.
Specifically, the computer device may obtain a previous face region of interest and a plurality of previous key points in a previous frame of image, perform face detection on a current frame of image, identify positions of local key organs of the face, including eyebrow, eye, nose, mouth, and cheek, by using a Discriminant Response Map Fitting (DRMF) method, and mark the plurality of key points. It may be typical to mark 66 facial keypoints for subsequent point tracking processing. The schematic diagram of the face key points can be seen from fig. 2a, and the key points are denoted by "+". It should be noted that, the manner of acquiring the keypoints of the previous frame of image and the manner of acquiring the current keypoints of the computer device may be the same, and the number and the category of the keypoints acquired by each frame are consistent, that is, the previous keypoints and the current keypoints are in one-to-one correspondence.
S113, obtaining the offset matrixes of the previous frame image and the current frame image according to the coordinate offset of the previous key point and the corresponding current key point.
And S114, determining the current face interesting region in the current frame image according to the previous face interesting region and the offset matrix.
Specifically, the computer device obtains offset matrices of a previous frame image and a current frame image according to coordinate offset representing coordinate changes of a previous key point and a corresponding current key point, and then superimposes the offset matrices according to the position of a previous face region of interest, so as to obtain a current face region of interest in the current frame image.
When the previous frame image is a first frame image, the acquiring a previous face region of interest and a plurality of previous key points in the previous frame image in the video includes: and carrying out face detection on the previous frame image to obtain the previous face interesting area and a plurality of previous key points of the previous frame image. Starting from the second frame image, other face interesting regions can be obtained in a tracking mode.
When the face generates rigid motion such as inclination, shaking and the like, the key points detected by the DRMF method are tracked under a Kanade-Lucas-Tomasi (KLT) frame, the displacement change of the key points of the face along with time is calculated, namely the coordinate change of the corresponding key points in the adjacent frame images, and the face interesting region of the back frame image is accurately obtained according to the face interesting region of the front frame image through matrix transformation. Typically, this region of face interest may be a rectangular box. First, a tracking process is initialized with a first frame image and its corresponding keypoints, and then, for a t-th (t ═ 2,3,4 …) frame image, the positions of the keypoints are tracked by using the KLT algorithm, which is expressed as: dt ═ d1(t), d2(t), …, dm (t) ]. Wherein m is the number of key points and is 1-66; at least three points can ensure that dm (t) is a vector representing the coordinates of the mth key point in the tth frame image. An approximate 2-dimensional affine transformation exists between the feature points of two adjacent frame images, namely Dt is TtDt-1, wherein Tt is an offset matrix. Therefore, the shift matrix Tt is estimated by using a random sampling consistency algorithm, and is applied to the previous frame of face frame to update 4 vertexes of the frame: and Ft is TtFt-1, wherein the Ft is [ f1(t), f2(t), f3(t), f4(t) ], f1(t) to f4(t) respectively represent the 1 st to 4 th vertex coordinates of the face frame of the t-th frame image.
In this embodiment, the computer device obtains a previous face region of interest and a plurality of previous key points in a previous frame of image in the video, and performs face detection on a current frame of image in the video to obtain a plurality of current key points of the current image. Because the previous key point corresponds to the current key point one by one, the computer equipment can obtain the offset matrixes of the previous frame image and the current frame image according to the coordinate offset of the previous key point and the corresponding current key point, and accurately determine the current face interesting area in the current frame image in a tracking mode according to the previous face interesting area and the offset matrix. When the previous frame image is the first frame image, the computer device performs face detection on the previous frame image to obtain a previous face interesting region and a plurality of previous key points of the previous frame image, so that the face interesting region acquired in a detection and tracking mode can greatly eliminate rigid motion, such as inclination and shaking, and further the acquired face interesting region is more accurate, and further the heart rate information is more accurate.
Optionally, on the basis of the foregoing embodiment, one possible implementation manner of step S50 may include: acquiring the power spectral density of each heart rate sub-signal; determining a power integration range according to the pulse rate approximate value and a preset integration offset; determining effective spectral energy within the power integration range according to the power spectral density of the heart rate sub-signal and the power integration range; determining spectral energy in a pass band according to the power spectral density and the pass band range of the heart rate sub-signal; and taking the ratio of the effective spectral energy to the difference between the spectral energy in the passband and the effective spectral energy as the frequency domain weight of each sub-image sequence. In particular, the computer device may acquire each heart rate sub-signal
Power spectral density of
Then, according to the pulse rate approximate value PR and the preset integral offset h, determining the power integral range [ PR-h, PR + h]Then according to
In [ PR-h, PR + h]And carrying out integration to obtain the effective spectral energy in the power integration range. Then according to the heart rate sub-signal
Power spectral density of
And passband range [ B1,B2]Determining the spectral energy within the passband, i.e.
In [ B ]1,B2]And (4) carrying out internal integration to obtain the spectral energy in the passband. Using the ratio of the effective spectral energy to the difference between the spectral energy in the pass band and the effective spectral energy as the frequency domain weight of each sub-image sequence, namely using the formula
Or the frequency domain weight w of each ith sub-image sequence is obtained by the deformation of the formulai。
In general, let the common notation C ═ Xf-αYfUnfolding the filtered signals of different sub-image sequences as:
where i ∈ {1,2 …, n } denotes the number of sub-image sequences, i.e. the number of sub-images in each ROI region, Ai denotes the potential PPG signal strength in sub-image Ri, determined by the pulse volume modulation parameter α i and the incident light intensity Ii, p (t) is the pulse volume variation signal, vi (t) represents the noise contribution caused by camera quantization, surface reflections and motion artifacts.
Which can be considered as p (t) signals received via different color channels, have different intensities and noise levels. Weighting and averaging signals of different channels to obtain blood volume pulsation change signal
Weight w of each channeliThe method can be calculated by a maximum pen diversity algorithm. The maximum ratio diversity algorithm indicates that: the assigned weights need to be proportional to the Root Mean Square (RMS) value of the signal component and inversely proportional to the mean square noise of the channel, so that the maximum can be achieved
The weight formula can generally be:
where Ai and vi are positions and therefore can be given to wiAn estimation is performed. Since the spectral energy of the PPG signal is concentrated in the pass band around PR, while the spectral energy of the noise vi (t) in ci (t) is distributed in the pass band of the band-pass filter [0.5Hz,5Hz ]]In addition, therefore, based on the spectral structure of the signal, the ratio of the spectral energy near PR to the spectral energy of the noise outside the passband can be used to approximate wiSee, in particular, the foregoing. Due to the weight wiFrequency dependent, then frequency weights can be written, i.e.
In this embodiment, the computer device obtains the power spectral density of each heart rate sub-signal; according to the pulse rate approximate value and the preset integral offset, a power integral range is determined, effective spectral energy in the power integral range is determined according to the power spectral density and the power integral range of the heart rate sub-signal, then spectral energy in a pass band is determined according to the power spectral density and the pass band range of the heart rate sub-signal, and finally the ratio of the effective spectral energy to the difference between the spectral energy in the pass band and the effective spectral energy is used as the frequency domain weight of each sub-image sequence.
Optionally, on the basis of the foregoing embodiment, one possible implementation manner of the foregoing step S50 may include: and weighting and summing each heart rate sub-signal according to the frequency domain weight and the space domain weight corresponding to the sub-image sequence to obtain the PPG signal. Specifically, the computer device may weight the heart rate sub-signal of each sub-image sequence according to a frequency domain weight and a spatial domain weight, that is, multiply the three, and finally, accumulate all the weighting results corresponding to each sub-image sequence to obtain a PPG signal representing pulse information in the video. Alternatively, formulas may be employed
Or a variant of this formula, the PPG signal s (t) is calculated. Wherein the content of the first and second substances,
representing the spatial weight of the ith sub-image sequence,
representing the spatial weight of the ith sub-image sequence,
a heart rate sub-signal representing a first sub-image sequence. In this embodiment, the PPG signal is obtained by performing weighted summation on each heart rate sub-signal according to the frequency domain weight and the spatial domain weight corresponding to the sub-image sequence, so that the features of different regions can be weighted based on the frequency domain weight of each sub-image sequence, and the region of interest is highlightedThe specific gravity of the image is reduced, regardless of the specific gravity of the middle area, and fine estimation of the image is achieved, therefore, a PPG signal representing pulse information in a video is determined through the frequency domain weight, the spatial domain weight and the heart rate sub-signal of each sub-image sequence, a fine heart rate estimation value is determined according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period, combination of coarse estimation and fine estimation is further achieved, non-rigid interference, such as interference generated by chewing, expression change and the like, can be eliminated, and interference of non-rigid movement is eliminated, so that the obtained fine heart rate estimation value is more reasonable and accurate.
Optionally, on the basis of the foregoing embodiments, the spatial weight is determined according to the image spatial gradient and a preset adjustment parameter. In particular, the computer device may employ a formula
Or the space domain weight is obtained by the deformation of the formula
The ▽ Lambda represents the gradient of the image Lambda, Lambda is an adjusting parameter, the small value of the differential operator of the ▽ Lambda is strengthened by using a negative exponential form, so that the spatial domain weight distributed to the sub-images with small spatial gradient is large, the spatial domain weight distributed to the sub-images with large spatial gradient is small, and therefore signals of a face flat region are selected more, and the influence degree described by the spatial domain weight is more reasonable and accurate.
Optionally, on the basis of the foregoing embodiments, one possible obtaining manner of the step S60 may include: taking a frequency corresponding to the maximum power spectral density of the PPG signal within a preset time period as a target frequency; and determining the preset multiple of the target frequency as the fine heart rate estimation value. In particular, the computer device maps the maximum power spectral density of the PPG signal within a preset time periodCorresponding frequency as target frequency fHRAnd determining the preset multiple of the target frequency as the heart rate fine estimation value HR. Alternatively, the preset multiple may be 60, that is, the formula HR-60 f is adoptedHROr a variant of this formula results in a fine estimate HR of the heart rate. Optionally, the preset multiple may also be other values, such as 61, 62, or 58, 59, etc., which is not limited in this embodiment. Wherein the PPG signal and the heart rate fine estimate respectively characterize the heart rate information from different ways. In this embodiment, a frequency corresponding to a maximum power spectral density of the PPG signal within a preset time period is used as a target frequency; and determining the preset multiple of the target frequency as the heart rate fine estimation value, so that the conversion between the PPG signal and the heart rate fine estimation value is realized, the expression of the estimation result is richer, and the user identification is facilitated.
For a more detailed description of the method of the embodiments of the present application, reference may be made to the flow chart illustrated in fig. 5. And carrying out face detection and tracking on the frame image in the video of the computer equipment to obtain an ROI (region of interest) of each frame image. On one hand, the computer equipment calculates the chrominance feature C, namely the pixel chrominance feature average value, by combining with the RGB channel, and further performs credit-pass filtering to obtain a rough estimation heart rate signal, performs Fourier transform on the rough estimation heart rate signal, obtains the maximum peak frequency, obtains the power spectral density of the signal, and obtains the frequency domain energy weight, namely the frequency domain weight by calculating according to the power spectral density. On the other hand, the computer device divides the ROI into grids to obtain image blocks of the grids, namely sub-images, and calculates spatial gradient weight, namely spatial weight according to the sub-images. Meanwhile, the heart lining sub-signals of the sub-images are subjected to band-pass filtering, and the accurately estimated heart rate signals are obtained by combining the spatial gradient weight and the frequency domain energy weight, so that the accurately estimated heart rate value is obtained.
The results of using the two-step heart rate estimation described above can be seen in fig. 5 a.
It should be understood that although the various steps in the flow charts of fig. 2-5 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least some of the steps in fig. 2-5 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternating with other steps or at least some of the sub-steps or stages of other steps.
In one embodiment, as shown in fig. 6, there is provided a heart rate information acquiring apparatus including:
an obtaining module 100, configured to obtain a pulse rate approximation value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image;
an extracting module 200, configured to divide a region of interest of a human face in each frame of image in the video into a plurality of sub-images, extract a heart rate sub-signal of each sub-image sequence in a time domain, and obtain a frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximation value; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence;
a processing module 300, configured to determine, according to the frequency domain weight, the spatial domain weight, and the heart rate sub-signal of each sub-image sequence, a PPG signal representing pulse information in a video, and determine a heart rate fine estimation value from a frequency corresponding to a maximum power spectral density of the PPG signal within a preset time period.
In an embodiment, the obtaining module 100 is specifically configured to obtain a face region of interest of each frame of image in the video; carrying out chrominance feature extraction and average value calculation on each face region of interest to obtain a pixel chrominance feature average value; performing band-pass filtering on the pixel chrominance characteristic average value to obtain a rough estimation heart rate signal; and taking the frequency component with the maximum power spectral density of the roughly estimated heart rate signal as the approximate pulse rate value of the target image.
In an embodiment, the obtaining module 100 is specifically configured to obtain a previous face region of interest and a plurality of previous key points in a previous frame of image in the video; carrying out face detection on a current frame image in the video to obtain a plurality of current key points of the current image; the previous key point and the current key point are in one-to-one correspondence; obtaining the offset matrixes of the previous frame image and the current frame image according to the coordinate offset of the previous key point and the corresponding current key point; determining a current face interesting region in the current frame image according to the previous face interesting region and the offset matrix; when the previous frame image is a first frame image, performing face detection on the previous frame image to obtain the previous face interesting area of the previous frame image and a plurality of previous key points.
In one embodiment, the extracting module 200 is specifically configured to obtain a power spectral density of each of the heart rate sub-signals; determining a power integration range according to the pulse rate approximate value and a preset integration offset; determining effective spectral energy within the power integration range according to the power spectral density of the heart rate sub-signal and the power integration range; determining spectral energy in a pass band according to the power spectral density and the pass band range of the heart rate sub-signal; and taking the ratio of the effective spectral energy to the difference between the spectral energy in the passband and the effective spectral energy as the frequency domain weight of each sub-image sequence.
In an embodiment, the processing module 300 is specifically configured to perform weighted summation on each heart rate sub-signal according to the frequency domain weight and the spatial domain weight corresponding to the sub-image sequence to obtain the PPG signal.
In one embodiment, the spatial weight is determined according to the image spatial gradient and a preset adjustment parameter.
In an embodiment, the processing module 300 is specifically configured to use, as the target frequency, a frequency corresponding to a maximum power spectral density of the PPG signal within a preset time period; and determining the preset multiple of the target frequency as the fine heart rate estimation value.
For specific limitations of the heart rate information acquisition device, reference may be made to the above limitations of the heart rate information acquisition method, which are not described herein again. The modules in the heart rate information acquisition device can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, comprising a memory and a processor, the memory having a computer program stored therein, the processor implementing the following steps when executing the computer program:
obtaining a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image; dividing the human face interesting region in each frame image into a plurality of sub-images; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence; extracting a heart rate sub-signal of each sub-image sequence in a time domain; obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value; determining a photoplethysmography (PPG) signal representing pulse information in a video according to the frequency domain weight, the space domain weight and the heart rate sub-signal of each sub-image sequence; and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring a human face region of interest of each frame of image in the video;
carrying out chrominance feature extraction and average value calculation on each face region of interest to obtain a pixel chrominance feature average value;
performing band-pass filtering on the pixel chrominance characteristic average value to obtain a rough estimation heart rate signal;
and taking the frequency component with the maximum power spectral density of the roughly estimated heart rate signal as the approximate pulse rate value of the target image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring a previous face interesting region and a plurality of previous key points in a previous frame of image in the video;
carrying out face detection on a current frame image in the video to obtain a plurality of current key points of the current image; the previous key point and the current key point are in one-to-one correspondence;
obtaining the offset matrixes of the previous frame image and the current frame image according to the coordinate offset of the previous key point and the corresponding current key point;
determining a current face interesting region in the current frame image according to the previous face interesting region and the offset matrix;
when the previous frame image is a first frame image, the acquiring a previous face region of interest and a plurality of previous key points in the previous frame image in the video includes:
and carrying out face detection on the previous frame image to obtain the previous face interesting area and a plurality of previous key points of the previous frame image.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
acquiring the power spectral density of each heart rate sub-signal;
determining a power integration range according to the pulse rate approximate value and a preset integration offset;
determining effective spectral energy within the power integration range according to the power spectral density of the heart rate sub-signal and the power integration range;
determining spectral energy in a pass band according to the power spectral density and the pass band range of the heart rate sub-signal;
and taking the ratio of the effective spectral energy to the difference between the spectral energy in the passband and the effective spectral energy as the frequency domain weight of each sub-image sequence.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
and weighting and summing each heart rate sub-signal according to the frequency domain weight and the space domain weight corresponding to the sub-image sequence to obtain the PPG signal.
In one embodiment, the spatial weight is determined according to the image spatial gradient and a preset adjustment parameter.
In one embodiment, the processor, when executing the computer program, further performs the steps of:
taking a frequency corresponding to the maximum power spectral density of the PPG signal within a preset time period as a target frequency;
and determining the preset multiple of the target frequency as the fine heart rate estimation value.
It should be clear that, in the embodiments of the present application, the process of executing the computer program by the processor is consistent with the process of executing the steps in the above method, and specific reference may be made to the description above.
In one embodiment, a computer-readable storage medium is provided, having a computer program stored thereon, which when executed by a processor, performs the steps of:
obtaining a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image; dividing the human face interesting region in each frame image into a plurality of sub-images; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence; extracting a heart rate sub-signal of each sub-image sequence in a time domain; obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value; determining a photoplethysmography (PPG) signal representing pulse information in a video according to the frequency domain weight, the space domain weight and the heart rate sub-signal of each sub-image sequence; and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring a human face region of interest of each frame of image in the video;
carrying out chrominance feature extraction and average value calculation on each face region of interest to obtain a pixel chrominance feature average value;
performing band-pass filtering on the pixel chrominance characteristic average value to obtain a rough estimation heart rate signal;
and taking the frequency component with the maximum power spectral density of the roughly estimated heart rate signal as the approximate pulse rate value of the target image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring a previous face interesting region and a plurality of previous key points in a previous frame of image in the video;
carrying out face detection on a current frame image in the video to obtain a plurality of current key points of the current image; the previous key point and the current key point are in one-to-one correspondence;
obtaining the offset matrixes of the previous frame image and the current frame image according to the coordinate offset of the previous key point and the corresponding current key point;
determining a current face interesting region in the current frame image according to the previous face interesting region and the offset matrix;
when the previous frame image is a first frame image, the acquiring a previous face region of interest and a plurality of previous key points in the previous frame image in the video includes:
and carrying out face detection on the previous frame image to obtain the previous face interesting area and a plurality of previous key points of the previous frame image.
In one embodiment, the computer program when executed by the processor further performs the steps of:
acquiring the power spectral density of each heart rate sub-signal;
determining a power integration range according to the pulse rate approximate value and a preset integration offset;
determining effective spectral energy within the power integration range according to the power spectral density of the heart rate sub-signal and the power integration range;
determining spectral energy in a pass band according to the power spectral density and the pass band range of the heart rate sub-signal;
and taking the ratio of the effective spectral energy to the difference between the spectral energy in the passband and the effective spectral energy as the frequency domain weight of each sub-image sequence.
In one embodiment, the computer program when executed by the processor further performs the steps of:
and weighting and summing each heart rate sub-signal according to the frequency domain weight and the space domain weight corresponding to the sub-image sequence to obtain the PPG signal.
In one embodiment, the spatial weight is determined according to the image spatial gradient and a preset adjustment parameter.
In one embodiment, the computer program when executed by the processor further performs the steps of:
taking a frequency corresponding to the maximum power spectral density of the PPG signal within a preset time period as a target frequency;
and determining the preset multiple of the target frequency as the fine heart rate estimation value.
It should be clear that, in the embodiments of the present application, the process of executing the computer program by the processor is consistent with the process of executing the steps in the above method, and specific reference may be made to the description above.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by hardware instructions of a computer program, which can be stored in a non-volatile computer-readable storage medium, and when executed, can include the processes of the embodiments of the methods described above. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the above embodiments can be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the above embodiments are not described, but should be considered as the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present application, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the concept of the present application, which falls within the scope of protection of the present application. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (10)
1. A heart rate information acquisition method, characterized in that the method comprises:
obtaining a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image;
dividing the human face interesting region in each frame image into a plurality of sub-images; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence;
extracting a heart rate sub-signal of each sub-image sequence in a time domain;
obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value;
determining a photoplethysmography (PPG) signal representing pulse information in a video according to the frequency domain weight, the space domain weight and the heart rate sub-signal of each sub-image sequence;
and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
2. The method of claim 1, wherein obtaining the pulse rate approximation of the target object in the video comprises:
acquiring a human face region of interest of each frame of image in the video;
carrying out chrominance feature extraction and average value calculation on each face region of interest to obtain a pixel chrominance feature average value;
performing band-pass filtering on the pixel chrominance characteristic average value to obtain a rough estimation heart rate signal;
and taking the frequency component with the maximum power spectral density of the roughly estimated heart rate signal as the approximate pulse rate value of the target image.
3. The method according to claim 2, wherein the obtaining the region of interest of the face of each frame of image in the video comprises:
acquiring a previous face interesting region and a plurality of previous key points in a previous frame of image in the video;
carrying out face detection on a current frame image in the video to obtain a plurality of current key points of the current image; the previous key point and the current key point are in one-to-one correspondence;
obtaining the offset matrixes of the previous frame image and the current frame image according to the coordinate offset of the previous key point and the corresponding current key point;
determining a current face interesting region in the current frame image according to the previous face interesting region and the offset matrix;
when the previous frame image is a first frame image, the acquiring a previous face region of interest and a plurality of previous key points in the previous frame image in the video includes:
and carrying out face detection on the previous frame image to obtain the previous face interesting area and a plurality of previous key points of the previous frame image.
4. The method of claim 1, wherein the deriving a frequency domain weight for each of the sub-image sequences from each of the heart rate sub-signals and the pulse rate approximation comprises:
acquiring the power spectral density of each heart rate sub-signal;
determining a power integration range according to the pulse rate approximate value and a preset integration offset;
determining effective spectral energy within the power integration range according to the power spectral density of the heart rate sub-signal and the power integration range;
determining spectral energy in a pass band according to the power spectral density and the pass band range of the heart rate sub-signal;
and taking the ratio of the effective spectral energy to the difference between the spectral energy in the passband and the effective spectral energy as the frequency domain weight of each sub-image sequence.
5. The method according to claim 1, wherein determining a photoplethysmography (PPG) signal characterizing pulse information in video from the frequency domain weights, spatial weights and the heart rate sub-signals of each of the sub-image sequences comprises:
and weighting and summing each heart rate sub-signal according to the frequency domain weight and the space domain weight corresponding to the sub-image sequence to obtain the PPG signal.
6. The method according to any one of claims 1 to 5, wherein the spatial weight is determined according to the image spatial gradient and a preset adjustment parameter.
7. The method according to any one of claims 1 to 4, wherein the determining a heart rate fine estimate of the frequency to which the maximum power spectral density of the PPG signal is mapped within a preset time period comprises:
taking a frequency corresponding to the maximum power spectral density of the PPG signal within a preset time period as a target frequency;
and determining the preset multiple of the target frequency as the fine heart rate estimation value.
8. A heart rate information acquisition apparatus, characterized in that the apparatus comprises:
the acquisition module is used for acquiring a pulse rate approximate value of a target object in a video; the pulse rate approximate value is obtained by extracting features based on the whole video image;
the extraction module is used for dividing a human face interesting region in each frame of image in the video into a plurality of sub-images, extracting a heart rate sub-signal of each sub-image sequence in a time domain, and obtaining the frequency domain weight of each sub-image sequence according to each heart rate sub-signal and the pulse rate approximate value; any sub-image of each frame image corresponds to a sub-image position of any other frame image, and each sub-image and the sub-images corresponding to the positions in the other frame images form a sub-image sequence according to the time sequence;
and the processing module is used for determining a PPG signal representing pulse information in a video according to the frequency domain weight, the spatial domain weight and the heart rate sub-signal of each sub-image sequence, and determining a heart rate fine estimation value according to the frequency corresponding to the maximum power spectral density of the PPG signal in a preset time period.
9. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor implements the steps of the method of any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911371181.8A CN111134650A (en) | 2019-12-26 | 2019-12-26 | Heart rate information acquisition method and device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911371181.8A CN111134650A (en) | 2019-12-26 | 2019-12-26 | Heart rate information acquisition method and device, computer equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111134650A true CN111134650A (en) | 2020-05-12 |
Family
ID=70520737
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911371181.8A Pending CN111134650A (en) | 2019-12-26 | 2019-12-26 | Heart rate information acquisition method and device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111134650A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111743524A (en) * | 2020-06-19 | 2020-10-09 | 联想(北京)有限公司 | Information processing method, terminal and computer readable storage medium |
| CN111932510A (en) * | 2020-08-03 | 2020-11-13 | 深圳回收宝科技有限公司 | Method and device for determining image definition |
| CN111938622A (en) * | 2020-07-16 | 2020-11-17 | 启航汽车有限公司 | Heart rate detection method, device and system and readable storage medium |
| CN112017155A (en) * | 2020-07-13 | 2020-12-01 | 浙江大华汽车技术有限公司 | Method, device and system for measuring health sign data and storage medium |
| CN112766094A (en) * | 2021-01-05 | 2021-05-07 | 清华大学 | Method and system for extracting PPG signal through video |
| CN113688985A (en) * | 2021-07-26 | 2021-11-23 | 浙江大华技术股份有限公司 | Training method of heart rate estimation model, heart rate estimation method and device |
| CN114041769A (en) * | 2021-11-25 | 2022-02-15 | 深圳市商汤科技有限公司 | Heart rate measuring method and device, electronic equipment and computer readable storage medium |
| CN114557685A (en) * | 2020-11-27 | 2022-05-31 | 上海交通大学 | Non-contact motion robust heart rate measuring method and measuring device |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109480808A (en) * | 2018-09-27 | 2019-03-19 | 深圳市君利信达科技有限公司 | A kind of heart rate detection method based on PPG, system, equipment and storage medium |
| CN109584213A (en) * | 2018-11-07 | 2019-04-05 | 复旦大学 | A kind of selected tracking of multiple target number |
| US20190192079A1 (en) * | 2017-12-22 | 2019-06-27 | Imec Vzw | System and a method for motion artifact reduction in a ppg signal |
| CN110251075A (en) * | 2019-07-29 | 2019-09-20 | 华中科技大学同济医学院附属同济医院 | A kind of portable pupil diameter measuring instrument strutting eyelid and its measurement method |
| CN110288698A (en) * | 2019-06-25 | 2019-09-27 | 诸暨市人民医院 | Meniscus three-dimensional reconstruction system based on MRI |
-
2019
- 2019-12-26 CN CN201911371181.8A patent/CN111134650A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190192079A1 (en) * | 2017-12-22 | 2019-06-27 | Imec Vzw | System and a method for motion artifact reduction in a ppg signal |
| CN109480808A (en) * | 2018-09-27 | 2019-03-19 | 深圳市君利信达科技有限公司 | A kind of heart rate detection method based on PPG, system, equipment and storage medium |
| CN109584213A (en) * | 2018-11-07 | 2019-04-05 | 复旦大学 | A kind of selected tracking of multiple target number |
| CN110288698A (en) * | 2019-06-25 | 2019-09-27 | 诸暨市人民医院 | Meniscus three-dimensional reconstruction system based on MRI |
| CN110251075A (en) * | 2019-07-29 | 2019-09-20 | 华中科技大学同济医学院附属同济医院 | A kind of portable pupil diameter measuring instrument strutting eyelid and its measurement method |
Non-Patent Citations (1)
| Title |
|---|
| 杨昭 等: "抗运动干扰的人脸视频心率估计", 《电子与信息学报》 * |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111743524A (en) * | 2020-06-19 | 2020-10-09 | 联想(北京)有限公司 | Information processing method, terminal and computer readable storage medium |
| CN112017155B (en) * | 2020-07-13 | 2023-12-26 | 浙江华锐捷技术有限公司 | Method, device, system and storage medium for measuring health sign data |
| CN112017155A (en) * | 2020-07-13 | 2020-12-01 | 浙江大华汽车技术有限公司 | Method, device and system for measuring health sign data and storage medium |
| CN111938622A (en) * | 2020-07-16 | 2020-11-17 | 启航汽车有限公司 | Heart rate detection method, device and system and readable storage medium |
| CN111938622B (en) * | 2020-07-16 | 2022-08-30 | 启航汽车有限公司 | Heart rate detection method, device and system and readable storage medium |
| CN111932510A (en) * | 2020-08-03 | 2020-11-13 | 深圳回收宝科技有限公司 | Method and device for determining image definition |
| CN111932510B (en) * | 2020-08-03 | 2024-03-05 | 深圳回收宝科技有限公司 | Method and device for determining image definition |
| CN114557685A (en) * | 2020-11-27 | 2022-05-31 | 上海交通大学 | Non-contact motion robust heart rate measuring method and measuring device |
| CN114557685B (en) * | 2020-11-27 | 2023-11-14 | 上海交通大学 | Non-contact type exercise robust heart rate measurement method and measurement device |
| CN112766094A (en) * | 2021-01-05 | 2021-05-07 | 清华大学 | Method and system for extracting PPG signal through video |
| CN113688985A (en) * | 2021-07-26 | 2021-11-23 | 浙江大华技术股份有限公司 | Training method of heart rate estimation model, heart rate estimation method and device |
| CN114041769A (en) * | 2021-11-25 | 2022-02-15 | 深圳市商汤科技有限公司 | Heart rate measuring method and device, electronic equipment and computer readable storage medium |
| WO2023093707A1 (en) * | 2021-11-25 | 2023-06-01 | 上海商汤智能科技有限公司 | Heart rate measurement method and apparatus, electronic device and computer-readable storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111134650A (en) | Heart rate information acquisition method and device, computer equipment and storage medium | |
| US6038339A (en) | White point determination using correlation matrix memory | |
| US8842906B2 (en) | Body measurement | |
| CN112040834A (en) | Eyeball tracking method and system | |
| WO2016006027A1 (en) | Pulse wave detection method, pulse wave detection program, and pulse wave detection device | |
| JP6957929B2 (en) | Pulse wave detector, pulse wave detection method, and program | |
| US8712182B2 (en) | Image processing device, image processing method, and program | |
| KR102215557B1 (en) | Heart rate estimation based on facial color variance and micro-movement | |
| JP2006508460A (en) | Image signal processing method | |
| EP2486543A1 (en) | Formation of a time-varying signal representative of at least variations in a value based on pixel values | |
| JP6927322B2 (en) | Pulse wave detector, pulse wave detection method, and program | |
| CN113408508A (en) | Transformer-based non-contact heart rate measurement method | |
| WO2012015020A1 (en) | Method and device for image enhancement | |
| JP2021060989A (en) | Multimodal dense correspondence imaging system | |
| CN111310584A (en) | Heart rate information acquisition method and device, computer equipment and storage medium | |
| JP7032913B2 (en) | Image processing device, image processing method, computer program | |
| CN110544259A (en) | method for detecting disguised human body target under complex background based on computer vision | |
| CN111513701A (en) | Heart rate detection method and device, computer equipment and readable storage medium | |
| KR100640761B1 (en) | Method of extracting 3 dimension coordinate of landmark image by single camera | |
| CN110321781B (en) | Signal processing method and device for non-contact measurement | |
| CN111382646B (en) | Living body identification method, storage medium and terminal equipment | |
| CN114913287B (en) | Three-dimensional human body model reconstruction method and system | |
| CN115565213B (en) | Image processing method and device | |
| KR101774158B1 (en) | Apparatus and method of processing fog in image based on statistical vertor ellipsoid | |
| JP2009258770A (en) | Image processing method, image processor, image processing program, and imaging device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200512 |
|
| RJ01 | Rejection of invention patent application after publication |