CN113827208A - Non-contact blood pressure measuring equipment based on face video - Google Patents
Non-contact blood pressure measuring equipment based on face video Download PDFInfo
- Publication number
- CN113827208A CN113827208A CN202111111390.6A CN202111111390A CN113827208A CN 113827208 A CN113827208 A CN 113827208A CN 202111111390 A CN202111111390 A CN 202111111390A CN 113827208 A CN113827208 A CN 113827208A
- Authority
- CN
- China
- Prior art keywords
- roi
- feature
- face video
- space
- sequence
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000036772 blood pressure Effects 0.000 title claims abstract description 17
- 239000013598 vector Substances 0.000 claims abstract description 24
- 238000009530 blood pressure measurement Methods 0.000 claims abstract description 16
- 238000000605 extraction Methods 0.000 claims abstract description 16
- 238000013507 mapping Methods 0.000 claims abstract description 15
- 238000013528 artificial neural network Methods 0.000 claims abstract description 14
- 239000008280 blood Substances 0.000 claims abstract description 14
- 210000004369 blood Anatomy 0.000 claims abstract description 14
- 230000002708 enhancing effect Effects 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000005728 strengthening Methods 0.000 claims abstract description 4
- 238000004364 calculation method Methods 0.000 claims description 15
- 230000035488 systolic blood pressure Effects 0.000 claims description 15
- 230000035487 diastolic blood pressure Effects 0.000 claims description 14
- 238000013527 convolutional neural network Methods 0.000 claims description 7
- 239000000284 extract Substances 0.000 claims description 5
- 238000010586 diagram Methods 0.000 claims description 4
- 230000000873 masking effect Effects 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000002347 injection Methods 0.000 claims 1
- 239000007924 injection Substances 0.000 claims 1
- 238000000034 method Methods 0.000 claims 1
- MJEREOGWNZGKRN-UHFFFAOYSA-N 1-(4-aminophenyl)-8-[4-(4-fluorophenyl)-4-oxobutyl]-1,3,8-triazaspiro[4.5]decan-4-one Chemical compound C1=CC(N)=CC=C1N1C2(CCN(CCCC(=O)C=3C=CC(F)=CC=3)CC2)C(=O)NC1 MJEREOGWNZGKRN-UHFFFAOYSA-N 0.000 description 7
- 238000005259 measurement Methods 0.000 description 7
- 238000012360 testing method Methods 0.000 description 5
- 238000000691 measurement method Methods 0.000 description 4
- 210000004204 blood vessel Anatomy 0.000 description 2
- 230000003205 diastolic effect Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000010989 Bland-Altman Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000002790 cross-validation Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 239000012634 fragment Substances 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 108091008695 photoreceptors Proteins 0.000 description 1
- 239000000126 substance Substances 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
- 238000012549 training Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/021—Measuring pressure in heart or blood vessels
- A61B5/02108—Measuring pressure in heart or blood vessels from analysis of pulse wave characteristics
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0002—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network
- A61B5/0004—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network characterised by the type of physiological signal transmitted
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0002—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network
- A61B5/0004—Remote monitoring of patients using telemetry, e.g. transmission of vital signals via a communication network characterised by the type of physiological signal transmitted
- A61B5/0013—Medical image data
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Health & Medical Sciences (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Animal Behavior & Ethology (AREA)
- Surgery (AREA)
- Medical Informatics (AREA)
- Theoretical Computer Science (AREA)
- Heart & Thoracic Surgery (AREA)
- Physiology (AREA)
- Public Health (AREA)
- Pathology (AREA)
- Veterinary Medicine (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- General Physics & Mathematics (AREA)
- Cardiology (AREA)
- Computer Networks & Wireless Communication (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Computing Systems (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Computational Biology (AREA)
- Vascular Medicine (AREA)
- Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
- Radiology & Medical Imaging (AREA)
- Fuzzy Systems (AREA)
- Psychiatry (AREA)
- Signal Processing (AREA)
- Measuring Pulse, Heart Rate, Blood Pressure Or Blood Flow (AREA)
Abstract
The invention relates to a non-contact blood pressure measuring device based on a face video, which comprises: the face video acquisition terminal is used for acquiring a face video, and performing ROI (region of interest) region splicing and feature extraction on each frame of image in the face video to obtain a feature sequence; the data enhancement module is used for enhancing the dimension characteristics of the blood volume pulse wave related information in a color space by adopting a YUVT color space according to the characteristic sequence obtained by the face video acquisition terminal to obtain a video sequence with enhanced dimension characteristics; the spatial feature processing module is used for calculating the feature vectors of the video sequence after the dimensionality feature enhancement, splicing the feature vectors of all frames into a space-time feature map, and performing spatial slicing on the space-time feature map to form multi-time domain spatial feature map; and the neural network computing module is used for carrying out high-dimensional feature extraction on the multi-time domain space feature mapping and strengthening the time domain feature correlation of the extracted high-dimensional features by using the LSTM, thereby obtaining the blood pressure measurement model.
Description
Technical Field
The invention relates to the technical field of human face video feature processing, in particular to non-contact blood pressure measuring equipment based on human face video.
Background
Blood pressure is an important physiological parameter of a human body, and blood pressure measurement is mainly divided into a direct measurement method and an indirect measurement method for a long time, wherein the direct measurement method is to directly monitor a biological signal of the human body by using an instrument, the indirect measurement method is to acquire a photoreceptor pulse wave on the surface of skin by using a photoelectric sensor or an electrode, extract characteristics according to the change of the pulse wave and then calculate systolic pressure and diastolic pressure of the pulse by using the characteristics.
However, the measurement accuracy of the conventional method is easily affected by many factors, such as the professional ability of the measuring person, and further, such as the height, weight, age, and measurement environment of the measured person. In the field of blood pressure measurement, the stability of traditional measuring equipment is strong, but is not portable, and the blood pressure measuring equipment based on PPG signal is suitable for the scene abundant, but its accuracy and stability remain to be improved. Moreover, both the traditional measurement equipment and the feature extraction measurement equipment based on the PPG technology belong to contact type passive measurement, and long-time contact type measurement inevitably causes physiological discomfort to the measured person.
Disclosure of Invention
The invention aims to improve the accuracy of blood pressure measurement and improve the comfort of testers, and provides non-contact blood pressure measuring equipment based on a human face video.
In order to achieve the above object, the embodiments of the present invention provide the following technical solutions:
non-contact blood pressure measurement equipment based on face video includes:
the face video acquisition terminal is used for acquiring a face video, and performing ROI (region of interest) region splicing and feature extraction on each frame of image in the face video to obtain a feature sequence;
the data enhancement module is used for extracting blood volume pulse wave related information from a face video acquired by the face video acquisition terminal, and enhancing the dimension characteristics of the blood volume pulse wave related information in a color space by adopting a YUVT color space according to the characteristic sequence acquired by the face video acquisition terminal to obtain a video sequence with enhanced dimension characteristics;
the spatial feature processing module is used for calculating the feature vectors of the video sequence after the dimensionality feature enhancement, splicing the feature vectors of all frames into a space-time feature map, and performing spatial slicing on the space-time feature map to form multi-time domain spatial feature map;
and the neural network computing module is used for carrying out high-dimensional feature extraction on the multi-time domain space feature mapping and strengthening the time domain feature correlation of the extracted high-dimensional features by using the LSTM, thereby obtaining the blood pressure measurement model.
In the scheme, the end-to-end non-contact blood pressure measuring equipment based on the face video is provided, the naked skin part (such as the face, the four limbs, the back, the neck and the like) of a human body is extracted from the face video, a plurality of interested ROI areas are divided on the naked skin part, a plurality of effective ROI areas of the face are spliced into a new feature sequence again, the feature sequence is subjected to dimension feature enhancement, space-time slicing and multiband feature mapping are carried out, blood vessel flow signals are extracted more effectively, finally, the high-dimensional space features of each time slice are extracted by using a neural network, time domain feature association is enhanced by using LSTM, a blood pressure measuring network model is obtained, and systolic pressure and diastolic pressure values are output.
Furthermore, the face video acquisition terminal comprises a face video recognition engine, a feature map generation module and a feature sequence generation module, wherein,
the face video recognition engine is used for acquiring a face video, extracting each frame image in the face video, and combining the images of all the frames into an image sequence A ═ A { (A)1,...At,...ATIn which A istRepresenting the t frame image;
the characteristic diagram is generatedThe forming module is connected with the data output end of the face video recognition engine and used for generating 4 ROI (region of interest) areas for each frame image of the face video and recombining and splicing the 4 ROI areas into an ROI feature map f of the frame imageroi＝(t，ri)，ri∈[r1，r2，r3，r4](ii) a Wherein t represents the t frame image of the face video, and ri represents 4 ROI areas; thereby obtaining ROI feature map f of T frame imageroi＝(t，ri)，t∈T；
The characteristic sequence generation module is connected with the data output end of the characteristic map generation module and is used for generating the ROI characteristic map f of the T-frame imageroi＝(t，ri) Randomly selects a segment f with the frame number length of V in the sequenceroi＝(v，ri) V is equal to V, and a segment f is randomly selectedroi＝(v，ri) P × q region of (1)
Masking to obtain a characteristic sequence
And put the f' togetherroi(v，ri) V ∈ V and froi＝(t，ri)，
And (3) splicing on a time dimension to form a new characteristic sequence:
F`roi(t，ri)，t∈T。
furthermore, the data enhancement module extracts the RGB color space pixel value corresponding to the blood volume pulse wave related information of the t-th frame from the face video acquired by the face video acquisition terminal as Rt(x，y)、Gt(x，y)、Bt(x，y)∈Froi(t，ri)；
Converting RGB color space pixel values corresponding to blood volume pulse wave related information of the t-th frame into YUVT color channel pixel values Y by adopting YUVT color spacet(x，y)、Ut(x，y)、Vt(x, y) is:
wherein the content of the first and second substances,
converting RGB color space pixel value into matrix of YUVT color channel pixel value, the RGB color space pixel value and YUVT color channel pixel value jointly form video sequence F with enhanced dimensionality characteristicroi(t，ri)，ri∈[r1，r2，r3，r4]。
Furthermore, the spatial feature processing module comprises a pixel mean value calculation module, a feature vector calculation module and a space-time slice mapping module;
the pixel mean value calculation module is used for enhancing the dimensionality characteristics of each frame of video sequence Froi(t，ri) Equally dividing each ROI area into N sub-ROIs, and performing average pooling on each sub-ROI to obtain pixel mean values AP (i, N, t) of each sub-ROI in the ROI area:
wherein v (x, y, t) represents the pixel value of the nth sub ROI position (x, y) in the ith ROI area of the tth frame of the face video, and num represents the pixel number of the ith ROI area;
the feature vector calculation module is used for expanding the obtained pixel mean values of all sub-ROIs and splicing the pixel mean values to form a feature vector SS (t) corresponding to the t-th frame image:
wherein, Flattern represents the expansion operation of the characteristic vector, and CAT represents the splicing operation of the characteristic vector;
the space-time slice mapping module is used for splicing the feature vectors SS (t) of all framesFollowed by a spatio-temporal feature map, which is divided into M separate spatio-temporal segments { TS } using a sliding window of step size and length cl1，TS2，...TSM} whereby the assembly maps STS (m) for multi-time-domain spatial features:
where m denotes the index of the spatio-temporal slice and N denotes the total number of sub-ROIs of 4 ROI regions in the t-th frame.
Furthermore, the neural network calculation module comprises a residual convolutional neural network, an LSTM network and a weight distributor;
the residual convolutional neural network is used for performing high-dimensional feature extraction on multi-time domain space mapping, and the feature extraction network corresponding to each time domain segment adopts the same parameters;
the LSTM network is used for enhancing time correlation of high-dimensional features extracted from the residual convolutional neural network;
the weight distributor is composed of a fully connected neural network and is used for fitting the high-dimensional characteristics with strengthened time correlation to the systolic pressure and the diastolic pressure so as to obtain a blood pressure measurement model, and the blood pressure measurement model outputs the blood pressure
Compared with the prior art, the invention has the beneficial effects that:
(1) the device realizes end-to-end non-contact blood pressure measurement through the collected face video information;
(2) the invention can realize the measurement of diastolic pressure and systolic pressure by only carrying out feature extraction on the extremely short face video, and has higher detection efficiency compared with the traditional blood pressure measurement mode.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present invention and therefore should not be considered as limiting the scope, and for those skilled in the art, other related drawings can be obtained according to the drawings without inventive efforts.
FIG. 1 is a block diagram of a blood pressure device module according to the present invention;
fig. 2 is a distribution of systolic pressure and diastolic pressure in the ASPD database in the embodiment 2 of the present invention, in which fig. 2(a) is a distribution of systolic pressure and fig. 2(b) is a distribution of diastolic pressure;
fig. 3 is a schematic diagram of an ASPD apparatus in embodiment 2 of the present invention;
FIG. 4 is a graph showing the results of a systolic blood pressure cross data test in example 2 of the present invention;
FIG. 5 is a graph showing the results of diastolic cross-correlation data testing in example 2 of the present invention.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. The components of embodiments of the present invention generally described and illustrated in the figures herein may be arranged and designed in a wide variety of different configurations. Thus, the following detailed description of the embodiments of the present invention, presented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments of the present invention without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined and explained in subsequent figures.
Example 1:
the invention is realized by the following technical scheme, as shown in fig. 1, the non-contact blood pressure measuring device based on the face video comprises a face video acquisition terminal, a data enhancement module, a spatial feature processing module and a neural network computing module, wherein:
the face video acquisition module is used for acquiring a face video, and performing ROI (region of interest) region splicing and feature extraction on each frame of image in the face video to obtain a feature sequence.
In detail, the face video acquisition module comprises a face video recognition engine, a feature map generation module and a feature sequence generation module.
The face video recognition engine is used for acquiring a face video, for example, a detected person is located in front of the face video acquisition module, the face video recognition engine can acquire the face video for 10s, extract each frame image in the face video, and combine images of all frames into an image sequence a ═ { a ═ a1,...At,...ATIn which A istRepresenting the t-th frame image. For example, when T is 10 frames, the image sequence is a ═ a1，...A5，...AT10}. The face video recognition engine is a SeetaFace face recognition engine.
The characteristic image generation module is connected with the data output end of the face video recognition engine and used for generating 4 ROI (region of interest) regions at the position of a brow chain house with rich blood vessel information in each frame image of the face video and recombining and splicing the 4 ROI regions into an ROI characteristic image f of the frame imageroi＝(t，ri)，ri∈[r1，r2，r3，r4](ii) a Where t represents the t-th frame image of the face video and ri represents 4 ROI areas.
For example, 4 ROI regions are generated from the 1 st frame image in the face video, the ROI feature map f of the 1 st frame image can be obtainedroi＝(1，ri)，ri∈[r1，r2，r3，r4]，r1Denotes the ROI area of block 1, r2Denotes the 2 nd ROI area, r3Denotes the 3 rd block ROI area, r4The 4 th block ROI area is shown.
In this way, the ROI feature map f of the T-frame image is obtainedroi＝(t，ri) And T ∈ T. That is, the ROI feature map f of 10 frames of images can be obtainedroi＝(1，ri)、froi＝(2，ri)、......froi＝(10，ri)。
The characteristic sequence generation module is connected with the data output end of the characteristic map generation module and is used for generating the ROI characteristic map f of the T-frame imageroi＝(t，ri) Randomly selects a segment f with the frame number length of V in the sequenceroi＝(v，ri) And V ∈ V. For example, if a segment with a frame number length of 4 is randomly selected from the ROI feature map of 10 frames of images, the 4 segments may be froi＝(1，ri)、froi＝(2，ri)、froi＝(4，ri)、froi＝(5，ri) That is, the 4 segments of the 1 st frame, the 2 nd frame, the 4 th frame and the 5 th frame form froi＝(v，ri)，v∈V。
Then randomly selecting the fragment froi＝(v，ri) P × q region of (1)
Masking to obtain a characteristic sequence
Respectively for the segments froi＝(1，ri)、froi＝(2，ri)、froi＝(4，ri)、froi＝(5，ri) The p × q region in (1) is masked, so as to obtain the characteristic sequence f ″roi(1，ri)、f`roi(2，ri)、f`roi(3，ri)、f`roi(5，ri)。
And put the f' togetherroi(v，ri) V ∈ V and froi＝(t，ri)，
Splicing in time dimension to form new characteristic sequence F ″roi(T, ri), T ∈ T. The characteristic sequence f' of the mask is processedroi(1，ri)、f`roi(2，ri)、f`roi(3，ri)、f`roi(5，ri) And unselected frame segmentsfroi＝(3，ri)、froi＝(6，ri)、froi＝(7，ri)、froi＝(8，ri)、froi＝(9，ri)、froi＝(10，ri) Splicing is performed to form a new signature sequence F ″roi(T, ri), T ∈ T. The new characteristic sequence includes F ″roi(1，ri)、F`roi(2，ri)、......F`roi(10，ri)。
The data enhancement module is used for extracting blood volume pulse wave related information from a face video acquired by the face video acquisition terminal, and enhancing the dimension characteristics of the blood volume pulse wave related information in a color space by adopting a YUVT color space according to the characteristic sequence acquired by the face video acquisition terminal to obtain a video sequence with enhanced dimension characteristics.
In detail, the data enhancement module extracts an RGB color space pixel value corresponding to the blood volume pulse wave related information of the t-th frame as R from the face video acquired by the face video acquisition terminalt(x，y)、Gt(x，y)、Bt(x，y)∈F`roi(t, ri). The face video acquisition terminal can acquire face videos in a certain time period and can acquire blood volume pulse waves simultaneously.
Converting RGB color space pixel values corresponding to blood volume pulse wave related information of the t-th frame into YUVT color channel pixel values Y by adopting YUVT color spacet(x，y)、Ut(x，y)、Vt(x, y) is:
wherein, RGB color space pixel value and YUVT color channel pixel value jointly form the t frame video sequence F with enhanced dimensionality characteristicroi(t，ri)，ri∈[r1，r2，r3，r4]. In this way, a 10-frame video sequence F is obtainedroi(t，ri) T ∈ T, including Froi(1，ri)、Froi(2，ri)、......Froi(10，ri)。
The spatial feature processing module is used for calculating feature vectors of the video sequence after the dimensionality feature enhancement, splicing the feature vectors of all frames into a space-time feature map, and performing spatial slicing on the space-time feature map to form multi-time-domain spatial feature map.
In detail, the spatial feature processing module includes a pixel mean calculation module, a feature vector calculation module, and a spatio-temporal slice mapping module.
The pixel mean value calculation module is used for enhancing the dimensionality characteristics of each frame of video sequence Froi(t，ri) Equally dividing each ROI area into N sub-ROIs, and performing average pooling on each sub-ROI to obtain pixel mean values AP (i, N, t) of each sub-ROI in the ROI area:
wherein v (x, y, t) represents the pixel value of the nth sub-ROI position (x, y) in the ith ROI area of the tth frame of the face video, and num represents the pixel number of the ith ROI area of the tth frame. For example, the pixel mean value AP (i, n, 1) of the 1 st frame image may include AP (1, n, 1), AP (2, n, 1), AP (3, n, 1), and AP (4, n, 1), and similarly, the pixel mean values AP (i, n, 1) to AP (i, n, 10) of the 10 th frame image may be obtained.
The feature vector calculation module is used for expanding the obtained pixel mean values of all sub-ROIs and splicing the pixel mean values to form a feature vector SS (t) corresponding to the t-th frame image:
wherein, Flattern represents the expansion operation of the eigenvector, and CAT represents the splicing operation of the eigenvector. Similarly, feature vectors SS (1) to SS (10) corresponding to 10 frames of images can be obtained.
The space-time slice mapping module is used for mapping all framesIs spliced into a space-time feature map, which is divided into M separate space-time segments { TS } using a sliding window with both step size and length cl1，TS2，...TSM} whereby the assembly maps STS (m) for multi-time-domain spatial features:
where m denotes the index of the spatio-temporal slice and N denotes the total number of sub-ROIs of 4 ROI regions in the t-th frame.
The neural network computing module is used for carrying out high-dimensional feature extraction on the multi-time domain space feature mapping and strengthening the time domain feature correlation of the extracted high-dimensional features by using the LSTM, so that a blood pressure measurement model is obtained. The neural network calculation module comprises a residual convolutional neural network, an LSTM network and a weight distributor.
The convolution kernel size of the residual convolution neural network is 3 multiplied by 3, the step length is 1, the residual convolution neural network is used for carrying out high-dimensional feature extraction on multi-time domain space mapping, and the feature extraction network corresponding to each time domain segment adopts the same parameters. The results are input into the LSTM network, again enhancing the temporal correlation. Then inputting the data into a weight distributor formed by a fully-connected neural network to obtain a blood pressure measurement model, and fitting the systolic pressure and diastolic pressure values by integrating the results of all time domain segments.
Example 2:
based on the basis of example 1, experimental verification was performed:
MMSE-HR: consisting of 102 RGB face videos from 40 testers with corresponding average HR and BP values, with the average number of face videos at 25 frames/s, is a public database for non-contact HR and BP estimation.
ASPD: physiological data and corresponding face videos of 124 testers are acquired by using an ohm dragon sphygmomanometer (OMRON HEM-1020), an electromyograph (BIOPAC M160) and a mobile phone camera, each group of data acquires 1min of face videos, the video frame rate is 30 frames/s, and the image resolution is 1920 x 1080. The distribution of systolic pressure and diastolic pressure in the ASPD database is shown in fig. 2(a) and fig. 2(b), respectively, and the ASPD device is shown in fig. 3.
According to the scheme, the results obtained by performing cross validation on the ASPD database are shown in table 1, the ASPD is used as a training set, and the results obtained by performing testing on the MMSE-HR cross data set are shown in table 2.
| SD(nmhg) | RMSE(nmhg) | MAE(nmhg) | |
| Systolic pressure | 10.06 | 10.33 | 8.44 |
| Diastolic blood pressure | 8.28 | 8.45 | 6.78 |
TABLE 1
| SD(nmhg) | RMSE(nmhg) | MAE(nmhg) | |
| Systolic pressure | 13.22 | 13.75 | 11.51 |
| Diastolic blood pressure | 10.71 | 10.79 | 8.47 |
TABLE 2
In the results of the crossover tests, the MAE and RMSE between the predicted and actual values of systolic blood pressure were 11.51 and 13.75, respectively, and the MAE and RMSE between the predicted and actual values of diastolic blood pressure were 8.47 and 1079, respectively.
In conclusion, the device provided by the invention has certain bloom capability and can have better prediction results in the interval of the systolic pressure [90, 160] and the interval of the diastolic pressure [50, 100 ].
A further Bland-Altman plot analyzes the consistency of the results of the systolic and diastolic cross-data set tests, as shown in FIGS. 4 and 5. The results show that the device proposed by the invention has good consistency both at systolic and diastolic pressures.
The above description is only for the specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily conceive of the changes or substitutions within the technical scope of the present invention, and all the changes or substitutions should be covered within the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (5)
1. Non-contact blood pressure measurement equipment based on face video, its characterized in that: the method comprises the following steps:
the face video acquisition terminal is used for acquiring a face video, and performing ROI (region of interest) region splicing and feature extraction on each frame of image in the face video to obtain a feature sequence;
the data enhancement module is used for extracting blood volume pulse wave related information from a face video acquired by the face video acquisition terminal, and enhancing the dimension characteristics of the blood volume pulse wave related information in a color space by adopting a YUVT color space according to the characteristic sequence acquired by the face video acquisition terminal to obtain a video sequence with enhanced dimension characteristics;
the spatial feature processing module is used for calculating the feature vectors of the video sequence after the dimensionality feature enhancement, splicing the feature vectors of all frames into a space-time feature map, and performing spatial slicing on the space-time feature map to form multi-time domain spatial feature map;
and the neural network computing module is used for carrying out high-dimensional feature extraction on the multi-time domain space feature mapping and strengthening the time domain feature correlation of the extracted high-dimensional features by using the LSTM, thereby obtaining the blood pressure measurement model.
2. The non-contact blood pressure measuring device based on human face video of claim 1, characterized in that: the face video acquisition terminal comprises a face video recognition engine, a characteristic diagram generation module and a characteristic sequence generation module, wherein,
the face video recognition engine is used for acquiring a face video, extracting each frame image in the face video, and combining the images of all the frames into an image sequence A ═ A { (A)1,...At,...ATIn which A istRepresenting the t frame image;
the characteristic image generation module is connected with the data output end of the face video recognition engine and used for generating 4 ROI (region of interest) areas for each frame image of the face video and recombining and splicing the 4 ROI areas into the ROI characteristic image f of the frame imageroi＝(t,ri)，ri∈[r1,r2,r3,r4](ii) a Wherein t represents the t frame image of the face video, and ri represents 4 ROI areas; thereby obtaining ROI feature map f of T frame imageroi＝(t,ri)，t∈T；
The characteristic sequence generation module is connected with the data output end of the characteristic map generation module and is used for generating the ROI characteristic map f of the T-frame imageroi＝(t,ri) Randomly selects a segment f with the frame number length of V in the sequenceroi＝(v,ri) V is equal to V, and a segment f is randomly selectedroi＝(v,ri) P × q region of (1)
Masking to obtain a characteristic sequence
And put the f' togetherroi(v,ri) V ∈ V and
v, splicing in a time dimension to form a new characteristic sequence:
F`roi(t,ri),t∈T。
3. the non-contact blood pressure measuring device based on human face video of claim 2, characterized in that: the data enhancement module extracts RGB color space pixel values corresponding to blood volume pulse wave related information of the t-th frame from a face video acquired by the face video acquisition terminal as Rt(x,y)、Gt(x,y)、Bt(x,y)∈F`roi(t,ri)；
Converting RGB color space pixel values corresponding to blood volume pulse wave related information of the t-th frame into YUVT color channel pixel values Y by adopting YUVT color spacet(x,y)、Ut(x,y)、Vt(x, y) is:
wherein, RGB color space pixel value and YUVT color channel pixel value jointly form the video sequence F with enhanced dimensionality characteristicsroi(t,ri),ri∈[r1,r2,r3,r4]。
4. The non-contact blood pressure measuring device based on human face video of claim 3, characterized in that: the spatial feature processing module comprises a pixel mean value calculation module, a feature vector calculation module and a space-time slice mapping module;
the pixel mean value calculation module is used for enhancing the dimensionality characteristics of each frame of video sequence Froi(t,ri) Equally dividing each ROI area into N sub-ROIs, and performing average pooling on each sub-ROI to obtain pixel mean values AP (i, N, t) of each sub-ROI in the ROI area:
wherein v (x, y, t) represents the pixel value of the nth sub ROI position (x, y) in the ith ROI area of the tth frame of the face video, and num represents the pixel number of the ith ROI area;
the feature vector calculation module is used for expanding the obtained pixel mean values of all sub-ROIs and splicing the pixel mean values to form a feature vector SS (t) corresponding to the t-th frame image:
wherein, Flattern represents the expansion operation of the characteristic vector, and CAT represents the splicing operation of the characteristic vector;
the space-time slice mapping module is used for splicing the feature vectors SS (t) of all frames into a space-time feature map, and the space-time feature map is divided into M independent space-time segments { TS) by using a sliding window with the step length and the length being cl1,TS2,...TSMThus the components are mapped for multi-time domain space(m) for injection of STS:
where m denotes the index of the spatio-temporal slice and N denotes the total number of sub-ROIs of 4 ROI regions in the t-th frame.
5. The non-contact blood pressure measuring device based on human face video of claim 4, characterized in that: the neural network calculation module comprises a residual convolutional neural network, an LSTM network and a weight distributor;
the residual convolutional neural network is used for performing high-dimensional feature extraction on multi-time domain space mapping, and the feature extraction network corresponding to each time domain segment adopts the same parameters;
the LSTM network is used for enhancing time correlation of high-dimensional features extracted from the residual convolutional neural network;
the weight distributor is composed of a fully connected neural network and is used for fitting the high-dimensional characteristics with strengthened time correlation to the systolic pressure and the diastolic pressure so as to obtain a blood pressure measurement model, and the blood pressure is output by the blood pressure measurement model.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111111390.6A CN113827208A (en) | 2021-09-18 | 2021-09-18 | Non-contact blood pressure measuring equipment based on face video |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111111390.6A CN113827208A (en) | 2021-09-18 | 2021-09-18 | Non-contact blood pressure measuring equipment based on face video |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113827208A true CN113827208A (en) | 2021-12-24 |
Family
ID=78969145
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111111390.6A Pending CN113827208A (en) | 2021-09-18 | 2021-09-18 | Non-contact blood pressure measuring equipment based on face video |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113827208A (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20160026942A (en) * | 2016-02-18 | 2016-03-09 | 삼성전자주식회사 | Portable device for measuring blood pressure and method thereof |
| CN109846469A (en) * | 2019-04-16 | 2019-06-07 | 合肥工业大学 | A kind of contactless method for measuring heart rate based on convolutional neural networks |
| CN111728602A (en) * | 2020-08-21 | 2020-10-02 | 之江实验室 | Non-contact blood pressure measuring device based on PPG |
| CN112168155A (en) * | 2020-10-28 | 2021-01-05 | 广东小天才科技有限公司 | Blood pressure detection method, wearable device and computer readable storage medium |
| CN113255585A (en) * | 2021-06-23 | 2021-08-13 | 之江实验室 | Face video heart rate estimation method based on color space learning |
-
2021
- 2021-09-18 CN CN202111111390.6A patent/CN113827208A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20160026942A (en) * | 2016-02-18 | 2016-03-09 | 삼성전자주식회사 | Portable device for measuring blood pressure and method thereof |
| CN109846469A (en) * | 2019-04-16 | 2019-06-07 | 合肥工业大学 | A kind of contactless method for measuring heart rate based on convolutional neural networks |
| CN111728602A (en) * | 2020-08-21 | 2020-10-02 | 之江实验室 | Non-contact blood pressure measuring device based on PPG |
| CN112168155A (en) * | 2020-10-28 | 2021-01-05 | 广东小天才科技有限公司 | Blood pressure detection method, wearable device and computer readable storage medium |
| CN113255585A (en) * | 2021-06-23 | 2021-08-13 | 之江实验室 | Face video heart rate estimation method based on color space learning |
Non-Patent Citations (1)
| Title |
|---|
| XUESONG NIU，等: "RhythmNet: End-to-End Heart Rate Estimation From Face via Spatial-TemporalRepresentation", IEEE TRANSACTIONS ON IMAGE PROCESSING, vol. 29, pages 2409, XP011765077, DOI: 10.1109/TIP.2019.2947204 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Hsu et al. | Deep learning with time-frequency representation for pulse estimation from facial videos | |
| CN110969124B (en) | Two-dimensional human body posture estimation method and system based on lightweight multi-branch network | |
| Shoieb et al. | Computer-aided model for skin diagnosis using deep learning | |
| CN111407245B (en) | Non-contact heart rate and body temperature measuring method based on camera | |
| CN109389045B (en) | Micro-expression identification method and device based on mixed space-time convolution model | |
| CN110706826B (en) | Non-contact real-time multi-person heart rate and blood pressure measuring method based on video image | |
| CN112819910A (en) | Hyperspectral image reconstruction method based on double-ghost attention machine mechanism network | |
| TW201315438A (en) | Method of contact-free heart rate estimation and system thereof | |
| US11227161B1 (en) | Physiological signal prediction method | |
| CN113408508A (en) | Transformer-based non-contact heart rate measurement method | |
| CN106031631A (en) | Method, device, and system for heart rate detection | |
| CN113782190B (en) | Image processing method based on multistage space-time characteristics and mixed attention network | |
| Prajapati et al. | Direct unsupervised super-resolution using generative adversarial network (DUS-GAN) for real-world data | |
| CN112541433B (en) | Two-stage human eye pupil accurate positioning method based on attention mechanism | |
| Wang et al. | Multimodal medical image fusion based on multichannel coupled neural P systems and max-cloud models in spectral total variation domain | |
| Zheng et al. | Heart rate prediction from facial video with masks using eye location and corrected by convolutional neural networks | |
| CN116758619B (en) | Facial video-based emotion classification method, system, storage medium and equipment | |
| He et al. | Remote Photoplethysmography Heart Rate Variability Detection Using Signal to Noise Ratio Bandpass Filtering | |
| CN113827208A (en) | Non-contact blood pressure measuring equipment based on face video | |
| CN113963427B (en) | Method and system for rapid in-vivo detection | |
| CN107731306B (en) | Non-contact heart rate extraction method based on thermal imaging technology | |
| CN109886212A (en) | From the method and apparatus of rolling fingerprint synthesis fingerprint on site | |
| JP6201520B2 (en) | Gaze analysis system and method using physiological indices | |
| CN110400250A (en) | Intelligent image preprocess method and system based on AI | |
| CN114587305A (en) | Non-contact blood pressure measuring equipment based on human body video |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |