CN116327133A - Multi-physiological index detection method, device and related equipment - Google Patents
Multi-physiological index detection method, device and related equipment Download PDFInfo
- Publication number
- CN116327133A CN116327133A CN202310611339.4A CN202310611339A CN116327133A CN 116327133 A CN116327133 A CN 116327133A CN 202310611339 A CN202310611339 A CN 202310611339A CN 116327133 A CN116327133 A CN 116327133A
- Authority
- CN
- China
- Prior art keywords
- user
- pulse wave
- video
- detected
- wave signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 32
- 238000000034 method Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 25
- 238000000605 extraction Methods 0.000 claims abstract description 22
- 230000001815 facial effect Effects 0.000 claims description 42
- 238000010586 diagram Methods 0.000 claims description 13
- 230000006870 function Effects 0.000 claims description 12
- 230000036387 respiratory rate Effects 0.000 claims description 11
- 238000005070 sampling Methods 0.000 claims description 11
- 238000004364 calculation method Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 6
- 238000011176 pooling Methods 0.000 claims description 6
- 230000007246 mechanism Effects 0.000 claims description 4
- 238000007499 fusion processing Methods 0.000 claims description 3
- 230000009191 jumping Effects 0.000 claims description 3
- 238000013507 mapping Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 description 12
- 210000004204 blood vessel Anatomy 0.000 description 6
- 239000008280 blood Substances 0.000 description 5
- 210000004369 blood Anatomy 0.000 description 5
- 238000012549 training Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000005259 measurement Methods 0.000 description 3
- 238000012935 Averaging Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000006835 compression Effects 0.000 description 2
- 238000007906 compression Methods 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000003745 diagnosis Methods 0.000 description 1
- 235000019800 disodium phosphate Nutrition 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008451 emotion Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 230000004886 head movement Effects 0.000 description 1
- 230000010365 information processing Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 230000031700 light absorption Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000000644 propagated effect Effects 0.000 description 1
- 230000010349 pulsation Effects 0.000 description 1
- 230000036391 respiratory frequency Effects 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 230000001131 transforming effect Effects 0.000 description 1
- 238000011282 treatment Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/15—Biometric patterns based on physiological signals, e.g. heartbeat, blood flow
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0033—Features or image-related aspects of imaging apparatus classified in A61B5/00, e.g. for MRI, optical tomography or impedance tomography apparatus; arrangements of imaging apparatus in a room
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/0059—Measuring for diagnostic purposes; Identification of persons using light, e.g. diagnosis by transillumination, diascopy, fluorescence
- A61B5/0077—Devices for viewing the surface of the body, e.g. camera, magnifying lens
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
- A61B5/02405—Determining heart rate variability
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/02—Detecting, measuring or recording pulse, heart rate, blood pressure or blood flow; Combined pulse/heart-rate/blood pressure determination; Evaluating a cardiovascular condition not otherwise provided for, e.g. using combinations of techniques provided for in this group with electrocardiography or electroauscultation; Heart catheters for measuring blood pressure
- A61B5/024—Detecting, measuring or recording pulse rate or heart rate
- A61B5/02416—Detecting, measuring or recording pulse rate or heart rate using photoplethysmograph signals, e.g. generated by infrared radiation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/08—Detecting, measuring or recording devices for evaluating the respiratory organs
- A61B5/0816—Measuring devices for examining respiratory frequency
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7253—Details of waveform analysis characterised by using transforms
- A61B5/7257—Details of waveform analysis characterised by using transforms using Fourier transforms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7271—Specific aspects of physiological measurement analysis
- A61B5/7275—Determining trends in physiological measurement data; Predicting development of a medical condition based on physiological measurements, e.g. determining a risk factor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
- G06V20/46—Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
Abstract
The invention discloses a multi-physiological index detection method, a multi-physiological index detection device, computer equipment and a storage medium, wherein the method comprises the following steps: acquiring a face video of a user to be detected; inputting a face video of a user to be detected into a pulse wave signal prediction model, wherein the model comprises a video quality enhancement module, a feature extraction module and a signal prediction module, the video quality enhancement module is used for carrying out quality enhancement processing on the face video of the user to be detected, the feature extraction module is used for extracting a space-time feature map of the face video after quality enhancement, and the signal prediction module predicts and obtains pulse wave information of the user to be detected according to the space-time feature map; and calculating to obtain multiple physiological indexes of the user to be tested according to the pulse wave signals of the user to be tested. The invention can improve the detection efficiency and the detection precision without direct contact with the detected person.
Description
Technical Field
The present invention relates to the field of physiological index detection and video quality enhancement technologies, and in particular, to a method, an apparatus, a computer device, and a storage medium for detecting multiple physiological indexes.
Background
The physiological indexes such as Heart Rate (HR), respiratory Rate (RF), heart rate variability (heart rate variability, HRV) and the like can be used for judging the health state and the emotion state of a person, and play an important role in the fields of medical diagnosis, health monitoring and the like. The pulse wave is generated by the ejection of blood from the heart of a human body and the propagation of blood along arterial blood vessels, is generated and propagated in the blood vessels of the human body and is directly influenced by heart pulsation, blood vessel pressure, blood vessel elasticity, blood vessel inner diameter and blood viscosity, and contains abundant human body blood vessel information, so that the pulse wave is processed and analyzed to extract various physiological indexes such as heart rate, respiratory rate, heart rate variability and the like.
At present, multiple physiological indexes can be measured in various modes, for example, the related physiological indexes can be further extracted from bioelectricity signals generated by recording heart activities in an electrocardiogram-based mode, and the measurement result obtained by the mode is accurate, but professional equipment such as an electrocardiograph monitor is required to be operated by professionals, and a measurement process is completed by connecting complex electrodes on a human body. The method can also be used for analyzing physiological indexes such as heart rate, respiratory rate and heart rate variability by measuring the periodical changes of the light absorption and reflection amounts of different wavelengths, namely blood volume pulse (blood volume pulse, BVP) signals, generated by the heart beat of a specific body part in real time, and the contact type devices such as a fingertip type pulse oximeter, an intelligent bracelet and the like can be used for completing the measurement of multiple physiological indexes. Although the method can measure multiple physiological indexes, the method has higher professional requirements, limited use scenes and needs to be contacted with a tested person.
Therefore, how to improve the flexibility of multi-physiological index detection and accurately detect the multi-physiological index is a problem to be solved urgently.
Disclosure of Invention
The embodiment of the invention provides a multi-physiological index detection method, a multi-physiological index detection device, computer equipment and a storage medium, which do not need to be in direct contact with a detected person, and can improve the flexibility of multi-physiological index detection and the detection efficiency and the detection precision.
In order to solve the above technical problems, an embodiment of the present application provides a method for detecting multiple physiological indexes, including the following steps: acquiring a face video of a user to be detected, wherein the face video comprises a plurality of face image sequences; inputting the facial video of the user to be detected into a pulse wave signal prediction model to obtain a pulse wave signal of the user to be detected, wherein the pulse wave signal prediction model comprises a video quality enhancement module, a feature extraction model and a signal prediction module, and the video quality enhancement module is used for carrying out video quality enhancement processing on the facial video of the user to be detected to obtain a facial video with enhanced quality; the feature extraction module is used for extracting a space-time feature map according to the facial video with enhanced quality, wherein the space-time feature map comprises time features and space features contained in the plurality of facial image sequences; the signal prediction module is used for predicting and obtaining the pulse wave signal of the user to be detected according to the space-time characteristic diagram; and calculating multiple physiological indexes of the user to be detected according to the pulse wave signals of the user to be detected.
In one possible implementation manner, inputting the face video of the user to be tested into a downsampling path network for convolution processing, and outputting characteristic information of different scales of the face video of the user to be tested, wherein the downsampling path network comprises a plurality of 3D convolution blocks; and inputting the feature information of different scales output by the downsampling path network into an upsampling path network for feature fusion processing, and outputting the facial video with enhanced quality, wherein the upsampling path network is connected with the downsampling path network through jumping, and the upsampling path network comprises a 3D deconvolution block symmetrical to the plurality of 3D convolution blocks.
In another possible implementation manner, the face video with enhanced quality is respectively input into a 3D convolution network and a 2D convolution network, wherein the 3D convolution network is used for extracting and obtaining time features in the face video of the user to be detected, and the 2D convolution network is used for extracting and obtaining space features in the face video of the user to be detected; and multiplying the spatial features extracted by the 2D convolution network by the time features extracted by the 3D convolution network through a convolution attention mechanism, and outputting the space-time feature map.
In another possible implementation manner, the space-time feature images are input into an average pooling layer network to obtain feature values corresponding to each space-time feature image; and inputting the characteristic value corresponding to each space-time characteristic diagram into a full-connection layer network, and outputting the pulse wave signal of the user to be detected, wherein the full-connection layer network is used for mapping the characteristic value into a corresponding signal value.
In another possible implementation manner, performing fast fourier transform on the pulse wave signal of the user to be detected to obtain a pulse wave signal spectrogram; filtering the pulse wave signal spectrogram by using filters with different cut-off frequencies and preset frequency values; and calculating the heart rate and the respiratory rate of the user to be detected based on the frequency of the corresponding maximum peak value in the filtered spectrogram respectively.
In another possible implementation manner, all peak points of the pulse wave signals of the user to be tested are determined; calculating to obtain pulse intervals according to the sampling period and the distance between each peak point; and calculating the heart rate variability of the user to be detected according to the pulse interval.
In another possible implementation manner, a sample face video and a real pulse wave signal corresponding to the sample face video are obtained; inputting the real pulse wave signals corresponding to the sample face video to an initial pulse wave signal prediction model; the initial pulse wave signal prediction model predicts a pulse wave signal corresponding to the sample face video, compares the obtained pulse wave signal with the real pulse wave signal, and calculates a signal loss function; and adjusting parameters and weights in the initial pulse wave signal prediction model according to the signal loss function to obtain the pulse wave signal prediction model.
In order to solve the above technical problem, an embodiment of the present application further provides a multi-physiological index detection device, including: the data acquisition module is used for acquiring a face video of a user to be detected, wherein the face video comprises a plurality of face image sequences; the prediction module is used for inputting the facial video of the user to be detected into a pulse wave signal prediction model to obtain a pulse wave signal of the user to be detected, wherein the pulse wave signal prediction model comprises a video quality enhancement module, a feature extraction model and a signal prediction module, and the video quality enhancement module is used for carrying out video quality enhancement processing on the facial video of the user to be detected to obtain a facial video with enhanced quality; the feature extraction module is used for extracting a space-time feature map according to the facial video with enhanced quality, wherein the space-time feature map comprises time features and space features contained in the plurality of facial image sequences; the signal prediction module is used for predicting and obtaining the pulse wave signal of the user to be detected according to the space-time characteristic diagram; and the calculation module is used for calculating and obtaining multiple physiological indexes of the user to be detected according to the pulse wave signals of the user to be detected.
To solve the above technical problem, embodiments of the present application further provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the above method when executing the computer program.
To solve the above technical problem, embodiments of the present application further provide a computer readable storage medium storing a computer program, which when executed by a processor, implements the steps of the above method.
According to the multi-physiological index detection method, device, computer equipment and storage medium provided by the embodiment of the invention, the face video of the user to be detected is obtained, the pulse wave signal containing physiological information in the face video is predicted by utilizing the network model based on deep learning, the problems of reduced video quality and reduced prediction precision caused by manual signal processing, light change during video shooting and movement of the user to be detected can be avoided, in addition, the face video of the user to be detected can be enhanced by the video quality enhancement module in the model, the loss of detail information in the pulse wave signal caused by video compression and the like can be improved, the prediction precision of the pulse wave signal is further improved, the detection precision of each physiological index of the user to be detected is further improved, the whole detection process does not need to be in direct contact with the user to be detected, the detection flexibility is improved, and the use scene is widened.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments of the present invention will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is an exemplary system architecture diagram in which the present application may be applied.
FIG. 2 is a flow chart of one embodiment of a multi-physiological index detection method of the present application.
Fig. 3 is a schematic diagram of a pulse wave signal prediction model structure of the present application.
FIG. 4 is a flow chart of one embodiment of a pulse wave signal prediction model training process of the present application.
Fig. 5 is a schematic structural diagram of an embodiment of a multi-physiological index detecting device according to the present application.
FIG. 6 is a schematic structural diagram of one embodiment of a computer device according to the present application.
Detailed Description
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description of the applications herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having" and any variations thereof in the description and claims of the present application and in the description of the figures above are intended to cover non-exclusive inclusions. The terms first, second and the like in the description and in the claims or in the above-described figures, are used for distinguishing between different objects and not necessarily for describing a sequential or chronological order.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those of skill in the art will explicitly and implicitly appreciate that the embodiments described herein may be combined with other embodiments.
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all embodiments of the invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, as shown in fig. 1, a system architecture 100 may include a terminal device 110, a network 120, and a server 130. Network 120 is the medium used to provide communication links between terminal equipment 110 and server 130. The network 120 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.
A user may interact with server 130 via network 120 using terminal device 110 to receive or send messages, etc.
The server 130 may be a server providing various services, such as a background server providing support for pages displayed on the terminal device 110.
It should be noted that, the multi-physiological index detection method provided by the embodiment of the present application is executed by a server, and accordingly, the multi-physiological index detection device is disposed in the server.
It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. Any number of terminal devices, networks and servers may be provided according to implementation requirements, and the terminal device 110 in the embodiment of the present application may specifically correspond to an application system in actual production.
Referring to fig. 2, fig. 2 shows a multi-physiological index detection method according to an embodiment of the present invention, and the method is applied to the server in fig. 1 for illustration, as follows.
S201, acquiring a face video of a user to be detected.
Specifically, the user to be tested can shoot a video containing the whole facial image of the user through a terminal device (such as a mobile phone), then the video is sent to a server through a network, or a pre-stored video containing the whole facial image is sent to the server, and the server further processes the video.
Optionally, after receiving the face video of the user to be tested, the server may perform preprocessing on the face video, for example, perform scaling processing on a face image included in the face video, scale the image to the same scale, or perform normalization and average reduction processing on the face image, so as to ensure that all data has uniformity in a subsequent processing process, and improve data processing efficiency.
S202, inputting the facial video of the user to be detected into a pulse wave signal prediction model to obtain the pulse wave signal of the user to be detected.
Specifically, after receiving the facial video of the user, the server may input the facial image sequence contained in the facial video into the trained pulse wave signal prediction model with the pulse wave prediction capability, so as to obtain the pulse wave signal of the user to be detected.
Exemplary, as shown in fig. 3, the pulse wave signal prediction model includes a video quality enhancement module, a feature extraction module and a signal prediction module, where the video quality enhancement module is configured to enhance a facial video of a user to be detected to obtain a facial video with enhanced quality, and then the feature extraction module performs feature extraction based on the facial video with enhanced quality, extracts temporal features and spatial features contained in the facial video to obtain a space-time feature map, and the signal prediction module predicts a pulse wave signal of the user to be detected according to the space-time feature map.
The pulse wave signal prediction model is a detection model constructed based on a convolutional neural network, and before the pulse wave signal prediction is performed by using the pulse wave signal prediction model, the pulse wave signal prediction model needs to be trained to have a prediction capability, and a training process of the pulse wave signal prediction model will be described in detail later.
In one possible implementation manner, the face video of the user to be detected is input to a downsampling path network to be subjected to convolution processing, the feature information of different scales of the face video of the user to be detected is output, then the feature information of different scales output by the downsampling path network is input to an upsampling path network to be subjected to feature fusion processing, and the face video with enhanced quality is output.
As shown in fig. 3, the video quality enhancement module is a symmetrical structure of a combination of a downsampling path and an upsampling path, so as to enhance the quality of the face video of the user to be detected, and obtain a high-quality face image sequence. The downsampling path consists of a plurality of 3D convolution blocks, and the face video is subjected to convolution processing to obtain feature information of different scales of the face image sequence. The up-sampling path is symmetrically connected with the down-sampling path through jumping by using a plurality of 3D deconvolution blocks, namely, different scale outputs of the down-sampling path are connected to the up-sampling path of a corresponding scale, by the connection mode, the up-sampling path can fuse the detail features of a lower layer with the semantic features of a higher layer, restore the size of the face video to the original size, and finally, the face video with enhanced quality is output to the feature extraction module to finish further feature extraction processing.
In another possible implementation manner, the face video with enhanced quality is respectively input into a 3D convolution network and a 2D convolution network, the 3D convolution network is used for extracting and obtaining time features in the face video of the user to be detected, the 2D convolution network is used for extracting and obtaining space features in the face video of the user to be detected, the space features extracted by the 2D convolution network are multiplied by the time features extracted by the 3D convolution network through a convolution attention mechanism, and the space-time feature map is output.
Exemplary, as shown in fig. 3, the feature extraction module includes a 3D convolution branch and a 2D convolution branch, where the 3D convolution branch performs convolution processing on the face video with enhanced quality to extract a temporal feature in the face video, and the 2D convolution branch also performs convolution processing on the face video with enhanced quality to extract a spatial feature in the face video. After the 2D convolution branch performs convolution processing on the facial video, the extracted spatial features need to be enhanced by a convolution attention mechanism, that is, a weight map is generated through a series of linear and nonlinear transformations, and then the weight map is multiplied by the feature map extracted by the 3D convolution branch, so that the channel and the space of the effective pulse wave signal are given greater weight, and finally the space-time feature map is output.
In another possible implementation manner, the space-time feature map is input to a global average pooling layer network, so as to obtain a feature value corresponding to each space-time feature map, the feature value corresponding to each space-time feature map is input to a full-connection layer network, and a pulse wave signal of a user to be tested is output.
For example, as shown in fig. 3, the signal prediction module includes a global averaging pooling layer and a fully-connected layer, which are connected in series, the global averaging pooling layer receives the space-time feature images output by the feature extraction module, for each space-time feature image, adds all pixel values in the space-time feature image to obtain an average value, uses the average value as a feature value to represent the space-time feature image, and then the fully-connected layer maps the feature values obtained after pooling into final signal values to output corresponding pulse wave signals.
It can be seen that the pulse wave signal prediction model has no limitation on the quality of the video image, complex pretreatment is not required to be performed on the obtained video image, in addition, after a series of treatments such as video quality enhancement, feature extraction and signal prediction, the problem of signal prediction accuracy reduction caused by artificial signal processing, ambient light change, head movement of a tested person and the like can be effectively avoided, meanwhile, loss of detail information in the pulse wave signal caused by video compression can be improved, the application scene can be widened, and the flexibility and accuracy of multi-physiological index detection are remarkably improved.
S203, calculating to obtain multiple physiological indexes of the user to be detected according to the pulse wave signals of the user to be detected.
Specifically, after the pulse wave signal of the user to be detected is obtained through the pulse wave signal prediction model, a plurality of physiological indexes can be further calculated according to the pulse wave signal, so that multi-physiological index detection is realized.
In one possible implementation manner, performing fast fourier transform on a pulse wave signal of a user to be detected to obtain a pulse wave signal spectrogram; filtering the pulse wave signal spectrogram by using filters with different cut-off frequencies and preset frequency values; and calculating the heart rate and the respiratory rate of the user to be detected based on the frequency of the corresponding maximum peak value in the filtered spectrogram respectively.
Specifically, firstly, carrying out fast Fourier transform on pulse wave signals of a user to be detected, transforming the signals from a time domain to a frequency domain to obtain a pulse wave signal spectrogram, then, carrying out filtering processing by using a filter with cut-off frequency being a preset frequency value, and finally, calculating to obtain corresponding heart rate or respiratory frequency based on the frequency at the maximum peak value in the filtered spectrogram.
Optionally, if the heart rate needs to be calculated, a Butterworth filter with a cut-off frequency of 0.4-3Hz can be used for filtering, then the frequency at the maximum peak value in the filtered spectrogram is detected to obtain the heart rate frequency, and the heart rate frequency is multiplied by 60 to obtain the heart rate in times per minute (bpm); similarly, if the respiratory rate needs to be calculated, a butterworth filter with a cut-off frequency of 0.15-0.6Hz may be used for filtering, then the frequency at the maximum peak in the filtered spectrogram is detected to obtain the respiratory rate, and the respiratory rate is multiplied by 60 to obtain the respiratory rate in times per minute (bpm).
In another possible implementation manner, all peak points of the pulse wave signal of the user to be detected are determined, a pulse interval is calculated according to a sampling period and a distance between each peak point, and heart rate variability of the user to be detected is calculated according to the pulse interval.
Specifically, all peak points in pulse wave signals of a user to be detected are detected, then the distance between each peak point is calculated and multiplied by a sampling period to obtain pulse intervals, and then the standard deviation of the pulse intervals is calculated and used as a heart rate variability reference index of the tested person.
Further, the specific calculation process can be calculated by the following formula:
where RR denotes the pulse interval.
Next, a training process of the pulse wave signal prediction model will be described, referring to fig. 4.
S401, acquiring a sample face video and a real pulse wave signal corresponding to the sample face video.
Specifically, based on the existing image acquisition terminal, sample face videos are acquired in a real natural open environment, or face videos are acquired from a face video database, the video time can be selected according to requirements, for example, face videos with 60 continuous seconds can be selected as the sample face videos, and the application is not limited to this. For each face video, the real-time pulse wave signals of the tested person corresponding to the face video and parameters such as heart rate, respiratory rate, heart rate variability indexes and the like can be obtained through a multi-parameter monitor.
In the process of acquiring the sample face video, the face video with different quality needs to be acquired, so that the generalization capability of the model on data is improved, and the application range of the model is enlarged.
S402, inputting a real pulse wave signal corresponding to the sample face video to an initial pulse wave signal prediction model for training.
Specifically, after the initial pulse wave signal prediction model inputs the sample face video, the quality of the sample face video is enhanced through a video quality enhancement module, then the feature extraction module is used for extracting features of the sample face video with enhanced quality to obtain a space-time feature map, the signal prediction module is used for predicting the space-time feature map to obtain a predicted pulse wave signal, the predicted pulse wave signal is compared with a real pulse wave signal, a signal loss function is obtained through calculation, and the mean square error between the predicted signal and the real signal can be used as the loss function.
S403, adjusting parameters and weights in the initial pulse wave signal prediction model according to the signal loss function to obtain the pulse wave signal prediction model.
Specifically, after the signal loss function is obtained by calculation, parameters and weights in the initial pulse wave signal prediction model are adjusted according to the signal loss function, and the parameters and weights in the network are continuously adjusted through continuous iterative execution until the error between the predicted signal output by the prediction model and the real signal is smaller than a threshold (or the signal loss function converges), at this time, the pulse wave signal prediction model is trained, and the pulse wave signal prediction model has the capability of pulse wave signal detection.
It should be understood that the sequence number of each step in the foregoing embodiment does not mean that the execution sequence of each process should be determined by the function and the internal logic, and should not limit the implementation process of the embodiment of the present invention.
Fig. 5 shows a schematic block diagram of a multi-physiological index detecting device corresponding to the multi-physiological index detecting method according to the above embodiment. As shown in fig. 5, the multi-physiological index detection device 500 includes a data acquisition module 510, a prediction module 520, and a calculation module 530. The functional modules are described in detail below.
The data obtaining module 510 is configured to obtain a face video of a user to be tested, where the face video includes a plurality of face image sequences, and the face video includes a plurality of face image sequences.
The prediction module 520 is configured to input the facial video of the user to be detected to a pulse wave signal prediction model to obtain a pulse wave signal of the user to be detected, where the pulse wave signal prediction model includes a video quality enhancement module, a feature extraction model and a signal prediction module, and the video quality enhancement module is configured to perform video quality enhancement processing on the facial video of the user to be detected to obtain a facial video with enhanced quality; the feature extraction module is used for extracting a space-time feature map according to the facial video with enhanced quality, wherein the space-time feature map comprises time features and space features contained in the plurality of facial image sequences; the signal prediction module is used for predicting and obtaining the pulse wave signal of the user to be detected according to the space-time characteristic diagram.
The calculating module 530 is configured to calculate multiple physiological indexes of the user to be measured according to the pulse wave signal of the user to be measured.
For specific limitation of the multi-physiological index detection device, reference may be made to the limitation of the multi-physiological index detection method hereinabove, and the description thereof will not be repeated here. The modules in the multi-physiological index detection device can be realized in whole or in part by software, hardware and a combination thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
In order to solve the technical problems, the embodiment of the application also provides computer equipment. Referring specifically to fig. 6, fig. 6 is a basic structural block diagram of a computer device according to the present embodiment.
The computer device 600 includes a memory 610, a processor 620, and a network interface 630 communicatively coupled to each other via a system bus. It should be noted that only a computer device 600 having components connected to a memory 610, a processor 620, a network interface 630 is shown, but it should be understood that not all of the illustrated components need be implemented and that more or fewer components may be implemented instead. It will be appreciated by those skilled in the art that the computer device herein is a device capable of automatically performing numerical calculations and/or information processing in accordance with predetermined or stored instructions, the hardware of which includes, but is not limited to, microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASICs), programmable gate arrays (fields-Programmable Gate Array, FPGAs), digital processors (Digital Signal Processor, DSPs), embedded devices, etc.
The computer equipment can be a desktop computer, a notebook computer, a palm computer, a cloud server and other computing equipment. The computer equipment can perform man-machine interaction with a user through a keyboard, a mouse, a remote controller, a touch pad or voice control equipment and the like.
The memory 610 includes at least one type of readable storage medium including flash memory, hard disk, multimedia card, card memory (e.g., SD or D interface display memory, etc.), random Access Memory (RAM), static Random Access Memory (SRAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), programmable Read Only Memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage 610 may be an internal storage unit of the computer device 600, such as a hard disk or a memory of the computer device 600. In other embodiments, the memory 610 may also be an external storage device of the computer device 600, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash Card (Flash Card) or the like, which are provided on the computer device 600. Of course, the memory 610 may also include both internal storage units and external storage devices of the computer device 600. In this embodiment, the memory 610 is typically used to store an operating system and various application software installed on the computer device 600, such as program codes for controlling electronic files. In addition, the memory 610 may also be used to temporarily store various types of data that have been output or are to be output.
The processor 620 may be a central processing unit (Central Processing Unit, CPU), controller, microcontroller, microprocessor, or other data processing chip in some embodiments. The processor 620 is generally used to control the overall operation of the computer device 600. In this embodiment, the processor 620 is configured to execute the program code stored in the memory 610 or process data, such as program code for executing control of an electronic file.
The network interface 630 may include a wireless network interface or a wired network interface, the network interface 630 typically being used to establish communication connections between the computer device 600 and other electronic devices.
The present application also provides another embodiment, namely, a computer readable storage medium, where an interface display program is stored, where the interface display program is executable by at least one processor, so that the at least one processor performs the steps of the multi-physiological index detection method as described above.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), comprising several instructions for causing a terminal device (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method described in the embodiments of the present application.
It is apparent that the embodiments described above are only some embodiments of the present application, but not all embodiments, the preferred embodiments of the present application are given in the drawings, but not limiting the patent scope of the present application. This application may be embodied in many different forms, but rather, embodiments are provided in order to provide a more thorough understanding of the present disclosure. Although the present application has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that modifications may be made to the embodiments described in the foregoing, or equivalents may be substituted for elements thereof. All equivalent structures made by the specification and the drawings of the application are directly or indirectly applied to other related technical fields, and are also within the protection scope of the application.
Claims (10)
1. A method for detecting multiple physiological indexes, the method comprising:
acquiring a face video of a user to be detected, wherein the face video comprises a plurality of face image sequences;
inputting the facial video of the user to be detected into a pulse wave signal prediction model to obtain a pulse wave signal of the user to be detected, wherein the pulse wave signal prediction model comprises a video quality enhancement module, a feature extraction model and a signal prediction module, and the video quality enhancement module is used for carrying out video quality enhancement processing on the facial video of the user to be detected to obtain a facial video with enhanced quality; the feature extraction module is used for extracting a space-time feature map according to the facial video with enhanced quality, wherein the space-time feature map comprises time features and space features contained in the plurality of facial image sequences; the signal prediction module is used for predicting and obtaining the pulse wave signal of the user to be detected according to the space-time characteristic diagram;
and calculating multiple physiological indexes of the user to be detected according to the pulse wave signals of the user to be detected.
2. The method of claim 1, wherein the video quality enhancement module is configured to perform video quality enhancement processing on the face video of the user to be detected, and obtaining the face video with enhanced quality comprises:
inputting the face video of the user to be tested to a downsampling path network for convolution processing, and outputting characteristic information of different scales of the face video of the user to be tested, wherein the downsampling path network comprises a plurality of 3D convolution blocks;
and inputting the feature information of different scales output by the downsampling path network into an upsampling path network for feature fusion processing, and outputting the facial video with enhanced quality, wherein the upsampling path network is connected with the downsampling path network through jumping, and the upsampling path network comprises a 3D deconvolution block symmetrical to the plurality of 3D convolution blocks.
3. The method of claim 2, wherein the feature extraction module is configured to extract a spatiotemporal feature map from the quality enhanced facial video, including:
the facial videos with the enhanced quality are respectively input into a 3D convolution network and a 2D convolution network, the 3D convolution network is used for extracting and obtaining time features in the facial videos of the users to be detected, and the 2D convolution network is used for extracting and obtaining space features in the facial videos of the users to be detected;
and multiplying the spatial features extracted by the 2D convolution network by the time features extracted by the 3D convolution network through a convolution attention mechanism, and outputting the space-time feature map.
4. The method of claim 3, wherein the signal prediction module is configured to predict, based on the spatiotemporal feature map, a pulse wave signal of the user under test comprising:
inputting the space-time feature images into an average pooling layer network to obtain feature values corresponding to each space-time feature image;
and inputting the characteristic value corresponding to each space-time characteristic diagram into a full-connection layer network, and outputting the pulse wave signal of the user to be detected, wherein the full-connection layer network is used for mapping the characteristic value into a corresponding signal value.
5. The method according to claim 1 or 2, wherein calculating the multiple physiological indexes of the user to be measured according to the pulse wave signal of the user to be measured comprises:
performing fast Fourier transform on the pulse wave signals of the user to be detected to obtain a pulse wave signal spectrogram;
filtering the pulse wave signal spectrogram by using filters with different cut-off frequencies and preset frequency values;
and calculating the heart rate and the respiratory rate of the user to be detected based on the frequency of the corresponding maximum peak value in the filtered spectrogram respectively.
6. The method according to claim 1 or 2, wherein calculating the multiple physiological indexes of the user to be measured according to the pulse wave signal of the user to be measured comprises:
determining all peak points of the pulse wave signals of the user to be detected;
calculating to obtain pulse intervals according to the sampling period and the distance between each peak point;
and calculating the heart rate variability of the user to be detected according to the pulse interval.
7. The method of claim 1 or 2, wherein the method further comprises:
acquiring a sampling personal face video and a real pulse wave signal corresponding to the sampling personal face video;
inputting the real pulse wave signals corresponding to the sample face video to an initial pulse wave signal prediction model;
the initial pulse wave signal prediction model predicts a pulse wave signal corresponding to the sample face video, compares the obtained pulse wave signal with the real pulse wave signal, and calculates a signal loss function;
and adjusting parameters and weights in the initial pulse wave signal prediction model according to the signal loss function to obtain the pulse wave signal prediction model.
8. A multiple physiological index detection device, the device comprising:
the data acquisition module is used for acquiring a face video of a user to be detected, wherein the face video comprises a plurality of face image sequences;
the prediction module is used for inputting the facial video of the user to be detected into a pulse wave signal prediction model to obtain a pulse wave signal of the user to be detected, wherein the pulse wave signal prediction model comprises a video quality enhancement module, a feature extraction model and a signal prediction module, and the video quality enhancement module is used for carrying out video quality enhancement processing on the facial video of the user to be detected to obtain a facial video with enhanced quality; the feature extraction module is used for extracting a space-time feature map according to the facial video with enhanced quality, wherein the space-time feature map comprises time features and space features contained in the plurality of facial image sequences; the signal prediction module is used for predicting and obtaining the pulse wave signal of the user to be detected according to the space-time characteristic diagram;
and the calculation module is used for calculating and obtaining multiple physiological indexes of the user to be detected according to the pulse wave signals of the user to be detected.
9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the method according to any of claims 1 to 7 when executing the computer program.
10. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the method according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310611339.4A CN116327133A (en) | 2023-05-29 | 2023-05-29 | Multi-physiological index detection method, device and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310611339.4A CN116327133A (en) | 2023-05-29 | 2023-05-29 | Multi-physiological index detection method, device and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116327133A true CN116327133A (en) | 2023-06-27 |
Family
ID=86876228
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310611339.4A Pending CN116327133A (en) | 2023-05-29 | 2023-05-29 | Multi-physiological index detection method, device and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116327133A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117064361A (en) * | 2023-10-16 | 2023-11-17 | 北京清雷科技有限公司 | Pulse wave-based heart rate variability analysis method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113223697A (en) * | 2021-04-23 | 2021-08-06 | 西北大学 | Remote plethysmography signal detection model construction and detection method, device and application |
| CN113674191A (en) * | 2021-08-23 | 2021-11-19 | 中国人民解放军国防科技大学 | Weak light image enhancement method and device based on conditional countermeasure network |
| CN113712526A (en) * | 2021-09-30 | 2021-11-30 | 四川大学 | Pulse wave extraction method and device, electronic equipment and storage medium |
-
2023
- 2023-05-29 CN CN202310611339.4A patent/CN116327133A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113223697A (en) * | 2021-04-23 | 2021-08-06 | 西北大学 | Remote plethysmography signal detection model construction and detection method, device and application |
| CN113674191A (en) * | 2021-08-23 | 2021-11-19 | 中国人民解放军国防科技大学 | Weak light image enhancement method and device based on conditional countermeasure network |
| CN113712526A (en) * | 2021-09-30 | 2021-11-30 | 四川大学 | Pulse wave extraction method and device, electronic equipment and storage medium |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117064361A (en) * | 2023-10-16 | 2023-11-17 | 北京清雷科技有限公司 | Pulse wave-based heart rate variability analysis method and device |
| CN117064361B (en) * | 2023-10-16 | 2023-12-19 | 北京清雷科技有限公司 | Pulse wave-based heart rate variability analysis method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111870237B (en) | Blood pressure detection device, blood pressure detection apparatus, and blood pressure detection medium | |
| KR101870121B1 (en) | System, method and program for analyzing blood flow by deep neural network | |
| CN108197592B (en) | Information acquisition method and device | |
| CN111611888B (en) | Non-contact blood pressure estimation device | |
| CN113569984B (en) | Brain perfusion state classification device, method, equipment and storage medium | |
| CN107374646A (en) | Blood sugar detection apparatus and method, electronic installation based near infrared technology | |
| CN116327133A (en) | Multi-physiological index detection method, device and related equipment | |
| EP3942567A1 (en) | Population-level gaussian processes for clinical time series forecasting | |
| CN104423574A (en) | Apparatus And Method For Enhancing System Usability | |
| CN114947886B (en) | Symbol digital conversion test method and system based on asynchronous brain-computer interface | |
| CN116712048A (en) | Blood pressure measurement method and device and related equipment | |
| CN113693611A (en) | Machine learning-based electrocardiogram data classification method and device | |
| CN113192639B (en) | Training method, device, equipment and storage medium of information prediction model | |
| CN110840401A (en) | Heart rate value measuring method, device and equipment and readable storage medium | |
| CN114557691B (en) | Non-invasive blood lipid detection method and system based on multi-wavelength PPG (photoplethysmography) signals | |
| CN116138756A (en) | Non-contact heart rate detection method, system and medium based on face feature point detection | |
| Zaki et al. | Smart medical chatbot with integrated contactless vital sign monitor | |
| CN115836847A (en) | Blood pressure prediction device and equipment | |
| CN112957018A (en) | Heart state detection method and device based on artificial intelligence | |
| CN115666376A (en) | System and method for hypertension monitoring | |
| CN113100736A (en) | Cerebral blood flow autonomic nerve disorder evaluation device, system and storage medium | |
| CN112971749B (en) | Fatigue detection method and device based on artificial intelligence | |
| CN110292388A (en) | A kind of measurement method and terminal of cognitive load and psychological pressure | |
| CN115966314B (en) | Data processing method and device, electronic equipment and storage medium | |
| CN113974554A (en) | Dicrotic wave identification method, apparatus, device and computer readable storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20230627 |