CN115624322B - Non-contact physiological signal detection method and system based on efficient space-time modeling - Google Patents

Abstract

The invention provides a non-contact physiological signal detection method and system based on efficient space-time modeling, and relates to the technical field of intelligent data processing. The method comprises the steps of obtaining an original video stream, preprocessing the original video stream, and obtaining a preprocessed image sequence; acquiring an image sequence, inputting the image sequence into a depth neural network based on a three-dimensional center differential convolution operator, and extracting space-time information by combining a attention mask mechanism of a convolution layer; based on the space-time information and the epsilon-insensitive Huber loss loss function, constructing a multi-task loss function to optimize the deep neural network; and filtering the optimized deep neural network by adopting a second-order Butterworth filter, and outputting the heart rate and the respiratory rate. The three-dimensional center difference convolution operator can be used for extracting pulse wave information by gathering time difference information, and cross-database evaluation and ablation research are carried out, so that the effectiveness and the robustness of the method are proved.

Description

A non-contact physiological signal detection method and system based on efficient spatio-temporal modeling

technical field

The invention relates to the technical field of intelligent data processing, in particular to a non-contact physiological signal detection method and system based on efficient spatio-temporal modeling.

Background technique

With the continuous development of modern society and the continuous improvement of people's living standards, the incidence of cardiovascular diseases is also increasing, which may be due to increased work pressure and accelerated pace of life. The detection of human physiological indicators is very important for the perception of human health. Early detection and treatment can effectively prevent and control cardiovascular diseases, and can avoid and reduce sudden death caused by cardiovascular problems. Traditional heart rate measurement methods, such as electrocardiograms, are contact heart rate measurements, which have the following limitations: First, they are not suitable for some specific application scenarios, such as scenarios that require long-term monitoring: newborns, patients with extensive burns, sleep monitoring, Driver monitoring, etc., and exposure measurements require the subject's subjective cooperation. The second is that when the position of the measuring instrument in contact with the skin deviates, it is easy to cause large deviations in the measurement results. Furthermore, despite the advantages of accurate measurements, an electrocardiograph is relatively expensive, requires professional operation, and is not suitable for routine physiological signal measurement. With the initial success of the non-contact remote photoplethysmography prototype method, Classical Signal Processing has demonstrated the feasibility of telephotoplethysmography-based heart rate measurement. However, these methods tend to degrade when encountering noise such as motion, lighting changes, and skin color. In practical applications, it is found that the method based on signal separation can only target specific interference, and cannot effectively deal with the coexistence of multiple interferences in real scenes.

For practical applications, recent research has started to focus on deep learning based methods due to their better representation capabilities. Several deep learning-based methods have been successfully applied to the task of remote photoplethysmography recovery with pulse or respiration as the target signal, but these methods still struggle to effectively model spatio-temporal information. Although methods exist to model spatiotemporal information by generating feature maps, the feature maps require preprocessing, including face detection, facial keypoint localization, face alignment, skin segmentation, and color space transformation, which are quite complex. In addition, due to the limitations of supervised learning, the performance of cross-database evaluation and real application will drop. The learned spatio-temporal features are still vulnerable to lighting conditions and motion, and they cannot fully utilize extensive temporal context information.

Contents of the invention

Aiming at the problem that in the prior art, the learned spatio-temporal features are still easily affected by lighting conditions and motion, and they cannot make full use of extensive temporal context information, the present invention proposes a non-contact physiological signal detection based on efficient spatio-temporal modeling methods and systems.

In order to solve the above technical problems, the present invention provides the following technical solutions:

In one aspect, a non-contact physiological signal detection method based on efficient spatio-temporal modeling is provided, the method is applied to electronic equipment, comprising the following steps:

S1: Obtain the original video stream, preprocess the original video stream, and obtain the preprocessed image sequence;

S2: Obtain the image sequence, input the image sequence into the deep neural network based on the three-dimensional central difference convolution operator, combine the attention mask mechanism of the convolution layer, and extract the spatiotemporal information;

S3: Based on spatiotemporal information and ε-insensitive Huber loss loss function, build a multi-task loss function to optimize the deep neural network;

S4: The second-order Butterworth filter is used to filter the optimized deep neural network, and the heart rate and respiration rate are output at the same time, and the non-contact physiological signal detection based on efficient spatiotemporal modeling is completed.

Optionally, in step S1, the original video stream is obtained, and the original video stream is preprocessed to obtain a preprocessed image sequence, including:

The original video stream is subjected to time-domain normalized difference and down-sampling processing to obtain a preprocessed image sequence;

Among them, the time domain normalized difference is calculated according to the following formula (1):

in Indicates the first skin pixels at time the RGB value, is the time-varying value.

Optionally, the image sequence includes: a time domain normalized difference image sequence and a downsampled image sequence.

Optionally, in step S2, an image sequence is obtained, and the image sequence is input into a deep neural network based on a three-dimensional central difference convolution operator, and combined with the attention mask mechanism of the convolution layer, spatiotemporal information is extracted, including:

S21: Taking the time-domain normalized difference image sequence as a motion branch, and inputting it into a deep neural network based on a three-dimensional central difference convolution operator;

The downsampled image sequence is used as the appearance branch and input into the deep neural network based on the three-dimensional central difference convolution operator;

S22: Through the attention mask mechanism, based on the appearance branch to model the skin area of interest, assist the motion branch to extract spatiotemporal information;

S23: Repeat steps S21-S22 to extract spatio-temporal information, and transmit the spatio-temporal information to the fully connected layer.

Optionally, in step S21, including:

The three-dimensional central difference convolution operator is obtained according to the following formula (2): :

in, is the input feature map, Represents the local receptive field cube, are learnable weights, Indicates the current position on the feature map, receptive field and adjacent time steps An enumeration of positions in the hyperparameter Used to balance spatial strength and gradients.

Optionally, in step S22, including:

The function of the attention masking mechanism is obtained according to the following formula (3): formula:

in, is the appearance branch The feature map of the layer convolutional layer; is the sports branch The feature map of the layer convolutional layer; and is the first The height and width of the feature map of the layer convolution layer; Indicates the sigmoid function, is the convolution kernel weight, is the convolution kernel bias, is the L1 norm, Represents an element-wise product.

Optionally, in step S3, based on the spatiotemporal information and the ε-insensitive Huber loss loss function, a multi-task loss function is constructed to optimize the deep neural network, including:

Calculate the intensity loss of pulse wave and respiratory wave according to the following formula (4) ε-insensitive Huber loss loss function :

in represents the truth value, Indicates input After the function The predicted value after mapping, Is the hyperparameter of Huberloss, the default value is 1, Is the hyperparameter of ε-insensitive loss, the default value is 0.1;

Combining the following formula (5) to construct a multi-task loss function :

in and is the loss function weight;

Optimize the weight of the deep neural network by backpropagating the loss function value, and stop the optimization after the loss function no longer decreases, that is, select the deep neural network with the lowest loss function value during the training process.

Optionally, in step S4, the second-order Butterworth filter is used to filter the optimized deep neural network, and the heart rate and respiration rate are simultaneously output to complete non-contact physiological signal detection based on efficient spatiotemporal modeling, including:

Using second-order Butterworth filter deep neural network, output heart rate and respiration rate at the same time;

Among them, the cut-off frequencies of heart rate are 0.75Hz and 2.5Hz, and the cut-off frequencies of respiratory frequency are 0.08Hz and 0.5Hz respectively;

Among them, the position of the highest peak in the power spectrum obtained by filtering the signal is selected as the output of heart rate and respiration rate, and the non-contact physiological signal detection based on efficient spatiotemporal modeling is completed.

In one aspect, a non-contact physiological signal detection system based on efficient spatiotemporal modeling is provided, the system is applied to electronic equipment, and the system includes:

The data acquisition module is used to obtain the original video stream, preprocess the original video stream, and obtain the preprocessed image sequence;

The spatiotemporal information extraction module is used to obtain image sequences, input the image sequences into the deep neural network based on the three-dimensional central difference convolution operator, and extract spatiotemporal information in combination with the attention mask mechanism of the convolutional layer;

The model optimization module is used to construct a multi-task loss function to optimize the deep neural network based on spatiotemporal information and ε-insensitive Huber loss loss function;

The data output module is used to use the second-order Butterworth filter to filter the optimized deep neural network, output heart rate and respiration rate at the same time, and complete non-contact physiological signal detection based on efficient spatiotemporal modeling.

Optionally, the data acquisition module is further configured to perform time-domain normalized difference and down-sampling processing on the original video stream, respectively, to obtain a preprocessed image sequence;

Among them, the time domain normalized difference is calculated according to the following formula (1):

in Indicates the first skin pixels at time the RGB value, is the time-varying value.

In one aspect, an electronic device is provided, the electronic device includes a processor and a memory, and at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to realize the above-mentioned high-efficiency-based Non-contact physiological signal detection method for spatiotemporal modeling.

In one aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the storage medium, and the at least one instruction is loaded and executed by a processor to realize the above-mentioned non-contact physiological signal based on efficient spatio-temporal modeling Detection method.

The above-mentioned technical solutions of the embodiments of the present invention have at least the following beneficial effects:

In the above scheme, an accurate non-contact physiological signal measurement method based on a 3D centered difference convolutional attention network is proposed for efficient spatiotemporal modeling, and the utilized 3D centered difference convolution operator can extract Pulse wave information.

The ε-insensitive Huber loss is proposed as the loss function of the non-contact pulse wave measurement network, because it can focus on the pulse wave intensity constraints, and the better performance of the ε-insensitive Huber loss loss function is shown by evaluating different loss functions and their combinations.

And further proposed a network for joint multi-task measurement of heart and respiratory motion, which has the advantage of sharing information between related physiological signals, can measure heart rate and respiratory rate at the same time, and further improve accuracy while reducing computational cost. Extensive experiments show that the proposed method has excellent performance on public databases. Cross-database evaluation and ablation studies are also conducted to demonstrate the effectiveness and robustness of the proposed method.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings that need to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those skilled in the art, other drawings can also be obtained based on these drawings without creative effort.

FIG. 1 is a flowchart of a non-contact physiological signal detection method based on efficient spatiotemporal modeling provided by an embodiment of the present invention;

2 is a flowchart of a non-contact physiological signal detection method based on efficient spatiotemporal modeling provided by an embodiment of the present invention;

Fig. 3 is a block diagram of a non-contact physiological signal detection system based on efficient spatio-temporal modeling provided by an embodiment of the present invention;

Fig. 4 is a schematic structural diagram of an electronic device provided by an embodiment of the present invention.

Detailed ways

In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will describe in detail with reference to the drawings and specific embodiments.

An embodiment of the present invention provides a non-contact physiological signal detection method based on efficient spatiotemporal modeling, which can be implemented by an electronic device, and the electronic device can be a terminal or a server. The flow chart of the non-contact physiological signal detection method based on efficient spatio-temporal modeling as shown in Figure 1, the processing flow of the method may include the following steps:

S101: Obtain the original video stream, perform preprocessing on the original video stream, and obtain a preprocessed image sequence;

S102: Obtain an image sequence, input the image sequence into a deep neural network based on a three-dimensional central difference convolution operator, combine the attention mask mechanism of the convolution layer, and extract spatiotemporal information;

S103: Based on spatiotemporal information and ε-insensitive Huber loss loss function, construct a multi-task loss function to optimize the deep neural network;

S104: Use the second-order Butterworth filter to filter the optimized deep neural network, output heart rate and respiration rate at the same time, and complete non-contact physiological signal detection based on efficient spatio-temporal modeling.

Optionally, in step S101, the original video stream is obtained, and the original video stream is preprocessed to obtain a preprocessed image sequence, including:

The original video stream is subjected to time-domain normalized difference and down-sampling processing to obtain the preprocessed image sequence:

Among them, the time domain normalized difference is calculated according to the following formula (1):

in Indicates the first skin pixels at time the RGB value, is the time-varying value.

Optionally, the image sequence includes: a time domain normalized difference image sequence and a downsampled image sequence.

Optionally, in step S102, an image sequence is obtained, and the image sequence is input into a deep neural network based on a three-dimensional central difference convolution operator, and combined with the attention mask mechanism of the convolution layer, spatiotemporal information is extracted, including:

S121: Taking the time-domain normalized difference image sequence as a motion branch, and inputting it into a deep neural network based on a three-dimensional central difference convolution operator;

The downsampled image sequence is used as the appearance branch and input into the deep neural network based on the three-dimensional central difference convolution operator;

S122: Through the attention mask mechanism, modeling the skin region of interest based on the appearance branch, assisting the motion branch to extract spatiotemporal information;

S123: Repeat steps S121-S122 to extract spatio-temporal information, and transmit the spatio-temporal information to the fully connected layer.

Optionally, in step S121, including:

The three-dimensional central difference convolution operator is obtained according to the following formula (2): :

in, is the input feature map, Represents the local receptive field cube, are learnable weights, represents the current position on the feature map, receptive field and adjacent time steps An enumeration of positions in the hyperparameter Used to balance spatial strength and gradients.

Optionally, in step S122, including:

The function of the attention masking mechanism is obtained according to the following formula (3): formula:

in, is the appearance branch The feature map of the layer convolutional layer; is the sports branch The feature map of the layer convolutional layer; and is the first The height and width of the feature map of the layer convolution layer; Indicates the sigmoid function, is the convolution kernel weight, is the convolution kernel bias, is the L1 norm, Represents an element-wise product.

Optionally, in step S103, based on spatiotemporal information and ε-insensitive Huber loss loss function, a multi-task loss function is constructed to optimize the deep neural network, including:

Calculate the intensity loss of pulse wave and respiratory wave according to the following formula (4) ε-insensitive Huber loss loss function :

in represents the truth value, Indicates input After the function The predicted value after mapping, Is the hyperparameter of Huber loss, the default value is 1, Is the hyperparameter of ε-insensitive loss, the default value is 0.1;

Combining the following formula (5) to construct a multi-task loss function :

in and For the loss function weights:

Optimize the weight of the deep neural network by backpropagating the loss function value, stop the optimization after the loss function no longer decreases, and select the deep neural network with the lowest loss function value during the training process.

Optionally, in step S104, the second-order Butterworth filter is used to filter the optimized deep neural network, and the heart rate and respiration rate are simultaneously output to complete non-contact physiological signal detection based on efficient spatiotemporal modeling, including:

Using second-order Butterworth filter deep neural network, output heart rate and respiration rate at the same time;

Among them, the cut-off frequencies of heart rate are 0.75Hz and 2.5Hz, and the cut-off frequencies of respiratory frequency are 0.08Hz and 0.5Hz respectively;

Among them, the position of the highest peak in the power spectrum obtained by filtering the signal is selected as the output of heart rate and respiration rate, and the non-contact physiological signal detection based on efficient spatiotemporal modeling is completed.

In the embodiment of the present invention, a remote photoplethysmography recovery method for non-contact measurement of physiological signals based on efficient spatiotemporal modeling is proposed. Its efficient spatiotemporal modeling is achieved by combining a 3D central difference convolution operator, a motion and appearance dual branch structure, and a soft attention mask. The 3D central difference convolution operator is good at describing the intrinsic pattern of the pulse wave through the combination of gradient and intensity information. The deep neural network based on 3D central difference convolution operator can provide more reliable spatiotemporal information modeling ability than traditional 3D convolution.

In addition, this patent first introduces the ε-insensitive Huberloss loss function in the remote photoplethysmography task, which has the advantages of L1 and L2 loss, and combines ε-insensitive so that the loss function can ignore the noise samples in the insensitive region, thereby increasing the robustness. stickiness, showing better performance.

An embodiment of the present invention provides a non-contact physiological signal detection method based on efficient spatiotemporal modeling, which can be implemented by an electronic device, and the electronic device can be a terminal or a server. As shown in Figure 2, the flow chart of the non-contact physiological signal detection method based on efficient spatiotemporal modeling, the processing flow of the method may include the following steps:

S201: Obtain an original video stream, perform preprocessing on the original video stream, and obtain a preprocessed image sequence;

In a feasible implementation manner, the original video stream is subjected to time-domain normalized difference and down-sampling processing, and the resolution after processing is 36×36×3. Obtain the preprocessed image sequence;

Among them, the time domain normalized difference is calculated according to the following formula (1):

.

in Indicates the first skin pixels at time the RGB value, is the time-varying value.

In a feasible implementation manner, the image sequence includes: a time-domain normalized difference image sequence and a downsampled image sequence. In this embodiment, the image sequences are all 10 frames.

S202: Taking the time-domain normalized difference image sequence as a motion branch, and inputting it into a deep neural network based on a three-dimensional central difference convolution operator;

The downsampled image sequence is used as the appearance branch and input into the deep neural network based on the three-dimensional central difference convolution operator;

S203: Through the attention mask mechanism, modeling skin ROI based on the appearance branch, assisting the motion branch to extract spatio-temporal information.

S204: Repeat steps S202-203 to extract spatio-temporal information, and transmit the spatio-temporal information to the fully connected layer.

In a feasible implementation manner, the three-dimensional central difference convolution operator is obtained according to the following formula (2):

in, is the input feature map, Represents the local receptive field cube, are learnable weights, Indicates the current position on the feature map, receptive field and adjacent time steps An enumeration of positions in the hyperparameter Used to balance spatial strength and gradients.

In a feasible implementation, the size of the convolution kernel is (3×3×3, 32), and the number of convolution layers is 2.

In a feasible implementation manner, the functional formula of the attention mask mechanism is obtained according to the following formula (2):

.

in, is the appearance branch The feature map of the layer convolutional layer; is the sports branch The feature map of the layer convolutional layer; and is the first The height and width of the feature map of the layer convolution layer; Indicates the sigmoid function, is the convolution kernel weight, is the convolution kernel bias, is the L1 norm, Represents an element-wise product.

In a feasible implementation, the stratified layer adopts average pooling, the kernel size is (2×2×2, 32), and it has a dropout probability of 0.5.

In a feasible implementation, the fully connected layer has input and output dimensions of 128 and 10, and has a dropout probability of 0.5.

S205: Based on spatiotemporal information and ε-insensitive Huber loss loss function, construct a multi-task loss function to optimize the deep neural network;

In a feasible implementation, the intensity loss of pulse wave and respiratory wave is calculated according to the following formula (4) ε-insensitive Huber loss loss function:

in represents the truth value, Indicates input After the function The predicted value after mapping, Is the hyperparameter of Huberloss, the default value is 1, Is the hyperparameter of ε-insensitive loss, the default value is 0.1;

Combining the following formula (5) as a multi-task loss function:

in and is the loss function weight;

Optimize the weight of the deep neural network by backpropagating the loss function value, stop the optimization after the loss function no longer decreases, and select the deep neural network with the lowest loss function value during the training process.

In a feasible implementation manner, the deep neural network model with the lowest loss function value is used as the final model.

In a feasible implementation, α=β=1 is adopted in an example, and finally optimize the weight of the neural network by backpropagating the loss function value, and stop when the loss function no longer decreases, and the model with the best effect at this time is used as the final The model adopts recovery pulse wave and respiration wave;

S206: Use the second-order Butterworth filter to filter the optimized deep neural network, output heart rate and respiration rate at the same time, and complete non-contact physiological signal detection based on efficient spatio-temporal modeling.

In a feasible implementation, a second-order Butterworth filter is used to further filter the network output heart rate and respiration rate, and the cut-off frequencies of the heart rate are 0.75 and 2.5 Hz, and the cut-off frequencies of the respiration frequency are 0.08 and 0.5 Hz, respectively;

The position of the highest peak in the power spectrum obtained by filtering the signal is selected as the output of heart rate and respiration rate, and the non-contact physiological signal detection based on efficient spatiotemporal modeling is completed.

In a feasible implementation mode, compared with the previous previous methods, the present invention mainly has two differences: one is the spatio-temporal network. Three-dimensional convolution and time-shifted convolution are used in traditional techniques, aiming to reduce the computational budget but without accuracy gain. The 3D central difference convolution operator used in this paper can replace the traditional convolution operation without additional parameters. The improved results are due to the enhanced spatio-temporal context modeling capability that facilitates the representation of appearance and motion information. At the same time, the central difference can be regarded as a regularization term to alleviate overfitting. The other is a network architecture, such as Dual-GAN, which is a design based on a generative confrontation network architecture for signal decoupling, and its performance in some metrics (such as the root mean square error on the UBFC dataset) is better than that of the present invention Methods. However, Dual GAN includes a preprocessing step called spatio-temporal map generation, including face detection, facial key point positioning, face alignment, skin segmentation, and color space transformation, which are relatively complex. Whereas the method of the present invention only requires a simple subtraction between frames as input to the motion branch.

In addition, in terms of loss function, the adopted ε-insensitive Huber loss when the loss between the predicted signal and the real signal is close to the minimum, the gradient decreases slowly with the Huber loss, so the model is more robust in signal prediction Stickiness. The multi-task network proposed by the present invention is capable of modeling internal dependencies, has accuracy advantages over the single-task version, and saves computing resources at the same time. Overall, the remote photoplethysmography measurement network based on efficient spatiotemporal modeling can capture rich temporal context by aggregating temporal difference information related to remote photoplethysmography, and obtain more robust and accurate non-contact physiological signal measurements result.

In the embodiment of the present invention, an accurate non-contact physiological signal measurement method based on a three-dimensional center difference convolution attention network is proposed for efficient spatio-temporal modeling. The three-dimensional center difference convolution operator used can gather time difference information to extract pulse wave information.

The ε-insensitive Huber loss is proposed as the loss function of the non-contact pulse wave measurement network, because it can focus on the pulse wave intensity constraints, and the better performance of the ε-insensitive Huber loss loss function is shown by evaluating different loss functions and their combinations.

A network for joint multi-task measurement of heart rate and respiratory motion is further proposed, which has the advantage of sharing information between related physiological signals, can measure heart rate and respiratory rate simultaneously, and further improves accuracy while reducing computational cost. Extensive experiments show that the proposed method has excellent performance on public databases. Cross-database evaluation and ablation studies are also conducted to demonstrate the effectiveness and robustness of the proposed method.

Fig. 3 is a block diagram of a non-contact physiological signal detection system based on efficient spatio-temporal modeling according to an exemplary embodiment. Referring to Figure 3, the system 300 includes:

The data collection module 310 is used to obtain the original video stream, preprocess the original video stream, and obtain a preprocessed image sequence;

The spatio-temporal information extraction module 320 is used to obtain the image sequence, input the image sequence into the deep neural network based on the three-dimensional central difference convolution operator, and extract the spatio-temporal information in combination with the attention mask mechanism of the convolutional layer;

The model optimization module 330 is used to construct a multi-task loss function to optimize the deep neural network based on the spatiotemporal information and the ε-insensitive Huber loss loss function;

The data output module 340 is configured to use the second-order Butterworth filter to filter the optimized deep neural network, output the heart rate and respiration rate simultaneously, and complete non-contact physiological signal detection based on efficient spatio-temporal modeling.

Optionally, the data acquisition module 310 is further configured to perform time-domain normalized difference and down-sampling processing on the original video stream, respectively, to obtain a preprocessed image sequence;

Among them, the time domain normalized difference is calculated according to the following formula (1):

in Indicates the first skin pixels at time the RGB value, is the time-varying value.

Optionally, the image sequence includes: a time domain normalized difference image sequence and a downsampled image sequence.

Optionally, the spatio-temporal information extraction module 320 is further configured to use the time-domain normalized difference image sequence as a motion branch, and input it into a deep neural network based on a three-dimensional central difference convolution operator;

The downsampled image sequence is used as the appearance branch and input into the deep neural network based on the three-dimensional central difference convolution operator;

Through the attention mask mechanism, the skin area of interest is modeled based on the appearance branch, and the motion branch is assisted to extract spatiotemporal information;

Repeat the extraction of spatio-temporal information, and pass the spatio-temporal information to the fully connected layer.

Optionally, the spatio-temporal information extraction module 320 is further used to

The three-dimensional central difference convolution operator is obtained according to the following formula (2): :

in, is the input feature map, Represents the local receptive field cube, are learnable weights, represents the current position on the feature map, receptive field and adjacent time steps An enumeration of positions in the hyperparameter Used to balance spatial strength and gradients.

Optionally, the spatio-temporal information extraction module 320 is further used to

The function of the attention masking mechanism is obtained according to the following formula (3): formula:

in, is the appearance branch The feature map of the layer convolutional layer; is the sports branch The feature map of the layer convolutional layer; and is the first The height and width of the feature map of the layer convolution layer; Indicates the sigmoid function, is the convolution kernel weight, is the convolution kernel bias, is the L1 norm, Represents an element-wise product.

Optionally, the model optimization module 330 is used to calculate the intensity loss of pulse wave and respiratory wave according to the following formula (4) ε-insensitive Huber loss loss function :

in represents the truth value, Indicates input After the function The predicted value after mapping, Is the hyperparameter of Huberloss, the default value is 1, Is the hyperparameter of ε-insensitive loss, the default value is 0.1;

Combine the following formula (5) to construct the multi-task loss function L ^Total :

in and is the loss function weight;

Optimize the weight of the deep neural network by backpropagating the loss function value, and stop the optimization after the loss function no longer decreases, that is, select the deep neural network with the lowest loss function value during the training process.

Optionally, the data output module 340 is configured to use a second-order Butterworth filter deep neural network to simultaneously output heart rate and respiration rate;

Among them, the cut-off frequencies of heart rate are 0.75Hz and 2.5Hz, and the cut-off frequencies of respiratory frequency are 0.08Hz and 0.5Hz respectively;

Among them, the position of the highest peak in the power spectrum obtained by filtering the signal is selected as the output of heart rate and respiration rate, and the non-contact physiological signal detection based on efficient spatiotemporal modeling is completed.

In the embodiment of the present invention, a remote photoplethysmography recovery method for non-contact measurement of physiological signals based on efficient spatiotemporal modeling is proposed. Its efficient spatiotemporal modeling is achieved by combining a 3D central difference convolution operator, a motion and appearance dual branch structure, and a soft attention mask. The 3D central difference convolution operator is good at describing the intrinsic pattern of the pulse wave through the combination of gradient and intensity information. The deep neural network based on 3D central difference convolution operator can provide more reliable spatiotemporal information modeling ability than traditional 3D convolution. In addition, this patent first introduces the ε-insensitive Huber loss loss function in the remote photoplethysmography task, and combines ε-insensitive so that the loss function can ignore the noise samples in the insensitive region to increase robustness and show better performance.

FIG. 4 is a schematic structural diagram of an electronic device 400 provided by an embodiment of the present invention. The electronic device 400 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 401 and one or more memory 402, wherein at least one instruction is stored in the memory 402, and the at least one instruction is loaded and executed by the processor 401 to realize the following non-contact physiological signal based on efficient spatiotemporal modeling The steps of the detection method:

S1: Obtain an original video stream, perform preprocessing on the original video stream, and obtain a preprocessed image sequence;

S2: Acquiring the image sequence, inputting the image sequence into a deep neural network based on a three-dimensional central difference convolution operator, and extracting spatiotemporal information in combination with the attention mask mechanism of the convolution layer;

S3: Based on the spatiotemporal information and the ε-insensitive Huber loss loss function, construct a multi-task loss function to optimize the deep neural network;

S4: The second-order Butterworth filter is used to filter the optimized deep neural network, and the heart rate and respiration rate are output at the same time, and the non-contact physiological signal detection based on efficient spatiotemporal modeling is completed.

In an exemplary embodiment, there is also provided a computer-readable storage medium, such as a memory including instructions, which can be executed by a processor in the terminal to implement the above-mentioned non-contact physiological signal detection method based on efficient spatio-temporal modeling. For example, the computer readable storage medium may be ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like.

Those of ordinary skill in the art can understand that all or part of the steps for implementing the above embodiments can be completed by hardware, and can also be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium. The above-mentioned The storage medium mentioned may be a read-only memory, a magnetic disk or an optical disk, and the like.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection of the present invention. within range.

Claims (2)

1. The non-contact physiological signal detection method based on the efficient space-time modeling is characterized by comprising the following steps of:

s1: acquiring an original video stream, preprocessing the original video stream, and acquiring a preprocessed image sequence;

s2: acquiring the image sequence, inputting the image sequence into a depth neural network based on a three-dimensional center differential convolution operator, and extracting space-time information by combining a notice mask mechanism of a convolution layer;

s3: constructing a multi-task loss function to optimize the deep neural network based on the space-time information and the epsilon-insensitive Huber loss loss function;

in the step S3, constructing a multi-task loss function to optimize the deep neural network based on the spatio-temporal information and the epsilon-insensitive Huber loss loss function includes:

calculating the intensity loss of pulse wave and respiratory wave according to the following equation (4) epsilon-insensitive Huber loss loss function

：

（4）

Wherein the method comprises the steps of

Indicating true value(s)>

Representation input +.>

Through a function->

Mapped prediction value ∈ ->

Is a superparameter of Huber loss, default 1, < >>

Is an superparameter of epsilon-intrinsic loss, and the default value is 0.1;

the following formula (5) is combinedBuilding a multitasking loss functionL ^Total ：

（5）

Wherein the method comprises the steps of

And->

Weights for a loss function;

optimizing the weight of the deep neural network through the back propagation loss function value, stopping optimizing after the loss function is not reduced any more, and selecting a model of the deep neural network with the lowest loss function value in the training process;

s4: and filtering the optimized deep neural network by adopting a second-order Butterworth filter, and outputting the heart rate and the respiratory rate simultaneously to finish non-contact physiological signal detection based on efficient space-time modeling.

2. A non-contact physiological signal detection system based on efficient spatiotemporal modeling, characterized in that the system is adapted for use in the method of claim 1 above, the system comprising:

the data acquisition module is used for acquiring an original video stream, preprocessing the original video stream and acquiring a preprocessed image sequence;

the space-time information extraction module is used for acquiring the image sequence, inputting the image sequence into a deep neural network based on a three-dimensional center difference convolution operator, and extracting space-time information by combining a attention mask mechanism of a convolution layer;

the model optimization module is used for constructing a multi-task loss function to optimize the deep neural network based on the space-time information and the epsilon-insensitive Huber loss loss function;

and the data output module is used for filtering the optimized deep neural network by adopting a second-order Butterworth filter, outputting heart rate and respiratory rate at the same time, and finishing non-contact physiological signal detection based on efficient space-time modeling.

Images