CN114041780B - Method for monitoring respiration based on data acquired by inertial sensor - Google Patents

Abstract

The invention provides a respiration monitoring model, which comprises a processing main road and a plurality of processing branches, wherein each processing branch and the main road comprise convolution filters, each convolution filter comprises a plurality of convolution networks for filtering input data and a multi-head self-attention mechanism layer arranged between the corresponding convolution networks for enhancing the global receptive field, and the respiration monitoring model is configured to: respectively inputting each modal data in the multi-modal data obtained based on the inertial sensor data into a corresponding processing branch circuit for convolution filtering to obtain a filtering result of each modal data; and inputting multi-modal respiration characteristics obtained by superposing filtering results of the modal data into a processing main path for convolution filtering to obtain correlation respiration characteristics, and generating a respiration waveform based on the correlation respiration characteristics. The invention generates the respiration waveform through the model, thereby monitoring the human respiration condition.

Description

A method for breathing monitoring based on data collected by inertial sensors

technical field

The invention relates to the fields of digital signal processing and deep learning, and in particular, to a method for breathing monitoring based on data collected by an inertial sensor.

Background technique

Respiratory monitoring is of self-evident importance in physical and mental health, as well as of great clinical medical and social significance. However, as a traditional medical device, the respiratory monitor is expensive and professional, and its use of an invasive airflow detection method is not conducive to long-term wearing. With the development of sensor technology, some civilian respiratory monitoring products have been developed. For example, the thoracic and abdominal winding respiration monitoring products use pressure-sensitive sensors or magnetic flux sensors to monitor the movement of the thoracic and abdominal cavity caused by respiration to generate respiratory waveforms. There are also breathing mask-type breathing monitoring products, which monitor breathing by wearing a cover in the mouth and nose and using a gas flow meter to sense the gas exchange generated by breathing. Although these methods reduce the price of medical devices, they are still expensive, lose some precision, and have poor wearability. As a result, people cannot wear them continuously in their daily life, nor can they use them as a portable wearable device to use the health services they provide.

Therefore, the realization of universal and universal respiratory monitoring has always been the goal of this field, as early as the 1930s, Starr et al proposed Ballistocardiography. In the same way, the breathing movement performed by the human body will also be reflected on the human limbs, producing weak breathing movement signals on the limbs. Sensors such as an accelerometer and a gyroscope in an inertial measurement unit (Inertial Measurement Unit, IMU) can be used to collect motion signals, and filter them to obtain a breathing waveform.

However, the human limbs are accompanied by daily behaviors, resulting in a large number of motion artifacts in the IMU data, and the weak breathing signal is aliased by the motion artifacts. As a result, the traditional filtering algorithm cannot firstly analyze and process the IMU data with complex interference generated by motion, that is, it does not filter from multiple perspectives of multimodal data, lacking generalization and low accuracy, and secondly, it does not use multiple Convolutional networks provide better filtering capabilities. Finally, an existing adaptive filtering algorithm relies on the partial knownness of the expected breathing signal to adjust parameters. However, because the amplitude, curvature and frequency of breathing are important indicators for reflecting the physiological state, they cannot be predicted in advance, and are constantly changing. Weak changes are carried out, and the multi-head attention mechanism is not used to make up for the excessive attention to local features in the filtering process, resulting in the problems of low precision and no global receptive field in the respiratory waveform extracted by the adaptive filtering algorithm.

Therefore, there is an urgent need for a respiration monitoring system based on the data collected by the IMU for filtering, which can overcome the problems of difficult extraction of weak features and complex interference caused by human motion, so as to obtain high-precision respiration waveforms.

SUMMARY OF THE INVENTION

Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art, and to provide a method for breathing monitoring based on data collected by an inertial sensor.

The purpose of this invention is to realize through the following technical solutions:

According to a first aspect of the present invention, a breathing monitoring model includes a processing general circuit and a plurality of processing branches, each processing branch and processing general circuit includes a convolution filter, and each convolution filter includes a For a plurality of convolutional networks for filtering input data and a multi-head self-attention mechanism layer arranged between the corresponding convolutional networks for enhancing the global receptive field, wherein the breathing monitoring model is configured to: based on inertial Each modal data in the multimodal data obtained from the sensor data is input to the corresponding processing branch for convolution filtering to obtain the filtering result of each modal data; and the filtering results of each modal data are superimposed to obtain The multimodal respiration features are input to the processing circuit for convolution filtering to obtain relevant respiration features, and a respiration waveform is generated based on the relevant respiration features.

In some embodiments of the present invention, the convolution filter further includes a downsampling layer and an upsampling layer arranged between the corresponding convolutional networks; the downsampling layer is used to filter the input data of the corresponding convolutional network to obtain The result is downsampled to obtain low-dimensional features; the upsampling layer is used to upsample the results obtained by filtering the corresponding low-dimensional features input by the corresponding convolutional network to obtain high-dimensional features; among them, low-dimensional features and high-dimensional features The features use multi-head self-attention mechanism layers to enhance their respective global receptive fields.

In some embodiments of the present invention, the convolutional filter includes a first convolutional network, a second convolutional network, a third convolutional network and a fourth convolutional network, all of which are one-dimensional convolutional networks, the first convolutional network The input end of the convolution network is the input end of the convolution filter, and the output end of the fourth convolution network is the output end of the convolution filter; there is a multi-head self-attention between the first convolution network and the fourth convolution network The mechanism layer has a downsampling layer between the second convolutional network and the third convolutional network, and a downsampling layer between the second convolutional network and the fourth convolutional network. There is a multi-head self-attention mechanism layer between the three convolutional networks; an upsampling layer is arranged between the third convolutional network and the fourth convolutional network.

In some embodiments of the present invention, the convolution filter is configured to: process the input data sequentially through the first convolution network and the multi-head self-attention mechanism layer to obtain the first high-dimensional feature; The data is sequentially processed by the first convolutional network, the downsampling layer, the second convolutional network and the upsampling layer to obtain the second high-dimensional feature; the input data is sequentially processed by the first convolutional network and the downsampling layer, Obtain the first low-dimensional feature; pass the input data through the first convolutional network, the downsampling layer, the second convolutional network, and the multi-head self-attention mechanism layer in turn to obtain the second low-dimensional feature; After superimposing the dimensional feature and the second low-dimensional feature, the third convolutional network and the upsampling layer are processed in turn to obtain the third high-dimensional feature; the first high-dimensional feature, the second high-dimensional feature and the third high-dimensional feature are combined. After superposition, the filtering result is obtained by processing the fourth convolutional network.

In some embodiments of the present invention, the processing branch or the processing general circuit includes one or more convolution filters; when the processing branch or the processing general circuit includes a plurality of convolution filters, many of them are Any two convolution filters in the convolution filters are connected in the manner of vertical stacking connection or horizontal stacking connection, wherein the connection mode of the first convolution filter and the second convolution filter of the vertical stack connection is The output end of the second convolution network of a convolution filter is connected to the input end of the second convolution filter, and the output end of the second convolution filter is connected to the input end of the third convolution network of the first convolution filter. The first convolution filter and the second convolution filter connected in a horizontal stack are connected in such a way that the output end of the first convolution filter and the input end of the second convolution filter are connected to each other.

According to a second aspect of the present invention, there is provided a training method for the respiration monitoring model described in the first aspect of the present invention, comprising performing multiple iterative training on the respiration monitoring model in the following manner: acquiring a training set, wherein all The input data of the samples in the training set is each modal data in the multi-modal data obtained based on the inertial sensor data corresponding to the corresponding time window, and the label of the sample is the standard breathing waveform corresponding to the corresponding time window; use the training set to train breathing The monitoring model performs convolution filtering on the input data to generate a respiratory waveform; based on the difference between the generated respiratory waveform and the standard respiratory waveform, the total loss value is calculated; based on the total loss value, the parameters of the respiratory monitoring model are updated to obtain a trained respiratory monitoring model.

In some embodiments of the present invention, the total loss value is calculated based on the Huber loss and the L2 regularization term calculated based on the generated respiratory waveform and the standard respiratory waveform, and the calculation method is as follows:

x_i是基于第i个样本生成的呼吸波形，y_i是第i个样本的标签，||x_i||₂表示x_i的2范数，||y_i||₂表示y_i的2范数，max(||x_i||₂·||y_i||₂，∈)表示取||x_i||₂·||y_i||₂和∈中的最大值作为分母，∈为标量值，α是L2正则化项的参数，Ω是L2正则化项。Among them, n is the total number of samples, _zi is the Huber loss of the ith sample, equal to

_xi is the respiratory waveform generated based on the ith sample, _yi is the label of the ith sample, || _xi || ₂ represents the 2 norm of _xi , ||y _i || ₂ represents the 2 of _yi Norm, max(||x _i || ₂ ·||y _i || ₂ , ∈) means taking the maximum value of ||x _i || ₂ ·||y _i || ₂ and ∈ as the denominator, ∈ is a scalar value, α is the parameter of the L2 regularization term, and Ω is the L2 regularization term.

According to a third aspect of the present invention, a system for respiratory monitoring based on data collected by an inertial sensor is provided, comprising: a data processing module for obtaining acceleration modal data, angular velocity model data based on data collected by an inertial sensor in a wearable device modal data and Euler angle modal data; the respiration monitoring model trained by the method in the second aspect of the present invention processes the acceleration modal data, angular velocity modal data and Euler angle modal data to generate a breathing waveform.

According to a fourth aspect of the present invention, there is provided a method for breathing monitoring based on data collected by an inertial sensor, comprising: obtaining acceleration modal data, angular velocity modal data and Euler angles based on data collected by an inertial sensor in a wearable device Modal data; the respiration monitoring model trained by the method in the second aspect of the present invention processes the acceleration modal data, angular velocity modal data and Euler angle modal data to generate a respiration waveform.

According to a fifth aspect of the present invention, there is provided an electronic device comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to execute the The instructions are executable to implement the steps of the method of any one of the second and fourth aspects of the present invention.

Compared with the prior art, the advantages of the present invention are:

1. In the breathing monitoring model of the present invention, first, the convolution filters of each processing branch respectively carry out convolution filtering on each modal data, wherein each modal data is obtained based on the processing of inertial sensor data, enhanced The weak inertial sensor data features multimodal data, while convolution filtering is performed from all angles of the enhanced multimodal data to avoid overfitting and lack of generalization; secondly, the convolution filter of the processing general The multi-modal respiratory features obtained by superimposing the filtering results of each modal data are subjected to convolution filtering, and the filtering results of the multi-modal data are combined to generate the respiratory waveform to improve the accuracy of the model. The convolution filter of the road performs convolution filtering based on the convolution network to enhance the filtering ability, and uses the multi-head self-attention mechanism to enhance the global receptive field of the breathing waveform.

2. The convolution filter in the model of the present invention, when performing convolution filtering on the input data, uses a very small amount of convolution network, and combines the multi-head self-attention mechanism layer, down-sampling layer and up-sampling layer to achieve Multiple encoding and decoding of the input data, again to avoid model overfitting and lack of generalization, to achieve better filtering capabilities. In addition, the convolutional filter has good modularity and scalability, and multiple convolutional filters can be connected vertically and horizontally according to the needs of the task. Vertical stacking connections can increase the number of down-sampling layers and up-sampling layers, enabling deeper encoding and decoding and enhancing filtering capabilities. Horizontal stacking connections can increase the number of fittings to the features and achieve better fitting of the features.

3. In the breathing monitoring method of the present invention, the acceleration modal data and the angular velocity modal data are fused together with low-pass filtering and attitude calculation complementary filtering algorithm, so as to enhance the characteristics of the multi-modal data of the inertial sensor data, and make the breathing monitoring model more accurate. Perform feature extraction well, and use the mean smoothing module to continuously infer and optimize the respiratory waveform covering amplitude, curvature, and frequency changes to make the waveform smoother.

Description of drawings

The embodiments of the present invention will be further described below with reference to the accompanying drawings, wherein:

1 is a schematic diagram of a system for performing respiratory monitoring according to an embodiment of the present invention;

2 is a schematic structural diagram of a breathing monitoring model according to an embodiment of the present invention;

3 is a schematic structural diagram of a convolution filter according to an embodiment of the present invention;

4 is a schematic structural diagram of a one-dimensional convolution network in a convolution filter according to an embodiment of the present invention;

FIG. 5 is a schematic structural diagram of a multi-head self-attention mechanism layer according to an embodiment of the present invention;

6 is a schematic structural diagram of a breathing monitoring model according to another embodiment of the present invention;

7 is a schematic structural diagram of vertical stacking and connection of multiple convolution filters according to an embodiment of the present invention;

8 is a schematic structural diagram of a horizontal stacking connection of multiple convolution filters according to an embodiment of the present invention;

FIG. 9 is a schematic diagram of a breathing waveform and a real breathing waveform output by a breathing monitoring model according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings through specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

As mentioned in the background art section, human limbs are accompanied by daily behaviors, so that a large number of motion artifacts are generated in the IMU data, and the weak breathing signal is greatly aliased by the motion artifacts. As a result, the traditional filtering algorithm cannot firstly analyze and process the IMU data with complex interference generated by motion, that is, it does not filter from multiple perspectives of multimodal data, lacking generalization and low accuracy, and secondly, it does not use multiple Convolutional networks provide better filtering capabilities. Finally, an existing adaptive filtering algorithm relies on the partial knownness of the expected breathing signal to adjust parameters. However, because the amplitude, curvature and frequency of breathing are important indicators for reflecting the physiological state, they cannot be predicted in advance, and are constantly changing. Weak changes are carried out, and the multi-head attention mechanism is not used to make up for the excessive attention to local features in the filtering process, resulting in the problems of low precision and no global receptive field in the respiratory waveform extracted by the adaptive filtering algorithm.

In view of the defects of the existing method, according to an embodiment of the present invention, a system for monitoring respiration based on data collected by an inertial sensor is provided. Referring to FIG. 1 , it includes a data processing module, a respiration monitoring model and a mean smoothing module. First, by The data processing module obtains the accelerometer data and gyroscope data of the inertial sensor worn on the human body, and processes the two kinds of data. During the processing, the characteristics of each data of the inertial sensor are enhanced to obtain acceleration modal data, angular velocity modal data and Euler angle modal data; secondly, convolution filtering is performed on each modal data through the convolution filter of the corresponding processing branch 2 of the breathing monitoring model, so as to realize multi-angle filtering from multi-modal data and avoid breathing To monitor the over-fitting and lack of generalization of the model, the convolution filter provided in the main circuit 1 superimposes the filtering results of each modal data to perform convolution filtering on the multi-modal respiratory features obtained by convolution filtering to obtain correlated respiratory features. , and generate the respiratory waveform based on the relevant respiratory features to improve the accuracy of the model; finally, the optimized respiratory waveform is obtained through the mean smoothing module.

Further, the convolution filters provided in each processing branch 2 and processing main circuit 1 perform convolution filtering on the basis of multiple convolutional networks to enhance the filtering ability, and use the multi-head self-attention mechanism layer to enhance the global experience of the breathing waveform. wild.

Before the embodiments of the present invention are specifically introduced, some terms used therein are explained as follows:

High-dimensional features: refers to the inclusion of many features that are irrelevant to the learning task of the respiratory monitoring model (for example, there are many features with only weak correlations), and many features that are redundant with the learning task of the respiratory monitoring model (for example, there are strong mutual correlation) and noise data. For example, the original features of each modal data in the present invention and the convolution filter output results of performing convolution filtering on each modal data are high-dimensional features.

Low-dimensional features: refers to the features obtained by reducing the dimensionality of high-dimensional features (excluding features that are irrelevant to the learning task of the breathing monitoring model and partially redundant features). For example, in the present invention, the original features of each modal data are down-sampled to achieve dimensionality reduction of the original features, and the obtained features are low-dimensional features.

In order to better understand the present invention, the structure of each part of the respiratory monitoring model will be described in detail below with reference to specific embodiments.

According to an embodiment of the present invention, a respiration monitoring model is provided, see FIG. 2 , its structure includes a processing main circuit 1 and three processing branches 2 , and each processing branch 2 and processing general circuit 1 are provided with convolution filtering device. The output ends of the three processing branches 2 superimpose the output results in a cascaded manner and input them to the input end of the processing main circuit 1 . In the input part of the three processing branches 2, each data in the array of (N, 3, 256) on the arrow indicates that the corresponding batch number is N, the channel number is 3 and the length is 256 when the corresponding data is input. One of the arrays in the processing branch 2 whose tail is (N, 8, 256) indicates that when the corresponding data is output, the corresponding batch number is N, the number of channels is 8 and the length is 256, and the head is (N, 8, The array of 128) indicates that the number of batches corresponding to the input of the corresponding data is N, the number of channels is 8 and the length is 128.

According to an embodiment of the present invention, the breathing monitoring model is configured to: input each modal data in the multimodal data obtained based on inertial sensor data into the corresponding processing branch 2 to perform convolution filtering to obtain each modal The filtering results of the data; and the multi-modal breathing features obtained by superimposing the filtering results of the modal data are input into the processing circuit 1 for convolution filtering to obtain relevant breathing features, and a breathing waveform is generated based on the relevant breathing features.

According to an embodiment of the present invention, each convolution filter in the processing bus 1 and the plurality of processing branches 2 includes a plurality of convolutional networks for filtering input data, which are arranged in the corresponding convolutional networks. The multi-head self-attention mechanism layer for enhancing the global receptive field between the two layers and the down-sampling layer and the up-sampling layer are arranged between the corresponding convolutional networks. Referring to FIG. 3 , the specific structure of each convolutional filter may be as follows: the multiple convolutional networks include a first convolutional network, a second convolutional network, a third convolutional network, and a fourth convolutional network, all of which are one-dimensional convolutional networks. Convolutional network, the input end of the first convolution network is the input end of the convolution filter, and the output end of the fourth convolution network is the output end of the convolution filter. Among them, a multi-head self-attention mechanism layer is arranged between the first convolutional network and the fourth convolutional network, a downsampling layer is arranged between the first convolutional network and the second convolutional network, and a downsampling layer is arranged between the third convolutional network; An upsampling layer is set between the second convolutional network and the fourth convolutional network, and a multi-head self-attention mechanism layer is set between the second convolutional network and the third convolutional network; an upsampling layer is set between the third convolutional network and the fourth convolutional network Floor.

According to an embodiment of the present invention, the down-sampling layer of the convolution filter is used to down-sample the result obtained by filtering the input data by the corresponding convolution network to obtain low-dimensional features; the up-sampling layer of the convolution filter uses Upsampling the results obtained by filtering the corresponding low-dimensional features input by the corresponding convolutional network to obtain high-dimensional features, wherein the low-dimensional features and high-dimensional features use multi-head self-attention mechanism layers to enhance their respective global receptive fields. .

According to an embodiment of the present invention, the downsampling layer adopts a maximum pooling strategy, and the upsampling layer adopts a linear interpolation algorithm, wherein the downsampling layer is set after the output end of the corresponding convolutional network, and the maximum pooling strategy is used to obtain the reserved The low-dimensional features of the core features are helpful for the feature abstraction of complex signals. It selects the regional maximum value for the output of the corresponding convolutional network, which effectively helps the convolution kernel to perform time-invariant feature discovery and increase robustness. The linear interpolation algorithm used in the downsampling layer expands the low-dimensional features to obtain high-dimensional features. The linear interpolation algorithm has the nature of low-pass filtering, and focuses more on the core features of the low-dimensional features, making the final generated respiratory waveform smoother.

According to an embodiment of the present invention, each convolutional network includes a multi-layer structure. For example, referring to FIG. 4 , each convolutional network includes a one-dimensional convolutional layer, a normalized operation layer, and a linear rectification activation function layer (Rectified LinearUnit, ReLU), 1D convolution layer, normalization operation layer and linear rectification activation function layer.

According to an embodiment of the present invention, referring to FIG. 5 , the multi-head self-attention mechanism layer includes h self-attention heads (Attention _i ) and a fully connected layer. The principle of the multi-head self-attention mechanism formed by the connection of the input terminals of the fully connected layer is as follows:

Each self-attention head performs linear transformation on the input feature F, and weights the attention of the h self-attention heads to obtain the processing result head of each self-attention head.

For example, the i-th self-attention layer performs linear transformation on the input feature F to obtain

为对应的权重矩阵，

为对应的偏置矩阵；index:

is the corresponding weight matrix,

is the corresponding bias matrix;

为对应的权重矩阵，

为对应的偏置矩阵；key:

is the corresponding weight matrix,

is the corresponding bias matrix;

为对应的权重矩阵，

为对应的偏置矩阵；value:

is the corresponding weight matrix,

is the corresponding bias matrix;

Carry out the attention weighting of the h self-attention heads to obtain the processing result of each self-attention head head _i =Attention(Q _i ,K _i ,V _i ), as follows:

为输入键K_i的维度，

为V_i的注意力权重矩阵，系数

用于对注意力权重矩阵进行标准化，最后，通过softmax层对V_i进行注意力加权。in,

is the dimension of the input key _Ki ,

is the attention weight matrix of V _i , the coefficient

It is used to normalize the attention weight matrix, and finally, the attention weights V _i through the softmax layer.

The outputs of the h self-attention heads are cascaded and input to the fully connected layer for processing to obtain the final output MultiHead(Q, K, V) of the multi-head self-attention mechanism layer, as follows:

MultiHead(Q, K, V) = Concat(head ₁ , . . . , head _h ) W ^O +B ^O ;

Among them, Q represents the final index of the h indices obtained by linearly transforming the features by the h self-attention heads, and K represents the final index of the h keys obtained by linearly transforming the features by the h self-attention heads. key, V represents the final value of h values obtained by linearly transforming the features of the h self-attention heads, Concat represents the combination of head ₁ , ..., head _h into a string, and W ^O is each The corresponding weight matrices linearly transformed from the attention heads, ^BO is the corresponding bias matrix of each self-attention head linearly transformed.

The multi-head self-attention mechanism layer of the convolutional filter has the ability of global analysis. For example, for the characteristics of long breathing cycle and weak features, global time-series correlation analysis of long sequences is performed to make up for the structure with multiple convolutional networks. The defect of lack of global receptive field due to local features. Moreover, the structure with multiple convolutional networks can also make up for the defect that the multi-head self-attention mechanism layer is suitable for high-semantic sequence feature processing, and has poor processing effect on low-semantic multimodal data, providing better filtering capabilities. Multiple convolutional networks are used to filter low-semantic multi-modal data to obtain high-semantic sequence features, and a multi-head self-attention mechanism layer is used to discover global temporal correlation feature signals to enhance the global receptive field.

According to an embodiment of the present invention, when filtering the input data, the convolution filter further includes using a very small number of convolutional networks and combining the down-sampling layer and the up-sampling layer to realize multiple encoding and decoding of the input data, Again, model overfitting and lack of generalization are avoided, and filtering capabilities are enhanced. Preferably, the convolution filter filters the input data in the following ways:

The convolution filter sequentially processes the input data through the first convolutional network and the multi-head self-attention mechanism layer to obtain the first high-dimensional feature.

The first encoding and decoding process for multiple encoding and decoding is: the convolution filter passes the input data through the first convolutional network, the downsampling layer, the second convolutional network and the upsampling layer in turn to obtain a second high-dimensional feature.

The second encoding and decoding process is as follows: the convolution filter passes the input data through the first convolution network and the downsampling layer in turn to obtain the first low-dimensional feature; the input data passes through the first convolution network, The downsampling layer, the second convolution network, and the multi-head self-attention mechanism layer are processed to obtain the second low-dimensional feature; the first low-dimensional feature and the second low-dimensional feature will be superimposed and then passed through the third convolutional network and the second low-dimensional feature in turn. The processing of the upsampling layer, the third high-dimensional feature is obtained.

The convolution filter superimposes the first high-dimensional feature, the second high-dimensional feature, and the third high-dimensional feature obtained above, and then processes the fourth convolutional network to obtain a filtering result.

According to an embodiment of the present invention, referring to FIG. 6 , each processing branch 2 of the respiration monitoring model is provided with one convolution filter, the processing circuit 1 is provided with two convolution filters, and the two convolution filters Connect in a vertical stack connection. The connection mode of the first convolution filter and the second convolution filter connected by vertical stacking is shown in FIG. 7 , the output end of the second convolution network including the first convolution filter and the output end of the second convolution filter The input end is connected, and the output end of the second convolution filter is connected with the input end of the third convolution network of the first convolution filter. By increasing the number of down-sampling layers and the number of up-sampling layers by means of the vertical stack connection, deeper encoding and decoding can be achieved.

According to another embodiment of the present invention, the two convolution filters of the processing bus 1 can also be connected in a horizontally stacked connection, see FIG. 8 , that is, the connection of the first convolution filter and the second convolution filter The manner includes interconnecting the output of the first convolution filter and the input of the second convolution filter. Horizontal stacking connections can increase the number of fittings to the features and achieve better fitting of the features.

According to another embodiment of the present invention, the convolution filter has good modularity and scalability, and can be connected vertically and horizontally according to the needs of the task. The processing branch 2 of the respiratory monitoring model can be set as a convolution filter, or a plurality of convolution filters connected by vertical stacking and/or horizontal stacking can be set, and the processing circuit 1 can also be set as a convolution filter , or set up multiple convolution filters connected vertically and/or horizontally.

According to another embodiment of the present invention, when multiple convolution filters are set in the processing branch 2 or the processing bus 1, any two convolution filters in the multiple convolution filters are stacked vertically Connection or horizontal stacking connection, if processing bus 1 is set with three convolution filters, the first convolution filter and the second convolution filter are vertically stacked and connected, and the second convolution filter Connect horizontally with the third convolutional filter; or all three convolutional filters are connected horizontally; or all are connected vertically.

According to an embodiment of the present invention, the three processing branches 2 of the respiration monitoring model respectively perform convolution filtering on acceleration modal data, angular velocity modal data and Euler angle modal data obtained by processing inertial sensor data. . Multi-angle discovery of respiratory features on multi-modal data to avoid the difficulty of extracting too weak features, overfitting and lack of generalization, while enhancing the filtering ability of the respiratory monitoring model.

According to an embodiment of the present invention, in order to ensure the accuracy of the breathing monitoring model, it needs to be trained through a large amount of sample data. According to one embodiment of the present invention, the sample can be obtained by:

When acquiring each modal data and collecting the standard respiratory waveform, each data frame will be marked with a timestamp, and the standard respiratory waveform and each modal data will be aligned with time and data frame number and divided into a certain time window size (for example, 256 data frame) to obtain the training set. The standard respiratory waveform corresponding to the corresponding time window is used as the label of the sample, and the modal data corresponding to the corresponding time window is used as the input data of the sample. Among them, each sample includes 256 sets of input data as follows:

(a _x , a _y , a _z , g _x , g _y , g _z , a' _x , a' _y , a' _z ), where each set of data corresponds to the three channels of X-axis, Y-axis, and Z-axis Acceleration modal data at the corresponding moment: a _x , a _y , _az ; angular velocity modal data of the three channels of X-axis, Y-axis, Z-axis at the corresponding moment: g _x , g _y , g _z ; Euler angle modal data at time: a' _x , a' _y , a' _z .

According to an embodiment of the present invention, after the training set is obtained, the respiration monitoring model is trained through the training set, and the parameters of the model are adjusted at the same time, so that the finally obtained respiration monitoring model can output a more accurate respiration waveform. The above-mentioned method for training a breathing monitoring model on the training set obtains a better breathing monitoring model. The method includes performing multiple iterations of training on the breathing monitoring model according to the following steps a1, a2, a3 and a4:

Step a1: Obtain a training set, wherein the input data of the samples in the training set is each modal data in the multi-modal data obtained from the inertial sensor-based data corresponding to the corresponding time window, and the label of the sample is the standard corresponding to the corresponding time window. Respiratory waveform.

Step a2: Use the training set to train a breathing monitoring model to perform convolution filtering on the input data to generate a breathing waveform.

Step a3: Calculate the total loss value based on the difference between the generated respiratory waveform and the standard respiratory waveform.

In some embodiments of the present invention, the total loss value is calculated based on the Huber loss and the L2 regularization term calculated based on the generated respiration waveform and the standard respiration waveform. The Huber loss and the cosine distance are combined, and the Huber loss helps To fit outliers and prevent exploding gradients, the cosine distance helps fit the overall waveform features. At the same time, in order to avoid overfitting of the model, the L2 regularization term can be used. It is calculated as follows:

x_i是基于第i个样本生成的呼吸波形，y_i是第i个样本的标签，||x_i||₂表示x_i的2范数，||y_i||₂表示y_i的2范数，max(||x_i||₂·||y_i||₂，∈)表示取||x_i||₂·||y_i||₂和∈中的最大值作为分母，∈为标量值且等于1e-8，以防止分母为0的情况，α是L2正则化项的参数，Ω是L2正则化项。Among them, n is the total number of samples, _zi is the Huber loss of the ith sample, equal to

_xi is the respiratory waveform generated based on the ith sample, _yi is the label of the ith sample, || _xi || ₂ represents the 2 norm of _xi , ||y _i || ₂ represents the 2 of _yi Norm, max(||x _i || ₂ ·||y _i || ₂ , ∈) means taking the maximum value of ||x _i || ₂ ·||y _i || ₂ and ∈ as the denominator, ∈ is a scalar value and is equal to 1e-8 to prevent the case where the denominator is 0, α is the parameter of the L2 regularization term, and Ω is the L2 regularization term.

Step a4: Update respiratory monitoring model parameters based on the total loss value to obtain a trained respiratory monitoring model. Until the preset number of iterations is reached or the total loss value is within the preset range, the parameters are stopped to be updated, and the trained respiratory monitoring model is obtained.

Through the above training of the breathing monitoring model, the trained breathing monitoring model finally obtained can be used for breathing monitoring. According to an embodiment of the present invention, a system for respiratory monitoring based on data collected by an inertial sensor is provided, including a data processing module, a trained respiratory monitoring model, and a mean smoothing module. The following describes each part of the system with reference to specific embodiments. Detailed description.

According to an embodiment of the present invention, the data processing module is configured to obtain acceleration modal data, angular velocity modal data and Euler angle modal data based on data collected by an inertial sensor in the wearable device.

According to an embodiment of the present invention, the wearable device is worn on the human body, the accelerometer data and gyroscope data of the inertial sensor in the wearable device are obtained, and the accelerometer data is subjected to low-pass filtering and attitude calculation to obtain the acceleration mode Data, perform low-pass filtering and attitude calculation on the gyroscope data to obtain angular velocity modal data, and process the acceleration modal data and angular velocity modal data through the attitude calculation complementary filtering algorithm to obtain Euler angle modal data.

According to an embodiment of the present invention, the accelerometer data includes three-channel accelerometer data of X-axis, Y-axis, and Z-axis, and the gyroscope data includes three-channel gyroscope data of X-axis, Y-axis, and Z-axis.

Use a low-pass filter with 3-5 times the maximum breathing frequency as the cut-off frequency, perform low-pass filtering on the accelerometer data on the three channels to obtain low-frequency accelerometer data, and then perform attitude calculation to obtain the X-axis, The acceleration directions of the three channels of the Y axis and the Z axis are used as the acceleration modal data.

Use a low-pass filter with 3-5 times the maximum breathing frequency as the cut-off frequency, and perform low-pass filtering on the gyroscope data on the three channels to obtain the low-frequency gyroscope data, and perform the attitude calculation to obtain the X-axis, Y-axis, Z-axis three-channel angular velocity, the three-channel angular velocity as the angular velocity modal data.

Based on the data acceleration direction and angular velocity on each channel, the complementary filtering algorithm for attitude calculation is used, as shown in Equation (1):

angle _i =K ₁ *(angle _i-1 +gyro _i *dt)+K ₂ *accel _i (1);

Among them, K ₁ is the weight coefficient of the corresponding item (angle _i-1 +gyro _i *dt), K ₂ is the weight coefficient of the corresponding item accel _i , and (K ₁ +K ₂ )=1, angle _i is the i-th moment The Euler angle of , accel _i is the acceleration direction at the i-th moment, and gyro _i is the angular velocity at the i-th moment.

The formula (1) continuously uses the acceleration direction to calibrate the integral of the angular velocity to avoid the accumulation of the sensor error of the gyroscope in the integral, and finally obtains the filtered Euler angle feature as the Euler angle modal data, including the horizontal Roll angle (Roll), pitch angle (Pitch), yaw angle (Yaw). The acceleration modal data, the angular velocity modal data, and the Euler angle modal data are respectively input into the corresponding processing branch 2 of the respiration monitoring model for convolution filtering.

The breathing monitoring model trained based on the above training method is used to process the input Euler angle modal data, acceleration modal data and angular velocity modal data to generate a breathing waveform.

After the respiratory monitoring model performs convolution filtering on the above modal data, the generated respiratory waveform contains noise data, and the respiratory waveform is oriented to flow data and has continuity. Therefore, the respiratory waveform generated by the respiratory monitoring model needs to be decomposed. Noise processing to obtain optimized breathing waveform. According to an embodiment of the present invention, the mean value smoothing module is configured to, at each moment, take the mean value of all candidate points at the current moment as the final amplitude of the respiratory waveform at that moment, so as to cover the amplitude, curvature, The frequency-changing respiratory waveform is de-noised to obtain an optimized respiratory waveform.

Based on the above-mentioned system for respiratory monitoring, according to an embodiment of the present invention, a method for respiratory monitoring is provided, including steps b1, b2 and b3:

Step b1: Acceleration modal data, angular velocity modal data and Euler angle modal data are obtained based on the data collected by the inertial sensor in the wearable device.

Step b2: Using the above method for training the breathing monitoring model to obtain a trained breathing monitoring model, and processing the input Euler angle modal data, acceleration modal data and angular velocity modal data to generate a breathing waveform.

Step b3: Use the mean smoothing module to optimize the respiration waveform to obtain a smoother respiration waveform with a sliding receptive field.

Fig. 9 shows the comparison result of the respiratory waveform output by the respiratory monitoring model of the present invention and the real respiratory waveform of the professional respiratory monitor, the ordinate represents the amplitude of the respiratory waveform, the abscissa represents the corresponding time, the respiratory waveform output by the respiratory monitoring model and the real respiratory waveform At the same time, the respiratory waveform amplitudes of the two are the same or similar, and the overall respiratory waveform curvature is mostly similar.

It should be noted that although the steps are described above in a specific order, it does not mean that the steps must be executed in the above-mentioned specific order. In fact, some of these steps can be executed concurrently, or even change the order, as long as it can be achieved The required function can be.

The present invention may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the present invention.

A computer-readable storage medium may be a tangible device that retains and stores instructions for use by the instruction execution device. Computer-readable storage media may include, but are not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing, for example. More specific examples (non-exhaustive list) of computer readable storage media include: portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM) or flash memory), static random access memory (SRAM), portable compact disk read only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically coded devices, such as printers with instructions stored thereon Hole cards or raised structures in grooves, and any suitable combination of the above.

Various embodiments of the present invention have been described above, and the foregoing descriptions are exemplary, not exhaustive, and not limiting of the disclosed embodiments. Numerous modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (11)

1. A device based on a respiration monitoring model is characterized in that the respiration monitoring model comprises a processing main path and a plurality of processing branch paths,

each processing branch and the processing main road comprise convolution filters, each convolution filter comprises a plurality of convolution networks for filtering input data and a multi-head self-attention mechanism layer arranged between the corresponding convolution networks for enhancing the global receptive field,

wherein the respiratory monitoring model is configured to:

respectively inputting each modal data in the multi-modal data obtained based on the data of the inertial sensor into a corresponding processing branch circuit for convolution filtering to obtain a filtering result of each modal data; and

and inputting multi-modal breathing characteristics obtained by superposing filtering results of the modal data into a processing main path for convolution filtering to obtain correlation breathing characteristics, and generating a breathing waveform based on the correlation breathing characteristics.

2. The apparatus of claim 1, wherein the convolution filter further comprises a downsampling layer and an upsampling layer disposed between respective convolution networks;

the down-sampling layer is used for down-sampling a result obtained by filtering input data by a corresponding convolution network to obtain a low-dimensional feature;

the up-sampling layer is used for up-sampling a result obtained by filtering the corresponding input low-dimensional features by the corresponding convolution network to obtain high-dimensional features;

wherein the low-dimensional features and the high-dimensional features respectively enhance the respective global receptive fields by utilizing a multi-head self-attention mechanism layer.

3. The apparatus of claim 1 or 2, wherein the convolution filter comprises a first convolution network, a second convolution network, a third convolution network and a fourth convolution network, all of which are one-dimensional convolution networks, wherein an input end of the first convolution network is an input end of the convolution filter, and an output end of the fourth convolution network is an output end of the convolution filter;

a multi-head self-attention mechanism layer is arranged between the first convolution network and the fourth convolution network, a down-sampling layer is arranged between the first convolution network and the second convolution network, and a down-sampling layer is arranged between the first convolution network and the third convolution network;

an upper sampling layer is arranged between the second convolution network and the fourth convolution network, and a multi-head self-attention mechanism layer is arranged between the second convolution network and the third convolution network;

an upper sampling layer is arranged between the third convolution network and the fourth convolution network.

4. The apparatus of claim 3, wherein the convolution filter is configured to:

processing input data sequentially through a first convolution network and a multi-head self-attention mechanism layer to obtain a first high-dimensional feature;

processing input data sequentially through a first convolution network, a down-sampling layer, a second convolution network and an up-sampling layer to obtain a second high-dimensional feature;

processing input data through a first convolution network and a down-sampling layer in sequence to obtain a first low-dimensional feature;

processing input data sequentially through a first convolution network, a down-sampling layer, a second convolution network and a multi-head self-attention mechanism layer to obtain a second low-dimensional feature;

overlapping the first low-dimensional feature and the second low-dimensional feature, and sequentially processing the first low-dimensional feature and the second low-dimensional feature through a third convolution network and an upper sampling layer to obtain a third high-dimensional feature;

and superposing the first high-dimensional feature, the second high-dimensional feature and the third high-dimensional feature, and processing the superposed high-dimensional features through a fourth convolution network to obtain a filtering result.

5. The apparatus of claim 3, wherein the processing branch or processing head comprises one or more convolution filters;

when the processing branch or the processing main comprises a plurality of convolution filters, any two convolution filters in the plurality of convolution filters are connected in a vertical stacking connection mode or a horizontal stacking connection mode,

the first convolution filter and the second convolution filter which are longitudinally stacked and connected are connected in a mode that the output end of a second convolution network of the first convolution filter is connected with the input end of the second convolution filter, and the output end of the second convolution filter is connected with the input end of a third convolution network of the first convolution filter;

the first convolution filter and the second convolution filter which are connected in a transverse stacking mode are connected in a mode that the output end of the first convolution filter is connected with the input end of the second convolution filter.

6. A training method for the respiration monitoring model according to any one of claims 1 to 5, comprising performing a plurality of iterative training of the respiration monitoring model in the following manner:

acquiring a training set, wherein input data of a sample in the training set is each modal data in multi-modal data which is obtained based on data of an inertial sensor and corresponds to a corresponding time window, and a label of the sample is a standard respiration waveform corresponding to the corresponding time window;

training a respiration monitoring model by using a training set, and performing convolution filtering on input data to generate a respiration waveform;

calculating a total loss value based on a difference of the generated respiration waveform and a standard respiration waveform;

and updating parameters of the respiration monitoring model based on the total loss value to obtain a trained respiration monitoring model.

7. The method of claim 6, wherein the total loss value is calculated for a huber loss calculated based on the generated respiratory waveform and a standard respiratory waveform and an L2 regularization term by:

wherein n is the total amount of samples, z _i Huber loss for the ith sample, equal to

x _i Is a respiratory waveform, y, generated based on the ith sample _i Is the label of the ith sample, | | x _i || ₂ Denotes x _i 2 norm, | | y _i || ₂ Denotes y _i 2 norm of, max (| | x) _i || ₂ ·||y _i || ₂ And e) represents taking x _i || ₂ ·||y _i || ₂ And the maximum value in e is used as a denominator, e is a scalar value, a is a parameter of the L2 regularization term, and Ω is the L2 regularization term.

8. A system for respiratory monitoring based on data collected by inertial sensors, comprising:

the data processing module is used for obtaining acceleration modal data, angular velocity modal data and Euler angle modal data based on data collected by an inertial sensor in the wearable device;

processing the acceleration modal data, the angular velocity modal data, and the euler angular modal data using a respiration monitoring model trained using the method of claim 6 or 7 to generate a respiration waveform.

9. A method for respiratory monitoring based on data collected by an inertial sensor, comprising:

obtaining acceleration modal data, angular velocity modal data and Euler angle modal data based on data acquired by an inertial sensor in the wearable device;

processing the acceleration mode data, the angular velocity mode data and the Euler angle mode data by using a respiration monitoring model trained by the method of claim 6 or 7 to generate a respiration waveform.

10. A computer-readable storage medium, having stored thereon a computer program executable by a processor for carrying out the steps of the method according to any one of claims 6-7 and 9.

11. An electronic device, comprising:

one or more processors; and

a memory, wherein the memory is to store executable instructions;

the one or more processors are configured to implement the steps of the method of any of claims 6-7 and 9 via execution of the executable instructions.

Images