CN114764575B - Multi-modal data classification method based on deep learning and time sequence attention mechanism - Google Patents
Multi-modal data classification method based on deep learning and time sequence attention mechanism Download PDFInfo
- Publication number
- CN114764575B CN114764575B CN202210376944.3A CN202210376944A CN114764575B CN 114764575 B CN114764575 B CN 114764575B CN 202210376944 A CN202210376944 A CN 202210376944A CN 114764575 B CN114764575 B CN 114764575B
- Authority
- CN
- China
- Prior art keywords
- signal
- layer
- convolution
- characteristic signal
- inputting
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000007246 mechanism Effects 0.000 title claims abstract description 31
- 238000000034 method Methods 0.000 title claims abstract description 26
- 238000013135 deep learning Methods 0.000 title claims abstract description 13
- 230000005236 sound signal Effects 0.000 claims abstract description 64
- 238000010606 normalization Methods 0.000 claims description 69
- 238000012549 training Methods 0.000 claims description 64
- 230000004913 activation Effects 0.000 claims description 45
- 238000012360 testing method Methods 0.000 claims description 27
- 238000011176 pooling Methods 0.000 claims description 21
- 230000002457 bidirectional effect Effects 0.000 claims description 19
- 230000006870 function Effects 0.000 claims description 17
- 230000004927 fusion Effects 0.000 claims description 15
- 238000012545 processing Methods 0.000 claims description 7
- 238000012952 Resampling Methods 0.000 claims description 6
- 230000002238 attenuated effect Effects 0.000 claims description 6
- 238000002790 cross-validation Methods 0.000 claims description 6
- 239000011159 matrix material Substances 0.000 claims description 6
- 238000005070 sampling Methods 0.000 claims description 6
- 238000002759 z-score normalization Methods 0.000 claims description 6
- GNFTZDOKVXKIBK-UHFFFAOYSA-N 3-(2-methoxyethoxy)benzohydrazide Chemical compound COCCOC1=CC=CC(C(=O)NN)=C1 GNFTZDOKVXKIBK-UHFFFAOYSA-N 0.000 claims description 3
- HCHKCACWOHOZIP-UHFFFAOYSA-N Zinc Chemical compound [Zn] HCHKCACWOHOZIP-UHFFFAOYSA-N 0.000 claims description 3
- YTAHJIFKAKIKAV-XNMGPUDCSA-N [(1R)-3-morpholin-4-yl-1-phenylpropyl] N-[(3S)-2-oxo-5-phenyl-1,3-dihydro-1,4-benzodiazepin-3-yl]carbamate Chemical compound O=C1[C@H](N=C(C2=C(N1)C=CC=C2)C1=CC=CC=C1)NC(O[C@H](CCN1CCOCC1)C1=CC=CC=C1)=O YTAHJIFKAKIKAV-XNMGPUDCSA-N 0.000 claims description 3
- 230000011218 segmentation Effects 0.000 claims description 3
- 230000002123 temporal effect Effects 0.000 claims 1
- 238000001514 detection method Methods 0.000 abstract description 4
- 238000007635 classification algorithm Methods 0.000 abstract description 3
- 238000000605 extraction Methods 0.000 abstract 1
- 230000010354 integration Effects 0.000 abstract 1
- 238000011160 research Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/08—Feature extraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
- G06F2218/12—Classification; Matching
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Biomedical Technology (AREA)
- Data Mining & Analysis (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Signal Processing (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
Abstract
A multi-mode data classification method based on deep learning and time sequence attention mechanisms comprises the steps of firstly utilizing PC-TBG-ECG and PC-TBG-PCG models to respectively realize feature extraction of electrocardiosignals and heart sound signals, and then adopting an XGboost integration classification algorithm to perform feature selection and classification on the extracted features. While the operation efficiency is increased, the regularization is added, and overfitting is effectively prevented. The method is suitable for classification detection of data in different modes, and can analyze signals from various angles, so that the classification accuracy is improved.
Description
Technical Field
The invention relates to the field of multi-modal data classification, in particular to a multi-modal data classification method based on deep learning and a time sequence attention mechanism.
Background
Electrocardiogram (ECG) and Phonocardiogram (PCG) are non-invasive and cost-effective signal acquisition tools, and potential features of two signals can be mined and analyzed from various angles according to complementarity between the two, so that the classification effect is improved. In the conventional research, related researchers mainly use single-mode data or a single classifier to classify signals, but the classification research using the method cannot classify the signals from the comprehensive point of view, so the classification method fusing multi-mode data proposed by the research is extremely suitable for the practical requirement.
Disclosure of Invention
In order to overcome the defects of the technology, the invention provides a method which is suitable for classification detection of data in different modes, can analyze signals from various angles and further improves the accuracy of classification.
The technical scheme adopted by the invention for overcoming the technical problems is as follows:
a multi-modal data classification method based on deep learning and time sequence attention mechanism comprises the following steps:
a) Selecting training-a in PhysioNet/CinC Challenge 2016 as a data set, expanding the data set, and dividing the expanded data set into a training set and a test set;
b) Establishing an electrocardiosignal model, wherein the electrocardiosignal model consists of a PC module, a TBG module and a classification module in sequence;
c) Resampling the electrocardiosignals in the training set and the testing set to 2048 sampling points, and then carrying out z-score normalization processing to obtain normalized electrocardiosignals x' ecg ；
d) Normalizing the electrocardiosignals x 'in the training set' ecg Inputting to PC module of electrocardiosignal model, outputting to obtain characteristic signal X 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
e) The characteristic signal X 1 Inputting the signal into a TBG module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 2 The TBG module consists of 3 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
f) The characteristic signal X 2 Inputting the prediction data into a classification module of the electrocardiosignal model, and outputting the prediction data to obtain a prediction category f ecg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
g) Repeating the steps d) to f) N times, and obtaining an optimal electrocardiosignal model after training by using an SGD optimizer through a minimized cross entropy loss function;
h) Establishing a heart sound signal model which sequentially consists of a PC module, a TBG module and a classification module;
i) Resampling the heart sound signals in the training set and the test set to 8000 sampling points, and then carrying out z-score normalization processing to obtain normalized heart sound signals x' pcg ；
j) The heart sound signals x 'after normalization in the training set' pcg Inputting the signal into PC module of heart sound signal model, outputting to obtain characteristic signal Y 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
k) The characteristic signal Y 1 Inputting the signal into a TBG module of the heart sound signal model, and outputting to obtain a characteristic signal Y 2 The TBG module consists of 4 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
l) applying the characteristic signal Y 2 Inputting the prediction into a classification module of a heart sound signal model, and outputting the prediction category f pcg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
m) repeating the steps j) to l) M times, and obtaining an optimal heart sound signal model after training by minimizing a cross entropy loss function by using an SGD optimizer;
n) manually dividing the data set into a new training set and a new testing set again according to the proportion of 4:1, inputting the new training set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 3 Inputting the new training set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 3 By the formula PP x ＝[X 3 ,Y 3 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP x ；
o) fusing featuresComposite signal PP x Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP x The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 x Selecting an optimal hyper-parameter by adopting 5-fold cross validation, and training the XGboost classifier by utilizing the optimal hyper-parameter to obtain an optimized XGboost classifier;
p) inputting the new test set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 4 Inputting the new test set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 4 By the formula PP c ＝[X 4 ,Y 4 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP c ；
q) feature fusion signal PP c Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP c The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 c 。
Preferably, the data set is expanded in step a) by using a sliding window segmentation method, and the data set is divided into 5 different training sets and test sets by using a five-fold cross validation method.
Further, in step c), the formula is used
Calculating to obtain a normalized electrocardiosignal x' ecg In the formula x ecg For training and testing the concentrated ECG signal, u ecg Is the mean value, σ, of the electrocardiosignal ecg Is the variance of the electrocardiosignals.
Further, step d) comprises the following steps:
d-1) the first convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg The input is to the first of the convolution branches,the output obtains a 32-dimensional characteristic signal E 1 ；
d-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 13 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Inputting the signal into the second convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 2 ；
d-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 9 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 3 ；
d-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 5 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 4 ；
d-5) converting the characteristic signal E 1 Characteristic signal E 2 Characteristic signal E 3 Characteristic signal E 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal E = [ E ] after cascade 1 ,E 2 ,E 3 ,E 4 ]；
d-6) 1 × 1 convolution block is composed of convolution layers with 16 channels, 1 × 1 convolution kernel size and 1 step size and ReLU active layer, and a 128-dimensional characteristic signal E = [ E ] 1 ,E 2 ,E 3 ,E 4 ]Inputting the signal into a 1 × 1 convolution block, and outputting to obtain a 16-dimensional characteristic signal X 1 。
Further, step e) comprises the steps of:
e-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels of 32 and the size of a convolutional kernel of 1 multiplied by 11, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 4, and a characteristic signal X is formed 1 Inputting the signal into a first convolution coding module, and outputting to obtain a 32-dimensional characteristic signal E 5 ；
e-2) second convolution encoding modeThe block comprises convolutional layer with channel number of 64 and convolution kernel size of 1 × 7, batch normalization layer, reLU activation layer, and pooling layer with size of 2, and the feature signal E is obtained 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal E 6 ；
E-3) the third convolutional coding module consists of a convolutional layer with the channel number of 128 and the convolutional kernel size of 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2 in sequence, and a characteristic signal E is generated 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal E 7 ；
E-4) converting the characteristic signal E 7 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal X 2 In the bidirectional GRU layer of TPA mechanism by formula
Calculating to obtain a characteristic signal X 2 Where i = {1,2.., n }, n =128, t is the transpose, τ i For the attention weight of the ith row vector,
sigma (-) is a sigmoid function,
is a time pattern matrix G C Row i of (1), G C Conv1d (·) is a one-dimensional convolution operation, G is a hidden state matrix,
g i i = {1,2., t-1}, t is the time, w is the hidden state vector of the ith bidirectional GRU k Is a weight coefficient, g t Is the hidden state vector of the bi-directional GRU at time t.
Further, in the step g), the value of N is 150, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 80 periods, and the formula is used
Calculating to obtain a cross entropy loss function cc (x), wherein L is the number of categories, and L =2,f i (x) As a prediction class f ecg The predictive label of the ith category of (c),
as a prediction class f ecg The real category of the corresponding ith category; in the step m), the value of N is 180, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 90 periods, and the formula is used
Calculating to obtain a cross entropy loss function cc (y), wherein L is the number of categories, and L =2,f i (y) is prediction class f pcg The predictive label of the ith category of (c),
as a prediction class f pcg True category of the ith category of (1).
Further, in step i), the formula is used
Calculating to obtain a normalized heart sound signal x' pcg In the formula x pcg For the heart sound signals of the training set and the test set, u pcg Is the mean value, σ, of the heart sound signal pcg Is the variance of the heart sound signal.
Further, step j) comprises the following steps:
j-1) the first convolution branch is composed of convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 2, batch normalization layer and ReLU activation layer in sequence, and the heart sound signal x 'after normalization in training set' pcg Inputting the signal into the first convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 1 ；
j-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 11 and step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the second convolution branch, and outputting to obtain 32-dimensional characteristic signal P 2 ；
j-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 9 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 3 ；
j-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 5 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 4 ；
j-5) converting the characteristic signal P 1 Characteristic signal P 2 Characteristic signal P 3 Characteristic signal P 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal P = [ P ] after cascade 1 ,P 2 ,P 3 ,P 4 ]；
j-6) 1 × 1 convolution block is composed of convolution layer with 32 channel number, convolution kernel size of 1 × 1 and step size of 1 and ReLU active layer, and 128-dimensional characteristic signal P = [ P = 1 ,P 2 ,P 3 ,P 4 ]Inputting into a 1 × 1 convolution block, and outputting to obtain a 32-dimensional characteristic signal Y 1 。
Further, step k) comprises the steps of:
k-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels being 16 and the convolutional kernel size being 1 multiplied by 1, a batch normalization layer, a ReLU activation layer and a pooling layer with the size being 4, and the feature signal Y is processed 1 Inputting the data into a first convolutional coding module, and outputting to obtain a 16-dimensional characteristic signal P 5 ；
k-2) the second convolutional coding module sequentially comprises a convolutional layer with the channel number of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 2, and a characteristic signal P is obtained 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 32-dimensional characteristic signal P 6 ；
k-3) the third convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and a characteristic signal P is obtained 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal P 7 ；
k-4) the fourth convolutional coding module sequentially comprises a convolutional layer with the channel number of 128 and the convolutional kernel size of 1 multiplied by 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a feature signal 7 Inputting the signal into a fourth convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal P 8 ；
k-5) converting the characteristic signal P 8 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal Y 2 In the bidirectional GRU layer of TPA mechanism by formula
Calculating to obtain a characteristic signal Y 2 。
The invention has the beneficial effects that: firstly, the characteristics of electrocardiosignals and heart sound signals are respectively extracted by utilizing a PC-TBG-ECG model and a PC-TBG-PCG model, and then the extracted characteristics are selected and classified by adopting an XGboost integrated classification algorithm. While the operation efficiency is increased, the regularization is added, and overfitting is effectively prevented. The method is suitable for classification detection of data in different modes, and can analyze signals from various angles, so that the classification accuracy is improved.
Drawings
FIG. 1 is a flow chart of a method of the present invention;
fig. 2 is a network configuration diagram of the PC module of the present invention.
Detailed Description
The invention will be further explained with reference to fig. 1 and 2.
A multi-modal data classification method based on deep learning and time sequence attention mechanism comprises the following steps:
a) Selecting training-a in PhysioNet/CinC Change 2016 as a data set, expanding the data set, and dividing the expanded data set into a training set and a test set.
b) And establishing an electrocardiosignal model (PC-TBG-ECG), wherein the electrocardiosignal model is sequentially composed of a PC module, a TBG module and a classification module.
c) Resampling the electrocardiosignals in the training set and the testing set to 2048 sampling points, and then carrying out z-score normalization processing to obtain a normalized electrocardiosignal x' ecg 。
d) Normalizing the electrocardiosignals x 'in the training set' ecg Inputting the signal into a PC module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 1 The PC module, in turn, is made up of four convolution branches and a 1 x 1 convolution block.
e) The characteristic signal X 1 Inputting the signal into TBG module of electrocardio signal model, outputting to obtain characteristic signal X 2 The TBG module consists of 3 convolutional coding modules and a Bi-directional GRU layer (TPA-Bi-GRU) with TPA mechanism. f) The characteristic signal X 2 Inputting the prediction data into a classification module of the electrocardiosignal model, and outputting the prediction data to obtain a prediction category f ecg The classification module is composed of a full connection layer and a Softmax activation layer in sequence.
g) And (f) repeating the steps d) to f) N times, and obtaining the trained optimal electrocardiosignal model by using an SGD optimizer and minimizing a cross entropy loss function.
h) And establishing a heart sound signal model (PC-TBG-PCG), wherein the heart sound signal model is composed of a PC module, a TBG module and a classification module in sequence.
i) Resampling the heart sound signals in the training set and the test set to 8000 sampling points, and then carrying out z-score normalization processing to obtain normalized heart sound signals x' pcg 。
j) The heart sound signals x 'after normalization in the training set' pcg Inputting the signal into PC module of heart sound signal model, outputting to obtain characteristic signal Y 1 The PC module, in turn, is made up of four convolution branches and a 1 x 1 convolution block.
k) The characteristic signal Y 1 Inputting the signal into a TBG module of the heart sound signal model, and outputting to obtain a characteristic signal Y 2 The TBG module consists of 4 convolutional coding modules and oneA Bi-directional GRU layer (TPA-Bi-GRU) with TPA mechanism. l) applying the characteristic signal Y 2 Inputting the prediction into a classification module of a heart sound signal model, and outputting the prediction category f pcg The classification module is composed of a full connection layer and a Softmax activation layer in sequence.
M) repeating the steps j) to l) M times, and obtaining the trained optimal heart sound signal model by minimizing a cross entropy loss function by using an SGD optimizer.
n) manually dividing the data set into a new training set and a new testing set again according to the proportion of 4:1, inputting the new training set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 3 Inputting the new training set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG (tunnel boring generator) module of the optimal heart sound signal model 3 By the formula PP x ＝[X 3 ,Y 3 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP x 。
o) fusing features into a signal PP x Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP x The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 x And selecting an optimal hyper-parameter by adopting 5-fold cross validation, and training the XGboost classifier by utilizing the optimal hyper-parameter to obtain the optimized XGboost classifier.
p) inputting the new test set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 4 Inputting the new test set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG (tunnel boring generator) module of the optimal heart sound signal model 4 By the formula PP c ＝[X 4 ,Y 4 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP c 。
q) feature fusion signal PP c Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP c The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 c 。
The signals do not need to be subjected to noise reduction, filtering and other processing, the problems of low classification accuracy rate or poor practicability and the like caused by unreasonable signal preprocessing in the past are avoided, and the robustness of the model is ensured. Firstly, the characteristics of electrocardiosignals and heart sound signals are respectively extracted by utilizing PC-TBG-ECG and PC-TBG-PCG models, and then the XGboost integrated classification algorithm is adopted to select and classify the extracted characteristics. While the operation efficiency is increased, the regularization is added, and overfitting is effectively prevented. The method is suitable for classification detection of data in different modes, and can analyze signals from various angles, so that the classification accuracy is improved.
Example 1:
in the step a), a sliding window segmentation method is used for expanding the data set, and a five-fold cross validation method is used for dividing the data set into 5 different training sets and test sets.
Example 2:
in step c) by the formula
Calculating to obtain a normalized electrocardiosignal x' ecg In the formula x ecg For training and testing the concentrated ECG signal, u ecg Is the mean value, σ, of the electrocardiosignal ecg Is the variance of the electrocardiosignals.
Example 3:
the step d) comprises the following steps:
d-1) the first convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Inputting the signal into the first convolution branch, and outputting to obtain 32-dimensional characteristic signal E 1 ；
d-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 13 and step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Input into the second convolution branchAnd outputting the obtained 32-dimensional characteristic signal E 2 ；
d-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 9 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 3 ；
d-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 5 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 4 ；
d-5) converting the characteristic signal E 1 Characteristic signal E 2 Characteristic signal E 3 Characteristic signal E 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal E = [ E ] after cascade 1 ,E 2 ,E 3 ,E 4 ]；
d-6) 1 × 1 convolution block is composed of convolution layer with 16 channel number, convolution kernel size 1 × 1 and step size 1 and ReLU active layer, and 128-dimensional feature signal E = [ 1 ,E 2 ,E 3 ,E 4 ]Inputting the signal into a 1 × 1 convolution block, and outputting to obtain a 16-dimensional characteristic signal X 1 。
Example 4:
step e) comprises the following steps:
e-1) the first convolutional coding module consists of a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 4 in sequence, and the characteristic signal X is converted into a characteristic signal 1 Inputting the signal into a first convolution coding module, and outputting to obtain a 32-dimensional characteristic signal E 5 ；
E-2) the second convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and a characteristic signal E is obtained 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal E 6 ；
e-3) The third convolutional coding module consists of a convolutional layer with the channel number of 128 and the convolutional kernel size of 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2 in sequence, and a characteristic signal E is formed 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal E 7 ；
E-4) converting the characteristic signal E 7 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal X 2 In the bidirectional GRU layer of TPA mechanism by formula
Calculating to obtain a characteristic signal X 2 Where i = {1,2.., n }, n =128, t is the transpose, τ i For the attention weight of the ith row vector,
sigma (-) is a sigmoid function,
is a time pattern matrix G C Row i of (1), G C Conv1d (G), conv1d (·) is a one-dimensional convolution operation, G is a hidden state matrix,
g i i = {1,2.., t-1}, t is the time, w is the hidden state vector for the ith bidirectional GRU k Is a weight coefficient, g t Is the hidden state vector of the bi-directional GRU at time t.
Example 5:
in the step g), the value of N is 150, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 80 periods, and the formula is used
Calculating to obtain a cross entropy loss function cc (x), wherein L is the number of categories, and L =2,f i (x) To predict class f ecg The predictive label of the ith category of (c),
as a prediction class f ecg The real category of the corresponding ith category; in the step m), the value of N is 180, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 90 periods, and the formula is used
Calculating to obtain a cross entropy loss function cc (y), wherein L is the number of categories, and L =2,f i (y) is prediction class f pcg The predictive tag of the ith category of (a),
as a prediction class f pcg True category of the ith category of (1).
Example 6:
in step i) by the formula
Calculating to obtain a normalized heart sound signal x' pcg In the formula x pcg For the heart sound signals in the training set and test set, u pcg Is the mean value, σ, of the heart sound signal pcg Is the variance of the heart sound signal.
Example 7:
step j) comprises the following steps:
j-1) the first convolution branch is composed of convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 2, batch normalization layer and ReLU activation layer in sequence, and the heart sound signal x 'after normalization in training set' pcg Inputting the signal into the first convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 1 ；
j-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 11 and step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the second convolution branch, and outputting to obtain 32-dimensional characteristic signal P 2 ；
j-3) the third convolution branch consists of 32 channels,Convolution layer with convolution kernel size of 1 × 9 and step size of 2, batch normalization layer, and ReLU activation layer, and training set is normalized to obtain normalized heart sound signal x' pcg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 3 ；
j-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 5 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 4 ；
j-5) converting the characteristic signal P 1 Characteristic signal P 2 Characteristic signal P 3 Characteristic signal P 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal P = [ P ] after cascade 1 ,P 2 ,P 3 ,P 4 ]；
j-6) 1 × 1 convolution block is composed of convolution layer with 32 channel number, convolution kernel size of 1 × 1 and step size of 1 and ReLU active layer, and 128-dimensional characteristic signal P = [ P = 1 ,P 2 ,P 3 ,P 4 ]Inputting into a 1 × 1 convolution block, and outputting to obtain a 32-dimensional characteristic signal Y 1 。
Example 8:
step k) comprises the following steps:
k-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels being 16 and the convolutional kernel size being 1 multiplied by 1, a batch normalization layer, a ReLU activation layer and a pooling layer with the size being 4, and the feature signal Y is processed 1 Inputting the signals into a first convolutional encoding module, and outputting to obtain a 16-dimensional characteristic signal P 5 ；
k-2) the second convolutional coding module sequentially comprises a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a linear convolution function 5 Inputting the data into a second convolutional coding module, and outputting to obtain a 32-dimensional characteristic signal P 6 ；
k-3) the third convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU active layer and a convolutional layer with the size of 2Formation of pooling layer by combining characteristic signal P 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal P 7 ；
k-4) the fourth convolutional coding module sequentially comprises a convolutional layer with the channel number of 128 and the convolutional kernel size of 1 multiplied by 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a feature signal 7 Inputting the signal into a fourth convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal P 8 ；
k-5) converting the characteristic signal P 8 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal Y 2 In the bidirectional GRU layer of TPA mechanism by formula
Calculating to obtain a characteristic signal Y 2 。
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims (7)
1. A multi-modal data classification method based on deep learning and time-series attention mechanism is characterized by comprising the following steps:
a) Selecting training-a in PhysioNet/CinC Challenge 2016 as a data set, expanding the data set, and dividing the expanded data set into a training set and a test set;
b) Establishing an electrocardiosignal model, wherein the electrocardiosignal model consists of a PC module, a TBG module and a classification module in sequence;
c) Resampling the electrocardiosignals in the training set and the testing set to 2048 sampling points, and then carrying out z-score normalization processing to obtain a regressionNormalized electrocardiosignal x' ecg ；
d) Normalizing the electrocardiosignals x 'in the training set' ecg Inputting the signal into a PC module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
e) The characteristic signal X 1 Inputting the signal into a TBG module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 2 The TBG module consists of 3 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
f) The characteristic signal X 2 Inputting the prediction data into a classification module of the electrocardiosignal model, and outputting the prediction data to obtain a prediction category f ecg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
g) Repeating the steps d) to f) N times, and obtaining an optimal electrocardiosignal model after training by using an SGD optimizer through a minimized cross entropy loss function;
h) Establishing a heart sound signal model which sequentially consists of a PC module, a TBG module and a classification module;
i) Resampling the heart sound signals in the training set and the test set to 8000 sampling points, and then carrying out z-score normalization processing to obtain normalized heart sound signals x' pcg ；
j) Normalizing the heart sound signals x 'in the training set' pcg Inputting the signal into PC module of heart sound signal model, outputting to obtain characteristic signal Y 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
k) The characteristic signal Y 1 Inputting the signal into a TBG module of the heart sound signal model, and outputting to obtain a characteristic signal Y 2 The TBG module consists of 4 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
l) applying the characteristic signal Y 2 Inputting the prediction into a classification module of a heart sound signal model, and outputting the prediction category f pcg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
m) repeating the steps j) to l) M times, and obtaining an optimal heart sound signal model after training by minimizing a cross entropy loss function by using an SGD optimizer;
n) manually dividing the data set into a new training set and a new testing set according to the proportion of 4:1, inputting the new training set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 3 Inputting the new training set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 3 By the formula PP x ＝[X 3 ,Y 3 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP x ；
o) fusing features into a signal PP x Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP x The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 x Selecting an optimal hyper-parameter by adopting 5-fold cross validation, and training the XGboost classifier by utilizing the optimal hyper-parameter to obtain an optimized XGboost classifier;
p) inputting the new test set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 4 Inputting the new test set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 4 By the formula PP c ＝[X 4 ,Y 4 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP c ；
q) feature fusion signal PP c Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP c The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 c ；
The step d) comprises the following steps:
d-1) the first convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Input into the first convolution branch, and output is obtained32-dimensional characteristic signal E 1 ；
d-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 13 and step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Inputting the signal into the second convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 2 ；
d-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 9 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 3 ；
d-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 5 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 4 ；
d-5) converting the characteristic signal E 1 Characteristic signal E 2 Characteristic signal E 3 Characteristic signal E 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal E = [ E ] after cascade 1 ,E 2 ,E 3 ,E 4 ]；
d-6) 1 × 1 convolution block is composed of convolution layers with 16 channels, 1 × 1 convolution kernel size and 1 step size and ReLU active layer, and a 128-dimensional characteristic signal E = [ E ] 1 ,E 2 ,E 3 ,E 4 ]Inputting the signal into a 1 × 1 convolution block, and outputting to obtain a 16-dimensional characteristic signal X 1 ；
The step j) comprises the following steps:
j-1) the first convolution branch is composed of convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 2, batch normalization layer and ReLU activation layer in sequence, and the heart sound signal x 'after normalization in training set' pcg Inputting the signal into the first convolution branch, and outputting to obtain 32-dimensional characteristic signal P 1 ；
j-2) the second convolution branch is composed of 32 channels and convolution kernelA convolution layer of 1 × 11 size and step size 2, a batch normalization layer, and a ReLU activation layer, and the normalized heart sound signal x 'in the training set' pcg Inputting the signal into the second convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 2 ；
j-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 9 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 3 ；
j-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 5 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the fourth convolution branch, and outputting to obtain 32-dimensional characteristic signal P 4 ；
j-5) converting the characteristic signal P 1 Characteristic signal P 2 Characteristic signal P 3 Characteristic signal P 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal P = [ P ] after cascade 1 ,P 2 ,P 3 ,P 4 ]；
j-6) 1 × 1 convolution block is composed of convolution layer with 32 channel number, convolution kernel size of 1 × 1 and step size of 1 and ReLU active layer, and 128-dimensional characteristic signal P = [ P = 1 ,P 2 ,P 3 ,P 4 ]Inputting into a 1 × 1 convolution block, and outputting to obtain a 32-dimensional feature signal Y 1 。
2. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in the step a), a sliding window segmentation method is used for expanding the data set, and a five-fold cross validation method is used for dividing the data set into 5 different training sets and test sets.
3. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in step c) byIs of the formula
Calculating to obtain a normalized electrocardiosignal x' ecg In the formula x ecg For training and testing the concentrated ECG signal u ecg Is the mean value, σ, of the electrocardiosignal ecg Is the variance of the electrocardiosignals.
4. The method for multi-modal data classification based on deep learning and temporal attention mechanism according to claim 1, wherein step e) comprises the following steps:
e-1) the first convolutional coding module consists of a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 4 in sequence, and the characteristic signal X is converted into a characteristic signal 1 Inputting the signal into a first convolution coding module, and outputting to obtain a 32-dimensional characteristic signal E 5 ；
E-2) the second convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 2, and a characteristic signal E is generated 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal E 6 ；
E-3) the third convolution coding module consists of a convolution layer with the channel number of 128 and the convolution kernel size of 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2 in sequence, and a characteristic signal E is generated 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal E 7 ；
E-4) converting the characteristic signal E 7 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal X 2 In the bidirectional GRU layer of TPA mechanism by formula
Calculating to obtain a characteristic signal X 2 Where i = {1,2.., n }, n =128, t is the transpose, τ i For the attention weight of the ith row vector,
σ (-) is a sigmoid function,
is a time pattern matrix G C Row i of (1), G C Conv1d (G), conv1d (·) is a one-dimensional convolution operation, G is a hidden state matrix,
g i i = {1,2., t-1}, t is the time, w is the hidden state vector of the ith bidirectional GRU k Is a weight coefficient, g t Is the hidden state vector of the bi-directional GRU at time t.
5. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in the step g), the value of N is 150, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 80 periods, and the formula is used
Calculating to obtain a cross entropy loss function cc (x), wherein L is the number of categories, and L =2,f i (x) As a prediction class f ecg The predictive label of the ith category of (c),
as a prediction class f ecg The real category of the corresponding ith category; in the step m), the value of N is 180, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 90 periods, and the N is obtained by a formula
Calculating to obtain a cross entropy loss function cc (y), wherein L is the number of categories, and L =2,f i (y) as a prediction class f pcg The predictive tag of the ith category of (a),
as a prediction class f pcg True category of the ith category of (1).
6. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in step i) by the formula
Calculating to obtain a normalized heart sound signal x' pcg In the formula x pcg For the heart sound signals in the training set and test set, u pcg Is the mean value, σ, of the heart sound signal pcg Is the variance of the heart sound signal.
7. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein step k) comprises the following steps:
k-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels being 16 and the convolutional kernel size being 1 multiplied by 1, a batch normalization layer, a ReLU activation layer and a pooling layer with the size being 4, and the feature signal Y is processed 1 Inputting the signals into a first convolutional encoding module, and outputting to obtain a 16-dimensional characteristic signal P 5 ；
k-2) the second convolutional coding module sequentially comprises a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a linear convolution function 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 32-dimensional characteristic signal P 6 ；
k-3) the third convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 2, and a characteristic signal P is obtained 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal P 7 ；
k-4) the fourth convolutional encoding module sequentially comprises a channel number of 128 and a convolutional kernel size of 1 x 3A convolution layer, a batch normalization layer, a ReLU active layer, a pooling layer of size 2, and a feature signal P 7 Inputting the signal into a fourth convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal P 8 ；
k-5) converting the characteristic signal P 8 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal Y 2 In the bidirectional GRU layer of TPA mechanism by formula
Calculating to obtain a characteristic signal Y 2 。
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210376944.3A CN114764575B (en) | 2022-04-11 | 2022-04-11 | Multi-modal data classification method based on deep learning and time sequence attention mechanism |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210376944.3A CN114764575B (en) | 2022-04-11 | 2022-04-11 | Multi-modal data classification method based on deep learning and time sequence attention mechanism |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114764575A CN114764575A (en) | 2022-07-19 |
| CN114764575B true CN114764575B (en) | 2023-02-28 |
Family
ID=82364741
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210376944.3A Active CN114764575B (en) | 2022-04-11 | 2022-04-11 | Multi-modal data classification method based on deep learning and time sequence attention mechanism |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114764575B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116186593B (en) * | 2023-03-10 | 2023-10-03 | 山东省人工智能研究院 | Electrocardiosignal detection method based on separable convolution and attention mechanism |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018130541A (en) * | 2017-02-16 | 2018-08-23 | タタ コンサルタンシー サービシズ リミテッドＴＡＴＡ Ｃｏｎｓｕｌｔａｎｃｙ Ｓｅｒｖｉｃｅｓ Ｌｉｍｉｔｅｄ | Method and system for detection of coronary artery disease in person using fusion approach |
| CN110236518A (en) * | 2019-04-02 | 2019-09-17 | 武汉大学 | The method and device of electrocardio and the heart shake combined signal classification neural network based |
| CN110537910A (en) * | 2019-09-18 | 2019-12-06 | 山东大学 | coronary heart disease noninvasive screening system based on electrocardio and heart sound signal joint analysis |
| CN113288163A (en) * | 2021-06-04 | 2021-08-24 | 浙江理工大学 | Multi-feature fusion electrocardiosignal classification model modeling method based on attention mechanism |
| CN113855063A (en) * | 2021-10-21 | 2021-12-31 | 华中科技大学 | Heart sound automatic diagnosis system based on deep learning |
| CN114190952A (en) * | 2021-12-01 | 2022-03-18 | 山东省人工智能研究院 | 12-lead electrocardiosignal multi-label classification method based on lead grouping |
-
2022
- 2022-04-11 CN CN202210376944.3A patent/CN114764575B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2018130541A (en) * | 2017-02-16 | 2018-08-23 | タタ コンサルタンシー サービシズ リミテッドＴＡＴＡ Ｃｏｎｓｕｌｔａｎｃｙ Ｓｅｒｖｉｃｅｓ Ｌｉｍｉｔｅｄ | Method and system for detection of coronary artery disease in person using fusion approach |
| CN110236518A (en) * | 2019-04-02 | 2019-09-17 | 武汉大学 | The method and device of electrocardio and the heart shake combined signal classification neural network based |
| CN110537910A (en) * | 2019-09-18 | 2019-12-06 | 山东大学 | coronary heart disease noninvasive screening system based on electrocardio and heart sound signal joint analysis |
| CN113288163A (en) * | 2021-06-04 | 2021-08-24 | 浙江理工大学 | Multi-feature fusion electrocardiosignal classification model modeling method based on attention mechanism |
| CN113855063A (en) * | 2021-10-21 | 2021-12-31 | 华中科技大学 | Heart sound automatic diagnosis system based on deep learning |
| CN114190952A (en) * | 2021-12-01 | 2022-03-18 | 山东省人工智能研究院 | 12-lead electrocardiosignal multi-label classification method based on lead grouping |
Non-Patent Citations (2)
| Title |
|---|
| Integrating multi-domain deep features of electrocardiogram and phonocardiogram for coronary artery disease detection;HanLi, et al.;《Computers in Biology and Medicine》;20211130;1-7 * |
| 基于心音心电信号的心衰分析系统;李俊杰 等;《Software Engineering and Applications》;20220209;1-10 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114764575A (en) | 2022-07-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112766355B (en) | Electroencephalogram signal emotion recognition method under label noise | |
| CN111414942A (en) | Remote sensing image classification method based on active learning and convolutional neural network | |
| CN112294341B (en) | Sleep electroencephalogram spindle wave identification method and system based on light convolutional neural network | |
| CN114176607B (en) | Electroencephalogram signal classification method based on vision transducer | |
| CN114564990B (en) | Electroencephalogram signal classification method based on multichannel feedback capsule network | |
| CN105095863A (en) | Similarity-weight-semi-supervised-dictionary-learning-based human behavior identification method | |
| CN113343860A (en) | Bimodal fusion emotion recognition method based on video image and voice | |
| CN113554110B (en) | Brain electricity emotion recognition method based on binary capsule network | |
| CN111582041A (en) | Electroencephalogram identification method based on CWT and MLMSFFCNN | |
| CN113749657A (en) | Brain wave emotion recognition method based on multitask capsules | |
| CN112732921A (en) | False user comment detection method and system | |
| CN116186593B (en) | Electrocardiosignal detection method based on separable convolution and attention mechanism | |
| CN116258914B (en) | Remote Sensing Image Classification Method Based on Machine Learning and Local and Global Feature Fusion | |
| CN114764575B (en) | Multi-modal data classification method based on deep learning and time sequence attention mechanism | |
| CN114595725B (en) | Electroencephalogram signal classification method based on addition network and supervised contrast learning | |
| CN114420151B (en) | Speech emotion recognition method based on parallel tensor decomposition convolutional neural network | |
| CN115965864A (en) | Lightweight attention mechanism network for crop disease identification | |
| CN117942038A (en) | Sleep stage identification system integrating cross prediction and discrimination tasks | |
| CN113283519A (en) | Deep neural network approximate model analysis method based on discrete coefficients | |
| CN116610846A (en) | Breeding screening method and system for cattle breeding | |
| CN112465054B (en) | FCN-based multivariate time series data classification method | |
| CN115801152A (en) | WiFi action identification method based on hierarchical transform model | |
| CN112883905B (en) | Human behavior recognition method based on coarse-grained time-frequency features and multi-layer fusion learning | |
| CN112560784B (en) | Electrocardiogram classification method based on dynamic multi-scale convolutional neural network | |
| CN114818823A (en) | Electroencephalogram channel selection method based on squeezing and activation graph convolution neural network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CP03 | Change of name, title or address |
Address after: No.19 Keyuan Road, Lixia District, Jinan City, Shandong Province Patentee after: Shandong Institute of artificial intelligence Country or region after: China Patentee after: Qilu University of Technology (Shandong Academy of Sciences) Address before: No.19 Keyuan Road, Lixia District, Jinan City, Shandong Province Patentee before: Shandong Institute of artificial intelligence Country or region before: China Patentee before: Qilu University of Technology |