CN114764575B - Multi-modal data classification method based on deep learning and time sequence attention mechanism - Google Patents

Multi-modal data classification method based on deep learning and time sequence attention mechanism Download PDF

Info

Publication number
CN114764575B
CN114764575B CN202210376944.3A CN202210376944A CN114764575B CN 114764575 B CN114764575 B CN 114764575B CN 202210376944 A CN202210376944 A CN 202210376944A CN 114764575 B CN114764575 B CN 114764575B
Authority
CN
China
Prior art keywords
signal
layer
convolution
characteristic signal
inputting
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210376944.3A
Other languages
Chinese (zh)
Other versions
CN114764575A (en
Inventor
舒明雷
朱佳媛
刘辉
陈达
谢小云
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qilu University of Technology
Shandong Institute of Artificial Intelligence
Original Assignee
Qilu University of Technology
Shandong Institute of Artificial Intelligence
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qilu University of Technology, Shandong Institute of Artificial Intelligence filed Critical Qilu University of Technology
Priority to CN202210376944.3A priority Critical patent/CN114764575B/en
Publication of CN114764575A publication Critical patent/CN114764575A/en
Application granted granted Critical
Publication of CN114764575B publication Critical patent/CN114764575B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/08—Feature extraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2218/00—Aspects of pattern recognition specially adapted for signal processing
    • G06F2218/12—Classification; Matching

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Signal Processing (AREA)
  • Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)

Abstract

A multi-mode data classification method based on deep learning and time sequence attention mechanisms comprises the steps of firstly utilizing PC-TBG-ECG and PC-TBG-PCG models to respectively realize feature extraction of electrocardiosignals and heart sound signals, and then adopting an XGboost integration classification algorithm to perform feature selection and classification on the extracted features. While the operation efficiency is increased, the regularization is added, and overfitting is effectively prevented. The method is suitable for classification detection of data in different modes, and can analyze signals from various angles, so that the classification accuracy is improved.

Description

Multi-modal data classification method based on deep learning and time sequence attention mechanism
Technical Field
The invention relates to the field of multi-modal data classification, in particular to a multi-modal data classification method based on deep learning and a time sequence attention mechanism.
Background
Electrocardiogram (ECG) and Phonocardiogram (PCG) are non-invasive and cost-effective signal acquisition tools, and potential features of two signals can be mined and analyzed from various angles according to complementarity between the two, so that the classification effect is improved. In the conventional research, related researchers mainly use single-mode data or a single classifier to classify signals, but the classification research using the method cannot classify the signals from the comprehensive point of view, so the classification method fusing multi-mode data proposed by the research is extremely suitable for the practical requirement.
Disclosure of Invention
In order to overcome the defects of the technology, the invention provides a method which is suitable for classification detection of data in different modes, can analyze signals from various angles and further improves the accuracy of classification.
The technical scheme adopted by the invention for overcoming the technical problems is as follows:
a multi-modal data classification method based on deep learning and time sequence attention mechanism comprises the following steps:
a) Selecting training-a in PhysioNet/CinC Challenge 2016 as a data set, expanding the data set, and dividing the expanded data set into a training set and a test set;
b) Establishing an electrocardiosignal model, wherein the electrocardiosignal model consists of a PC module, a TBG module and a classification module in sequence;
c) Resampling the electrocardiosignals in the training set and the testing set to 2048 sampling points, and then carrying out z-score normalization processing to obtain normalized electrocardiosignals x' ecg ；
d) Normalizing the electrocardiosignals x 'in the training set' ecg Inputting to PC module of electrocardiosignal model, outputting to obtain characteristic signal X 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
e) The characteristic signal X 1 Inputting the signal into a TBG module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 2 The TBG module consists of 3 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
f) The characteristic signal X 2 Inputting the prediction data into a classification module of the electrocardiosignal model, and outputting the prediction data to obtain a prediction category f ecg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
g) Repeating the steps d) to f) N times, and obtaining an optimal electrocardiosignal model after training by using an SGD optimizer through a minimized cross entropy loss function;
h) Establishing a heart sound signal model which sequentially consists of a PC module, a TBG module and a classification module;
i) Resampling the heart sound signals in the training set and the test set to 8000 sampling points, and then carrying out z-score normalization processing to obtain normalized heart sound signals x' pcg ；
j) The heart sound signals x 'after normalization in the training set' pcg Inputting the signal into PC module of heart sound signal model, outputting to obtain characteristic signal Y 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
k) The characteristic signal Y 1 Inputting the signal into a TBG module of the heart sound signal model, and outputting to obtain a characteristic signal Y 2 The TBG module consists of 4 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
l) applying the characteristic signal Y 2 Inputting the prediction into a classification module of a heart sound signal model, and outputting the prediction category f pcg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
m) repeating the steps j) to l) M times, and obtaining an optimal heart sound signal model after training by minimizing a cross entropy loss function by using an SGD optimizer;
n) manually dividing the data set into a new training set and a new testing set again according to the proportion of 4:1, inputting the new training set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 3 Inputting the new training set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 3 By the formula PP x ＝[X 3 ,Y 3 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP x ；
o) fusing featuresComposite signal PP x Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP x The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 x Selecting an optimal hyper-parameter by adopting 5-fold cross validation, and training the XGboost classifier by utilizing the optimal hyper-parameter to obtain an optimized XGboost classifier;
p) inputting the new test set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 4 Inputting the new test set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 4 By the formula PP c ＝[X 4 ,Y 4 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP c ；
q) feature fusion signal PP c Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP c The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 c 。
Preferably, the data set is expanded in step a) by using a sliding window segmentation method, and the data set is divided into 5 different training sets and test sets by using a five-fold cross validation method.
Further, in step c), the formula is used
Figure BDA0003589207160000031
Calculating to obtain a normalized electrocardiosignal x' ecg In the formula x ecg For training and testing the concentrated ECG signal, u ecg Is the mean value, σ, of the electrocardiosignal ecg Is the variance of the electrocardiosignals.
Further, step d) comprises the following steps:
d-1) the first convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg The input is to the first of the convolution branches,the output obtains a 32-dimensional characteristic signal E 1 ；
d-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 13 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Inputting the signal into the second convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 2 ；
d-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 9 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 3 ；
d-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 5 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 4 ；
d-5) converting the characteristic signal E 1 Characteristic signal E 2 Characteristic signal E 3 Characteristic signal E 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal E = [ E ] after cascade 1 ,E 2 ,E 3 ,E 4 ]；
d-6) 1 × 1 convolution block is composed of convolution layers with 16 channels, 1 × 1 convolution kernel size and 1 step size and ReLU active layer, and a 128-dimensional characteristic signal E = [ E ] 1 ,E 2 ,E 3 ,E 4 ]Inputting the signal into a 1 × 1 convolution block, and outputting to obtain a 16-dimensional characteristic signal X 1 。
Further, step e) comprises the steps of:
e-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels of 32 and the size of a convolutional kernel of 1 multiplied by 11, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 4, and a characteristic signal X is formed 1 Inputting the signal into a first convolution coding module, and outputting to obtain a 32-dimensional characteristic signal E 5 ；
e-2) second convolution encoding modeThe block comprises convolutional layer with channel number of 64 and convolution kernel size of 1 × 7, batch normalization layer, reLU activation layer, and pooling layer with size of 2, and the feature signal E is obtained 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal E 6 ；
E-3) the third convolutional coding module consists of a convolutional layer with the channel number of 128 and the convolutional kernel size of 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2 in sequence, and a characteristic signal E is generated 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal E 7 ；
E-4) converting the characteristic signal E 7 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal X 2 In the bidirectional GRU layer of TPA mechanism by formula
Figure BDA0003589207160000041
Calculating to obtain a characteristic signal X 2 Where i = {1,2.., n }, n =128, t is the transpose, τ i For the attention weight of the ith row vector,
Figure BDA0003589207160000042
sigma (-) is a sigmoid function,
Figure BDA0003589207160000043
is a time pattern matrix G C Row i of (1), G C Conv1d (·) is a one-dimensional convolution operation, G is a hidden state matrix,
Figure BDA0003589207160000044
g i i = {1,2., t-1}, t is the time, w is the hidden state vector of the ith bidirectional GRU k Is a weight coefficient, g t Is the hidden state vector of the bi-directional GRU at time t.
Further, in the step g), the value of N is 150, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 80 periods, and the formula is used
Figure BDA0003589207160000051
Calculating to obtain a cross entropy loss function cc (x), wherein L is the number of categories, and L =2,f i (x) As a prediction class f ecg The predictive label of the ith category of (c),
Figure BDA0003589207160000052
as a prediction class f ecg The real category of the corresponding ith category; in the step m), the value of N is 180, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 90 periods, and the formula is used
Figure BDA0003589207160000053
Calculating to obtain a cross entropy loss function cc (y), wherein L is the number of categories, and L =2,f i (y) is prediction class f pcg The predictive label of the ith category of (c),
Figure BDA0003589207160000054
as a prediction class f pcg True category of the ith category of (1).
Further, in step i), the formula is used
Figure BDA0003589207160000055
Calculating to obtain a normalized heart sound signal x' pcg In the formula x pcg For the heart sound signals of the training set and the test set, u pcg Is the mean value, σ, of the heart sound signal pcg Is the variance of the heart sound signal.
Further, step j) comprises the following steps:
j-1) the first convolution branch is composed of convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 2, batch normalization layer and ReLU activation layer in sequence, and the heart sound signal x 'after normalization in training set' pcg Inputting the signal into the first convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 1 ；
j-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 11 and step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the second convolution branch, and outputting to obtain 32-dimensional characteristic signal P 2 ；
j-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 9 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 3 ；
j-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 5 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 4 ；
j-5) converting the characteristic signal P 1 Characteristic signal P 2 Characteristic signal P 3 Characteristic signal P 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal P = [ P ] after cascade 1 ,P 2 ,P 3 ,P 4 ]；
j-6) 1 × 1 convolution block is composed of convolution layer with 32 channel number, convolution kernel size of 1 × 1 and step size of 1 and ReLU active layer, and 128-dimensional characteristic signal P = [ P = 1 ,P 2 ,P 3 ,P 4 ]Inputting into a 1 × 1 convolution block, and outputting to obtain a 32-dimensional characteristic signal Y 1 。
Further, step k) comprises the steps of:
k-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels being 16 and the convolutional kernel size being 1 multiplied by 1, a batch normalization layer, a ReLU activation layer and a pooling layer with the size being 4, and the feature signal Y is processed 1 Inputting the data into a first convolutional coding module, and outputting to obtain a 16-dimensional characteristic signal P 5 ；
k-2) the second convolutional coding module sequentially comprises a convolutional layer with the channel number of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 2, and a characteristic signal P is obtained 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 32-dimensional characteristic signal P 6 ；
k-3) the third convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and a characteristic signal P is obtained 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal P 7 ；
k-4) the fourth convolutional coding module sequentially comprises a convolutional layer with the channel number of 128 and the convolutional kernel size of 1 multiplied by 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a feature signal 7 Inputting the signal into a fourth convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal P 8 ；
k-5) converting the characteristic signal P 8 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal Y 2 In the bidirectional GRU layer of TPA mechanism by formula
Figure BDA0003589207160000071
Calculating to obtain a characteristic signal Y 2 。
The invention has the beneficial effects that: firstly, the characteristics of electrocardiosignals and heart sound signals are respectively extracted by utilizing a PC-TBG-ECG model and a PC-TBG-PCG model, and then the extracted characteristics are selected and classified by adopting an XGboost integrated classification algorithm. While the operation efficiency is increased, the regularization is added, and overfitting is effectively prevented. The method is suitable for classification detection of data in different modes, and can analyze signals from various angles, so that the classification accuracy is improved.
Drawings
FIG. 1 is a flow chart of a method of the present invention;
fig. 2 is a network configuration diagram of the PC module of the present invention.
Detailed Description
The invention will be further explained with reference to fig. 1 and 2.
A multi-modal data classification method based on deep learning and time sequence attention mechanism comprises the following steps:
a) Selecting training-a in PhysioNet/CinC Change 2016 as a data set, expanding the data set, and dividing the expanded data set into a training set and a test set.
b) And establishing an electrocardiosignal model (PC-TBG-ECG), wherein the electrocardiosignal model is sequentially composed of a PC module, a TBG module and a classification module.
c) Resampling the electrocardiosignals in the training set and the testing set to 2048 sampling points, and then carrying out z-score normalization processing to obtain a normalized electrocardiosignal x' ecg 。
d) Normalizing the electrocardiosignals x 'in the training set' ecg Inputting the signal into a PC module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 1 The PC module, in turn, is made up of four convolution branches and a 1 x 1 convolution block.
e) The characteristic signal X 1 Inputting the signal into TBG module of electrocardio signal model, outputting to obtain characteristic signal X 2 The TBG module consists of 3 convolutional coding modules and a Bi-directional GRU layer (TPA-Bi-GRU) with TPA mechanism. f) The characteristic signal X 2 Inputting the prediction data into a classification module of the electrocardiosignal model, and outputting the prediction data to obtain a prediction category f ecg The classification module is composed of a full connection layer and a Softmax activation layer in sequence.
g) And (f) repeating the steps d) to f) N times, and obtaining the trained optimal electrocardiosignal model by using an SGD optimizer and minimizing a cross entropy loss function.
h) And establishing a heart sound signal model (PC-TBG-PCG), wherein the heart sound signal model is composed of a PC module, a TBG module and a classification module in sequence.
i) Resampling the heart sound signals in the training set and the test set to 8000 sampling points, and then carrying out z-score normalization processing to obtain normalized heart sound signals x' pcg 。
j) The heart sound signals x 'after normalization in the training set' pcg Inputting the signal into PC module of heart sound signal model, outputting to obtain characteristic signal Y 1 The PC module, in turn, is made up of four convolution branches and a 1 x 1 convolution block.
k) The characteristic signal Y 1 Inputting the signal into a TBG module of the heart sound signal model, and outputting to obtain a characteristic signal Y 2 The TBG module consists of 4 convolutional coding modules and oneA Bi-directional GRU layer (TPA-Bi-GRU) with TPA mechanism. l) applying the characteristic signal Y 2 Inputting the prediction into a classification module of a heart sound signal model, and outputting the prediction category f pcg The classification module is composed of a full connection layer and a Softmax activation layer in sequence.
M) repeating the steps j) to l) M times, and obtaining the trained optimal heart sound signal model by minimizing a cross entropy loss function by using an SGD optimizer.
n) manually dividing the data set into a new training set and a new testing set again according to the proportion of 4:1, inputting the new training set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 3 Inputting the new training set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG (tunnel boring generator) module of the optimal heart sound signal model 3 By the formula PP x ＝[X 3 ,Y 3 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP x 。
o) fusing features into a signal PP x Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP x The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 x And selecting an optimal hyper-parameter by adopting 5-fold cross validation, and training the XGboost classifier by utilizing the optimal hyper-parameter to obtain the optimized XGboost classifier.
p) inputting the new test set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 4 Inputting the new test set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG (tunnel boring generator) module of the optimal heart sound signal model 4 By the formula PP c ＝[X 4 ,Y 4 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP c 。
q) feature fusion signal PP c Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP c The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 c 。
The signals do not need to be subjected to noise reduction, filtering and other processing, the problems of low classification accuracy rate or poor practicability and the like caused by unreasonable signal preprocessing in the past are avoided, and the robustness of the model is ensured. Firstly, the characteristics of electrocardiosignals and heart sound signals are respectively extracted by utilizing PC-TBG-ECG and PC-TBG-PCG models, and then the XGboost integrated classification algorithm is adopted to select and classify the extracted characteristics. While the operation efficiency is increased, the regularization is added, and overfitting is effectively prevented. The method is suitable for classification detection of data in different modes, and can analyze signals from various angles, so that the classification accuracy is improved.
Example 1:
in the step a), a sliding window segmentation method is used for expanding the data set, and a five-fold cross validation method is used for dividing the data set into 5 different training sets and test sets.
Example 2:
in step c) by the formula
Figure BDA0003589207160000091
Calculating to obtain a normalized electrocardiosignal x' ecg In the formula x ecg For training and testing the concentrated ECG signal, u ecg Is the mean value, σ, of the electrocardiosignal ecg Is the variance of the electrocardiosignals.
Example 3:
the step d) comprises the following steps:
d-1) the first convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Inputting the signal into the first convolution branch, and outputting to obtain 32-dimensional characteristic signal E 1 ；
d-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 13 and step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Input into the second convolution branchAnd outputting the obtained 32-dimensional characteristic signal E 2 ；
d-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 9 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 3 ；
d-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 5 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 4 ；
d-5) converting the characteristic signal E 1 Characteristic signal E 2 Characteristic signal E 3 Characteristic signal E 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal E = [ E ] after cascade 1 ,E 2 ,E 3 ,E 4 ]；
d-6) 1 × 1 convolution block is composed of convolution layer with 16 channel number, convolution kernel size 1 × 1 and step size 1 and ReLU active layer, and 128-dimensional feature signal E = [ 1 ,E 2 ,E 3 ,E 4 ]Inputting the signal into a 1 × 1 convolution block, and outputting to obtain a 16-dimensional characteristic signal X 1 。
Example 4:
step e) comprises the following steps:
e-1) the first convolutional coding module consists of a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 4 in sequence, and the characteristic signal X is converted into a characteristic signal 1 Inputting the signal into a first convolution coding module, and outputting to obtain a 32-dimensional characteristic signal E 5 ；
E-2) the second convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and a characteristic signal E is obtained 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal E 6 ；
e-3) The third convolutional coding module consists of a convolutional layer with the channel number of 128 and the convolutional kernel size of 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2 in sequence, and a characteristic signal E is formed 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal E 7 ；
E-4) converting the characteristic signal E 7 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal X 2 In the bidirectional GRU layer of TPA mechanism by formula
Figure BDA0003589207160000111
Calculating to obtain a characteristic signal X 2 Where i = {1,2.., n }, n =128, t is the transpose, τ i For the attention weight of the ith row vector,
Figure BDA0003589207160000112
sigma (-) is a sigmoid function,
Figure BDA0003589207160000113
is a time pattern matrix G C Row i of (1), G C Conv1d (G), conv1d (·) is a one-dimensional convolution operation, G is a hidden state matrix,
Figure BDA0003589207160000114
g i i = {1,2.., t-1}, t is the time, w is the hidden state vector for the ith bidirectional GRU k Is a weight coefficient, g t Is the hidden state vector of the bi-directional GRU at time t.
Example 5:
in the step g), the value of N is 150, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 80 periods, and the formula is used
Figure BDA0003589207160000115
Calculating to obtain a cross entropy loss function cc (x), wherein L is the number of categories, and L =2,f i (x) To predict class f ecg The predictive label of the ith category of (c),
Figure BDA0003589207160000116
as a prediction class f ecg The real category of the corresponding ith category; in the step m), the value of N is 180, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 90 periods, and the formula is used
Figure BDA0003589207160000117
Calculating to obtain a cross entropy loss function cc (y), wherein L is the number of categories, and L =2,f i (y) is prediction class f pcg The predictive tag of the ith category of (a),
Figure BDA0003589207160000118
as a prediction class f pcg True category of the ith category of (1).
Example 6:
in step i) by the formula
Figure BDA0003589207160000119
Calculating to obtain a normalized heart sound signal x' pcg In the formula x pcg For the heart sound signals in the training set and test set, u pcg Is the mean value, σ, of the heart sound signal pcg Is the variance of the heart sound signal.
Example 7:
step j) comprises the following steps:
j-1) the first convolution branch is composed of convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 2, batch normalization layer and ReLU activation layer in sequence, and the heart sound signal x 'after normalization in training set' pcg Inputting the signal into the first convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 1 ；
j-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 11 and step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the second convolution branch, and outputting to obtain 32-dimensional characteristic signal P 2 ；
j-3) the third convolution branch consists of 32 channels,Convolution layer with convolution kernel size of 1 × 9 and step size of 2, batch normalization layer, and ReLU activation layer, and training set is normalized to obtain normalized heart sound signal x' pcg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 3 ；
j-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 5 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 4 ；
j-5) converting the characteristic signal P 1 Characteristic signal P 2 Characteristic signal P 3 Characteristic signal P 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal P = [ P ] after cascade 1 ,P 2 ,P 3 ,P 4 ]；
j-6) 1 × 1 convolution block is composed of convolution layer with 32 channel number, convolution kernel size of 1 × 1 and step size of 1 and ReLU active layer, and 128-dimensional characteristic signal P = [ P = 1 ,P 2 ,P 3 ,P 4 ]Inputting into a 1 × 1 convolution block, and outputting to obtain a 32-dimensional characteristic signal Y 1 。
Example 8:
step k) comprises the following steps:
k-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels being 16 and the convolutional kernel size being 1 multiplied by 1, a batch normalization layer, a ReLU activation layer and a pooling layer with the size being 4, and the feature signal Y is processed 1 Inputting the signals into a first convolutional encoding module, and outputting to obtain a 16-dimensional characteristic signal P 5 ；
k-2) the second convolutional coding module sequentially comprises a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a linear convolution function 5 Inputting the data into a second convolutional coding module, and outputting to obtain a 32-dimensional characteristic signal P 6 ；
k-3) the third convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU active layer and a convolutional layer with the size of 2Formation of pooling layer by combining characteristic signal P 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal P 7 ；
k-4) the fourth convolutional coding module sequentially comprises a convolutional layer with the channel number of 128 and the convolutional kernel size of 1 multiplied by 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a feature signal 7 Inputting the signal into a fourth convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal P 8 ；
k-5) converting the characteristic signal P 8 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal Y 2 In the bidirectional GRU layer of TPA mechanism by formula
Figure BDA0003589207160000131
Calculating to obtain a characteristic signal Y 2 。
Finally, it should be noted that: although the present invention has been described in detail with reference to the foregoing embodiments, it will be apparent to those skilled in the art that changes may be made in the embodiments and/or equivalents thereof without departing from the spirit and scope of the invention. Any modification, equivalent replacement, or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims (7)

1. A multi-modal data classification method based on deep learning and time-series attention mechanism is characterized by comprising the following steps:
a) Selecting training-a in PhysioNet/CinC Challenge 2016 as a data set, expanding the data set, and dividing the expanded data set into a training set and a test set;
b) Establishing an electrocardiosignal model, wherein the electrocardiosignal model consists of a PC module, a TBG module and a classification module in sequence;
c) Resampling the electrocardiosignals in the training set and the testing set to 2048 sampling points, and then carrying out z-score normalization processing to obtain a regressionNormalized electrocardiosignal x' ecg ；
d) Normalizing the electrocardiosignals x 'in the training set' ecg Inputting the signal into a PC module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
e) The characteristic signal X 1 Inputting the signal into a TBG module of the electrocardiosignal model, and outputting the signal to obtain a characteristic signal X 2 The TBG module consists of 3 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
f) The characteristic signal X 2 Inputting the prediction data into a classification module of the electrocardiosignal model, and outputting the prediction data to obtain a prediction category f ecg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
g) Repeating the steps d) to f) N times, and obtaining an optimal electrocardiosignal model after training by using an SGD optimizer through a minimized cross entropy loss function;
h) Establishing a heart sound signal model which sequentially consists of a PC module, a TBG module and a classification module;
i) Resampling the heart sound signals in the training set and the test set to 8000 sampling points, and then carrying out z-score normalization processing to obtain normalized heart sound signals x' pcg ；
j) Normalizing the heart sound signals x 'in the training set' pcg Inputting the signal into PC module of heart sound signal model, outputting to obtain characteristic signal Y 1 The PC module is composed of four convolution branches and a 1 multiplied by 1 convolution block in sequence;
k) The characteristic signal Y 1 Inputting the signal into a TBG module of the heart sound signal model, and outputting to obtain a characteristic signal Y 2 The TBG module consists of 4 convolutional coding modules and a bidirectional GRU layer with a TPA mechanism;
l) applying the characteristic signal Y 2 Inputting the prediction into a classification module of a heart sound signal model, and outputting the prediction category f pcg The classification module is sequentially composed of a full connection layer and a Softmax activation layer;
m) repeating the steps j) to l) M times, and obtaining an optimal heart sound signal model after training by minimizing a cross entropy loss function by using an SGD optimizer;
n) manually dividing the data set into a new training set and a new testing set according to the proportion of 4:1, inputting the new training set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 3 Inputting the new training set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 3 By the formula PP x ＝[X 3 ,Y 3 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP x ；
o) fusing features into a signal PP x Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP x The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 x Selecting an optimal hyper-parameter by adopting 5-fold cross validation, and training the XGboost classifier by utilizing the optimal hyper-parameter to obtain an optimized XGboost classifier;
p) inputting the new test set into the optimal electrocardiosignal model, and outputting a 64-dimensional characteristic signal X through a TBG (tunnel boring generator) module of the optimal electrocardiosignal model 4 Inputting the new test set into the optimal heart sound signal model, and outputting a 64-dimensional characteristic signal Y through a TBG module of the optimal heart sound signal model 4 By the formula PP c ＝[X 4 ,Y 4 ]Calculating to obtain spliced 128-dimensional feature fusion signal PP c ；
q) feature fusion signal PP c Inputting the signals into an XGboost classifier to obtain a feature fusion signal PP c The importance score ranking of (2) and selecting the signal of the top 64 of the importance score ranking as the characteristic signal PP 1 c ；
The step d) comprises the following steps:
d-1) the first convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Input into the first convolution branch, and output is obtained32-dimensional characteristic signal E 1 ；
d-2) the second convolution branch comprises a convolution layer with 32 channels, convolution kernel size of 1 × 13 and step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignal x 'after normalization in the training set' ecg Inputting the signal into the second convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 2 ；
d-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 9 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 3 ；
d-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 x 5 and a step length of 1, a batch normalization layer and a ReLU activation layer in sequence, and the electrocardiosignals x 'after normalization in the training set' ecg Inputting the signal into the fourth convolution branch, and outputting to obtain a 32-dimensional characteristic signal E 4 ；
d-5) converting the characteristic signal E 1 Characteristic signal E 2 Characteristic signal E 3 Characteristic signal E 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal E = [ E ] after cascade 1 ,E 2 ,E 3 ,E 4 ]；
d-6) 1 × 1 convolution block is composed of convolution layers with 16 channels, 1 × 1 convolution kernel size and 1 step size and ReLU active layer, and a 128-dimensional characteristic signal E = [ E ] 1 ,E 2 ,E 3 ,E 4 ]Inputting the signal into a 1 × 1 convolution block, and outputting to obtain a 16-dimensional characteristic signal X 1 ；
The step j) comprises the following steps:
j-1) the first convolution branch is composed of convolution layer with 32 channels, convolution kernel size of 1 × 15 and step size of 2, batch normalization layer and ReLU activation layer in sequence, and the heart sound signal x 'after normalization in training set' pcg Inputting the signal into the first convolution branch, and outputting to obtain 32-dimensional characteristic signal P 1 ；
j-2) the second convolution branch is composed of 32 channels and convolution kernelA convolution layer of 1 × 11 size and step size 2, a batch normalization layer, and a ReLU activation layer, and the normalized heart sound signal x 'in the training set' pcg Inputting the signal into the second convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 2 ；
j-3) the third convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 9 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into a third convolution branch, and outputting to obtain a 32-dimensional characteristic signal P 3 ；
j-4) the fourth convolution branch comprises a convolution layer with a channel number of 32, a convolution kernel size of 1 × 5 and a step size of 2, a batch normalization layer and a ReLU activation layer in sequence, and the heart sound signal x 'after normalization in the training set' pcg Inputting the signal into the fourth convolution branch, and outputting to obtain 32-dimensional characteristic signal P 4 ；
j-5) converting the characteristic signal P 1 Characteristic signal P 2 Characteristic signal P 3 Characteristic signal P 4 Performing characteristic cascade to obtain a 128-dimensional characteristic signal P = [ P ] after cascade 1 ,P 2 ,P 3 ,P 4 ]；
j-6) 1 × 1 convolution block is composed of convolution layer with 32 channel number, convolution kernel size of 1 × 1 and step size of 1 and ReLU active layer, and 128-dimensional characteristic signal P = [ P = 1 ,P 2 ,P 3 ,P 4 ]Inputting into a 1 × 1 convolution block, and outputting to obtain a 32-dimensional feature signal Y 1 。
2. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in the step a), a sliding window segmentation method is used for expanding the data set, and a five-fold cross validation method is used for dividing the data set into 5 different training sets and test sets.
3. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in step c) byIs of the formula
Figure FDA0003935458000000041
Calculating to obtain a normalized electrocardiosignal x' ecg In the formula x ecg For training and testing the concentrated ECG signal u ecg Is the mean value, σ, of the electrocardiosignal ecg Is the variance of the electrocardiosignals.
4. The method for multi-modal data classification based on deep learning and temporal attention mechanism according to claim 1, wherein step e) comprises the following steps:
e-1) the first convolutional coding module consists of a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 4 in sequence, and the characteristic signal X is converted into a characteristic signal 1 Inputting the signal into a first convolution coding module, and outputting to obtain a 32-dimensional characteristic signal E 5 ；
E-2) the second convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 2, and a characteristic signal E is generated 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal E 6 ；
E-3) the third convolution coding module consists of a convolution layer with the channel number of 128 and the convolution kernel size of 3, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2 in sequence, and a characteristic signal E is generated 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal E 7 ；
E-4) converting the characteristic signal E 7 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal X 2 In the bidirectional GRU layer of TPA mechanism by formula
Figure FDA0003935458000000051
Calculating to obtain a characteristic signal X 2 Where i = {1,2.., n }, n =128, t is the transpose, τ i For the attention weight of the ith row vector,
Figure FDA0003935458000000052
σ (-) is a sigmoid function,
Figure FDA0003935458000000053
is a time pattern matrix G C Row i of (1), G C Conv1d (G), conv1d (·) is a one-dimensional convolution operation, G is a hidden state matrix,
Figure FDA0003935458000000054
g i i = {1,2., t-1}, t is the time, w is the hidden state vector of the ith bidirectional GRU k Is a weight coefficient, g t Is the hidden state vector of the bi-directional GRU at time t.
5. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in the step g), the value of N is 150, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 80 periods, and the formula is used
Figure FDA0003935458000000055
Calculating to obtain a cross entropy loss function cc (x), wherein L is the number of categories, and L =2,f i (x) As a prediction class f ecg The predictive label of the ith category of (c),
Figure FDA0003935458000000056
as a prediction class f ecg The real category of the corresponding ith category; in the step m), the value of N is 180, the learning rate of the SGD optimizer is 0.001, the learning rate is attenuated to be 0.1 at every 90 periods, and the N is obtained by a formula
Figure FDA0003935458000000057
Calculating to obtain a cross entropy loss function cc (y), wherein L is the number of categories, and L =2,f i (y) as a prediction class f pcg The predictive tag of the ith category of (a),
Figure FDA0003935458000000058
as a prediction class f pcg True category of the ith category of (1).
6. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein: in step i) by the formula
Figure FDA0003935458000000061
Calculating to obtain a normalized heart sound signal x' pcg In the formula x pcg For the heart sound signals in the training set and test set, u pcg Is the mean value, σ, of the heart sound signal pcg Is the variance of the heart sound signal.
7. The multi-modal data classification method based on deep learning and time series attention mechanism as claimed in claim 1, wherein step k) comprises the following steps:
k-1) the first convolutional coding module sequentially comprises a convolutional layer with the number of channels being 16 and the convolutional kernel size being 1 multiplied by 1, a batch normalization layer, a ReLU activation layer and a pooling layer with the size being 4, and the feature signal Y is processed 1 Inputting the signals into a first convolutional encoding module, and outputting to obtain a 16-dimensional characteristic signal P 5 ；
k-2) the second convolutional coding module sequentially comprises a convolutional layer with the number of channels of 32 and the convolutional kernel size of 1 multiplied by 11, a batch normalization layer, a ReLU activation layer and a pooling layer with the size of 2, and the feature signal P is converted into a linear convolution function 5 Inputting the signal into a second convolutional coding module, and outputting to obtain a 32-dimensional characteristic signal P 6 ；
k-3) the third convolutional coding module sequentially comprises a convolutional layer with the channel number of 64 and the convolutional kernel size of 1 multiplied by 7, a batch normalization layer, a ReLU active layer and a pooling layer with the size of 2, and a characteristic signal P is obtained 6 Inputting the signal into a third convolutional coding module, and outputting to obtain a 64-dimensional characteristic signal P 7 ；
k-4) the fourth convolutional encoding module sequentially comprises a channel number of 128 and a convolutional kernel size of 1 x 3A convolution layer, a batch normalization layer, a ReLU active layer, a pooling layer of size 2, and a feature signal P 7 Inputting the signal into a fourth convolutional coding module, and outputting to obtain a 128-dimensional characteristic signal P 8 ；
k-5) converting the characteristic signal P 8 Inputting into 32-unit bidirectional GRU layer with TPA mechanism, and outputting to obtain 64-dimensional characteristic signal Y 2 In the bidirectional GRU layer of TPA mechanism by formula
Figure FDA0003935458000000062
Calculating to obtain a characteristic signal Y 2 。
CN202210376944.3A 2022-04-11 2022-04-11 Multi-modal data classification method based on deep learning and time sequence attention mechanism Active CN114764575B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210376944.3A CN114764575B (en) 2022-04-11 2022-04-11 Multi-modal data classification method based on deep learning and time sequence attention mechanism

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210376944.3A CN114764575B (en) 2022-04-11 2022-04-11 Multi-modal data classification method based on deep learning and time sequence attention mechanism

Publications (2)

Publication Number Publication Date
CN114764575A CN114764575A (en) 2022-07-19
CN114764575B true CN114764575B (en) 2023-02-28

Family

ID=82364741

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210376944.3A Active CN114764575B (en) 2022-04-11 2022-04-11 Multi-modal data classification method based on deep learning and time sequence attention mechanism

Country Status (1)

Country Link
CN (1) CN114764575B (en)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116186593B (en) * 2023-03-10 2023-10-03 山东省人工智能研究院 Electrocardiosignal detection method based on separable convolution and attention mechanism

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018130541A (en) * 2017-02-16 2018-08-23 タタ コンサルタンシー サービシズ リミテッドＴＡＴＡ Ｃｏｎｓｕｌｔａｎｃｙ Ｓｅｒｖｉｃｅｓ Ｌｉｍｉｔｅｄ Method and system for detection of coronary artery disease in person using fusion approach
CN110236518A (en) * 2019-04-02 2019-09-17 武汉大学 The method and device of electrocardio and the heart shake combined signal classification neural network based
CN110537910A (en) * 2019-09-18 2019-12-06 山东大学 coronary heart disease noninvasive screening system based on electrocardio and heart sound signal joint analysis
CN113288163A (en) * 2021-06-04 2021-08-24 浙江理工大学 Multi-feature fusion electrocardiosignal classification model modeling method based on attention mechanism
CN113855063A (en) * 2021-10-21 2021-12-31 华中科技大学 Heart sound automatic diagnosis system based on deep learning
CN114190952A (en) * 2021-12-01 2022-03-18 山东省人工智能研究院 12-lead electrocardiosignal multi-label classification method based on lead grouping

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2018130541A (en) * 2017-02-16 2018-08-23 タタ コンサルタンシー サービシズ リミテッドＴＡＴＡ Ｃｏｎｓｕｌｔａｎｃｙ Ｓｅｒｖｉｃｅｓ Ｌｉｍｉｔｅｄ Method and system for detection of coronary artery disease in person using fusion approach
CN110236518A (en) * 2019-04-02 2019-09-17 武汉大学 The method and device of electrocardio and the heart shake combined signal classification neural network based
CN110537910A (en) * 2019-09-18 2019-12-06 山东大学 coronary heart disease noninvasive screening system based on electrocardio and heart sound signal joint analysis
CN113288163A (en) * 2021-06-04 2021-08-24 浙江理工大学 Multi-feature fusion electrocardiosignal classification model modeling method based on attention mechanism
CN113855063A (en) * 2021-10-21 2021-12-31 华中科技大学 Heart sound automatic diagnosis system based on deep learning
CN114190952A (en) * 2021-12-01 2022-03-18 山东省人工智能研究院 12-lead electrocardiosignal multi-label classification method based on lead grouping

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Integrating multi-domain deep features of electrocardiogram and phonocardiogram for coronary artery disease detection;HanLi, et al.;《Computers in Biology and Medicine》;20211130;1-7 *
基于心音心电信号的心衰分析系统;李俊杰 等;《Software Engineering and Applications》;20220209;1-10 *

Also Published As

Publication number Publication date
CN114764575A (en) 2022-07-19

Similar Documents

Publication Publication Date Title
CN112766355B (en) Electroencephalogram signal emotion recognition method under label noise
Che et al. Hybrid multimodal fusion with deep learning for rolling bearing fault diagnosis
CN111414942A (en) Remote sensing image classification method based on active learning and convolutional neural network
CN112294341B (en) Sleep electroencephalogram spindle wave identification method and system based on light convolutional neural network
CN110390952A (en) City sound event classification method based on bicharacteristic 2-DenseNet parallel connection
CN114564990B (en) Electroencephalogram signal classification method based on multichannel feedback capsule network
CN112732921B (en) False user comment detection method and system
CN114176607B (en) Electroencephalogram signal classification method based on vision transducer
CN113554110B (en) Brain electricity emotion recognition method based on binary capsule network
CN110909928B (en) Energy load short-term prediction method and device, computer equipment and storage medium
CN113343860A (en) Bimodal fusion emotion recognition method based on video image and voice
CN113749657A (en) Brain wave emotion recognition method based on multitask capsules
CN111582041A (en) Electroencephalogram identification method based on CWT and MLMSFFCNN
CN116186593B (en) Electrocardiosignal detection method based on separable convolution and attention mechanism
CN114764575B (en) Multi-modal data classification method based on deep learning and time sequence attention mechanism
CN113069117A (en) Electroencephalogram emotion recognition method and system based on time convolution neural network
CN114595725B (en) Electroencephalogram signal classification method based on addition network and supervised contrast learning
CN115273236A (en) Multi-mode human gait emotion recognition method
CN116258914B (en) Remote Sensing Image Classification Method Based on Machine Learning and Local and Global Feature Fusion
Cho et al. Fruit ripeness prediction based on DNN feature induction from sparse dataset
CN116610846A (en) Breeding screening method and system for cattle breeding
CN112465054B (en) FCN-based multivariate time series data classification method
CN114420151B (en) Speech emotion recognition method based on parallel tensor decomposition convolutional neural network
CN112883905B (en) Human behavior recognition method based on coarse-grained time-frequency features and multi-layer fusion learning
CN112560784B (en) Electrocardiogram classification method based on dynamic multi-scale convolutional neural network

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant