KR102639807B1 - Multimodal Multitask Deep Learning Model for Alzheimer's Disease Progression Detection based on Time Series Data - Google Patents

Abstract

A multi-mode multi-task learning method and device for detecting Alzheimer's disease progression based on time series data are presented. The multi-mode multi-task learning method for detecting the progression of Alzheimer's disease based on time series data proposed in the present invention receives multiple types of multi-mode time series data and a series of background data and learns the local characteristics and characteristics of each mode of the multi-mode time series data. Extracting longitudinal features, performing preprocessing to improve the data quality of the features of the extracted multimode time series data, and learning about the characteristics of the preprocessed multimode time series data through a stacked CNN-BiLSTM network. step, extract common features in all modes using a series of dense layers for the learned features, a decision step to extract features using a series of dense layers for the background data, and the extracted It involves learning task-specific characteristics using a set of dense layers for common characteristics of multi-mode time series data and characteristics of background data, and finally detecting the results by classification or regression.

Description

{Multimodal Multitask Deep Learning Model for Alzheimer's Disease Progression Detection based on Time Series Data}

The present invention relates to a multi-mode multi-task deep learning method and device for detecting Alzheimer's disease progression based on time series data.

Alzheimer's disease (AD) accounts for 60% to 70% of dementia in the elderly, and it is expected that 115.4 million people will suffer from dementia in 2050 [1]. There is no cure for AD, and current treatments only slow the progression of AD [2]. As a result, its early prediction is of fundamental importance for timely treatment and delay in progression. Mild Cognitive Impairment (MCI) is a broad, ill-defined, and highly heterogeneous phenotypic spectrum that causes memory deficits that are relatively less noticeable than AD [3]. Each year, approximately 10% to 20% of MCI patients progress to AD [4]. The gradual change from MCI to AD takes years, if not decades. It is a difficult task to distinguish between stable MCI (sMCI) patients who will not progress to AD and progressive MCI (pMCI) patients who will later develop AD [6]. Machine Learning (ML) techniques can play an important role in helping medical professionals analyze patient data. AD symptomatology is multimodal and longitudinal [7,8]. Patient data consists of a collection of heterogeneous but complementary data of different types, including Magnetic Resonance Imaging (MRI), Positron Emission Tomography (PET), genetics, and Cerebrospinal Fluid (CSF) [ 9]. The combination of multimodes facilitates the detection and differentiation of all subtle changes in the patient's progress and supports reliable diagnosis [5]. Over the past decade, regular ML algorithms (particularly Support Vector Machine (SVM) and Random Forest (RF)) have been utilized for MCI transformation prediction [9-13]. Most studies utilized unimodal and single-task models, such as sMCI versus pMCI classification [5,14] or cognitive score regression [15]. This design paradigm is called single https format single operation, where the model optimizes only a single objective function based on one type of data [16]. In these models, neither correlations between tasks nor complementary information between modes are investigated [17]. Multimodal systems generally provide comprehensive insights, more accurate results, and more stable actions, and as a result, have been shown to be more acceptable in healthcare [13,18-20]. Liu [21] and Duchesne [22] used regular machine learning techniques to study multimode single-task classification and regression analysis, respectively. Multimodal data can be fused in different ways, and selecting the best mode combination and appropriate fusion scheme is a difficult task [13,23]. Additionally, single task models lack the ability to provide useful knowledge to healthcare professionals regarding the patient's possible cognitive behavior during progression. Some studies design MCI progression as a multimodal single-task regression model, in which some cognitive scores, such as the Mini-Mental State Examination (MMSE) and the Alzheimer's Diseases Assessment Scale (ADAS), are associated with AD It is an indicator of progress [22,24]. Conversely, Zhou [25] studied AD progression as a single-mode multitask regression problem. However, in real medical environments, many modes must be chronically analyzed and several clinical variables predicted. ML models that can perform this task are called multimodal multitask models, where every task has features from multiple sources and multiple tasks are related chronologically [7,8,17,26]. Zhang and Shen [9] proposed an SVM-based method to jointly predict multiple medical scores (i.e., MMSE, ADAS, and diagnostic features) by fusing multimodal data (i.e., MRI, PET, and CSF). Recently, Ding [8] argued that most AD studies only consider a limited number of factors, which is potentially insufficient to understand the complex and multifactorial nature of the disease. Most studies consider MCI progression as a binary sMCI vs. pMCI classification problem based on primary data [4]. This is an optimal strategy because baseline data are less discriminatory for progression detection than considering patients' longitudinal data, resulting in a less accurate model. Because AD is a chronic disease, patient data is always a time series. Patient data accumulates from different visits and forms continuous patient supervision. The disease state at a particular point in time is not independent of the state at a previous point in time. As a result, AD data is not only multimodal but also time series. Therefore, considering AD multimode data as a time series is an intuitive solution to the AD progression problem. However, the majority of studies do not consider these temporal/sequential characteristics of AD data [1]. Some studies have adopted traditional time series algorithms to the AD progression detection problem [27,28], and the correlation between patients' multimodal data and how they evolve has not been analyzed [6]. Recently, Li [4] argued for the urgency of multimodal and longitudinal analysis of AD data. Because multi-task learning serves as a regularizer for all tasks, multi-mode multi-task modeling of AD progress based on time series data is a task that promises significant improvement in model performance [29]. Moreover, most AD classifiers, such as SVM, are based on two independent steps: dimensionality reduction and classification. These two models are mathematically independent and include different assumptions. Additionally, these techniques require the use of a kernel selected from a pre-specified set. Recently, deep learning techniques have demonstrated promising prediction results in several fields [30,31]. All previous tasks could be effectively managed using Deep Learning (DL) [5,32-37]. In the AD context, Choi and Jin [38] utilized CNN to detect pMCI cases based on a single source (PET image) single task model. Spasov [39] proposed a CNN-based multimodal single-task classification model to detect AD progression based on late fusion of MRI, demographic, neuropsychological, and apolipoprotein E4 (APOe4) genetic data. Liu [7] proposed a CNN-based model for joint AD classification and clinical score regression analysis. The model is based on the fusion of MRI and three demographic characteristics collected only at the baseline visit. Most Alzheimer's DL models are based on CNNs and a single (baseline) MRI scan [5]. These models are inaccurate, insufficient, and medically unacceptable. This is because medical experts typically study longitudinal multimodal patient data before making decisions to proceed [8].

The technical task to be achieved by the present invention is to provide a deep learning model based on a stacked CNN (Convolutional Neural Network) and BiLSTM (Bidirectional Long Short Term Memory) network. The proposed multi-mode multi-task model jointly predicts multiple variables based on the fusion of five types of multi-mode time series data and a set of BGs, and uses stacked CNN and BiLSTM networks to predict the local and longitudinal characteristics of each mode ( longitudinal features are extracted, and the resulting features are fused in a deep network to detect common patterns that are jointly used to predict classification and regression tasks.

In one aspect, the multi-mode multi-task learning method for detecting the progression of Alzheimer's disease based on time series data proposed in the present invention receives a plurality of types of multi-mode time series data and a series of background data and learns each mode of the multi-mode time series data. Extracting local features and longitudinal features for, performing preprocessing to improve the data quality of the characteristics of the extracted multimode time series data, Stacked CNN- Steps of learning with a BiLSTM network, extracting common features from all modes using a series of dense layers for the learned features, and pseudo-processing to extract features using a series of dense layers for the background data. It involves learning task-specific characteristics using a set of dense layers for the common characteristics of the decision stage and the extracted multi-mode time series data and the characteristics of the background data, and finally detecting the results by classification or regression.

The step of learning about the characteristics of the preprocessed multimode time series data through a stacked CNN-BiLSTM network uses a CNN using 1D convolution to learn each mode about the characteristics of the multimode time series data, and for each mode To transform time series data of multiple tensors into a new feature space using one CNN layer, 1D convolution is performed separately along the time dimension for all input vectors, and based on the number of filters, 1D convolution is performed on all the CNNs. The input vector for univariate time series data is mapped to a new feature space and expanded into a feature map for LSTM prediction.

The step of learning through a stacked CNN-BiLSTM network regarding the characteristics of the preprocessed multimode time series data is to input the characteristics learned from the CNN and learn the characteristics of the longitudinal data through the BiLSTM layer to utilize the temporal correlation of the multimode time series data. It finds patterns and controls the updating, maintenance, and deletion of information contained in the cell state through LSTM cells, which include input gates, forget gates, and output gates.

The step of learning through a stacked CNN-BiLSTM network regarding the characteristics of the preprocessed multi-mode time series data is learning through a stacked CNN-BiLSTM network including multiple concurrent stacked CNN-BiLSTM pipelines and one feedforward neural network. Perform, the CNN for each CNN-BiLSTM pipeline includes one 1D convolutional layer followed by max pooling, the BiLSTM includes multiple stacked BiLSTM layers, an L2 regularization layer and a dropout layer, and the CNN for each multimode It is applied to extract local features from time series data, and BiLSTM is applied to learn features of the same type and temporal relationships within a single time series data by receiving features learned from CNN.

The decision-making step to extract common features across all modes using a series of dense layers for the learned features, and to extract features using a series of dense layers for the background data features learned in BiLSTM. is input into multiple dense layers to perform deep feature learning, and then fused by multiple dense layers.

In another aspect, the multi-mode multi-task learning device for detecting the progression of Alzheimer's disease based on time series data proposed in the present invention receives a plurality of types of multi-mode time series data and a series of background data and generates multi-mode time series data. A feature extraction unit that extracts local characteristics and longitudinal characteristics for each mode, a preprocessing unit that performs preprocessing to improve the data quality of the characteristics of the extracted multimode time series data, and a preprocessing unit that performs preprocessing to improve the data quality of the characteristics of the extracted multimode time series data. Regarding the CNN-BiLSTM training unit, which learns through a stacked CNN-BiLSTM network, extracts common features in all modes using a series of dense layers for the learned features, and a series of dense layers for the background data. A decision-making unit that extracts characteristics using a decision-making unit and a judgment unit that learns task-specific characteristics using a dense layer set for the common characteristics of the extracted multi-mode time series data and the characteristics of the background data and finally detects the results by classifying or regressing. Includes.

Five types of multi-mode time series data and Based on the fusion of a series of BGs, multiple variables can be jointly predicted, and local and longitudinal features of each mode are extracted using stacked CNN and BiLSTM networks, and the resulting features predict classification and regression tasks. They can be fused in deep networks to detect common patterns that are jointly used to

Figure 1 is a flowchart illustrating a multi-mode multi-task learning method for detecting the progression of Alzheimer's disease based on time series data according to an embodiment of the present invention.
Figure 2 is a diagram illustrating a deep learning model in which a multi-mode multi-task learning method for detecting the progression of Alzheimer's disease based on time series data according to an embodiment of the present invention is performed.
Figure 3 is a diagram illustrating a CNN network that learns local features using a 1D convolution layer according to an embodiment of the present invention.
Figure 4 is a diagram for explaining the architecture of a BiLSTM network according to an embodiment of the present invention.
Figure 5 is a diagram showing the structure of a multi-mode multi-task learning device for detecting the progression of Alzheimer's disease based on time series data according to an embodiment of the present invention.

Early prediction of Alzheimer's disease is very important in delaying the progression of Alzheimer's disease (AD). As a chronic disease, ignoring the temporal dimension of AD data impacts progression detection performance and is medically unacceptable. Furthermore, AD patients are represented by heterogeneous but complementary multimodality. Multi-task modeling improves progress-detection performance, robustness, and stability. However, multimodal multitask modeling has not been evaluated using time series and deep learning paradigms specifically for AD progression detection.

In the present invention, we use time-series multi-modality data from AD patients to investigate prediction performance and its impact on progression, and utilize CNN and Recurrent Neural Network (RNN) to capture local and long-term temporal dependencies, respectively [40 ]. Wang [40] proposed a Long Short-Term Memory (LSTM)-based regression model to predict AD progression from time series data with non-constant visit intervals. Unlike the AD progression problem, advanced DL techniques based on the combination of CNN and RNN models have been proposed in other areas of the industry [26,30,31,41,42] and have achieved superior performance compared to deep learning techniques. Cui [43] proposed a CNN-BiLSTM model for AD diagnosis based on six levels of MRI time series data. Most DL models in the AD domain are implemented in binary classification, but multi-class models still do not reach satisfactory results for clinical applicability [44]. Because AD data is complex, a DL model based on combined CNN-BiLSTM can outperform models based on CNN or LSTM alone. Additionally, increasing the number of time steps used for longitudinal data improves system performance [43]. Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

In the present invention, we propose a powerful ensemble deep learning model based on a stacked Convolutional Neural Network (CNN) and Bidirectional Long Short Term Memory (BiLSTM) network. This multi-mode multi-task model jointly predicts multiple variables based on the fusion of five types of multi-mode time series data and a set of background knowledge. Predictors include the AD multiclass progression test and the four important cognitive score regression tests. The proposed model uses stacked CNN and BiLSTM networks to extract local and longitudinal features of each mode. At the same time, local features are extracted from BG data using a feedforward neural network. The resulting features are fused in a deep network to detect common patterns that are jointly used to predict classification and regression tasks.

To verify the proposed model, six experiments were performed on five modes of the Alzheimer's Disease Neuroimaging Initiative (ADNI) in 1536 subjects. The results of the proposed model achieve state-of-the-art performance for both multi-class progressive and regression tasks. Moreover, the proposed model can be generalized to other medial domains to analyze heterogeneous temporal data to predict the patient's future status.

According to an embodiment of the present invention, multimodal joint prediction of multiple categorical and continuous variables based on late fusion of time series and static data can perform better than predicting each individual variable individually.

In the present invention, we propose an advanced multi-mode task DL architecture for detecting AD progression. This framework leverages patient time series data to jointly predict multiple variables from multiple sources. The resulting comprehensive system is more medically intuitive, stable, and accurate than recent prior art. According to the prior art, no prior art has proposed a method for integrating multimodal and multitask architectures based on time series data to create a personalized, accurate, and medically reliable AD progression model using deep learning.

The model proposed in the present invention jointly learns how to simultaneously predict the patient's progress status and the values of four important cognitive scores at the time of progress. The predicted clinical scores are ADAS, MMSE, Functional Assessment Questionnaire (FAQ), and Clinical Dementia Rating Sum of Boxes (CDRSB) scores, which are implemented in four regression tasks. Progression detection is a multi-class classification task (i.e., Cognitive Normal (CN) vs. sMCI vs. pMCI vs. AD). Medically, these related tasks share a subset of commonly related characteristics.

Compared with the prior art, the proposed model is the first attempt to extract temporal features from five heterogeneous data sources and a set of static baseline features. Each time series source is trained separately using a pipeline of stacked CNN-BiLSTM blocks. CNN automatically extracts local features from each time series, and LSTM extracts temporal features from each feature and the temporal relationship between features. Then, the learned shapes from all modes are fused to extract context-aware common features.

Since little effort has been put into exploring the role of static data as background knowledge (BG) for improving model performance [45], we studied the impact of such data on the performance of models. To prepare BG data, we collected several types of baseline data (e.g., age, gender, CSF, symptoms, etc.) from the patient's first visit and a series of statistical measurements extracted from time series data. These features are fed into the model, where they are simultaneously learned by separate feedforward neural networks. The learned deep features are then fused back with the learned features of the mode to jointly predict the patient's progress status and cognitive score.

In the present invention, we performed extensive experiments to evaluate the model in various environments using a dataset of 1536 patient samples from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. Six experiments were conducted and tested: (1) The deep-stack CNN-BiLSTM was proven to be more accurate than the connected CNN-BiLSTM network structure. (2) Late fusion of time series and BG achieved better performance compared to early fusion structure. (3) The multi-task model of classification and regression tasks provided a more stable and more accurate system than the single-task model. (4) Adding more data to the DL model (i.e., more time steps or more modes in the same mode) improved its performance even if the data was noisy. (5) MRI and PET based on an extended mode selection process are the most useful modes, and finally (6) statistical properties extracted from time series data are more important than reference data.

Data used in this study were accessed from ADNI 1, ADNI GO and ADNI 2 on March 18, 2019. We included 1536 patients (54.7% male) divided into 4 groups based on individual clinical diagnosis at baseline and future time points. Participants are divided into four categories. The first category includes 419 patients who were diagnosed with CN at baseline and remained CN when ready for testing according to embodiments of the invention. The second category includes 473 patients diagnosed with stable MCI at all time points in the trial according to embodiments of the present invention. The third category includes 305 patients who were assessed as having progressive MCI at the baseline visit and who progressed to AD at some point during the trial period (84 months) according to an embodiment of the invention. Finally, the fourth category includes 339 patients with a clinical diagnosis of AD at all visits. During follow-up, patients who were clinically diagnosed with MCI but returned to CN, or patients who were clinically diagnosed with AD but returned to MCI or CN were excluded from the study considering that the clinical diagnosis was an irreversible form of dementia. Additionally, cases of direct conversion from CN to AD were also removed. Demographic and clinical information about the subjects is in Table 1.

<Table 1>

Figure 1 is a flowchart illustrating a multi-mode multi-task learning method for detecting the progression of Alzheimer's disease based on time series data according to an embodiment of the present invention.

The proposed multi-mode multi-task learning method for detecting Alzheimer's disease progression based on time series data receives multiple types of multi-mode time series data and a series of background data and determines the local characteristics and longitudinal data for each mode of the multi-mode time series data. ) Step of extracting features (110), step of performing preprocessing to improve the data quality of the features of the extracted multi-mode time series data (120), stacked CNN-BiLSTM network with respect to the characteristics of the pre-processed multi-mode time series data Step 130 of learning through, extracting common features in all modes using a series of dense layers for the learned features, and extracting features using a series of dense layers for the background data. It includes a decision-making step (140) and a step (150) of learning task-specific characteristics using a set of dense layers for the common characteristics of the extracted multi-mode time series data and the characteristics of the background data, and finally detecting the results by classifying or regressing. do.

In step 110, a plurality of types of multi-mode time series data and a set of background data are input, local characteristics and longitudinal characteristics for each mode of the multi-mode time series data are extracted, and in step 120, the extracted Preprocessing is performed to improve the data quality of the characteristics of multi-mode time series data.

In step 130, the characteristics of the preprocessed multimodal time series data are learned through a stacked CNN-BiLSTM network.

In step 130, a CNN using 1D convolution is used to learn each mode about the characteristics of the multi-mode time series data, and one CNN layer for each mode is used to transform the time series data of the plurality of tensors into new characteristics. To transform to space, 1D convolution is performed separately along the time dimension for all input vectors. 1D convolution is based on the number of filters, and CNN maps the input vector for all univariate time series data to a new feature space, extending it into a feature map for LSTM prediction.

Afterwards, in order to take advantage of the temporal correlation of the multi-mode time series data by receiving the learned features from the CNN, temporal patterns are found in the longitudinal data through a BiLSTM layer, and the cells are analyzed through an LSTM cell including an input gate, a forgetting gate, and an output gate. Controls the updating, maintenance, and deletion of information contained in the state.

In step 130, training is performed through a stacked CNN-BiLSTM network including multiple concurrently stacked CNN-BiLSTM pipelines and one feedforward neural network. For each CNN-BiLSTM pipeline, the CNN contains one 1D convolutional layer followed by max pooling, and the BiLSTM contains multiple stacked BiLSTM layers, an L2 regularization layer, and a dropout layer. At this time, CNN is applied to extract local features from each multi-mode time series data, and BiLSTM is applied to receive features learned from CNN and learn the same type of features and temporal relationships within single time series data.

In step 140, a decision is made to extract common features across all modes using a series of dense layers for the learned features and to extract features using a series of dense layers for the background data. Perform. In step 140, deep feature learning is performed by inputting the features learned from BiLSTM into a plurality of dense layers and then fused by the plurality of dense layers.

In step 150, the characteristics of each task are learned using a dense layer set for the common characteristics of the extracted multi-mode time series data and the characteristics of the background data, and the results are finally detected by classification or regression. 2 to 4, the multi-mode multi-task learning process for detecting the progression of Alzheimer's disease based on time series data according to an embodiment of the present invention will be described in more detail.

Figure 2 is a diagram illustrating a deep learning model in which a multi-mode multi-task learning method for detecting the progression of Alzheimer's disease based on time series data according to an embodiment of the present invention is performed.

The proposed model was designed for multimodal multitask purposes to learn AD progression and four cognitive scores based on multivariate time series data. Initially, five time series modes with 15 regular time steps (i.e. baseline, M06, M12, ..., M84) were fed to the model separately along with BG data. Learning local and temporal features of the model is based on stacked CNN and BiLSTM subnetworks.

As shown in Figure 2, the proposed model initially prepares multimodal data (Multimodal data) 211, which is time series data of five modes. Neuroimaging features of MRI and PET features are extracted using FreeSurfer. Features extracted from multimodality (i.e. cognitive scores, neuropsychological battery, MRI, PET, and assessment modes) are preprocessed to improve data quality. The PCA technique for feature reduction is used to extract principal components from high-dimensional MRI and PET data. Afterwards, deep features are separately learned from each time series multimode using a stacked CNN-BiLSTM model. The abstract deep features learned in the previous step are fused in a deep learning structure 220 to extract common features across all modes using a series of dense layers 231 . At the same time, background knowledge (BG) data 212 is preprocessed and trained using a series of dense layers 232 to extract representative deep features. To obtain common deep features of the time series mode and representative deep features of the background data, a second decision fusion 230 by the dense layer 233 is used to obtain more abstract deep features. The final step is multitask learning (240), in which a set of dense layers is used to learn features for each task. Softmax or Sigmoid are then used for classification or regression tasks, respectively (250).

Figure 3 is a diagram illustrating a CNN network that learns local features using a 1D convolution layer according to an embodiment of the present invention.

As explained in FIG. 2, a separate CNN subnetwork 220 is used for learning each mode of multi-mode data. CNN introduced 1D convolution (Conv1D), which can learn univariate time series data, which is multi-mode data. Convolution is performed separately along the time dimension for all input vectors (see Figure 3). Formally, the input vector and if the kernel r is m×1, Conv1D defines x as the new feature space maps to where d is the step size. Based on the number of filters, CNN expands any univariate time series into more abstract and informative features called feature maps, which are more suitable for LSTM prediction. Each value f _i in feature map f is is provided as an activation function g to calculate , where g is a non-linear activation function (according to an embodiment of the present invention, , b is bias, is the j observation at x. We propose to separately transform time series of multiple tensors into a new feature space using one CNN layer for each mode. what mode For , using k filters of (m×1), the corresponding output tensor is , where n is the number of samples, l is the number of time steps, and f is the number of features. Max pooling layers are used to smooth the input, prevent overfitting, and learn higher-level abstractions.

Figure 4 is a diagram for explaining the architecture of a BiLSTM network according to an embodiment of the present invention.

To take advantage of the temporal correlation of time series data, we added a BiLSTM layer to find temporal patterns in longitudinal data. The input to the BiLSTM block is the features learned from the CNN layer described above. The core structure of an LSTM cell is the input gate ( ), Forget Gate ( ), output gate ( ) using three gates. This gate controls the updating, maintenance, and deletion of information contained in the cell state.

, and are the updates of the current cell state value, the last time step cell state value, and the current cell state value at time t _n , respectively, is the output value of each memory cell of the hidden layer at the previous time step, and is at time t _n class It is the value of the hidden layer based on and bs are sets of weight matrices and bias vectors, respectively, updated according to backpropagation through a time algorithm. also, represents the Hadamard product. is the standard logistic sigmoid function, is the concatenation operator, is the output activation function, for example SoftMax or Tanh. Equations (1)-(7) transmit information of memory cells at each step.

A single LSTM only captures previous context but does not utilize future context. BiLSTM [46] combines two separate hidden LSTM layers into the same output. BiLSTM is an input sequence Forward hidden sequence in the opposite direction and backward hidden sequence Process until. hidden layer , The output vector of is and It is a connection of and as shown in equations (8)-(11) am.

The output y _t is then used as input to the next hidden layer, etc. Each BiLSTM block in Figure 2 consists of three stacked BiLSTM layers, an L2 normalization layer, and a dropout layer. As can be seen in Figure 4, the output y _t from the lower layer becomes the input to the upper layer. According to an embodiment of the invention, the three BiLSTM layers do not increase the computational load because the time series is not very long. CNNs (before BiLSTM) perform a preprocessing step to learn local features, and the result is a shorter time series with high-level features. Separate BiLSTM subnetworks are used for different modes viz. are trained for

The proposed model includes five concurrent stack CNN-BiLSTM pipelines and one feed-forward neural network. For each CNN-BiLSTM pipeline, the CNN block has one Conv1D layer followed by max pooling. The BiLSTM block has three stacked BiLSTM layers, an L2 regularization layer, and a dropout layer. The model is based on a state-of-the-art fusion of five different temporal data sources, including neuroimaging, neuropsychological batteries, etc.

A CNN subnetwork is applied to extract local features from each time series feature, as shown in Figure 3. Afterwards, the stacked BiLSTM subnetwork is applied to learn the same type of features and temporal relationships within a single time series, as shown in Figure 4. Then, the features learned from the BiLSTM block are input into two dense layers to perform deeper feature learning. The outputs of the five streams are then fused by three dense layers to form more unique and deep features. Additionally, reference data serves as BG to increase the accuracy and reliability of the learning process. These baseline data are static characteristics of the patient, such as demographics and some statistical features extracted from longitudinal time series data. Other unique and deep features are extracted from the baseline data separately using a feedforward neural network. The results of these two feature extraction steps are fused by a series of shared dense layers to learn finer-grained common features for classification and regression tasks. The proposed model simultaneously learns many related tasks, including a multi-class classification problem (i.e., AD progression) and four regression problems (i.e., the most medically sensitive cognitive score related to Alzheimer's disease [7]). This type of information is expected to be very important for doctors to trust the results of DL models. The proposed model uses parameter sharing of the DL network for multi-task learning. As shown in Figure 2, a hidden layer is shared and applied between all tasks. Multitask learning acts as a regularizer and reduces the risk of overfitting. The resulting model is more general and stable than a single task model [47].

The proposed DL framework jointly learns a set of related operations Y based on a set of modes M. M mode of data Considering that it is displayed as, the multitask to learn is It is displayed as . Each jth task is am. Each ^mode It is expressed as, and in each case Is For a set of time steps and univariate time series f, a multivariate time series, am.

For N patients, each patient i is , It is expressed as, and is the label of the first task of the ith example. is the tth test value of the ith case, It is displayed as . The parameters to be optimized are shared parameters ( ) and task-specific parameters ( )am. The parametric hypothesis for each task is And the loss function for each task is am. A general optimization problem is gradient-based multi-objective optimization for task-specific loss, as shown in Equation (12).

silver It is a loss function for each task defined as follows.

The proposed model predicts two types of tasks: classification tasks and regression tasks. Classification and regression tasks according to the proposed model are handled equally with the objective function defined in equation (13). Here m is and It is the number of parameters. Here, the first term is the weighted cross-entropy loss for multi-class classification, the second term is the mean square loss for the four regression tasks, and the last term is the regularization term. T is the number of regression tasks, and are the actual and predicted values of the regression task t for patient i, respectively. is an indicator function, where I(true statement) = 1 and 0 otherwise. W _C is a vector of class weights calculated according to the number of cases in each class. Multi-class classification is based on the SoftMax function, where labels y are K different values. You can have For each input x, the model for Calculate the probability of The output is a K-dimensional vector of K estimated probabilities that sum to 1. In the present invention, the class labels and n clinical scores are used to update the network weights of the convolution and BiLSTM layers in the backpropagation procedure and learn the most relevant features in the dense layer.

According to an embodiment of the present invention, a plurality of types of multi-mode time series data and a series of background data are input, local characteristics and longitudinal characteristics for each mode of the multi-mode time series data are extracted, and the extracted multi-mode time series Preprocessing is performed to improve the data quality of data characteristics.

In the data preprocessing process, missing data is first processed. Because the nature of the dataset is numerical and categorical, missing values were handled according to data type. For the basic static data, we first removed all features with more than 30% missing features. Next, the K-Nearest Neighbors (KNN) algorithm was used to impute missing values, and missing values were replaced using information from other subjects with the same diagnosis. That is, after finding k neighbors, the imputed value was calculated by averaging the values of those neighbors. According to an embodiment of the present invention, k was empirically set to 10 through experiment. For numeric values, the Euclidean distance was also used, and for categorical values, a distance of 0 was taken if the two values were the same, otherwise a distance of 1 was taken. For time series data, all features with more than 30% missing were also removed. All patient cases with missing baseline readings were excluded from the trial. Some important features, such as CSF tau (83%), were missing in more than 30% of the time series but were not missing at baseline. In the present invention, we preferred to collect these characteristics at baseline and consider them as BGs with static data. Important characteristics were determined according to ADNI recommendations and AD diagnosis and progression literature [14]. To handle non-existent time series values, we followed two sequential strategies following ADNI's intuition. First, values that were not applicable to all data categories were filled in according to ADNI procedures. For example, an ADNI 1 patient with CN will not have an MRI scan performed when visiting M18. Therefore, these types of values should not be considered missing or non-random. Many laboratory tests, cognitive tests, and neuroimaging scans are not performed for a specific diagnosis at a specific visit. According to an embodiment of the invention, a precise procedure was followed to fill in these inapplicable values. If the diagnosis did not change, forward filling was used with the previous values. If the diagnosis changed, the value was considered missing. This method is commonly found in the Alzheimer's literature [48]. The second step is to use statistical or ML techniques to determine missing values in the existing data. According to embodiments of the present invention, a medically intuitive and well-known method is used to address this problem. For numerical data, average values according to various classes such as CN, sMCI, pMCI, and AD were used. For categorical features, mode values were used according to patient class. The resulting time series is regular with 6 months between two consecutive visits. As a result, LSTM and CNN models can be applied directly.

The amount of participant data available for both baseline and time series varies. Training ML models using such data directly makes convergence difficult. To ensure that all features in the data have equal importance, the z-score method, i.e. The features were standardized using . here is the participant's original value for feature j and is the normalized value is the average of the features is the standard deviation of the feature. The z-score method converts data to mean and unit standard deviation of 0 and helps remove outliers.

According to an embodiment of the present invention, principal component analysis (PCA) was utilized to reduce the number of features in MRI and PET data. PCA was implemented with variance retained over 91%. Initially, the MRI had 326 features extracted from UCSF using FreeSurfer software [49], and after applying PCA, the number of principal components increased to 110. After adding the 7 manually calculated features, the entire MRI imaging equipment had 117 features. PCA reduced PET features from 288 to 75. The total number of features across all modes was 259: MRI (117), PET (75), cognitive score (9), neuropathology (7), and assessment (51). PCA with the same settings was applied to the 131 calculated statistical features to generate 30 components. As a result, the baseline BG had 158 features (124 ADNI features + 30 PCA components).

Figure 5 is a diagram showing the structure of a multi-mode multi-task learning device for detecting the progression of Alzheimer's disease based on time series data according to an embodiment of the present invention.

The proposed multi-mode multi-task learning device 500 for detecting Alzheimer's disease progression based on time series data includes a feature extraction unit 510, a preprocessing unit 520, a CNN-BiLSTM learning unit 530, and a decision making unit 540. and a judgment unit 550.

The feature extraction unit 510, preprocessing unit 520, CNN-BiLSTM learning unit 530, decision making unit 540, and judgment unit 550 are configured to perform steps 110 to 150 in FIG. 1. It can be.

The feature extraction unit 510 receives a plurality of types of multi-mode time series data and a series of background data and extracts local characteristics and longitudinal characteristics for each mode of the multi-mode time series data.

The preprocessing unit 520 performs preprocessing to improve the data quality of the characteristics of the extracted multi-mode time series data.

The CNN-BiLSTM learning unit 530 learns about the characteristics of preprocessed multi-mode time series data through a stacked CNN-BiLSTM network.

The CNN-BiLSTM learning unit 530 uses CNN using 1D convolution to learn each mode regarding the characteristics of multi-mode time series data, and uses one CNN layer for each mode to learn time series data of multiple tensors. To transform to a new feature space, a 1D convolution is performed separately along the time dimension for all input vectors. 1D convolution is based on the number of filters, and CNN maps the input vector for all univariate time series data to a new feature space, extending it into a feature map for LSTM prediction.

Afterwards, in order to take advantage of the temporal correlation of the multi-mode time series data by receiving the learned features from the CNN, temporal patterns are found in the longitudinal data through a BiLSTM layer, and the cells are analyzed through an LSTM cell including an input gate, a forgetting gate, and an output gate. Controls the updating, maintenance, and deletion of information contained in the state.

The CNN-BiLSTM learning unit 530 performs learning through a stacked CNN-BiLSTM network including multiple simultaneous stack CNN-BiLSTM pipelines and one feedforward neural network. For each CNN-BiLSTM pipeline, the CNN contains one 1D convolutional layer followed by max pooling, and the BiLSTM contains multiple stacked BiLSTM layers, an L2 regularization layer, and a dropout layer. At this time, CNN is applied to extract local features from each multi-mode time series data, and BiLSTM is applied to receive features learned from CNN and learn the same type of features and temporal relationships within single time series data.

The decision-making unit 540 extracts common features in all modes using a series of dense layers for the learned features, and makes a decision to extract features using a series of dense layers for the background data. Perform. The decision-making unit 540 performs deep feature learning by inputting the features learned from BiLSTM into a plurality of dense layers and then fuse them through the plurality of dense layers.

The decision unit 550 uses a set of dense layers to learn task-specific characteristics for the common characteristics of the extracted multi-mode time series data and the characteristics of the background data, and performs classification or regression to finally detect the results.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

<Reference materials>

[1] A. Alberdi, A. Aztiria, A. Basarab, On the early diagnosis of Alzheimer's disease from multimodal signals: a survey, Artificial Intelligence in Medicine 71 (2016) 1-29.

[2] C.L. Masters, K. Beyreuther, Alzheimer's centennial legacy: prospects for rational therapeutic intervention targeting the ab amyloid pathway, Brain 129 (11) (2006) 2823-2839.

[3] R.A. Sperling, P.S. Aisen, L.A. Beckett, D.A. Bennett, S. Craft, A.M. Fagan, T. Iwatsubo, C.R. Jack Jr, J. Kaye, T.J. Montine, et al., Toward defining the preclinical stages of alzheimer's disease: recommendations from the national institute on aging-alzheimer's association workgroups on diagnostic guidelines for alzheimer's disease, Alzheimer's & Dementia 7 (3) (2011) 280- 292.

[4] H. Li, M. Habes, D.A. Wolk, Y. Fan, A deep learning model for early prediction of Alzheimer's disease dementia based on hippocampal mri, 2019, ArXiv abs/ 1904.07282.

[5] S. Qiu, G.H. Chang, M. Panagia, D.M. Gopal, R. Au, V.B. Kolachalama, Fusion of deep learning models of MRI scans, mini-mental state examination, and logical memory test enhances diagnosis of mild cognitive impairment, Alzheimer's & Dementia: Diagnosis, Assessment & Disease Monitoring 10 (2018) 737-749.

[6] G. Lee, K. Nho, B. Kang, K.-A. Sohn, D. Kim, Predicting alzheimer's disease progression using multi-modal deep learning approach, Scientific Reports 9 (1) (2019) 1952.

[7] M. Liu, J. Zhang, E. Adeli, D. Shen, Joint classification and regression via deep multi-task multi-channel learning for Alzheimer's disease diagnosis, IEEE Transactions on Biomedical Engineering 66 (5) (2018) 1195 -1206.

[8] X. Ding, M. Bucholc, H. Wang, D.H. Glass, H. Wang, D.H. Clarke, A.J. Bjourson, C. D. Le Roy, M. O'Kane, G. Prasad, et al., A hybrid computational approach for efficient Alzheimer's disease classification based on heterogeneous data, Scientific Reports 8 (1) (2018) 9774.

[9] D. Zhang, D. Shen, A.D.N. Initiative, et al., Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease, NeuroImage 59 (2) (2012) 895-907.

[10] B. Cheng, M. Liu, D. Zhang, B.C. Munsell, D. Shen, Domain transfer learning for mci conversion prediction, IEEE Transactions on Biomedical Engineering 62 (7) (2015) 1805-1817.

[11] C.-Y. Wee, P.-T. Yap, D. Shen, A.D.N. Initiative, Prediction of Alzheimer's disease and mild cognitive impairment using cortical morphological patterns, Human Brain Mapping 34 (12) (2013) 3411-3425.

[12] P. Moore, T. Lyons, J. Gallacher, A.D.N. Initiative, et al., Random forest prediction of Alzheimer's disease using pairwise selection from time series data, PloS One 14 (2) (2019), e0211558.

[13] E. Moradi, A. Pepe, C. Gaser, H. Huttunen, J. Tohka, A.D.N. Initiative, et al., Machine learning framework for early MRI-based alzheimer's conversion prediction in mci subjects, Neuroimage 104 (2015) 398-412.

[14] M.W. Weiner, D.P. Veitch, P.S. Aisen, L.A. Beckett, N.J. Cairns, R.C. Green, D. Harvey, C.R. Jack Jr, W. Jagust, J.C. Morris, et al., Recent publications from the Alzheimer's disease neuroimaging initiative: Reviewing progress toward improved ad clinical trials, Alzheimer's & Dementia 13 (4) (2017) e1-e85.

[15] K. Ito, B. Corrigan, Q. Zhao, J. French, R. Miller, H. Soares, E. Katz, T. Nicholas, B. Billing, R. Anziano, et al., Disease progression model for cognitive deterioration from Alzheimer's disease neuroimaging initiative database, Alzheimer's & Dementia 7 (2) (2011) 151-160.

[16] T. Tong, Q. Gao, R. Guerrero, C. Ledig, L. Chen, D. Rueckert, A.D.N. Initiative, et al., A novel grading biomarker for the prediction of conversion from mild cognitive impairment to Alzheimer's disease, IEEE Transactions on Biomedical Engineering 64 (1) (2016) 155-165.

[17] L. Nie, L. Zhang, L. Meng, X. Song, X. Chang, X. Li, Modeling disease progression via multisource multitask learners: a case study with Alzheimer's disease, IEEE Transactions on Neural Networks and Learning Systems 28 (7) (2016) 1508-1519.

[18] P.S. Pillai, T.-Y. Leong, Fusing heterogeneous data for Alzheimer's disease classification, Stud. Health Technology. Inform.

[19] M. Ewers, C. Walsh, J.Q. Trojanowski, L.M. Shaw, R.C. Petersen, C.R. Jack Jr, H.H. Feldman, A.L. Bokde, G.E. Alexander, P. Scheltens, et al., Prediction of conversion from mild cognitive impairment to Alzheimer's disease dementia based upon biomarkers and neuropsychological test performance, Neurobiology of Aging 33 (7) (2012) 1203-1214.

[20] K. Li, R. O'Brien, M. Lutz, S. Luo, A.D.N. Initiative, et al., A prognostic model of Alzheimer's disease relying on multiple longitudinal measures and time-toevent data, Alzheimer's & Dementia 14 (5) (2018) 644-651.

[21] F. Liu, L. Zhou, C. Shen, J. Yin, Multiple kernel learning in the primal for multimodal Alzheimer's disease classification, IEEE Journal of Biomedical and Health Informatics 18 (3) (2013) 984-990.

[22] S. Duchesne, A. Caroli, C. Geroldi, D.L. Collins, G.B. Frisoni, Relating one-year cognitive change in mild cognitive impairment to baseline MRI features, Neuroimage 47 (4) (2009) 1363-1370.

[23] D. Ramachandram, G.W. Taylor, Deep multimodal learning: a survey on recent advances and trends, IEEE Signal Processing Magazine 34 (6) (2017) 96-108.

[24] Y. Wang, Y. Fan, P. Bhatt, C. Davatzikos, High-dimensional pattern regression using machine learning: from medical images to continuous clinical variables, Neuroimage 50 (4) (2010) 1519-1535.

[25] J. Zhou, J. Liu, V.A. Narayan, J. Ye, A.D.N. Initiative, et al., Modeling disease progression via multi-task learning, NeuroImage 78 (2013) 233-248.

[26] Q. Liao, Y. Ding, Z.L. Jiang, X. Wang, C. Zhang, Q. Zhang, Multi-task deep convolutional neural network for cancer diagnosis, Neurocomputing 348 (2019) 66-73.

[27] W. Liu, B. Zhang, Z. Zhang, X.-H. Zhou, Joint modeling of transitional patterns of Alzheimer's disease, PloS One 8 (9) (2013), e75487.

[28] L. Huang, Y. Jin, Y. Gao, K.-H. Thung, D. Shen, A.D.N. Initiative, et al., Longitudinal clinical score prediction in Alzheimer's disease with soft-split sparse regression based random forest, Neurobiology of Aging 46 (2016) 180- 191.

[29] H. Harutyunyan, H. Khachatrian, D. C. Kale, G. Ver Steeg, A. Galstyan, Multitask learning and benchmarking with clinical time series data, Scientific data 6 (1) (2019) 96.

[30] H. Suresh, N. Hunt, A. Johnson, L. A. Celi, P. Szolovits, M. Ghassemi, Clinical intervention prediction and understanding using deep networks, arXiv preprint arXiv:1705.08498.

[31] C. Tian, J. Ma, C. Zhang, P. Zhan, A deep neural network model for short-term load forecast based on long short-term memory network and convolutional neural network, Energies 11 (12) (2018) ) 3493.

[32] D. Lu, K. Popuri, G.W. Ding, R. Balachandar, M.F. Beg, Multimodal and multiscale deep neural networks for the early diagnosis of Alzheimer's disease using structural mr and fdg-pet images, Scientific Reports 8 (1) (2018) 5697.

[33] M. Liu, D. Cheng, K. Wang, Y. Wang, A.D.N. Initiative, et al., Multi-modality cascaded convolutional neural networks for Alzheimer's disease diagnosis, Neuroinformatics 16 (3-4) (2018) 295-308.

[34] E. Goceri, Formulas behind deep learning success, in: Int. Conf. Appl. Anal. Math. Model, 2018.

[35] E. Goceri, Diagnosis of Alzheimer's disease with Sobolev gradient-based optimization and 3d convolutional neural network, International journal for numerical methods in biomedical engineering 35 (7) (2019), e3225.

[36] E. Goceri, Challenges and recent solutions for image segmentation in the era of deep learning, 2019, pp. 1-6.

[37] E. Goceri, Capsnet topology to classify tumors from brain images and comparative evaluation, IET Image Processing 14 (2020) 882-889(7).

[38] H. Choi, K.H. Jin, A.D.N. Initiative, et al., Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging, Behavioral Brain Research 344 (2018) 103-109.

[39] S. Spasov, L. Passamonti, A. Duggento, P. Lio, N. Toschi, A.D.N. Initiative, et al., A parameter-efficient deep learning approach to predict conversion from mild cognitive impairment to Alzheimer's disease, Neuroimage 189 (2019) 276- 287.

[40] S. Hochreiter, J. Schmidhuber, Long short-term memory, Neural Computation 9 (8) (1997) 1735-1780.

[41] A. Yousif, Z. Niu, J. Chambua, Z.Y. Khan, Multi-task learning model based on recurrent convolutional neural networks for citation sentiment and purpose classification, Neurocomputing 335 (2019) 195-205.

[42] Z. Hu, Z. Zhang, H. Yang, Q. Chen, R. Zhu, D. Zuo, Predicting the quality of online health expert question-answering services with temporal features in a deep learning framework, Neurocomputing 275 ( 2018) 2769-2782.

[43] R. Cui, M. Liu, A.D.N. Initiative, et al., Rnn-based longitudinal analysis for diagnosis of Alzheimer's disease, Computerized Medical Imaging and Graphics 73 (2019) 1-10.

[44] N. Amoroso, D. Diacono, A. Fanizzi, M. La Rocca, A. Monaco, A. Lombardi, C. Guaragnella, R. Bellotti, S. Tangaro, A.D.N. Initiative, et al., Deep learning reveals Alzheimer's disease onset in mci subjects: results from an international challenge, Journal of Neuroscience Methods 302 (2018) 3-9.

[45] N.P. Oxtoby, D.C. Alexander, Imaging plus x: multimodal models of neurodegenerative disease, Current Opinion in Neurology 30 (4) (2017) 371.

[46] M. Schuster, K.K. Paliwal, Bidirectional recurrent neural networks, IEEE Transactions on Signal Processing 45 (11) (1997) 2673-2681.

[47] S. Ruder, An overview of multi-task learning in deep neural networks, arXiv preprint arXiv:1706.05098.

[48] H. Runtti, J. Mattila, M. van Gils, J. Koikkalainen, H. Soininen, J. Lϕtjϕnen, A.D.N. Initiative, et al., Quantitative evaluation of disease progression in a longitudinal mild cognitive impairment cohort, Journal of Alzheimer's Disease 39 (1) (2014) 49-61.

[49] M. Reuter, N.J. Schumansky, H.D. Rosas, B. Fischl, Within-subject template estimation for unbiased longitudinal image analysis, Neuroimage 61 (4) (2012) 1402-1418.

Claims (8)

A step of receiving a plurality of types of multi-mode time series data and a series of background data through a feature extraction unit and extracting local characteristics and longitudinal characteristics for each mode of the multi-mode time series data;
Performing pre-processing to improve data quality of the characteristics of the extracted multi-mode time series data through a pre-processing unit;
Learning about the characteristics of the preprocessed multi-mode time series data through a CNN-BiLSTM learning unit through a stacked CNN-BiLSTM network;
Through the decision-making unit, a series of dense layers is used for the learned characteristics to extract common characteristics common to each mode of the multi-mode time series data, and a series of A decision-making step to extract features using dense layers; and
A step of learning task-specific characteristics using a dense layer set for the common characteristics of the extracted multi-mode time series data and characteristics of the background data through the judgment unit, and finally detecting the results by classifying or regressing.
A multi-mode multi-task learning method including. According to paragraph 1,
The step of learning through a stacked CNN-BiLSTM network about the characteristics of the multi-mode time series data preprocessed through the CNN-BiLSTM learning unit,
We use a CNN using 1D convolution to learn each mode about the characteristics of multimode time series data, and use one CNN layer for each mode to transform the time series data of multiple tensors into a new feature space. Convolution is performed separately along the time dimension for all input vectors, 1D convolution is based on the number of filters, and CNN maps the input vectors for all univariate time series data to a new feature space into feature maps for LSTM prediction. expanding
Multimodal multitask learning method. According to paragraph 2,
The step of learning through a stacked CNN-BiLSTM network about the characteristics of the preprocessed multi-mode time series data through the CNN-BiLSTM learning unit,
In order to take advantage of the temporal correlation of multi-mode time series data by receiving the learned features from the CNN, temporal patterns are found in the longitudinal data through a BiLSTM layer, and the cell state is determined through an LSTM cell including an input gate, a forget gate, and an output gate. Controlling the updating, maintenance and deletion of the information contained therein.
Multimodal multitask learning method. According to paragraph 3,
The step of learning through a stacked CNN-BiLSTM network about the characteristics of the preprocessed multi-mode time series data through the CNN-BiLSTM learning unit,
Perform training through a stacked CNN-BiLSTM network containing multiple concurrently stacked CNN-BiLSTM pipelines and one feedforward neural network,
For each CNN-BiLSTM pipeline, the CNN contains one 1D convolutional layer followed by max pooling, the BiLSTM contains multiple stacked BiLSTM layers, an L2 regularization layer and a dropout layer;
CNN is applied to extract local features from each multi-mode time series data, and BiLSTM is applied to receive features learned from CNN as input and learn the same type of features and temporal relationships within single time series data.
The decision-making steps for extracting common features in all modes using a series of dense layers for the learned features and extracting features using a series of dense layers for the background data are:
The features learned from BiLSTM are input into multiple dense layers to perform deep feature learning, and then fused by multiple dense layers.
Multimodal multitask learning method. a feature extraction unit that receives a plurality of types of multi-mode time series data and a series of background data and extracts local characteristics and longitudinal characteristics for each mode of the multi-mode time series data;
A pre-processing unit that performs pre-processing to improve the data quality of the characteristics of the extracted multi-mode time series data;
CNN-BiLSTM learning unit that learns about the characteristics of preprocessed multi-mode time series data through a stacked CNN-BiLSTM network;
A series of dense layers is used for learned characteristics to extract common characteristics common to each mode of multimode time series data, and a series of dense layers is used for background data other than multimode time series data. a decision-making unit that extracts characteristics; and
A judgment unit that learns task-specific characteristics using a set of dense layers for the common characteristics of the extracted multi-mode time series data and the characteristics of the background data and finally detects the results by classifying or regressing.
A multi-mode multi-task learning device including. According to clause 5,
CNN-BiLSTM learning department,
We use a CNN using 1D convolution to learn each mode about the characteristics of multimode time series data, and use one CNN layer for each mode to transform the time series data of multiple tensors into a new feature space. Convolution is performed separately along the time dimension for all input vectors, 1D convolution is based on the number of filters, CNN maps the input vectors for all univariate time series data to a new feature space into feature maps for LSTM prediction. expanding
Multi-mode multi-task learning device. According to clause 6,
CNN-BiLSTM learning department,
In order to take advantage of the temporal correlation of multi-mode time series data by receiving the learned features from the CNN, temporal patterns are found in the longitudinal data through a BiLSTM layer, and the cell state is determined through an LSTM cell including an input gate, a forget gate, and an output gate. Controlling the updating, maintenance and deletion of the information contained therein.
Multi-mode multi-task learning device. In clause 7,
CNN-BiLSTM learning department,
Perform training through a stacked CNN-BiLSTM network containing multiple concurrently stacked CNN-BiLSTM pipelines and one feedforward neural network,
For each CNN-BiLSTM pipeline, the CNN contains one 1D convolutional layer followed by max pooling, the BiLSTM contains multiple stacked BiLSTM layers, an L2 regularization layer and a dropout layer;
CNN is applied to extract local features from each multi-mode time series data, and BiLSTM is applied to receive features learned from CNN as input and learn the same type of features and temporal relationships within single time series data.
The decision-making department,
The features learned from BiLSTM are input into multiple dense layers to perform deep feature learning, and then fused by multiple dense layers.
Multi-mode multi-task learning device.