KR20230070084A - Providing method of diagnostic information for classification of advanced stages of Alzheimer's disease using 3D convolutional neural network - Google Patents

Abstract

The present invention relates to a method of providing diagnostic information for classification of advanced stages of alzheimer's disease through image analysis using a 3D convolution neural network. The method includes the steps of: acquiring a magnetic resonance imaging (MRI) or positron emission tomography (PET) image; extracting a feature; visualizing the feature; applying a final FCL weight; and classifying advanced stages of alzheimer's disease.

Description

Method for providing diagnostic information for classification of advanced stages of Alzheimer's disease using 3D convolutional neural network {Providing method of diagnostic information for classification of advanced stages of Alzheimer's disease using 3D convolutional neural network}

The present invention relates to a method for providing diagnostic information for classifying advanced stages of Alzheimer's disease using a 3D convolutional neural network.

Molecular and neurological causes of Alzheimer's disease (AD) are known to be caused by damage to synapses, which are brain nerve cell connection areas, and beta-amyloid proteins and tau proteins around the synaptic areas This is caused by excessive deposition. Beta-amyloid disrupts communication between neurons at the synapse, causing neuronal death, while tau protein blocks the supply of nutrients and other essential molecules inside neurons. As Alzheimer's disease progresses, the brain shrinks dramatically due to cell loss, inflammation, and neuron death, which affects brain function and causes problems such as memory loss. Alterations in anatomical brain structures associated with Alzheimer's disease include enlargement of the ventricles, shrinkage of the hippocampus, changes in cortical thickness, cerebrospinal fluid and other cerebral regions including white matter and gray matter brain tissue. It can be visualized through various medical imaging techniques such as (MRI), positron emission tomography (PET), and computed tomography (CT).

Mild cognitive impairment (MCI) is considered an early stage of Alzheimer's disease as an intermediate stage between normal aging and dementia, and maintains the status with progressive mild cognitive impairment (pMCI), which progresses to Alzheimer's disease within a few years. stable mild cognitive impairment (sMCI). Since MCI has a relatively high possibility of progressing to dementia, it is clinically very important to identify it early and potentially delay or prevent the transition from MCI to Alzheimer's disease.

On the other hand, deep learning is a field of machine learning, which is an artificial intelligence technology, and extracts necessary information from a large amount of data by combining various nonlinear transform methods. Recently, many studies have been conducted on how to diagnose diseases early using deep learning and how to increase the accuracy of diagnosis, and active research is being conducted in the field of medical imaging. Image processing can distinguish between normal and MCI by finding a distinctive pattern of image features from images such as MRI. When the MRI is converted into an image at the magnetic resonance frequency, it represents a pixel value for each structure, and these pixels are assigned to a class. Based on the features extracted from these brain image pixels, Alzheimer's disease can be distinguished. Major markers of Alzheimer's disease variation in brain anatomy include ventricular size, hippocampal shape, cortical thickness, and brain volume. Network identification features trained based on brain phenotypes reflected in images like this can help identify images prone to Alzheimer's disease.

A convolutional neural network (CNN) is an algorithm that shows excellent performance in speech recognition and image recognition. It is useful for finding patterns for recognizing images, self-learns features from data, and uses patterns to recognize images. to classify Unlike the single-node multiplication of artificial neural networks (ANNs), CNNs have additional feature investigators such as convolutional filter elements (weights), pooling forms, and activation features. CNN-based topologies include residual (Resnet50, ResNet101), recurrent (RCNN), inception (GoogLeNet), encoder-decoder (U-net), etc. All topologies are encoder units, that is, for feature generation. It consists of the basic units convolution-normalization-activation-pooling.

2D CNNs are easily misleading because target domain-trained CNNs can only give probability scores for each trained class, and even slight changes in pixels can degrade predictions. Training a CNN using 3D whole-brain structures can improve performance over 2D images due to the deeper architecture, but a deeper architecture means more parameters to train (weights of the layers) and at the same time larger and better Training materials are required. 2D or 3D CNNs follow a general feature extraction pattern, where the general features are basically CNN features, called ready-made CNN features, which are image features extracted from multiple convolutional layers with the weights (decimal numbers) of the network learned by applying various activation functions. can suggest Usually, the final feature weights of the FCL are graphed to determine the CNN's performance, which means that a well-separated class-based segment graph is usually a well-trained classifier.

In 2D CNN, it is important to select an appropriate slice and its direction as a training input, and in many literatures, best scan or best multiple slices are proposed for efficient performance, which makes the slice selection process somewhat ambiguous and impractical. However, the widely used models AlexNet, ResNet, GoogLeNet, and ZNet are all 2D-based architectures. Since focusing only on a limited scan or orientation may miss some important information, it is recommended to use the whole brain volume, where three-dimensional pixel values are provided, i.e., pixel values for the x, y, and z dimensions of a planar geometry. Previous studies have demonstrated poor classification performance of 2D CNNs when trained with a smaller number of MRI images, and the selection of slices is still ambiguous. Due to the dimensionality constraints of 2D CNNs, the architecture needs to be made deeper and larger to accommodate thousands of images per class. Therefore, 3D CNN may be suitable for generalization of MRI classification. Using 3D input requires fewer pre-processing steps such as slice correction, selection, and extraction, reducing manual processing steps and making the system more automated.

3D CNNs are multipatch-based architectures trained primarily on top patches or CNN ensembles. Whereas the best patch selects a single region of the brain based on a recommended region of interest (ROI) or passively assists in anatomical regions such as the hippocampus or amygdala atrophy, multiple patches trained for CNN ensembles Several CNNs in the ROI are trained separately for each region and then perform feature concatenation in the last fully connected layer (FCL) before classification. The reason why only limited/selected/informative pixels are used as feed in 3D CNN is to improve the quality of information and GPU memory constraints. Non-discriminant parts constitute features at a low level, but may not support cohort classification, so using a whole-brain model can result in information duplication, and selecting ROI patches or simply the best region makes the system semi-automatic, reducing the risk of automatic feature extraction. Since the true meaning is not applied, a technology that can implement classification more simply and accurately is required.

Huang et al. proposed a multimodal 3D CNN using a hippocampal ROI on MRI and a hippocampal and/or cortical ROI on PET, training a VGG architecture based CNN on the ROI-based MRI and PET, and on the final FCL before final classification. connected. As another example of multi-mode based 3D CNN, Liu et al. Instead of concatenating the final FCL, concatenation was made in the convolutional layers of each CNN (trained using PET and MRI patches) for sequential convolution until features were extracted from the FCL, and features were extracted using a 3D CNN. We used another 2D CNN to combine multimodal features for task-specific classification with T1-MRI and FDG-PET-based cascaded CNNs. In 2016 Asl et al. proposed a deeply supervised and adaptable 3D CNN (DSA-3D-CNN) trained on structural MRI images, which could predict AD, MCI and cognitively normal (CN). Similarly, Payan and Montana classified MRI using data set segmentation via a sparse auto encoder (SAE) patch-based 3D CNN, and used volumetric or convolutional auto encoder (CAE) for AD and CN classification. )-based 3D CNN was used to perform 5-fold cross-validations (CV), and supervised transfer learning was performed for sMCI and pMCI classification.

CNN learns the basic shapes, borders, edges and patterns of objects, and can choose the best architecture that is easy to train and has good performance, depending on the performance results, training time, validation period, reliability of prediction, generalizability and other factors.

According to recent studies, unlike manually supervised learning algorithms, CNNs can extract convenient features directly from raw images, and can find key points and features in object detection tasks for natural images. These features were explored in a region-based convolutional neural network (RCNN) for region-based detection in 2D images. Other tasks of segmentation using CNN have been proposed that CNNs can learn useful features without labeling the voxels themselves, which is not a prerequisite for classification tasks because the segmentation results themselves do not contain the information needed for classification, which suggests that We support supporting the general feature extraction properties of CNNs. However, going deeper requires extracting more meaningful features from the training dataset to perform the relevant task of classification or segmentation, which results in more feature vectors with more layers and a larger pool of features to extract. This can help judge which of the good features is the best, but depth doesn't necessarily mean a deep learning model. For example, He et al. ResNet shows that deeper networks of 1,202 layers show no significant improvement with aggressive depth compared to 50, 101, and 152 convolutional layers. Greater depth can increase the tendency to overfit, which can make model building difficult.

In the present invention, a CNN was implemented considering the performance of different architectures, the role of hyperparameters, data selection, and data set size for optimal network design, and a divergent architecture 'divNET' supporting MRI and PET was developed.

1. Korea Patent Registration No. 2125127 2. Korea Patent Registration No. 2313656

An object of the present invention is to provide a diagnostic information providing method for classifying Alzheimer's disease or mild cognitive impairment from a normal person by applying magnetic resonance imaging (MRI) and positron emission tomography (PET) images to a 3D convolutional neural network.

The present invention comprises: 1) obtaining a magnetic resonance imaging (MRI) or positron emission tomography (PET) image; 2) extracting activated features of a convolution layer from a single image; 3) Visualizing the activated features of the fully connected layer (FCL) on the entire test set using T-distributed stochastic embedding (T-SNE) projections; 4) applying the last FCL weight of the trained network for direct visualization; and 5) classifying the stage of progression of Alzheimer's disease; Provides a method for providing diagnostic information for classification of advanced stages of Alzheimer's disease, including.

According to one embodiment of the present invention, step 2) includes the filter feature (F _l ) = the weights of filters in layer l of size k×k×k×f for the trained network N of layer l, Here, k×k×k is the size of the filter, f is the number of filters in layer l, and step 2) includes the initial activated feature (Al ₎ = conv (F _l , I), where the initial Activated features (A _l ) include batch normalization and max pooling steps to downsample the size and pass through the activation layer, step 2) above for visualization = A of size k×k×k _lmax is scaled to 64x64x64 and each slice is displayed individually in the 2D domain.

According to another embodiment of the present invention, step 3) includes FCL feature (FCL _l ) = weights of FCL of size T×S for trained network N of layer l, where T is the number of test objects. is a number, S = the size of O × I, O and I represent the output and input of the FCL in the first layer, respectively, step 3) includes FCL _l-tsne = T-SNE (FCL _l ), Here, T-SNE is to perform T-SNE to find a feature matrix of T×2 while performing feature reduction in N dimensions, and step 3) above is performed by FCL _l-tsne to visualize the identification pattern. It is to display on the xy plane about the progress of

According to another embodiment of the present invention, step 4) above includes FCL _l = weights of FCL l of size _O ×n for a network N with l as the final FCL and n number of classification categories, where O is the output size of the second-to-last FCL, n is the output size of the final FCL in layer l, which is the number of progressive stages of Alzheimer's disease, and step 4) above FCL _l is simply displayed as a linear graph on the XY plane It is an O × n matrix that becomes

According to another embodiment of the present invention, the advanced stage of Alzheimer's disease is one selected from the group consisting of cognitive normal, mild cognitive impairment, and Alzheimer's disease.

The method for providing diagnostic information for classifying Alzheimer's disease or mild cognitive impairment from a normal person according to the present invention has an effect of providing an optimal diagnostic means capable of early diagnosis of a patient's condition using only an image scan.

Figure 1 shows MRI and PET scans. (a) AD prone MRI (AD prone MRI); (b) Healthy MRI; (c) MCI affected MRI; (d) AD prone PET; (e) Healthy PET; (f) MCI affected PET.
Figure 2 shows (a) the workflow of the experiment and (b) a picture of the proposed 3D CNN architecture for MRI/PET classification based on the divergent region of reception, called 'divNet'.
3 shows pseudo-codes 1, 2 and 3.
4 (a) to (f) show graphs of training and validation loss (y-axis) at each iteration (x-axis) of 100 epochs for L1 to L6 convolutions.
5 shows training and test results for the divergent architecture by changing the number of layers specified in the parameter column. Here, C[WWWN, S] denotes a convolutional layer with N filters of size W in each dimension, moving with Stride S and N biases. TC [WWWN, S] represents a Transposed Convolutional Layer with N filters of W size in each dimension, and moves with Stride S and N bias. BN[N] denotes batch normalization using offsets of N and N scale values as learnable parameters, and R denotes ReLU activation. M[WW WS] denotes the maximum pooling of the W kernel with A stride S, FC[OI] denotes the fully connected layer with input I and output O, CT, D, S and C are Concatenation, Dropout, and C respectively. Softmax and Classification Layer are shown, and the training pattern is shown in FIG.
6 shows test results using various types of architectures. Parameters are indexed as in FIG. 5 .
Figure 7 shows a convolutional layer visualization of maximally activated features using a single MRI scan with the original size scaled to [64 64 64] using pseudocode 1. The network used is L4 divergent. Each convolutional layer for a typical MRI in the categories AD, CN and MCI.
8 shows the classification performance results of BASELINE_MRI data for the hyperparameters investigated in the L4 branch architecture.
Figure 9 shows classification results for different data set sizes using L4 divergence.
10 shows FCL feature visualization results using t-SNE 2D feature projection for various architectures during testing. Colored dots represent single MRI scan features in the test set of the first three FCLs, namely FC1, FC2 and FC3. This feature is visualized by starting to show class domain attributes in the FCL and clusters of the same color starting to form. Based on visual inspection, we confirmed that the diverging architecture-based features, such as (d)–(f), are better clustered and separated than the other features. On the other hand, the separation is poor in the case of U-net-based architecture as shown in (j)-(l). Here, the educational environment and educational materials used for education were all the same. The generated model is detailed in FIG. 6 . X-axis and Y-axis represent the 1-dimensional and 2-dimensional values obtained from the t-SNE 2D projection, respectively.
11 shows the feature visualization results using t-SNE 2D projection on L4 divNet for 296 test images of BASELINE_MRI data. Each color point represents a feature of a single MRI of an indexed class. It starts with a 1st order convolution to a 4th order convolution (i.e. from (a) to (d)). Similar groups of features begin to separate and can be clearly visualized in the first FCL (i.e. FC1, (e)). It continues until the last FCL (i.e. FC4), where a few colored dots are found in the false cluster (near the green CN group and a few in the blue MCI group). This region of overlap can be due to possible false positive or false negative predictions that can lead to errors in test set predictions. X-axis and Y-axis represent the 1-dimensional and 2-dimensional values obtained from the t-SNE 2D projection, respectively.
12 shows training graphs plotted against training loss and validation loss on the Y-axis and corresponding number of iterations on the X-axis. The more iterations, the longer the epoch. The training plot of the convergent architecture has a validation loss much higher than the training loss. This can lead to performance degradation, which is similar to the case of equivalent architectures. However, validation loss is significantly reduced in a divergent architecture. thus making it the optimal choice. Here, the educational materials and educational environment are the same in all three cases.
13 shows the final FCL weight values plotted directly on the Y-axis for the three target domains individually for each architecture tested using pseudocode 3. The X-axis extends from 0-100 for the first 3 graphs whereas it extends from 0-512 in (d). The first three graphs have 100 parameters before generating the last three outputs for the softmax classifier, whereas the U-net has 512 parameters.
Figure 14 shows the generality test results with a completely different data set not included in the training and obtained from another ADNI project.

Hereinafter, the present invention will be described in more detail through examples. Since these examples are intended to illustrate the present invention only, the scope of the present invention is not to be construed as being limited to these examples.

Example 1 Data as fuel for CNN, but how big should our data be?

The ImageNet dataset implemented in Alexnet suggests that better data yields better results. To support this theory, artificial datasets have also been created with various augmentation techniques, and the results are supported by the use of a wide range of synthetic MRIs for improved performance in segmentation and classification tasks. In the case of ImageNet, it is divided into 1,000 classes with about 8,000 images in each class. That means more classes with more unique images, and similarly for other datasets like CIFAR101, Caltech, etc., where data works like oil for AI. On the other hand, in the case of medical images, the task is more difficult with image-based features, especially MRI or PET (Fig. 1) of AD versus MCI or MCI versus NC can hardly detect atrophy patterns. To solve this problem, experiments were conducted with data sets of various sizes for MRI and PET tests.

<Example 2> Visualization features: What did CNN extract and learn?

A generalized CNN follows the reduction of features from the input to the final classification layer. In the present invention, the input to the CNN is a 3D MRI obtained in NIfTi (Neuroimaging Informatics Technology Initiative) format with a .nii extension. It can be resized from its original size of 256×256×256 or 256×256×170 to 64×64×64 by reading the input using MATLAB’s built-in niftiread function. After downsampling several times using the max-pool operation for each convolutional layer, it is reduced to 1,728 for the first FCL. To reduce overfitting, we used other conceding FCLs such as dropout to make the final output 100 features per class, which is the softmax layer input. This idea of mapping a target domain using multiple FCLs is often referred to as target domain coordination, which is the basis for transfer learning while using pretrained networks. Features activated in the initial convolutional layer can detect pixel changes based on attributes such as lines, edges, and color in a small window filter. These edge-based features are passed through the middle layers of the CNN, combined into a number of filters, and the weights (initially kept at random weights or initialized using Xavier, He, Gaussian) are stochastic gradient descent (SDG) or Adam and are updated using backpropagation training, such as specific optimization paths. These middle layers detect the active part of the image, while the final layer learns the distinguishing features in shape and pattern between the target areas. When training reaches convergence, no more weight changes occur and training stops when training accuracy reaches maximum, whereby the network has been learned and is a generic feature extractor similar to existing algorithms that generate features. The generated feature is the discriminating feature used to differentiate the classes. In the present invention, several 3D filters are used, each layer providing a 4D output, that is, one 3D feature map per filter (FIG. 2). Convolution of an image using these filters creates a feature map that detects the presence of that feature in the image. This property of CNN is the essence of automatic feature extraction and can help automatic CAD (Computer Aided Design) systems.

It is difficult to predict which features a CNN can learn without training, and thus analyzing the features is a tedious task. This is because a single network can contain millions of parameters, and the final convergence value of each filter cannot be mathematically predicted without training. Therefore, whenever we train a CNN, we need to examine the learned features. After training, the CNN is loaded with filter weights, which are then used to make predictions with test images. It was convoluted at each layer to obtain different results for MRI, and the trained network is used to obtain the features described in pseudo-codes 1, 2, and 3 (Fig. 3).

< Example 3> Parameter initialization

Assuming that MRI and PET images have a 64×64×64 matrix represented by I (ie, I=[I _xiyizi ] _{i=1 to 64} ), a total of 262,144 gray-scale values are generated. These values are called voxels, and each voxel has 3D values of x, y, and z coordinates. MRI was simply represented as a cube (Fig. 1). Equation 1 below represents the first convolution of the first layer.

와

는 초기화 알고리즘을 사용하는 N번째 필터에서 첫 번째 컨볼루션 커널의 초기 바이어스와 가중치를 의미하며,

는 요소별 곱셈을 의미한다. 컨볼루션 연산 윈도우(winodw)는 스트라이드(stride) 크기에 따라 연속적으로 움직이며, 상기 수학식 1은 하기 수학식 2와 같이 더욱 간단하게 나타낼 수 있다. 3D 컨볼루션 필터의 각 노드에 대해 :remind

and

denotes the initial bias and weight of the first convolution kernel in the N -th filter using the initialization algorithm,

means element-by-element multiplication. The convolution operation window (winodw) continuously moves according to the size of the stride, and Equation 1 can be more simply expressed as Equation 2 below. For each node of the 3D convolution filter:

는 입력(input),

는 레이어 l에서 k번째 뉴런의 바이어스(bias),

는 레이어 l-1에서 i번째 뉴런의 출력(output),

는 레이어 l-1의 i번째 뉴런부터 레이어 l의 k번째 뉴런까지의 커널(가중치)을 의미한다. conv _.3은 [3×3×3] 커널 크기의 요소별 곱을 나타낸다. 최초의 컨볼루션 레이어의 경우, 입력

는 동일한 크기의 윈도우(window)로 스캔되는 이미지 픽셀 값(정규화될 수 있음)의 3×3×3 행렬이다.Here, conv _.3 is a regular 3D convolution without zero padding on the boundary,

is the input,

is the bias of the kth neuron in layer l ,

is the output of the ith neuron in layer l-1 ,

means the kernel (weight) from the i -th neuron of layer l-1 to the k -th neuron of layer l . conv _.3 denotes the element-wise product of the [3×3×3] kernel size. For the first convolutional layer, the input

is a 3x3x3 matrix of image pixel values (which can be normalized) scanned over equally sized windows.

N-dimensional convolution for discrete N-dimensional variables A and B when expressed in matrix or discrete form can be defined as in Equation 3 below.

Each k _i is executed for all valid values of A and B, and the 3D convolution can be executed as follows. The layer moves the filter vertically and horizontally along the input, convolves the input, computes the dot product of the weight and the input, and then adds a bias term. As the filter moves along the input, it uses the same set of weights and the same bias for convolution. Thus, a feature map is formed.

는 t번째 반복에 대한 l번째 컨볼루션 계층의 가중치를 나타내고, E는 크기 N의 미니 배치에 대한 비용 함수(비용 함수를 최소화하기 위해 역전파를 이용하여 업데이트 됨)를 나타낸다.During optimization in the SDG algorithm, filter weights are repeatedly updated as shown in Equations 4 and 5 below. here

denotes the weight of the lth convolutional layer for the tth iteration, and E denotes the cost function for a mini-batch of size N (updated using backpropagation to minimize the cost function).

α _l is the learning rate for the l ^th layer, m is the momentum due to the previous weight update in the current iteration, and γ is the scheduling ratio that reduces the learning rate for the completion of each epoch. When α _l is 0, it depends on the value of l , 1: all layers of l have the same weights, and the weights are transferred in the final version of the trained model.

< Example 2> Parameter Learning

)에서 학습 데이터(t _i ))의 편차 MSE 값을 더한 값이다. 여기서 상기 윗첨자 L은 최종 레이어의 출력을 나타낸다. 오차(E)를 기반으로 역전파(BP)를 수행하여 하기 수학식 7과 같이 파라미터에 대한 가중치를 업데이트 한다.The error in Equation 6 is the mean square error, and each sample (predicted value (

) to the value of the deviation MSE of the training data ( t _i )). Here, the superscript L represents the output of the final layer. Backpropagation (BP) is performed based on the error ( E ) to update the weight for the parameter as shown in Equation 7 below.

필터 k는 l번째 레이어의 필터 수이며, 이전 레이어 l+1의 가중치는 역전파 동안 l번째 레이어 출력

를 제공한다. 바이어스(bias)는 하기 수학식 8로 업데이트된다.here, above

filter k is the number of filters in the lth layer, and the weight of the previous layer l+1 is the output of the lth layer during backpropagation

provides The bias is updated to Equation 8 below.

As a result, it is written for the entire length of the 1 to 1 +1 layer, and thus, to obtain y in the 1 th layer, N filters of the 1 + 1 layer can be expressed by Equation 9 below.

의 기울기를 역전파하고 배치 정규화(batch normalization, BN) 변환으로 파라미터에 대한 기울기를 계산해야 한다.Error during training through this transformation

We need to backpropagate the gradient of , and calculate the gradient for the parameters with a batch normalization (BN) transform.

All experiments were performed using MATLAB R2019a academic software on Windows 10 OS. The network model was trained on a 24GB NVIDIA GeForce RTX 2070 GPU and tested on a 32GB Intel®Core ^TM i5-9600K CPU @ 3.70 GHz.

< Example 4> Test on different CNNs

To define the optimal number of layers for the input of a 64×64×64 3D scan, we tested on the initial layer of a single encoder (i.e. convolution-batch normalization-ReLU-maxpooling) and specified it as the L1 layer. Similarly, encoder blocks were further implemented successively in the L2, L3, L4, L5 and L6 layers. The final feature size of the 6th convolution at L6 is [2 2 2] for each of the 64 filters, which means that the filter kernel is 2 pixels long for each filter. Figure 5 shows classification results for this layer-wise CNN, and Figure 6 shows classification results using four different architectures according to the window size of the reception area, that is, the convolution kernel. Similarly, training and validation graphs were studied to observe how architecture affects learning and helps to better understand the convergence process of each CNN. Accordingly, in order to understand the features extracted from each convolutional layer, a single MRI was passed in each target domain, and the features were observed as shown in FIG. 7 . In microscopic observation, we were able to find differences in lines, edges, intensities and other patterns based on class domains. Also, since the FCL layer was visualized using t-SNE projection as shown in Fig. 10 for each architecture, it can support our results. Here, the features are visualized over the entire test set, which helps determine which architecture segregates the features in a better way. Finally, the results of various hyperparameter settings and data sets are shown in Figures 8 and 9, respectively.

<Embodiment 5> Why diversify the architecture?

The filter size determines the scanning window during convolution, and the size of this window can be inferred as the receiving area. The filter size was increased by 2 strides in each successive layer so that the extracted features could be sequentially extracted at low, medium, and high levels. Low-level features are extracted on a 3 × 3 × 3 filter window and max-pooled by a 2 × 2 × 2 window with a stride of 1 in the first convolutional layer (i.e., from conv_1 to max-1) (Fig. 1(b )). This is called a divergent network in the sense that the size of the filter kernel continues to increase as the step size or stride increases. However, the number of filters in each layer is the same (i.e., 64) to keep the channel size for an input of 64×64×64. We start with the first convolutional layer with a filter size of 3x3x3, so fine details can be easily captured. As the layers get deeper, you can increase the window size of each layer to accumulate features. As a result, the maximum full stride is also increased to reduce feature redundancy. Conversely, the receiving area continues to decrease according to the initial filter size. A uniform kernel size of 3×3×3 is used for each, whereas in convergent networks it is 9×9×9. All details of the architecture and experimental results after training and testing are highlighted in Fig. 6, which includes the parameters in the second column.

<Example 6> PET or MRI or both?

In order to examine the effect of the size of the training data, L4 divergence networks were trained with various data sets, and the results are shown in FIG. 9 . Both MR and PET images used were obtained from patients attending ADNI BL under the ADNI 1 project. We used 3D scans of T1-weighted structural MR images of the whole brain. PET scans were also obtained in ADNI BL (processed for smoothing, co-registration and some normalization), while normalized and processed using the ADNI pipeline and rarely resized. The present invention showed that MRI is a better imaging modality than PET for 3D CNN classification. When training the network with the smallest data set including MRI1 (see column 5 in Fig. 9), the network underfits. Therefore, the test accuracy was low at 74.5%, which is slightly lower than the validation accuracy. However, training achieved convergence with accuracy reaching 100%. The same network trained with BASELINE_MRI data (MRI2 type, Fig. 9) in the same environment achieved the highest test accuracy of 94.5%. The reason for the increased accuracy may be due to the higher scan-per-patient ratio (SPR). This reduces the variability of each scan and loses generality in the network. PET scans showed the worst performance on L4 divNet with increasing training time. The test accuracy of PET1, the BASELINE_PET_SMALL data set, is only 66.34%, whereas the test accuracy of the largest PET data sets (e.g., BASELINE_PET_ALL, PET2) is only 50.21%. It had difficulty achieving nearly 3x convergence with 100 epochs and GPU training time. It is 10 times larger than PET1. Finally, the MRI2+PET1 dataset was merged and trained on a single network, but after convergence, it could only reach 90% training accuracy and up to 82% test accuracy. As a result, MRI appears to be a better choice for CNN and PET only plays a complementary role for predicting AD. It is worth noting that PET images are not so visually discriminated by subject class compared to MRI images (see Fig. 1), which may result in better performance of MRI.

<Test Example 1> Test for CNNs of different layers

Figure 5 highlights the results of various architecture-based configurations using different layers, starting with two to six layers of convolutional encoding. The Parameters column details the indexed filter size, number of filters, maximum pool filter size, stride, and number of FCL inputs and outputs in each row. The training accuracy reaches nearly 100% for each configuration, while the validation and test accuracy start to drop after the L4 layer. This can be the best case plotted on the training and validation losses against the number of epochs, as in Fig. 4(a)-4(f), with an account of overfitting or underfitting cases.

<Example 2> Test for different architectures

The results of using different architectures based on the reception area of the evolution filter size, that is, the results of the four architectures, that is, diverging, equivalent, converging, and U-net, are shown in FIG. 6. Parameter columns are indexed in the same way as in FIG. 5 .

<Test Example 3> Test for different hyperparameter settings

Hyperparameters are a very important factor in optimizing a network for best performance. Several activation functions, initialization techniques, and optimization algorithms were experimented with to find the optimal condition (Fig. 8).

<Test Example 4> Illustration of convolution transformation of each architecture

The convolutional transform is visualized using pseudocode 1. Here we present Figure 7 for each class domain analysis visualized using a single patient MRI scan. The number of features continues to decrease from the former convolutional layer to the latter. The results of the L4 divergent architecture network are displayed in a slice view scaled to 64 × 64 for better visualization.

<Test Example 5> Test on different data sets

Although the network is complete, it is still necessary to determine the size of the data set as it can significantly affect network performance. In the present invention, experiments were performed on various data sets to confirm the effect of the number of training data on test accuracy (FIG. 9).

<Test Example 6> Illustration of FCL T-SNE conversion of each architecture

The FC layer weights are visualized using the T-SNE transform as described in pseudocode 2, and the experimental results for each architecture type are shown in Fig. 10. Here we present a class-by-class representation of the figure for the last three FCLs used.

<Test Example 7> Performance analysis and discussion

Figure 11 shows the feature distribution of a test image set consisting of 296 scans separated by layer during classification from the first convolution to the last FCL. The classification performance of the convergent and divergent architectures is the best among the four selected architectures (Fig. 6). Nevertheless, based on the FCL patient-level visualization, as shown in Fig. 11, it can be seen that the features of each class start to separate better in the divergent architecture than in the convergent architecture. Data visualization using t-SNE from the first FCL FC1 to the third FCL FC3 shows a better separation in the second case (i.e. divergence, Fig. 11). Similarly, based on the final FCL graph plotted as a separate color curve for each cohort domain (Fig. 13) against the actual weights of the final 100 parameters of the network trained without projection, the parameters 512 of the U-net architecture shows a better boundary between each color graph than Then we moved back to the training curves of these three networks (Fig. 10) to complete the best performance. It is observed that the validation loss is much higher than the training loss in convergence and equivalence architectures. This indicates that the network can still be optimized, which has been achieved with different architectures and appropriate hyperparameter selection.

Regarding hyperparameter selection, proper and timely training and maintaining good performance of the trained model are important. Regarding the experiment, the L4 branch architecture was the best among the selected architectures as indicated in Figs. 6 and 8, and important hyperparameters such as initialization, activation and optimization algorithms were selected using Fig. 8.

<Test Example 8> Generalization and overfitting problems

Looking at recent architecture and performance results, the reported precision and accuracy are very high at over 90%. In MR-guided image acquisition, various technical specifications such as acquisition tool, spatial location, contrast intensity, planar orientation, registration template, calibration method, and wrapping protocol can bring variability to MRI of a suspected class. Thus, a neural network trained on one variety of MRIs may be ambiguous in detecting MRIs of the same object class, and acquiring them differently introduces generalization errors to the network. Generalization error is one of the major challenges in medical imaging diagnosis. In the present invention, the network/model was tested with other data from ADNI, marked as MRI_adapted. This is because each participant in the ADNI project partially adjusted the different parts. The MRI_adapted data set was used only for generalization testing, consisting of 135 AD, 162 CN, and 134 MCI 3D scans. The test results are shown in FIG. 10 . Another way to investigate in detail is to visualize features, and by extracting better features, CNNs can learn better. Similarly, overting occurs with generalization errors, and models that do not generalize learn too well to remember only the training patterns that cause overting. Therefore, generality can also be achieved by solving the overting problem.

<Conclusion>

It is very important to design the architectural unit for the development of CNN's optimization algorithm. The deep learning process is highly dependent on the selection of training materials, and closely related images can improve training performance, but problems due to overfitting also occur, so good data is needed rather than big data to create a good network. The DL-based CNN of the present invention can classify MRI/PET based on the separated features learned in the convolutional layer through training, which can help classify Ad, MCI, and MC.

The above description is merely illustrative of the present invention, and those skilled in the art will be able to make various modifications without departing from the essential characteristics of the present invention. Accordingly, the embodiments disclosed in this specification are intended to explain, not limit, the present invention, and the spirit and scope of the present invention are not limited by these embodiments. The protection scope of the present invention should be construed according to the following claims, and all technologies within the equivalent range should be construed as being included in the scope of the present invention.

Claims (10)

1) acquiring a magnetic resonance imaging (MRI) or positron emission tomography (PET) image;
2) extracting activated features of a convolution layer from a single image;
3) Visualizing the activated features of the fully connected layer (FCL) on the entire test set using T-distributed stochastic embedding (T-SNE) projections;
4) applying the final FCL weights of the trained network for direct visualization; and
5) classifying the stage of progression of Alzheimer's disease; Method for providing diagnostic information for classification of advanced stage of Alzheimer's disease including The method of claim 1, wherein step 2) comprises the weights of the filters in layer l with filter characteristics (F _l ) = size k×k×k×f for the trained network N in layer l, where k A method for providing diagnostic information for classification of advanced stages of Alzheimer's disease, in which ×k ×k is the size of a filter and f is the number of filters in layer l. The method of claim 1, wherein the step 2) comprises an initial activated feature (A _l ) = conv (F _l , I), wherein the initially activated feature (A _l ) is downsampled in magnitude and an activation layer is formed. Method for providing diagnostic information for classification of advanced stages of Alzheimer's disease, comprising batch normalization and maximum pooling steps to pass The method of claim 1, wherein in step 2), the feature activated for visualization = _Almax of size k×k×k is scaled to 64×64×64 and each slice is individually displayed in the 2D domain. , Method for providing diagnostic information for classification of advanced stage of Alzheimer's disease The method of claim 1, wherein step 3) includes FCL features (FCL _l ) = weights of an FCL of size T×S for a trained network N of layer l, where T is the number of test objects, A method for providing diagnostic information for classification of advanced stages of Alzheimer's disease, in which S = the size of O × I, and O and I represent the output and input of FCL in the first layer, respectively. The method of claim 1, wherein step 3) includes FCL _l-tsne = T-SNE (FCL _l ), where T-SNE performs feature reduction in N dimensions to find a feature matrix of T×2. Method for providing diagnostic information for classification of advanced stages of Alzheimer's disease, which is to perform T-SNE The method according to claim 1, wherein in step 3), the FCL _l-tsne displays the progress stage of the target Alzheimer's disease on an xy plane to visualize the identification pattern. The method of claim 1, wherein the step 4) includes FCL _l = weights of FCL l of size O×n for a network N having l as the final FCL and n number of classification categories, where O is at the end _. A method for providing diagnostic information for classification of advanced stages of Alzheimer's disease, wherein n is the output size of the second FCL, and n is the output size of the final FCL in layer l, which is the number of advanced stages of Alzheimer's disease. The method of claim 1, wherein in step 4), FCL _l is an Oxn matrix simply displayed as a linear graph on an XY plane. The diagnostic information for classifying the advanced stage of Alzheimer's disease according to claim 1, wherein the advanced stage of Alzheimer's disease is one selected from the group consisting of cognitive normal, mild cognitive impairment, and Alzheimer's disease. How to provide

Images