CN110689543A

CN110689543A - Improved convolutional neural network brain tumor image segmentation method based on attention mechanism

Info

Publication number: CN110689543A
Application number: CN201910884954.6A
Authority: CN
Inventors: 白柯鑫; 李锵; 关欣
Original assignee: Tianjin University
Current assignee: Tianjin University
Priority date: 2019-09-19
Filing date: 2019-09-19
Publication date: 2020-01-14

Abstract

The invention relates to an attention mechanism-based improved convolutional neural network brain tumor image segmentation method, which comprises the following steps of: data preprocessing: firstly, carrying out unbiased field effect processing of an N4ITK algorithm on three-dimensional brain tumor MRI image data, and secondly, respectively carrying out gray level normalization pretreatment on four modal images of Flair, T1, T1C and T2 in an original MRI image; an attention mechanism-based improved convolutional neural network model is built and trained, and in the training process, four modal data of a patient are input into the network model as four channels of a neural network for training so as to fuse different characteristics of different modes and perform more accurate segmentation; and dividing the preprocessed image into a training set and a testing set, and training the attention mechanism-based improved convolutional neural network model by using the training set.

Description

Improved convolutional neural network brain tumor image segmentation method based on attention mechanism

Technical Field

The method is applied to an important field in the field of medical image processing, and combines medical image processing and a deep learning algorithm to finish the accurate segmentation of the three-dimensional brain tumor nuclear magnetic resonance image.

Background

Brain tumors are a serious health-threatening disease for humans, and due to their apparent differences in size, shape and location, accurate characterization and localization of brain tumor tissue types play a critical role in brain tumor diagnosis and treatment. Because of the characteristics of good soft tissue contrast and non-invasion, Magnetic Resonance Imaging (MRI) -based brain tumor segmentation research becomes a hot spot. Currently, due to the wide application of MRI equipment in brain examination, a large amount of brain MRI image data is generated in clinic, it is impossible for a doctor to manually annotate and segment all images in time, and manual segmentation of brain tumor tissue is dependent on the personal experience of the doctor. Therefore, how to segment brain tumors efficiently, accurately and automatically becomes the focus of research.

The brain tumor image segmentation method mainly comprises the methods based on regions, fuzzy clustering, graph theory, energy and machine learning.

In recent years, deep learning methods, particularly Convolutional Neural Networks (CNNs), have received much attention from researchers by representing very accurate brain tumor segmentation results. The neural network basic models commonly used at present include a CNN model, a Full Convolutional Neural Network (FCNN) model and a U-Net model. Based on the above models, researchers at home and abroad propose a plurality of improved models for segmenting brain tumor images.

Based on a CNN model, Pereira and the like adopt a CNN structure with a deeper layer number, and a plurality of convolution kernels with the size of 3 multiplied by 3 are used in the model to replace large convolution kernels with the sizes of 7 multiplied by 7 and 5 multiplied by 5 so as to improve the operation speed of a convolution network and enhance the extraction of brain tumor characteristics, so that the segmentation precision can reach about 87%. Havaei et al use multi-scale structures by combining features with different filter size paths and further improve the segmentation results by cascading their models, but the network training is difficult and the segmentation accuracy is not ideal due to the more complex network structure and the more model parameters. Kamnitsas et al first used a 3D convolution method to propose a fully connected multi-scale CNN, comprising a high resolution path and a low resolution path, which are recombined to form the final segmented output.

Based on an FCNN model, Chen and the like introduce a multi-scale receptive field on the basis of FCNN to perform accurate voxel classification, the model is built on a dense connecting block, different types of brain tumors are considered by utilizing a layered architecture, and a block-level training mode is used in the training process to relieve the problem of unbalanced brain tumor image categories. In order to fully utilize the strong capability of deep residual learning, Chen and the like provide a deep voxel-level residual network called VoxResNet, which expands two-dimensional deep residual into three-dimensional and integrates multi-level context information with depth supervision so as to further improve the segmentation performance of the 3D brain tumor image. Zhao et al achieved brain tumor segmentation by integrating FCNN and CRF, training CRF with image slices with FCNN parameters in the cross, coronal, and sagittal view directions, respectively, resulting in 3 segmentation models, and fusing the 3 models using a voting-based strategy.

Based on the U-Net model, the method,

by replacing all 2D operations with 3D counterparts on the basis of U-Net, a three-dimensional fully-convolutional neural network 3D U-Net based on voxel segmentation is proposed. Sherman et al propose to segment MRI brain tumor images with V-Net, the network expands U-Net to three dimensions, adds a residual structure between convolutions of the same layer, and uses convolution to replace pooling for down-sampling, which can significantly reduce memory usage. Stawiaski and the like input image data with the resolution of each layer in the analysis path based on the original U-Net network structure, thereby effectively avoiding the loss of brain tumor characteristics in the model training process; in the synthetic path, a multi-scale depth supervision mode is adopted to provide a more accurate segmentation result.

Disclosure of Invention

Aiming at overcoming the defects of the prior art, aiming at the problems of small brain tumor image data set, serious category imbalance, low segmentation precision of the prior algorithm and the like, the invention aims to provide an improved convolutional neural network based on an attention mechanism to realize accurate segmentation of a three-dimensional brain tumor MRI image. The technical scheme adopted by the invention is as follows:

an improved convolution neural network brain tumor image segmentation method based on an attention mechanism comprises the following steps:

1) data preprocessing: firstly, carrying out unbiased field effect processing of an N4ITK algorithm on three-dimensional brain tumor MRI image data, and secondly, respectively carrying out gray level normalization pretreatment on four modal images of Flair, T1, T1C and T2 in an original MRI image;

2) the improved convolutional neural network model based on the attention mechanism is built and trained, and in the training process, four modal data of a patient are input into the network model as four channels of the neural network for training so as to fuse different characteristics of different modes and perform more accurate segmentation: based on a convolutional network 3D U-Net for biomedical image segmentation, the network includes an analysis path for analyzing the whole image, obtaining context information, and a continuously extended synthesis path to achieve precise positioning to generate full resolution segmentation output;

each path has four resolution step layers, each of the two paths comprises two convolution layers with the kernel size of 3 multiplied by 3, and each convolution layer is followed by a ReLu activation function; the maximum pooling layer and the upper sampling layer are respectively arranged between two adjacent layers, and the step length is 2, and the kernel size is 2 multiplied by 2; in order to avoid the bottleneck, the number of channels in the analysis path is doubled before the maximum pooling layer, and the same principle is adopted in the synthesis path; in the last layer, the convolution layer with the kernel size of 1 multiplied by 1 reduces the number of output channels to the number of labels;

adding an Attention Gate (AG) model to a shortcut connection from the same layer in an analysis path for providing basic high-resolution features for a synthesis path, using coarse-scale information extracted from the next resolution layer as a gating signal for deleting irrelevant features in a skipped connection, highlighting salient features passed through the skipped connection, and calling the network as AG _ UNet;

adopting a leak ReLu activation function for the nonlinear parts of all the convolutional layers, adopting instance standardization in a standardization mode, carrying out data enhancement by using data enhancement technologies such as random overturning, random scaling, random elastic deformation and mirror image in the training process, and selecting various kinds of Dice loss functions as loss functions;

dividing the preprocessed image into a training set and a testing set, and training an improved convolutional neural network model based on an attention mechanism by using the training set;

3) and testing a segmentation result: after the improved convolutional neural network model based on the attention mechanism is trained, the model is tested on a test set.

The invention provides an improved convolutional neural network brain tumor image segmentation method based on an attention mechanism, aiming at the problems of small brain tumor image data set, serious category imbalance, low segmentation precision of the existing algorithm and the like. Compared with some classical methods, the advantages are mainly reflected in that:

1) ease of use: the improved convolution neural network based on the attention mechanism is a neural network which can be trained end to end, can be directly applied to the whole three-dimensional image data for processing, and is more convenient and easy to use;

2) the innovation is as follows: the invention applies an attention mechanism to the field of brain tumor segmentation for the first time and provides an attention mechanism-based improved convolutional neural network. Based on the convolutional network 3D U-Net for biomedical image segmentation, AG was added to the standard 3D U-Net network to highlight salient features passed through the skipped connections.

3) The accuracy is as follows: the average Dice evaluation of the algorithm in the whole tumor, the tumor core and the enhanced tumor can respectively reach 0.8623, 0.7846 and 0.6517, and compared with the original 3D U-Net network, the improved convolutional neural network based on the attention mechanism has higher accuracy.

Drawings

FIG. 1 is a flow chart of the segmentation algorithm of the present invention

FIG. 2AG schematically

FIG. 3 is a diagram of an improved convolutional neural network architecture based on an attention mechanism

FIG. 4 is a graph comparing the segmentation results of different convolutional network models

Detailed Description

The invention combines medical image processing and deep learning algorithm to finish the accurate segmentation of the three-dimensional brain tumor nuclear magnetic resonance image. The invention provides an improved convolutional neural network brain tumor image segmentation method based on an attention mechanism, aiming at the problems of small brain tumor image data set, serious category imbalance, low segmentation precision of the existing algorithm and the like. FIG. 1 is a block diagram of the algorithm proposed by the present invention, first preprocessing four modalities in the original MRI image, respectively; secondly, dividing the preprocessed image into a training set and a test set, and building and training an attention mechanism-based improved convolutional neural network model on the training set; and finally, applying the trained model to a test set to test the model, and evaluating the segmentation result by using the corresponding evaluation index.

1) Data pre-processing

Since MRI intensity values are non-standardized, it is important to normalize the MRI data. However, the data come from different research institutes, and the scanners and acquisition protocols used are different, so it is important to process the data with the same algorithm. During the process, it is necessary to ensure that the range of data values is matched not only between patients but also between modalities of the same patient to avoid initial bias of the network.

The method firstly carries out the unbiased field effect processing of the N4ITK algorithm on the three-dimensional brain tumor MRI image data. Secondly, respectively carrying out gray normalization preprocessing on four modal images, namely Flair, T1, T1C and T2 in the original MRI image, and firstly, independently normalizing each modality of each patient by subtracting the average value and dividing the average value by the standard deviation of a brain region; the resulting image is then cropped to [ -5,5] to remove outliers, then normalized to [0,1] again, and the non-brain regions are set to 0. In the training process, the four modal data of the patient are taken as four channels and input into the network model for training, so that the network learns different characteristics of different modalities to perform more accurate segmentation.

2) Building and training improved convolutional neural network model based on attention mechanism

The present invention is based on a convolutional network 3DU-Net for biomedical image segmentation, which includes an analysis path for analyzing the entire image, obtaining context information, and a continuously extended synthesis path to achieve accurate positioning to produce full resolution segmentation output.

Each path has four resolution step layers, each of the two paths containing two convolutional layers with kernel size of 3 × 3 × 3, each convolutional layer being followed by a ReLu activation function. The maximum pooling layer and the upper sampling layer are respectively arranged between two adjacent layers, and the step length is 2, and the kernel size is 2 multiplied by 2. To avoid bottlenecks, the number of channels in the analysis path has been doubled before the max pooling layer, the same in the synthesis path. At the last layer, a convolutional layer with a kernel size of 1 × 1 × 1 reduces the number of output channels to the number of tags.

Note that mechanisms have first gained popularity in the field of natural language processing, such as machine translation. In computer vision, attention mechanisms are applied to various questions including image classification, segmentation, motion recognition, image description, and visual question answering. In the field of medical image analysis, attention mechanisms have been applied to medical report generation and joint image and text classification. However, for the medical image segmentation problem, only a few research works have used attention mechanisms, although local information is also very important.

Oktay et al propose a novel image mesh-based Attention Gate (AG) model for medical imaging that can automatically learn to focus attention on target structures of various shapes and sizes. The model trained using AG implicitly learns to suppress irrelevant areas in the input image while highlighting salient features relevant to a particular task. The AG diagram is shown in FIG. 2.

Note the coefficient α_i∈[0,1]For identifying salient image regions and pruning feature maps, retaining only features relevant to a particular task. The output of the AG is an elemental multiplication of the input feature map and the attention coefficient:

in the default setting, for each pixel vectorA single scalar attention value is calculated, where Fl corresponds to the number of feature maps in layer l. In the case of multi-target structures, it is proposed to select multi-dimensional attention coefficients for learning, where each AG focuses on learning a subset of the target structure. Gated vector

An attention area is determined for each pixel i. The gating vector contains context information, so that the feature mapping of lower layers is deleted. The attention coefficient is calculated using the superimposed attention, which is defined as follows:

wherein

Is an S-shaped activation function. Parameter theta_attComprises the following steps: linear transformation

Sum of deviation term

The linear transformation is computed by a 1 × 1 × 1 convolution and standard back-propagation updates can be used to train the AG parameters, which is also a significant advantage of the AG model.

The invention adds an attention gate AG model to a shortcut connection from the same layer in the analysis path that provides basic high resolution features for the synthesis path, highlighting the salient features delivered by the skip connection, and the specific network structure is shown in fig. 3, which is called AG _ UNet.

The coarse-scale information extracted from the next resolution layer is used as a gating signal to prune irrelevant features in the skipped connection. This operation is performed just before the stitching operation, which ensures that only features relevant to the target task are merged. In the training process, the AG filters features during both forward and backward propagation, so that information from background areas is clipped during the transfer process, so that the parameters of the model are updated based largely on the area associated with a given target task. The update rule of the convolution parameters in layer l-1 can be expressed as follows:

wherein the first gradient term on the right side is

Scaling is performed. In each AG, the supplemental information is extracted and fused to define the output of the skipped connection.

The method adopts a leak ReLu activation function for the nonlinear parts of all the convolution layers, adopts instance standardization in a standardization mode, uses data enhancement technologies such as random inversion, random scaling, random elastic deformation and mirror image to enhance data in the training process, and selects various kinds of Dice loss functions as loss functions.

And dividing the preprocessed image into a training set and a testing set, and training the attention mechanism-based improved convolutional neural network model by using the training set.

3) Testing segmentation results

And after the improved convolutional neural network model based on the attention mechanism is trained, testing the model on the test set, and evaluating the segmentation result by using a corresponding evaluation index.

In order to verify the effectiveness of the improved 3D U-Net network, the improved convolutional neural network based on the attention mechanism, which is provided by the invention, and the original 3D U-Net network adopt the same depth and the same filter base number, and model training, verification and testing are carried out on the same training set, verification set and testing set.

First, qualitative analysis is performed from a segmentation result graph in a model test process. FIG. 4 is a comparison chart of the segmentation results in three directions of the transverse plane, the coronal plane and the sagittal plane after an example of data in the test set is segmented by respectively adopting an attention-based improved convolutional neural network model and an original 3D U-Net model. As can be seen from fig. 4, the segmentation result using the 3D U-Net network model can approximately segment the larger target objects, such as the whole tumor and the tumor core, compared with the label manually segmented by experts, although the detail information of the edge part is not fine enough. However, for small tissue structures such as enhanced tumors, the 3D U-Net network model cannot segment them out. The result of segmentation by using the AG _ UNet network model has better effect compared with the former. It can be seen that the size and margins of the segmented object are both closer to the label and that fine structures like enhanced tumors are also segmented better.

And secondly, carrying out quantitative analysis on the Dice similarity coefficient evaluation indexes of the segmentation results in the model test process. Table 1 shows the Dice mean results of three segmented targets, i.e., the whole tumor, the tumor core, and the enhanced tumor, after the test set data are segmented by using different convolutional network models. As can be seen from Table 1, the improved convolutional neural network based on attention mechanism proposed by the present invention has a certain improvement over the original 3D U-Net network, which is also consistent with the above qualitative analysis results. Experimental results show that the attention mechanism is introduced into the U-Net network structure to be used for segmenting the brain tumor image, and the accuracy of the segmentation model can be improved.

TABLE 1

Claims

1. An improved convolution neural network brain tumor image segmentation method based on an attention mechanism comprises the following steps:

1) data preprocessing: firstly, carrying out unbiased field effect processing of an N4ITK algorithm on three-dimensional brain tumor MRI image data, and secondly, respectively carrying out gray level normalization preprocessing on four modal images of Flair, T1, T1C and T2 in an original MRI image.

each path has four resolution step layers, each of the two paths comprises two convolution layers with the kernel size of 3 multiplied by 3, and each convolution layer is followed by a ReLu activation function; the maximum pooling layer and the upper sampling layer are respectively arranged between two adjacent layers, and the step length is 2, and the kernel size is 2 multiplied by 2; in order to avoid bottleneck, the number of channels in the analysis path is doubled before the maximum pooling layer, and the same principle is adopted in the synthesis path; in the last layer, the convolution layer with the kernel size of 1 multiplied by 1 reduces the number of output channels to the number of labels;

adding an Attention Gate (AG) model into a quick connection from the same layer in an analysis path for providing basic high-resolution features for a synthetic path, taking coarse-scale information extracted from the next resolution layer as a gating signal to delete irrelevant features in a skip connection, highlighting a significant feature transmitted by the skip connection, and constructing an improved convolutional neural network model based on an Attention mechanism, wherein the AG _ UNet model is called AG _ UNet;