CN111047589B - Attention-enhanced brain tumor auxiliary intelligent detection and identification method - Google Patents

Abstract

The invention realizes a set of attention-enhanced brain tumor auxiliary intelligent detection and identification method, the technical scheme is improved on the basis of a U-Net model, the attention enhancement mechanism using the training of the segmentation task as the classification task is provided, the accuracy of the classification task is improved by paying attention to the segmentation task, the focus area and the edge information, the segmentation task and the classification task are optimized simultaneously through a multi-task loss measurement and training method, the expected effects on the two tasks of the segmentation task and the classification task are achieved, and the design purpose and the application target are realized.

Description

An attention-enhanced brain tumor-assisted intelligent detection and recognition method

technical field

The invention relates to the field of image processing, in particular to an attention-enhanced brain tumor-assisted intelligent detection and identification method in the field of medical imaging and computer-aided diagnosis.

Background technique

Tumors that grow in the brain are collectively referred to as brain tumors, which refer to tumors of the nervous system that occur in the cranial cavity, including tumors originating from neuroepithelial, peripheral nerve, meninges, and germ cells, lymphoid and hematopoietic tissue tumors, and craniopharynx in the sella region. Angiomas and granulosa cell tumors, and metastatic tumors. Tumors that arise from the brain parenchyma are called primary intracranial tumors, and those that metastasize to the brain from malignant tumors of other organs and tissues of the body are called secondary intracranial tumors. Intracranial tumors can occur at any age, with 20-50 years being the most common. In recent years, with the development of neuroimaging technology and functional examination technology, auxiliary examination has become the main means of diagnosing intracranial tumors.

Glioma is the most common primary intracranial malignant tumor, accounting for more than 75%. Brain gliomas are divided into localized gliomas and diffuse gliomas, which can be classified into WHO grades I-IV according to the degree of tumor malignancy. Brain gliomas can be divided into low-grade gliomas (LGG, WHO grades I-II) and high-grade gliomas (HGG, WHO grades III-IV). There are differences in the treatment methods and prognosis of gliomas with different grades and gene mutations. Therefore, if the tumor area and tumor grade can be accurately segmented and judged before surgical treatment, it will help to guide the treatment plan and the selection of surgical resection area, which is of great value to improve the treatment effect and prognosis of patients.

Magnetic Resonance Imaging (MRI) is a medical imaging technology that uses the hydrogen nuclei of the human body, namely hydrogen protons, in a strong external magnetic field to generate magnetic resonance phenomena under the action of specific radio frequency pulses. MRI is the main imaging technique for various intracranial diseases, and it can be used as the first-choice examination method for some diseases, and it is also an important supplement to CT examination. MRI examination has the advantages of high tissue resolution, multi-sequence, multi-parameter, multi-directional and various fMRI examinations. Common MRI images used for the diagnosis of glioma include three planes of axial, sagittal and coronal planes and four modalities of T1, T1 enhancement, T2 and T2 water suppression. In clinical practice, the information of the above three planes and four modalities is often combined to comprehensively determine the location, extent and grade of the tumor. However, due to the changeable appearance and shape of brain tumors, the segmentation of brain tumors in multimodal MRI images is one of the most challenging and difficult figures in medical image processing. Grading, and even the judgment and classification of genotypes, are the research directions of great concern in clinical practice.

Localized gliomas are more common in children and relatively rare in adults. Most patients can be cured by complete resection. The low degree of malignancy is not the research focus of this patent. MRI images were segmented, and a segmentation task was used as an attention-enhancing mechanism to aid in the detection of diffuse gliomas to grade and classify tumors.

The current application of deep learning on brain MRI images, especially other related patents, focuses on the field of brain tumor segmentation, and the classification and grading of brain tumors are less involved in the fields directly related to diagnosis and treatment, which are clinical It is more concerned about and currently the human eye is difficult to achieve in non-invasive imaging examinations. At present, most of the methods that have achieved good results in brain tumor segmentation tasks use U-Net as the basic framework, and make improvements on this basis. On the basis of the original U-Net, it is optimized with reference to the latest advances in machine learning and deep learning in the field of computer vision to achieve better results. At the same time, compared with other work, it focuses more on tumor grading, classification and diagnosis. As an attention enhancement method, the classification task focuses on abnormal regions in brain MRI images to realize the task of diagnosing tumors.

SUMMARY OF THE INVENTION

At present, most of the applications of U-Net to brain tumor MRI image processing are image segmentation tasks. These applications only segment the region where the tumor is located, and do not further use the segmented images to obtain valuable medical information. In the process of communicating with clinicians, they are more concerned about the issues that we need to focus on.

To achieve the above object, the present invention has adopted the following technical solutions:

An attention-enhanced brain tumor-assisted intelligent detection and identification method, characterized by: comprising;

Step 1: Establish a three-dimensional convolutional network model of a multi-task neural network based on U-Net, which is suitable for segmentation and diagnosis of glioma lesions in brain MRI images;

Step 2: Multi-task joint training target;

Step 3: Measure the loss of multi-tasking and optimize the results;

Step 4: Model training and result evaluation and output.

The steps of establishing a U-Net-based 3D convolutional network model of a multi-task neural network suitable for segmentation and diagnosis of glioma lesions in brain MRI images include:

Based on the original U-Net network, a three-dimensional convolution processing method is used to build a three-dimensional model of the attention region based on the tumor area as a model. The architecture of the model includes a down-sampling data path and an up-sampling data path. Each layer contains two 3×3×3 convolutional layers, and each convolutional layer adopts a dropout method to prevent overfitting, using the activation function of ReLU. After two convolutional layers, a max pooling layer with stride 2 and size 3×3×3 is used for downsampling; deconvolution is used for upsampling between the two layers of the upsampling datapath operation, splicing the upsampled features with the features of the left downsampling layer, and then performing the same convolution operation as the left upsampling layer twice on the spliced features. After the features of the shallow information, two convolutions with the convolution kernel size of 3 × 3 × 3 are performed, and after the convolutional features are obtained, the model is divided into two branches: one continues to perform a convolution operation, and the output channel is the same as the one. The classification in the semantic segmentation results is the same, and then after softmax calculation, the output results including background, edema, tumor parenchyma, necrosis and enhancement kernel are obtained; the other branch performs a convolution operation and uses Global Average Pooling (Global Average Pooling) , followed by two fully connected layers, the output of the second fully connected layer is the same as the number of pathological diagnosis classifications;

Then traverse all the brain MRI images imported from the cases, count the mean and variance information, and reserve it for the standardization operation in the training and prediction process, and receive the input of the four modalities of the brain: T1, T1 enhanced, T2, and T2-Flair. MRI image sequence, each modality is used as a channel, then all slice images under the scanning sequence and the segmentation result labeling mask are spliced respectively to form a 3D picture and labeling sequence, and then these two sequences are combined with the corresponding diagnosis of the sequence. The result is bound as a sample, and all images are processed.

The multi-task joint training target steps include:

On the fully convolutional model, a classification branch is added after merging shallow and deep information, and a single task segmentation and classification model is used to obtain semantic segmentation and classification results at the same time, so that the segmentation task and classification task of brain tumors can be performed at the same time. layer features;

After obtaining the features that integrate the deep and shallow information, perform two convolutions with a convolution kernel size of 3 × 3 × 3. After obtaining the convolved features, the model is divided into two branches: one continues to perform a convolution Operation, the output channel is the same as the classification in the semantic segmentation result, and then after softmax calculation, the output results including background, edema, tumor parenchyma, necrosis and enhancement kernel are obtained; after another convolution operation, the global average pooling is used (Global Average Pooling), followed by two fully connected layers, the output of the second fully connected layer is the same as the number of pathological diagnosis classifications, including the following categories: oligodendroglioma, anaplastic oligodendroglioma , astrocytoma, anaplastic astrocytoma, glioblastoma.

In the step of measuring the loss of multiple tasks, the optimization result step uses the loss function of the multi-task combination to measure the segmentation result and the classification result, and optimize the segmentation and classification results, wherein:

The loss function of the image segmentation model adopts the Dice loss function;

The loss function of the tumor classification module selects the cross-entropy function.

The model training and result evaluation and output steps include:

The model training step is set to allow only the Dice loss of the image segmentation model or the cross-entropy loss function of the tumor classification model to perform back-propagation for iterative training, and combine the loss value of the image segmentation module with the loss value of the tumor classification model in a certain proportion as Then perform backpropagation for iterative training;

After the training effect converges and a relatively ideal result is obtained, the effect is evaluated on the training set;

A new case image is input to the evaluated model, and the detection and recognition results are output.

The advantages of the present invention relative to the prior art are:

This design scheme makes full use of the segmentation information, expands the classification task module for the medical information hidden in the segmented images, so as to play a role as a doctor's auxiliary medical system in the classification and grading of brain tumors, and provide suggestions and improve the diagnosis process. The ability of medical institutions to diagnose related diseases, and the role of feeding back the pathological research of brain tumors in the medical field.

And at present, most of the applications of U-Net to brain tumor MRI image processing are image segmentation tasks. These applications only segment the region where the tumor is located, and do not further use the segmented images to obtain valuable medical-level information. The images are processed and analyzed, and the segmentation and diagnosis results of the lesion area are obtained at the same time.

Description of drawings

Figure 1 Design framework of brain tumor auxiliary detection and recognition system;

Figure 2. Multi-task learning model for brain tumor segmentation and classification diagnosis tasks;

Detailed ways

Referring to the accompanying drawings 1-2 of the description, the present invention proposes a method that combines medical images with deep learning and computer vision methods, and uses an analysis method of computer vision to analyze and process three-dimensional brain MRI images, and perform glue on the brain MRI images. Segmentation and image-based classification and diagnosis tasks of plasmoma lesion regions. Aiming at the problems of small amount of data in medical image datasets, serious category imbalance, and existing methods focus on segmentation of lesion areas while ignoring the task of classification and diagnosis, an improved 3D U-Net convolutional neural network is proposed to increase classification and diagnosis. Branch, through the multi-task joint training, the segmentation and classification results are obtained at the same time. Fig. 1 is the algorithm design flow proposed by the present invention. First, preprocess the MRI image and its corresponding manually labeled segmentation results and the classification and diagnosis information obtained from the pathological information to obtain a three-dimensional image sequence including four modalities. The processed images are divided into a training set and a test set proportionally, and the training is carried out on the training set to optimize the performance of the model on the two tasks of segmentation and classification. Evaluate the effect above.

Before establishing the model, the brain MRI image data we used came directly from the hospital's medical record system. Image preprocessing operations such as denoising, brightness and contrast adjustment to enhance visibility had been completed, and the image data had been manually annotated. Different parts, including edema, tumor parenchyma, enhanced nucleus, and necrosis, were marked, and diagnostic information such as tumor type and stage were obtained from the case's medical record diagnosis and postoperative pathological information.

Traverse all MRI images, statistics such as mean and variance, and reserve it for normalization during training and prediction. In addition, four modal images of the same slice position in the same scanning sequence are stacked, and each modality is used as a channel, and then all slice images under the scanning sequence and the segmentation result labeling mask are spliced respectively to form a three-dimensional Picture and label sequence, and then bind these two sequences with the diagnostic result corresponding to the sequence as an example.

After all samples are processed, they are divided into training set and test set according to a certain proportion for later use.

Then, based on the original U-Net network, a three-dimensional model is built using the three-dimensional convolution processing method.

Figure 2 shows the architecture of the model used in the present invention, including a down-sampled data path on the left and an up-sampled data path on the right. Each layer on the downsampling path on the left contains two 3×3×3 convolutional layers, and each convolutional layer adopts a dropout method to prevent overfitting, using the activation function of ReLU. After two convolutional layers, a max pooling layer with stride 2 and size 3×3×3 is used for downsampling; deconvolution is used for upsampling between the two layers of the right data path operation, splicing the upsampled features with the features of the left downsampling layer, and then performing the same convolution operation twice with the left upsampling layer on the spliced features.

The downsampling layer improves the receptive field of each pixel in the final deepest feature by continuously reducing the resolution of the feature, and obtains a more abstract high-level representation of the feature, which is the deep information. For image classification and the category of pixels in the image The judgment has higher representation ability, but due to the loss of resolution, the accuracy of pixel-level classification and the resolution of the obtained classification results are greatly reduced. The cross-layer connection in the middle is the shallow layer information. By merging with the up-sampling path feature on the right, the problem that the resolution of the result is reduced compared with the input is solved, and the result after fusion is better expressed.

From the above-mentioned upsampling path, after finally obtaining the features that integrate the deep and shallow information, perform two convolutions with a convolution kernel size of 3 × 3 × 3. After the convolutional features are obtained, the model is divided into two branches: One continues to perform a convolution operation, and the output channel is the same as the classification in the semantic segmentation result, and then after softmax calculation, the output results including background, edema, tumor parenchyma, necrosis and enhancement kernel are obtained; the other branch performs a convolution operation. Then, use Global Average Pooling, followed by two fully connected layers, the output of the second fully connected layer is the same as the number of pathological diagnosis classifications, including the following categories: oligodendroglioma, anaplastic oligodendroglioma, astrocytoma, anaplastic astrocytoma, glioblastoma.

The above process can be regarded as a process of accurately identifying the brain tumor category based on the tumor region as the attention region of the model combined with the background. The above process uses a multi-task learning method, so that the segmentation task and the classification task of brain tumors can be performed at the same time, and due to the shared shallow features, the two can be jointly improved under the interaction.

The multi-task learning method used is measured by the following loss function and the model performance is optimized using the following model training method.

1. Loss function

1) Loss function selection of brain tumor segmentation task model:

One of the challenges in medical image segmentation is the problem of class imbalance in the data. For example, in brain tumor MRI images, the target object to be segmented occupies a particularly small proportion of the entire data, resulting in severe class imbalance. In this case, using the traditional categorical cross-entropy loss function will hinder training, while the Dice loss function can effectively deal with the class imbalance problem, so the present invention adopts this loss function as the loss function of the segmentation model, which is specifically expressed as follows:

where u is the segmentation result output of the network, v is the label segmentation, and i is the number of pixels in the training block

2) Loss function selection for brain tumor classification task:

This problem is a classification problem, and the cross-entropy function widely used in classification problems is used as the loss function of this module, which is specifically expressed as follows:

where N is the total number of samples, K is the total number of categories, y _{i, j} are the label values, and p _{i, j} are the predicted values

2. The model training method is as follows:

1) The setting only allows the Dice loss of the image segmentation model or the cross entropy loss function of the tumor classification model to perform back-propagation for iterative training, that is, the overall loss function is expressed as:

2) The loss value of the image segmentation module and the loss value of the tumor classification model are combined in a certain proportion, and then back-propagation is performed for iterative training, that is, the overall loss function is expressed as:

Loss ₂ = Loss _Dice + αLoss _cross

where α is an adjustable scaling factor.

Claims (3)

1. An attention-enhanced brain tumor auxiliary intelligent detection and identification method is characterized by comprising the following steps: comprises the following steps of;

the method comprises the following steps: establishing a three-dimensional convolution network model of a multitask neural network based on U-Net and suitable for segmentation and diagnosis of a brain glioma lesion region in a brain MRI image;

step two: a multi-task joint training objective;

step three: measuring the loss of multiple tasks and optimizing the result;

step four: model training and result evaluation and output;

the step of establishing a three-dimensional convolution network model of a multitask neural network based on U-Net and suitable for segmentation and diagnosis of a brain glioma lesion region in a brain MRI image comprises the following steps:

building a three-dimensional model of an attention area based on a tumor area as the model based on an original U-Net network by using a three-dimensional convolution processing method, wherein the framework of the model comprises a down-sampling data path and an up-sampling data path, each layer on the down-sampling path comprises two convolution layers of 3 multiplied by 3, each convolution layer adopts a dropout mode to prevent overfitting, an activation function of a ReLU is used, and after the two convolution layers, a maximum pooling layer with the step length of 2 and the size of 3 multiplied by 3 is used for carrying out down-sampling operation; the up-sampling operation is carried out between two layers of the up-sampling data path in a deconvolution mode, the features after up-sampling are spliced with the features of the left down-sampling layer, the spliced features are subjected to convolution operation twice which is the same as that of the left up-sampling layer, after the up-sampling path finally obtains the features fused with deep layer and shallow layer information, convolution with convolution kernel size of 3 x 3 is carried out twice, and after the features after convolution are obtained, the model is divided into two parts: one continues to carry out convolution operation once, the output channel is the same as the classification in the semantic segmentation result, and then the output result containing the background, the edema, the tumor parenchyma, the necrosis and the enhancement kernel is obtained through softmax calculation; after the other branch is subjected to convolution operation for one time, Global Average Pooling (Global Average Pooling) is used, then two full-connection layers are followed, and the output of the second full-connection layer is the same as the classification number of pathological diagnosis;

traversing all brain MRI images imported from a case, counting average value and variance information, reserving for standardized operation in the training and predicting process, receiving input brain MRI image sequences of four modes of T1, T1 enhancement, T2 and T2-Flair, wherein each mode is used as a channel, then respectively splicing all slice images in a scanning sequence and a segmentation result labeling mask to form a three-dimensional image and a labeling sequence, binding the two sequences and a diagnosis result corresponding to the sequences to be used as a sample, and processing all images;

the multi-task joint training target step comprises the following steps:

on the full convolution model, a classification branch is added after superficial layer information and deep layer information are fused, and a single task segmentation and classification model is used to obtain semantic segmentation and classification results at the same time, so that a segmentation task and a classification task of the brain tumor can be executed at the same time and share the superficial layer characteristics;

after the features of the deep layer information and the shallow layer information are fused, convolution with convolution kernel size of 3 multiplied by 3 is carried out twice, and after the features after convolution are obtained, the model is divided into two branches: one continues to carry out convolution operation once, the output channel is the same as the classification in the semantic segmentation result, and then the output result containing the background, the edema, the tumor parenchyma, the necrosis and the enhancement kernel is obtained through softmax calculation; the other branch, after one convolution operation, uses Global Average Pooling (Global Average plating), followed by two fully-connected layers, the output of the second fully-connected layer being the same as the number of pathological diagnosis categories, including the following categories: oligodendroglioma, anaplastic oligodendroglioma, astrocytoma, anaplastic astrocytoma, glioblastoma.

2. The method for aided intelligent detection and identification of attention-enhanced brain tumors according to claim 1, characterized in that: the step of measuring the loss of the multiple tasks and the step of optimizing the result use a loss function of the multiple task combination to measure the segmentation result and the classification result and optimize the segmentation and classification result, wherein:

adopting a Dice loss function as a loss function of the image segmentation model;

the loss function of the tumor classification module selects a cross-entropy function.

3. The method for aided intelligent detection and identification of brain tumors according to claim 2, wherein the method comprises the following steps: the model training and result evaluation and output step comprises the following steps:

a model training step, namely setting a Dice pass of an image segmentation model or a cross entropy loss function of a tumor classification model to carry out back propagation for iterative training, and carrying out back propagation for iterative training after combining a pass value of the image segmentation model and a pass value of the tumor classification model in a certain proportion;

after the training effect is converged and a more ideal result is obtained, evaluating the effect on a training set;

and inputting a new case image to the evaluated model, and outputting a detection and identification result.

Images