WO2022127500A1

WO2022127500A1 - Multiple neural networks-based mri image segmentation method and apparatus, and device

Info

Publication number: WO2022127500A1
Application number: PCT/CN2021/131340
Authority: WO
Inventors: 黄钢; 聂生东; 张小兵
Original assignee: 上海健康医学院; 上海理工大学
Priority date: 2020-12-14
Filing date: 2021-11-18
Publication date: 2022-06-23
Also published as: CN112634211A

Abstract

A multiple neural networks-based MRI image segmentation method and apparatus, and a device. Said method comprises the following steps: acquiring an original MRI image, pre-processing the original MRI image, and generating a processed MRI image; extracting an axial plane image block, a coronal plane image block and a sagittal plane image block from the processed MRI image; using the axial plane image block, the coronal plane image block and the sagittal plane image block respectively as inputs of a trained axial segmentation model, coronal segmentation model and sagittal segmentation model correspondingly, and obtaining segmentation results of the models; and fusing the segmentation results of the models, and obtaining a final segmentation result on the basis of a fusing result and a set volume constraint, wherein the axial segmentation model, the coronal segmentation model and the sagittal segmentation model are closely connected 2D-CNN neural network segmentation models with the same structure. Compared with the prior art, the method has the advantages of short operation time and high accuracy.

Description

MRI image segmentation method, device and equipment based on multiple neural networks

technical field

The invention belongs to the technical field of medical image processing, and in particular relates to a method, device and equipment for MRI image segmentation based on multiple neural networks.

Background technique

Glioma is one of the most common and aggressive types of brain tumors in primary brain tumors, with a very short life expectancy and extremely low survival rate. However, gliomas can appear anywhere in the brain with different sizes and shapes. In addition, manual segmentation of gliomas is time-consuming and labor-intensive and subject to subjective interference. Therefore, fully automated and reliable glioma segmentation method technology is an important research direction. Magnetic Resonance Imaging (MRI) technology is a common non-invasive imaging method that can provide high tissue contrast and is especially suitable for imaging special structures similar to tumors in the human brain. Multiple sequences can be measured by MRI. Such as T1-weighted images, T1C (Contrast enhanced T1-weighted images), T2-weighted images, and Flair (Fluid attenuated inversion recovery images), etc., four image sequences are used in combination for common diagnosis and segmentation Glioma.

The research on glioma segmentation in magnetic resonance images mainly analyzes the diseased tissue in images from the fields of image processing, pattern recognition, artificial intelligence, etc. Methods of discriminating models, etc. In recent years, with the great success of deep learning technology in the general image field, it has also been widely studied in the research of glioma segmentation, especially the convolutional neural network (CNN) based on deep learning. ) has achieved certain success in brain glioma segmentation. CNN is assembled from input layer, convolution layer, nonlinear layer, pooling layer and fully connected layer. The supervised learning algorithm based on training samples and labels passes Learn to obtain a classification model to achieve the task of segmentation of brain glioma images. CNN-based segmentation methods automatically learn a set of increasingly complex features directly from the data to represent learning without relying on manually extracted features. By analyzing the existing CNN-based segmentation methods, it is found that CNN-based glioma image segmentation has the following problems: (1) 2D images are used as input, and the spatial domain relationship of pixels is not considered; (2) 3D image blocks As the input, the computational load of the neural network is increased, and the running time and the required storage space are greatly increased; (3) the features of each layer of the CNN are only used once, which is easy to miss important information.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide an MRI image segmentation method, device and equipment based on multiple neural networks with reduced running time and high precision in order to overcome the above-mentioned defects in the prior art.

The object of the present invention can be realized through the following technical solutions:

An MRI image segmentation method based on multiple neural networks, comprising the following steps:

acquiring an original MRI image, preprocessing the original MRI image, and generating a processed MRI image;

extracting an axial plane image block, a coronal plane image block and a sagittal plane image block from the processed MRI image;

The axial plane image block, the coronal plane image block and the sagittal plane image block are respectively used as the input of the trained axial segmentation model, coronal segmentation model and sagittal segmentation model, and the segmentation result of each model is obtained;

Fusion of the segmentation results of the various models, and obtain the final segmentation result based on the fusion results;

Wherein, the axial segmentation model, coronal segmentation model and sagittal segmentation model are densely connected 2D-CNN neural network segmentation models with the same structure.

Further, the original MRI image is a multimodal MRI image, and the preprocessing includes superposition fusion processing and normalization processing of image data of different modalities.

Further, the modalities include Flair, T1, T1c and T2.

Further, the original MRI image is an original glioma MRI image, and the superposition fusion processing is to perform algebraic superposition fusion of images of two modalities of Flair and T1.

Further, the densely connected 2D-CNN neural network segmentation model includes a first convolution layer, a first densely connected block for extracting low-level feature data, a second densely connected block for extracting high-level feature data, The second convolution layer, the fully connected layer and the classification layer, the low-level feature data and the high-level feature data are jointly used as the input of the second convolution layer.

Further, the first dense connection block and the second dense connection block respectively include several convolutional layers, and the outputs of all the previous layers are used as the input of the next convolutional layer.

Further, the axial segmentation model, the coronal segmentation model and the sagittal segmentation model are obtained by training respectively based on the training image blocks of the axial plane, the coronal plane and the sagittal plane of the same training data set.

Further, in the densely connected 2D-CNN neural network segmentation model, each convolutional layer is provided with a batch regularization layer, and the fully connected layer is added with Dropout technology.

Further, the segmentation results of the fusion models are specifically:

Denote the segmentation results of the respective models as _ra , rc and _rs , and denote the fusion result as _r ;

If ra =rc = _rs , then _r ₌ _ra ;

If any two of _r _a , rc , and _rs are equal, then r is equal to two equal values;

If _r _a , rc , and _rs are not equal, if two are greater than 1, take r=2, otherwise take r=0.

Further, the method further includes: obtaining a final segmentation result based on the fusion result and the set volume constraint;

The volume constraint is: remove the area with a volume smaller than the set number of pixels and fill it with zeros.

The present invention also provides an MRI image segmentation device based on multiple neural networks, including:

a receiving module for acquiring an original MRI image, preprocessing the original MRI image, and generating a processed MRI image;

an image block providing module for extracting an axial plane image block, a coronal plane image block and a sagittal plane image block from the processed MRI image;

The segmentation module is used to respectively correspond the axial plane image block, coronal plane image block and sagittal plane image block as the input of the trained axial segmentation model, coronal segmentation model and sagittal segmentation model, and obtain the input of each model. segmentation result;

a fusion module, used to fuse the segmentation results of the various models, and obtain the final segmentation result based on the fusion results and the set volume constraints;

In the segmentation module, the axial segmentation model, the coronal segmentation model and the sagittal segmentation model are densely connected 2D-CNN neural network segmentation models with the same structure.

The present invention also provides an electronic device, comprising:

one or more processors;

memory; and

One or more programs stored in memory, the one or more programs including instructions for performing the multiple neural network based MRI image segmentation method as described above.

Compared with the prior art, the present invention has the following beneficial effects:

1. The present invention extracts image blocks from three views of the axial plane, coronal plane and sagittal plane of the original MRI image, obtains the segmentation results of each view, and then fuses the segmentation results to obtain the final segmentation result. The 2D-CNN model divides the image, not only using the plane neighborhood relationship of each pixel point, but also using the spatial characteristics of each pixel point, ensuring high segmentation accuracy and segmentation accuracy, and greatly reducing the running time. and storage space.

2. The multi-modal MRI image of the present invention performs preprocessing such as superposition fusion, normalization, etc., to strengthen the image features and improve the image standardization, thereby improving the segmentation accuracy.

3. For glioma MRI images of the present invention, by fusing the glioma data of T1 and Flair modalities, the edema part of gliomas is effectively enhanced, and then the MRI images are reduced by normalization operation. Interference due to uneven light.

4. The present invention adds dense connection blocks to the network structure, and realizes the extraction of different features in stages, and the features are reused without missing important information, thereby effectively improving the segmentation accuracy.

5. After the multi-view segmentation results are fused, the present invention also imposes volume constraints on the fusion results, and removes regions with a volume smaller than the set number of pixels and fills them with zeros to improve segmentation accuracy.

6. The present invention can be applied to glioma segmentation of multimodal MRI images, and can ensure high segmentation accuracy and segmentation accuracy, so that different sub-regions of glioma can be accurately and finely segmented, which is beneficial for doctors. Follow-up work provides important reference.

Description of drawings

Fig. 1 is the flow chart of the three-dimensional segmentation method of the present invention;

Fig. 2 is the dense connection block 1 constructed in the present invention;

Fig. 3 is the structure diagram of densely connected 2D-CNN constructed in the present invention;

4 is a flow chart of three-dimensional segmentation of MRI images of glioma in the embodiment;

Figure 5 is a comparison diagram of three-dimensional segmentation results of MRI images of brain gliomas in Example, (a) original image, (b) gold standard, (c) axial segmentation results, (d) coronal segmentation results, (e) sagittal segmentation results Segmentation results, (f) segmentation results after fusion processing, (g) final segmentation results after adding volume constraints.

Detailed ways

The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented on the premise of the technical solution of the present invention, and provides a detailed implementation manner and a specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

Example 1

As shown in FIG. 1 , this embodiment provides an MRI image segmentation method based on multiple neural networks, including:

Step S1, acquiring an original MRI image, where the original MRI image is a multimodal MRI image. In this embodiment, the original MRI image is a multimodal glioma MRI image.

In step S2, the original MRI image is preprocessed to generate a processed MRI image sequence.

Preprocessing includes superposition fusion processing and normalization processing of image data of different modalities. In this example, the three-dimensional glioma MRI image data is segmented, which includes four modalities: Flair, T1, T1c, and T2. Since the boundary of the edema inside the glioma is ambiguous, and the edema is in the Flair modal image There is a high-brightness signal in the middle, and the T1 modality can well display the internal structural information of the glioma. Therefore, in order to display the complete edema part of the glioma, the images of the two modalities of Flair and T1 are algebraically superimposed and fused. , get I _ehance , and then normalize the four data of Flair, I _ehance , T1c, T2.

Step S3, extracting an axial plane image block, a coronal plane image block and a sagittal plane image block from the processed MRI image. In this embodiment, the size of each image block is 33*33.

Step S4, transforming the glioma segmentation task into a multi-classification problem based on multimodal MRI images. The axial plane image block, coronal plane image block and sagittal plane image block are respectively used as the input of the trained axial segmentation model, coronal segmentation model and sagittal segmentation model, and the node value of the classification layer is obtained. value, obtain the segmentation result map of MRI image axial plane, coronal plane, sagittal plane, and record the segmentation results _ra , _rc and _rs of each model.

The axial segmentation model, coronal segmentation model and sagittal segmentation model are densely connected 2D-CNN neural network segmentation models with the same structure. The densely connected 2D-CNN neural network segmentation model includes a first convolutional layer, a first densely connected block for extracting low-level feature data, a second densely connected block for extracting high-level feature data, and a second convolutional layer. , fully connected layer and classification layer. The input data is first extracted through the first convolution layer unsupervised once; then it is sent to the first dense connection block, and the output data of the first dense connection block is simultaneously sent to two channels, the pooling layer of channel 1, and the image data comparison is obtained. The shallow feature data is sent to the second channel containing the second dense connection block and the pooling layer to obtain the high-level feature data of the image data; then the low-level and high-level feature data are connected and sent to the second convolutional layer. , without missing important information; finally, it is sent to the fully connected layer, and the softmax function of the classification layer outputs the probability that the center point of the image block belongs to each class. In addition, in order to improve the network convergence speed and avoid network overfitting, the model also adds a Batch Normalization (BN) layer after each convolutional layer, and adds Dropout technology to the fully connected layer.

The first densely connected block and the second densely connected block respectively contain several convolutional layers. As shown in Figure 2, the first dense connection block contains 8 layers of convolutional layers. The outputs of all previous layers are used as the input of the next layer of convolutional layers. The number of convolution kernels in each layer is 24 and the size is 3*3. Similarly, a second dense connection block can be designed, which can include 6 layers of convolution layers, the number of convolution kernels in each layer is 12, and the size is 3*3.

As shown in Figure 3, in the densely connected 2D-CNN neural network segmentation model constructed in this embodiment, the input image size is 33*33*4, and the output size is 1*1*4.

The training process of the densely connected 2D-CNN neural network segmentation model is as follows:

1.1. Obtain 3D MRI image data of glioma in four modalities of Flair, T1, T1c, and T2 and the corresponding label data by manual segmentation, label 4 represents the enhanced tumor part, label 2 represents the edema part, and label 1 represents necrosis and the non-enhanced part, label 0 is other tissue parts; then the glioma data of T1 and Flair modalities are fused to obtain I _ehance , and then the four data of Flair, I _ehance , T1c, T2 are set aside by The method divides the training set and the test set;

1.2. Perform axial, coronal, and sagittal slice processing on the 3D MRI image data of the training set to obtain the corresponding slice images and label images corresponding to the slice images, and then perform grayscale normalization on the slice images;

1.3. Obtain 33*33*4 image blocks from the normalized axial, coronal, and sagittal image slices as training image blocks for the axial segmentation model, coronal segmentation model, and sagittal segmentation model;

1.4. Use an unsupervised step-by-step layer-by-layer training method to extract glioma features, and use backpropagation algorithm and stochastic gradient descent algorithm to supervised to minimize the loss function to optimize network parameters, and finally obtain three optimized dense Connected 2D-CNN neural network segmentation model;

1.5. The axial, coronal, and sagittal slice images of the 3D MRI image data of the test set are respectively sent to the three trained neural networks, and each view obtains a 3D MRI image segmentation result, a voxel Click to get three segmentation results of _ra , _rc , and _rs to verify the effect of the model.

The loss function used in the training of the densely connected 2D-CNN neural network segmentation model is:

in,

is the cross-entropy loss function,

is the newly added loss term, which imposes a control factor ε.

Step S6, fuse the segmentation results of the various models. In this embodiment, the MRI image sequence is fused according to the majority voting strategy.

In this embodiment, the obtained segmentation results ra , rc , and _rs under the three views are fused according to the fusion strategy. The fusion rules are: (1) If _ra ₌ rc ₌ _rs , then _r ₌ ra, _r is the fusion result; (2) if any two of _ra , rc, _rs are equal, then _r is equal to two equal values; (3) if _ra , rc, _rs are different Equal, if there are two greater than 1, take r=2, otherwise take r=0.

In step S5, post-processing is performed.

In order to reduce mis-segmentation and over-segmentation, after the fusion process, a volume constraint is added, and the area with a volume of less than 200 pixels is removed and filled with zeros to improve the accuracy of segmentation.

experiment:

In order to illustrate the effectiveness and adaptability of the above method, the data used in the experiment is the training data set in the Brats2018 challenge database, which contains a total of 210 groups of high-grade glioma (HGG) data and 75 groups of low-grade gliomas. Tumor (LGG) data, each group of data includes MRI data of four modalities of Flair, T1, T1C, T2 and the gold standard data of manual segmentation, label 4 represents the enhanced tumor part, label 2 represents the edema part, label 1 represents necrosis and Non-enhanced part, label 0 is other tissue parts; this experiment selected 80% of the patient data as the training set and 20% of the data as the test set. The evaluation indicators of the test results include three items: Dice coefficient (Dice Score), positive predictive value (PPV, Positive Predictive Value), sensitivity (Sensitivity), they are defined as follows:

Among them, TP is true positive, FP false positive, FN false negative, TN true negative. In this experiment, there are 57 groups (42 groups of HGG, 15 groups of LGG) on the test set divided into the data of the Brats18 challenge. The average segmentation evaluation index is shown in Table 1, where Comp represents the entire tumor area (label 1+2+4 part), Core represents the tumor core region (label 1+4 part), and Enh represents the tumor enhancement region (label 4 part). Table 1 shows that the present invention performs well on the validation set, and the steps of fusion processing and post-processing can effectively improve the segmentation accuracy of gliomas. Figure 5 shows a set of data randomly tested during the experiment.

Table 1 The present invention averages the segmentation evaluation index on the test set

Example 2

This embodiment provides an MRI image segmentation device based on multiple neural networks, including a receiving module, an image block providing module, a segmentation module and a fusion module, wherein the receiving module is used to acquire an original MRI image, and perform a Preprocessing to generate a processed MRI image; the image block providing module is used to extract the axial plane image block, the coronal plane image block and the sagittal plane image block from the processed MRI image; the segmentation module is used to extract the axial plane image block. The plane image block, coronal plane image block and sagittal plane image block are respectively used as the input of the trained axial segmentation model, coronal segmentation model and sagittal segmentation model, and the segmentation results of each model are obtained; the fusion module is used to fuse the For the segmentation results of each model, the final segmentation results are obtained based on the fusion results and the set volume constraints; in the segmentation module, the axial segmentation model, coronal segmentation model and sagittal segmentation model are densely connected 2D- CNN neural network segmentation model. The rest are the same as in Example 1.

Example 3

This embodiment provides an electronic device, including one or more processors, a memory, and one or more programs stored in the memory, wherein the one or more programs include a method for executing the multi-based method described in Embodiment 1. Instructions for a neural network-based MRI image segmentation method.

The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make many modifications and changes according to the concept of the present invention without creative efforts. Therefore, all technical solutions that can be obtained by those skilled in the art through logical analysis, reasoning or limited experiments on the basis of the prior art according to the concept of the present invention shall fall within the protection scope determined by the claims.

Claims

A method for MRI image segmentation based on multiple neural networks, characterized in that it comprises the following steps:

acquiring an original MRI image, preprocessing the original MRI image, and generating a processed MRI image;

extracting an axial plane image block, a coronal plane image block and a sagittal plane image block from the processed MRI image;

The axial plane image block, the coronal plane image block and the sagittal plane image block are respectively used as the input of the trained axial segmentation model, coronal segmentation model and sagittal segmentation model, and the segmentation result of each model is obtained;

Fusion of the segmentation results of the various models, and obtain the final segmentation result based on the fusion results;

Wherein, the axial segmentation model, coronal segmentation model and sagittal segmentation model are densely connected 2D-CNN neural network segmentation models with the same structure.
The MRI image segmentation method based on multiple neural networks according to claim 1, wherein the original MRI image is a multimodal MRI image, and the preprocessing includes superposition fusion processing and normalization of image data of different modalities. Unified processing.
The MRI image segmentation method based on multiple neural networks according to claim 2, wherein the modalities include Flair, T1, T1c and T2.
The MRI image segmentation method based on multiple neural networks according to claim 3, wherein the original MRI image is an original glioma MRI image, and the superposition fusion process is to combine two modalities of Flair and T1. Algebraic overlay fusion of the images.
The MRI image segmentation method based on multiple neural networks according to claim 1, wherein the densely connected 2D-CNN neural network segmentation model comprises a first convolution layer, a first convolution layer for extracting low-level feature data A densely connected block, a second densely connected block for extracting high-level feature data, a second convolutional layer, a fully-connected layer and a classification layer, the low-level feature data and high-level feature data jointly serve as the second convolutional layer input of.
The MRI image segmentation method based on multiple neural networks according to claim 5, wherein the first densely connected block and the second densely connected block respectively comprise several convolutional layers, and the outputs of all the previous layers are used as Input to the next convolutional layer.
The MRI image segmentation method based on a plurality of neural networks according to claim 1, wherein the segmentation results of the fusion models are specifically:

Denote the segmentation results of the respective models as ra , rc and rs , and denote the fusion result as r ;

If ra =rc = rs , then r = ra ;

If any two of r a , rc , and rs are equal, then r is equal to two equal values;

If r a , rc , and rs are not equal, if two are greater than 1, take r=2, otherwise take r=0.
The MRI image segmentation method based on multiple neural networks according to claim 1, wherein the method further comprises: obtaining a final segmentation result based on the fusion result and the set volume constraint;

The volume constraint is: remove the area with a volume smaller than the set number of pixels and fill it with zeros.
An MRI image segmentation device based on multiple neural networks, characterized in that, comprising:

a receiving module, configured to obtain an original MRI image, preprocess the original MRI image, and generate a processed MRI image;

an image block providing module for extracting an axial plane image block, a coronal plane image block and a sagittal plane image block from the processed MRI image;

The segmentation module is used to respectively correspond the axial plane image block, coronal plane image block and sagittal plane image block as the input of the trained axial segmentation model, coronal segmentation model and sagittal segmentation model, and obtain the input of each model. segmentation result;

a fusion module, used to fuse the segmentation results of the various models, and obtain the final segmentation result based on the fusion results and the set volume constraints;

In the segmentation module, the axial segmentation model, the coronal segmentation model and the sagittal segmentation model are densely connected 2D-CNN neural network segmentation models with the same structure.
An electronic device, comprising:

one or more processors;

memory; and

One or more programs stored in a memory, the one or more programs comprising instructions for performing the multiple neural network based MRI image segmentation method of any of claims 1-8.