CN110706209B - Method for positioning tumor in brain magnetic resonance image of grid network - Google Patents

Abstract

The invention provides a method for automatically locating a tumor in a brain magnetic resonance image of a grid network, and proposes a new three-dimensional object detection method, which adopts a shallow three-dimensional convolutional neural network model to extract image features, and adopts Grid method for classification and positioning. The invention includes: obtaining image features from a backbone network of a three-dimensional deep convolutional neural network based on a residual network, and performing brain tumor localization based on the feature images obtained by the backbone network. The present invention can be better applied to the brain nuclear magnetic resonance image, realizes the location of the tumor region in the three-dimensional nuclear magnetic resonance image, the positioning result is accurate, and the cost of computing resources is low.

Description

A Grid Network-Based Tumor Localization Method in Brain Magnetic Resonance Images

technical field

The invention belongs to the field of digital images and relates to a magnetic resonance image processing method, in particular to a grid network-based method for automatically locating tumors in a brain magnetic resonance image.

Background technique

With the continuous development of computer vision technology, with the help of widely existing cameras, many technologies have been applied to everyone's daily life. Although two-dimensional images are more widely available, in some specific scenarios, three-dimensional images can bring a more realistic reflection of the real world. For example, with the help of nuclear magnetic resonance technology, we can obtain the state information of the internal organs of the organism, and with the help of RGB-D images, the safety of autonomous driving can be further improved, etc. Therefore, it is of great practical significance to study high-performance 3D visual models.

MRI is an imaging technology for the internal structure of the human body, and the research on MRI has high value: First, the automatic localization of tumors can reduce the workload of doctors to a certain extent. The research of technology can give more patients the chance to get a diagnosis; secondly, the combination of the MRI image inspection by the algorithm and the doctor's inspection can reduce the risk of misjudgment or missed diagnosis. If a missed diagnosis occurs in the process of tumor diagnosis , the patient is likely to miss the best time for treatment, and the consequences are very serious. While using 3D images for tumor segmentation and localization, we have greater confidence to generate high-precision results due to the richer spatial information of 3D images.

A series of visual challenges such as ImageNet have spawned a large number of excellent two-dimensional visual models such as VGG, ResNet and so on. These high-performance two-dimensional models all have the problem of high requirements for computing resources. Although many algorithms have been able to run at a high speed with the help of the parallel computing power of GPU, if these models are directly converted into corresponding three-dimensional models, it will inevitably cause Dozens of times, or even hundreds of times, the increase in computational cost. In addition, the storage of intermediate results and the growth of model parameters will also bring huge memory overhead. At the same time, the rapidly growing number of model parameters will also make the model in

SUMMARY OF THE INVENTION

In order to solve the above problems, the present invention proposes a new three-dimensional object detection method by exploring the existing deep learning model of three-dimensional images and continuously optimizing the method of adjusting the network structure, using a shallow three-dimensional convolutional neural network. The network model extracts image features, and uses grid method for classification and positioning.

In order to achieve the above object, the present invention provides the following technical solutions:

A method for automatically localizing tumors in a brain magnetic resonance image using a grid network, comprising the following steps:

Step 1, define the backbone network of the three-dimensional deep convolutional neural network based on the residual network, which specifically includes the following sub-steps:

1-1, perform a three-dimensional convolution operation with a step size of 2 and a kernel of [3, 3, 3] on the input three-dimensional MRI image data X: (L, W, H, 1), and the convolution kernel of the three-dimensional convolution The number is set to C1, and the generated data Y1: (L/2, W/2, H/2, C1), where L, W, and H are the length, width, and height of the original image, respectively;

1-2, define a three-dimensional convolutional SSCNN with a step size of 1 and a kernel of [3, 3, 3];

1-3, perform an SSCNN convolution on the data Y1: (L/2, W/2, H/2, C1), set the number of convolution kernels to C1, and generate the convolutional result Y1_1: (L/ 2, W/2, H/2, C1), perform another SSCNN convolution, set the number of convolution kernels to C1, and generate the convolution result Y1_2: (L/2, W/2, H/2 , C1), and finally add each element of Y1 and Y1_2 to generate data Y2: (L/2, W/2, H/2, C1);

1-4, perform a SSCNN convolution on the data Y2: (L/2, W/2, H/2, C1), the number of convolution kernels must be set to C1, and generate the convolution result Y2_1: (L /2,W/2,H/2,C1), perform another SSCNN convolution, the number of convolution kernels must be set to C1, and generate the convolution result Y2_2: (L/2,W/2,H /2, C1), and finally add each element of Y2 and Y2_2 to generate data Y3: (L/2, W/2, H/2, C1);

1-5, perform a three-dimensional convolution operation with a step size of 2 and a kernel of [3, 3, 3] for the data Y3: (L/2, W/2, H/2, C1), and the convolution of the three-dimensional convolution The number of cores is designated as C2, and the generated data Y4: (L/4, W/4, H/4, C2);

1-6, repeat steps 1-3, 1-4, 1-5 twice, and finally perform steps 1-3, 1-4 to get the feature Y extracted from the image: (L/16, W/16, H/ 16, C), where C is the number of convolution kernels in the last repetitions 1-4;

Step 2, define a grid tumor localization network, which specifically includes the following sub-steps:

2-1. Perform a SSCNN convolution on the feature map Y: (L/16, W/16, H/16, C) output by the backbone network, and set the number of convolution kernels to C3 to obtain the data G1 (L/ 16,W/16,H/16,C3);

2-2, run the data G1 (L/16, W/16, H/16, C3) once again. The kernel is 1×1×1, and the length, width and height are respectively (L/16)/ The three-dimensional convolution operation of N, (W/16)/M, (H/16)/K, the number of convolution kernels is set to 2, and the data G is obtained: (N, M, K, 2), where N, M and K respectively represent the number of divisions of the set original image in terms of length, width and height;

2-3, the obtained data G: (N, M, K, 2) is transformed into the original image through Softmax operation, and each grid contains tumor blocks and does not contain tumor blocks. Class probability G1: (N, M, K, 2), according to the size of the category probability, assign the category with a large category probability value to the current image block, and finally integrate all the image blocks containing the tumor to form a complete tumor block location;

Step 3, performing tumor localization on the three-dimensional brain MRI image, which specifically includes the following sub-steps:

3-1. Divide the three-dimensional nuclear magnetic resonance image into N×M×K three-dimensional image blocks, and the size of each three-dimensional image block is (H/N)×(W/M)×(L/K), where H , W, L represent the length, width and height of the original 3D MRI image, respectively;

3-2, labeling gridded 3D MRI images, and the grid categories include image blocks that contain tumors and image blocks that do not contain tumors;

3-2, put the three-dimensional nuclear magnetic resonance image into the network of step 1, and generate a feature map of (L/16)×(W/16)×(H/16)×C;

3-3, put the obtained feature image into the network structure of step 2, and generate each grid classification of N×M×K×2, which is classified into two types including tumor blocks and non-tumor blocks, which will contain tumor blocks. And two categories with high probability that do not contain tumor block are assigned to the current tumor block to form the final N×M×K result, and complete the localization of the tumor.

Further, the step 3-2 also includes the following process: setting a threshold, and calling this threshold a "grid positive threshold", and setting the three-dimensional pixel points marked as tumors in the image block to occupy the pixels of the three-dimensional image block. When the tumor proportion is greater than or equal to the grid positive example threshold, the image block is marked as the image block containing the tumor; when the tumor proportion is less than the grid positive example threshold, the image block is marked as Image blocks that do not contain tumors.

Further, the three-dimensional convolution operation in the step 1 is realized by the following formula:

为三维卷积核，

为偏置项，f为非线性的激活函数。

is the three-dimensional convolution kernel,

is the bias term, and f is the nonlinear activation function.

Further, in the described step 2, the Softmax operation is completed by the following formula:

表示对于输入X在第c层的特征图的分类激活响应。where P represents the pseudo-probability of being a positive example of the predicted class c for the input X,

represents the categorical activation response to the feature map of input X at layer c.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

The present invention can be better applied to the brain nuclear magnetic resonance image, realizes the location of the tumor region in the three-dimensional nuclear magnetic resonance image, the positioning result is accurate, and the cost of computing resources is low. By setting the positive threshold of the grid, the computing resources can be reduced, the segmentation accuracy can be improved, and the problem of uneven distribution of samples containing tumor blocks and non-tumor blocks can be solved.

Description of drawings

Fig. 1 is a three-dimensional backbone network structure based on residual learning provided by the present invention

FIG. 2 is a grid tumor localization network provided by the present invention.

Fig. 3 shows the tumor localization result using the method of the present invention. For brats_tcia_pat463_0001, 0.3 is the grid positive threshold value, and resnet_small is used as the visualization of the localization result of the backbone network.

Figure 4 shows the tumor localization results using the method of the present invention, wherein (a) the grid positive threshold localization results of 0.1 are visualized, (b) the grid positive threshold localization results of 0.2 are visualized, and (c) the grid positive thresholds of 0.3 are visualized The grid positive threshold localization results are visualized, and (d) the grid positive threshold localization results of 0.4 are visualized.

Detailed ways

The technical solutions provided by the present invention will be described in detail below with reference to specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and not to limit the scope of the present invention.

The method for automatically locating a brain magnetic resonance image tumor in a grid network provided by the invention extracts high-level features from the brain image, and locates the tumor part of the image according to the extracted features. First, a feature image is obtained from the 3D backbone network based on residual learning; then, according to the feature image, convolution is performed to predict whether the 3D image block contains a tumor. Finally, through the prediction of the three-dimensional image block, the tumor block is integrated into the whole. Specifically, the inventive method comprises the steps:

Step 1, obtain the features of the image from the backbone network of the 3D deep convolutional neural network based on the residual network:

1-1. The structure of the 3D backbone network based on residual learning is shown in Figure 1. The input 3D MRI image data X: (L, W, H, 1) has a step size of 2, and the kernel (convolution kernel) is For the three-dimensional convolution operation of [3, 3, 3], the number of convolution kernels of the three-dimensional convolution can specify a suitable value C1 to generate data Y1: (L/2, W/2, H/2, C1). L, W, H are the length, width and height of the original image, respectively.

The three-dimensional convolution operation is implemented by the following formula:

加上一个偏置项(bias)

并使用非线性的激活函数f得到一个新的特征图。每一个卷积核

本质上都是一个通过学习得到的参数

当l＝0时，

就对应着输入的第一层，对应着原始输入图片的不同通道。The formula describes that each channel of the previous layer is passed through a three-dimensional convolution kernel

add a bias

And use the nonlinear activation function f to get a new feature map. Each convolution kernel

In essence, it is a parameter obtained by learning

When l=0,

It corresponds to the first layer of the input, corresponding to the different channels of the original input picture.

1-2, define a three-dimensional convolutional SSCNN (Single StrideConvolution Neural Network) with a stride of 1 and a kernel of [3, 3, 3].

1-3, perform an SSCNN convolution on the data Y1: (L/2, W/2, H/2, C1), the number of convolution kernels must be set to C1, and generate the convolution result Y1_1: (L /2,W/2,H/2,C1), perform another SSCNN convolution, the number of convolution kernels must be set to C1, and generate the convolution result Y1_2: (L/2,W/2,H /2, C1), and finally add each element of Y1 and Y1_2 (Element-wisedAddition) to generate data Y2: (L/2, W/2, H/2, C1).

1-4, perform a SSCNN convolution on the data Y2: (L/2, W/2, H/2, C1), the number of convolution kernels must be set to C1, and generate the convolution result Y2_1: (L /2,W/2,H/2,C1), perform another SSCNN convolution, the number of convolution kernels must be set to C1, and generate the convolution result Y2_2: (L/2,W/2,H /2, C1), and finally add each element of Y2 and Y2_2 (Element-wisedAddition) to generate data Y3: (L/2, W/2, H/2, C1). The operations in steps 1-3 and 1-4 include 2 SSCNNs, 1 element addition operation, 2 SSCNNs, and 1 element addition operation to define a sub-network structure, Pyramidal Level (pyramid layer).

1-5, perform a three-dimensional convolution operation with a step size of 2 and a kernel of [3, 3, 3] for the data Y3: (L/2, W/2, H/2, C1), and the convolution of the three-dimensional convolution The number of cores can be specified with a suitable value C2 to generate data Y4: (L/4, W/4, H/4, C2)

1-6, repeat steps 1-3, 1-4, 1-5 twice, and finally perform steps 1-3, 1-4 to get the feature Y extracted from the image: (L/16, W/16, H/ 16, C). where C is the number of convolution kernels in the last repetitions 1-4.

Step 2, obtaining feature images based on the backbone network for brain tumor localization, specifically including:

2-1. The grid tumor localization network is shown in Figure 2. The feature map Y: (L/16, W/16, H/16, C) output by the backbone network is subjected to an SSCNN convolution, and the number of convolution kernels Set the appropriate value C3 to obtain data G1 (L/16, W/16, H/16, C3).

2-2, the data G1 (L/16, W/16, H/16, C3) is processed once again. The kernel is 1×1×1, and the step size is (L/16)/N in the length, width and height, respectively. For the three-dimensional convolution operation of (W/16)/M, (H/16)/K, the number of convolution kernels must be set to 2 to obtain data G: (N, M, K, 2), where N, M , and K respectively represent the number of divisions of the set original image in terms of length, width and height.

2-3, the obtained data G: (N, M, K, 2) is transformed into the original image through Softmax operation, and each grid contains tumor blocks and does not contain tumor blocks. Class probability G1: (N, M, K, 2) According to the size of the class probability, assign the class with a large class probability value to the current image block, and finally integrate all the image blocks containing the tumor to form a complete localization of the tumor block.

The Softmax operation is implemented by the following formula:

则是对于输入X在第c层的特征图的分类激活响应。where P represents the pseudo-probability of being a positive example of the predicted class c for the input X,

is the classification activation response to the feature map of the input X at layer c.

Step 3: Use the network defined in Steps 1 and 2 to perform the entire process of tumor localization on the three-dimensional brain MRI image, specifically including:

3-1. Divide the 3D MRI image into N×M×K 3D image blocks, the size of each 3D image block is (H/N)×(W/M)×(L/K), H,W , L represent the length, width and height of the original 3D MRI image, respectively.

3-2, labeling gridded 3D MRI images. The categories of the grid include image patches that contain tumors and image patches that do not. A threshold is set, which is called "grid positive threshold", and the proportion of three-dimensional pixels marked as tumors in the three-dimensional image block pixels in the image block is set as "tumor ratio". When the tumor proportion is greater than or equal to the grid positive threshold, we demarcate the image block as the image block containing the tumor, and when the tumor proportion is less than the grid positive threshold, we demarcate the image block as the image block not containing the tumor .

3-2, put the three-dimensional nuclear magnetic resonance image into the network of step 1, and generate a feature map of (L/16)×(W/16)×(H/16)×C.

3-3, put the obtained feature image into the network structure of step 2, and generate each grid classification of N×M×K×2, which is classified into two types including tumor blocks and non-tumor blocks. Complete localization of the tumor.

The following takes the data of the Brats 2015 dataset as an example to illustrate the automatic tumor localization method of the nuclear magnetic resonance image of the present invention.

Experimental conditions: Now select a computer for the experiment, the computer is equipped with an Intel processor (3.4GHz) and 10GB random access memory, a 64-bit operating system, and the programming language is Python.

The experimental data are brain magnetic resonance images from the Brats 2015 dataset. The Brats Brain Tumor Image Segmentation Challenge has been held annually since 2012 in conjunction with the MICCAI conference and this Unbiased Image Segmentation Challenge. This dataset for the Brats Brain Tumor Segmentation Challenge provides high-quality manual segmentation annotations as well as MRI images generated using different imaging methods.

On the label, Brats provides 5 labels, namely: 1: necrosis, 2: edema, 3: non-enhanced regions of tumors, 4: enhanced tumor regions ( enhanced regions of tumors) and 5: healthy brain tissue. Among them, the complete tumor of the patient is represented by label 1, label 2, label 3 and label 4, and the complete tumor label of the synthetic data is label 1 and label 2; the tumor core of the patient (Tumor Core) part uses label 1, label 3 and label 4 represent that the tumor core of the synthetic data is label 2; the enhancing tumor part of the patient is label 4, and this part has no samples of synthetic data. The imaging methods used in the dataset are: pre-contrast T1, post-contrast T1, T2 and T2 FLAIR methods. All images were aligned using anatomical samples and scaled by linear interpolation to a scale corresponding to 1〖mm〗^3 for each voxel point, and the original dataset resolution was (155, 240, 240). There are 220 HGG (high grade) training sets and 54 LGG (low grade) training sets in the Brats2015 dataset we use. The test set consists of 53 images that mix HGG and LGG. FIG. 3 is a visualization of the positioning result of using resnet_small as the backbone network by using the method of the present invention for the brats_tcia_pat463_0001 sample, when the grid positive example threshold is 0.3. Figure 4 is the tumor localization result of the present invention, (a) the visualization of the grid positive threshold value of 0.1, (b) the visualization of the grid positive threshold of 0.2, and (c) the grid positive example of 0.3 Visualization of the threshold localization results, (d) visualization of the grid positive example threshold localization results of 0.4. Obviously, based on the method of the present invention, the tumor and non-tumor regions in the three-dimensional nuclear magnetic resonance image can be classified, thereby realizing the localization of the tumor. Using the grid network as the region proposal region generation method for semantic segmentation, the grid positive threshold can be appropriately increased to reduce the prediction of the generated RoI (Region of Interest) region to reduce the regions that need to be segmented and reduce computing resources, or Decrease the grid positive threshold to increase the number of RoI regions to improve segmentation accuracy. Decrease the grid threshold, increase the number of positive samples including the number of tumors, and increase the grid threshold, and reduce the number of positive samples to include the number of tumors. Therefore, the setting of the grid positive threshold is helpful to solve the problem. Samples that contain tumor masses and do not contain uneven distribution of tumor masses.

The technical means disclosed in the solution of the present invention are not limited to the technical means disclosed in the above embodiments, but also include technical solutions composed of any combination of the above technical features. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications are also regarded as the protection scope of the present invention.

Claims (4)

1. a brain magnetic resonance image tumor automatic localization method of grid network, is characterized in that, comprises the steps: Step 1, define the backbone network of the three-dimensional deep convolutional neural network based on the residual network, which specifically includes the following sub-steps: 1-1, perform a three-dimensional convolution operation with a step size of 2 and a kernel of [3, 3, 3] on the input three-dimensional MRI image data X: (L, W, H, 1), and the convolution kernel of the three-dimensional convolution The number is set to C1, and the generated data Y1: (L/2, W/2, H/2, C1), where L, W, and H are the length, width, and height of the original image, respectively; 1-2, define a three-dimensional convolutional SSCNN with a step size of 1 and a kernel of [3, 3, 3]; 1-3, perform an SSCNN convolution on the data Y1: (L/2, W/2, H/2, C1), set the number of convolution kernels to C1, and generate the convolutional result Y1_1: (L/ 2, W/2, H/2, C1), perform another SSCNN convolution, set the number of convolution kernels to C1, and generate the convolution result Y1_2: (L/2, W/2, H/2 , C1), and finally add each element of Y1 and Y1_2 to generate data Y2: (L/2, W/2, H/2, C1); 1-4, perform a SSCNN convolution on the data Y2: (L/2, W/2, H/2, C1), the number of convolution kernels must be set to C1, and generate the convolution result Y2_1: (L /2,W/2,H/2,C1), perform another SSCNN convolution, the number of convolution kernels must be set to C1, and generate the convolution result Y2_2: (L/2,W/2,H /2, C1), and finally add each element of Y2 and Y2_2 to generate data Y3: (L/2, W/2, H/2, C1); 1-5, perform a three-dimensional convolution operation with a step size of 2 and a kernel of [3, 3, 3] for the data Y3: (L/2, W/2, H/2, C1), and the convolution of the three-dimensional convolution The number of cores is designated as C2, and the generated data Y4: (L/4, W/4, H/4, C2); 1-6, repeat steps 1-3, 1-4, 1-5 twice, and finally perform steps 1-3, 1-4 to get the feature Y extracted from the image: (L/16, W/16, H/ 16, C), where C is the number of convolution kernels in the last repetitions 1-4; Step 2, define a grid tumor localization network, which specifically includes the following sub-steps: 2-1. Perform a SSCNN convolution on the feature map Y: (L/16, W/16, H/16, C) output by the backbone network, and set the number of convolution kernels to C3 to obtain the data G1 (L/ 16,W/16,H/16,C3); 2-2, run the data G1 (L/16, W/16, H/16, C3) once again. The kernel is 1×1×1, and the length, width and height are respectively (L/16)/ The three-dimensional convolution operation of N, (W/16)/M, (H/16)/K, the number of convolution kernels is set to 2, and the data G is obtained: (N, M, K, 2), where N, M and K respectively represent the number of divisions of the set original image in terms of length, width and height; 2-3, the obtained data G: (N, M, K, 2) is transformed into the original image through Softmax operation, and each grid contains tumor blocks and does not contain tumor blocks. Class probability G1: (N, M, K, 2), according to the size of the category probability, assign the category with a large category probability value to the current image block, and finally integrate all the image blocks containing the tumor to form a complete tumor block location; Step 3, performing tumor localization on the three-dimensional brain MRI image, which specifically includes the following sub-steps: 3-1. Divide the three-dimensional nuclear magnetic resonance image into N×M×K three-dimensional image blocks, and the size of each three-dimensional image block is (H/N)×(W/M)×(L/K), where H , W, L represent the length, width and height of the original 3D MRI image, respectively; 3-2, labeling gridded 3D MRI images, and the grid categories include image blocks that contain tumors and image blocks that do not contain tumors; 3-2, put the three-dimensional nuclear magnetic resonance image into the network of step 1, and generate a feature map of (L/16)×(W/16)×(H/16)×C; 3-3, put the obtained feature image into the network structure of step 2, and generate each grid classification of N×M×K×2, which is classified into two types that include tumor blocks and those that do not contain tumor blocks, and complete the localization of tumors. . 2. The brain magnetic resonance image tumor automatic localization method of grid network according to claim 1, is characterized in that, in described step 3-2, also comprises the following process: set a threshold, call this threshold as "grid. "Positive threshold", and set the proportion of the three-dimensional pixels marked as tumor in the image block to the pixels of the three-dimensional image block as "tumor proportion"; when the tumor proportion is greater than or equal to the grid positive threshold, the image block is demarcated as The image block containing the tumor, when the proportion of the tumor is less than the grid positive threshold, the image block is marked as the image block that does not contain the tumor. 3. The brain magnetic resonance image tumor automatic localization method of grid network according to claim 1, is characterized in that, in described step 1, three-dimensional convolution operation is realized by following formula:

is the three-dimensional convolution kernel,

is the bias term, and f is the nonlinear activation function. 4. the brain magnetic resonance image tumor automatic localization method of grid network according to claim 1, is characterized in that, in described step 2, Softmax operation is completed by following formula:

where P represents the pseudo-probability of being a positive example of the predicted class c for the input X,

represents the categorical activation response of the feature map at layer C for input X.

Images