CN114266794B - Pathological section image cancer region segmentation system based on full convolution neural network - Google Patents

Abstract

The invention discloses a pathological slice image cancer area segmentation system based on a full convolutional neural network. The realization of the system includes: extracting the mask of the tissue area in the pathological slice image, removing the blank background; Combine the labeled cancer areas, cut the full-field pathological slice images to obtain training sample data; expand the sample data; build a Unet segmentation network with Resnet50 as the encoder, and replace the convolution unit of the first-level encoder with a combined volume The product unit extracts information of different scales in the input image, and introduces a feature fusion module in the decoder to make full use of the information output by each stage of the decoder; the segmentation network is trained and tuned on the data-enhanced dataset; grid processing is adopted The algorithm makes predictions on the entire pathological slide image and identifies areas of cancer within it. The invention introduces multi-scale information in the feature extraction stage, and improves the segmentation accuracy of the cancer area.

Description

Cancer region segmentation system in pathological slice images based on fully convolutional neural network

technical field

The invention relates to the technical field of image recognition, in particular to a pathological slice image cancer region segmentation system based on a full convolutional neural network.

Background technique

Before the popularity of neural networks, researchers have begun to use image processing technology to assist in the diagnosis of pathological slice images, mainly from the statistical features, texture features and morphological features of the image to extract features from the image to determine whether the image is in the image. Include certain structures and features and isolate them. Although traditional image processing methods can already play an auxiliary role in the diagnosis of doctors, there are generally problems such as low precision and unstable effect.

However, using neural networks to identify pathological slice images, most of the research only stops at judging whether there is cancer in the image, and does not specifically segment the area with cancer. The reliance on pathologists is still very heavy, and pathologists can only play a minor auxiliary role.

SUMMARY OF THE INVENTION

In order to overcome the defects and deficiencies of the prior art, the present invention provides a pathological slice image cancer region segmentation system based on a full convolutional neural network. The present invention introduces combined convolution into the network structure to fully extract different scales in the input image. Compared with the network that does not introduce different scale information, it has a better segmentation effect.

In order to achieve the above object, the present invention adopts the following technical solutions:

A pathological slice image cancer region segmentation system based on a full convolutional neural network, comprising: a tissue region extraction module, a sample data construction module, a sample data expansion module, a segmentation network construction module, a network training module, and a cancer region identification module;

The tissue region extraction module is used to extract the tissue region in the pathological slice image, obtain the mask of the tissue region, and remove the blank background;

The sample data building module is used to combine the extracted tissue area with the cancer area marked in the pathological section image, and cut the full-field pathological section image to obtain sample data for training;

The sample data expansion module is used to expand the sample data by using image enhancement;

The segmentation network building module is used to construct a Unet segmentation network with Resnet50 as the encoder, and replace the convolutional unit of the first-stage encoder of the encoder Resnet50 with a combined convolutional unit, and the combined convolutional unit is provided with a variety of The convolution of different size convolution kernels extracts the information of different scales in the input image, and introduces a feature fusion module in the decoder part to make full use of the information output by each stage of the decoder;

After the feature map output by each level of decoder is subjected to the upsampling operation, the convolution with different parameters is performed, and the overall feature map T is obtained by splicing on the channel;

The overall feature map T passes through a maximum pooling layer and an average pooling layer to obtain two vectors V ₁ and V ₂ , and the two vectors V ₁ and V ₂ are added through a shared parameter multilayer perceptron to obtain a vector C ;

The Sigmoid function is applied to the vector C and multiplied with the overall feature map T to obtain a feature map T' containing channel attention information;

After adding the overall feature map T and the feature map T' , the final segmentation result is obtained through the output convolution module;

The network training module is used for training and tuning the segmentation network on the data-enhanced dataset;

The cancer region identification module is used for using the trained model and based on the grid processing algorithm to predict the entire pathological slice image, and identify the cancer region therein.

As a preferred technical solution, the tissue region extraction module is used to extract tissue regions in pathological slice images, and the specific steps include:

Convert the original pathological slice image from RGB color space to HSV color space;

The segmentation threshold of the S component of the converted HSV image is obtained by using the Otsu algorithm;

The image is binarized based on the segmentation threshold to obtain the mask of the tissue area.

As a preferred technical solution, the pathological slice images of the full field of view are cut to obtain sample data for training, and the specific steps include:

The mask of the tissue region and the mask of the originally marked cancer region are phased together to obtain the final marked mask of the tissue region of the entire pathological slice image;

The original pathological slice and the final annotation mask are simultaneously cut to obtain image blocks and their annotations.

As a preferred technical solution, the sample data expansion module is used to expand the sample data by using image enhancement, and the specific steps include:

Image transformation is performed on the pathological slice image itself and its annotated image, and the image transformation includes any one or more of horizontal flipping, random angle rotation, color change, noise interference and blurring.

As a preferred technical solution, the combined convolution unit adopts 4 convolutions of different sizes, including 3*3, 5*5, 7*7 and 1*1 convolution kernels, and the input image passes through these 4 convolutions respectively. product, and then concatenate the 4 convolution results on the channel as the output of the final combined convolution.

As a preferred technical solution, the output convolution module includes a 3*3 convolution and a sigmoid activation function.

As a preferred technical solution, the multilayer perceptron includes a 1*1 convolutional layer, a ReLu activation function and another 1*1 convolutional layer for compressing information and generating on-channel attention results.

As a preferred technical solution, the cancer region identification module is used to predict the entire pathological slice image based on the grid processing algorithm using the trained model and identify the cancer region. The specific steps include:

The whole pathological slice image is divided into grids by grid algorithm, and each grid is used as a prediction sample;

Use a sliding window to take the image blocks in the grid each time and input the segmentation network to obtain the segmentation result;

The segmentation results of each sliding window are spliced to obtain the segmentation results of the entire pathological slice image.

Compared with the prior art, the present invention has the following advantages and beneficial effects:

(1) The present invention introduces combined convolution in the network structure to fully extract the feature information of different scales in the input image. Compared with the network without introducing different scale information, it has a better segmentation effect. Segmentation accuracy of the region.

(2) When generating the segmentation result, the present invention extracts the output of the 5-level decoder respectively and uses the feature fusion module to perform feature fusion. Such processing can make full use of the information such as the spatial position in the low-level feature map and the information in the high-level feature map. Advanced semantic information makes the segmentation results more accurate.

Description of drawings

Fig. 1 is the realization flow schematic diagram of the pathological slice image cancer region segmentation system based on the fully convolutional neural network of the present invention;

2 is a schematic diagram of a fully convolutional network structure of the present invention;

3 is a schematic diagram of the implementation process of the grid processing algorithm of the present invention

Figure 4(a) is a schematic diagram of the original image of the entire pathological slice image used for prediction in the present invention;

Figure 4(b) is a schematic diagram of the labeling mask of the entire pathological slice image used for prediction in the present invention;

FIG. 4( c ) is a schematic diagram of the segmentation result of the entire pathological slice image according to the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

Example

This embodiment provides a pathological slice image cancer region segmentation system based on a fully convolutional neural network, including: a tissue region extraction module, a sample data construction module, a sample data expansion module, a segmentation network construction module, a network training module, and a cancer region identification module. module;

In this embodiment, the tissue region extraction module is used to extract the tissue region in the pathological slice image, obtain the mask of the tissue region, and remove the blank background;

In this embodiment, the sample data building module is used to combine the extracted tissue area with the cancer area marked in the pathological slice image, and cut the full-field pathological slice image to obtain sample data for training;

In this embodiment, the sample data expansion module is used to expand the sample data by using image enhancement;

In this embodiment, the segmentation network building module is used to construct the Unet segmentation network with Resnet50 as the encoder, and the convolutional unit of the first-stage encoder of the encoder Resnet50 is replaced by a combined convolutional unit, the combined convolutional unit There are multiple convolution kernels of different sizes, extracting information of different scales in the input image, and introducing a feature fusion module in the decoder part to make full use of the information output by each level of decoder;

After the feature map output by each level of decoder is subjected to the upsampling operation, the convolution with different parameters is performed, and the overall feature map T is obtained by splicing on the channel;

The overall feature map T passes through a maximum pooling layer and an average pooling layer to obtain two vectors V ₁ and V ₂ , and the two vectors V ₁ and V ₂ are added through a shared parameter multilayer perceptron to obtain a vector C ;

The Sigmoid function is applied to the vector C and multiplied with the overall feature map T to obtain a feature map T' containing channel attention information;

After adding the overall feature map T and the feature map T' , the final segmentation result is obtained through the output convolution module;

In this embodiment, the network training module is used to train and optimize the segmentation network on the data-enhanced dataset;

In this embodiment, the cancer region identification module is used for using the trained model and based on the grid processing algorithm to predict the entire pathological slice image, and identify the cancer region therein.

As shown in FIG. 1 , in the specific implementation method of the pathological slice image cancer region segmentation system based on the fully convolutional neural network in this embodiment, the breast cancer data set disclosed in the Grand Challenge competition Camelyon16 is used in the specific implementation. The set contains 111 annotated breast cancer pathological section images containing cancer and 159 normal pathological section images, and the test set contains 129 pathological section images (of which only 48 have annotations of tumor regions), including the following steps:

S1, use Otsu algorithm and color space conversion, extract the tissue area in the pathological slice image, remove blank background; The concrete process of this embodiment is as follows:

S11. Convert the original pathological slice image from the RGB color space to the HSV color space;

S12, applying the Otsu algorithm to obtain the segmentation threshold of the S component of the converted HSV image;

S13. Binarize the image by using the threshold obtained in step S12 to obtain a mask of the tissue area. In the mask, the value of the area belonging to the tissue is 255, and the value of the area not belonging to the tissue is 0.

S2. Combine the extracted tissue area with the cancer area marked in the pathological section image, and cut the full-field pathological section image to obtain sample data for training. The specific processing process is as follows:

S21. In the original labeled cancer region mask, the value of the cancer region is 1, and the value of other regions is 0. In order to make the mask of the tissue area not affect the marked mask value, the tissue area mask obtained in step S1 and the original marked cancer area mask are bitwise ANDed, and then the labeling of the tissue area of the entire pathological slice image can be obtained. Mask, denoted as Mask, an area not 0 in Mask indicates that the area is both tissue and cancer, and the mask value of other areas is 0;

S22. Cut the original pathological slice image and the Mask obtained in step S21 at the same time to obtain small image blocks and their labels. The size of the image blocks is denoted as W, and the value of W can be 1024, 512, 256, etc. Efficiency and reducing computational complexity, the W used is 1024, but the cut image block is downsampled to make its size 256, which can ensure that the image block contains enough information and speed up the calculation;

S3. Use image enhancement technology to expand the sample data to improve the generalization performance of the model. The specific implementation process is as follows:

According to the fact that pathological slice images do not have a fixed direction, when the image is input to the network for training, image transformations such as horizontal flipping, random angle rotation, color change, noise interference and blurring are applied to the image itself and the label at the same time with a certain probability, so as to expand the sample. the purpose of the data.

S4, constructing a segmentation network that introduces information of different scales, and training and tuning. The specific process of this embodiment is as follows:

S41. Construct a Unet segmentation network with Resnet50 as the encoder. Its structure can be summarized as a structure in which the encoder and the decoder are connected. The decoder and the encoder each have 5 levels. ), each stage of the decoder consists of a deconvolution with a convolution kernel size of 2*2 and a stride of 2 and two layers of 3*3 convolutions to restore image resolution;

S42. Replace the convolution unit of the first-stage encoder of the encoder Resnet50 in the segmentation network constructed in step S41 (that is, the first residual module of the encoder Resnet50) with a combined convolution unit to fully extract the difference in the input image. The scale information has a gain in improving the segmentation accuracy; the combined convolution used in this embodiment consists of 4 convolutions of different sizes, such as 3*3, 5*5, 7*7 and 1*1 convolutions respectively. Kernel, the combination of convolution kernels of different sizes is not limited. The input image undergoes these 4 convolutions respectively, and then the 4 convolution results are spliced on the channel as the output of the final combined convolution;

S43. Introduce a feature fusion module into the decoder of the segmentation network constructed in step S42, and make full use of the information of the 5-level decoder to make the segmentation effect better. The process of introducing the feature fusion module is as follows:

；The feature maps output by the 5-level decoder are all up-sampled, so that the size of these feature maps is consistent with the final target output size, denoted as

;

5个特征图都经过参数不同的1*1卷积，主要作用是减少特征图的通道数量，可以起到降低计算量的作用，所获得的特征图记为

；obtained above

The 5 feature maps are all subjected to 1*1 convolution with different parameters. The main function is to reduce the number of channels in the feature map, which can reduce the amount of calculation. The obtained feature map is recorded as

;

5个特征图在通道上拼接获得一个整体特征图T；Will

5 feature maps are spliced on the channel to obtain an overall feature map T ;

Pass the overall feature map T through a maximum pooling layer and an average pooling layer to obtain two vectors V ₁ and V ₂ , and add the two vectors V ₁ and V ₂ through a shared parameter multi-layer perceptron to obtain A vector C , the multi-layer perceptron is a three-layer structure composed of a 1*1 convolutional layer, a ReLu activation function and another 1*1 convolutional layer, which is used to compress information and generate attention on the channel force result;

The Sigmoid function is applied to the vector C and multiplied with the overall feature map T to obtain a feature map T' containing channel attention information;

After adding the overall feature map T and the feature map T' , an output convolution module composed of a 3*3 convolution and a sigmoid activation function is used to obtain the final segmentation result.

S44. As shown in Figure 2, the encoders at all levels in the figure correspond to each residual module in the original Resnet50 respectively, and the improved segmentation network is trained and optimized on the data-enhanced data set. The specific implementation adopts the following scheme:

Use the Adam optimizer to update the model parameters, and use the exponential transformation method to update the learning rate. During the training process, monitor the mIoU index of the validation set. Every time the mIoU index is improved, the corresponding model parameters are saved for later use. Test and predict.

The test results of the training model are shown in Table 1 below. The mIoU index and the Dice index in the table are common indicators for segmentation task evaluation.

Table 1 Model test result table

S5, adopt the trained model and adopt the grid processing algorithm to predict the entire pathological slice image, and identify the cancer area therein, and the specific process of the present embodiment is as follows:

S51. As shown in Figure 3, the entire pathological slice image is divided into small blocks by using a grid algorithm to form a grid, and each small grid is used as a prediction sample. The specific steps are as follows:

Assuming that the size of the sliding window is W, and the size of the overlapping area of each sliding window is O, then the moving step size of the sliding window is S=W-O;

Calculate the number of image blocks n _x and _ny that can be obtained in the two directions of length X and width Y of the original pathological slice image according to the step size S, and the calculation formulas are:

Divide the length X and width Y of the entire pathological slice image according to n _x and _ny , and save the coordinate points in X _a and Y _a , X _a =[ X ₁ , X ₂ ,…, X _n ], Y _a =[ Y ₁ , Y ₂ ,…, Y _n ];

According to the starting point coordinates ( X _i , Y _i ) of each window and the size of the sliding window, the small image blocks in the sliced image are obtained. The coordinates of the upper left corner of each small cell in the grid are the indices when the small image blocks are cut. If the size of the image block exceeds the boundary of the sliced image (right and lower boundaries), move the starting point of the image block to the left or up, so that the image block beyond the boundary of the sliced image is exactly the same as the sliced image. The borders match.

S52, adopting the sliding window to take a small block in the grid each time to input the network to obtain the segmentation result;

S53. As shown in Figure 4(c), combined with Figures 4(a) and 4(b), splicing the segmentation results of each sliding window to obtain the segmentation results of the entire pathological slice image. Segmentation results of pathological section images.

The above-mentioned embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited by the above-mentioned embodiments, and any other changes, modifications, substitutions, combinations, The simplification should be equivalent replacement manners, which are all included in the protection scope of the present invention.

Claims (7)

1. a pathological slice image cancer region segmentation system based on full convolutional neural network, is characterized in that, comprises: tissue region extraction module, sample data building module, sample data expansion module, segmentation network building module, network training module, cancer area identification module; The tissue region extraction module is used to extract the tissue region in the pathological slice image, obtain the mask of the tissue region, and remove the blank background; The sample data building module is used to combine the extracted tissue area with the cancer area marked in the pathological section image, and cut the full-field pathological section image to obtain sample data for training; The mask of the tissue region and the mask of the originally marked cancer region are bitwise added to obtain the final marked mask of the tissue region of the entire pathological slice image; Cut the original pathological slice and the final annotation mask at the same time to obtain image blocks and their annotations; The sample data expansion module is used to expand the sample data by using image enhancement; The segmentation network building module is used to construct a Unet segmentation network with Resnet50 as the encoder, and replace the convolutional unit of the first-stage encoder of the encoder Resnet50 with a combined convolutional unit, and the combined convolutional unit is provided with a variety of The convolution of different size convolution kernels extracts the information of different scales in the input image, and introduces a feature fusion module in the decoder part to make full use of the information output by each stage of the decoder; After the feature map output by each level of decoder is subjected to the upsampling operation, the convolution with different parameters is performed, and the overall feature map T is obtained by splicing on the channel; The overall feature map T passes through a maximum pooling layer and an average pooling layer to obtain two vectors V ₁ and V ₂ , and the two vectors V ₁ and V ₂ are added through a shared parameter multilayer perceptron to obtain a vector C ; The Sigmoid function is applied to the vector C and multiplied with the overall feature map T to obtain a feature map T' containing channel attention information; After adding the overall feature map T and the feature map T' , the final segmentation result is obtained through the output convolution module; The network training module is used for training and tuning the segmentation network on the data-enhanced dataset; The cancer region identification module is used for using the trained model and based on the grid processing algorithm to predict the entire pathological slice image, and identify the cancer region therein. 2. The pathological slice image cancer region segmentation system based on a full convolutional neural network according to claim 1, wherein the tissue region extraction module is used to extract the tissue region in the pathological slice image, and the concrete steps include: Convert the original pathological slice image from RGB color space to HSV color space; The segmentation threshold of the S component of the converted HSV image is obtained by using the Otsu algorithm; The image is binarized based on the segmentation threshold to obtain the mask of the tissue area. 3. the pathological slice image cancer region segmentation system based on full convolutional neural network according to claim 1, is characterized in that, described sample data expansion module is used for adopting image enhancement to expand sample data, and concrete steps comprise: Image transformation is performed on the pathological slice image itself and its annotated image, and the image transformation includes any one or more of horizontal flipping, random angle rotation, color change, noise interference and blurring. 4. The pathological slice image cancer region segmentation system based on a fully convolutional neural network according to claim 1, wherein the combined convolution unit adopts 4 convolutions of different sizes, including 3*3, 5* For the convolution kernels of 5, 7*7 and 1*1, the input image undergoes these 4 convolutions respectively, and then the 4 convolution results are spliced on the channel as the output of the final combined convolution. 5. The full convolutional neural network-based pathological slice image cancer region segmentation system according to claim 1, wherein the output convolution module comprises a 3*3 convolution and a sigmoid activation function. 6. The pathological slice image cancer region segmentation system based on a fully convolutional neural network according to claim 1, wherein the multilayer perceptron comprises a layer of 1*1 convolution layer, a ReLu activation function and another Layer 1*1 convolutional layer for compressing information and generating on-channel attention results. 7. The pathological slice image cancer region segmentation system based on a fully convolutional neural network according to claim 1, wherein the cancer region identification module is used for adopting a trained model and based on a grid processing algorithm. The pathological section image is predicted to identify the cancer area, and the specific steps include: The whole pathological slice image is divided into grids by grid algorithm, and each grid is used as a prediction sample; Use a sliding window to take the image blocks in the grid each time and input the segmentation network to obtain the segmentation result; The segmentation results of each sliding window are spliced to obtain the segmentation results of the entire pathological slice image.

Images