CN116524315A - A Method for Recognition and Segmentation of Lung Cancer Pathological Tissue Slices Based on Mask R-CNN - Google Patents

Abstract

The invention belongs to the technical field of medical image processing, and in particular relates to a disease identification and segmentation method for pathological tissue slices of lung cancer. A method for identifying and segmenting lung cancer pathological tissue slices based on Mask R-CNN, comprising the following steps: S1, obtaining scanned images of patient lung cancer pathological tissue slices, and performing preprocessing; S2, inputting the preprocessed scanned images into In the pre-trained disease classification and segmentation model, the slice disease category is judged, and the visualization class activation map of lesion area segmentation is obtained; the disease classification and segmentation model is an improved Mask R-CNN neural network; S3, according to the acquired Visualize the class activation map and calculate the proportion of the lesion area to the global pathological tissue slice. The invention has the advantages of high classification accuracy, smooth region segmentation, accurate quantitative calculation, multi-indicator image analysis, and assisting doctors to conduct pathological research and judgment more quickly, conveniently and accurately.

Description

A Method for Recognition and Segmentation of Lung Cancer Pathological Tissue Slices Based on Mask R-CNN

technical field

The invention belongs to the technical field of medical image processing, and in particular relates to a disease identification and segmentation method for pathological tissue slices of lung cancer

Background technique

Lung cancer is the disease with the largest number of cancer deaths in China, which seriously threatens the health of Chinese residents. Histopathological examination is the most reliable basis for diagnosis. Doctors conduct biopsy through H&E stained pathological tissue sections, which is the "gold standard" for diagnosing lung cancer and distinguishing its type and severity. With the development of computer recognition technology, digital pathology technology combines the current digital imaging system with traditional optical imaging devices to provide doctors with higher resolution, higher definition, and more stable imaging conditions for the diagnosis and analysis process.

Medical image processing is one of the hot research fields in the world today. The emergence of computer-aided diagnosis has reduced the workload of doctors, but the traditional "expert" system still needs to manually extract lesion features, and its system development cycle is long. Development costs are high. With the development of deep learning, the field of image processing has made great progress. In the recognition competition of lung tumor nodules (CT image data set) held by Kaggle in 2017, the average recall rate (AR) reached 89.7%. He Kelei designed an end-to-end multi-instance deep convolutional network based on prototype learning, which realized the weakly labeled environmental noise filtering recognition of lung cancer pathological cell images. Existing research work focuses on tumor classification and segmentation, which has poor interpretability. However, with its unique medical ethics, it is still too early for machine learning to replace manual conclusive diagnosis at this stage. Therefore, adding multi-dimensional evaluation indicators for doctors in the auxiliary diagnosis system and designing explanatory functions that fit pathology will help doctors to provide more accurate and convenient diagnostic references.

For routine evaluation of pathological changes in tissue sections, a 4-level method (ie mild, mild, moderate, and severe) is often used. This is a classic method for evaluating tissue lesions, and it is also the current mainstream. However, with the popularization of whole-slice scanning and quantitative analysis concepts, quantitative analysis of tissue lesions has gradually become popular. Sometimes, data obtained after quantitative analysis can more accurately reflect the actual extent and extent of the lesion. In addition, the obtained data are also more convenient for statistical analysis of differences between groups. Area measurement is an important indicator in quantitative analysis. The existing measurement software needs to manually mark the lesion area to calculate the area. The present invention integrates qualitative analysis and quantitative measurement functions to create a smart medical assistance system.

In recent years, convolutional neural network is one of the most popular methods applied in the field of image processing. Deeply integrating convolutional neural network with pathological images and designing specific functions and evaluation indicators in combination with pathological characteristics is a direction worth studying in the future.

Contents of the invention

The purpose of the present invention is to provide a method based on deep learning for the identification, segmentation and quantitative calculation of lung cancer pathological tissue slices in order to address the shortcomings of the above existing methods, and apply the Mask R-CNN model in image recognition to images in stages Combined with the regression algorithm of quantitative calculation, the disease classification and lesion area positioning of lung cancer pathological tissue slices are realized, and the area occupied by the global medical record tissue slices is calculated. The invention has the advantages of high classification accuracy, smooth region segmentation, accurate quantitative calculation, multi-indicator image analysis, and assisting doctors to conduct pathological research and judgment more quickly, conveniently and accurately.

In order to solve the above-mentioned technical problems, the technical solution adopted in the present invention is: a method for identifying and segmenting lung cancer pathological tissue slices based on Mask R-CNN, comprising the following steps:

S1, obtaining the scanned image of the patient's lung cancer pathological tissue slice, and performing preprocessing;

S2, inputting the pre-processed scanned image into the pre-trained disease classification and segmentation model, judging the slice disease category, and obtaining the visualization class activation map of lesion area segmentation; the disease classification and segmentation model is improved Mask R-CNN neural network;

S3. Calculate the area ratio of the lesion area to the global pathological tissue slice according to the obtained visualization class activation map.

Further, in the S1, the specific method of the pretreatment is:

The scanned images of lung cancer pathological tissue slices were magnified by 20×, and converted from TIFP format to jpeg format.

Further, the improved Mask R-CNN neural network specifically includes:

A feature extraction network, the feature extraction network includes an improved residual network ResNet; the improved residual network ResNet adds a fully connected layer and a dropout layer before the last classification layer;

FPN network, add FPN network in described feature extraction network, carry out multi-scale fusion to extraction feature;

The RPN network is used to generate the target area for the features after FPN fusion, and input the set number of candidate areas with the highest score value into the Mask R-CNN network;

The Mask R-CNN network classifies the input candidate regions for disease classification and the segmentation of the lesion area, and generates the segmentation Mask of the background and lesion area.

Further, in said S3, the method for calculating the area ratio of the lesion area to the global pathological tissue slice includes:

(1), Gaussian blur processing is performed on the obtained visualization activation image, the gray level of the lesion area is set to 0, and the gray level of the background area is 255;

(2) Traverse the image pixels, count the pixels of the lesion area, and calculate the area and proportion of the lesion area.

Further, the Gaussian blur processing uses a normal distribution to calculate the transformation of each pixel in the image, and the two-dimensional space normal distribution equation is:

Among them, (u, v) is the two-dimensional coordinates of image pixels, r is the blur radius, and σ is the standard deviation of the normal distribution.

Further, the training data of the lesion recognition model is obtained by the following method:

(1) Segment the full-scan pathological tissue slice image after 20× magnification, and convert from TIFP format to jpeg format.

(2) According to the classification instructions, all images were divided into four categories: normal, lung adenocarcinoma, lung squamous cell carcinoma and lung small cell carcinoma;

(3) For the lesion area, apply labelme software to carry out manual segmentation;

(4) Divide all data into training set, verification set and test set according to the ratio of 8:1:1.

The lung cancer pathological tissue section identification and segmentation method provided by the present invention is based on the improved Mask R-CNN neural network, which has important reference significance for improving the accuracy of lung cancer diagnosis, and its beneficial effects are reflected in the following aspects:

In the present invention, the image information of tumor pathological tissue slices of patients in the medical image database is applied to lung cancer lesion segmentation and disease identification through the improved Mask R-CNN neural network, and the classification and segmentation results obtained by automatic learning are realized, and normalized The transformed lesion area image and its corresponding binarized mask image. The lesion feature extraction network extracts relevant geometric and morphological feature parameters as a reference for subsequent quantitative calculation of lesion area and proportion, assisting pathologists to improve the detection efficiency of lung cancer disease identification and the accuracy of tumor differentiation assessment. In addition, the present invention greatly reduces the reading time of clinicians and relieves the pressure of human resource shortage.

Description of drawings

Fig. 1 is a schematic diagram of the system structure of the present invention;

Figure 2 is a schematic diagram of the ResNet structure with improved generalization capabilities;

Fig. 3 is a schematic diagram of a neural network structure for disease identification and lesion region segmentation;

Fig. 4 is a flow chart for calculating the area ratio of the binarized lesion area.

Detailed ways

In order to facilitate the understanding of the present invention, the present invention will be described in more detail below in conjunction with the accompanying drawings and specific embodiments. Preferred embodiments of the invention are shown in the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described in this specification. On the contrary, these embodiments are provided to make the understanding of the present disclosure more thorough and comprehensive.

A method for identification and segmentation of pathological tissue slices of lung cancer based on Mask R-CNN provided in this embodiment, the process is shown in Figure 1, including the following specific steps:

1. Build a data set

The scanned image of the patient's lung cancer pathological tissue slice is obtained from the medical image database, and the acquired pathological tissue slice scanned image is preprocessed and marked to obtain the preprocessed image.

Among them, preprocessing mainly includes the following steps:

(1) Segment the full-scan pathological tissue slice image after 20 times magnification, and convert from TIFP format to jpeg format.

(2) Divide all images into four categories: normal, lung adenocarcinoma, lung squamous cell carcinoma, and lung small cell carcinoma according to the document description.

(3) Pathologists use labelme software to manually segment the lesion area. The obtained annotation information is saved in a json file

(4) Divide all data into training set, verification set and test set in proportion of 8:1:1 for training and testing models.

The labeling includes: (1) applying the json file containing the labeling information to generate a binary mask map of the lesion area of each image; (2) labeling the classification information of lung cancer.

2. Construction and training of disease classification and segmentation models

1. Feature extraction network

The feature extraction network selects the improved residual network ResNet, reduces the number of convolutional layers, and adds a fully connected layer and a dropout layer before the final classification layer to improve the generalization ability of the neural network. At the same time, FPN network is added to the feature extraction network , to perform multi-scale fusion of extracted features.

The feature extraction convolutional network shown in Figure 2 is an improved ResNet network. The first convolution part of the network consists of a convolution layer + BatchNorm layer + Relu activation layer + maximum pooling layer, where the convolution kernel size is 7x7, and the convolution kernel step size of the maximum pooling layer is 2. The second convolutional part of the network consists of 3 residual blocks ResidualBlock. Each residual block includes 1 1x1 convolutional layer + 1 3x3 convolutional layer + 3 BatchNorm layers + 3 Relu activation layers. For the first layer feature map of each residual block, you must Perform upconvolution to ensure that the extracted feature map size is consistent with the size of the second layer convolution feature map.

The FPN fused features are sent to the RPN network to generate the target area, and the candidate area with the highest score value (the number is set as a hyperparameter by itself) is input into the Mask R-CNN network and the frame regression operation is used to refine the position of the candidate frame. Get the final target box.

2. Based on the classification and segmentation neural network of Mask R-CNN, the classification of lesions and the segmentation of lesion areas are realized, as shown in Figure 3.

3. Input the pathological tissue image training set constructed above into the Mask R-CNN neural network for training, after verification and testing, a disease classification and segmentation model is obtained.

3. Disease identification and segmentation of pathological tissue slices of lung cancer

1. The scanned image of the obtained patient's cancer pathological tissue slice is magnified by 20 times, then segmented, and converted from TIFP format to jpeg format.

2. Input the above image into the disease classification and segmentation model

First, the improved ResNet network performs preliminary convolution to extract image abstract features; secondly, the FPN feature map pyramid network performs multi-scale fusion of multi-layer abstract feature maps; then, the FPN fused features are sent to the RPN network for target area Generate, use RoI Align to extract the features corresponding to each RoI on the full image features, and then classify through the fully connected layer. Parallel to fully connected layer classification is the segmentation task - RoI Align uses bilinear interpolation:

Among them, Q ₁₁ = (x ₁ , y ₁ ), Q ₁₂ = (x ₁₁ , y ₂ ), Q ₂₁ = (x ₂ , y ₁ ), Q ₂₂ = (x ₂ , y ₂ ) are used for interpolation positioning four points.

Lesion area detection target result refinement: Obtain the class score and coordinates of the recommended area with the highest score for each target recommended area, delete the recommended area with the highest score as the background, and eliminate the recommended area whose highest score does not reach the threshold. For the same category Non-maximum value suppression NMS is performed on the candidate frame of NMS, the -1 placeholder is removed from the frame index after NMS, the top n is obtained, and the information of each frame (y1, x1, y2, x2, Class_ID, Score) is returned at last.

Segmentation Mask generation of the lesion area image: The target recommended area is obtained as input and sent to the FCN network to output a 2-layer Mask. Each layer represents a different class, and the log output is used to perform binarization with a threshold to generate the background and lesion area. Split Mask.

3. Calculation of the proportion of lesion area

(1) Perform Gaussian blur processing on the image of "segmentation mask of background and lesion area" obtained above, set the gray level of the lesion area to 0, and set the gray level of the background area to 255.

Gaussian blur is an image blur filter that computes the transformation of each pixel in an image using a normal distribution.

The two-dimensional space normal distribution equation is:

Where (u,v) is the two-dimensional coordinates of image pixels, r is the blur radius, and σ is the standard deviation of the normal distribution. In a two-dimensional image, the contour line concentric circles that conform to the formula and are normally distributed from the center are convolved with the original corresponding pixels of the image, and each pixel value after convolution is the weight of the surrounding adjacent pixel values average.

(2) Traverse the image pixels, count the pixels of the lesion area, and calculate the area and proportion of the lesion area.

The grayscale image was binarized, the pixel value in the lesion area was 0, and the pixel value in the background area was 255. The lesion pixel count variable count is initially set to 0. Use a loop statement to traverse all pixels in the image and judge the gray value of each pixel. The process is shown in Figure 4, specifically:

Suppose for the pixel at (h_x,w_x):

When the pixel value value(h_x, w_x) is 0, count=count+1; otherwise, it does not count.

Finally, the lesion ratio ratio is obtained:

proportion=count/pic_shape

Where pic_shape is the image size.

Claims (7)

1. A Mask R-CNN-based lung cancer pathological tissue section identification and segmentation method is characterized by comprising the following steps of: the method comprises the following steps:

s1, acquiring a lung cancer pathological tissue section scanning image of a patient, and preprocessing;

s2, inputting the preprocessed scanning image into a pre-trained disease classification and segmentation model, judging the type of the slice disease, and obtaining a visual activation diagram for lesion region segmentation; the disease classification and segmentation model is an improved Mask R-CNN neural network;

s3, calculating the area proportion of the lesion area to the global pathological tissue section according to the acquired visual activation diagram.

2. The Mask R-CNN-based lung cancer pathological tissue section identification and segmentation method according to claim 1, wherein the method comprises the following steps: in the step S1, the specific method for preprocessing is as follows:

the scanned image of lung cancer pathological tissue section of patient is magnified 20 times and converted into jpeg format.

3. The Mask R-CNN-based lung cancer pathological tissue section identification and segmentation method according to claim 1, wherein the method comprises the following steps: the improved Mask R-CNN neural network specifically comprises:

a feature extraction network comprising an improved residual network, res net; the improved residual error network ResNet is added with a full connection layer and a dropout layer before the last classification layer;

the FPN network is added into the feature extraction network, and multiscale fusion is carried out on the extracted features;

the RPN network is used for generating target areas of the features after the FPN fusion and inputting a set number of candidate areas with the highest score values into the Mask R-CNN network;

and (3) classifying the input candidate areas by using a Mask R-CNN network, and dividing the lesion areas to generate a segmentation Mask of the background and the lesion areas.

4. The Mask R-CNN-based lung cancer pathological tissue section recognition and segmentation method according to claim 3, wherein: in the step S3, the area ratio calculation method of the lesion area to the global pathological tissue section comprises the following steps:

(1) Performing Gaussian blur processing on the obtained visual activation image, setting the gray level of a lesion area to be 0 and setting the gray level of a background area to be 255;

(2) And traversing the image pixels, counting the pixels of the lesion area, and calculating the area and the duty ratio of the lesion area.

5. The Mask R-CNN-based lung cancer pathological tissue section identification and segmentation method according to claim 4, wherein the method comprises the following steps of: the Gaussian blur processing calculates the transformation of each pixel in the image by using normal distribution, and a two-dimensional space normal distribution equation is as follows:

where (u, v) is the two-dimensional coordinates of the image pixel point, r is the blur radius, and σ is the standard deviation of the normal distribution.

6. The Mask R-CNN-based lung cancer pathological tissue section recognition and segmentation method according to any one of claims 1 to 5, wherein: training data of the lesion recognition model is obtained by adopting the following method:

(1) The full-scanning pathological tissue slice image is segmented after 20 times magnification, and is converted from a TIFP format to a jpeg format;

(2) Classifying all images into four types of normal lung adenocarcinoma, lung squamous carcinoma and lung small cell carcinoma according to classification;

(3) Manually segmenting a lesion area by using labelme software;

(4) All data are scaled 8:1:1 into a training set, a validation set and a test set.

7. The Mask R-CNN-based lung cancer pathological tissue section identification and segmentation method according to claim 6, wherein the method comprises the following steps: the preprocessed image also needs to be marked, including: (1) Generating a binary mask map of a lesion area of each image by applying a json file containing labeling information; and (2) marking lung cancer classification information.