CN112580748B - Method for counting classified cells of stain image - Google Patents

Abstract

The invention relates to a method for counting classified cells of a stain image, which comprises the steps of firstly manufacturing a precisely classified cell data set, then training a model by adopting a target detection method in deep learning so as to identify a first cell and a second cell in the current stain image, and simultaneously analyzing a detection result and optimizing the model so as to obtain more accurate number of the first cell. Comparing the detection result with the manual annotation of a pathologist, the F1 score of the first A-type cells in the current staining image of the first-type cells can reach about 95%, and the F1 score of the first B-type cells is about 91%.

Description

A Method for Differential Cell Counting of Stained Images

technical field

The present invention relates to deep learning technology and target detection technology in computer vision.

Background technique

As an important part of image understanding, object detection is one of the core problems in the field of machine vision. Convolutional Neural Network CNN is one of the basic tools of deep learning and is usually used for image analysis. Convolutional neural networks such as VGG, GoogleLenet, resnet, etc., have shown excellent performance in object detection and semantic segmentation. Unlike image classification, object detection requires locating objects within an image. Object detection based deep learning models can be divided into two stages. One stage is to generate region proposals, and the other stage is to classify regions and provide confidence for each object. Some related methods include Fast R-CNN, and the improved Faster R-CNN, SPP, R-FCN and Mask R-CNN. Deep learning has achieved unprecedented performance on a variety of tasks, especially in the biomedical field. Furthermore, end-to-end detection methods based on deep learning, such as SSD, YOLO and RON, etc. The size, location and label of objects can be directly predicted without any intermediate steps, which improves the speed of detection compared to the two-stage Faster-RCNN. Despite the attractive qualities of CNNs, it is necessary to make the training set large enough.

Ki-67 proliferation index is an important biomarker of cancer cell proliferation, which is closely related to tumor differentiation, invasion, metastasis and prognosis. Quickly obtaining an accurate Ki-67 index is of great significance for clinical research. The Ki-67 index is the ratio of the number of positive cancer cells to the total number of cancer cells. However, due to the extremely similar morphology and color of the nuclei of various types of cells in the Ki-67 staining images, some traditional methods will regard non-tumor cells as non-tumor cells. tumor cells, resulting in a large number of count errors. Ruihan Zhang et al. of Nanjing University of Aeronautics and Astronautics used GAN network (Generation Adversarial Network), a successful new method for generative model training, by generating more artificial samples for data enhancement, combined with CNN and SSD to improve Ki67 accuracy. Dayong Wang et al. divided the entire segmented breast cancer images into patches and classified them according to the patches. Saha et al. established an automatic scoring system Ki-67 using a Gamma mixture model GMM with expectation maximization for seed point detection, patch selection and deep learning, resulting in a final precision of 93% and a recall rate of 88%. Practical analysis of Ki-67 shows that limited labeled datasets may lead to insufficient CNN, which in turn leads to overfitting of the training set and affects accuracy.

Many computerized methods rely on color features to detect and classify cells for Ki-67 scoring. Al-Lahham et al. first applied K-means clustering to the transformed color space and subsequently used mathematical morphology and connected component analysis to segment and count cells on Ki-67-stained histological images. Tumor cells are quantified using an image analysis system in which appropriate selection of color intensity thresholds is required. Markiewicz used a watershed algorithm to separate contact cells and a support vector machine (SVM) classifier to distinguish immunopositive cells from immunonegative cells. However, these methods cannot simultaneously accurately distinguish tumor and non-tumor cells and separate contact cells. Ki-67 images belong to immunohistochemical staining images (IHC). In recent years, the research on automatic nucleus segmentation of IHC staining images has recently attracted people's attention. Most of the related research focuses on image segmentation methods based on thresholding, edge detection or pixel classification based on machine learning. The pixel intensity thresholding method will utilize pixel intensities in the red, green and blue (RGB) color space and apply intensity transformation and global thresholding based on the difference between brown and blue. In both supervised and unsupervised learning methods, individual pixels are studied, while pixels in the same class collectively make up each component of the organization. Researchers need to select representative regions of each tissue component, including all cell types, as training samples before performing supervised classification, the performance of which largely depends on the quality and comprehensiveness of the pre-defined training samples. Since the application of deep learning in medicine is not very extensive, there is no dataset that can be directly used for Ki-67 stained images. How to effectively improve the detection index of Ki-67 stained images is still the focus of current research.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to propose a method for detecting the number of classified cells of a color image based on the deep learning and full supervision method.

The technical solution adopted by the present invention to solve the above-mentioned technical problems is a method for classifying cells of stained images, comprising the following steps:

1) Create a training set:

1-1) Collect the specimen image, manually mark part of the label image, and cut out small image patches on the area of the scanned whole film that has been marked; the labels used for labeling are divided into four categories: the first type A cells, the first type B cells Class cells, second class A cells and second class B cells; wherein the first class A cells and the first class B belong to the first class of cells, and the second class A cells and the second class B cells belong to the second class of cells;

1-2) Screen out patches with a high background ratio or blurred cells, and form a pre-training set of the filtered patches and their corresponding labels;

1-3) Input the pre-training set into the Libra-RCNN network model to complete the pre-training;

1-4) input a part of unlabeled specimen images into the pre-training network model obtained by pre-training for testing, and obtain the test results output by the pre-training network model;

1-5) Manually correct the test results output by the pre-training network model, and then screen out patches with a high proportion of background or blurred cells, and add the manually corrected and screened test results to the pre-training set to determine whether it meets the requirements. Pre-training end condition, if so, take the current pre-training set as the training set, and then enter step 2), otherwise, return to step 1-3);

2) Input the final perfect training set into the Libra-RCNN network model for training to obtain a training model;

3) Detection steps:

3-1) Select the region of interest ROI on the input scan full slice to be detected; the number of cells of the first type determined by machine learning can be used, and then the region with more cells of the first type can be selected as the ROI or freely selected by the doctor ROI;

3-2) Cut the slice patch in the selected ROI;

3-3) Detect and classify all patches to obtain the final detection results of all patches; the detection and classification process of each patch is as follows:

3-3-1) Use the trained model to detect each patch, and deduplicate the duplicate detection frame of the patch in the preset overlapping area overlap;

3-3-2) Traverse all detection frames on the current patch, and deduplicate detection frames between different classes, and obtain the final detection result of the patch after the traversal; wherein, the specific method of deduplication between detection frames between different classes is: The size of the intersection ratio of the IOU is calculated for the detection boxes of two different categories. When the IOU is larger than the preset threshold, the one with the lower confidence in the detection boxes of the different categories is deleted, and the detection frame with the higher reliability is retained;

4) The coordinates of all patch detection results are mapped to the entire scanning film, and the number of the first type of cells is calculated according to the number of the first type A cells and the first type B cells obtained by statistics. In the present invention, the final data set is formed by the steps of collecting specimens, sorting and screening, dividing patches, and re-screening. And the cell types of the dataset are divided into four categories, of which there are two types of cells of the first type and two types of cells of the second type. The detection of the second type of cells can not only make the discrimination of the first type of cells more accurate, but also can be used To expand the analysis of other medical indicators. The Libra-RCNN network is an improvement of the Faster-RCNN network, which greatly improves the detection performance. The problem that the same cell has different categories of detection boxes has not been solved in the detection results. In particular, a method of deduplication between detection boxes between different categories is proposed.

The beneficial effect of the invention is that detection and classification are accurate, which provides a basis for the accurate counting of the classified cells in the stained image, and the number of classified cells is accurately provided, which can provide better assistance for clinical medicine.

Description of drawings

Figure 1 shows the typical morphology of four types of cells;

Figure 2. Visualization of the detection results on the whole film;

Figure 3 compares the experimental data.

Detailed ways

The present invention will be described in further detail below according to the accompanying drawings and examples.

The examples apply the counting of sorted cells of the present invention. Among the four types of labels, the first type A cells correspond to positive cancer cells, the first type B cells correspond to negative cancer cells, the second type A cells correspond to lymphocytes, and the second type B cells correspond to stromal cells.

The invention mainly studies the Ki-67 staining images in breast cancer, and the cells in the Ki-67 staining images are roughly divided into two colors, brown and blue, with different shapes. Among them, positive cancer cells are generally brown, round in shape, and larger in size; negative cancer cells are generally blue, with a circular distribution, and the volume is the same as that of positive cancer cells; while normal cells (stromal cells, lymphocytes, etc.), Most are blue, a few are brown, and the shapes are different. We divided the cells into four categories, positive tumor cells, negative tumor cells, lymphocytes and stromal cells. The typical morphology of the four types of cells is shown in Figure 1. The typical lymphocytes and interstitial cells are mostly blue, and the morphology is easy to distinguish from cancer cells, but some special lymphoid and interstitial cells are very similar in shape to cancer cells. It is easy to be confused with cancer cells (especially negative cancer cells).

Fully supervised object detection methods, such as R-CNN, Fast R-CNN and Faster-RCNN, are two-stage based object detection methods: first extract the region of interest ROI, and then classify it. Subsequently, one-stage object detection methods also emerged, such as SSD, YOLOv2 and RetinaNet. Compared to the two-stage method, these methods will be faster, but not as accurate as the two-stage method. The present invention pays more attention to the size of the precision, so the two-stage network Libra-rcnn is selected. The overall network architecture of Libra-RCNN consists of two main modules: (1) Region Proposal Network (RPN), which returns ROIs in images; (2) Detection Network, which classifies objects within regions while performing bounding box regression . The anchors of RPN have three scales and three aspect ratios. Since the detection target is irregularly shaped cells, it is necessary to set multiple aspect ratios. The present invention uses three aspect ratios, namely 1:1, 1:2 and 2:1, depending on the size of the cells. At the same time, the ResNeXt model acts as a basic convolutional neural network to extract more complete and powerful feature maps from the input image.

The data set of the fully supervised network is particularly important, so the processing of the data must be more accurate. First, we selected the whole Ki-67 films with clear scans and different proportions of positive cancer cells. Segment the patch (small image) in the area marked by the pathologist, the size is 1024*1024, and set a certain overlap (overlapping part) according to your own needs. In order to reduce the labeling workload of pathologists, we first asked doctors to label three or four wsi (full films), segmented patches in the labeling area, screened out the patches with high background proportion and blurred cells, and sorted out all the patches. The patch performs preliminary training to obtain the training model, and then the training model is used to test more wsi, and the test results are sorted and returned to the doctor for modification. Perform fine tune, and so on.

The limited training set data will make the model overfit and reduce the accuracy. In order to increase the diversity of the data, we use some other dyed pictures to generate Ki-67 dyed images through style transfer, so as to achieve the purpose of enhancing the data. To achieve this, we employ CycleGAN, a network that enables unpaired image-to-image translation, learning a mapping function between two image domains X and Y from unpaired examples. Whereas the mapping G:X→Y and an inverse mapping F:Y→X are jointly learned using CNN. We train CycleGAN to learn a mapping function between source domain Xs and target domain Xt, converting other types of stained images to Ki-67 images. More data samples are generated through CycleGAN to achieve the purpose of data enhancement.

After that, the training model was started. With the increase of data diversity, a series of comparative experiments were carried out. The optimal model was selected for detection, and the detection results were visually analyzed to analyze the details of misdetected and missed cells. The problems are: 1. Missing detection of overlapping cells; 2. There are two detection boxes for the same cell; 3. The wrong cell type is detected. In the embodiment, changing nms to soft-nms alleviates the missed detection of overlapping cells, and for the second problem, the detection result is processed by the intersection ratio IOU, and the confidence levels of the two detection frames of the same cell are compared, Retain detection boxes with high confidence. The optimized detection results have improved by three or four percentage points. The last question covers the misdetection between different categories, which can be roughly divided into two cases: 1) Misdetection between negative cancer cells and normal cells; 2) Misdetection between positive cancer cells and lymphocytes; the first The first case is the main error case, and the second case is rare. These two false detection situations will be improved with the improvement of the data set, but the mitigation range is small, which is also an improvement point that can continue to improve the performance on the basis of the present invention.

Specific steps are as follows:

1) Create a training set:

1-1) Collect specimen images, manually mark part of the label images, and cut out small image patches on the area of the entire Ki-67 scan that has been marked;

1-2) Screen out patches with a high background ratio or blurred cells, and form a pre-training set of the filtered patches and their corresponding labels;

1-3) Convert images of dyeing methods other than Ki67 to Ki67 images through the CycleGAN network, so as to achieve the purpose of enhancing the pre-training set;

1-4) Input the pre-training set into the Libra-RCNN network model to complete the pre-training;

1-5) input a part of unlabeled specimen images into the pre-training network model obtained by pre-training for testing, and obtain the test results output by the pre-training network model;

1-6) Manually correct the test results output by the pre-training network model, and then screen out patches with a high background ratio or blurred cells, and add the manually corrected and screened test results to the pre-training set to determine whether the results are satisfied. The pre-training end condition, if so, take the current pre-training set as the training set, and then enter step 2), otherwise, return to step 1-4);

2) Input the final perfect training set into the Libra-RCNN network model for training to obtain a training model;

3) Detection steps:

3-1) Select the ROI of the region of interest on the input Ki-67 scan to be detected; the number of cancer cells determined by machine learning can be used, and then the region with more cancer cells can be selected as the ROI or the ROI can be freely selected by the doctor ;

3-2) Cut the slice patch in the selected ROI;

3-3) Detect and classify all patches to obtain the final detection results of all patches; the detection and classification process of each patch is as follows:

3-3-1) Use the trained model to detect each patch, and repeat the deduplication of the detection frame for the cells in the overlap when the patch is removed;

3-3-2) Traverse all the detection frames on the current patch, and deduplicate the detection frames of two different categories of the same cell, and obtain the final detection result of the patch after the traversal; among them, the detection frames between different classes are deduplicated The specific method is as follows: when calculating the size of the intersection and union ratio IOU for the detection boxes of different categories, when the IOU is larger than the preset threshold, delete the one with lower confidence in the detection boxes of different categories, and keep the reliability. higher detection frame;

4) Map the coordinates of all patch detection results to the whole Ki-67 scan, and calculate the Ki67 index according to the number of positive and negative cancer cells obtained by statistics.

Figure 2 shows the detection results on the whole Ki-67 film. Because the Ki-67 indicator is only related to cancer cells, we only label the cancer cells in the end. In order to be more clear, we convert the detection frame to dots on the cells, in which the dark black is positive cancer cells, and the light gray is negative cancer cells , the area arbitrarily framed in the figure is a rectangle (irregular areas are also supported). So what we end up giving the doctor is the Ki-67 indicator and the cancer cells we detect.

The detection evaluation index of the embodiment is F1 score. The F1 score is the harmonic mean of recall (recall rate) and precise precision), where recall=TP/(TP+FN) and precision=TP/(TP+FP). (TP: number of true positives, FP=number of false positives, FN=number of false negatives).

Comparing the detection results with the pathologist's manual annotation, in the Ki-67 images of breast cancer, the F1 score of positive cancer cells was around 95%, the F1 score of negative cancer cells was around 91%, and the F1 score of recall and The harmonic mean of precision. The specific indicators are shown in Figure 3, and the position of the detection frame is more accurate. The indicators of negative cancer cells are slightly lower than that of positive cancer cells. The reason is that negative cancer cells are more likely to be confused with normal cells, resulting in detection indicators lower than positive cancer cells. cell. The test results prove that it can be used initially to assist doctors in making clinical judgments.

Claims (2)

1. A method of classifying a cell count for a stain image, comprising the steps of:

1) Creating a training set:

1-1) collecting a specimen image, manually marking a part of label image, and cutting a small image patch on a marked scanning whole area; labels used for labeling fall into four categories: a first class a cell, a first class B cell, a second class a cell, and a second class B cell; wherein the first A-type cell and the first B-type cell belong to a first type of cell, and the second A-type cell and the second B-type cell belong to a second type of cell;

1-2) screening out the patch with high background occupation ratio or fuzzy cells, and forming a pre-training set by the screened patch and the corresponding label;

1-3) inputting the pre-training set into a Libra-RCNN network model to complete pre-training;

1-4) inputting a part of unlabelled specimen images into a pre-training network model obtained by pre-training for testing to obtain a test result output by the pre-training network model;

1-5) manually correcting the test result output by the pre-training network model, screening out the patch with high background occupation ratio or fuzzy cells, adding the manually corrected and screened test result into a pre-training set, and judging whether a pre-training finishing condition is met, wherein if yes, the current pre-training set is used as a training set, and then the step 2) is carried out, and if not, the step 1-3) is carried out;

2) Inputting the finally obtained perfect training set into a Libra-RCNN network model for training to obtain a training model;

3) A detection step:

3-1) selecting a region of interest ROI on the input scanning full sheet to be detected; according to the number of the first type cells judged by machine learning, selecting a region with more first type cells as an ROI or freely selecting the ROI by a doctor;

3-2) cutting slices patch in the selected ROI;

3-3) carrying out detection classification on all the patches to obtain the final detection results of all the patches; the detection classification process of each patch is as follows:

3-3-1) detecting each patch by using the trained model, and removing the duplicate of the repeated detection frame of the patch in the preset overlap region overlap;

3-3-2) traversing all the detection frames on the current patch, and performing duplication removal on the detection frames among different classes to obtain a final detection result of the patch after traversing is finished; the specific way of removing the duplicate of the detection frame among different classes is as follows: calculating the size of the intersection comparison IOU of every two different types of detection frames, and when the IOU is larger than a preset threshold value, deleting the detection frame with lower confidence level from every two different types of detection frames, and keeping the detection frame with higher confidence level;

4) And mapping all patch detection result coordinates to the whole scanning whole slice, and calculating the number of the first type cells according to the counted number of the first type A cells and the first type B cells.

2. The method of claim 1, wherein the specimen image includes a current stain image and a current stain image into which other types of stain images than the current stain image are converted, and the conversion of the other types of stain images into the current stain image is accomplished using a CycleGAN network.

Images