CN113192047A

CN113192047A - Method for automatically interpreting KI67 pathological section based on deep learning

Info

Publication number: CN113192047A
Application number: CN202110529098.XA
Authority: CN
Inventors: 蔡佳桐; 杨林; 祝骋路; 吴同
Original assignee: Hangzhou Diyingjia Technology Co ltd
Current assignee: Hangzhou Diyingjia Technology Co ltd
Priority date: 2021-05-14
Filing date: 2021-05-14
Publication date: 2021-07-30

Abstract

The invention discloses a method for automatically interpreting KI67 pathological sections based on deep learning. The method comprises the following steps of firstly, collecting data and marking; secondly, manufacturing a true standard; dividing all the labeled data into a training set, a verification set and a test set; thirdly, designing a model architecture and a loss function, and training the model until convergence; fourthly, model prediction and post-processing are carried out; fifthly, evaluating the model performance; and sixthly, analyzing the model output, and visualizing the result. The main purpose of the invention is to position and distinguish all cells in the visual field under a KI67 section lens, then count positive tumor cells and negative tumor cells respectively, and finally calculate the KI67 value, wherein the value is used as the judgment standard of KI67 pathological section.

Description

Method for automatically interpreting KI67 pathological section based on deep learning

Technical Field

The invention relates to the technical field of machine learning, in particular to a method for automatically interpreting KI67 pathological sections based on deep learning.

Background

In recent years, with the development of artificial intelligence and machine vision technologies, digital image processing and artificial intelligence assist doctors to interpret pathological images in clinical application are becoming more and more widespread. The method adopts a deep learning model and combines the marks of doctors with rich experience, so that the pathological data can be interpreted with high accuracy. KI67 is a common immunohistochemical technique in the pathology department, and generally uses a KI67 index as an evaluation index for measuring the malignancy degree of tumors, and the index has diagnosis and prognosis values in various cancers. The KI67 index may be obtained by analyzing KI67 immunohistochemical pathology images. In clinic, a doctor selects a region with more positive cells on a pathological section by moving a microscope visual field, calculates the ratio of the positive tumor cells to all the tumor cells in not less than ten under-lens visual fields, and then takes the average of the ratios as a standard for judging pathological images. However, it is time-consuming and laborious for the clinician to count the tumor cells in the under-the-lens field, and it is very boring and tedious for the clinician to make misjudgment due to fatigue.

Disclosure of Invention

The invention aims to provide a method for automatically judging KI67 pathological sections based on deep learning, which comprises the steps of positioning all cells in a visual field under a KI67 section lens and judging the types of the cells, then respectively counting positive tumor cells and negative tumor cells, and finally calculating a KI67 value which is used as a judgment standard of the KI67 pathological sections.

In order to achieve the purpose, the invention provides the following technical scheme: a method for automatically interpreting KI67 pathological sections based on deep learning comprises the following steps:

firstly, collecting visual field data under a KI67 section lens, wherein the data volume is not less than 100, and labeling the data by a professional pathologist;

secondly, making a true standard, and analyzing the marked file of marked data into a smooth Gaussian mask as the true standard of the deep learning model, wherein the mask is in smooth transition from the center of the cell to the edge; dividing all the labeled data into a training set, a verification set and a test set;

thirdly, designing a model architecture and a loss function, and training the model until convergence;

fourthly, model prediction and post-processing are carried out, the test set is input into a trained model to obtain a prediction result thermodynamic diagram, and finally, Gaussian smoothing and local peak value processing are carried out to obtain a result which is used as a model to be output;

fifthly, evaluating the model performance, matching the model prediction cells with the labeled cells by adopting a Hungarian method, and measuring accuracy rate, recall rate and F1-score index;

and sixthly, analyzing the model output, and visualizing the result.

Preferably, the labeling in the first step is a dot labeling.

Preferably, in the first step, all the collected data are amplified with a uniform magnification, all the collected data cannot be from the same case, and the collected data of the same case does not exceed ten.

Preferably, the mask in the second step is represented by a floating point number in the range of [0,1], and the operation is to perform gaussian smoothing on each labeled point in the image, and the gaussian superposition part takes the maximum value to prevent the peaks from merging.

Preferably, in the third step, the model architecture selects a deep learning model rescet 34 as an unet model of the encoder, the learning rate is 5e-4, the optimizer adopts an Adam optimizer, the batch size is 8, the maximum iteration number is 500 epochs, the maximum iteration number is gradually attenuated after 100 epochs, the model is trained until convergence, the loss function is composed of two parts, and one part is a cross entropy loss function:

some are the Jaccard coefficients:

wherein X and Y are respectively the network output and the true standard mask, and the integral loss function is the sum of the two.

Preferably, in the fourth step, the trained model is used for predicting a test set, and thermodynamic diagrams of various categories are output;

the thermodynamic diagram is a three-dimensional image, wherein the first two dimensions respectively represent the height and the width of the image, and the third dimension is a cell category, namely, each category of cells corresponds to a two-dimensional thermodynamic diagram;

processing the prediction output result of the model, screening all cell centers in the whole image, summing the cell category dimensions of the three-dimensional thermodynamic diagram to obtain a two-dimensional thermodynamic diagram of all cells, performing Gaussian smoothing on the thermodynamic diagram to eliminate isolated points predicted by the model, then using a maximum filter to search a local maximum value in the smooth image and returning all local peak value coordinates, wherein the local peak value coordinates correspond to the centers of the predicted cells in the whole image by detecting all cells predicted by the model;

and then, classifying the cell center to obtain a thermodynamic value of a three-dimensional thermodynamic diagram corresponding to the local peak value coordinate, wherein the third dimension of the thermodynamic diagram, namely the number of layers of the two-dimensional thermodynamic diagram, corresponds to the number of cell types, and the cell type corresponding to the highest thermodynamic value is the cell type to which the coordinate belongs.

Preferably, in the fifth step, the Hungarian method is adopted to match the model prediction cells with the annotation cells, so that each annotation cell is not matched with more than one prediction cell, and each prediction cell is not matched with more than one annotation cell; the matching range is a circular area with the center point of the marked cell as the center of a circle and 16 pixels as the radius, and the area beyond the range does not participate in matching; in the matching process, only whether the cells are effectively detected or not is considered, and the cell types are not taken as the matching standards;

and the evaluation indexes of KI67 index difference, MAE and RMSE of label and network prediction are adopted as evaluation indexes. The calculation formula for MAE and RMSE is as follows:

where pred represents model prediction, anno represents annotation or true annotation, and m represents cell number.

Compared with the prior art, the invention has the following beneficial effects:

according to the invention, pathological doctor labels are used as gold standards, and the prediction results of the training model are close to the doctor labels, so that manual work can be replaced, counting of various cells in the under-mirror visual field and calculation of KI67 indexes can be completed, and time cost and labor cost are greatly reduced for colleagues meeting clinical interpretation precision.

Drawings

FIG. 1 is an image of a field under a collected KI67 mirror;

FIG. 2 is a visualization diagram of the labeling result of the pathologist in FIG. 1;

FIG. 3 is a visualization of the true standard mask corresponding to FIG. 2;

fig. 4 is a structure diagram of UNET network;

FIG. 5 is a graph of the match of the model prediction results to the true criteria using the Hungarian algorithm;

FIG. 6 is a visualization graph of deep learning predicted cell localization and classification results;

FIG. 7 is a flow chart of the method of the present invention.

Detailed Description

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1 to 7, the present invention provides a technical solution: a method for automatically interpreting KI67 pathological sections based on deep learning comprises the following steps:

step one, data acquisition and doctor labeling: collecting field data under a KI67 section mirror, wherein the data volume is not less than 100; the acquisition requirements are as follows: all the collected data adopt a unified magnification, such as 20x mirrors or 40x mirrors; the cell contour is clear, and can not be too large to occupy the whole picture and too small to be distinguished by human eyes; in order to ensure the generalization of the model, the collected data can not come from the same case; the size of the collected image and the dyeing style of the pathological section do not need to be set; marking the data of all the acquired images meeting the requirements by a professional pathologist; the marking mode is point marking; the deep learning method takes the label of the pathology department doctor as the gold standard, so the clinical case department doctor is required to label the data; the point labeling method is defined as labeling points at the center of the cell without drawing the outline of the cell; the pathologist needs to exhaust the cells in the visual field under the marking mirror; the cell types are classified into 6 types, which are: negative fibers, lymph, negative tumors, positive fibers, positive tumors, and other cells, and can be stained sequentially with blue, yellow, green, brown, red, purple, with only one label per cell; the label file is exported after the pathologist exhaustively labels the complete picture.

Secondly, manufacturing a true standard; different from a common deep learning segmentation task which adopts 0 and 1 to respectively represent a background and a foreground, the mask in the invention is expressed by a floating point (float) number in a range of [0,1 ]; the specific operation is that every marking point in the image is subjected to Gaussian smoothing, and the Gaussian superposition part takes the maximum value to prevent the wave crests from merging; the mask after Gaussian smoothing is in a form of high central intensity and low edge intensity; not only can emphasize the center position of the cell, but also can approximate the edge of the cell; the closer the mask pixel value is to 1, the higher the probability that the pixel is the foreground, whereas, the closer the mask pixel value is to 0, the higher the probability that the pixel is the background; in FIG. 3, a total of 6 masks are used, corresponding to 6 cell types; the first row from left to right is: negative fiber cells, lymphocytes, negative tumor cells; the second row, from left to right, is: positive fibroblasts, positive tumor cells and other cells; and dividing a training set, a verification set and a test set, training the model on the training set until convergence, storing the model on the verification set to realize optimal model storage, and testing the model precision by adopting the test set.

Thirdly, selecting a deep learning model resnet34 as the UNET model of the encoder by the model architecture, wherein the learning rate is 5e-4, an Adam optimizer is adopted by the optimizer, the batch size is 8, the maximum iteration frequency is 500 epochs, the model is gradually attenuated after 100 epochs, and the model is trained until convergence;

the loss function is composed of two parts, one part is a cross entropy loss function:

some are the Jaccard coefficients:

And fourthly, model prediction and post-processing. And predicting the test set by the trained model, and outputting the thermodynamic diagrams of all classes. The thermodynamic diagram is a three-dimensional image in which the first two dimensions represent the height and width of the image, respectively, and the third dimension is the cell class. I.e. one two-dimensional thermodynamic diagram for each class of cells. And processing the output result of the model prediction, and screening all cell centers in the whole image. And summing the cell class dimensions of the three-dimensional thermodynamic diagrams to obtain the two-dimensional thermodynamic diagrams of all cells, performing Gaussian smoothing on the thermodynamic diagrams, eliminating isolated points predicted by the model, then using a maximum filter to search a local maximum value in the smooth diagrams, and returning all local peak value coordinates. This step is to detect all cells predicted by the model in the global map, with local peak coordinates corresponding to the center of the predicted cell. Subsequently, the cell centers are classified. And acquiring a thermodynamic value of a three-dimensional thermodynamic diagram corresponding to the local peak value coordinate, wherein the third dimension of the thermodynamic diagram, namely the number of layers of the two-dimensional thermodynamic diagram, corresponds to the number of cell types, and the cell type corresponding to the highest thermodynamic value is the cell type to which the coordinate belongs. The corresponding classes of all cells predicted by the full-map model can be obtained by the method.

And fifthly, evaluating the model performance. And matching the model prediction cells and the labeled cells by adopting a Hungarian algorithm, so that each labeled cell is not matched with more than one prediction cell, and each prediction cell is not matched with more than one labeled cell. The matching range is a circular area with the center point of the marked cell as the center and 16 pixels as the radius, and the area beyond the range does not participate in matching. In the matching process, only whether the cells are effectively detected or not is considered, and the cell class is not taken as the matching standard. According to the matching result, indexes such as Precision (Precision), Recall (Recall), Euclidean distance, F1-score and the like can be calculated. Further, we can calculate an index of classification for the matched result. FIG. 5 is a result of matching the model prediction result with the true standard using the Hungarian algorithm. Wherein the cell center and the predicted cell center are labeled with red line representation pair, the length of red line represents the distance between the two, green represents False negative, i.e. False negative, and blue represents False positive. Besides, the evaluation indexes of KI67 index difference, MAE and RMSE of the label and the network prediction are adopted as evaluation indexes. The calculation formula for MAE and RMSE is as follows:

And sixthly, analyzing the model output, and visualizing the result as shown in fig. 6. In order to demonstrate the output result of the model and facilitate comparison, a graph I is still selected and input into the model, and FIG. 5 is a visual graph of deep learning prediction cell positioning and classification results. The correspondence of color points to corresponding cell classes is consistent with fig. 1. Negative fibers (blue), lymph (yellow), negative tumors (green), positive fibers (brown), positive tumors (red) and other cells (purple). As can be seen, the cell localization and classification results predicted by the deep learning model are basically consistent with the labeling results of doctors.

Experiment and analysis:

47 images not participating in model training were used for testing, and the resulting model was represented as follows:

evaluation criteria	Detectable Rate (precision)	Recall rate (recall)	F1-Score	European distance (Unit: pixel)
					Cell detection	0.8686	0.8570	0.8628	5.59±3.17
Cell sorting	0.8668	0.8278	0.8397	—

TABLE 1

TABLE 2

As can be seen from tables 1 and 2, the model has higher detection rate, recall rate and F1-Score, the Euclidean distance is smaller, the predicted cell center of the model is closer to the true standard, the error between the index marked KI67 and the index predicted KI67 is within 1, and the MAE and the RMSE are both within 0.1. The model prediction result can meet the clinical requirements in both precision and stability.

Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims

1. A method for automatically interpreting KI67 pathological sections based on deep learning is characterized by comprising the following steps:

firstly, collecting visual field data under a KI67 section lens, wherein the data volume is not less than 100, and manually marking the data;

fifthly, evaluating the model performance, matching the model prediction cells with the labeled cells by adopting a Hungarian algorithm, and measuring accuracy rate, recall rate and F1-score index;

and sixthly, analyzing the model output, and visualizing the result.

2. The method for automatically interpreting KI67 pathological sections based on deep learning according to claim 1, wherein: the labeling in the first step is point labeling.

3. The method for automatically interpreting KI67 pathological sections based on deep learning according to claim 1, wherein: in the first step, all the collected data adopt a unified magnification, all the collected data can not come from the same case, and the collected data of the same case does not exceed ten.

4. The method for automatically interpreting KI67 pathological sections based on deep learning according to claim 1, wherein: in the second step, the mask is represented by a floating point number in a range of [0,1], the specific operation is to perform Gaussian smoothing on each marked point in the image, and the Gaussian superposition part takes the maximum value to prevent the wave crests from merging.

5. The method for automatically interpreting KI67 pathological sections based on deep learning according to claim 1, wherein: in the third step, a model architecture selects a deep learning model rescet 34 as an unet model of an encoder, the learning rate is 5e-4, an Adam optimizer is adopted by the optimizer, the batch size is 8, the maximum iteration frequency is 500 epochs, the model is gradually attenuated after 100 epochs, the model is trained until convergence, a loss function is formed by two parts, and one part is a cross entropy loss function:

some are the Jaccard coefficients:

6. The method for automatically interpreting KI67 pathological sections based on deep learning according to claim 1, wherein: in the fourth step, the trained model is used for predicting a test set, and thermodynamic diagrams of all categories are output;

7. The method for automatically interpreting KI67 pathological sections based on deep learning according to claim 1, wherein: in the fifth step, the Hungarian algorithm is adopted to match the model prediction cells with the labeled cells, so that each labeled cell is not matched with more than one prediction cell, and each prediction cell is not matched with more than one labeled cell; the matching range is a circular area with the center point of the marked cell as the center of a circle and 16 pixels as the radius, and the area beyond the range does not participate in matching; in the matching process, only whether the cells are effectively detected or not is considered, and the cell types are not taken as the matching standards;

using the marked and network predicted KI67 index difference, MAE and RMSE as evaluation indexes; the calculation formula for MAE and RMSE is as follows: