KR102495367B1 - Multiple instance learning for histopathology classification - Google Patents

Abstract

The present invention relates to a multi-instance learning method for histopathology classification, which is performed by at least one processor in a computing device or a computing network, and features a feature extraction model (Fθ(•)). to convert the ith slide-derived instance (pij) into a low dimensional embedding (gij), and after confirming the positivity of the instance (pij) using a binary classifier, the instance level of all bags (bags) An instance selection step of sampling the highest instance per slide for learning by classifying instance level probabilities, learning using the instances obtained in the instance selection step, and sequentially performing instance level learning and vowel level learning to final loss It may include a learning step of obtaining , and a soft assignment-based inference step of distributing vowel level embeddings (zi) to learned centroids using a kernel that detects a similarity between two points.

Description

Multi-instance learning method for histopathology classification {MULTIPLE INSTANCE LEARNING FOR HISTOPATHOLOGY CLASSIFICATION}

The present invention relates to a multi-instance learning method for histopathology classification, and more particularly, to a multi-instance learning method for histopathology classification capable of accurately predicting instance labels.

Recently, digitization of glass slides obtained using a whole-slide image (WSI) scanner into histopathological images has played an important role as a standard for cancer diagnosis in a clinical environment.

A single WSI is a very large volume of 100k pixels, and selective analysis needs to consider both analysis difficulty and time consumption.

In particular, due to the high computational cost and bias of observers' subjective judgments, automated and accurate analysis of WSI is suitable for improved diagnosis and better treatment strategies.

Deep learning has become a widely used solution and can yield improved results when provided with sufficient training data.

However, pixel-level analysis has been difficult and costly.

In order to solve this problem, learning of multiple instance learning (MIL)-based neural networks presents a solution that can alleviate the challenges of final WSI diagnosis without accurate interpretation.

For example, "Campanella, G., Hanna, M.G., Geneslaw, L., Miraor, A., Silva, V.W.K., Busam, K.J., Brogi, E., Reuter, V.E., Klimstra, D.S., Fuchs, T.J.: Clinical- grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine 25(8), 1301{1309 (2019)" etc. proposed methods for performing histopathology classification using MIL-based neural network learning.

However, because MIL's instance labels are ambiguous, learning strong instance embeddings is very difficult.

To solve this problem, "Hashimoto, N., Fukushima, D., Koga, R., Takagi, Y., Ko, K., Kohno, K., Nakaguro, M., Nakamura, S., Hontani, H., In Takeuchi, I.: Multi-scale domain adversarial multiple-instance cnn for cancer subtype classication with nonannotated histopathological images. arXiv preprint arXiv:2001.01599 (2020)", (1) learning an instance encoder based on a region sampled from WSI and (2) learning an aggregation model incorporating instance-level information for slide-level prediction using a learned instance encoder.

However, even when using the above two-step approach, although it is successful in some problem settings, this approach often fails when a large number of ambiguous instances are used during learning, and the feature does not match the actual label. There was a problem that was exacerbated in the second step of training the aggregation model because it does not represent

Therefore, accurate classification is difficult, and reliability of classification may be lowered.

An object to be solved by the present invention in view of the above problems is to provide a multi-instance learning method for histopathology classification capable of more accurate classification by changing an aggregation model.

In addition, it is to provide a multi-instance learning method for histopathology classification that follows the standard histopathology process.

More specifically, an object of the present invention is to provide an end-to-end model for histopathology classification, capable of assigning accurate instances and bag labels.

The multi-instance learning method for histopathology classification of the present invention to solve the above technical problem is a multi-instance learning method for histopathology classification performed by at least one processor in a computing device or a computing network, and includes a feature extraction model. (Fθ(ㆍ)) is executed to convert the ith slide-derived instance (pij) into a low dimensional embedding (gij), and after confirming the positivity of the instance (pij) using a binary classifier, all collections An instance selection step of sampling the highest instance per slide for learning by classifying instance level probabilities of bags, and learning using the instances obtained in the instance selection step, but instance-level learning and collection-level learning and a soft assignment-based inference step of distributing vowel level embeddings (zi) to a learned centroid using a kernel that detects a similarity between two points. there is.

The present invention has an effect of performing more accurate classification by improving the learning of bag features through center loss and also improving the uncertainty of instance labels.

In addition, by considering both instance-based MIL and embedding-based MIL, there is an effect of reducing false positives by improving classification performance.

1 is a framework of a multi-instance learning method for histopathology classification according to a preferred embodiment of the present invention.
Figure 2 shows the qualitative results of the present invention in two aspects: k patches, which are model samples per slide class, and the effect of the learned model on interpretability through segmentation.

In order to fully understand the configuration and effects of the present invention, preferred embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be implemented in various forms and various changes may be applied. However, the description of the present embodiment is provided to complete the disclosure of the present invention, and to completely inform those skilled in the art of the scope of the invention to which the present invention belongs. In the accompanying drawings, the size of the components is enlarged from the actual size for convenience of description, and the ratio of each component may be exaggerated or reduced.

Terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. The above terms may only be used for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a 'first element' may be named a 'second element', and similarly, a 'second element' may also be named a 'first element'. can Also, singular expressions include plural expressions unless the context clearly indicates otherwise. Terms used in the embodiments of the present invention may be interpreted as meanings commonly known to those skilled in the art unless otherwise defined.

In addition, the present invention relates to a method for performing histopathology classification by digitizing an image acquired by a whole slide image (WSI) scanner and learning the result.

The present invention provides a processor such as a microprocessor that performs the proposed framework, a storage means for storing digitized WSI, a memory means for temporarily recalling data when classifying and learning is performed, and a user's control command input. It can be performed in a computing device including an input means and a display means for displaying a classification result.

In addition to this, a WSI scanner may be further included.

The computing device may be a single device such as a personal computer, server, smart pad, or smart phone, as well as a network device including terminals that collect WSI data and a server that learns and classifies the collected WSI data.

That is, the present invention is a method using a computing system constituting a single or network, and can be particularly understood as a process processed by a processor.

Therefore, even if there is no special mention, the performer of each step constituting the present invention may be a normal processor although there may be differences in expression.

1 is a framework of a multi-instance learning method for histopathology classification according to a preferred embodiment of the present invention.

Referring to this, the present invention includes an instance selection step (S10) of sampling k instances based on the highest predicted probability per slide, a learning step (S20) of learning using the instances obtained in the instance selection step (S10), Soft allocation based inference step for the learned center (S30).

Hereinafter, the configuration and operation of the present invention configured as described above will be described in more detail.

First, some definitions are made prior to detailed description of the present invention.

The WSI dataset (D) is a data set and can be expressed as {S1, ..., Sn}.

Here, each Si (i is 1 to n, n is a positive integer) is a WSI image having a label (yi), and the label is assumed to have a value of 0 or 1.

Si, which is data included in the WSI dataset, may have m instances.

That is, Si can be expressed as {s _i1 , s _i2 , ..., s _im }.

where m is a positive integer less than or equal to M, the total number of instances obtained in non-background areas within the slide.

In the present invention, in order to assign the correct label (yi) to each slide, only set labels can be used during learning, with 1 indicating positive and 0 indicating negative.

Multi-instance learning (MIL) must meet the following conditions:

- If data (Si) is negative, then all instances of Si are negative. That is, if the instance label yi is 0, ∀(yij) is 0.

- If data (Si) is positive, then at least one instance of Si is positive. That is, if the instance label yi is 1, Σyij is greater than or equal to 1.

In order to satisfy these conditions, the present invention proposes a learning model (CNN model) that can perform both instance level discrimination and slide level classification in the end-to-end framework shown in FIG. .

First, in the instance selection step (S10), the feature extraction model (Fθ(•)), which is a neural network, is executed to convert the i-th slide-derived instance into a low dimensional embedding (gij).

The low-dimensional embedding gij can be expressed as Fθ(s _ij ).

The conversion at this time is performed through the instance branch L _I including the shared embedding module Eθ.

The classification module then outputs the positivity of the instance pij.

At this time, the output of positivity uses a binary classifier (H _I ), and as mentioned above, 1 indicates positive, and 0 indicates negative.

A binary classifier (H _I ) classifying an instance (pij) can be expressed as H _I (pij).

Then classify the instance level probabilities of all bags to get the top instance per slide for training.

That is, in the present invention, all of the data D is not used, but the selected k pieces of data Dk are used. We call this top-k instance selection.

At this time, the selected data Dk is a subset of the data D.

In this process, F, E and θ, which is a parameter of the vowel module (B) to be described later, are not updated or stored like other modules.

In other words, in the instance selection step of the present invention, when each training starts based on the expected highest probability per slide through the instance module, k instances are sampled.

Then, using the obtained k instances, a learning step (S20) is performed.

The learning step (S20) of the present invention includes instance level learning and vowel level learning.

That is, learning is performed considering both the instance level and the collection level.

First, instance-level learning obtains embeddings (gi _ij ) after total mean pooling through a neural network (Fθ) for k instances inputs given in each learning step.

The obtained embedding (g _ij ) is supplied to the shared embedding module (Eθ), and a predicted value for the instance classifier (H _I ) using cross entropy can be obtained.

At this time, the k number of instance inputs (s _ij ) are included in the set of the k number of data (Dk).

Each instance is assigned a vowel level label (y), and the above condition is satisfied and used to calculate instance loss (L _I ).

The formula for calculating instance loss can be expressed as Equation 1 below.

In addition, in pyramidal bag-level learning (Bθ), three feature maps obtained in the instance-level learning are used in addition to the previously obtained embedding (g _ij ).

The three feature maps are low-dimensional embeddings, instances, and instance-level probabilities of bags.

The three feature maps are input to convolutional blocks corresponding to feature map sizes having a serial input size of [512, 256, 128], and features reduced to a single channel can be obtained.

Then upsampling the map using linear interpolation to match the size of the previous block and concatenate it with the previous feature map to obtain the final platen feature (zi). A single spatial map is obtained with k channels used to

Then, the final platen feature (zi) is input to the vowel classifier (H _B (zi)) through the shared embedding module (Eθ).

)은 교차 엔트로피를 사용하여 모음 로스(L_B(

,y)을 구하는데 사용된다.vowel prediction (

) is the vowel loss (L _B (

,y) is used to find

The equation for obtaining the bag loss is as shown in Equation 2 below.

In addition, the present invention introduces the concept of center loss in order to improve the discriminative ability of deep features.

Center loss characterizes intra-class variation by learning embeddings that minimize instance distances in the same collection.

The center loss can be expressed by Equation 3 below.

In Equation 3, c _yi is the center-dip feature of the yij th class of the same dimension as the embedding (g _ij ), and u is the mini-batch size.

In the present invention, the class center is initialized from a standard normal distribution and parameterized by the center module ( _Cθ ( )) trained with the center loss (L _C ) and the vowel loss (LB ).

Intuitively, instance embeddings of the same collection should have similar features that can be clustered into similar points in the embedding space.

Class centers are updated based on instance embeddings in mini-batch, not the full data set.

The final loss function can be expressed as Equation 4.

In Equation 4 above, α, β, and λ are balance values of the loss, respectively.

Then, in the soft assignment-based inference step (S30), considering the vowel embedding obtained through the vowel level learning for accurate classification, it is necessary to distribute an accurate label as a final diagnosis.

To do this, instance-embeddings of the same collection must match a single centroid representing the collection label.

The vowel level embedding (zi) for the slide (Si) is B{gi1, ... , gik}, and the vowel level embedding (zi) is the learned center through Equation 5, a kernel that detects the similarity between two points. (learned centroid).

In Equation 5, qi is the probability of distributing the vowel level embedding (zi) to the class center, and ψ is the degree of freedom of the Student distribution.

In the above example, the label of each slide is determined based on the distance between the predicted value obtained from the center loss part and the center loss, and as another example, the predicted value obtained from the vowel level loss can also be determined.

In addition, when learning is performed in the present invention, a label in an instance level unit is not included, but a prediction value can be obtained in an instance unit.

If instances can be predicted, malignant areas (tumors) can be automatically detected.

An example of an experiment for performing histopathology classification using the present invention configured as described above and the result thereof will be described below.

- Data sets and settings

As an example of the present invention, two small intestine cancer (CRC) data sets collected from an anonymous medical center scanned under different scanning conditions are prepared.

The data set contains normal and malignant tissue slides stained with Hematoxylin and Eosin and scanned at x40 magnification on different scanners.

CRC is the third most common cancer in humans and a common cause of death in both men and women.

In this data set, malignant slides contain microsatellite instable (MSI) CRC, a molecular phenotype due to defective DNA.

Specialist pathologists use immunohistochemical analysis (IHC) and PCR-based amplification to detect and treat MSI. Determination of MSI status in CRC has prognostic and therapeutic implications.

Among the two data sets, the first data set consists of a total of 173 WSIs, including 59 normal and 114 malicious WSIs, and the second data set consists of 85 normal WSIs and 108 malicious WSIs, totaling 193 WSIs.

In the present invention, each data set is divided into non-overlapping sets of 40%, 10%, and 50% of the total for training, verification, and testing.

Then, after converting to HSV color space for each WSI, "Otsu, N.: A threshold selection method from gray-level histograms. IEEE transactions on systems, man, and cybernetics 9(1), 62{66 (1979 )" to remove non-tissue areas.

Patch candidate locations are randomly selected for extraction per each slide during each training and validation.

During training and inference, the number k of instances is set to 50, and instances of size 256X256 are used.

"He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770{778 (2016) "'s ResNet-34 was used as the refinement and feature extraction model (Fθ).

In addition, instance and vowel classifiers (H _I , H _B ) fully connected layers were used.

The number of features of the embedding and central modules is set to 512, the entire framework is trained end-to-end, and the training rate is set to 1e ^-4 at 40 times.

In addition, the balance of the loss α, β, and λ are set to 1.0, 0.01, and 0.01, respectively. That is, the balance value can be set to any positive decimal value.

- Comparison method

In order to explain the effect of the present invention, the classification result is the latest technology in the art to which the present invention belongs, "Campanella, G., Hanna, M.G., Geneslaw, L., Miraor, A., Silva, V.W.K., Busam, K.J., Brogi , E., Reuter, V.E., Klimstra, D.S., Fuchs, T.J.: Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine 25(8), 1301{1309 (2019)" . The above technology is abbreviated as Comparative Technology 1.

In addition, comparative technique 2 (Ilse, M., Tomczak, J.M., Welling, M.: Attention-based deep multiple instance learning. arXiv preprint arXiv:1802.04712 (2018)) and comparative technique 3 (Nazeri, K., Aminpour, A ., Ebrahimi, M.: Two-stage convolutional neural network for breast cancer histology image classication. In: International Conference Image Analysis and Recognition. pp. 717{726. Springer (2018)".

For fair comparison, the same backbone Fθ is used for all cases.

Comparative technique 1 and comparative technique 3 both use a two-step learning procedure of instance-level learning and slide-level aggregation.

Comparative technique 2 uses an end-to-end approach together with permutation invariant pooling based on an attention mechanism.

- Quantitative results

Table 1 below is a performance comparison table of the first data set classification results and comparison techniques of the present invention.

As can be seen from Table 1 above, the present invention including the soft allocation-based reasoning step (S30) achieved the best results.

Comparison Technique 1 suffers from poor performance because it considers the probability of the top instance as the final slide.

In addition, when the present invention was evaluated using only vowel classification, it achieved a 9.97% improved performance compared to Comparative Technique 1, and the assignment-based approach showed an improvement of 15.56%.

In particular, comparative technique 2 was the best way to show the advantage of attention-based aggregation among the compared techniques. However, the performance can be further improved by applying the collection module of the present invention than described in the comparative technology 2 document.

In most cases, soft assignment is an excellent alternative to vowel classifiers. In the present invention, we discussed that the learned center has the maximum information among other similar instance embeddings, and the soft assignment-based inference step (S30) It shows more powerful performance compared to other methods used.

Table 2 is a performance comparison table of the second data set classification results and comparison techniques of the present invention.

As shown in Table 2 above, the performance of the present invention is significantly higher than that of comparable technologies.

Although the RNN-based aggregation of Comparative Technique 1 is the best in this set, the present invention shows a 1.28% improvement over it.

Also, most methods appeared higher in the second data set compared to the first data set. This is because performance may appear different due to differences in scanning protocols.

The color normalization method was not used in the preprocessing because it follows the standard protocol introduced in Comparative Technique 1 using the recent normalization technique.

- Qualitative results

Figure 2 shows the qualitative results of the present invention in two aspects: k patches, which are model samples per slide class, and the effect of the learned model on interpretability through segmentation.

When making decisions to assist experts in clinical research, it is advantageous to look at the model's areas of focus.

Therefore, the present invention visually verifies the effect of the present invention by collecting patches with a k value of 5 with high and low positives predicted by the instance classifier.

In particular, the cases predicted with the lowest values all correspond to actual normal tissue, while the ones with high probability are areas of malignant tumors clustered together.

This indicates that the model can accurately classify ambiguous labels in each slide.

In addition, by performing patch-by-patch classification on the entire slide using the trained model, it is possible to obtain a heat map showing regions with a high tumor probability with a threshold value.

In particular, these predictions are in exact agreement with the expert's comments.

This is because at each stage of the present invention, sampling is used to accurately select suspicious cases and avoid misidentification of negative decisions.

These results verify the effectiveness of the present invention and indicate a good balance between instance and meeting feature learning.

Embodiments according to the present invention have been described above, but these are merely examples, and those skilled in the art will understand that various modifications and embodiments of equivalent scope are possible therefrom. Therefore, the true technical protection scope of the present invention should be defined by the following claims.

Claims (8)

A multi-instance learning method for histopathology classification performed by at least one processor in a computing device or computing network, comprising:
After executing the feature extraction model (Fθ(ㆍ)), the ith slide-derived instance (pij) is converted into a low-dimensional embedding (low dimensional embedding, gij), and the positiveness of the instance (pij) is confirmed using a binary classifier. , instance selection step of sampling the top instance per slide for learning by classifying instance level probabilities of all bags;
A learning step of learning using the instances obtained in the instance selection step, and obtaining a final loss by sequentially performing instance-level learning and vowel-level learning; and
Including a soft assignment-based inference step of distributing vowel level embeddings (zi) to learned centroids using a kernel that detects a similarity between two points,
In the learning phase,
The instance-level learning is performed to obtain an instance loss, the bag loss is obtained by performing the bag-level learning based on the instance-level learning, and the bag loss is obtained based on the instance-level learning and the bag-level learning. A multi-instance learning method for histopathology classification, characterized in that by obtaining a center loss with , and obtaining the final loss through the instance loss, the vowel loss, and the center loss. delete According to claim 1,
A multi-instance learning method for histopathology classification, characterized in that processing so that features of instances extracted from the same collection are similar using the center loss. According to claim 1 or 3,
The instance loss,
A multi-instance learning method for histopathology classification represented by Equation 1 below.
Equation 1

L _I is the instance loss, yij is the instance label, pij is the instance According to claim 1 or 3,
The vowel Ross,
A multi-instance learning method for histopathology classification represented by Equation 2 below.
Equation 2

L _B is the vowel loss, yi is the instance label,

is vowel prediction According to claim 1 or 3,
The center loss,
Multi-instance learning method for histopathology classification represented by Equation 3 below.
Equation 3

L _C is the center loss, g _ij is the embedding, c _yi is the center-dip feature of the yij-th class of the same dimension as the embedding (g _ij ), u is the center-dip of the yij-th class of the same dimension as the mini-batch size characteristic According to claim 1 or 3,
The final loss is,
A multi-instance learning method for histopathology classification represented by Equation 4 below.
Equation 4

L is the final loss, L _I is the instance loss, L _B is the vowel loss, L _C is the center loss, and α, β, and λ are the balance of the loss, respectively, and are arbitrary positive decimal values. According to claim 1 or 3,
The kernel for detecting the degree of similarity between two points in the soft assignment-based inference step is a multi-instance learning method for histopathology classification represented by Equation 5 below.
Equation 5

qi is the probability of distributing vowel level embeddings (zi) to class centers, ψ is the degree of freedom of the Student distribution

Images