KR20220144687A - Dual attention multiple instance learning method - Google Patents

Abstract

The present invention relates to a multiple instance learning device for analyzing a 3D image, which comprises: a memory in which a multiple instance learning model is stored; and at least one processor electrically connected to the memory. The multiple instance learning model may comprise: a convolution block which derives a feature map of each 2D instance of an input 3D image; a spatial attention block which derives a spatial attention map of instances from the feature map derived from the convolution block; and an instance attention block which derives an attention score for each instance by receiving a synthesis result of the feature map and spatial attention map, and derives an aggregated embedding for the 3D image by aggregating embeddings of instances according to the attention score. The present invention provides an attention-based end-to-end weak supervised framework for rapid diagnosis based on multiple instance learning.

Description

DUAL ATTENTION MULTIPLE INSTANCE LEARNING METHOD

The present invention relates to a dual attention multi-instance learning method, and more particularly, to a multi-instance learning method using unsupervised complementary loss.

Chest Computed Tomography (Chest CT)-based diagnosis and analysis plays a key role in combating outbreaks of rapidly spreading pandemics worldwide, such as coronavirus disease 2019 (COVID-19).

However, accurate screening is challenging due to difficulties in annotating infected areas, classifying and structuring large datasets, and distinguishing minute discrepancies between the disease to be diagnosed and other diseases.

There is also the problem that the sensitivity of automated screening is limited and may not be equivalent to performance at the radiological level.

Therefore, there is an urgent need to develop and improve robust screening methods based on chest CT.

On the other hand, deep learning-based solutions can extract rich features from clinical datasets and have applications in wide fields such as long-term segmentation and disease diagnosis. The analysis has shown considerable success.

For example, Korean Patent Laid-Open Publication No. 2021-0030730 discloses a method for predicting lung cancer incidence using an artificial intelligence model that interprets a medical image and a medical image analysis apparatus.

However, although deep learning-based solutions show promising performance, most methods are supervised learning methods, and there is a problem that a considerable amount of labeling effort is required.

Accordingly, an unsupervised or weakly supervised learning method that does not require excessive data pre-processing and/or strong prior knowledge may be a preferred option for accurate diagnosis.

The problem to be solved by the present invention in consideration of the above problems is an attention-based end-to-end weak map for rapid diagnosis based on Multiple Instance Learning (MIL). It is to provide a framework.

In addition, the problem to be solved by the present invention is to provide an unsupervised contrastive learning method using attention applied in both spatial and potential contexts, and dual attention contrast based MIL (Dual Attention Contrastive based) MIL: DA-CMIL).

In addition, the problem to be solved by the present invention is to provide (a) patch-based, (b) slice-based, (c) 3D CT-based methods for a diagnosis decision of a disease.

In addition, the problem to be solved by the present invention is to provide a novel end-to-end attention-based weak supervised framework, and MIL and self-supervised contrastive learning for features for accurate disease diagnosis learning) is provided.

A multi-instance learning apparatus for analyzing a 3D image according to an embodiment of the present disclosure for solving the above technical problem includes a memory in which a multi-instance learning model is stored and at least one processor electrically connected to the memory, The multi-instance learning model is derived from a convolution block that derives a feature map of each instance that is a 2D slice/patch of an input 3D image, and a convolution block A spatial attention block for deriving a spatial attention map of the instances from the feature map, a spatial attention block for deriving a spatial attention map of the instances, and a result of combining a feature map and the spatial attention map are input to derive an attention score for each instance, and an instance according to the attention score An output block for outputting an analysis result for the 3D image based on an instance attention block (Latent Attention Block) that derives aggregated embedding for the 3D image by synthesizing the embeddings of the 3D image may include

In addition, the processor, in the training phase of the multi-instance learning model, using the training data including the 3D image labeled with the ground-truth for the analysis result, the overall loss function (L) of the learning model is the minimum value. An operation of training the learning model to have

Here, the overall loss function may be a combination of a bag level loss function ( _{LB ) for the result of the output block and a contrast loss function (L F )} between instance embeddings and sum embeddings.

The present invention enables more accurate analysis of 3D images while providing unsupervised or weakly supervised learning methods that do not require excessive data pre-processing and/or strong prior knowledge.

In addition, by using supervised learning and contrast loss together, the model of the present invention can avoid overfitting even when trained on smaller data sets and feature robustness at the instance level without sacrificing accuracy. ) can be improved.

In addition, the multi-instance learning method of the present invention may have robustness to different CT sizes when the instance (eg, slice/patch) count is changed.

1 illustrates a framework of a multi-instance learning apparatus and a multi-instance learning model according to an embodiment of the present disclosure.
2 is a diagram for explaining contrast learning applied to a multi-instance learning setting according to an embodiment of the present disclosure.
3 is an example of CT images that are pre-processed in a multi-instance learning method according to an embodiment of the present disclosure.
4 illustrates a graph comparing characteristics of a multi-instance learning model and other learning models according to an embodiment of the present disclosure.
5 illustrates a graph comparing the prediction performance of a multi-instance learning model according to an embodiment of the present disclosure and other learning models in a CT slice dataset.
6 illustrates a graph comparing the prediction performance of a multi-instance learning model according to another embodiment of the present disclosure and other learning models in a CT patch dataset.
7 illustrates a spatial attention map and an instance attention score derived from a multi-instance learning model according to an embodiment of the present disclosure.
8 shows CT slices represented in an embedding space in a multi-instance learning model according to an embodiment of the present disclosure.
9 is a flowchart illustrating a learning method of a multi-instance learning model according to an embodiment of the present disclosure.

In order to fully understand the configuration and effect of the present invention, preferred embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, and may be embodied in various forms and various modifications may be made. However, the description of the present embodiment is provided so that the disclosure of the present invention is complete, and to fully inform those of ordinary skill in the art to which the present invention pertains the scope of the invention. In the accompanying drawings, components are enlarged in size than actual for convenience of description, and ratios of each component may be exaggerated or reduced.

Terms such as 'first' and 'second' may be used to describe various elements, but the elements should not be limited by the above terms. The above term may be used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, a 'first component' may be termed a 'second component', and similarly, a 'second component' may also be termed a 'first component'. can Also, the singular expression includes the plural expression unless the context clearly dictates otherwise. Unless otherwise defined, terms used in the embodiments of the present invention may be interpreted as meanings commonly known to those of ordinary skill in the art.

In the multi-instance learning apparatus and method according to the present disclosure, a more accurate image analysis is possible by using a double attention block and using a contrast loss function in addition to a vowel level loss function.

The multi-instance learning apparatus and method according to the present disclosure can be used for various 3D image analysis, but below, the concept of the present invention is explained using the case of distinguishing pneumonia caused by corona and bacterial pneumonia by analyzing 3D CT images. do.

Multi-instance learning according to the present disclosure may be referred to as Dual Attention Contrastive based Multiple Instance Learning (DA-CMIL). DA-CMIL can receive patient CT slices (treated as a collection of instances) derived from a 3D CT image and output a single label.

Attention-based pooling can be applied to select key slices in the potential space, and spatial attention can learn the context of the slice space for interpretable diagnosis. Contrastive Loss may be applied at the instance level to encode similar features from the same patient CT relative to the features of the typically pooled patient CT.

The goal of DA-CMIL is to assign one category label (eg, corona pneumonia or bacterial pneumonia) to patient CT when the CT volume of multiple 2D slices is given as input.

In general, each patient CT scan can be thought of as a bag of instances, which can be either positive or negative. It would also be beneficial to identify which slides/instances contribute to the final patient diagnosis along with identifying the location of the infected area.

In the present disclosure, we will propose an attention-based permutation-invariant MIL method for pooling slices to obtain a single representative feature for patient CT.

Additionally, spatial attention can be applied together to learn spatial features for discovering an infected area. In the multi-instance learning method of the present disclosure, contrast learning may be used at the instance level in an unsupervised manner so that the patient-level aggregate feature and the instance feature from the same patient are semantically similar.

To this end, during training of the learning model, the unsupervised contrast loss can be used together with the patient category labels for the supervised loss.

Existing tasks using MIL applied to different domains separate instance-level learning and vowel-level learning into a two-step procedure. For example, we first train the instance level encoders, and then train the aggregate models for inference using the encoders trained with MIL pooling. However, training a robust encoder can be challenging due to the uncertainty and noise of the instance labels.

Therefore, the multi-instance learning model according to the present disclosure solves the above problem through longitudinal learning, and instance selection can be achieved through attention-based pooling of CT slices with model optimization focused only on accurate patient labels.

In the present disclosure, we propose a novel end-to-end model for the classification of corona pneumonia and bacterial pneumonia in a weakly supervised manner, and show that contrast learning of instance features and patient-level features is possible together in the MIL setting.

1 illustrates a framework of a multi-instance learning apparatus and a multi-instance learning model according to an embodiment of the present disclosure.

의 세트를 수신할 수 잇다.In this example, chest CT data set D = {S ₁ , ... , S _n }, the training model is modeled on m labeled example scans taken from a joint distribution defined as S × Y.

can receive a set of

where S _i is the patient CT scan with instances (eg, 2D CT slices or patches), Y is the label set of patient-level labels, { for binary classification of COVID-19 and other cases { 0, 1}.

In addition, S _i is S1 = {s ₁ , s ₂ , ... , s _N } may be considered a bag of instances, where N may represent the total number of instances of the collection.

It can be assumed that each instance s _n has a label y _n ∈ {0, 1}, but not all instances may be negative or positive.

Moreover, not all slices of the scan represent areas of infection that are important for diagnosis. This is because other slices may be noisy artifacts that are not useful for training.

Therefore, in this embodiment, the MIL can assume that if a bag S _i is negative, then all corresponding instances must be negative, and in the case of a positive bag, one or more instances must be positive. . This can be expressed as Equation (1).

Given that both sets of bags considered in this example (eg, COVID-19 pneumonia and other pneumonias) contain negative and positive instances (lesions), the above conditions are strictly may not be applicable. Therefore, in this embodiment, an attention mechanism can be applied to learn a corresponding label by considering a more relaxed version of the constraint, and implicitly assigning weights to instances.

In this embodiment, a CNN model for diagnosis at the patient CT scan level between COVID-19 pneumonia and another pneumonia in one end-to-end framework can be implemented. In this embodiment, a dual-attention multi-instance learning deep model using unsupervised contrast learning (DA-CMIL) may be applied.

As shown in FIG. 1 , in this embodiment, a CT scan with unlabeled instances is used as an input, and a key semantic representation can be learned. Also, in this embodiment, an attention-based pooling method may be used to transform patient instances into a single bag representation for a final prediction.

And in this embodiment, unsupervised contrast learning can be used to make instances of a bag semantically similar to a bag representative during training.

는 CT 모음(bag)의 j번째 인스턴스를 형상(shape) C×H×W의 공간 차수(spatial dimension)를 가진 저차원 임베딩

으로 변환하기 위한 특징 맵 추출기로 구현될 수 있다. 여기서, C는 채널의 크기, H는 높이, W는 너비를 나타낼 수 있다.Meanwhile, in the framework proposed in this embodiment, the backbone CNN model

is a low-dimensional embedding of the j-th instance of the CT bag with a spatial dimension of shape C×H×W.

It can be implemented as a feature map extractor for transforming into . Here, C may represent a size of a channel, H may represent a height, and W may represent a width.

에 제공될 수 있다.And in this embodiment, g _ij is a spatial attention module to learn spatial representative features, and to output spatial attention maps of size 1×H ^*× W ^* per instance with C = 1

can be provided on

,

을 얻을 수 있다. 여기서, D는 특징 맵의 치수(dimension size)를 의미할 수 있다.The maps obtained at this time can highlight key regions, and are used to weight the features of all initial instances, resulting in a single spatial pooling feature.

,

can get Here, D may mean a dimension size of the feature map.

In this case, a summary operation between the attention map and the backbone feature map may be achieved through Einstein summation.

을 집계(aggregate)하기 위해, 일관성을 위해 동일한 차원을 갖는 단일 모음(bag) 표현

,

를 얻기 위해 어텐션 기반 순열 불변 풀링(permutation invariant pooling)을 수행하는 두번째 모듈

를 구현할 수 있다.In this embodiment, instance characteristics for each CT scan

To aggregate , a single bag representation with the same dimension for consistency

,

A second module that performs attention-based permutation invariant pooling to obtain

can be implemented.

에 대한 예측을 얻기 위해 환자 레벨 분류기

로 전달될 수 있으며, 여기서

는 CT 스캔이 COVID-19 폐렴 또는 기타 폐렴으로 라벨링 될 확률을 의미할 수 있다.And in this embodiment, z _n is the whole bag.

A patient-level classifier to get predictions for

can be passed to, where

can mean the probability that a CT scan will be labeled as COVID-19 pneumonia or other pneumonia.

을 사용할 수 있다.Meanwhile, in this embodiment, as expressed in Equation 2, a bag loss using cross entropy

can be used

Dual Attention-Based Learning

Attention can be essential to learning robust features in a MIL setting. In particular, attention-based pooling may be preferred over existing pooling methods such as maximal or average because the existing methods are not differentiable or not applicable for end-to-end model update.

) 및 잠재 임베딩(

) 기반 어텐션 풀링을 모두 이용한다. 공간 모듈에서, 입력

가 주어지면, 각각 쌍곡 탄젠트(tanh) 및 시그모이드(sigm) 비선형성이 뒤 따르는 두 개의 컨볼루션 레이어를 사용할 수 있다.In this embodiment, spatial embedding (

) and latent embeddings (

) based attention pooling. In the spatial module, input

Given , we can use two convolutional layers followed by hyperbolic tangent (tanh) and sigmoid (sigm) nonlinearities, respectively.

In this embodiment, the feature maps g _ij may be passed successively to each module and then to the final convolutional layer with a single channel output indicating the presence of an infection.

을 얻기 위해 최종 레이어에 전달하기 전에 컨볼루션 레이어(또는 합성곱 계층)들의 각 분기(branch) 출력간에 요소 별 곱셈(element-wise multiplication)을 수행할 수 있다. 다음으로, 공간 스코어들은 소프트 맥스 연산에 의해 정규화될 수 있으며, 이는 높이와 무게 치수(dimension) 모두에서 총합 행렬 곱셈(예를 들어,

, 여기서

이며, 일관성을 위해

를

로 함)을 통해 얻은 최종 공간 풀링 특징 들을 의미할 수 있다.In particular, in this embodiment, spatial scores

, element-wise multiplication can be performed between each branch output of the convolutional layers (or convolutional layers) before being passed to the final layer to obtain . Next, the spatial scores can be normalized by a soft max operation, which is a summation matrix multiplication (e.g., in both height and weight dimensions)

, here

and for consistency

cast

It can mean the final spatial pooling features obtained through

At this time, it is worth noting that in this embodiment, gated spatial attention is implemented instead of global average pooling (GAP) which is generally applied to the initial backbone features.

In addition, initial normalized spatial maps can be used to visually show the area the model focuses on to make a decision.

을 집계하기 위해, 인스턴스 어텐션 모듈

에서 어텐션 기반 풀링을 사용할 수 있다. 즉 본 실시예에서는, 인스턴스 임베딩들에 어텐션이 적용되기 때문에 모든 컨볼루션 레이어가 풀리- 커넥티드(fully-connected) 레이어로 대체된다는 점을 제외하고는 초기 백본 특징 맵에서 게이트 공간 어텐션을 위해 이전에 적용된 동일한 아키텍처 디자인을 고려할 수도 있다. On the other hand, in this embodiment, the features

To aggregate , the instance attention module

Attention-based pooling can be used in That is, in this embodiment, since attention is applied to the instance embeddings, all the convolutional layers are replaced with fully-connected layers previously for gate space attention in the initial backbone feature map, except that The same architectural design applied may be considered.

,

를 N개의 인스턴스 특징들이 있는 모음(bag)으로 표현할 수 있다. 그리고 본 실시예에서는, 게이팅 메커니즘(gating mechanism)을 사용한 어텐션 기반 풀링 MIL을 수학식 3 및 수학식 4와 같이 나타낼 수 있다.In this embodiment,

,

can be expressed as a bag with N instance features. And in this embodiment, the attention-based pooling MIL using a gating mechanism can be expressed as Equations 3 and 4.

,

및

는 훈련 가능한 파라미터들이다. 그리고 tanh(*)와 sigm(*)는 요소별 비선형성을 나타낼 수 있으며,

는 요소별 곱셈을 나타낼 수 있다. 또한, a_n은 전체 모음(bag) 예측에 대한 주어진 인스턴스의 관련성을 나타내는 인스턴스당 어텐션 점수를 의미할 수 있다.here,

,

and

are trainable parameters. And tanh(*) and sigm(*) can represent nonlinearity for each element,

can represent element-wise multiplication. In addition, a _n may mean an attention score per instance indicating the relevance of a given instance to the prediction of the entire bag.

From a technical point of view, attention-based pooling allows you to assign different weights to instances, reducing the need for explicit instance selection. Also, the final bag expression can be more informative. That is, in this embodiment, enhanced training for learning powerful and interpretable features can be enabled through a synergistic combination of spatial and attention-based pooling.

contrastive MIL

2 is a diagram for explaining contrast learning applied to a multi-instance learning setting according to an embodiment of the present disclosure.

In this embodiment, in order to improve the learning of instance-level feature maps, in addition to the MIL method presented above, the unsupervised loss may be integrated and implemented. The model of this embodiment can learn a representation that maximizes agreement between the instance feature maps of the same patient and the aggregated bag feature map through contrasting loss values in the latent space. Fig. 2 can be said to show the overall concept of the technology applied in this embodiment.

According to the previously proposed self-supervising framework using contrast loss values, probabilistic data augmentation is applied to 2D data samples to generate two correlated views of the same example. Augmentation may include random cropping, color distortion and random Gaussian bluring, and the like.

In addition, control loss values can be applied to define control prediction tasks for unlabeled samples, and positive and negative pairs can be identified for a given sample. That is, in the present embodiment, since the contrast loss is applied to the latent space, the probabilistic data increase can be omitted. Also, in this embodiment, for a given patient CT scan, each slice may be considered a pseudo-enhancement of the overall patient characteristic. Thus, in this embodiment, it can be considered that stochastic augmentation is potentially applied, for example different views of the same patient.

In this embodiment, z` may be expressed as a latent instance-level feature of a patient, and z may be expressed as a patient bag-level feature obtained through the module proposed above. And after l <sub>2</sub> normalization of z` and z features, a contrast loss function can be defined as in Equation 5 below.

는 1(iff k ≠ i)로 산출되는 인디케이터(indicator) 함수(function)이고,

는 온도 파라미터(temperature parameter)를 나타낼 수 있다. 온도 파라미터는 유사도의 차이가 로스에 반영되는 정도를 조절하는 파라미터이며, 모델의 실시형태에 따라 경험적으로 임의의 숫자를 넣으며 찾아지는 최적변수이다. 그리고

은 유사도 함수(similarity function)일 수 있으며, 예를 들어, 코사인 유사도(cosine similarity)일 수 있다. 여기서, z_i, z_j는 i번째 패치(슬라이스)의 특징, j번째 패치(슬라이스)의 특징을 의미할 수 있다. 또한, N은 학습 페이즈에서 정의되는 배치사이즈이며, 도 2와 같이 두 데이터를 함께 비교하는 경우 배치사이즈가 2N이 되어 시그마에는 2N이 기재될 수 있다.

를 구하기 위해 각 인스턴스들(i=1, …, 총 인스턴스 수)과 각 모음(bag) 레벨 특징 맵들과 비교를 하게 되고 실질적으로 이 비교 값들을 모두 합한 값에 기초하여

가 결정될 수 있다. 손실(loss)은 모든 환자 슬라이스 특징들 및 각각의 모음(bag) 레벨 특징들에 걸쳐 계산될 수 있으며, 본 실시예에서는 미니 배치(mini-batch) 당 증강으로 간주될 수 있다. 본 실시예의 전체 프레임 워크의 최종 손실 함수(전체 손실 함수)는 다음 수학식 6과 같이 정의할 수 있다.here,

is an indicator function calculated as 1 (iff k ≠ i),

may represent a temperature parameter. The temperature parameter is a parameter that adjusts the degree to which the difference in similarity is reflected in the loss, and is an optimal variable found by empirically entering a random number according to the embodiment of the model. and

may be a similarity function, for example, cosine similarity. Here, z _i , z _j may mean a characteristic of an i-th patch (slice) and a characteristic of a j-th patch (slice). In addition, N is a batch size defined in the learning phase, and when comparing two data together as shown in FIG. 2 , the batch size becomes 2N, and 2N may be described in sigma.

Each instance (i=1, ..., total number of instances) and each bag level feature map are compared to obtain

can be determined. A loss may be computed over all patient slice features and each bag level features, and in this example may be considered an enhancement per mini-batch. The final loss function (total loss function) of the entire framework of this embodiment can be defined as in Equation 6 below.

는 모음(bag) 손실과 대조 손실의 기여도에 따라 가중치를 부여하는 파라미터이며 0과 1사이의 값이다.here,

is a parameter that is weighted according to the contribution of bag loss and contrast loss, and is a value between 0 and 1.

For a detailed algorithm for the above content, refer to the following algorithm table.

[Algorithm Table]

In this example, a chest CT dataset consisting of 173 samples was collected at Yeungnam University Medical Center (YUMC) located in Daegu, Korea. The dataset includes 75 CT samples for patients with COVID-19 and 98 CT samples for patients with bacterial pneumonia, collected between February and April 2020.

2D CT slices or patches are used as instances in the MIL framework according to this embodiment, and the learning model according to this embodiment is evaluated for both the slice case and the patch case. Additionally, the 3D CT volume dataset is processed to train and test 3D-based methods under full supervised settings.

For pre-processing, lung regions were segmented for all CT samples. For this, we can use the ResNeSt model for segmentation training and inference.

3 is an example of CT images that are pre-processed in a multi-instance learning method according to an embodiment of the present disclosure.

The upper images in Figure 3 are lung CT slices (three left) and patch samples (two rightmost) of a patient with pre-processed COVID-19 pneumonia, and the lower images in Figure 3 are pre-processed Lung CT slices (3 left) and patch samples (2 far right) from a patient with bacterial pneumonia.

All datasets were classified as data for training, validation and testing by patient ID and the ratios were 0.5, 0.1 and 0.4, respectively. The same classification was used for all dataset variants in all versions using only truncated lung regions. CT samples may be 512x512, 128x128 and 256x256x256 in size in slices, patches and 3D CT volume sets, respectively. Each CT slice can be resized from 512x512 to 256x256, and patch slices can be resized from 256 to 128. In particular, a set of slices may contain about 14,000 slices, and a patch version may contain 64,000 patches, which may primarily show more than 30% of lung tissue. For 3D CT volumes, all slices belonging to one patient can be used to build the volume using nearest neighbor smapling applied to obtain the desired input sizes.

로 이용되는 단일 풀리-커넥티드 레이어와 함께 특징 추출 모듈

로 사용될 수 있다. 피쳐들의 차원은 512로 고정될 수 있고 이는 C=512인 512×8×8을 가지는

로부터 획득되는 피쳐 맵들을 포함할 수 있다. 공간 풀링에 이어서, 피쳐들은 512로 다시 리쉐이핑될 수 있다.The model of the present disclosure may be implemented in Pytorch. ResNet-34 fine-tuned from ImageNet pretrained weights is a collection classifier

A feature extraction module with a single fully-connected layer used as

can be used as The dimensions of the features can be fixed at 512, which has 512×8×8 with C=512.

It may include feature maps obtained from Following spatial pooling, the features may be reshaped back to 512.

In order to evaluate the effectiveness of the method according to the present embodiment, comparison with recent MIL-based methods such as DeepAttention MIL, ClassicMIL and JointMIL is made. In addition, recent 3D-based methods, DeCovNet and Zhang3DCNN, are also included in the comparison. For a fair evaluation, the same backbone feature extractor is used except for the 3D methods in which publicly available implementations are used.

의 효과들에 대한 절삭 연구들이 이루어진다.Quantitative and qualitative results of the method proposed according to an embodiment of the present disclosure are presented below. In addition, vowel size, attention modules with and without contrast learning, weight parameters

Cutting studies on the effects of

를 가지는 DA-CMIL이 98.6%의 정확도 및 98.4%의 AUC(Area Under the curve)로 최고의 전체 성능을 달성함을 보여준다. 훈련 중에

가 적용되지 않는 경우에도, 본 개시의 실시예에 따른 모델이 93%(JointMIL 비교 +2.9)의 정확도 및 93.4%(JointMIL 비교 +2.5)의 AUC를 보여주어 가장 우수한 약한 방식의 지도학습 방법인 JointMIL보다 우수함을 알 수 있다. Table 1 shows control loss according to an embodiment of the present disclosure.

It shows that DA-CMIL with 98.6% accuracy and 98.4% AUC (Area Under the Curve) achieves the best overall performance. during training

Even when is not applied, the model according to the embodiment of the present disclosure shows an AUC of 93% (JointMIL comparison +2.9) and an AUC of 93.4% (JointMIL comparison +2.5), which is the best weak supervised learning method JointMIL It can be seen that the better

To further validate the method according to an embodiment of the present disclosure, DA-CMIL can be applied to randomly cut patches of CT samples. Referring to Table 2, it can be seen that even in the case of patches, the DA-CMIL method according to the embodiment of the present disclosure consistently shows better performance than the comparative methods.

4 illustrates a graph comparing characteristics of a multi-instance learning model and other learning models according to an embodiment of the present disclosure.

4 shows receiver operating characteristic (ROC) curves of methods compared on different datasets. 5 illustrates a graph comparing the prediction performance of the multi-instance learning model and other learning models according to an embodiment of the present disclosure. 6 illustrates a graph comparing prediction performance of a multi-instance learning model and other learning models according to another embodiment of the present disclosure.

Referring to FIG. 4 , as a whole, the method according to an embodiment of the present disclosure exhibits a higher True Positive Rate (TPR) and a lower False Positive Rate (FPR) in all settings. This can also be demonstrated in the summaries of the confusion matrices for the comparison methods shown in FIGS. 5 and 6 . 5 and 6, CP (Common Pneumonia) means general pneumonia, in this embodiment may mean bacterial pneumonia, and NCP means COVID-19 pneumonia.

, 및 학습상 이중 어텐션 모듈의 효과 Collection size, weight parameter

, and the effect of the dual attention module on learning

To evaluate the effect of bag size during training of the method according to an embodiment of the present disclosure, construct a collection while changing k when each collection is composed of up to k instances (slices/patches) and conduct cutting research. As shown in Table 3, it can be seen that the DA-CMIL performance also improves as the collection size increases.

를 가지는 다중 인스턴스들의 대조 특징 학습을 이용할 수 있다. 표 4를 참조하면, 가중치 파라미터

=1.0인 경우, 즉

를 사용하지 않는 경우,

는 학습에서 아무런 효과도 가지지 못하며

를 사용하는 경우에 비해 93%의 더 낮은 성능을 보인다. DA-CMIL is a weighting parameter to balance the effect of losses.

It is possible to use contrast feature learning of multiple instances with . Referring to Table 4, weight parameters

=1.0, i.e.

If you are not using

has no effect on learning.

93% lower performance compared to the case of using

In the framework of this embodiment, in order to evaluate the effect of attention, several settings in which both the contrast module and the attention module are used or not used (see Table 1 and Table 2) may be considered.

)의 풀링 없이, 단순성을 위해 인스턴스 특징 맵의 글로벌 평균 풀링(GAP)을 기본으로 사용하고, (2) (

)를 통한 어텐션 기반 모음(bag) 레벨 특징의 집계(aggregation) 없이, 인스턴스 특징들의 평균을 사용하여 z`와 함께 전체 모음(bag) 수준 특징 z를 얻을 수 있다. 이러한 수정 후 평가를 보다 쉽게 수행할 수 있을 것이다.In the framework of this embodiment, except for attention, modifications may be required in the following two aspects. (1) Spatial attention-based feature map (

), using the global average pooling (GAP) of instance feature maps as a basis for simplicity, (2) (

), without aggregation of the attention-based bag-level features, we can use the average of the instance features to obtain the entire bag-level feature z together with z'. This post-correction evaluation will be easier to perform.

와

가 학습의 일부일 때 최고의 성능을 달성함을 확인할 수 있다. 반면에 어텐션 모듈을 제외하면 전체 성능이 현저히 감소함을 확인할 수 있다(최고 성능 방법에 비해 -20%). 그리고 유사한 성능 저하가 CT 패치 데이터 세트에서도 나타날 수 있다. clearly,

Wow

It can be seen that the best performance is achieved when is part of the training. On the other hand, if the attention module is excluded, it can be seen that the overall performance is significantly reduced (-20% compared to the highest performance method). And a similar performance degradation can be seen in the CT patch data set.

It can be seen that using only the contrast feature loss without the attention module deteriorates the result without improving the performance compared to the comparison method. It can be confirmed that the combination of the techniques of the present embodiment (ie, both attention and feature map loss) has an advantage, and it can be confirmed that improved results are obtained through complementary learning.

Qualitative Results

7 illustrates a spatial attention map and an instance attention score derived from a multi-instance learning model according to an embodiment of the present disclosure;

Referring to FIG. 7 , a qualitative result may be confirmed based on each of the spatial attention maps and the attention score. This may show that DA-CMIL can use coarse maps to find major slices related to the infected area. At this time, a low attention score may be observed for slices such as noisy slices/artifacts without an infected region, further indicating the usefulness of the method of this embodiment. In addition, it can be seen that the attention maps focus on key areas such as ground-glass opacities and consolidations that are consistent with clinical outcomes.

In addition, attention maps can be highlighted when contrast learning is not applied. In general, the results can show a map similar to the case where the loss is applied. However, it can be seen that for some CT slices, both the spatial map and the score have slight changes, while the localization of the main region is particularly degraded due to the large difference in the attention score.

This can be largely predicted because the contrast loss is to encourage similarity between representative features of the subject. The benefits of using both losses can be better verified through quantitative evaluation of classification performance. That is, the attention mechanism of the present embodiment is relevant in both cases and can be very usefully applied to clinical evaluation.

Multi-focal ground-glass opacities (GGO) of the lateral side of the lower lobes are the most common early findings in CT, and other characteristics such as pleural thickening are can be observed less commonly in expression. This may be consistent with the fact that most spatial attention maps are mainly concentrated in lower regions.

In general, CAM (Class Activation Maps) may not indicate the exact lesion location due to a resolution problem. Moreover, in this embodiment, especially in a weakly supervised setting, it may be difficult to enforcing the model to generate a limited map of the tissue without access to the actual lesion location.

In this example, we are confident in our current results even if the map is normalized only to tissue regions, and it may be clear that the high-density regions are clinically relevant regions corresponding to lesions and/or GGOs.

RT-PCR is a standard method of diagnosing COVID-19, but it can take a long time to test and can take a lot of time to get results. Therefore, the examination method through CT, which can obtain results within a few minutes, can be considered as a reasonable alternative to the test method.

Accordingly, in this embodiment, it is possible to implement a new approach to deep learning (CNN) for diagnosing COVID-19 under weak supervision with clinical implication. For rapid evaluation, it is important to use methods that are fully automated and interpretable in the real world. Moreover, since COVID-19 and other pneumonias have very subtle differences, it is difficult for field experts to distinguish them, so accurate diagnosis is very important in terms of image characteristics.

The method of the present embodiment can be evaluated through a data set in which only patient diagnostic labels can be used without lesion-infected regions of interest, as is common in conventional methods.

In addition, in order to further verify the above-described approach of this embodiment, in this embodiment, a region focused by the model can be qualitatively identified through a coarse attention map together with an attention score. It can be seen that the method of this embodiment achieves an AUC of 98.4%, an accuracy of 98.6% and a true positive rate (TPR) of 96.8%. In addition, in the present embodiment, the obtained attention map may highlight the major infection areas in most samples together with the attention scores corresponding to the major pieces.

In addition, this embodiment can empirically demonstrate the advantage of using unsupervised contrast loss to complement supervised learning of patient labels, and can serve as a basis for more complex methods. In addition, it can be confirmed that the method of this embodiment greatly outperforms the 3D-based method. This may be due to the limited size of the 3D CT volume data set collected using a large cohort. In addition, it can be seen that DeCovNet performs poorly because it has been trained from scratch and has a custom deep architecture.

And, it can be seen that the performance of ZhangCNN is significantly improved than later, but it still does not achieve similar performance even when the model is trained for more time.

That is, it is worth noting that the models trained with the existing broad-based augmentation did not achieve significant improvement across evaluation indicators, as COVID-19 and bacterial pneumonia exhibit similar characteristics.

8 shows CT slices represented in an embedding space in a multi-instance learning model according to an embodiment of the present disclosure.

To further demonstrate the benefit of the techniques of this embodiment when capturing overall statistics for a single subject, Figure 8 depicts instances and representative features represented in 2D space. In particular, the aggregated feature map (top left of the figure: blue dots) can capture the features of the key slice (red) ignoring the noisy artifacts of other slices that are well clustered together. This can show that although no explicit label is used for instance search, the model of this embodiment can effectively learn slices useful for patient classification.

In this example, a 2D CNN framework with dual attention module and contrast feature map learning under a multi-instance learning (MIL) framework to discriminate sub-types of COVID-19 and bacterial pneumonia in chest CT. can be applied. The performance of the method of this embodiment can be confirmed in both the CT patch and slice-based data set versions. In addition, ablation experiments can show the benefits of using a large bag size during training and the effect of weight loss for stable learning. Therefore, in this embodiment, it is possible to more accurately distinguish COVID-19 from bacterial pneumonia through multi-instance learning based on a weakly supervised method for testing for COVID-19.

9 is a flowchart illustrating a learning method of a multi-instance learning model according to an embodiment of the present disclosure.

The multi-instance learning method for 3D image analysis according to an embodiment of the present disclosure may be performed by at least one processor in a computing device or a computing network.

Referring to FIG. 1 , the multi-instance learning method according to an embodiment of the present disclosure may be performed by the processor 110 of the multi-instance learning apparatus 100 .

The multi-instance learning apparatus 100 may include a processor 110 and a memory 130 , and a multi-instance learning model and several instructions may be stored in the memory 130 . Although only one memory is disclosed in FIG. 1, the memory in which the multi-instance learning model is stored and the memory in which several instructions are stored may be configured separately.

According to the multi-instance learning method according to the embodiment of the present disclosure, the processor 110 may derive a feature map of each of the 2D instances of the 3D image input to the multi-instance learning model (S100). ). A block or module for deriving such a feature map may include a convolution layer (convolution layer), and may be referred to as a convolution module or block.

The processor 110 may derive a spatial attention map of the instances from the feature map derived from the convolution block (S200). A block or module deriving such a spatial attention map may be referred to as a spatial attention block. The spatial attention map may determine where the part to be emphasized in order to analyze what the present model intends to determine in each instance. However, the parameters of the initial spatial attention block may be randomly determined, but may be optimized through the training process as described above.

Thereafter, the processor 110 may receive the result of combining the feature map and the spatial attention map to derive an attention score for each instance ( S300 ). Here, the synthesis of the specific map and the spatial attention map may be element-wise multiplication. Also, the processor 110 may derive an aggregated embedding for the 3D image by aggregating the embeddings of the instances according to the attention score ( S400 ). These operations may be performed by an instance attention block (or latent attention block). However, the parameters of the initial instance attention block may be randomly determined, but may be optimized through the above-described training process.

The processor 110 may output an analysis result for the 3D image based on the total embedding (S500). Here, as described above, the analysis result may be whether the CT scan image indicates pneumonia caused by COVID-19 or pneumonia caused by another disease, and other diseases or the meaning of the image may be of course. This output may be performed by an output block.

Meanwhile, in the training process of the multi-instance learning model, the input 3D image of the 2D instance may be data labeled as ground-truth. Here, the labeling is performed only on the 3D image at the bag level, and the labeling is not performed on each 2D slice. As mentioned above, even if the 3D images are images labeled as COVID-19 pneumonia, some of the 2D instances may not show any lesions of COVID-19 pneumonia at all.

In order to train the multi-instance learning model, the processor 110 may perform learning such that the overall loss function value of the multi-instance learning model is minimized by using training data including the labeled 3D image ( S600 ).

Here, performing learning means deriving a feature map so that the overall loss function (L) of the multi-instance learning model has a minimum value, deriving a spatial attention map, deriving an attention score, deriving a sum embedding It may mean adjusting parameters set in blocks used in the step.

Meanwhile, the overall loss function may be a combination of a bag level loss function ( _{LB ) for the result of the output block and a contrast loss function (L F )} between the instance embeddings and the sum embedding.

As described above, the present disclosure proposes a technique for automatically extracting which part plays a major role in classification along with an important instance by simultaneously utilizing a spatial attention structure and an instance attention structure.

In addition, the deep learning network is trained so that the embedding representing the instance and the distinction between each instance are well made by performing the learning in the direction of minimizing the contrastive loss function together with the vowel level loss function. When the proposed technique is used, the instances are well selected to improve the collection level classification performance, the variation within the class of the selected instances is reduced, and the features are embedded so that the distance between instances of other classes is increased.

Therefore, better results can be obtained when the final backlevel classification is performed based on the extracted features. In addition, by using the spatial attention structure and the instance-level attention structure in one network, it was qualitatively confirmed that the abnormal lesion site could be more accurately found even with only the image unit label.

The embodiment according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded in a computer-readable medium. In this case, the medium includes a hard disk, a magnetic medium such as a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and DVD, a magneto-optical medium such as a floppy disk, and a ROM. , RAM, flash memory, and the like, hardware devices specially configured to store and execute program instructions.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and used by those skilled in the computer software field. Examples of the computer program may include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

In the specification of the present invention (especially in the claims), the use of the term "above" and similar referential terms may be used in both the singular and the plural. In addition, when a range is described in the present invention, each individual value constituting the range is described in the detailed description of the invention as including the invention to which individual values belonging to the range are applied (unless there is a description to the contrary). same as

The steps constituting the method according to the present invention may be performed in an appropriate order, unless the order is explicitly stated or there is no description to the contrary. The present invention is not necessarily limited to the order in which the steps are described. The use of all examples or exemplary terms (eg, etc.) in the present invention is merely for the purpose of describing the present invention in detail, and the scope of the present invention is limited by the examples or exemplary terms unless defined by the claims. it's not going to be In addition, those skilled in the art will appreciate that various modifications, combinations, and changes may be made in accordance with design conditions and factors within the scope of the appended claims or their equivalents.

Although the embodiments according to the present invention have been described above, these are merely exemplary, and those of ordinary skill in the art will understand that various modifications and equivalent ranges of embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the following claims.

Claims (15)

A multi-instance learning device for analysis of 3D images, comprising:
memory in which multi-instance training models are stored; and
at least one processor electrically connected to the memory;
The multi-instance learning model is
a convolution block for deriving a feature map of each of the 2D instances of the input 3D image;
a spatial attention block for deriving a spatial attention map of the instances from the feature map derived from the convolution block;
The result of combining the feature map and the spatial attention map is input to derive an attention score for each instance, and according to the attention score, the embeddings of instances are synthesized to perform aggregated embedding of the 3D image. deriving instance attention block (Instance Attention Block); and
An output block for outputting an analysis result for the 3D image based on the sum embedding,
Multi-instance learning device.
The method of claim 1,
The processor, in the training phase of the learning model, using training data including a 3D image labeled with a ground-truth for the analysis result, the overall loss function (L) of the learning model has a minimum value. To perform the operation of training the learning model to
wherein the overall loss function is a combination of a Bag Level loss function ( _{LB ) for the result of the output block and a contrast loss function (L F )} between the instance embeddings and the sum embedding,
Multi-instance learning device.
3. The method of claim 2,
The operation of training the multi-instance learning model includes adjusting parameters of the convolution block, the spatial attention block, and the instance attention block so that an overall loss function (L) of the multi-instance learning model has a minimum value. doing,
Multi-instance learning device.
3. The method of claim 2,
The total loss function (L) is expressed by Equation 1 below,
Multi-instance learning device.
Equation 1

is a value between 0 and 1, and is a parameter representing the weight of the loss function at the vowel level.
3. The method of claim 2,
The loss function L _B of the bag level is expressed by Equation 2 below, and the contrast loss function L _F is expressed by Equation 3 below,
Multi-instance learning device.
Equation 2

is the instance label,

is the label of the 3D image

probability to be
Equation 3

is an instance-level characteristic,

is a characteristic of the vowel level,

, has a value of 1 if k is not equal to i, and a value of 0 if k = i,

is the temperature parameter,

is the similarity function, N is the number of instances of the 3D image
A multi-instance learning device for analysis of 3D images, comprising:
a memory storing at least one instruction; and
Comprising at least one processor interlocked with the memory to execute the instruction,
The instructions, when executed by the processor, cause the processor to:
deriving a feature map of each of the 2D instances of the 3D image input to the multi-instance learning model;
deriving a spatial attention map of the instances from the derived feature map;
deriving an attention score for each instance by receiving a synthesis result of the feature map and the spatial attention map;
deriving an aggregated embedding for the 3D image by synthesizing embeddings of instances according to the attention score; and
to perform an operation of outputting an analysis result for the 3D image based on the total embedding,
Multi-instance learning device.
7. The method of claim 6,
The processor, in a training phase of the multi-instance learning model, using training data including a 3D image labeled with a ground-truth for the analysis result, a total loss function (L) of the multi-instance learning model training the multi-instance learning model to have a minimum value,
wherein the overall loss function is a combination of a Bag Level loss function ( _{LB ) for the result of the output block and a contrast loss function (L F )} between the instance embeddings and the sum embedding,
Multi-instance learning device.
8. The method of claim 7,
The training of the multi-instance learning model may include deriving the feature map so that the overall loss function L of the multi-instance learning model has a minimum value, deriving the spatial attention map, and deriving the attention score. Including the operation of adjusting parameters used in the operation of deriving the sum embedding,
Multi-instance learning device.
8. The method of claim 7,
The total loss function (L) is expressed by Equation 1 below,
Multi-instance learning device.
Equation 1

is a value between 0 and 1, and is a parameter representing the weight of the loss function at the vowel level.
8. The method of claim 7,
The loss function L _B of the bag level is expressed by Equation 2 below, and the contrast loss function L _F is expressed by Equation 3 below,
Multi-instance learning device.
Equation 2

is the instance label,

is the label of the 3D image

probability to be
Equation 3

is an instance-level characteristic,

is a characteristic of the vowel level,

, has a value of 1 if k is not equal to i, and a value of 0 if k = i,

is the temperature parameter,

is the similarity function, N is the number of instances of the 3D image
A multi-instance learning method for 3D image analysis performed by at least one processor in a computing device or computing network, comprising:
deriving a feature map of each of the 2D instances of the 3D image input to the multi-instance learning model;
deriving a spatial attention map of the instances from the derived feature map;
deriving an attention score for each instance by receiving a synthesis result of the feature map and the spatial attention map;
deriving an aggregated embedding for the 3D image by synthesizing embeddings of instances according to the attention score; and
Comprising the step of outputting an analysis result for the 3D image based on the total embedding,
Multi-instance learning method.
12. The method of claim 11,
The 3D image is labeled with a ground-truth for the analysis result,
The multi-instance learning method is
In the training phase of the multi-instance learning model, using training data including the labeled 3D image to train the multi-instance learning model so that the overall loss function (L) of the multi-instance learning model has a minimum value including,
wherein the overall loss function is a combination of a Bag Level loss function ( _{LB ) for the result of the output block and a contrast loss function (L F )} between the instance embeddings and the sum embedding,
Multi-instance learning method.
13. The method of claim 12,
The step of training the multi-instance learning model includes deriving the feature map so that the overall loss function L of the multi-instance learning model has a minimum value, deriving the spatial attention map, and deriving the attention score. and adjusting parameters used in the step of deriving the sum embedding.
Multi-instance training method.
13. The method of claim 12,
The total loss function (L) is expressed by Equation 1 below,
Multi-instance training method.
Equation 1

is a value between 0 and 1, and is a parameter representing the weight of the loss function at the vowel level.
13. The method of claim 12,
The loss function L _B of the bag level is expressed by Equation 2 below, and the contrast loss function L _F is expressed by Equation 3 below,
Multi-instance learning method.
Equation 2

is the instance label,

is the label of the 3D image

probability to be
Equation 3

is an instance-level characteristic,

is a characteristic of the vowel level,

, has a value of 1 if k is not equal to i, and a value of 0 if k = i,

is the temperature parameter,

is the similarity function, N is the number of instances of the 3D image

Images