KR102522734B1 - Transfer Learning method and apparatus based on self-supervised learning for 3D MRI analysis - Google Patents

Abstract

Disclosed are a self-learning-based transfer learning method and apparatus for analyzing 3D medical images. A self-learning-based transfer learning framework device for 3D medical image analysis divides a 3D medical image into three planes, expresses them as 2D sequential data for each plane, and then learns pre-representation to integrate them into 3D spatial feature information. (representation learning) a central model; and a prediction module configured of a fully connected layer and performing prediction on a target task based on the 3D spatial feature information, wherein the pretrained central module and the prediction module may be fine-tuned according to the target task. there is.

Description

Transfer learning method and apparatus based on self-supervised learning for 3D MRI analysis for 3D medical image analysis

The present invention relates to a self-learning-based transfer learning method and apparatus for analyzing 3D medical images.

Transfer learning refers to a learning technique that reuses a trained model in a data-rich field to build a model that lacks training data. This encompasses the concept of transferring a model learned for a specific task to another task to learn the corresponding model ex post facto.

Transfer learning reduces the training time of deep learning models, which take a relatively long learning time, and shows high prediction performance with only a small number of training samples, so research is being actively conducted in various fields of deep learning.

Recently, transfer learning is attracting attention in the field of high-dimensional medical image analysis with few labeled samples, and through this, it is actively used for various medical image analysis-related tasks such as image classification and segmentation.

Most of these transfer learning frameworks for medical image analysis utilize 3D Convolutional Neural Network (CNN) networks after pre-learning through supervised learning or self-supervised learning methods.

However, the supervised learning-based pre-learning method has a problem in that it requires a large number of labeled training datasets for learning. In addition, 3D convolutional neural network-based methods have limitations in actually learning high-dimensional medical image datasets due to the large number of parameters required for model learning.

An object of the present invention is to provide a self-learning-based transfer learning method and apparatus capable of deriving high performance with only a small number of training samples in the field of 3D medical image analysis where labeled training datasets are insufficient.

In addition, the present invention divides a 3D medical image into three planes, expresses them as 2D sequential data for each plane, and integrates them into 3D spatial features for representation learning, thereby providing 3D spatial features. It is to provide a self-learning-based transfer learning method and apparatus capable of efficiently reducing learning parameters as well as effectively learning.

According to one aspect of the present invention, an apparatus for self-learning-based transfer learning for 3D medical image analysis is provided.

According to an embodiment of the present invention, a 3D medical image is divided into three planes, and each plane is expressed as 2D sequential data, and then integrated into 3D spatial feature information to perform pre-representation learning; It consists of a fully connected layer and includes a prediction module that performs prediction on a target task based on the 3D spatial feature information, wherein the pretrained central module and the prediction module are fine-tuned according to the target task. A characterized self-learning-based transfer learning framework device may be provided.

The central axis module may include a 2D convolutional neural network module trained by a self-learning method to extract an encoding vector for a 2D slice image for each plane; and a transformer that performs expression learning to combine the encoding vectors and integrate them into the 3D spatial feature information, wherein the 2D convolutional neural network module may be pre-learned based on a triplet loss function.

The transformer may be trained to randomly mask a portion of a 2D slice image and predict an encoding vector of the masked slice image using an encoding vector of an unmasked slice image.

When learning the transformer, the 2D convolutional neural network module may be frozen so as not to be learned.

The pretrained central module and the prediction module are fine-tuned to each target task in an end-to-end manner, and the target task may be at least one of brain disease diagnosis, brain age prediction, and brain tumor segmentation.

The prediction module, according to the target task, a single-scale technique using a high-level encoding vector of the 2D convolutional neural network module and a low-level (low level) of the 2D convolutional neural network module -level) Any one of multi-scale techniques for integrating encoding vectors may be applied.

According to another aspect of the present invention, a self-learning-based transfer learning method for analyzing 3D medical images is provided.

According to an embodiment of the present invention, pre-representation learning is performed on a model to divide a 3D medical image into three planes, express them as 2D sequential data for each plane, and integrate them into 3D spatial feature information. ; and fine-tuning the model according to the target task to perform prediction on the target task based on the 3D spatial feature information.

By providing a self-learning-based transfer learning method and apparatus for analyzing 3D medical images according to an embodiment of the present invention, high performance is obtained with only a small number of training samples in the field of 3D medical image analysis lacking labeled training datasets. can do.

In addition, the present invention divides a 3D medical image into three planes, expresses them as 2D sequential data for each plane, and integrates them into 3D spatial features for representation learning, thereby providing 3D spatial features. It not only learns effectively, but also has the advantage of efficiently reducing the learning parameters.

1 is a block diagram schematically showing the internal configuration of a self-learning-based transfer learning framework device according to an embodiment of the present invention.
Figure 2 is a view showing the internal configuration of the heavy duty module according to an embodiment of the present invention.
3 is a diagram for explaining triplet loss-based learning of a two-dimensional convolutional neural network module according to an embodiment of the present invention.
4 is a diagram for explaining fine adjustment according to an embodiment of the present invention;
5 is a flowchart illustrating a self-learning-based transfer learning method according to an embodiment of the present invention.
6 is a diagram showing the overall flow of a self-learning-based transfer learning framework according to an embodiment of the present invention.
Figure 7 is a comparison of brain tumor segmentation results of the prior art and the present invention.

Singular expressions used herein include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or some of the steps It should be construed that it may not be included, or may further include additional components or steps. In addition, terms such as "...unit" and "module" described in the specification mean a unit that processes at least one function or operation, which may be implemented as hardware or software or a combination of hardware and software. .

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram schematically showing the internal configuration of a self-learning-based transfer learning framework device according to an embodiment of the present invention, and FIG. 2 is a block diagram showing the internal configuration of a centralized module according to an embodiment of the present invention. 3 is a diagram for explaining triplet loss-based learning of a two-dimensional convolutional neural network module according to an embodiment of the present invention, and FIG. 4 is a diagram for explaining fine tuning according to an embodiment of the present invention. It is a drawing shown for

Referring to FIG. 1, a self-learning-based transfer learning framework apparatus 100 according to an embodiment of the present invention includes a centralization module 110, a prediction module 120, a memory 130 and a processor 140, It consists of

According to an embodiment of the present invention, the self-learning-based transfer learning framework apparatus 100 may include a centralized module 110 pre-trained by an SSL proxy task and a fully-connected layer-based prediction module 120 for final prediction. can

The central axis module 110 may extract relational features by modeling inter-slice dependencies by dividing the 3D medical image into planes and then extracting high-level spatial features of the 2D slices for each plane.

Since a 3D medical image is composed of three planes, axial, coronal, and sagittal, each plane having a different view, the central axis module 110 may use an independent convolutional encoder for each plane that does not share parameters. there is.

Before the spatial features extracted by the convolutional encoder 212 are supplied to the transformer 220, position encoding for specifying a slice order and segment encoding for an identification plane may be performed.

After passing through the transformer 220, a 3D spatial feature map (information) is formed as a combination of features of each slice for all planes, and the prediction module 120 predicts according to the target task using the corresponding 3D spatial feature map. can be performed.

It will be described in more detail below.

According to an embodiment of the present invention, the central axis module 110 divides the 3D medical image into three planes, expresses them as 2D sequential data for each plane, and then integrates them into 3D spatial feature information. expression is learned.

When the pre-learning of the central module 110 is completed, the pre-trained central module 110 and the prediction module 120 may be fine-tuned to be suitable for the target task.

Prediction module 120 consists of a fully connected layer, but the pre-trained central module 110 and the prediction module can be fine-tuned to suit the target task in an end-to-end manner.

The internal structure of the central axis module 110 is shown in FIG. 2 . As shown in FIG. 2 , the central axis module 110 includes a 2D convolutional neural network module 210 and a transformer 220 .

The 2D convolutional neural network module 210 divides the 3D medical image into three planes and then extracts slice encoding vectors as spatial features of the 2D slice image generated for each plane.

The transformer 220 is a means for combining slice encoding vectors corresponding to each plane into 3D spatial feature information.

라 가정하면, 각 평면의 2D 슬라이스 영상의 집합을

로 나타낼 수 있다.

이다.

는 상응하는 평면의 슬라이스 개수를 나타내고, w, d, h는 3차원 의료 영상의 너비, 깊이, 및 높이를 각각 나타낸다. For example, 3D medical images

Assuming that, a set of 2D slice images in each plane

can be expressed as

am.

represents the number of slices of the corresponding plane, and w, d, and h represent the width, depth, and height of the 3D medical image, respectively.

를 추출할 수 있다. 여기서,

는 임베딩 벡터의 차원을 나타낸다. The 2D slice image is passed to the 2D convolutional neural network module 210, and the 2D convolutional neural network module 210 embedding vectors for the slice image for each plane.

can be extracted. here,

represents the dimension of the embedding vector.

를 생성할 수 있으며, 해당 인코딩 벡터가 트랜스포머(220)로 전달될 수 있다. Subsequently, the 2D convolutional neural network module 210 performs position and segment encoding on the embedding vectors for the slice images of each plane, respectively, to encode the encoding vectors.

can be generated, and the corresponding encoding vector can be passed to the transformer 220.

The central axis module 110 according to an embodiment of the present invention may be pre-learned. Hereinafter, the pre-learning process of the central axis module 110 will be first described.

The central axis module 110 may be pre-learned based on self-learning twice.

This will be described in more detail with reference to FIG. 3 .

The 2D convolutional neural network module 210 may be trained by a self-learning method so that the 2D convolutional neural network module 210 extracts a spatially meaningful embedding vector.

)이 계산될 수 있다. The two-dimensional convolutional neural network module 210 includes a convolutional encoder 212, and the convolutional encoder 212 may be trained based on a triplet loss function. For each plane, all slice-by-slice features are averaged and compared to positive pairs (transformed features) and negative pairs (sample features from different batches), resulting in triplet loss (

) can be calculated.

는 슬라이스에 대해 평균을 낸 랜덤 훈련 샘플의 임베딩 벡터를 나타내고,

는 트랜스폼된 버전(포지티브 예제)을 나타내고,

는 데이터세트로부터의 다른 샘플(네거티브 예제)을 나타내며,

는 훈련된 샘플의 개수를 나타내고,

는 포지티브 샘플과 네거티브 샘플 사이의 마진을 나타내며, D는 거리 측정을 위한 함수로 예를 들어, L2거리일 수 있다. 최종 손실은 세 평면의 모든 손실을 평균하여 정의될 수 있다. here,

denotes the embedding vector of random training samples averaged over slices,

denotes the transformed version (positive example),

represents another sample (negative example) from the dataset,

represents the number of trained samples,

Represents a margin between positive and negative samples, and D is a function for measuring a distance, and may be, for example, an L2 distance. The final loss can be defined by averaging all losses in the three planes.

를 토큰으로 간주하고, 이미지 수준 특징 집합인 인코딩 벡터의 집합

을 문장으로 간주할 수 있다. After the two-dimensional slice space feature (embedding vector) for each plane is extracted by the convolutional encoder 212, position encoding for specifying the slice order and segment encoding for the identification plane are performed, and the encoding vector as an output value is converted to a transformer ( 220). Slice Unit Encoding Vector

as a token, and a set of encoding vectors, which are image-level feature sets.

can be regarded as a sentence.

A masked encoding vector predicting an encoder vector masked arbitrarily after the training of the two-dimensional convolutional neural network module 210 is performed based on the triplet loss function, and then the learned convolutional neural network module 210 is frozen so that it is not learned. The transformer 220 may perform learning through the prediction task.

In this way, after the 2D convolutional neural network module 210 is pre-trained, the transformer 220 may be trained.

During pre-learning of the transformer 220 according to an embodiment of the present invention, some slice images for each plane are randomly masked, and an encoding vector of the masked slice image is predicted from encoding vectors for the remaining unmasked slice images The transformer 220 may be learned to do so.

At this time, since the 2D convolutional neural network module 210 pre-trained in the previous step is fixed and only the transformer 220 is trained to predict the masked encoding vector, the transformer 220 determines the inter-slice dependency and determines the masked encoding vector. can predict

라고 가정하기로 한다. For example, slice-by-slice encoding vectors

Let's assume that

The transformer 220 may model relationships (dependencies) between slices through a concentration mechanism.

If this is expressed as an equation, it can be expressed as in Equation 2.

,

,

}는 학습 가능한 가중치 행렬의 집합을 나타내고,

와

는 두개의 위치 단위 피드포워드 합성곱(point-wise feed-forward convolutions)을 나타낸다. here, {

,

,

} denotes a set of learnable weight matrices,

and

denotes two point-wise feed-forward convolutions.

A loss function for masked encoding vector prediction can be expressed as Equation 3.

는 마스킹된 슬라이스들의 인덱스 세트를 나타낸다. here,

denotes a set of indices of masked slices.

In this way, after the pre-learning of the central module 110 is completed, the pre-learned central module 110 and the prediction module 120 may be fine-tuned.

As already mentioned, the pre-trained weighting module 110 and prediction module 120 can be fine-tuned according to the target task end-to-end.

This will be described in more detail with reference to FIG. 4 .

The target task may be any one of brain disease diagnosis, brain age prediction, and brain tumor segmentation.

That is, the pretrained central axis module 110 and the prediction module 120 may be differently fine-tuned according to any one of brain disease diagnosis, brain age prediction, and brain tumor segmentation.

Depending on the target task, a multi-scale approach to capture both low-level and high-level features and a single-scale approach to capture high-level features for classification and regression tasks can be taken. .

The overall flow for pre-learning and fine-tuning is as shown in FIG. 4 .

The memory 130 stores various commands for the self-learning-based transfer learning method according to an embodiment of the present invention.

The processor 140 includes internal components (eg, the central module 110, the prediction module 120, the memory 130) of the self-learning-based transfer learning framework device 100 according to an embodiment of the present invention. etc.) as a means to control.

In addition, the processor 140 controls the central module 110 to be pre-learned, and controls the pre-trained central module 110 and the prediction module 120 to be fine-tuned according to the target task.

5 is a flowchart showing a self-learning-based transfer learning method according to an embodiment of the present invention, and FIG. 6 is a diagram showing the overall flow of a self-learning-based transfer learning framework according to an embodiment of the present invention. 7 is a diagram comparing the brain tumor segmentation results of the prior art and the present invention.

In step 510, the self-learning-based transfer learning framework apparatus 100 pre-trains the backbone network.

As shown in FIG. 6 , the backbone network includes a central axis module 110 and includes a 2D convolutional neural network module 210 and a transformer 220 . The 2D convolutional neural network module 210 is pre-trained to extract high-level spatial features of 2D slices for each plane of the 3D medical image and extract relational features by modeling dependencies between slices through the transformer 220. can

Since a 3D medical image is composed of three planes, axial, coronal and sagittal, each having a different view, an independent 2D convolutional neural network module 210 is used for each plane that does not share parameters.

Spatial features may be transmitted to the transformer 220 after passing through position encoding for order designation of slices and segment encoding for an identification plane before being transferred to the transformer 220 .

The transformer 220 can be taught to capture dependencies between adjacent slices and far slices through an autoattention mechanism. That is, the transformer 220 may learn to predict encoding vectors of some masked slices by using encoding vectors of non-masked slices.

As already described above, during pre-learning of the transformer 220, the 2D convolutional neural network module 210 may be frozen and not learned.

In step 515, the self-learning-based transfer learning framework device 100 fine-tunes the pretrained central module 110 and the prediction module 120 according to the target task.

For example, the target task may be brain disease diagnosis, brain age prediction, and brain tumor segmentation.

The pretrained central axis module 110 and the prediction module 120 adopt a multi-scale approach using both low-level and high-level features and high-level features according to the target task. A single scale approach may be used.

Accordingly, the pre-trained centralization module 110 and the prediction module 120 can be fine-tuned in an end-to-end manner according to the target task.

In this way, when the fine adjustment is completed, the 3D medical image is divided into each plane through the central axis module 110, high-level spatial features of the 2D slice for each plane are extracted, and 3D spatial feature information is formed by combining them. can do.

Next, the prediction module 120 may predict the target task using the 3D spatial feature information.

7 compares brain tumor segmentation results for 3D medical images according to the prior art and an embodiment of the present invention. It can be seen that the brain tumor segmentation results of the present invention are similar to those of 3D-SSL, and tumor types are best distinguished compared to the conventional ones.

Devices and methods according to embodiments of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in computer readable media. Computer readable media may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on a computer readable medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art in the field of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - Includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media and ROM, RAM, flash memory, etc. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been looked at mainly by its embodiments. Those skilled in the art to which the present invention pertains will be able to understand that the present invention can be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from a descriptive point of view rather than a limiting point of view. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the equivalent scope will be construed as being included in the present invention.

100: Self-learning based transfer learning framework device.
110: central axis module
120: prediction module
130: memory
140: processor

Claims (7)

A central axis module for pre-representation learning to divide the 3D medical image into three planes, express them as 2D sequential data for each plane, and integrate them into 3D spatial feature information;
It consists of a fully connected layer and includes a prediction module that performs prediction on a target task based on the 3D spatial feature information,
The central axis module may include a 2D convolutional neural network module trained by a self-learning method to extract an encoding vector for a 2D slice image for each plane; and
A transformer for pre-representation learning to combine the encoding vectors and integrate them into the 3-dimensional spatial feature information, wherein the 2-dimensional convolutional neural network module is pre-learned based on a triplet loss function,
The pre-representation learned condensation module and the prediction module are fine-tuned to suit each target task in an end-to-end manner,
Each of the target tasks is a self-learning-based transfer learning framework device, characterized in that at least one of brain disease diagnosis, brain age prediction, and brain tumor segmentation.
delete According to claim 1,
The transformer randomly masks a part of the 2D slice image and then learns to predict the encoding vector of the masked slice image using the encoding vector of the unmasked slice. Self-learning-based transfer learning framework Device.
According to claim 3,
Self-learning-based transfer learning framework device, characterized in that, when learning the transformer, the two-dimensional convolutional neural network module is frozen so as not to be learned.
delete According to claim 1,
The prediction module,
A single-scale technique using a high-level encoding vector of the 2D convolutional neural network module and a low-level encoding vector of the 2D convolutional neural network module according to the target task A self-learning-based transfer learning framework device, characterized in that any one of the multi-scale techniques for integrating them is applied.
delete

Images