KR102620790B1 - Adaptive Instance Normalization-Based Chest X-Ray Image Segmentation Mehtod and Apparatus Therefor - Google Patents

Abstract

A chest radiology image segmentation method and device based on adaptive instance normalization are disclosed. A chest radiation image segmentation method according to an embodiment of the present invention includes receiving a chest radiation image; and segmenting an organ region in the chest radiology image using a single neural network based on a pre-learned segmentation model with adaptive instance normalization, wherein the step of segmenting the organ region includes the single neural network. Organ regions can be segmented from the lungs of the chest radiology image using a neural network and a pre-built adaptive instance normalization code.

Description

Adaptive Instance Normalization-Based Chest X-Ray Image Segmentation Method and Apparatus Therefor {Adaptive Instance Normalization-Based Chest X-Ray Image Segmentation Mehtod and Apparatus Therefor}

The present invention relates to a chest radiology image segmentation technology based on adaptive instance normalization, and more specifically, to detect abnormalities in chest radiology images by utilizing a neural network based on adaptive instance normalization (AdaIN). This relates to a chest radiography image segmentation method and device capable of segmenting organ regions in lungs and normal lungs, respectively.

High-accuracy image segmentation often serves as the first step in a variety of medical imaging tasks. Recently, deep learning (DL) approaches have become the state-of-the-art (SOTA) technique for medical image segmentation tasks due to their superior performance compared to classical methods.

The performance of DL-based segmentation algorithms usually depends on a large amount of labels, but segmentation masks are sparse due to expensive and time-consuming annotation procedures. Another difficulty in DL-based medical image segmentation is so-called domain shift. In other words, segmentation networks trained with data from a specific domain often experience rapid performance degradation when applied to new test domains (unseen test domains).

For example, in chest radiograph (CXR) lung segmentation for infectious disease diagnosis, segmentation networks trained on normal CXR data often produce under-segmentation when applied to abnormal CXR images due to serious infectious diseases such as bacterial pneumonia. do. The areas missed in under-segmentation mostly contain important features, such as lung consolidation, for diagnosing infectious diseases. Therefore, highly accurate segmentation without under-segmentation for normal and abnormal lung CXR is required to ensure that DL-based classification algorithms fully learn whole lung features while mitigating extraneous factors outside the lung.

To address label sparsity and domain shift problems, there has been extensive research to train segmentation networks using limited training labels or using unlabeled datasets in self-supervised or unsupervised manner. For example, teacher-student architecture and domain adaptation were introduced to take advantage of features learned in supervised learning tasks and distill the knowledge into unsupervised learning tasks. However, these approaches appear different, and there is no consensus on how these different approaches can be combined synergistically.

Embodiments of the present invention utilize a neural network based on adaptive instance normalization (AdaIN) in chest radiology images to segment organ regions from abnormal lungs and normal lungs, respectively. Provides a method and device for segmenting chest radiographic images.

A chest radiation image segmentation method according to an embodiment of the present invention includes receiving a chest radiation image; and segmenting an organ region in the chest radiology image using a single neural network based on a pre-trained segmentation model with adaptive instance normalization.

In the step of segmenting the organ area, the organ area in the lung of the chest radiology image may be segmented using the single neural network and a pre-built adaptive instance normalization code.

The step of dividing the organ region is by changing each adaptive instance normalization code for the encoder and decoder of the single neural network, thereby dividing the organ region in the normal lung based on supervised learning and domain adaptation (Domain adaptation). Self-supervised learning method in which the knowledge learned through domain adaptation and supervised learning and domain adaptation of the organ area by domain conversion from abnormal lung to normal lung based on the adaptation method is distilled. Based on this, the organ area can be divided into normal lung and abnormal lung respectively.

The single neural network learns lung region segmentation based on supervised learning from normal lungs, and then learns a first adaptive instance normalization code that converts abnormal lungs into normal lungs through domain adaptation. And, by distilling the knowledge learned through supervised learning and domain adaptation into a second adaptive instance normalization code and learning with self-supervised learning, domain adaptation and segmentation are integrated. Thus, the organ areas can be divided into normal lungs and abnormal lungs, respectively.

The step of dividing the organ area is performed using the single neural network, based on a supervised learning technique in normal lungs where organ labels exist in units of image pixels. Areas can be divided.

In the step of dividing the organ region, the domain of the abnormal lung is converted to a normal lung using an adaptive instance normalization technique using the single neural network, and the organ region can be divided from the abnormal lung whose domain has been converted to the normal lung.

In the step of dividing the organ region, the organ region may be divided from each of the abnormal lung domain-converted to the normal lung and the labeled normal lung based on self-supervised learning using the single neural network.

A chest radiation image segmentation apparatus according to an embodiment of the present invention includes a receiver for receiving a chest radiation image; and a segmentation unit for segmenting organ regions in the chest radiology image using a single neural network based on a pre-learned segmentation model with adaptive instance normalization.

The segmentation unit may segment an organ region in the lungs of the chest radiology image using the single neural network and a pre-built adaptive instance normalization code.

The division unit changes the adaptive instance normalization code for each encoder and decoder of the single neural network, thereby dividing the organ region in the normal lung based on supervised learning and domain adaptation method. Division of the organ area by domain conversion from abnormal lung to normal lung, and a self-supervised learning method in which knowledge learned through supervised learning and domain adaptation is distilled, normal lung and The organ region can be segmented from each abnormal lung.

The single neural network learns lung region segmentation based on supervised learning from normal lungs, and then learns a first adaptive instance normalization code to convert abnormal lungs into normal lungs through domain adaptation. And, by distilling the knowledge learned through supervised learning and domain adaptation into a second adaptive instance normalization code and learning with self-supervised learning, domain adaptation and segmentation are integrated. Thus, the organ areas can be divided into normal lungs and abnormal lungs, respectively.

The segmentation unit can use the single neural network to segment organ areas in the chest radiology image based on a supervised learning technique in normal lungs where organ labels exist in pixel units. there is.

The segmentation unit may convert the domain of an abnormal lung into a normal lung using an adaptive instance normalization technique using the single neural network, thereby dividing the organ region from the abnormal lung whose domain has been converted into the normal lung.

The segmentation unit may divide the organ region from each of the abnormal lung domain-converted to normal lung and the labeled normal lung based on self-supervised learning using the single neural network.

According to embodiments of the present invention, organ regions can be segmented from abnormal lungs and normal lungs respectively by using a neural network based on adaptive instance normalization in chest radiology images. You can.

According to embodiments of the present invention, generating an organ area label in units of image pixels is costly and time-consuming, and in particular, there is a lack of labels for abnormal lungs, so first, an integration is performed. In the algorithm, lung region segmentation is learned from normal lungs based on supervised learning techniques, and then a first adaptive instance normalization code is learned to convert abnormal lungs into normal lungs through domain adaptation. The knowledge learned through supervised learning and domain adaptation is distilled into another second adaptive instance normalization code, and the normal lung and the domain are converted to the abnormal lung through self-supervised learning. In each case, organs can be divided with excellent performance.

Figure 1 shows an operation flowchart for a chest radiology image segmentation method according to an embodiment of the present invention.
Figure 2 shows an example diagram to explain the integrated domain adaptation and segmentation framework.
Figure 3 shows an example diagram of the architecture of the neural network of the present invention.
Figure 4 shows an example diagram of the architecture of a multi-head discriminator, AdaIN code generator, and style encoder.
Figure 5 shows an example diagram comparing organ segmentation results for abnormal lungs in chest radiography images according to an embodiment of the present invention.
Figure 6 shows the configuration of a chest radiation image segmentation device according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms, but the present embodiments only serve to ensure that the disclosure of the present invention is complete and are within the scope of common knowledge in the technical field to which the present invention pertains. It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used herein, “comprises” and/or “comprising” refers to the presence of one or more other components, steps, operations and/or elements. or does not rule out addition.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the attached drawings. The same reference numerals are used for the same components in the drawings, and duplicate descriptions for the same components are omitted.

Embodiments of the present invention utilize a neural network based on adaptive instance normalization (AdaIN) in chest radiology images to segment organ regions in abnormal lungs and normal lungs, respectively. That is the gist of it.

Here, the present invention proposes an integrated knowledge distillation framework that can provide improved segmentation by synergistically utilizing domain adaptation, self-supervised, and supervised training. Although segmentation masks can only be used on normal data, the trained method can be applied to both normal and abnormal images.

The main idea of the present invention is that a single generator trained with Adaptive Instance Normalization (AdaIN) can perform segmentation on normal data as well as domain adaptation between normal and abnormal domains, simply by changing the AdaIN code. The network can also be trained in a self-supervised manner to ensure that direct segmentation results match other segmentation results through domain adaptation and subsequent segmentation. Then, other AdaIN codes can be used to accurately perform direct segmentation of abnormal images. A single generator is used for all tasks with simple changes to the AdaIN code, allowing the network to synergistically learn common features across all tasks through knowledge distillation.

Before explaining the present invention, existing technology related to the present invention will be described as follows.

Video Style Transfer

The purpose of video style transfer is to convert content video into a specific stylized video. Currently, two types of approaches are most commonly used for video style transfer. First, the content video and the style reference video are transmitted to the neural network, and the content video is converted from a style reference to imitating the style. For example, Adaptive Instance Normalization (AdaIN) has been proposed as a simple but powerful method. Specifically, the AdaIN layer estimates the mean and standard deviation of the referenced style features and uses them to modify the mean and standard deviation of the content features. For example, one embodiment technique is Multimodal Unsupervised Image Interchange (MUNIT), which decomposes an image into content and style codes, and the encoded style code for the desired domain replaces the affine parameter of the AdaIN layer at the decoder.

On the other hand, unsupervised style transfer approaches such as cycleGAN learn a target data distribution rather than a single style image. However, the cycleGAN approach requires N(N-1) generators to enable translation between N domains. To handle this, a multi-domain image transformation approach was proposed. In particular, StarGANv2 (Choi, Y., Uh, Y., Yoo, J., Ha, J.W., 2020. Stargan v2: Diverse image synthesis for multiple domains, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8188-8197.) introduced an advanced single generator-based framework for transferring styles across multiple domains by changing the domain-specific style code in the AdaIN layer.

Semi-supervised learning through domain adaptation

In domain adaptive approaches for medical image segmentation, a DL-based model trained using labeled datasets from a specific domain (e.g., generic CXR) is trained on different domain datasets in a semi-supervised, self-supervised, or unsupervised manner. is refined for. These approaches leverage features learned from supervised learning in a specific domain and attempt to distill that knowledge into unsupervised learning tasks in an unseen domain. Specifically, the technique in one embodiment applies a semi-supervised learning approach to the task of cardiac MR image segmentation and generates pseudo labels for an unlabeled dataset with a pre-trained model and By training the network on a pseudo-labeled dataset, it consistently achieved higher performance compared to a supervised baseline model. Another embodiment of the technique introduces a semi-supervised approach to the task of pneumonia lung segmentation by generating pseudo-images via an image-to-image style transfer framework, followed by another segmentation network using both labeled and pseudo-labeled datasets. trained.

Teacher-Student Approach

The teacher-student framework consists of two separate networks: the student and teacher models. Student models can be trained in a supervised manner as well as self-supervised, which forces the student model output to match the output of a teacher model with an unlabeled dataset.

Specifically, the technology in one embodiment introduces a dual teacher framework for partitioning tasks, which consists of two teacher models: a traditional teacher model for intra-domain knowledge and an additive teacher model for cross-domain knowledge distillation. To utilize the inter-domain dataset with segmentation labels, images from the domain shift dataset are style-transferred as pseudo-images. Additional cross-domain teachers and students trained in a supervised manner over pseudo-labeled images are forced to predict consistent output when considering style transfer images. Therefore, the main goal is to ensure that the student model fully utilizes the information in labeled, unlabeled, and pseudo-labeled cross-domain datasets.

self-directed learning

Recently, new research on self-supervised learning has brought about significant improvements in medical domain adaptation or image segmentation tasks by promoting consistency between model outputs given the same input with different perturbations or by training auxiliary proxy tasks. In general, it is demonstrated that DL models trained with auxiliary self-supervised loss achieve better baseline task performance as well as better generalization capabilities, especially when trained using labeled and rich unlabeled datasets. .

For example, the technique of one embodiment configured an auxiliary edge generation task that leverages an unlabeled domain dataset to achieve SOTA performance for cross-modality domain adaptation with the help of self-supervised learning. Another embodiment of the technique introduced loss of output consistency between the student and teacher models when given unlabeled input with different augmentations.

Figure 1 shows an operation flowchart for a chest radiology image segmentation method according to an embodiment of the present invention.

Referring to FIG. 1, the chest radiology image segmentation method according to an embodiment of the present invention receives a chest radiology image to be segmented, and uses a single neural segment based on a pre-learned segmentation model with adaptive instance normalization. Organ regions are segmented from chest radiology images using a network (S110, S120).

Here, step S120 can segment the organ region in the lungs of the chest radiology image using a single neural network and a pre-built adaptive instance normalization code.

Here, step S120 changes the adaptive instance normalization code for each encoder and decoder of a single neural network, thereby dividing the organ region in the normal lung based on supervised learning and domain adaptation method. Normal lungs and abnormal lungs are divided into organ regions by domain conversion from abnormal lungs to normal lungs and are based on a self-supervised learning method that distills the knowledge learned through supervised learning and domain adaptation. Organ regions can be segmented from each lung.

At this time, a single neural network learns lung region segmentation based on a supervised learning technique from normal lungs, and then uses a first adaptive instance normalization code that converts abnormal lungs into normal lungs through domain adaptation. Domain adaptation and segmentation by learning through self-supervised learning by distilling the knowledge learned through supervised learning and domain adaptation into a second adaptive instance normalization code. By integrating, it is possible to segment organ areas in normal lungs and abnormal lungs, respectively.

Here, step S120 uses a single neural network to segment the organ area in the chest radiology image based on a supervised learning technique in normal lungs where organ labels exist in pixel units. You can.

Here, in step S120, the domain of the abnormal lung is converted to a normal lung using an adaptive instance normalization technique using a single neural network, thereby dividing the organ region from the abnormal lung that has been domain converted to a normal lung.

Here, step S120 is based on self-supervised learning using a single neural network, and the organ region can be divided from each of the abnormal lungs domain-converted to normal lungs and the labeled normal lungs.

The method of the present invention will be described in detail with reference to FIGS. 2 to 5 as follows.

One of the unique features of the StarGANv2 image transformation approach is that it can fully utilize all training datasets by synergizing common features across multiple domains through a shared network layer, but still uses different AdaIN codes to enable domain-specific This means that adaptation is possible. The network of the present invention can integrate domain adaptation and segmentation through AdaIN-based knowledge distillation.

Specifically, the framework of the present invention classifies the training data into three groups: segmentation mask (MASK), matched input image domain (INTRA), and segmentation unlabeled domain moving input image (INTER). More specifically, as shown in Figure 2, training data in the INTRA domain matches segmentation masks in the MASK domain, while many training datasets in the INTER domain do not have segmentation masks.

Due to the domain shift between the INTRA and INTER domains, a supervised network trained using INTRA data does not generalize well to INTER domain images. To alleviate this problem, the present invention allows a single generator trained with Adaptive Instance Normalization (AdaIN) to not only perform supervised segmentation using INTRA data for a specific AdaIN code, as shown in Figure 2A. Rather, as shown in Figure 2b, domain adaptation between the INTER domain and the INTRA domain can be performed using a different AdaIN code. The network is then trained in a self-supervised manner that forces the direct segmentation results to match other segmentation results through domain adaptation post-segmentation, as shown in Figure 2c, so direct segmentation using INTER images uses another AadIN code. It can be done by doing this. This allows knowledge to be distilled across different tasks.

Once the network is trained, only a single generator and pre-built AdaIN code can be used in the inference step for INTRA and INTER domain image segmentation.

As shown in Figure 3, the overall architecture of the neural network of the present invention will be composed of a single generator G, a style encoder S, AdaIN code generators (F _e and F _d ) of each encoder and decoder, and a multi-head discriminator D. You can. A single generator is connected to two AdaIN code generators and a style encoder, and can be divided into an encoder and a decoder consisting of a series of residual blocks, and the output code generated from the AdaIN code generators or style encoder is the output code of each residual block. Can be connected to the AdaIN layer. One of the reasons for introducing the style encoder S is to place additional constraints on the AdaIN code generation so that the AdaIN code generator can be normalized to generate meaningful code. Here, the style encoder, AdaIN code generator and discriminator may have a multi-head structure.

One of the main differences between our network and StarGANv2 is the encoder-decoder architecture with independent AdaIN code layers for the encoder and decoder. Because of the two separate AdaIN codes for the encoder and decoder, the generator in the present invention can perform supervised segmentation, domain adaptation, and self-map by simply changing the AdaIN code combination of the two AdaIN code generators, as shown in Table 1 below. Learning can be done.

For example, for segmentation operations, the constant AdaIN code (0, 1) is used for both encoder and decoder, leading to standard instance normalization. Meanwhile, the learnable AdaIN code is used in the decoder AdaIN layer in the conversion to INTRA and INTER domains. However, another learnable AdaIN code can be used in the encoder AdaIN layer for a self-supervised learning task to enforce consistent reconstruction across various paths.

In particular, if _X , Y, and _Z refer to INTRA, _INTER , and MASK domains related to probability distributions P

<Equation 1>

Here, u refers to the _input image of X or _Y , and a∈{ ^a _seg , a ^' _seg , _a Depending on the input data and AdaIN code, the network output v can be one of X, Y, or Z.

In the present invention, the training loss extends from that of StarGANv2 and can include segmentation tasks and self-supervised learning.

split map

Figure 2a shows a map segmentation that can be considered a conversion from INTRA to MASK domain, and the generator can be learned as shown in Equation 2 below.

<Equation 2>

Here, λ _seg and λ _style mean hyperparameters, and the segmentation loss l _seg is defined by the cross-entropy loss between the generated output and the matching label, and the segmentation loss l _seg is defined by the equation below: 3> It can be expressed as follows.

<Equation 3>

Here, z _i refers to the ith pixel of the ground truth segmentation mask z ∈ Z with respect to the input image x ∈ may refer to the softmax probability function of the ith pixel in the fake image generated by G(x, a _seg ).

Once a segmentation result for a given AdaIN style code is generated, the style encoder with the segmentation result as input is forced to match the dummy AdaIN code a' _seg generated by the style encoder in the MASK domain. This can be achieved using the style loss in <Equation 4> below.

<Equation 4>

Here, a' _seg may mean the AdaIN code for each MASK domain. Although this code is not used directly for segmentation, generating this AdaIN code from a style code generator or style code generator can be important for training shared layers in the multi-head structure of the AdaIN code generator and style encoder.

domain adaptation

Domain adaptive training is basically similar to StarGANv2. Figure 2b shows a domain adaptation training method between

<Equation 5>

Here, the domain adaptation loss (l _da ) can be expressed as <Equation> below.

Here, λ _cycle , λ _style , and λ _div mean hyperparameters, and l _adv means adversarial loss. The adversarial loss can be expressed as <Equation 6> below.

<Equation 6>

Here, the domains S and T refer to the source and target domains, and are randomly selected from All combinations of the source domain and the other domain can be considered. Additionally, the learnable part of the AdaIN code a ^τ _da is generated from the encoder AdaIN code generator F _d or the style encoder S(x) given the reference target t ∈ T, as shown in Figures 4b and 4c, so that the header can also be optimized. there is. This technology can follow the procedures used in StarGANv2.

The present invention can use similar random selection of the source domain and target domain to define the cycle consistency loss l _cycle as shown in Equation 7 below.

<Equation 7>

Similar to the cycle consistency loss lcycle in the video, the style loss _lstyle of Equation 4 above can be introduced to strengthen cycle consistency in the AdaIN code domain. More specifically, when a fake image is generated using a domain-specific AdaIN code, a style encoder using the fake image as input must reproduce the original AdaIN code. This can be achieved by using a style loss as shown in Equation 8 below.

<Equation 8>

Lastly, to make the generated fake images diverse, we need to maximize the difference between two fake images generated with two different AdaIN codes. This can be achieved by maximizing the loss as shown in Equation 9 below.

<Equation 9>

Here, a ^τ _da and a' ^τ _da can be generated randomly from the AdaIN code generator or from a style encoder S given two different images.

self-directed learning

The goal of self-supervised learning is that images in the INTER domain can directly generate segmentation results, as illustrated by the red arrow in Figure 2c.

In particular, the INTER domain does not have a segmentation mask, so the knowledge learned from supervised learning and domain adaptation must be distilled. Therefore, the contribution of the present invention comes from introducing new constraints. Segmentation outputs trained in a self-supervised manner should match other segmentation outputs generated through domain adaptation, illustrated by green and black arrows in Figure 2c, respectively. Additionally, in the inference stage, it is often difficult to know which domain the input came from. Therefore, one AdaIN code must process both INTRA and INTER domain image segmentation. This results in self-consistency loss as shown in Equation 10 below.

<Equation 10>

Here, l _inter and l _intra may mean inter-domain autocorrelation loss and inter-domain autocorrelation loss, respectively, and λ _inter and λ _intra may mean hyperparameters for inter-domain and inter-domain, respectively.

In fact, this procedure can be considered a teacher-student approach. The indirect path in Figure 2c is a teacher network that guides the training procedure of the student network in the direct path in Figure 2c. However, in contrast to existing teacher-student approaches, our approach does not require separate networks for teachers and students. Instead, the same generator with different AdaIN combinations acts as teacher or student, which is another great advantage of the method of the present invention.

To train a single-generator framework for self-supervised learning tasks without sacrificing supervised segmentation performance, we introduce an AdaIN code generator F _e in the encoder, while the decoder is a fixed generator that still corresponds to standard instance normalization as listed in Table 1 above. AdaIN codes (1, 0) can be used.

network architecture

Generator G of the present invention may be composed of an encoder and a decoder module, as shown in FIG. 3. In particular, the encoder part consists of 4 downsampling residual blocks and 2 middle residual blocks, and the decoder part consists of 2 middle residual blocks and 4 upsampling residual blocks. Each residual block consists of an AdaIN layer, an activation layer, and a convolution layer, respectively. All AdaIN layers are connected to the AdaIN code generator, and in the case of the decoder module, a style encoder is also connected for domain adaptation tasks.

Similar to StarGANv2, the discriminator D of the present invention is composed of a multi-head non-shared convolution layer for each image domain, that is, INTER and INTRA, as shown in Figure 4a. In the discriminator, the input image can be classified as 1 or 0 for each domain individually, where 1 can mean a real image and 0 can mean a fake image. The AdaIN code generator consists of a shared linear layer followed by a multi-head non-shared linear layer for each domain, as shown in Figure 4b. Therefore, the AdaIN code for each domain can be generated through a shared layer, respectively, and then through a non-shared layer for each domain. Additionally, the style encoder consists of a shared convolutional layer followed by a multi-head non-shared linear layer for each domain, as shown in Figure 4c. In the style encoder, the input video can be decoded into a domain-specific AdaIN code for each domain through a shared layer and a non-shared layer.

Thanks to the existence of a shared layer, the knowledge of a specific domain translation can be distilled to other domains, improving the overall performance of the discriminator.

dataset

The public CXR dataset can be utilized to evaluate the performance of the domain adaptation and segmentation tasks of the present invention. For supervised segmentation tasks, regular CXR images from the JSRT dataset with paired lung annotations from the SCR dataset can be used as INTRA and MASK domains, respectively. For abnormal CXR images, that is, INTER domain, pneumonia CXR datasets can be collected from four sources, including RSNA COVID19 dataset, Cohen pneumonia dataset, BRIXIA COVID-19 dataset, and BIMCV dataset, and the characteristics of each are Table 2 As shown in .

Implementation details

The network of the present invention can be trained by feeding input images from two randomly selected pairs of domains, one for the source domain and the other for the target domain. For example, if you feed the network a domain pair consisting of an INTER domain and an INTRA domain, the network is trained for the domain adaptation task. Once the network is fed with a domain pair consisting of INTRA as the source and MASK as the target domain, the network is trained for the supervised segmentation task. In the case of self-supervised learning, an image can be supplied from INTRA to the MASK domain as a source domain for outputting not only the MASK domain from INTRA but also the segmentation mask of the MASK domain. In terms of training sequence, self-supervised training starts after the supervised segmentation and domain adaptation tasks are performed.

The method of the present invention can be implemented with the PyTorch library, and the Adam optimizer can be applied to train the model and set the batch size to 1. This model can be trained using an NVIDIA GeForce GTX 1080Ti GPU, the hyperparameters can be selected as λcycle = 2, λstyle = 1, λdiv = 1, λseg = 5, λinter = 10, λintra = 1, and the learning rate is can be optimized to 0.0001.

All input CXR images and labels can be resized to 256Х256, and no preprocessing or data augmentation can be performed except normalizing pixel intensities to [-1,0,1.0]. The network can be trained for 20K iterations to simultaneously train domain adaptation and supervised segmentation tasks, and an early stopping strategy can be adopted based on the validation performance of the supervised segmentation task. After training until the segmentation and domain adaptation tasks guarantee a certain performance, the network can continue to be trained in a self-supervised manner for additional 5K iterations, and once the training iterations reach a certain fixed iteration point across all iterations, the learning rate can be reduced by a factor of 10.

For the domain adaptation task, MUNIT and the original StarGANv2 can be used as reference models for comparative studies, and the model can be trained with INTRA and INTER domain images with the same experimental settings as that of the model of the present invention. For the segmentation task, the present invention uses U-Net as a reference and can be trained with the same experimental settings as the settings for the segmentation task. XLSor can be added as an additional baseline to compare the synergy of domain adaptation and segmentation operations. XLSor can be tested utilizing pre-trained weights, which can be trained in a semi-supervised manner using a pseudo-labeled pneumonia dataset generated at scale via MUNIT.

In the inference phase, as a post-processing step of the segmentation task, the two largest contours can be automatically selected based on the contour area and all holes within each contour can be filled, and the same post-processing steps will be applied to all methods to ensure a fair comparison. You can.

For CXR datasets, the segmentation performance relative to new normal CXR data with a ground-truth segmentation mask can be quantified using the Dice similarity score (Dice) index of both lung contours. Meanwhile, due to the lack of ground truth (or ground truth) labels, the domain adaptation and self-supervised segmentation performance of the INTER domain can be evaluated based on the generation of predicted lung structures covered by highly hardened regions, completely covering each lung structure. Rectangular boxes can be manually drawn on the CXR image, and then the goodness of fit between the generated fake images and the segmentation results can be qualitatively assessed.

Figure 5 shows an example diagram comparing the results of unsupervised segmentation using an external test dataset. The external test dataset consisting of two COVID-19 pneumonia databases has various intensity distributions, as shown in Figure 5a. And, as shown in Figure 5b, U-Net mostly fails to segment the normal lung shape from pneumonia lung, and shows the intensity distribution of the source domain. And, as shown in Figure 5c, XLSor shows excellent segmentation performance in mild opaque pneumonia cases (4th and 12th columns), but for severe pneumonia cases, XLSor does not segment lung regions or generalizes to domain-shifted inputs. (1st, 5th and 10th columns). The method of the present invention shows the best segmentation performance in a new dataset. It can be seen that the method of the present invention shown in FIGS. 5D to 5E successfully segments the lung structure from the domain shift input without under-segmentation or severe over-segmentation. Moreover, it can be seen that the segmentation results generated by our model trained with self-consistency loss show a better fit to the expected lung area compared to our method without self-supervised learning. That is, as can be seen from Figure 5, the qualitative performance of the model of the present invention for organ segmentation in abnormal lung images shows that it shows significantly superior lung segmentation performance compared to other segmentation algorithms.

<Table 3> below shows the segmentation performance of the general CXR dataset, and compares the map segmentation performance for general CXR images using the network of the present invention and other networks. As can be seen from the Dice index of the lung segmentation results in Table 3, compared with U-Net and XLSor with a Dice index of 0.976, which is the current SOTA performance of deep learning-based normal lung segmentation, our method shows excellent performance for abnormal CXR. It can be seen that it not only provides , but also shows similar Dice indices. In other words, it can be confirmed that the method of the present invention produces performance comparable to that of the model currently considered the best in the industry.

As such, the chest radiology image segmentation method according to embodiments of the present invention utilizes a neural network based on adaptive instance normalization in the chest radiology image to separate abnormal lungs and normal lungs. In the lungs, each organ region can be divided.

Generating organ area labels in units of image pixels is costly and time-consuming, and in particular, labels are insufficient for abnormal lungs, so the chest radiography image segmentation method according to embodiments of the present invention The first adaptive instance normalization code learns lung region segmentation based on supervised learning from normal lungs in one integrated algorithm and then converts abnormal lungs into normal lungs through domain adaptation. Learning, and distilling the knowledge learned through supervised learning and domain adaptation into another second adaptive instance normalization code, normal lung and domain through self-supervised learning. Organs can be segmented with excellent performance from each converted abnormal lung.

FIG. 6 shows the configuration of a chest radiation image segmentation device according to an embodiment of the present invention, and shows the conceptual configuration of the device that performs the method of FIGS. 1 to 5.

Referring to FIG. 6, the chest radiation image segmentation apparatus 600 according to an embodiment of the present invention includes a receiving unit 610 and a dividing unit 620.

The receiver 610 receives a chest radiation image for segmenting an organ region.

The segmentation unit 620 divides organ regions in the chest radiology image using a single neural network based on a pre-learned segmentation model with adaptive instance normalization.

Here, the segmentation unit 620 may segment the organ region in the lungs of the chest radiology image using a single neural network and a pre-built adaptive instance normalization code.

Here, the segmentation unit 620 changes the adaptive instance normalization code for each encoder and decoder of a single neural network, thereby performing domain adaptation and division of the organ region in the normal lung based on supervised learning. Division of organ areas by domain conversion from abnormal lungs to normal lungs based on method and self-supervised learning method based on distillation of knowledge learned through supervised learning and domain adaptation. Organ regions can be segmented from lungs and abnormal lungs respectively.

At this time, a single neural network learns lung region segmentation based on a supervised learning technique from normal lungs, and then uses a first adaptive instance normalization code that converts abnormal lungs into normal lungs through domain adaptation. Domain adaptation and segmentation by learning through self-supervised learning by distilling the knowledge learned through supervised learning and domain adaptation into a second adaptive instance normalization code. By integrating, it is possible to segment organ areas in normal lungs and abnormal lungs, respectively.

Here, the segmentation unit 620 uses a single neural network to determine the organ area in the chest radiology image based on a supervised learning technique in normal lungs where organ labels exist in units of image pixels. can be divided.

Here, the segmentation unit 620 converts the domain of the abnormal lung into the normal lung using an adaptive instance normalization technique using a single neural network, thereby dividing the organ region from the abnormal lung that has been domain-converted into the normal lung.

Here, the segmentation unit 620 can segment the organ regions from each of the abnormal lungs that have been domain converted to normal lungs and the labeled normal lungs based on self-supervised learning using a single neural network.

Even if the description is omitted in the device of FIG. 6, each component constituting FIG. 6 may include all the contents described in FIGS. 1 to 5, and this is obvious to those skilled in the art.

The device described above may be implemented with hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general-purpose or special-purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may execute an operating system (OS) and one or more software applications that run on the operating system. Additionally, a processing device may access, store, manipulate, process, and generate data in response to the execution of software. For ease of understanding, a single processing device may be described as being used; however, those skilled in the art will understand that a processing device includes multiple processing elements and/or multiple types of processing elements. It can be seen that it may include. For example, a processing device may include a plurality of processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are possible.

Software may include a computer program, code, instructions, or a combination of one or more of these, which may configure a processing unit to operate as desired, or may be processed independently or collectively. You can command the device. Software and/or data may be used on any type of machine, component, physical device, virtual equipment, computer storage medium or device to be interpreted by or to provide instructions or data to a processing device. It can be embodied in . Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the medium may be specially designed and configured for the embodiment or may be known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -Includes optical media (magneto-optical media) and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, etc. Examples of program instructions include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

As described above, although the embodiments have been described with limited examples and drawings, various modifications and variations can be made by those skilled in the art from the above description. For example, the described techniques are performed in a different order than the described method, and/or components of the described system, structure, device, circuit, etc. are combined or combined in a different form than the described method, or other components are used. Alternatively, appropriate results may be achieved even if substituted or substituted by an equivalent.

Therefore, other implementations, other embodiments, and equivalents of the claims also fall within the scope of the claims described below.

Claims (14)

Receiving a chest radiology image; and
Segmenting an organ region in the chest radiology image using a single neural network based on a pre-trained segmentation model with adaptive instance normalization.
Including,
The step of dividing the organ area is
A chest radiology image segmentation method, characterized in that the organ region is segmented from each of the abnormal lungs domain-converted to normal lungs and the labeled normal lungs based on self-supervised learning using the single neural network.
According to paragraph 1,
The step of dividing the organ area is
A chest radiology image segmentation method, characterized in that segmenting an organ region in the lungs of the chest radiology image using the single neural network and a pre-built adaptive instance normalization code.
Receiving a chest radiology image; and
Segmenting an organ region in the chest radiology image using a single neural network based on a pre-trained segmentation model with adaptive instance normalization.
Including,
The step of dividing the organ area is
By changing each adaptive instance normalization code for the encoder and decoder of the single neural network, segmentation of the organ region in normal lung based on supervised learning and abnormal lung based on domain adaptation method are performed. Normal lung and abnormal lung respectively based on a self-supervised learning method that distills the knowledge learned through division and supervised learning and domain adaptation of the organ area by domain conversion to normal lung. A chest radiography image segmentation method, characterized in that segmenting the organ region.
Receiving a chest radiology image; and
Segmenting an organ region in the chest radiology image using a single neural network based on a pre-trained segmentation model with adaptive instance normalization.
Including,
The single neural network is
After learning lung region segmentation from normal lungs based on supervised learning techniques, a first adaptive instance normalization code is learned to convert abnormal lungs into normal lungs through domain adaptation, and supervised learning and Knowledge learned through the domain adaptation method is distilled into a second adaptive instance normalization code and learned using a self-supervised learning method, integrating domain adaptation and segmentation into normal lung and abnormal lung. A chest radiography image segmentation method characterized by segmenting the organ region in each lung.
According to paragraph 1,
The step of dividing the organ area is
Characterized by segmenting the organ area in the chest radiology image based on a supervised learning technique in normal lungs where organ labels exist in pixel units using the single neural network. Chest radiography image segmentation method.
According to paragraph 1,
The step of dividing the organ area is
A chest radiology image segmentation method comprising converting the domain of an abnormal lung into a normal lung using an adaptive instance normalization technique using the single neural network, thereby segmenting the organ region from the abnormal lung whose domain has been converted to the normal lung.
delete A receiving unit that receives a chest radiography image; and
A segmentation unit that segments organ regions in the chest radiology image using a single neural network based on a pre-trained segmentation model with adaptive instance normalization.
Including,
The division part is
A chest radiology image segmentation device, characterized in that the organ region is segmented from each of the abnormal lungs domain-converted to normal lungs and the labeled normal lungs based on self-supervised learning using the single neural network.
According to clause 8,
The division part is
A chest radiology image segmentation device, characterized in that segmenting an organ region in the lungs of the chest radiology image using the single neural network and a pre-built adaptive instance normalization code.
A receiving unit that receives a chest radiography image; and
A segmentation unit that segments organ regions in the chest radiology image using a single neural network based on a pre-trained segmentation model with adaptive instance normalization.
Including,
The division part is
By changing each adaptive instance normalization code for the encoder and decoder of the single neural network, segmentation of the organ region in normal lung based on supervised learning and abnormal lung based on domain adaptation method are performed. Normal lung and abnormal lung respectively based on a self-supervised learning method that distills the knowledge learned through division and supervised learning and domain adaptation of the organ area by domain conversion to normal lung. A chest radiography image segmentation device, characterized in that segmenting the organ region.
A receiving unit that receives a chest radiography image; and
A segmentation unit that segments organ regions in the chest radiology image using a single neural network based on a pre-trained segmentation model with adaptive instance normalization.
Including,
The single neural network is
After learning lung region segmentation from normal lungs based on supervised learning techniques, a first adaptive instance normalization code is learned to convert abnormal lungs into normal lungs through domain adaptation, and supervised learning and Knowledge learned through the domain adaptation method is distilled into a second adaptive instance normalization code and learned using a self-supervised learning method, integrating domain adaptation and segmentation into normal lung and abnormal lung. A chest radiography image segmentation device, characterized in that segmenting the organ region in each lung.
According to clause 8,
The division part is
Characterized by segmenting the organ area in the chest radiology image based on a supervised learning technique in normal lungs where organ labels exist in pixel units using the single neural network. Chest radiography image segmentation device.
According to clause 8,
The division part is
A chest radiology image segmentation device that converts the domain of an abnormal lung into a normal lung using an adaptive instance normalization technique using the single neural network, thereby segmenting the organ region from the abnormal lung whose domain has been converted to the normal lung.
delete